2022-07-03T09:58:34.615 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-03T09:58:34.616 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/ffsb}} --name rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847 --owner scheduled_rishabh@teuthology --verbose -- /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847/orig.config.yaml 2022-07-03T09:58:34.681 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-03T09:58:34.776 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847 branch: wip-rishabh-testing-20220702-1631 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/ffsb}} email: rishabhddave@gmail.com,ridave@redhat.com first_in_suite: false job_id: '6910847' kernel: client: branch: testing flavor: default kdb: true sha1: 1c8086aad75988b46f5ad0acdbe9e00b8086e530 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-rishabh-testing-20220702-1631 ceph: cephfs: session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.dir_split install: ceph: flavor: default sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20220702-1631 sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d owner: scheduled_rishabh@teuthology priority: 200 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d sleep_before_teardown: 0 suite: fs suite_branch: wip-rishabh-testing-20220702-1631 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFv4EiKKl146MaeH0JesX5rTY7IWeGzuKVNnXzU7qZBGJTNJMW8l5x1aRDjr4Cp1DskhQWW8ClVSgZDQVJP6ct/qRMxmOiRGMgNxKFOHD0Zb1jSeQaaOjPhsx+t8FTtws8Y76Mvle5ahJD67wWb0N65zDeLkkK0pLiHihE5ETul+jXn5iRNqSl3LReGXQtMv+SXi8084NU6Szl4KT1O+TaQt5yT9KhMsHd34iFyBjyQNRk9fEYFt7Eu02uwwIyFtA7EpB9B0NqWJ0w4086wtn5RoCe734WkCzFlqxbpYxChqiz7CwP2CllYmcjc5fuT8p+FqvU4XYnNBtf26121lmxvG/pGsW9lBK6IpzdRjzPWzP3LTGkApJGXZFAAJ2rcx0QrzjgvXXxwT5uJIVGL1YiB4HiaSLPSDUrT/soOatH4SJGVPZIbCEeyNKI4jASrwmWfdQ2xU5JAuPQp/4LIwHbTfYpfdeu3d5Ot0MWoU6j40Igapwp9R1paEfuIrt7Kn0= smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXBdn5CyL5FwcV5JyvPrE2JU41c7SPRdK/TRAfFKrt23fvfZEPyRsMTMQmX1wVHutvev/lH7Zf7L702XbhJOPoHia3p9+XOgJwsBAtDQnaYwqDHxl8XFOJurwoT0fmaMhTwrYGFi8gjDB6OSzG87ZDduOtjVuwtUn0auPNF/S/pBzrpvmM6XNKaghOS4J90AbTFekrJDiofzvmYcrgzCgmI5NEaAlUIuBlqwxRRSekHzzNP0q170JJiqYlGh9M+x1l6hnWqtRElShkP+pdunlTtVbbjNNbIbbs1pCcHohTdwTvCIhLm1wbrnrHlDvEvYv6yveeA56UsLu/Hjylk2fm5BqsK9P/7MbZMbPxhpEpwtRvzcGbp4eQ32hreU/5sP4CBmFPnLK8ZNXg6zXEyDwmIjof7wyriwez901Wfmn3nN5AmM15uBqWgCKb1t84MOXmCMcc4UFIc5RVEY9fvF5rCvIyHddD2fn8P2tAaLlJaii/XKnGXGgmout3nBL78YE= smithi188.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA2eWna7XGeRV5RLmb1Sb9XDT/Vqp33IjbjG/fUls5JeROvKMe3JXu1f6oAusJQq92u2pbHBDf+BU2/5DQHQmQPdpGkFweliuoi1ISgT3i7eCwt44g+wNPJvwb5Cy3FNJR0PQYBZfmcEk9XKONPtqLh7giH+cE/BDFWS8Ov8SIF1ORmmmLXVjq+BIVN/EmSAW5olm89N6GA5+RJRHqvPSdhGMBACc6KYpOWg9c4hK49/UBKh4R/ZN9l09x3/UDvXpmE3ijZk/NNcO/W4BnqOLjoIESZfHFgf/C6DQxOCWcPWJh949XnNXNYWf1XcFEtvms5ams6NJ++1oRdlA5QxNTwogdzjQEFedV7iAVEcqjpG6mlFzSS68iWFLpUC3G1ihC9D+fvbyql0xoaMSmsglhBK6/zy31qDVyoRKTYC83ubi0fmBKXaY7bxyRZBst89mh9nD57X1b6dJzsVvMJylE+ME1s9YBVOGHG9e5s7wmGFhD6eMO7ykUzS6aU85hMf0= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - check-counter: {} - workunit: clients: all: - suites/ffsb.sh teuthology_branch: main teuthology_sha1: f58709e3fda190343157063e5edb0d7a36a41afc timestamp: 2022-07-02_14:14:52 tube: smithi user: rishabh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-03T09:58:34.776 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa; will attempt to use it 2022-07-03T09:58:34.777 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks 2022-07-03T09:58:34.778 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-03T09:58:34.780 INFO:teuthology.task.internal:Checking packages... 2022-07-03T09:58:34.801 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '61829bae536fb4126cfaa87ca353b9576e018e7d' 2022-07-03T09:58:34.801 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-03T09:58:34.802 INFO:teuthology.packaging:ref: None 2022-07-03T09:58:34.802 INFO:teuthology.packaging:tag: None 2022-07-03T09:58:34.803 INFO:teuthology.packaging:branch: wip-rishabh-testing-20220702-1631 2022-07-03T09:58:34.803 INFO:teuthology.packaging:sha1: 61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T09:58:34.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-testing-20220702-1631 2022-07-03T09:58:34.966 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-13322.g61829bae 2022-07-03T09:58:34.967 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-03T09:58:34.974 INFO:teuthology.task.internal:no buildpackages task found 2022-07-03T09:58:34.975 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-03T09:58:34.982 INFO:teuthology.task.internal:Saving configuration 2022-07-03T09:58:35.006 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-03T09:58:35.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-03T09:58:35.038 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-03 09:53:25.174500', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1mb5C1REBnp85GNfY0F9FQiHhejxx6IJwW7koAZwYqy27nyY6tPEnB8dRrn9vb//L+eW1EovyFUVnJFb4PLhM='} 2022-07-03T09:58:35.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-03 09:53:25.173421', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSqCscaEZnfXLIPgPsPkuBsHSnFS0Qowvb74X7tsX+U6e5KuD/k1JxmLMT8GjZPlFPHxki10/enCzb/J5t4aMw='} 2022-07-03T09:58:35.096 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910847', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-03 09:53:25.171962', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMoVwFXcfiDzydmY/77ub/iB5pefOgMnhoA/xTbLZWzI95gWi+iUFmS6Wd59cVlkHiK52jSGTBt3HotWVdO8MdQ='} 2022-07-03T09:58:35.096 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-03T09:58:35.103 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-07-03T09:58:35.103 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-07-03T09:58:35.104 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-07-03T09:58:35.104 INFO:teuthology.run_tasks:Running task console_log... 2022-07-03T09:58:35.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3bd1432598>, signals=[15]) 2022-07-03T09:58:35.230 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-03T09:58:35.236 INFO:teuthology.task.internal:Opening connections... 2022-07-03T09:58:35.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi101.front.sepia.ceph.com 2022-07-03T09:58:35.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T09:58:35.325 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2022-07-03T09:58:35.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T09:58:35.410 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2022-07-03T09:58:35.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T09:58:35.498 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-03T09:58:35.507 DEBUG:teuthology.orchestra.run.smithi101:> uname -m 2022-07-03T09:58:35.528 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2022-07-03T09:58:35.529 DEBUG:teuthology.orchestra.run.smithi101:> cat /etc/os-release 2022-07-03T09:58:35.588 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="Red Hat Enterprise Linux" 2022-07-03T09:58:35.589 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8.6 (Ootpa)" 2022-07-03T09:58:35.589 INFO:teuthology.orchestra.run.smithi101.stdout:ID="rhel" 2022-07-03T09:58:35.589 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="fedora" 2022-07-03T09:58:35.590 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8.6" 2022-07-03T09:58:35.590 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8" 2022-07-03T09:58:35.591 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-07-03T09:58:35.591 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2022-07-03T09:58:35.591 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-07-03T09:58:35.591 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://www.redhat.com/" 2022-07-03T09:58:35.591 INFO:teuthology.orchestra.run.smithi101.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-07-03T09:58:35.592 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-03T09:58:35.592 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T09:58:35.592 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-03T09:58:35.592 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-07-03T09:58:35.593 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-07-03T09:58:35.593 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-07-03T09:58:35.594 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2022-07-03T09:58:35.616 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2022-07-03T09:58:35.639 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2022-07-03T09:58:35.639 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2022-07-03T09:58:35.699 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Red Hat Enterprise Linux" 2022-07-03T09:58:35.699 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8.6 (Ootpa)" 2022-07-03T09:58:35.699 INFO:teuthology.orchestra.run.smithi161.stdout:ID="rhel" 2022-07-03T09:58:35.700 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="fedora" 2022-07-03T09:58:35.700 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8.6" 2022-07-03T09:58:35.701 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2022-07-03T09:58:35.701 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-07-03T09:58:35.701 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2022-07-03T09:58:35.702 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-07-03T09:58:35.702 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.redhat.com/" 2022-07-03T09:58:35.702 INFO:teuthology.orchestra.run.smithi161.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-07-03T09:58:35.703 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-03T09:58:35.703 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T09:58:35.703 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-03T09:58:35.704 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-07-03T09:58:35.705 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-07-03T09:58:35.705 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-07-03T09:58:35.706 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-07-03T09:58:35.729 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2022-07-03T09:58:35.751 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2022-07-03T09:58:35.752 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2022-07-03T09:58:35.810 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2022-07-03T09:58:35.810 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.6 (Ootpa)" 2022-07-03T09:58:35.810 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2022-07-03T09:58:35.811 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2022-07-03T09:58:35.811 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.6" 2022-07-03T09:58:35.811 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2022-07-03T09:58:35.811 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-07-03T09:58:35.812 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2022-07-03T09:58:35.812 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-07-03T09:58:35.812 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2022-07-03T09:58:35.812 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-07-03T09:58:35.813 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-03T09:58:35.813 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T09:58:35.813 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-03T09:58:35.813 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-07-03T09:58:35.813 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-07-03T09:58:35.814 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-07-03T09:58:35.814 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2022-07-03T09:58:35.837 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-03T09:58:35.848 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-03T09:58:35.854 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-03T09:58:35.854 DEBUG:teuthology.orchestra.run.smithi101:> test '!' -e /home/ubuntu/cephtest 2022-07-03T09:58:35.857 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2022-07-03T09:58:35.860 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2022-07-03T09:58:35.878 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-03T09:58:35.886 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-03T09:58:35.887 DEBUG:teuthology.orchestra.run.smithi101:> test -z $(ls -A /var/lib/ceph) 2022-07-03T09:58:35.917 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2022-07-03T09:58:35.920 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2022-07-03T09:58:35.950 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-03T09:58:36.038 INFO:teuthology.run_tasks:Running task kernel... 2022-07-03T09:58:36.054 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1c8086aad75988b46f5ad0acdbe9e00b8086e530'} 2022-07-03T09:58:36.055 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-07-03T09:58:36.056 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-07-03T09:58:36.056 DEBUG:teuthology.orchestra.run.smithi101:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-03T09:58:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T09:58:36.077 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2022-07-03T09:58:36.175 INFO:teuthology.orchestra.run.smithi101.stdout:RedHatEnterprise 2022-07-03T09:58:36.177 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T09:58:36.178 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-07-03T09:58:36.756 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1c8086aad75988b46f5ad0acdbe9e00b8086e530 2022-07-03T09:58:36.757 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1c8086aad75988b46f5ad0acdbe9e00b8086e530"... 2022-07-03T09:58:36.757 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2022-07-03T09:58:36.778 INFO:teuthology.orchestra.run.smithi101.stdout:4.18.0-372.9.1.el8.x86_64 2022-07-03T09:58:36.778 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 1c8086aad75988b46f5ad0acdbe9e00b8086e530 2022-07-03T09:58:36.779 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-07-03T09:58:36.781 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum upgrade -y linux-firmware 2022-07-03T09:58:37.798 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2022-07-03T09:58:37.798 INFO:teuthology.orchestra.run.smithi101.stdout:Unable to read consumer identity 2022-07-03T09:58:37.893 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T09:58:37.893 INFO:teuthology.orchestra.run.smithi101.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-07-03T09:58:37.893 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T09:58:41.728 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-07-03T09:58:42.871 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 11 MB/s | 12 MB 00:01 2022-07-03T09:58:45.508 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 571 kB/s | 24 kB 00:00 2022-07-03T09:58:46.183 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2022-07-03T09:58:46.184 INFO:teuthology.orchestra.run.smithi101.stdout:Nothing to do. 2022-07-03T09:58:46.184 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2022-07-03T09:58:46.275 INFO:teuthology.task.kernel:Downloading kernel 1c8086aad75988b46f5ad0acdbe9e00b8086e530 on client.0... 2022-07-03T09:58:46.276 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2022-07-03T09:58:46.317 INFO:teuthology.orchestra.run.smithi101.stdout:RedHatEnterprise 2022-07-03T09:58:46.318 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T09:58:46.318 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c8086aad75988b46f5ad0acdbe9e00b8086e530 2022-07-03T09:58:46.463 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/1c8086aad75988b46f5ad0acdbe9e00b8086e530/centos/8/flavors/default/x86_64/ 2022-07-03T09:58:46.463 DEBUG:teuthology.orchestra.run.smithi101:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.19.0_rc4_ceph_g1c8086aad759-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/1c8086aad75988b46f5ad0acdbe9e00b8086e530/centos/8/flavors/default/x86_64/ --input-file=- 2022-07-03T09:58:46.466 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi101.front.sepia.ceph.com to complete... 2022-07-03T09:59:03.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03 09:59:03 URL:https://2.chacra.ceph.com/r/kernel/testing/1c8086aad75988b46f5ad0acdbe9e00b8086e530/centos/8/flavors/default/x86_64/kernel-5.19.0_rc4_ceph_g1c8086aad759-1.x86_64.rpm [514252248/514252248] -> "/tmp/kernel.x86_64.rpm" [1] 2022-07-03T09:59:03.638 INFO:teuthology.orchestra.run.smithi101.stderr:FINISHED --2022-07-03 09:59:03-- 2022-07-03T09:59:03.638 INFO:teuthology.orchestra.run.smithi101.stderr:Total wall clock time: 17s 2022-07-03T09:59:03.638 INFO:teuthology.orchestra.run.smithi101.stderr:Downloaded: 1 files, 490M in 17s (28.9 MB/s) 2022-07-03T09:59:03.640 INFO:teuthology.task.kernel:Installing kernel 1c8086aad75988b46f5ad0acdbe9e00b8086e530 on client.0... 2022-07-03T09:59:03.640 DEBUG:teuthology.orchestra.run.smithi101:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-07-03T09:59:05.724 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying... ######################################## 2022-07-03T09:59:05.834 INFO:teuthology.orchestra.run.smithi101.stdout:Preparing... ######################################## 2022-07-03T09:59:05.925 INFO:teuthology.orchestra.run.smithi101.stdout:Updating / installing... 2022-07-03T09:59:44.471 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-5.19.0_rc4_ceph_g1c8086aad759-1######################################## 2022-07-03T10:00:08.406 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi101.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-07-03T10:00:08.407 DEBUG:teuthology.orchestra.run.smithi101:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-07-03T10:00:08.489 INFO:teuthology.orchestra.run.smithi101.stdout:/boot/System.map-5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.490 INFO:teuthology.orchestra.run.smithi101.stdout:/boot/config-5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.490 INFO:teuthology.orchestra.run.smithi101.stdout:/boot/vmlinux-5.19.0-rc4-ceph-g1c8086aad759.bz2 2022-07-03T10:00:08.490 INFO:teuthology.orchestra.run.smithi101.stdout:/boot/vmlinuz-5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.490 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.490 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel 2022-07-03T10:00:08.491 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch 2022-07-03T10:00:08.491 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86 2022-07-03T10:00:08.491 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto 2022-07-03T10:00:08.491 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-07-03T10:00:08.492 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-07-03T10:00:08.492 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-07-03T10:00:08.492 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-07-03T10:00:08.492 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-07-03T10:00:08.492 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-07-03T10:00:08.493 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-07-03T10:00:08.493 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/crc32c-intel.ko 2022-07-03T10:00:08.493 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-07-03T10:00:08.493 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-07-03T10:00:08.494 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-07-03T10:00:08.494 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-07-03T10:00:08.494 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/serpent-avx2.ko 2022-07-03T10:00:08.494 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-07-03T10:00:08.494 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-07-03T10:00:08.495 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-07-03T10:00:08.495 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-07-03T10:00:08.495 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-07-03T10:00:08.495 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-07-03T10:00:08.495 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-07-03T10:00:08.496 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kernel 2022-07-03T10:00:08.496 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kernel/cpu 2022-07-03T10:00:08.496 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kernel/cpu/mce 2022-07-03T10:00:08.496 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-07-03T10:00:08.497 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kvm 2022-07-03T10:00:08.497 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kvm/kvm-amd.ko 2022-07-03T10:00:08.497 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kvm/kvm-intel.ko 2022-07-03T10:00:08.497 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/arch/x86/kvm/kvm.ko 2022-07-03T10:00:08.497 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto 2022-07-03T10:00:08.498 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/ansi_cprng.ko 2022-07-03T10:00:08.498 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/anubis.ko 2022-07-03T10:00:08.498 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/arc4.ko 2022-07-03T10:00:08.498 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx 2022-07-03T10:00:08.499 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/async_memcpy.ko 2022-07-03T10:00:08.499 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/async_pq.ko 2022-07-03T10:00:08.499 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/async_raid6_recov.ko 2022-07-03T10:00:08.499 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/async_tx.ko 2022-07-03T10:00:08.499 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/async_xor.ko 2022-07-03T10:00:08.500 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/async_tx/raid6test.ko 2022-07-03T10:00:08.500 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/authenc.ko 2022-07-03T10:00:08.500 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/authencesn.ko 2022-07-03T10:00:08.500 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/blake2b_generic.ko 2022-07-03T10:00:08.501 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/blowfish_common.ko 2022-07-03T10:00:08.501 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/blowfish_generic.ko 2022-07-03T10:00:08.501 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/camellia_generic.ko 2022-07-03T10:00:08.501 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/cast5_generic.ko 2022-07-03T10:00:08.501 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/cast6_generic.ko 2022-07-03T10:00:08.502 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/cast_common.ko 2022-07-03T10:00:08.502 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/ccm.ko 2022-07-03T10:00:08.502 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/cmac.ko 2022-07-03T10:00:08.502 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/crc32_generic.ko 2022-07-03T10:00:08.503 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/crypto_engine.ko 2022-07-03T10:00:08.503 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/crypto_user.ko 2022-07-03T10:00:08.503 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/cts.ko 2022-07-03T10:00:08.503 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/des_generic.ko 2022-07-03T10:00:08.504 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/dh_generic.ko 2022-07-03T10:00:08.504 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/echainiv.ko 2022-07-03T10:00:08.504 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/essiv.ko 2022-07-03T10:00:08.504 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/fcrypt.ko 2022-07-03T10:00:08.504 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/khazad.ko 2022-07-03T10:00:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/lz4.ko 2022-07-03T10:00:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/lz4hc.ko 2022-07-03T10:00:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/md4.ko 2022-07-03T10:00:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/michael_mic.ko 2022-07-03T10:00:08.505 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/pcbc.ko 2022-07-03T10:00:08.506 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/pcrypt.ko 2022-07-03T10:00:08.506 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/rmd160.ko 2022-07-03T10:00:08.506 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/seed.ko 2022-07-03T10:00:08.506 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/serpent_generic.ko 2022-07-03T10:00:08.507 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/tcrypt.ko 2022-07-03T10:00:08.507 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/tea.ko 2022-07-03T10:00:08.507 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/twofish_common.ko 2022-07-03T10:00:08.507 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/twofish_generic.ko 2022-07-03T10:00:08.507 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/vmac.ko 2022-07-03T10:00:08.508 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/wp512.ko 2022-07-03T10:00:08.508 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/xcbc.ko 2022-07-03T10:00:08.508 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/xor.ko 2022-07-03T10:00:08.508 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/crypto/xxhash_generic.ko 2022-07-03T10:00:08.508 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers 2022-07-03T10:00:08.509 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi 2022-07-03T10:00:08.509 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/acpi_ipmi.ko 2022-07-03T10:00:08.509 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/acpi_pad.ko 2022-07-03T10:00:08.509 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/custom_method.ko 2022-07-03T10:00:08.509 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/ec_sys.ko 2022-07-03T10:00:08.510 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/platform_profile.ko 2022-07-03T10:00:08.510 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/sbs.ko 2022-07-03T10:00:08.510 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/sbshc.ko 2022-07-03T10:00:08.510 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/acpi/video.ko 2022-07-03T10:00:08.511 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata 2022-07-03T10:00:08.511 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/acard-ahci.ko 2022-07-03T10:00:08.511 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/ahci_platform.ko 2022-07-03T10:00:08.511 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/ata_generic.ko 2022-07-03T10:00:08.511 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/libahci_platform.ko 2022-07-03T10:00:08.512 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_acpi.ko 2022-07-03T10:00:08.512 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_ali.ko 2022-07-03T10:00:08.512 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_amd.ko 2022-07-03T10:00:08.512 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_artop.ko 2022-07-03T10:00:08.512 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_atiixp.ko 2022-07-03T10:00:08.513 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_atp867x.ko 2022-07-03T10:00:08.513 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_cmd640.ko 2022-07-03T10:00:08.513 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_cmd64x.ko 2022-07-03T10:00:08.513 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_cypress.ko 2022-07-03T10:00:08.513 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_efar.ko 2022-07-03T10:00:08.514 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_hpt366.ko 2022-07-03T10:00:08.514 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_hpt37x.ko 2022-07-03T10:00:08.514 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_hpt3x2n.ko 2022-07-03T10:00:08.514 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_hpt3x3.ko 2022-07-03T10:00:08.514 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_it8213.ko 2022-07-03T10:00:08.515 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_it821x.ko 2022-07-03T10:00:08.515 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_jmicron.ko 2022-07-03T10:00:08.515 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_marvell.ko 2022-07-03T10:00:08.515 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_mpiix.ko 2022-07-03T10:00:08.515 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_netcell.ko 2022-07-03T10:00:08.516 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_ninja32.ko 2022-07-03T10:00:08.516 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_ns87410.ko 2022-07-03T10:00:08.516 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_ns87415.ko 2022-07-03T10:00:08.516 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_oldpiix.ko 2022-07-03T10:00:08.517 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_opti.ko 2022-07-03T10:00:08.517 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_optidma.ko 2022-07-03T10:00:08.517 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_pcmcia.ko 2022-07-03T10:00:08.517 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_pdc2027x.ko 2022-07-03T10:00:08.517 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-07-03T10:00:08.518 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_piccolo.ko 2022-07-03T10:00:08.518 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_rdc.ko 2022-07-03T10:00:08.518 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_sch.ko 2022-07-03T10:00:08.518 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_serverworks.ko 2022-07-03T10:00:08.519 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_sil680.ko 2022-07-03T10:00:08.519 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_sis.ko 2022-07-03T10:00:08.519 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_sl82c105.ko 2022-07-03T10:00:08.519 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_triflex.ko 2022-07-03T10:00:08.519 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pata_via.ko 2022-07-03T10:00:08.520 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/pdc_adma.ko 2022-07-03T10:00:08.520 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_inic162x.ko 2022-07-03T10:00:08.520 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_mv.ko 2022-07-03T10:00:08.520 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_nv.ko 2022-07-03T10:00:08.520 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_promise.ko 2022-07-03T10:00:08.521 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_qstor.ko 2022-07-03T10:00:08.521 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_sil.ko 2022-07-03T10:00:08.521 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_sil24.ko 2022-07-03T10:00:08.521 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_sis.ko 2022-07-03T10:00:08.521 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_svw.ko 2022-07-03T10:00:08.522 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_sx4.ko 2022-07-03T10:00:08.522 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_uli.ko 2022-07-03T10:00:08.522 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_via.ko 2022-07-03T10:00:08.522 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ata/sata_vsc.ko 2022-07-03T10:00:08.522 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/auxdisplay 2022-07-03T10:00:08.523 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/auxdisplay/ks0108.ko 2022-07-03T10:00:08.523 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/base 2022-07-03T10:00:08.523 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/base/regmap 2022-07-03T10:00:08.523 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/base/regmap/regmap-i2c.ko 2022-07-03T10:00:08.523 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/bcma 2022-07-03T10:00:08.524 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/bcma/bcma.ko 2022-07-03T10:00:08.524 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block 2022-07-03T10:00:08.524 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/aoe 2022-07-03T10:00:08.524 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/aoe/aoe.ko 2022-07-03T10:00:08.524 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/brd.ko 2022-07-03T10:00:08.525 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/drbd 2022-07-03T10:00:08.525 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/drbd/drbd.ko 2022-07-03T10:00:08.525 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/floppy.ko 2022-07-03T10:00:08.525 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/loop.ko 2022-07-03T10:00:08.526 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/mtip32xx 2022-07-03T10:00:08.526 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-07-03T10:00:08.526 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/nbd.ko 2022-07-03T10:00:08.526 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/null_blk 2022-07-03T10:00:08.526 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/null_blk/null_blk.ko 2022-07-03T10:00:08.527 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/pktcdvd.ko 2022-07-03T10:00:08.527 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/rbd.ko 2022-07-03T10:00:08.527 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/sx8.ko 2022-07-03T10:00:08.527 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/virtio_blk.ko 2022-07-03T10:00:08.527 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/xen-blkback 2022-07-03T10:00:08.528 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-07-03T10:00:08.528 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/xen-blkfront.ko 2022-07-03T10:00:08.528 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/zram 2022-07-03T10:00:08.528 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/block/zram/zram.ko 2022-07-03T10:00:08.528 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char 2022-07-03T10:00:08.529 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hangcheck-timer.ko 2022-07-03T10:00:08.529 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random 2022-07-03T10:00:08.529 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random/amd-rng.ko 2022-07-03T10:00:08.529 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random/intel-rng.ko 2022-07-03T10:00:08.530 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-07-03T10:00:08.530 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random/via-rng.ko 2022-07-03T10:00:08.530 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/hw_random/virtio-rng.ko 2022-07-03T10:00:08.530 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi 2022-07-03T10:00:08.530 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-07-03T10:00:08.531 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-07-03T10:00:08.531 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-07-03T10:00:08.531 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_si.ko 2022-07-03T10:00:08.531 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-07-03T10:00:08.531 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-07-03T10:00:08.532 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/lp.ko 2022-07-03T10:00:08.532 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/mwave 2022-07-03T10:00:08.532 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/mwave/mwave.ko 2022-07-03T10:00:08.532 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/pcmcia 2022-07-03T10:00:08.532 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-07-03T10:00:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-07-03T10:00:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/ppdev.ko 2022-07-03T10:00:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tlclk.ko 2022-07-03T10:00:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm 2022-07-03T10:00:08.533 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm.ko 2022-07-03T10:00:08.534 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm_atmel.ko 2022-07-03T10:00:08.534 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm_infineon.ko 2022-07-03T10:00:08.534 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm_nsc.ko 2022-07-03T10:00:08.534 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm_tis.ko 2022-07-03T10:00:08.535 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-07-03T10:00:08.535 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/uv_mmtimer.ko 2022-07-03T10:00:08.535 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/char/virtio_console.ko 2022-07-03T10:00:08.535 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq 2022-07-03T10:00:08.535 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-07-03T10:00:08.536 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-07-03T10:00:08.536 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/p4-clockmod.ko 2022-07-03T10:00:08.536 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-07-03T10:00:08.536 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/powernow-k8.ko 2022-07-03T10:00:08.536 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/cpufreq/speedstep-lib.ko 2022-07-03T10:00:08.537 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto 2022-07-03T10:00:08.537 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/ccp 2022-07-03T10:00:08.537 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-07-03T10:00:08.537 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/ccp/ccp.ko 2022-07-03T10:00:08.537 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/padlock-aes.ko 2022-07-03T10:00:08.538 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/padlock-sha.ko 2022-07-03T10:00:08.538 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/qat 2022-07-03T10:00:08.538 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/qat/qat_common 2022-07-03T10:00:08.538 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-07-03T10:00:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/qat/qat_dh895xcc 2022-07-03T10:00:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-07-03T10:00:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/virtio 2022-07-03T10:00:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-07-03T10:00:08.539 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dca 2022-07-03T10:00:08.540 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dca/dca.ko 2022-07-03T10:00:08.540 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/devfreq 2022-07-03T10:00:08.540 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-07-03T10:00:08.540 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dma 2022-07-03T10:00:08.540 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dma/dw 2022-07-03T10:00:08.541 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dma/dw/dw_dmac.ko 2022-07-03T10:00:08.541 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dma/ioat 2022-07-03T10:00:08.541 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/dma/ioat/ioatdma.ko 2022-07-03T10:00:08.541 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac 2022-07-03T10:00:08.541 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/amd64_edac.ko 2022-07-03T10:00:08.542 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/e752x_edac.ko 2022-07-03T10:00:08.542 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/edac_mce_amd.ko 2022-07-03T10:00:08.542 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i3000_edac.ko 2022-07-03T10:00:08.542 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i3200_edac.ko 2022-07-03T10:00:08.542 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i5000_edac.ko 2022-07-03T10:00:08.543 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i5100_edac.ko 2022-07-03T10:00:08.543 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i5400_edac.ko 2022-07-03T10:00:08.543 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i7300_edac.ko 2022-07-03T10:00:08.543 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i7core_edac.ko 2022-07-03T10:00:08.544 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/i82975x_edac.ko 2022-07-03T10:00:08.544 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/ie31200_edac.ko 2022-07-03T10:00:08.544 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/sb_edac.ko 2022-07-03T10:00:08.544 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/edac/x38_edac.ko 2022-07-03T10:00:08.544 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/firmware 2022-07-03T10:00:08.545 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/firmware/edd.ko 2022-07-03T10:00:08.545 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/firmware/iscsi_ibft.ko 2022-07-03T10:00:08.545 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/gpio 2022-07-03T10:00:08.545 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/gpio/gpio-ich.ko 2022-07-03T10:00:08.545 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/gpio/gpio-viperboard.ko 2022-07-03T10:00:08.546 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid 2022-07-03T10:00:08.546 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-appleir.ko 2022-07-03T10:00:08.546 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-aureal.ko 2022-07-03T10:00:08.546 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-axff.ko 2022-07-03T10:00:08.546 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-dr.ko 2022-07-03T10:00:08.547 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-elecom.ko 2022-07-03T10:00:08.547 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-elo.ko 2022-07-03T10:00:08.547 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-emsff.ko 2022-07-03T10:00:08.547 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-gaff.ko 2022-07-03T10:00:08.547 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-gt683r.ko 2022-07-03T10:00:08.548 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-gyration.ko 2022-07-03T10:00:08.548 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-holtek-kbd.ko 2022-07-03T10:00:08.548 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-holtek-mouse.ko 2022-07-03T10:00:08.548 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-holtekff.ko 2022-07-03T10:00:08.548 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-hyperv.ko 2022-07-03T10:00:08.549 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-icade.ko 2022-07-03T10:00:08.549 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-keytouch.ko 2022-07-03T10:00:08.549 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-kye.ko 2022-07-03T10:00:08.549 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-lcpower.ko 2022-07-03T10:00:08.550 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-led.ko 2022-07-03T10:00:08.550 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-lenovo.ko 2022-07-03T10:00:08.550 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-logitech-dj.ko 2022-07-03T10:00:08.550 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-07-03T10:00:08.550 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-multitouch.ko 2022-07-03T10:00:08.551 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-ortek.ko 2022-07-03T10:00:08.551 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-penmount.ko 2022-07-03T10:00:08.551 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-petalynx.ko 2022-07-03T10:00:08.551 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-picolcd.ko 2022-07-03T10:00:08.551 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-pl.ko 2022-07-03T10:00:08.552 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-plantronics.ko 2022-07-03T10:00:08.552 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-primax.ko 2022-07-03T10:00:08.552 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-rmi.ko 2022-07-03T10:00:08.552 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-arvo.ko 2022-07-03T10:00:08.552 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-common.ko 2022-07-03T10:00:08.553 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-isku.ko 2022-07-03T10:00:08.553 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-kone.ko 2022-07-03T10:00:08.553 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-07-03T10:00:08.553 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-konepure.ko 2022-07-03T10:00:08.554 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-07-03T10:00:08.554 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-lua.ko 2022-07-03T10:00:08.555 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-pyra.ko 2022-07-03T10:00:08.555 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-ryos.ko 2022-07-03T10:00:08.555 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat-savu.ko 2022-07-03T10:00:08.556 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-roccat.ko 2022-07-03T10:00:08.556 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-saitek.ko 2022-07-03T10:00:08.556 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-samsung.ko 2022-07-03T10:00:08.557 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-sensor-hub.ko 2022-07-03T10:00:08.557 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-sjoy.ko 2022-07-03T10:00:08.557 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-sony.ko 2022-07-03T10:00:08.557 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-speedlink.ko 2022-07-03T10:00:08.558 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-steelseries.ko 2022-07-03T10:00:08.558 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-sunplus.ko 2022-07-03T10:00:08.558 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-thrustmaster.ko 2022-07-03T10:00:08.559 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-tivo.ko 2022-07-03T10:00:08.559 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-tmff.ko 2022-07-03T10:00:08.560 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-topseed.ko 2022-07-03T10:00:08.560 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-twinhan.ko 2022-07-03T10:00:08.560 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-uclogic.ko 2022-07-03T10:00:08.560 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-waltop.ko 2022-07-03T10:00:08.561 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-wiimote.ko 2022-07-03T10:00:08.561 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-xinmo.ko 2022-07-03T10:00:08.561 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-zpff.ko 2022-07-03T10:00:08.562 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/hid-zydacron.ko 2022-07-03T10:00:08.562 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/uhid.ko 2022-07-03T10:00:08.562 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hid/wacom.ko 2022-07-03T10:00:08.563 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hv 2022-07-03T10:00:08.563 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hv/hv_balloon.ko 2022-07-03T10:00:08.563 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hv/hv_utils.ko 2022-07-03T10:00:08.564 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hv/hv_vmbus.ko 2022-07-03T10:00:08.564 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon 2022-07-03T10:00:08.564 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/abituguru.ko 2022-07-03T10:00:08.565 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/abituguru3.ko 2022-07-03T10:00:08.565 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/acpi_power_meter.ko 2022-07-03T10:00:08.565 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ad7414.ko 2022-07-03T10:00:08.566 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ad7418.ko 2022-07-03T10:00:08.566 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adc128d818.ko 2022-07-03T10:00:08.566 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm1021.ko 2022-07-03T10:00:08.567 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm1025.ko 2022-07-03T10:00:08.567 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm1026.ko 2022-07-03T10:00:08.567 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm1029.ko 2022-07-03T10:00:08.568 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm1031.ko 2022-07-03T10:00:08.568 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adm9240.ko 2022-07-03T10:00:08.568 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ads7828.ko 2022-07-03T10:00:08.568 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7410.ko 2022-07-03T10:00:08.569 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7411.ko 2022-07-03T10:00:08.569 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7462.ko 2022-07-03T10:00:08.569 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7470.ko 2022-07-03T10:00:08.569 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7475.ko 2022-07-03T10:00:08.569 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/adt7x10.ko 2022-07-03T10:00:08.570 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/amc6821.ko 2022-07-03T10:00:08.570 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/applesmc.ko 2022-07-03T10:00:08.570 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/asb100.ko 2022-07-03T10:00:08.570 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/asc7621.ko 2022-07-03T10:00:08.571 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/asus_atk0110.ko 2022-07-03T10:00:08.571 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/atxp1.ko 2022-07-03T10:00:08.571 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/coretemp.ko 2022-07-03T10:00:08.571 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-07-03T10:00:08.571 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/dme1737.ko 2022-07-03T10:00:08.572 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ds1621.ko 2022-07-03T10:00:08.572 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ds620.ko 2022-07-03T10:00:08.572 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/emc1403.ko 2022-07-03T10:00:08.572 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/emc6w201.ko 2022-07-03T10:00:08.572 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/f71805f.ko 2022-07-03T10:00:08.573 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/f71882fg.ko 2022-07-03T10:00:08.573 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/f75375s.ko 2022-07-03T10:00:08.573 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/fam15h_power.ko 2022-07-03T10:00:08.573 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/fschmd.ko 2022-07-03T10:00:08.573 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/g760a.ko 2022-07-03T10:00:08.574 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/g762.ko 2022-07-03T10:00:08.574 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/gl518sm.ko 2022-07-03T10:00:08.574 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/gl520sm.ko 2022-07-03T10:00:08.574 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/hwmon-vid.ko 2022-07-03T10:00:08.575 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/i5500_temp.ko 2022-07-03T10:00:08.575 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/i5k_amb.ko 2022-07-03T10:00:08.575 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ibmaem.ko 2022-07-03T10:00:08.575 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ibmpex.ko 2022-07-03T10:00:08.575 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ina209.ko 2022-07-03T10:00:08.576 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ina2xx.ko 2022-07-03T10:00:08.576 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/it87.ko 2022-07-03T10:00:08.576 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/k10temp.ko 2022-07-03T10:00:08.576 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/k8temp.ko 2022-07-03T10:00:08.576 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lineage-pem.ko 2022-07-03T10:00:08.577 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm63.ko 2022-07-03T10:00:08.577 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm73.ko 2022-07-03T10:00:08.577 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm75.ko 2022-07-03T10:00:08.577 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm77.ko 2022-07-03T10:00:08.577 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm78.ko 2022-07-03T10:00:08.578 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm80.ko 2022-07-03T10:00:08.578 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm83.ko 2022-07-03T10:00:08.578 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm85.ko 2022-07-03T10:00:08.579 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm87.ko 2022-07-03T10:00:08.579 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm90.ko 2022-07-03T10:00:08.579 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm92.ko 2022-07-03T10:00:08.579 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm93.ko 2022-07-03T10:00:08.580 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm95234.ko 2022-07-03T10:00:08.580 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm95241.ko 2022-07-03T10:00:08.580 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/lm95245.ko 2022-07-03T10:00:08.581 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc2945.ko 2022-07-03T10:00:08.581 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4151.ko 2022-07-03T10:00:08.581 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4215.ko 2022-07-03T10:00:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4222.ko 2022-07-03T10:00:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4245.ko 2022-07-03T10:00:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4260.ko 2022-07-03T10:00:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/ltc4261.ko 2022-07-03T10:00:08.582 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max16065.ko 2022-07-03T10:00:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max1619.ko 2022-07-03T10:00:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max1668.ko 2022-07-03T10:00:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max197.ko 2022-07-03T10:00:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max6639.ko 2022-07-03T10:00:08.583 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max6642.ko 2022-07-03T10:00:08.584 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max6650.ko 2022-07-03T10:00:08.584 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/max6697.ko 2022-07-03T10:00:08.584 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/mcp3021.ko 2022-07-03T10:00:08.584 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/nct6683.ko 2022-07-03T10:00:08.585 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/nct6775-core.ko 2022-07-03T10:00:08.585 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/nct6775.ko 2022-07-03T10:00:08.585 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/nct7802.ko 2022-07-03T10:00:08.585 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pc87360.ko 2022-07-03T10:00:08.586 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pc87427.ko 2022-07-03T10:00:08.586 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pcf8591.ko 2022-07-03T10:00:08.586 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus 2022-07-03T10:00:08.587 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-07-03T10:00:08.587 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-07-03T10:00:08.587 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-07-03T10:00:08.587 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/max16064.ko 2022-07-03T10:00:08.587 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/max34440.ko 2022-07-03T10:00:08.588 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/max8688.ko 2022-07-03T10:00:08.588 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-07-03T10:00:08.588 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-07-03T10:00:08.588 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-07-03T10:00:08.589 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-07-03T10:00:08.589 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-07-03T10:00:08.589 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-07-03T10:00:08.589 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/powr1220.ko 2022-07-03T10:00:08.590 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sch5627.ko 2022-07-03T10:00:08.590 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sch5636.ko 2022-07-03T10:00:08.590 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sch56xx-common.ko 2022-07-03T10:00:08.590 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sht15.ko 2022-07-03T10:00:08.590 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sht21.ko 2022-07-03T10:00:08.591 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/shtc1.ko 2022-07-03T10:00:08.591 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/sis5595.ko 2022-07-03T10:00:08.591 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/smsc47b397.ko 2022-07-03T10:00:08.591 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/smsc47m1.ko 2022-07-03T10:00:08.591 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/smsc47m192.ko 2022-07-03T10:00:08.592 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/thmc50.ko 2022-07-03T10:00:08.592 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/tmp102.ko 2022-07-03T10:00:08.592 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/tmp103.ko 2022-07-03T10:00:08.593 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/tmp401.ko 2022-07-03T10:00:08.593 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/tmp421.ko 2022-07-03T10:00:08.594 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/via-cputemp.ko 2022-07-03T10:00:08.594 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/via686a.ko 2022-07-03T10:00:08.594 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/vt1211.ko 2022-07-03T10:00:08.594 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/vt8231.ko 2022-07-03T10:00:08.595 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83627ehf.ko 2022-07-03T10:00:08.595 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83627hf.ko 2022-07-03T10:00:08.595 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83781d.ko 2022-07-03T10:00:08.595 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83791d.ko 2022-07-03T10:00:08.595 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83792d.ko 2022-07-03T10:00:08.596 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83793.ko 2022-07-03T10:00:08.596 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83795.ko 2022-07-03T10:00:08.596 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83l785ts.ko 2022-07-03T10:00:08.596 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/hwmon/w83l786ng.ko 2022-07-03T10:00:08.597 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c 2022-07-03T10:00:08.597 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/algos 2022-07-03T10:00:08.597 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-07-03T10:00:08.598 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-07-03T10:00:08.598 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses 2022-07-03T10:00:08.598 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-07-03T10:00:08.598 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-07-03T10:00:08.599 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-07-03T10:00:08.599 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2022-07-03T10:00:08.599 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-07-03T10:00:08.599 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-07-03T10:00:08.600 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-07-03T10:00:08.600 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-07-03T10:00:08.600 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-i801.ko 2022-07-03T10:00:08.600 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-isch.ko 2022-07-03T10:00:08.601 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-07-03T10:00:08.601 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-07-03T10:00:08.601 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-07-03T10:00:08.601 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-parport.ko 2022-07-03T10:00:08.602 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-07-03T10:00:08.602 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-07-03T10:00:08.602 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-07-03T10:00:08.602 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-07-03T10:00:08.602 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-07-03T10:00:08.603 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-07-03T10:00:08.603 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-via.ko 2022-07-03T10:00:08.603 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-07-03T10:00:08.603 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-07-03T10:00:08.604 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/i2c-dev.ko 2022-07-03T10:00:08.604 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/i2c-mux.ko 2022-07-03T10:00:08.604 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-07-03T10:00:08.604 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/i2c-smbus.ko 2022-07-03T10:00:08.605 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/i2c/i2c-stub.ko 2022-07-03T10:00:08.605 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband 2022-07-03T10:00:08.605 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core 2022-07-03T10:00:08.605 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/ib_cm.ko 2022-07-03T10:00:08.606 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/ib_core.ko 2022-07-03T10:00:08.606 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/ib_umad.ko 2022-07-03T10:00:08.606 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-07-03T10:00:08.606 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/iw_cm.ko 2022-07-03T10:00:08.607 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/rdma_cm.ko 2022-07-03T10:00:08.607 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-07-03T10:00:08.607 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw 2022-07-03T10:00:08.607 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/cxgb4 2022-07-03T10:00:08.607 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-07-03T10:00:08.608 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mlx4 2022-07-03T10:00:08.608 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-07-03T10:00:08.608 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mlx5 2022-07-03T10:00:08.608 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-07-03T10:00:08.609 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mthca 2022-07-03T10:00:08.609 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-07-03T10:00:08.609 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp 2022-07-03T10:00:08.610 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/ipoib 2022-07-03T10:00:08.610 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-07-03T10:00:08.610 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/iser 2022-07-03T10:00:08.611 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-07-03T10:00:08.611 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/isert 2022-07-03T10:00:08.611 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-07-03T10:00:08.611 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/srp 2022-07-03T10:00:08.611 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-07-03T10:00:08.612 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/srpt 2022-07-03T10:00:08.612 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-07-03T10:00:08.612 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input 2022-07-03T10:00:08.612 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport 2022-07-03T10:00:08.612 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-07-03T10:00:08.613 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport/fm801-gp.ko 2022-07-03T10:00:08.613 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport/gameport.ko 2022-07-03T10:00:08.613 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport/lightning.ko 2022-07-03T10:00:08.613 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/gameport/ns558.ko 2022-07-03T10:00:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/joydev.ko 2022-07-03T10:00:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/keyboard 2022-07-03T10:00:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/keyboard/gpio_keys.ko 2022-07-03T10:00:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse 2022-07-03T10:00:08.614 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/appletouch.ko 2022-07-03T10:00:08.615 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/bcm5974.ko 2022-07-03T10:00:08.615 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/cyapatp.ko 2022-07-03T10:00:08.615 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/elan_i2c.ko 2022-07-03T10:00:08.615 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/sermouse.ko 2022-07-03T10:00:08.615 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-07-03T10:00:08.616 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/synaptics_usb.ko 2022-07-03T10:00:08.616 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/mouse/vsxxxaa.ko 2022-07-03T10:00:08.616 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/rmi4 2022-07-03T10:00:08.616 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/rmi4/rmi_core.ko 2022-07-03T10:00:08.616 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/serio 2022-07-03T10:00:08.617 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/serio/altera_ps2.ko 2022-07-03T10:00:08.617 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/serio/arc_ps2.ko 2022-07-03T10:00:08.617 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-07-03T10:00:08.617 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/serio/serio_raw.ko 2022-07-03T10:00:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/input/sparse-keymap.ko 2022-07-03T10:00:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/iommu 2022-07-03T10:00:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/iommu/amd 2022-07-03T10:00:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/iommu/amd/iommu_v2.ko 2022-07-03T10:00:08.618 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds 2022-07-03T10:00:08.619 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/leds-blinkm.ko 2022-07-03T10:00:08.619 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/leds-clevo-mail.ko 2022-07-03T10:00:08.619 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/leds-lm3530.ko 2022-07-03T10:00:08.619 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/leds-lp3944.ko 2022-07-03T10:00:08.619 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/leds-ss4200.ko 2022-07-03T10:00:08.620 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger 2022-07-03T10:00:08.620 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-07-03T10:00:08.620 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-07-03T10:00:08.620 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-07-03T10:00:08.620 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-07-03T10:00:08.621 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-07-03T10:00:08.621 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-07-03T10:00:08.621 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-07-03T10:00:08.621 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-07-03T10:00:08.622 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-07-03T10:00:08.622 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md 2022-07-03T10:00:08.622 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/bcache 2022-07-03T10:00:08.623 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/bcache/bcache.ko 2022-07-03T10:00:08.623 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-bio-prison.ko 2022-07-03T10:00:08.623 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-cache-smq.ko 2022-07-03T10:00:08.623 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-cache.ko 2022-07-03T10:00:08.624 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-crypt.ko 2022-07-03T10:00:08.624 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-delay.ko 2022-07-03T10:00:08.624 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-flakey.ko 2022-07-03T10:00:08.624 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-log-userspace.ko 2022-07-03T10:00:08.624 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-multipath.ko 2022-07-03T10:00:08.625 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-queue-length.ko 2022-07-03T10:00:08.625 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-raid.ko 2022-07-03T10:00:08.625 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-round-robin.ko 2022-07-03T10:00:08.625 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-service-time.ko 2022-07-03T10:00:08.625 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-switch.ko 2022-07-03T10:00:08.626 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-thin-pool.ko 2022-07-03T10:00:08.626 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/dm-verity.ko 2022-07-03T10:00:08.626 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/faulty.ko 2022-07-03T10:00:08.626 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/linear.ko 2022-07-03T10:00:08.627 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/multipath.ko 2022-07-03T10:00:08.627 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/persistent-data 2022-07-03T10:00:08.627 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-07-03T10:00:08.627 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/raid0.ko 2022-07-03T10:00:08.627 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/raid1.ko 2022-07-03T10:00:08.628 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/raid10.ko 2022-07-03T10:00:08.628 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/md/raid456.ko 2022-07-03T10:00:08.628 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message 2022-07-03T10:00:08.628 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion 2022-07-03T10:00:08.628 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptbase.ko 2022-07-03T10:00:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptctl.ko 2022-07-03T10:00:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptfc.ko 2022-07-03T10:00:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptlan.ko 2022-07-03T10:00:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptsas.ko 2022-07-03T10:00:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptscsih.ko 2022-07-03T10:00:08.630 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/message/fusion/mptspi.ko 2022-07-03T10:00:08.630 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd 2022-07-03T10:00:08.630 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/lpc_ich.ko 2022-07-03T10:00:08.630 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/lpc_sch.ko 2022-07-03T10:00:08.630 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/mfd-core.ko 2022-07-03T10:00:08.631 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/sm501.ko 2022-07-03T10:00:08.631 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/viperboard.ko 2022-07-03T10:00:08.631 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/vx855.ko 2022-07-03T10:00:08.631 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/mfd/wl1273-core.ko 2022-07-03T10:00:08.631 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc 2022-07-03T10:00:08.632 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/altera-stapl 2022-07-03T10:00:08.632 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-07-03T10:00:08.632 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/apds9802als.ko 2022-07-03T10:00:08.632 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/apds990x.ko 2022-07-03T10:00:08.633 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/bh1770glc.ko 2022-07-03T10:00:08.633 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/cb710 2022-07-03T10:00:08.633 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/cb710/cb710.ko 2022-07-03T10:00:08.633 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/eeprom 2022-07-03T10:00:08.633 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/eeprom/at24.ko 2022-07-03T10:00:08.634 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/eeprom/eeprom.ko 2022-07-03T10:00:08.634 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-07-03T10:00:08.634 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/eeprom/max6875.ko 2022-07-03T10:00:08.634 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/enclosure.ko 2022-07-03T10:00:08.634 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/hpilo.ko 2022-07-03T10:00:08.635 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/isl29003.ko 2022-07-03T10:00:08.635 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/isl29020.ko 2022-07-03T10:00:08.635 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/lis3lv02d 2022-07-03T10:00:08.635 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-07-03T10:00:08.636 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-07-03T10:00:08.636 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/mei 2022-07-03T10:00:08.636 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/mei/mei-me.ko 2022-07-03T10:00:08.636 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/mei/mei-txe.ko 2022-07-03T10:00:08.636 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/mei/mei.ko 2022-07-03T10:00:08.637 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-gru 2022-07-03T10:00:08.637 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-gru/gru.ko 2022-07-03T10:00:08.637 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-xp 2022-07-03T10:00:08.637 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-xp/xp.ko 2022-07-03T10:00:08.638 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-xp/xpc.ko 2022-07-03T10:00:08.638 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-07-03T10:00:08.638 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/tifm_7xx1.ko 2022-07-03T10:00:08.638 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/tifm_core.ko 2022-07-03T10:00:08.638 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/tsl2550.ko 2022-07-03T10:00:08.639 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/vmw_balloon.ko 2022-07-03T10:00:08.639 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/vmw_vmci 2022-07-03T10:00:08.639 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-07-03T10:00:08.639 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net 2022-07-03T10:00:08.639 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/bonding 2022-07-03T10:00:08.640 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/bonding/bonding.ko 2022-07-03T10:00:08.640 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa 2022-07-03T10:00:08.640 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/b53 2022-07-03T10:00:08.640 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/b53/b53_common.ko 2022-07-03T10:00:08.640 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/bcm-sf2.ko 2022-07-03T10:00:08.641 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/mv88e6060.ko 2022-07-03T10:00:08.641 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/mv88e6xxx 2022-07-03T10:00:08.641 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-07-03T10:00:08.641 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/dummy.ko 2022-07-03T10:00:08.642 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/eql.ko 2022-07-03T10:00:08.642 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet 2022-07-03T10:00:08.642 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/3com 2022-07-03T10:00:08.642 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-07-03T10:00:08.642 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-07-03T10:00:08.643 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-07-03T10:00:08.643 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-07-03T10:00:08.643 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/8390 2022-07-03T10:00:08.643 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/8390/8390.ko 2022-07-03T10:00:08.643 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-07-03T10:00:08.644 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-07-03T10:00:08.644 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-07-03T10:00:08.644 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/adaptec 2022-07-03T10:00:08.644 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-07-03T10:00:08.644 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/agere 2022-07-03T10:00:08.645 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/agere/et131x.ko 2022-07-03T10:00:08.645 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/alteon 2022-07-03T10:00:08.645 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-07-03T10:00:08.645 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/altera 2022-07-03T10:00:08.645 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-07-03T10:00:08.646 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/amd 2022-07-03T10:00:08.646 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-07-03T10:00:08.646 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-07-03T10:00:08.647 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-07-03T10:00:08.647 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros 2022-07-03T10:00:08.647 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/alx 2022-07-03T10:00:08.647 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-07-03T10:00:08.648 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atl1c 2022-07-03T10:00:08.648 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-07-03T10:00:08.648 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atl1e 2022-07-03T10:00:08.648 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-07-03T10:00:08.648 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atlx 2022-07-03T10:00:08.649 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-07-03T10:00:08.649 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-07-03T10:00:08.649 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom 2022-07-03T10:00:08.649 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-07-03T10:00:08.650 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-07-03T10:00:08.650 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-07-03T10:00:08.650 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-07-03T10:00:08.650 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-07-03T10:00:08.650 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/genet 2022-07-03T10:00:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-07-03T10:00:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-07-03T10:00:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/brocade 2022-07-03T10:00:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/brocade/bna 2022-07-03T10:00:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-07-03T10:00:08.652 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio 2022-07-03T10:00:08.652 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb 2022-07-03T10:00:08.652 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-07-03T10:00:08.652 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-07-03T10:00:08.652 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-07-03T10:00:08.653 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-07-03T10:00:08.653 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-07-03T10:00:08.653 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-07-03T10:00:08.653 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-07-03T10:00:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-07-03T10:00:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-07-03T10:00:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/cisco 2022-07-03T10:00:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/cisco/enic 2022-07-03T10:00:08.654 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-07-03T10:00:08.655 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec 2022-07-03T10:00:08.655 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip 2022-07-03T10:00:08.655 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-07-03T10:00:08.655 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-07-03T10:00:08.655 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-07-03T10:00:08.656 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-07-03T10:00:08.656 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-07-03T10:00:08.656 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-07-03T10:00:08.656 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dlink 2022-07-03T10:00:08.656 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-07-03T10:00:08.657 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-07-03T10:00:08.657 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/dnet.ko 2022-07-03T10:00:08.657 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/emulex 2022-07-03T10:00:08.657 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/emulex/benet 2022-07-03T10:00:08.657 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-07-03T10:00:08.658 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/ethoc.ko 2022-07-03T10:00:08.658 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/fealnx.ko 2022-07-03T10:00:08.658 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel 2022-07-03T10:00:08.658 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/e100.ko 2022-07-03T10:00:08.659 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/e1000 2022-07-03T10:00:08.659 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-07-03T10:00:08.659 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/e1000e 2022-07-03T10:00:08.659 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-07-03T10:00:08.659 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/fm10k 2022-07-03T10:00:08.660 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-07-03T10:00:08.660 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/i40e 2022-07-03T10:00:08.660 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-07-03T10:00:08.660 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/iavf 2022-07-03T10:00:08.660 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-07-03T10:00:08.661 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/igb 2022-07-03T10:00:08.661 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-07-03T10:00:08.661 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/igbvf 2022-07-03T10:00:08.661 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-07-03T10:00:08.661 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgb 2022-07-03T10:00:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-07-03T10:00:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgbe 2022-07-03T10:00:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-07-03T10:00:08.662 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgbevf 2022-07-03T10:00:08.663 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-07-03T10:00:08.663 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/jme.ko 2022-07-03T10:00:08.663 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/marvell 2022-07-03T10:00:08.663 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-07-03T10:00:08.663 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/marvell/skge.ko 2022-07-03T10:00:08.664 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-07-03T10:00:08.664 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox 2022-07-03T10:00:08.664 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx4 2022-07-03T10:00:08.664 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-07-03T10:00:08.664 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-07-03T10:00:08.665 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx5 2022-07-03T10:00:08.665 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-07-03T10:00:08.665 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-07-03T10:00:08.665 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/micrel 2022-07-03T10:00:08.665 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-07-03T10:00:08.666 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/myricom 2022-07-03T10:00:08.666 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/myricom/myri10ge 2022-07-03T10:00:08.666 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-07-03T10:00:08.666 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/natsemi 2022-07-03T10:00:08.666 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-07-03T10:00:08.667 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-07-03T10:00:08.667 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/neterion 2022-07-03T10:00:08.667 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-07-03T10:00:08.667 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/neterion/vxge 2022-07-03T10:00:08.668 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2022-07-03T10:00:08.668 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/nvidia 2022-07-03T10:00:08.668 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-07-03T10:00:08.668 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/packetengines 2022-07-03T10:00:08.668 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-07-03T10:00:08.669 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-07-03T10:00:08.669 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic 2022-07-03T10:00:08.669 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic/netxen 2022-07-03T10:00:08.669 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-07-03T10:00:08.669 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-07-03T10:00:08.670 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-07-03T10:00:08.670 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-07-03T10:00:08.670 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/rdc 2022-07-03T10:00:08.670 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-07-03T10:00:08.670 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/realtek 2022-07-03T10:00:08.671 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-07-03T10:00:08.671 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-07-03T10:00:08.671 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/realtek/atp.ko 2022-07-03T10:00:08.671 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-07-03T10:00:08.672 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/rocker 2022-07-03T10:00:08.672 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-07-03T10:00:08.672 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sfc 2022-07-03T10:00:08.672 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-07-03T10:00:08.672 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/silan 2022-07-03T10:00:08.673 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-07-03T10:00:08.673 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sis 2022-07-03T10:00:08.673 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sis/sis190.ko 2022-07-03T10:00:08.673 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sis/sis900.ko 2022-07-03T10:00:08.673 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/smsc 2022-07-03T10:00:08.674 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-07-03T10:00:08.674 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-07-03T10:00:08.674 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-07-03T10:00:08.674 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-07-03T10:00:08.674 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/stmicro 2022-07-03T10:00:08.675 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/stmicro/stmmac 2022-07-03T10:00:08.675 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-07-03T10:00:08.675 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-07-03T10:00:08.675 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sun 2022-07-03T10:00:08.676 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sun/cassini.ko 2022-07-03T10:00:08.676 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sun/niu.ko 2022-07-03T10:00:08.676 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sun/sungem.ko 2022-07-03T10:00:08.676 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-07-03T10:00:08.676 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/tehuti 2022-07-03T10:00:08.677 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-07-03T10:00:08.677 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/ti 2022-07-03T10:00:08.677 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/ti/tlan.ko 2022-07-03T10:00:08.677 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/via 2022-07-03T10:00:08.677 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-07-03T10:00:08.678 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-07-03T10:00:08.678 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/wiznet 2022-07-03T10:00:08.678 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-07-03T10:00:08.678 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-07-03T10:00:08.678 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/xircom 2022-07-03T10:00:08.679 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-07-03T10:00:08.679 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/geneve.ko 2022-07-03T10:00:08.679 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/hyperv 2022-07-03T10:00:08.679 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-07-03T10:00:08.680 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ieee802154 2022-07-03T10:00:08.680 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ieee802154/fakelb.ko 2022-07-03T10:00:08.680 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ifb.ko 2022-07-03T10:00:08.680 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ipvlan 2022-07-03T10:00:08.680 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ipvlan/ipvlan.ko 2022-07-03T10:00:08.681 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/macvlan.ko 2022-07-03T10:00:08.681 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/macvtap.ko 2022-07-03T10:00:08.681 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/mdio 2022-07-03T10:00:08.681 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/mdio.ko 2022-07-03T10:00:08.681 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-07-03T10:00:08.682 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-07-03T10:00:08.682 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/mii.ko 2022-07-03T10:00:08.682 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/net_failover.ko 2022-07-03T10:00:08.682 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/netconsole.ko 2022-07-03T10:00:08.682 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/nlmon.ko 2022-07-03T10:00:08.683 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/pcs 2022-07-03T10:00:08.683 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-07-03T10:00:08.683 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy 2022-07-03T10:00:08.683 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/amd.ko 2022-07-03T10:00:08.683 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-07-03T10:00:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/bcm7xxx.ko 2022-07-03T10:00:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/bcm87xx.ko 2022-07-03T10:00:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/broadcom.ko 2022-07-03T10:00:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/cicada.ko 2022-07-03T10:00:08.684 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/davicom.ko 2022-07-03T10:00:08.685 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/dp83640.ko 2022-07-03T10:00:08.685 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/et1011c.ko 2022-07-03T10:00:08.685 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/icplus.ko 2022-07-03T10:00:08.685 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/lxt.ko 2022-07-03T10:00:08.686 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/marvell.ko 2022-07-03T10:00:08.686 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/micrel.ko 2022-07-03T10:00:08.686 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/national.ko 2022-07-03T10:00:08.686 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/phylink.ko 2022-07-03T10:00:08.686 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/qsemi.ko 2022-07-03T10:00:08.687 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/realtek.ko 2022-07-03T10:00:08.687 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/smsc.ko 2022-07-03T10:00:08.687 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/ste10Xp.ko 2022-07-03T10:00:08.687 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/phy/vitesse.ko 2022-07-03T10:00:08.687 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp 2022-07-03T10:00:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/bsd_comp.ko 2022-07-03T10:00:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/ppp_async.ko 2022-07-03T10:00:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/ppp_deflate.ko 2022-07-03T10:00:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/ppp_generic.ko 2022-07-03T10:00:08.688 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/ppp_mppe.ko 2022-07-03T10:00:08.689 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/ppp_synctty.ko 2022-07-03T10:00:08.689 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/pppoe.ko 2022-07-03T10:00:08.689 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/pppox.ko 2022-07-03T10:00:08.689 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/ppp/pptp.ko 2022-07-03T10:00:08.689 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/slip 2022-07-03T10:00:08.690 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/slip/slhc.ko 2022-07-03T10:00:08.690 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/slip/slip.ko 2022-07-03T10:00:08.690 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/sungem_phy.ko 2022-07-03T10:00:08.690 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/tap.ko 2022-07-03T10:00:08.691 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team 2022-07-03T10:00:08.691 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team.ko 2022-07-03T10:00:08.691 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team_mode_activebackup.ko 2022-07-03T10:00:08.691 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team_mode_broadcast.ko 2022-07-03T10:00:08.691 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-07-03T10:00:08.692 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team_mode_random.ko 2022-07-03T10:00:08.692 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-07-03T10:00:08.692 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/tun.ko 2022-07-03T10:00:08.692 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/veth.ko 2022-07-03T10:00:08.693 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/virtio_net.ko 2022-07-03T10:00:08.693 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/vmxnet3 2022-07-03T10:00:08.693 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-07-03T10:00:08.693 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/vxlan 2022-07-03T10:00:08.693 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/vxlan/vxlan.ko 2022-07-03T10:00:08.694 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/xen-netback 2022-07-03T10:00:08.694 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/xen-netback/xen-netback.ko 2022-07-03T10:00:08.694 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/net/xen-netfront.ko 2022-07-03T10:00:08.694 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ntb 2022-07-03T10:00:08.694 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ntb/ntb.ko 2022-07-03T10:00:08.695 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/nvme 2022-07-03T10:00:08.695 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/nvme/host 2022-07-03T10:00:08.695 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/nvme/host/nvme-core.ko 2022-07-03T10:00:08.695 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/nvme/host/nvme.ko 2022-07-03T10:00:08.695 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/parport 2022-07-03T10:00:08.696 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/parport/parport.ko 2022-07-03T10:00:08.696 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/parport/parport_cs.ko 2022-07-03T10:00:08.696 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/parport/parport_pc.ko 2022-07-03T10:00:08.696 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/parport/parport_serial.ko 2022-07-03T10:00:08.697 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci 2022-07-03T10:00:08.697 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci/hotplug 2022-07-03T10:00:08.697 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-07-03T10:00:08.697 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci/pcie 2022-07-03T10:00:08.697 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci/pcie/aer_inject.ko 2022-07-03T10:00:08.698 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pci/xen-pcifront.ko 2022-07-03T10:00:08.698 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pcmcia 2022-07-03T10:00:08.698 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pcmcia/i82092.ko 2022-07-03T10:00:08.698 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pcmcia/pd6729.ko 2022-07-03T10:00:08.698 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pcmcia/yenta_socket.ko 2022-07-03T10:00:08.699 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pinctrl 2022-07-03T10:00:08.699 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pinctrl/intel 2022-07-03T10:00:08.699 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-07-03T10:00:08.699 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform 2022-07-03T10:00:08.699 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86 2022-07-03T10:00:08.700 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/acer-wmi.ko 2022-07-03T10:00:08.700 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/acerhdf.ko 2022-07-03T10:00:08.700 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/apple-gmux.ko 2022-07-03T10:00:08.700 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/asus-laptop.ko 2022-07-03T10:00:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-07-03T10:00:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/asus-wmi.ko 2022-07-03T10:00:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/classmate-laptop.ko 2022-07-03T10:00:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-07-03T10:00:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-07-03T10:00:08.702 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-07-03T10:00:08.702 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-07-03T10:00:08.702 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/hdaps.ko 2022-07-03T10:00:08.702 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/hp-wmi.ko 2022-07-03T10:00:08.703 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/hp_accel.ko 2022-07-03T10:00:08.703 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/intel 2022-07-03T10:00:08.703 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-07-03T10:00:08.703 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/intel_ips.ko 2022-07-03T10:00:08.703 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/msi-wmi.ko 2022-07-03T10:00:08.704 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/mxm-wmi.ko 2022-07-03T10:00:08.704 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-07-03T10:00:08.704 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/samsung-laptop.ko 2022-07-03T10:00:08.704 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/samsung-q10.ko 2022-07-03T10:00:08.704 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/topstar-laptop.ko 2022-07-03T10:00:08.705 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-07-03T10:00:08.705 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/toshiba_haps.ko 2022-07-03T10:00:08.705 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/wmi-bmof.ko 2022-07-03T10:00:08.705 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/platform/x86/wmi.ko 2022-07-03T10:00:08.705 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/powercap 2022-07-03T10:00:08.706 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/powercap/intel_rapl_common.ko 2022-07-03T10:00:08.706 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/powercap/intel_rapl_msr.ko 2022-07-03T10:00:08.706 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps 2022-07-03T10:00:08.706 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps/clients 2022-07-03T10:00:08.707 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps/clients/pps-gpio.ko 2022-07-03T10:00:08.707 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps/clients/pps-ldisc.ko 2022-07-03T10:00:08.707 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps/clients/pps_parport.ko 2022-07-03T10:00:08.707 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/pps/pps_core.ko 2022-07-03T10:00:08.707 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ptp 2022-07-03T10:00:08.708 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ptp/ptp.ko 2022-07-03T10:00:08.708 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ptp/ptp_kvm.ko 2022-07-03T10:00:08.708 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc 2022-07-03T10:00:08.709 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-bq32k.ko 2022-07-03T10:00:08.709 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-bq4802.ko 2022-07-03T10:00:08.709 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1286.ko 2022-07-03T10:00:08.710 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1307.ko 2022-07-03T10:00:08.710 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1374.ko 2022-07-03T10:00:08.710 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1511.ko 2022-07-03T10:00:08.710 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1553.ko 2022-07-03T10:00:08.711 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1672.ko 2022-07-03T10:00:08.711 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds1742.ko 2022-07-03T10:00:08.711 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds2404.ko 2022-07-03T10:00:08.712 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-ds3232.ko 2022-07-03T10:00:08.712 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-em3027.ko 2022-07-03T10:00:08.712 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-fm3130.ko 2022-07-03T10:00:08.713 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-isl12022.ko 2022-07-03T10:00:08.713 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-isl1208.ko 2022-07-03T10:00:08.713 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-m41t80.ko 2022-07-03T10:00:08.713 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-m48t35.ko 2022-07-03T10:00:08.713 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-m48t59.ko 2022-07-03T10:00:08.714 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-max6900.ko 2022-07-03T10:00:08.714 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-msm6242.ko 2022-07-03T10:00:08.714 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-pcf2127.ko 2022-07-03T10:00:08.714 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-pcf85063.ko 2022-07-03T10:00:08.714 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-pcf8523.ko 2022-07-03T10:00:08.715 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-pcf8563.ko 2022-07-03T10:00:08.715 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-pcf8583.ko 2022-07-03T10:00:08.715 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-rp5c01.ko 2022-07-03T10:00:08.715 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-rs5c372.ko 2022-07-03T10:00:08.715 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-07-03T10:00:08.716 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-rx8025.ko 2022-07-03T10:00:08.716 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-rx8581.ko 2022-07-03T10:00:08.716 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-07-03T10:00:08.716 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-v3020.ko 2022-07-03T10:00:08.717 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/rtc/rtc-x1205.ko 2022-07-03T10:00:08.717 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi 2022-07-03T10:00:08.717 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/3w-9xxx.ko 2022-07-03T10:00:08.717 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/3w-sas.ko 2022-07-03T10:00:08.717 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/3w-xxxx.ko 2022-07-03T10:00:08.718 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/BusLogic.ko 2022-07-03T10:00:08.718 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/a100u2w.ko 2022-07-03T10:00:08.718 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/aacraid 2022-07-03T10:00:08.718 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/aacraid/aacraid.ko 2022-07-03T10:00:08.718 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/advansys.ko 2022-07-03T10:00:08.719 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/aic7xxx 2022-07-03T10:00:08.719 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-07-03T10:00:08.719 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-07-03T10:00:08.719 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/am53c974.ko 2022-07-03T10:00:08.719 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/arcmsr 2022-07-03T10:00:08.720 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-07-03T10:00:08.720 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/atp870u.ko 2022-07-03T10:00:08.720 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/be2iscsi 2022-07-03T10:00:08.720 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-07-03T10:00:08.720 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bfa 2022-07-03T10:00:08.721 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bfa/bfa.ko 2022-07-03T10:00:08.721 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bnx2fc 2022-07-03T10:00:08.721 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-07-03T10:00:08.721 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bnx2i 2022-07-03T10:00:08.722 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-07-03T10:00:08.722 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/ch.ko 2022-07-03T10:00:08.722 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/csiostor 2022-07-03T10:00:08.722 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/csiostor/csiostor.ko 2022-07-03T10:00:08.722 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi 2022-07-03T10:00:08.723 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi/cxgb3i 2022-07-03T10:00:08.723 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-07-03T10:00:08.723 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi/cxgb4i 2022-07-03T10:00:08.723 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-07-03T10:00:08.723 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-07-03T10:00:08.724 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/dc395x.ko 2022-07-03T10:00:08.724 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/device_handler 2022-07-03T10:00:08.724 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-07-03T10:00:08.724 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-07-03T10:00:08.725 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-07-03T10:00:08.725 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-07-03T10:00:08.725 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/dmx3191d.ko 2022-07-03T10:00:08.725 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/esas2r 2022-07-03T10:00:08.725 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/esas2r/esas2r.ko 2022-07-03T10:00:08.726 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/esp_scsi.ko 2022-07-03T10:00:08.726 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/fcoe 2022-07-03T10:00:08.726 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/fcoe/fcoe.ko 2022-07-03T10:00:08.726 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-07-03T10:00:08.726 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/fnic 2022-07-03T10:00:08.727 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/fnic/fnic.ko 2022-07-03T10:00:08.727 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/hpsa.ko 2022-07-03T10:00:08.727 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/hptiop.ko 2022-07-03T10:00:08.727 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/hv_storvsc.ko 2022-07-03T10:00:08.727 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/initio.ko 2022-07-03T10:00:08.728 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/ipr.ko 2022-07-03T10:00:08.728 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/ips.ko 2022-07-03T10:00:08.728 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/isci 2022-07-03T10:00:08.728 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/isci/isci.ko 2022-07-03T10:00:08.729 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-07-03T10:00:08.729 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/iscsi_tcp.ko 2022-07-03T10:00:08.729 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libfc 2022-07-03T10:00:08.729 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libfc/libfc.ko 2022-07-03T10:00:08.729 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libiscsi.ko 2022-07-03T10:00:08.730 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libiscsi_tcp.ko 2022-07-03T10:00:08.730 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libsas 2022-07-03T10:00:08.730 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/libsas/libsas.ko 2022-07-03T10:00:08.730 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/lpfc 2022-07-03T10:00:08.730 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/lpfc/lpfc.ko 2022-07-03T10:00:08.731 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/megaraid 2022-07-03T10:00:08.731 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/megaraid.ko 2022-07-03T10:00:08.731 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-07-03T10:00:08.731 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-07-03T10:00:08.731 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-07-03T10:00:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/mpt3sas 2022-07-03T10:00:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-07-03T10:00:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/mvsas 2022-07-03T10:00:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/mvsas/mvsas.ko 2022-07-03T10:00:08.732 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/mvumi.ko 2022-07-03T10:00:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/pm8001 2022-07-03T10:00:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-07-03T10:00:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/pmcraid.ko 2022-07-03T10:00:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla1280.ko 2022-07-03T10:00:08.733 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla2xxx 2022-07-03T10:00:08.734 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-07-03T10:00:08.734 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-07-03T10:00:08.734 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla4xxx 2022-07-03T10:00:08.734 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-07-03T10:00:08.735 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/raid_class.ko 2022-07-03T10:00:08.735 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_debug.ko 2022-07-03T10:00:08.735 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_transport_fc.ko 2022-07-03T10:00:08.735 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-07-03T10:00:08.735 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_transport_sas.ko 2022-07-03T10:00:08.736 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_transport_spi.ko 2022-07-03T10:00:08.736 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/scsi_transport_srp.ko 2022-07-03T10:00:08.736 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/ses.ko 2022-07-03T10:00:08.736 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/st.ko 2022-07-03T10:00:08.736 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/stex.ko 2022-07-03T10:00:08.737 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/sym53c8xx_2 2022-07-03T10:00:08.737 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-07-03T10:00:08.737 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/virtio_scsi.ko 2022-07-03T10:00:08.737 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/vmw_pvscsi.ko 2022-07-03T10:00:08.737 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/scsi/wd719x.ko 2022-07-03T10:00:08.738 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ssb 2022-07-03T10:00:08.738 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ssb/ssb.ko 2022-07-03T10:00:08.738 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target 2022-07-03T10:00:08.738 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/iscsi 2022-07-03T10:00:08.739 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-07-03T10:00:08.739 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/loopback 2022-07-03T10:00:08.739 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/loopback/tcm_loop.ko 2022-07-03T10:00:08.739 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/target_core_file.ko 2022-07-03T10:00:08.739 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/target_core_iblock.ko 2022-07-03T10:00:08.740 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/target_core_mod.ko 2022-07-03T10:00:08.740 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/target_core_pscsi.ko 2022-07-03T10:00:08.740 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/target_core_user.ko 2022-07-03T10:00:08.740 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/tcm_fc 2022-07-03T10:00:08.741 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-07-03T10:00:08.741 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal 2022-07-03T10:00:08.741 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel 2022-07-03T10:00:08.741 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal 2022-07-03T10:00:08.741 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-07-03T10:00:08.742 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-07-03T10:00:08.742 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-07-03T10:00:08.742 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-07-03T10:00:08.742 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-07-03T10:00:08.742 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-07-03T10:00:08.743 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-07-03T10:00:08.743 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-07-03T10:00:08.743 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-07-03T10:00:08.743 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-07-03T10:00:08.744 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-07-03T10:00:08.744 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-07-03T10:00:08.744 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-07-03T10:00:08.744 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-07-03T10:00:08.744 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-07-03T10:00:08.745 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty 2022-07-03T10:00:08.745 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/ipwireless 2022-07-03T10:00:08.745 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-07-03T10:00:08.745 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/n_gsm.ko 2022-07-03T10:00:08.745 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/n_hdlc.ko 2022-07-03T10:00:08.746 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/nozomi.ko 2022-07-03T10:00:08.746 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial 2022-07-03T10:00:08.746 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial/8250 2022-07-03T10:00:08.746 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-07-03T10:00:08.747 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial/arc_uart.ko 2022-07-03T10:00:08.747 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial/jsm 2022-07-03T10:00:08.747 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/serial/jsm/jsm.ko 2022-07-03T10:00:08.747 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/tty/synclink_gt.ko 2022-07-03T10:00:08.747 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ufs 2022-07-03T10:00:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ufs/core 2022-07-03T10:00:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ufs/core/ufshcd-core.ko 2022-07-03T10:00:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ufs/host 2022-07-03T10:00:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/ufs/host/ufshcd-pci.ko 2022-07-03T10:00:08.748 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio 2022-07-03T10:00:08.749 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio/uio.ko 2022-07-03T10:00:08.749 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio/uio_aec.ko 2022-07-03T10:00:08.749 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio/uio_cif.ko 2022-07-03T10:00:08.749 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio/uio_pci_generic.ko 2022-07-03T10:00:08.749 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/uio/uio_sercos3.ko 2022-07-03T10:00:08.750 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb 2022-07-03T10:00:08.750 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/class 2022-07-03T10:00:08.750 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/class/cdc-acm.ko 2022-07-03T10:00:08.750 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/class/cdc-wdm.ko 2022-07-03T10:00:08.750 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/class/usblp.ko 2022-07-03T10:00:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/class/usbtmc.ko 2022-07-03T10:00:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/host 2022-07-03T10:00:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/host/sl811-hcd.ko 2022-07-03T10:00:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/host/u132-hcd.ko 2022-07-03T10:00:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/image 2022-07-03T10:00:08.752 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/image/mdc800.ko 2022-07-03T10:00:08.752 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/image/microtek.ko 2022-07-03T10:00:08.752 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc 2022-07-03T10:00:08.752 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/adutux.ko 2022-07-03T10:00:08.753 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/appledisplay.ko 2022-07-03T10:00:08.753 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/emi26.ko 2022-07-03T10:00:08.753 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/emi62.ko 2022-07-03T10:00:08.753 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/ezusb.ko 2022-07-03T10:00:08.753 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/ftdi-elan.ko 2022-07-03T10:00:08.754 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/idmouse.ko 2022-07-03T10:00:08.754 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/iowarrior.ko 2022-07-03T10:00:08.754 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/isight_firmware.ko 2022-07-03T10:00:08.754 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/ldusb.ko 2022-07-03T10:00:08.754 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/legousbtower.ko 2022-07-03T10:00:08.755 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/sisusbvga 2022-07-03T10:00:08.755 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-07-03T10:00:08.755 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/trancevibrator.ko 2022-07-03T10:00:08.755 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/usb3503.ko 2022-07-03T10:00:08.756 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/usblcd.ko 2022-07-03T10:00:08.756 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/usbsevseg.ko 2022-07-03T10:00:08.756 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/uss720.ko 2022-07-03T10:00:08.756 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/misc/yurex.ko 2022-07-03T10:00:08.756 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/phy 2022-07-03T10:00:08.757 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/phy/phy-generic.ko 2022-07-03T10:00:08.757 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial 2022-07-03T10:00:08.757 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/aircable.ko 2022-07-03T10:00:08.757 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ark3116.ko 2022-07-03T10:00:08.757 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/belkin_sa.ko 2022-07-03T10:00:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ch341.ko 2022-07-03T10:00:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/cp210x.ko 2022-07-03T10:00:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/cyberjack.ko 2022-07-03T10:00:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/cypress_m8.ko 2022-07-03T10:00:08.758 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/digi_acceleport.ko 2022-07-03T10:00:08.759 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/empeg.ko 2022-07-03T10:00:08.759 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ftdi_sio.ko 2022-07-03T10:00:08.759 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/garmin_gps.ko 2022-07-03T10:00:08.759 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/io_edgeport.ko 2022-07-03T10:00:08.759 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/io_ti.ko 2022-07-03T10:00:08.760 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ipaq.ko 2022-07-03T10:00:08.760 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ipw.ko 2022-07-03T10:00:08.760 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ir-usb.ko 2022-07-03T10:00:08.760 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-07-03T10:00:08.760 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/keyspan.ko 2022-07-03T10:00:08.761 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/keyspan_pda.ko 2022-07-03T10:00:08.761 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/kl5kusb105.ko 2022-07-03T10:00:08.761 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/kobil_sct.ko 2022-07-03T10:00:08.761 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/mct_u232.ko 2022-07-03T10:00:08.762 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/mos7720.ko 2022-07-03T10:00:08.762 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/mos7840.ko 2022-07-03T10:00:08.762 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/navman.ko 2022-07-03T10:00:08.762 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/omninet.ko 2022-07-03T10:00:08.762 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/opticon.ko 2022-07-03T10:00:08.763 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/option.ko 2022-07-03T10:00:08.763 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/oti6858.ko 2022-07-03T10:00:08.763 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/pl2303.ko 2022-07-03T10:00:08.763 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/qcaux.ko 2022-07-03T10:00:08.764 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/qcserial.ko 2022-07-03T10:00:08.764 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/quatech2.ko 2022-07-03T10:00:08.764 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/safe_serial.ko 2022-07-03T10:00:08.764 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/sierra.ko 2022-07-03T10:00:08.764 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/spcp8x5.ko 2022-07-03T10:00:08.765 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ssu100.ko 2022-07-03T10:00:08.765 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/symbolserial.ko 2022-07-03T10:00:08.765 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-07-03T10:00:08.765 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-07-03T10:00:08.765 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/usb_debug.ko 2022-07-03T10:00:08.766 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/usb_wwan.ko 2022-07-03T10:00:08.766 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/visor.ko 2022-07-03T10:00:08.766 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/whiteheat.ko 2022-07-03T10:00:08.766 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/serial/xsens_mt.ko 2022-07-03T10:00:08.767 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage 2022-07-03T10:00:08.767 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/uas.ko 2022-07-03T10:00:08.767 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-alauda.ko 2022-07-03T10:00:08.767 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-cypress.ko 2022-07-03T10:00:08.768 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-datafab.ko 2022-07-03T10:00:08.768 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-07-03T10:00:08.768 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-freecom.ko 2022-07-03T10:00:08.768 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-isd200.ko 2022-07-03T10:00:08.768 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-07-03T10:00:08.769 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-karma.ko 2022-07-03T10:00:08.769 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-onetouch.ko 2022-07-03T10:00:08.769 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-realtek.ko 2022-07-03T10:00:08.769 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-sddr09.ko 2022-07-03T10:00:08.769 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-sddr55.ko 2022-07-03T10:00:08.770 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/ums-usbat.ko 2022-07-03T10:00:08.770 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/storage/usb-storage.ko 2022-07-03T10:00:08.770 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/usbip 2022-07-03T10:00:08.770 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/usbip/usbip-core.ko 2022-07-03T10:00:08.770 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/usbip/usbip-host.ko 2022-07-03T10:00:08.771 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-07-03T10:00:08.771 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio 2022-07-03T10:00:08.771 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/pci 2022-07-03T10:00:08.771 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-07-03T10:00:08.771 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/pci/vfio-pci.ko 2022-07-03T10:00:08.772 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/vfio.ko 2022-07-03T10:00:08.772 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-07-03T10:00:08.772 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vfio/vfio_virqfd.ko 2022-07-03T10:00:08.772 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vhost 2022-07-03T10:00:08.773 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vhost/vhost.ko 2022-07-03T10:00:08.773 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vhost/vhost_iotlb.ko 2022-07-03T10:00:08.773 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vhost/vhost_net.ko 2022-07-03T10:00:08.773 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/vhost/vhost_scsi.ko 2022-07-03T10:00:08.773 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/video 2022-07-03T10:00:08.774 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/video/backlight 2022-07-03T10:00:08.774 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/video/backlight/apple_bl.ko 2022-07-03T10:00:08.774 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/video/backlight/lcd.ko 2022-07-03T10:00:08.774 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/video/backlight/platform_lcd.ko 2022-07-03T10:00:08.774 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio 2022-07-03T10:00:08.775 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio.ko 2022-07-03T10:00:08.775 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_balloon.ko 2022-07-03T10:00:08.775 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_mmio.ko 2022-07-03T10:00:08.775 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_pci.ko 2022-07-03T10:00:08.775 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-07-03T10:00:08.776 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-07-03T10:00:08.776 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/virtio/virtio_ring.ko 2022-07-03T10:00:08.776 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1 2022-07-03T10:00:08.776 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/masters 2022-07-03T10:00:08.776 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/masters/ds1wm.ko 2022-07-03T10:00:08.777 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/masters/ds2482.ko 2022-07-03T10:00:08.777 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/masters/ds2490.ko 2022-07-03T10:00:08.777 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves 2022-07-03T10:00:08.777 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-07-03T10:00:08.777 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-07-03T10:00:08.778 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-07-03T10:00:08.778 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-07-03T10:00:08.778 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-07-03T10:00:08.778 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-07-03T10:00:08.779 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-07-03T10:00:08.779 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-07-03T10:00:08.779 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-07-03T10:00:08.779 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_smem.ko 2022-07-03T10:00:08.779 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/slaves/w1_therm.ko 2022-07-03T10:00:08.780 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/w1/wire.ko 2022-07-03T10:00:08.780 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog 2022-07-03T10:00:08.780 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/alim1535_wdt.ko 2022-07-03T10:00:08.780 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/alim7101_wdt.ko 2022-07-03T10:00:08.780 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/f71808e_wdt.ko 2022-07-03T10:00:08.781 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/hpwdt.ko 2022-07-03T10:00:08.781 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/i6300esb.ko 2022-07-03T10:00:08.781 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-07-03T10:00:08.781 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/iTCO_wdt.ko 2022-07-03T10:00:08.781 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/ib700wdt.ko 2022-07-03T10:00:08.782 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/ibmasr.ko 2022-07-03T10:00:08.782 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-07-03T10:00:08.782 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/it8712f_wdt.ko 2022-07-03T10:00:08.782 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/it87_wdt.ko 2022-07-03T10:00:08.783 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/machzwd.ko 2022-07-03T10:00:08.783 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/nv_tco.ko 2022-07-03T10:00:08.783 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/pcwd_pci.ko 2022-07-03T10:00:08.783 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/pcwd_usb.ko 2022-07-03T10:00:08.783 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-07-03T10:00:08.784 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/sch311x_wdt.ko 2022-07-03T10:00:08.784 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/softdog.ko 2022-07-03T10:00:08.784 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/sp5100_tco.ko 2022-07-03T10:00:08.784 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/via_wdt.ko 2022-07-03T10:00:08.784 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-07-03T10:00:08.785 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/w83877f_wdt.ko 2022-07-03T10:00:08.785 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/w83977f_wdt.ko 2022-07-03T10:00:08.785 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/wdt_pci.ko 2022-07-03T10:00:08.785 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/watchdog/xen_wdt.ko 2022-07-03T10:00:08.785 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen 2022-07-03T10:00:08.786 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-acpi-processor.ko 2022-07-03T10:00:08.786 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-evtchn.ko 2022-07-03T10:00:08.786 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-gntalloc.ko 2022-07-03T10:00:08.786 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-gntdev.ko 2022-07-03T10:00:08.786 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-pciback 2022-07-03T10:00:08.787 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-07-03T10:00:08.787 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xen-privcmd.ko 2022-07-03T10:00:08.787 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xenfs 2022-07-03T10:00:08.787 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/drivers/xen/xenfs/xenfs.ko 2022-07-03T10:00:08.787 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs 2022-07-03T10:00:08.788 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/9p 2022-07-03T10:00:08.788 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/9p/9p.ko 2022-07-03T10:00:08.788 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/affs 2022-07-03T10:00:08.788 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/affs/affs.ko 2022-07-03T10:00:08.788 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/befs 2022-07-03T10:00:08.789 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/befs/befs.ko 2022-07-03T10:00:08.789 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/binfmt_misc.ko 2022-07-03T10:00:08.789 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/btrfs 2022-07-03T10:00:08.789 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/btrfs/btrfs.ko 2022-07-03T10:00:08.790 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cachefiles 2022-07-03T10:00:08.790 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cachefiles/cachefiles.ko 2022-07-03T10:00:08.790 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ceph 2022-07-03T10:00:08.790 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ceph/ceph.ko 2022-07-03T10:00:08.790 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cifs 2022-07-03T10:00:08.791 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cifs/cifs.ko 2022-07-03T10:00:08.791 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/coda 2022-07-03T10:00:08.791 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/coda/coda.ko 2022-07-03T10:00:08.791 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cramfs 2022-07-03T10:00:08.791 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/cramfs/cramfs.ko 2022-07-03T10:00:08.792 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/dlm 2022-07-03T10:00:08.792 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/dlm/dlm.ko 2022-07-03T10:00:08.792 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ecryptfs 2022-07-03T10:00:08.792 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ecryptfs/ecryptfs.ko 2022-07-03T10:00:08.793 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/f2fs 2022-07-03T10:00:08.793 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/f2fs/f2fs.ko 2022-07-03T10:00:08.793 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fat 2022-07-03T10:00:08.793 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fat/fat.ko 2022-07-03T10:00:08.793 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fat/msdos.ko 2022-07-03T10:00:08.794 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fat/vfat.ko 2022-07-03T10:00:08.794 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fscache 2022-07-03T10:00:08.794 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fscache/fscache.ko 2022-07-03T10:00:08.794 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fuse 2022-07-03T10:00:08.794 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fuse/cuse.ko 2022-07-03T10:00:08.795 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/fuse/fuse.ko 2022-07-03T10:00:08.795 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/gfs2 2022-07-03T10:00:08.795 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/gfs2/gfs2.ko 2022-07-03T10:00:08.795 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/hfs 2022-07-03T10:00:08.796 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/hfs/hfs.ko 2022-07-03T10:00:08.796 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/hfsplus 2022-07-03T10:00:08.796 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/hfsplus/hfsplus.ko 2022-07-03T10:00:08.796 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/isofs 2022-07-03T10:00:08.796 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/isofs/isofs.ko 2022-07-03T10:00:08.797 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/jfs 2022-07-03T10:00:08.797 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/jfs/jfs.ko 2022-07-03T10:00:08.797 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/lockd 2022-07-03T10:00:08.797 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/lockd/lockd.ko 2022-07-03T10:00:08.797 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/minix 2022-07-03T10:00:08.798 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/minix/minix.ko 2022-07-03T10:00:08.798 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/netfs 2022-07-03T10:00:08.798 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/netfs/netfs.ko 2022-07-03T10:00:08.798 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs 2022-07-03T10:00:08.798 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/blocklayout 2022-07-03T10:00:08.799 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-07-03T10:00:08.799 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/filelayout 2022-07-03T10:00:08.799 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-07-03T10:00:08.799 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/flexfilelayout 2022-07-03T10:00:08.799 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-07-03T10:00:08.800 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/nfs.ko 2022-07-03T10:00:08.800 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/nfsv3.ko 2022-07-03T10:00:08.800 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs/nfsv4.ko 2022-07-03T10:00:08.800 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs_common 2022-07-03T10:00:08.801 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs_common/grace.ko 2022-07-03T10:00:08.801 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfs_common/nfs_acl.ko 2022-07-03T10:00:08.801 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfsd 2022-07-03T10:00:08.801 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nfsd/nfsd.ko 2022-07-03T10:00:08.801 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nilfs2 2022-07-03T10:00:08.802 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nilfs2/nilfs2.ko 2022-07-03T10:00:08.802 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls 2022-07-03T10:00:08.802 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-celtic.ko 2022-07-03T10:00:08.802 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-centeuro.ko 2022-07-03T10:00:08.803 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-croatian.ko 2022-07-03T10:00:08.803 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-cyrillic.ko 2022-07-03T10:00:08.804 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-gaelic.ko 2022-07-03T10:00:08.804 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-greek.ko 2022-07-03T10:00:08.804 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-iceland.ko 2022-07-03T10:00:08.805 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-inuit.ko 2022-07-03T10:00:08.805 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-roman.ko 2022-07-03T10:00:08.805 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-romanian.ko 2022-07-03T10:00:08.805 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/mac-turkish.ko 2022-07-03T10:00:08.806 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp1250.ko 2022-07-03T10:00:08.806 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp1251.ko 2022-07-03T10:00:08.806 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp1255.ko 2022-07-03T10:00:08.807 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp737.ko 2022-07-03T10:00:08.807 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp775.ko 2022-07-03T10:00:08.807 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp850.ko 2022-07-03T10:00:08.807 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp852.ko 2022-07-03T10:00:08.807 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp855.ko 2022-07-03T10:00:08.808 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp857.ko 2022-07-03T10:00:08.808 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp860.ko 2022-07-03T10:00:08.808 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp861.ko 2022-07-03T10:00:08.808 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp862.ko 2022-07-03T10:00:08.808 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp863.ko 2022-07-03T10:00:08.809 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp864.ko 2022-07-03T10:00:08.809 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp865.ko 2022-07-03T10:00:08.809 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp866.ko 2022-07-03T10:00:08.809 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp869.ko 2022-07-03T10:00:08.809 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp874.ko 2022-07-03T10:00:08.810 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp932.ko 2022-07-03T10:00:08.810 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp936.ko 2022-07-03T10:00:08.810 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp949.ko 2022-07-03T10:00:08.810 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_cp950.ko 2022-07-03T10:00:08.811 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_euc-jp.ko 2022-07-03T10:00:08.811 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-1.ko 2022-07-03T10:00:08.811 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-13.ko 2022-07-03T10:00:08.811 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-14.ko 2022-07-03T10:00:08.812 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-15.ko 2022-07-03T10:00:08.812 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-2.ko 2022-07-03T10:00:08.812 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-3.ko 2022-07-03T10:00:08.812 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-4.ko 2022-07-03T10:00:08.812 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-5.ko 2022-07-03T10:00:08.813 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-6.ko 2022-07-03T10:00:08.813 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-7.ko 2022-07-03T10:00:08.813 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_iso8859-9.ko 2022-07-03T10:00:08.813 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_koi8-r.ko 2022-07-03T10:00:08.813 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_koi8-ru.ko 2022-07-03T10:00:08.814 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_koi8-u.ko 2022-07-03T10:00:08.814 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/nls/nls_utf8.ko 2022-07-03T10:00:08.814 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2 2022-07-03T10:00:08.814 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/cluster 2022-07-03T10:00:08.815 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-07-03T10:00:08.815 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/dlm 2022-07-03T10:00:08.815 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-07-03T10:00:08.816 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/dlmfs 2022-07-03T10:00:08.816 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-07-03T10:00:08.816 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/ocfs2.ko 2022-07-03T10:00:08.816 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-07-03T10:00:08.817 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-07-03T10:00:08.817 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-07-03T10:00:08.817 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/overlayfs 2022-07-03T10:00:08.817 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/overlayfs/overlay.ko 2022-07-03T10:00:08.818 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/pstore 2022-07-03T10:00:08.818 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/pstore/ramoops.ko 2022-07-03T10:00:08.818 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/reiserfs 2022-07-03T10:00:08.818 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/reiserfs/reiserfs.ko 2022-07-03T10:00:08.819 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/romfs 2022-07-03T10:00:08.819 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/romfs/romfs.ko 2022-07-03T10:00:08.819 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/smbfs_common 2022-07-03T10:00:08.819 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/smbfs_common/cifs_arc4.ko 2022-07-03T10:00:08.819 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/smbfs_common/cifs_md4.ko 2022-07-03T10:00:08.820 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/squashfs 2022-07-03T10:00:08.820 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/squashfs/squashfs.ko 2022-07-03T10:00:08.820 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/sysv 2022-07-03T10:00:08.820 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/sysv/sysv.ko 2022-07-03T10:00:08.820 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/udf 2022-07-03T10:00:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/udf/udf.ko 2022-07-03T10:00:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ufs 2022-07-03T10:00:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/ufs/ufs.ko 2022-07-03T10:00:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/xfs 2022-07-03T10:00:08.821 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/fs/xfs/xfs.ko 2022-07-03T10:00:08.822 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel 2022-07-03T10:00:08.822 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel/rcu 2022-07-03T10:00:08.822 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel/rcu/rcutorture.ko 2022-07-03T10:00:08.822 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel/torture.ko 2022-07-03T10:00:08.822 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel/trace 2022-07-03T10:00:08.823 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-07-03T10:00:08.823 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib 2022-07-03T10:00:08.823 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/asn1_encoder.ko 2022-07-03T10:00:08.823 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/crc-itu-t.ko 2022-07-03T10:00:08.824 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/crc8.ko 2022-07-03T10:00:08.824 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/crypto 2022-07-03T10:00:08.824 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/crypto/libarc4.ko 2022-07-03T10:00:08.824 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/crypto/libdes.ko 2022-07-03T10:00:08.824 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/libcrc32c.ko 2022-07-03T10:00:08.825 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/lru_cache.ko 2022-07-03T10:00:08.825 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/lz4 2022-07-03T10:00:08.825 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/lz4/lz4_compress.ko 2022-07-03T10:00:08.825 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/lz4/lz4hc_compress.ko 2022-07-03T10:00:08.826 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/math 2022-07-03T10:00:08.826 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/math/cordic.ko 2022-07-03T10:00:08.826 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/raid6 2022-07-03T10:00:08.826 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/raid6/raid6_pq.ko 2022-07-03T10:00:08.827 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/reed_solomon 2022-07-03T10:00:08.827 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/reed_solomon/reed_solomon.ko 2022-07-03T10:00:08.827 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/ts_bm.ko 2022-07-03T10:00:08.827 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/ts_fsm.ko 2022-07-03T10:00:08.828 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/ts_kmp.ko 2022-07-03T10:00:08.828 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/zstd 2022-07-03T10:00:08.828 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/lib/zstd/zstd_compress.ko 2022-07-03T10:00:08.828 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/mm 2022-07-03T10:00:08.828 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/mm/hwpoison-inject.ko 2022-07-03T10:00:08.829 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net 2022-07-03T10:00:08.829 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan 2022-07-03T10:00:08.829 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/6lowpan.ko 2022-07-03T10:00:08.830 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_dest.ko 2022-07-03T10:00:08.830 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_fragment.ko 2022-07-03T10:00:08.830 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_hop.ko 2022-07-03T10:00:08.831 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_ipv6.ko 2022-07-03T10:00:08.831 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_mobility.ko 2022-07-03T10:00:08.831 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_routing.ko 2022-07-03T10:00:08.832 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/6lowpan/nhc_udp.ko 2022-07-03T10:00:08.832 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802 2022-07-03T10:00:08.832 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802/garp.ko 2022-07-03T10:00:08.832 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802/mrp.ko 2022-07-03T10:00:08.832 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802/p8022.ko 2022-07-03T10:00:08.833 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802/psnap.ko 2022-07-03T10:00:08.833 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/802/stp.ko 2022-07-03T10:00:08.833 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/8021q 2022-07-03T10:00:08.833 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/8021q/8021q.ko 2022-07-03T10:00:08.833 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/9p 2022-07-03T10:00:08.834 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/9p/9pnet.ko 2022-07-03T10:00:08.834 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/9p/9pnet_fd.ko 2022-07-03T10:00:08.834 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/9p/9pnet_rdma.ko 2022-07-03T10:00:08.834 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/9p/9pnet_virtio.ko 2022-07-03T10:00:08.835 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/batman-adv 2022-07-03T10:00:08.835 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/batman-adv/batman-adv.ko 2022-07-03T10:00:08.835 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge 2022-07-03T10:00:08.835 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/br_netfilter.ko 2022-07-03T10:00:08.835 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/bridge.ko 2022-07-03T10:00:08.836 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter 2022-07-03T10:00:08.836 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-07-03T10:00:08.836 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_among.ko 2022-07-03T10:00:08.837 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_arp.ko 2022-07-03T10:00:08.837 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-07-03T10:00:08.837 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-07-03T10:00:08.837 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_ip.ko 2022-07-03T10:00:08.837 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-07-03T10:00:08.838 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_limit.ko 2022-07-03T10:00:08.838 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_log.ko 2022-07-03T10:00:08.838 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_mark.ko 2022-07-03T10:00:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-07-03T10:00:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-07-03T10:00:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-07-03T10:00:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-07-03T10:00:08.839 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_snat.ko 2022-07-03T10:00:08.840 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_stp.ko 2022-07-03T10:00:08.840 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-07-03T10:00:08.840 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-07-03T10:00:08.840 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-07-03T10:00:08.841 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-07-03T10:00:08.841 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/ebtables.ko 2022-07-03T10:00:08.841 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-07-03T10:00:08.841 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-07-03T10:00:08.841 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-07-03T10:00:08.842 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ceph 2022-07-03T10:00:08.842 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ceph/libceph.ko 2022-07-03T10:00:08.842 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/core 2022-07-03T10:00:08.842 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/core/failover.ko 2022-07-03T10:00:08.842 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/core/pktgen.ko 2022-07-03T10:00:08.843 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dccp 2022-07-03T10:00:08.843 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dccp/dccp.ko 2022-07-03T10:00:08.843 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dccp/dccp_diag.ko 2022-07-03T10:00:08.843 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dccp/dccp_ipv4.ko 2022-07-03T10:00:08.843 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dccp/dccp_ipv6.ko 2022-07-03T10:00:08.844 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dns_resolver 2022-07-03T10:00:08.844 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dns_resolver/dns_resolver.ko 2022-07-03T10:00:08.844 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dsa 2022-07-03T10:00:08.844 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dsa/dsa_core.ko 2022-07-03T10:00:08.845 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dsa/tag_brcm.ko 2022-07-03T10:00:08.845 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dsa/tag_dsa.ko 2022-07-03T10:00:08.845 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/dsa/tag_trailer.ko 2022-07-03T10:00:08.845 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ieee802154 2022-07-03T10:00:08.846 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ieee802154/6lowpan 2022-07-03T10:00:08.846 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-07-03T10:00:08.846 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ieee802154/ieee802154.ko 2022-07-03T10:00:08.846 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ieee802154/ieee802154_socket.ko 2022-07-03T10:00:08.846 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4 2022-07-03T10:00:08.847 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ah4.ko 2022-07-03T10:00:08.847 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/esp4.ko 2022-07-03T10:00:08.847 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/fou.ko 2022-07-03T10:00:08.847 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/gre.ko 2022-07-03T10:00:08.848 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/inet_diag.ko 2022-07-03T10:00:08.848 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ip_gre.ko 2022-07-03T10:00:08.848 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ip_tunnel.ko 2022-07-03T10:00:08.848 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ip_vti.ko 2022-07-03T10:00:08.848 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ipcomp.ko 2022-07-03T10:00:08.849 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/ipip.ko 2022-07-03T10:00:08.849 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter 2022-07-03T10:00:08.849 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/arp_tables.ko 2022-07-03T10:00:08.849 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-07-03T10:00:08.849 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-07-03T10:00:08.850 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-07-03T10:00:08.850 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-07-03T10:00:08.850 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-07-03T10:00:08.850 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-07-03T10:00:08.851 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-07-03T10:00:08.851 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-07-03T10:00:08.851 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-07-03T10:00:08.851 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-07-03T10:00:08.851 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/iptable_security.ko 2022-07-03T10:00:08.852 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-07-03T10:00:08.852 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-07-03T10:00:08.852 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-07-03T10:00:08.852 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-07-03T10:00:08.853 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-07-03T10:00:08.853 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-07-03T10:00:08.853 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-07-03T10:00:08.853 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-07-03T10:00:08.853 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-07-03T10:00:08.854 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-07-03T10:00:08.854 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_bic.ko 2022-07-03T10:00:08.854 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_dctcp.ko 2022-07-03T10:00:08.854 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_diag.ko 2022-07-03T10:00:08.855 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_highspeed.ko 2022-07-03T10:00:08.855 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_htcp.ko 2022-07-03T10:00:08.855 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_hybla.ko 2022-07-03T10:00:08.855 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_illinois.ko 2022-07-03T10:00:08.855 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_lp.ko 2022-07-03T10:00:08.856 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_scalable.ko 2022-07-03T10:00:08.856 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_vegas.ko 2022-07-03T10:00:08.856 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_veno.ko 2022-07-03T10:00:08.856 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_westwood.ko 2022-07-03T10:00:08.857 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tcp_yeah.ko 2022-07-03T10:00:08.857 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/tunnel4.ko 2022-07-03T10:00:08.857 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/udp_diag.ko 2022-07-03T10:00:08.857 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/udp_tunnel.ko 2022-07-03T10:00:08.857 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv4/xfrm4_tunnel.ko 2022-07-03T10:00:08.858 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6 2022-07-03T10:00:08.858 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/ah6.ko 2022-07-03T10:00:08.858 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/esp6.ko 2022-07-03T10:00:08.858 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/fou6.ko 2022-07-03T10:00:08.858 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/ip6_tunnel.ko 2022-07-03T10:00:08.859 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-07-03T10:00:08.859 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/ip6_vti.ko 2022-07-03T10:00:08.859 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/ipcomp6.ko 2022-07-03T10:00:08.859 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter 2022-07-03T10:00:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-07-03T10:00:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-07-03T10:00:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-07-03T10:00:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-07-03T10:00:08.860 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-07-03T10:00:08.861 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-07-03T10:00:08.861 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-07-03T10:00:08.861 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-07-03T10:00:08.861 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-07-03T10:00:08.861 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-07-03T10:00:08.862 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-07-03T10:00:08.862 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-07-03T10:00:08.862 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-07-03T10:00:08.862 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-07-03T10:00:08.862 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-07-03T10:00:08.863 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-07-03T10:00:08.863 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-07-03T10:00:08.863 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-07-03T10:00:08.863 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-07-03T10:00:08.864 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-07-03T10:00:08.864 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-07-03T10:00:08.864 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-07-03T10:00:08.864 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-07-03T10:00:08.864 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-07-03T10:00:08.865 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/sit.ko 2022-07-03T10:00:08.865 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/tunnel6.ko 2022-07-03T10:00:08.865 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/ipv6/xfrm6_tunnel.ko 2022-07-03T10:00:08.865 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/key 2022-07-03T10:00:08.865 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/key/af_key.ko 2022-07-03T10:00:08.866 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp 2022-07-03T10:00:08.866 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_core.ko 2022-07-03T10:00:08.866 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_debugfs.ko 2022-07-03T10:00:08.866 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_eth.ko 2022-07-03T10:00:08.866 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_ip.ko 2022-07-03T10:00:08.867 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_ip6.ko 2022-07-03T10:00:08.867 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_netlink.ko 2022-07-03T10:00:08.867 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/l2tp/l2tp_ppp.ko 2022-07-03T10:00:08.867 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/llc 2022-07-03T10:00:08.867 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/llc/llc.ko 2022-07-03T10:00:08.868 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/mac802154 2022-07-03T10:00:08.868 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/mac802154/mac802154.ko 2022-07-03T10:00:08.868 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/mpls 2022-07-03T10:00:08.868 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/mpls/mpls_gso.ko 2022-07-03T10:00:08.868 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter 2022-07-03T10:00:08.869 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset 2022-07-03T10:00:08.869 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set.ko 2022-07-03T10:00:08.869 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-07-03T10:00:08.869 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-07-03T10:00:08.870 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-07-03T10:00:08.870 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-07-03T10:00:08.870 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-07-03T10:00:08.870 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-07-03T10:00:08.870 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-07-03T10:00:08.871 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-07-03T10:00:08.871 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-07-03T10:00:08.871 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-07-03T10:00:08.871 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-07-03T10:00:08.871 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-07-03T10:00:08.872 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-07-03T10:00:08.872 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-07-03T10:00:08.872 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-07-03T10:00:08.872 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs 2022-07-03T10:00:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs.ko 2022-07-03T10:00:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-07-03T10:00:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-07-03T10:00:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-07-03T10:00:08.873 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-07-03T10:00:08.874 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-07-03T10:00:08.874 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-07-03T10:00:08.874 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-07-03T10:00:08.874 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-07-03T10:00:08.874 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-07-03T10:00:08.875 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-07-03T10:00:08.875 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-07-03T10:00:08.875 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-07-03T10:00:08.875 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-07-03T10:00:08.875 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conncount.ko 2022-07-03T10:00:08.876 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack.ko 2022-07-03T10:00:08.876 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-07-03T10:00:08.876 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-07-03T10:00:08.876 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-07-03T10:00:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_h323.ko 2022-07-03T10:00:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_irc.ko 2022-07-03T10:00:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-07-03T10:00:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-07-03T10:00:08.877 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-07-03T10:00:08.878 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_sane.ko 2022-07-03T10:00:08.878 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_sip.ko 2022-07-03T10:00:08.878 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-07-03T10:00:08.878 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-07-03T10:00:08.878 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_dup_netdev.ko 2022-07-03T10:00:08.879 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_flow_table.ko 2022-07-03T10:00:08.879 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_flow_table_inet.ko 2022-07-03T10:00:08.879 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_log_syslog.ko 2022-07-03T10:00:08.879 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat.ko 2022-07-03T10:00:08.879 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat_amanda.ko 2022-07-03T10:00:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat_ftp.ko 2022-07-03T10:00:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat_irc.ko 2022-07-03T10:00:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat_sip.ko 2022-07-03T10:00:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_nat_tftp.ko 2022-07-03T10:00:08.880 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_synproxy_core.ko 2022-07-03T10:00:08.881 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nf_tables.ko 2022-07-03T10:00:08.881 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nfnetlink.ko 2022-07-03T10:00:08.881 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nfnetlink_acct.ko 2022-07-03T10:00:08.881 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nfnetlink_log.ko 2022-07-03T10:00:08.882 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nfnetlink_osf.ko 2022-07-03T10:00:08.882 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nfnetlink_queue.ko 2022-07-03T10:00:08.882 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_chain_nat.ko 2022-07-03T10:00:08.882 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_compat.ko 2022-07-03T10:00:08.882 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_ct.ko 2022-07-03T10:00:08.883 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_dup_netdev.ko 2022-07-03T10:00:08.883 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_fib.ko 2022-07-03T10:00:08.883 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_fwd_netdev.ko 2022-07-03T10:00:08.883 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_hash.ko 2022-07-03T10:00:08.883 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_limit.ko 2022-07-03T10:00:08.884 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_log.ko 2022-07-03T10:00:08.884 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_masq.ko 2022-07-03T10:00:08.884 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_nat.ko 2022-07-03T10:00:08.884 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_numgen.ko 2022-07-03T10:00:08.884 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_objref.ko 2022-07-03T10:00:08.885 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_queue.ko 2022-07-03T10:00:08.885 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_quota.ko 2022-07-03T10:00:08.885 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_redir.ko 2022-07-03T10:00:08.885 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_reject.ko 2022-07-03T10:00:08.885 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_reject_inet.ko 2022-07-03T10:00:08.886 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_socket.ko 2022-07-03T10:00:08.886 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/nft_tunnel.ko 2022-07-03T10:00:08.886 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_AUDIT.ko 2022-07-03T10:00:08.886 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_CHECKSUM.ko 2022-07-03T10:00:08.887 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_CLASSIFY.ko 2022-07-03T10:00:08.887 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-07-03T10:00:08.887 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_CT.ko 2022-07-03T10:00:08.887 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_DSCP.ko 2022-07-03T10:00:08.887 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_HL.ko 2022-07-03T10:00:08.888 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_HMARK.ko 2022-07-03T10:00:08.888 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_IDLETIMER.ko 2022-07-03T10:00:08.888 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_LED.ko 2022-07-03T10:00:08.888 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_LOG.ko 2022-07-03T10:00:08.888 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_MASQUERADE.ko 2022-07-03T10:00:08.889 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_NETMAP.ko 2022-07-03T10:00:08.889 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_NFLOG.ko 2022-07-03T10:00:08.889 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_NFQUEUE.ko 2022-07-03T10:00:08.889 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_RATEEST.ko 2022-07-03T10:00:08.889 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_REDIRECT.ko 2022-07-03T10:00:08.890 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_SECMARK.ko 2022-07-03T10:00:08.890 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_TCPMSS.ko 2022-07-03T10:00:08.890 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-07-03T10:00:08.890 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_TEE.ko 2022-07-03T10:00:08.891 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_TPROXY.ko 2022-07-03T10:00:08.891 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_TRACE.ko 2022-07-03T10:00:08.891 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_addrtype.ko 2022-07-03T10:00:08.891 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_bpf.ko 2022-07-03T10:00:08.891 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_cgroup.ko 2022-07-03T10:00:08.892 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_cluster.ko 2022-07-03T10:00:08.892 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_comment.ko 2022-07-03T10:00:08.892 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_connbytes.ko 2022-07-03T10:00:08.892 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_connlabel.ko 2022-07-03T10:00:08.892 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_connlimit.ko 2022-07-03T10:00:08.893 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_connmark.ko 2022-07-03T10:00:08.893 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_conntrack.ko 2022-07-03T10:00:08.893 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_cpu.ko 2022-07-03T10:00:08.893 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_dccp.ko 2022-07-03T10:00:08.893 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_devgroup.ko 2022-07-03T10:00:08.894 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_dscp.ko 2022-07-03T10:00:08.894 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_ecn.ko 2022-07-03T10:00:08.894 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_esp.ko 2022-07-03T10:00:08.894 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_hashlimit.ko 2022-07-03T10:00:08.895 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_helper.ko 2022-07-03T10:00:08.895 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_hl.ko 2022-07-03T10:00:08.895 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_ipcomp.ko 2022-07-03T10:00:08.895 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_iprange.ko 2022-07-03T10:00:08.896 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_ipvs.ko 2022-07-03T10:00:08.896 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_l2tp.ko 2022-07-03T10:00:08.896 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_length.ko 2022-07-03T10:00:08.897 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_limit.ko 2022-07-03T10:00:08.897 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_mac.ko 2022-07-03T10:00:08.897 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_mark.ko 2022-07-03T10:00:08.897 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_multiport.ko 2022-07-03T10:00:08.898 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_nat.ko 2022-07-03T10:00:08.898 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_nfacct.ko 2022-07-03T10:00:08.898 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_osf.ko 2022-07-03T10:00:08.898 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_owner.ko 2022-07-03T10:00:08.898 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_physdev.ko 2022-07-03T10:00:08.899 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_pkttype.ko 2022-07-03T10:00:08.899 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_policy.ko 2022-07-03T10:00:08.899 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_quota.ko 2022-07-03T10:00:08.899 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_rateest.ko 2022-07-03T10:00:08.900 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_realm.ko 2022-07-03T10:00:08.900 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_recent.ko 2022-07-03T10:00:08.900 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_sctp.ko 2022-07-03T10:00:08.900 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_set.ko 2022-07-03T10:00:08.901 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_socket.ko 2022-07-03T10:00:08.901 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_state.ko 2022-07-03T10:00:08.901 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_statistic.ko 2022-07-03T10:00:08.901 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_string.ko 2022-07-03T10:00:08.901 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_tcpmss.ko 2022-07-03T10:00:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_time.ko 2022-07-03T10:00:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netfilter/xt_u32.ko 2022-07-03T10:00:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netlink 2022-07-03T10:00:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/netlink/netlink_diag.ko 2022-07-03T10:00:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/nsh 2022-07-03T10:00:08.903 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/nsh/nsh.ko 2022-07-03T10:00:08.903 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/openvswitch 2022-07-03T10:00:08.903 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/openvswitch/openvswitch.ko 2022-07-03T10:00:08.903 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/openvswitch/vport-geneve.ko 2022-07-03T10:00:08.904 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/openvswitch/vport-gre.ko 2022-07-03T10:00:08.904 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/openvswitch/vport-vxlan.ko 2022-07-03T10:00:08.904 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/packet 2022-07-03T10:00:08.905 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/packet/af_packet_diag.ko 2022-07-03T10:00:08.905 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/rds 2022-07-03T10:00:08.905 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/rds/rds.ko 2022-07-03T10:00:08.905 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/rds/rds_rdma.ko 2022-07-03T10:00:08.906 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/rds/rds_tcp.ko 2022-07-03T10:00:08.906 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched 2022-07-03T10:00:08.906 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_csum.ko 2022-07-03T10:00:08.906 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_gact.ko 2022-07-03T10:00:08.907 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_ipt.ko 2022-07-03T10:00:08.907 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_mirred.ko 2022-07-03T10:00:08.907 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_nat.ko 2022-07-03T10:00:08.907 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_pedit.ko 2022-07-03T10:00:08.907 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_police.ko 2022-07-03T10:00:08.908 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_simple.ko 2022-07-03T10:00:08.908 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_skbedit.ko 2022-07-03T10:00:08.908 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/act_vlan.ko 2022-07-03T10:00:08.908 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_basic.ko 2022-07-03T10:00:08.908 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_bpf.ko 2022-07-03T10:00:08.909 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_flow.ko 2022-07-03T10:00:08.909 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_fw.ko 2022-07-03T10:00:08.909 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_route.ko 2022-07-03T10:00:08.909 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_rsvp.ko 2022-07-03T10:00:08.909 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_rsvp6.ko 2022-07-03T10:00:08.910 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_tcindex.ko 2022-07-03T10:00:08.910 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/cls_u32.ko 2022-07-03T10:00:08.910 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_cmp.ko 2022-07-03T10:00:08.910 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_ipset.ko 2022-07-03T10:00:08.910 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_meta.ko 2022-07-03T10:00:08.911 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_nbyte.ko 2022-07-03T10:00:08.911 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_text.ko 2022-07-03T10:00:08.911 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/em_u32.ko 2022-07-03T10:00:08.911 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_cbq.ko 2022-07-03T10:00:08.911 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_choke.ko 2022-07-03T10:00:08.912 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_codel.ko 2022-07-03T10:00:08.912 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_drr.ko 2022-07-03T10:00:08.912 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_dsmark.ko 2022-07-03T10:00:08.912 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_fq.ko 2022-07-03T10:00:08.912 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_gred.ko 2022-07-03T10:00:08.913 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_hfsc.ko 2022-07-03T10:00:08.913 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_hhf.ko 2022-07-03T10:00:08.913 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_htb.ko 2022-07-03T10:00:08.913 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_ingress.ko 2022-07-03T10:00:08.914 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_mqprio.ko 2022-07-03T10:00:08.914 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_multiq.ko 2022-07-03T10:00:08.914 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_netem.ko 2022-07-03T10:00:08.914 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_pie.ko 2022-07-03T10:00:08.914 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_plug.ko 2022-07-03T10:00:08.915 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_prio.ko 2022-07-03T10:00:08.915 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_qfq.ko 2022-07-03T10:00:08.915 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_red.ko 2022-07-03T10:00:08.915 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_sfb.ko 2022-07-03T10:00:08.915 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_sfq.ko 2022-07-03T10:00:08.916 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_tbf.ko 2022-07-03T10:00:08.916 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sched/sch_teql.ko 2022-07-03T10:00:08.916 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sctp 2022-07-03T10:00:08.916 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sctp/sctp.ko 2022-07-03T10:00:08.916 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sctp/sctp_diag.ko 2022-07-03T10:00:08.917 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc 2022-07-03T10:00:08.917 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/auth_gss 2022-07-03T10:00:08.917 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-07-03T10:00:08.917 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-07-03T10:00:08.917 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/sunrpc.ko 2022-07-03T10:00:08.918 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/xprtrdma 2022-07-03T10:00:08.918 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-07-03T10:00:08.918 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/tipc 2022-07-03T10:00:08.918 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/tipc/diag.ko 2022-07-03T10:00:08.919 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/tipc/tipc.ko 2022-07-03T10:00:08.919 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/unix 2022-07-03T10:00:08.919 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/unix/unix_diag.ko 2022-07-03T10:00:08.919 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock 2022-07-03T10:00:08.919 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-07-03T10:00:08.920 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-07-03T10:00:08.920 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock/vsock.ko 2022-07-03T10:00:08.920 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock/vsock_diag.ko 2022-07-03T10:00:08.920 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/vmw_vsock/vsock_loopback.ko 2022-07-03T10:00:08.920 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/xfrm 2022-07-03T10:00:08.921 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/net/xfrm/xfrm_ipcomp.ko 2022-07-03T10:00:08.921 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security 2022-07-03T10:00:08.921 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security/keys 2022-07-03T10:00:08.921 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security/keys/encrypted-keys 2022-07-03T10:00:08.921 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-07-03T10:00:08.922 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security/keys/trusted-keys 2022-07-03T10:00:08.922 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/security/keys/trusted-keys/trusted.ko 2022-07-03T10:00:08.922 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/virt 2022-07-03T10:00:08.922 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/virt/lib 2022-07-03T10:00:08.923 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/kernel/virt/lib/irqbypass.ko 2022-07-03T10:00:08.923 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.alias 2022-07-03T10:00:08.923 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.alias.bin 2022-07-03T10:00:08.923 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.builtin 2022-07-03T10:00:08.923 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.builtin.bin 2022-07-03T10:00:08.924 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.builtin.modinfo 2022-07-03T10:00:08.924 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.dep 2022-07-03T10:00:08.924 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.dep.bin 2022-07-03T10:00:08.924 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.devname 2022-07-03T10:00:08.924 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.order 2022-07-03T10:00:08.925 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.softdep 2022-07-03T10:00:08.925 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.symbols 2022-07-03T10:00:08.925 INFO:teuthology.orchestra.run.smithi101.stdout:/lib/modules/5.19.0-rc4-ceph-g1c8086aad759/modules.symbols.bin 2022-07-03T10:00:08.925 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/lib/.build-id 2022-07-03T10:00:08.925 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/lib/.build-id/db 2022-07-03T10:00:08.926 INFO:teuthology.orchestra.run.smithi101.stdout:/usr/lib/.build-id/db/ab8ff14b786fe111c66bc761f735a0032bc016 2022-07-03T10:00:08.926 DEBUG:teuthology.task.kernel:get_image_version: 5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.927 DEBUG:teuthology.orchestra.run.smithi101:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-07-03T10:00:08.955 INFO:teuthology.orchestra.run.smithi101.stdout:postinstall scriptlet (using /bin/sh): 2022-07-03T10:00:08.955 INFO:teuthology.orchestra.run.smithi101.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.19.0-rc4-ceph-g1c8086aad759 -a -r /boot/System.map-5.19.0-rc4-ceph-g1c8086aad759 ]; then 2022-07-03T10:00:08.956 INFO:teuthology.orchestra.run.smithi101.stdout:cp /boot/vmlinuz-5.19.0-rc4-ceph-g1c8086aad759 /boot/.vmlinuz-5.19.0-rc4-ceph-g1c8086aad759-rpm 2022-07-03T10:00:08.956 INFO:teuthology.orchestra.run.smithi101.stdout:cp /boot/System.map-5.19.0-rc4-ceph-g1c8086aad759 /boot/.System.map-5.19.0-rc4-ceph-g1c8086aad759-rpm 2022-07-03T10:00:08.956 INFO:teuthology.orchestra.run.smithi101.stdout:rm -f /boot/vmlinuz-5.19.0-rc4-ceph-g1c8086aad759 /boot/System.map-5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.956 INFO:teuthology.orchestra.run.smithi101.stdout:/sbin/installkernel 5.19.0-rc4-ceph-g1c8086aad759 /boot/.vmlinuz-5.19.0-rc4-ceph-g1c8086aad759-rpm /boot/.System.map-5.19.0-rc4-ceph-g1c8086aad759-rpm 2022-07-03T10:00:08.957 INFO:teuthology.orchestra.run.smithi101.stdout:rm -f /boot/.vmlinuz-5.19.0-rc4-ceph-g1c8086aad759-rpm /boot/.System.map-5.19.0-rc4-ceph-g1c8086aad759-rpm 2022-07-03T10:00:08.957 INFO:teuthology.orchestra.run.smithi101.stdout:fi 2022-07-03T10:00:08.957 INFO:teuthology.orchestra.run.smithi101.stdout:preuninstall scriptlet (using /bin/sh): 2022-07-03T10:00:08.957 INFO:teuthology.orchestra.run.smithi101.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-07-03T10:00:08.958 INFO:teuthology.orchestra.run.smithi101.stdout:new-kernel-pkg --remove 5.19.0-rc4-ceph-g1c8086aad759 --rminitrd --initrdfile=/boot/initramfs-5.19.0-rc4-ceph-g1c8086aad759.img 2022-07-03T10:00:08.958 INFO:teuthology.orchestra.run.smithi101.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-07-03T10:00:08.958 INFO:teuthology.orchestra.run.smithi101.stdout:kernel-install remove 5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.959 INFO:teuthology.orchestra.run.smithi101.stdout:fi 2022-07-03T10:00:08.959 INFO:teuthology.orchestra.run.smithi101.stdout:postuninstall scriptlet (using /bin/sh): 2022-07-03T10:00:08.959 INFO:teuthology.orchestra.run.smithi101.stdout:if [ -x /sbin/update-bootloader ]; then 2022-07-03T10:00:08.959 INFO:teuthology.orchestra.run.smithi101.stdout:/sbin/update-bootloader --remove 5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:08.960 INFO:teuthology.orchestra.run.smithi101.stdout:fi 2022-07-03T10:00:08.961 DEBUG:teuthology.orchestra.run.smithi101:> sudo rpm -qi grub2-tools 2022-07-03T10:00:09.041 INFO:teuthology.orchestra.run.smithi101.stdout:Name : grub2-tools 2022-07-03T10:00:09.042 INFO:teuthology.orchestra.run.smithi101.stdout:Epoch : 1 2022-07-03T10:00:09.042 INFO:teuthology.orchestra.run.smithi101.stdout:Version : 2.02 2022-07-03T10:00:09.042 INFO:teuthology.orchestra.run.smithi101.stdout:Release : 123.el8 2022-07-03T10:00:09.042 INFO:teuthology.orchestra.run.smithi101.stdout:Architecture: x86_64 2022-07-03T10:00:09.042 INFO:teuthology.orchestra.run.smithi101.stdout:Install Date: Tue 28 Jun 2022 03:31:14 PM UTC 2022-07-03T10:00:09.043 INFO:teuthology.orchestra.run.smithi101.stdout:Group : System Environment/Base 2022-07-03T10:00:09.043 INFO:teuthology.orchestra.run.smithi101.stdout:Size : 9265450 2022-07-03T10:00:09.043 INFO:teuthology.orchestra.run.smithi101.stdout:License : GPLv3+ 2022-07-03T10:00:09.043 INFO:teuthology.orchestra.run.smithi101.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2022-07-03T10:00:09.044 INFO:teuthology.orchestra.run.smithi101.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2022-07-03T10:00:09.044 INFO:teuthology.orchestra.run.smithi101.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2022-07-03T10:00:09.044 INFO:teuthology.orchestra.run.smithi101.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2022-07-03T10:00:09.044 INFO:teuthology.orchestra.run.smithi101.stdout:Relocations : (not relocatable) 2022-07-03T10:00:09.044 INFO:teuthology.orchestra.run.smithi101.stdout:Packager : Red Hat, Inc. 2022-07-03T10:00:09.045 INFO:teuthology.orchestra.run.smithi101.stdout:Vendor : Red Hat, Inc. 2022-07-03T10:00:09.045 INFO:teuthology.orchestra.run.smithi101.stdout:URL : http://www.gnu.org/software/grub/ 2022-07-03T10:00:09.045 INFO:teuthology.orchestra.run.smithi101.stdout:Summary : Support tools for GRUB. 2022-07-03T10:00:09.045 INFO:teuthology.orchestra.run.smithi101.stdout:Description : 2022-07-03T10:00:09.046 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:00:09.046 INFO:teuthology.orchestra.run.smithi101.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-07-03T10:00:09.046 INFO:teuthology.orchestra.run.smithi101.stdout:customizable bootloader with modular architecture. It supports a rich 2022-07-03T10:00:09.046 INFO:teuthology.orchestra.run.smithi101.stdout:variety of kernel formats, file systems, computer architectures and 2022-07-03T10:00:09.046 INFO:teuthology.orchestra.run.smithi101.stdout:hardware devices. 2022-07-03T10:00:09.047 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:00:09.047 INFO:teuthology.orchestra.run.smithi101.stdout:This subpackage provides tools for support of all platforms. 2022-07-03T10:00:09.048 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-07-03T10:00:09.048 INFO:teuthology.task.kernel:Updating grub on smithi101 to boot 5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:09.049 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-07-03T10:00:12.505 INFO:teuthology.orchestra.run.smithi101.stderr:Generating grub configuration file ... 2022-07-03T10:00:15.067 INFO:teuthology.orchestra.run.smithi101.stderr:done 2022-07-03T10:00:15.068 DEBUG:teuthology.orchestra.run.smithi101:> mktemp 2022-07-03T10:00:15.114 INFO:teuthology.orchestra.run.smithi101.stdout:/tmp/tmp.FjvpdBglPx 2022-07-03T10:00:15.115 DEBUG:teuthology.orchestra.run.smithi101:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FjvpdBglPx 2022-07-03T10:00:15.149 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0666 /tmp/tmp.FjvpdBglPx 2022-07-03T10:00:15.317 DEBUG:teuthology.orchestra.remote:smithi101:/tmp/tmp.FjvpdBglPx is 6KB 2022-07-03T10:00:15.366 DEBUG:teuthology.orchestra.run.smithi101:> rm -fr /tmp/tmp.FjvpdBglPx 2022-07-03T10:00:15.384 DEBUG:teuthology.orchestra.run.smithi101:> sudo /bin/ls /boot/loader/entries || true 2022-07-03T10:00:15.451 INFO:teuthology.orchestra.run.smithi101.stdout:d2015723b439415ead6b8a39910a4af9-0-rescue.conf 2022-07-03T10:00:15.452 INFO:teuthology.orchestra.run.smithi101.stdout:d2015723b439415ead6b8a39910a4af9-4.18.0-372.9.1.el8.x86_64.conf 2022-07-03T10:00:15.452 INFO:teuthology.orchestra.run.smithi101.stdout:d2015723b439415ead6b8a39910a4af9-5.19.0-rc4-ceph-g1c8086aad759.conf 2022-07-03T10:00:15.454 DEBUG:teuthology.orchestra.run.smithi101:> sudo grub2-set-default d2015723b439415ead6b8a39910a4af9-5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:00:15.628 DEBUG:teuthology.orchestra.run.smithi101:> sudo shutdown -r now 2022-07-03T10:00:45.632 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-07-03T10:00:45.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-07-03T10:00:45.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:01:03.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.101 2022-07-03T10:01:33.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-07-03T10:01:33.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:01:52.378 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.101 2022-07-03T10:02:22.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-07-03T10:02:22.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:02:22.862 DEBUG:teuthology.orchestra.run.smithi101:> true 2022-07-03T10:02:23.254 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-07-03T10:02:23.254 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.19.0-rc4-ceph-g1c8086aad759"... 2022-07-03T10:02:23.255 DEBUG:teuthology.orchestra.run.smithi101:> uname -r 2022-07-03T10:02:23.312 INFO:teuthology.orchestra.run.smithi101.stdout:5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:02:23.313 DEBUG:teuthology.task.kernel:current kernel version is 5.19.0-rc4-ceph-g1c8086aad759 vs 5.19.0-rc4-ceph-g1c8086aad759 2022-07-03T10:02:23.313 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-03T10:02:23.313 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-07-03T10:02:24.315 DEBUG:teuthology.parallel:result is None 2022-07-03T10:02:24.316 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-03T10:02:24.339 INFO:teuthology.task.internal:Creating test directory... 2022-07-03T10:02:24.340 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-03T10:02:24.345 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-03T10:02:24.349 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-03T10:02:24.399 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-03T10:02:24.416 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-03T10:02:24.438 INFO:teuthology.task.internal:Creating archive directory... 2022-07-03T10:02:24.439 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-03T10:02:24.444 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-03T10:02:24.447 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-03T10:02:24.707 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-03T10:02:24.718 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-03T10:02:24.719 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-03T10:02:24.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:02:24.739 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-03T10:02:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:02:24.759 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-03T10:02:24.766 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-03T10:02:24.782 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-03T10:02:24.827 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.846 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.850 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.870 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.901 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.943 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-03T10:02:24.949 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-03T10:02:24.966 INFO:teuthology.task.internal:Configuring sudo... 2022-07-03T10:02:24.967 DEBUG:teuthology.orchestra.run.smithi101:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-03T10:02:24.972 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-03T10:02:24.975 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-03T10:02:25.056 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-03T10:02:25.070 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-03T10:02:25.071 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-03T10:02:25.077 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-03T10:02:25.079 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-03T10:02:25.129 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.221 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.308 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:25.394 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:25.481 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:02:25.482 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-03T10:02:25.569 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.671 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:25.757 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:25.801 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:02:25.801 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-03T10:02:25.870 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-03T10:02:25.970 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:26.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-03T10:02:26.101 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:02:26.102 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-03T10:02:26.172 DEBUG:teuthology.orchestra.run.smithi101:> sudo service rsyslog restart 2022-07-03T10:02:26.177 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2022-07-03T10:02:26.180 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2022-07-03T10:02:26.233 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-07-03T10:02:26.262 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-07-03T10:02:26.302 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-07-03T10:02:26.776 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-03T10:02:26.790 INFO:teuthology.task.internal:Starting timer... 2022-07-03T10:02:26.790 INFO:teuthology.run_tasks:Running task pcp... 2022-07-03T10:02:26.817 INFO:teuthology.run_tasks:Running task selinux... 2022-07-03T10:02:26.835 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-07-03T10:02:26.835 DEBUG:teuthology.orchestra.run.smithi101:> sudo service auditd rotate 2022-07-03T10:02:26.984 INFO:teuthology.orchestra.run.smithi101.stdout:Rotating logs: 2022-07-03T10:02:26.990 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2022-07-03T10:02:27.047 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2022-07-03T10:02:27.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2022-07-03T10:02:27.103 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2022-07-03T10:02:27.105 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-03T10:02:27.106 DEBUG:teuthology.orchestra.run.smithi101:> /usr/sbin/getenforce 2022-07-03T10:02:27.180 INFO:teuthology.orchestra.run.smithi101.stdout:Permissive 2022-07-03T10:02:27.181 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2022-07-03T10:02:27.217 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2022-07-03T10:02:27.218 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2022-07-03T10:02:27.236 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2022-07-03T10:02:27.237 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi101.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2022-07-03T10:02:27.237 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-07-03T10:02:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:02:27.328 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-07-03T10:02:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:02:27.357 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-07-03T10:02:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:02:27.385 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-03T10:02:27.386 DEBUG:teuthology.orchestra.run.smithi101:> sudo /usr/sbin/setenforce permissive 2022-07-03T10:02:27.475 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2022-07-03T10:02:27.503 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2022-07-03T10:02:27.531 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-03T10:02:27.551 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-07-03T10:02:27.559 INFO:teuthology.repo_utils:Fetching main from origin 2022-07-03T10:02:27.605 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-03T10:02:27.631 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-03T10:02:27.633 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi101.front.sepia.ceph.com,smithi161.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-03T10:13:10.051 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi101.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2022-07-03T10:13:10.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2022-07-03T10:13:10.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:13:10.165 DEBUG:teuthology.orchestra.run.smithi101:> true 2022-07-03T10:13:10.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-07-03T10:13:10.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:13:10.367 DEBUG:teuthology.orchestra.run.smithi161:> true 2022-07-03T10:13:10.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2022-07-03T10:13:10.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-03T10:13:10.525 DEBUG:teuthology.orchestra.run.smithi188:> true 2022-07-03T10:13:10.593 INFO:teuthology.run_tasks:Running task clock... 2022-07-03T10:13:10.609 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-03T10:13:10.610 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-03T10:13:10.613 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-03T10:13:10.618 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-03T10:13:10.618 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-03T10:13:10.621 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-03T10:13:10.621 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-03T10:13:10.656 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-07-03T10:13:10.667 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-07-03T10:13:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-07-03T10:13:10.682 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-07-03T10:13:10.704 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-07-03T10:13:10.737 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2022-07-03T10:13:10.745 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-07-03T10:13:10.750 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2022-07-03T10:13:10.763 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2022-07-03T10:13:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-07-03T10:13:10.775 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2022-07-03T10:13:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-07-03T10:13:10.789 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-07-03T10:13:10.803 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-07-03T10:13:10.847 INFO:teuthology.orchestra.run.smithi101.stderr:sudo: ntpd: command not found 2022-07-03T10:13:10.857 INFO:teuthology.orchestra.run.smithi161.stderr:bash: ntpq: command not found 2022-07-03T10:13:10.861 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-07-03T10:13:10.861 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2022-07-03T10:13:10.861 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.861 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.862 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.862 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.881 INFO:teuthology.orchestra.run.smithi101.stdout:506 Cannot talk to daemon 2022-07-03T10:13:10.889 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2022-07-03T10:13:10.892 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-07-03T10:13:10.892 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2022-07-03T10:13:10.893 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.893 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.893 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.893 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:10.921 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-07-03T10:13:10.961 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-07-03T10:13:11.061 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2022-07-03T10:13:11.070 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-07-03T10:13:11.071 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2022-07-03T10:13:11.071 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:11.071 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:11.071 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:11.072 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-07-03T10:13:11.073 INFO:teuthology.run_tasks:Running task pexec... 2022-07-03T10:13:11.083 INFO:teuthology.task.pexec:Executing custom commands... 2022-07-03T10:13:11.084 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi101.front.sepia.ceph.com 2022-07-03T10:13:11.084 DEBUG:teuthology.orchestra.run.smithi101:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-07-03T10:13:11.085 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi161.front.sepia.ceph.com 2022-07-03T10:13:11.085 DEBUG:teuthology.orchestra.run.smithi161:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-07-03T10:13:11.086 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2022-07-03T10:13:11.086 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-07-03T10:13:11.461 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:11.463 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:11.607 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:12.276 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2022-07-03T10:13:12.359 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2022-07-03T10:13:12.520 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2022-07-03T10:13:12.550 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 87 kB/s | 2.4 kB 00:00 2022-07-03T10:13:12.551 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2022-07-03T10:13:12.845 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2022-07-03T10:13:14.293 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2022-07-03T10:13:14.294 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2022-07-03T10:13:14.294 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2022-07-03T10:13:14.295 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2022-07-03T10:13:14.295 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2022-07-03T10:13:14.295 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2022-07-03T10:13:14.295 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:14.296 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2022-07-03T10:13:14.296 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2022-07-03T10:13:14.296 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:14.372 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-07-03T10:13:14.373 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-07-03T10:13:14.373 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2022-07-03T10:13:14.373 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-07-03T10:13:14.373 INFO:teuthology.orchestra.run.smithi161.stdout:Resetting modules: 2022-07-03T10:13:14.374 INFO:teuthology.orchestra.run.smithi161.stdout: container-tools 2022-07-03T10:13:14.374 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:14.374 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-07-03T10:13:14.374 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2022-07-03T10:13:14.375 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:14.701 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2022-07-03T10:13:14.702 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2022-07-03T10:13:14.702 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2022-07-03T10:13:14.702 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2022-07-03T10:13:14.703 INFO:teuthology.orchestra.run.smithi101.stdout:Resetting modules: 2022-07-03T10:13:14.703 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools 2022-07-03T10:13:14.703 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:14.703 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2022-07-03T10:13:14.704 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2022-07-03T10:13:14.704 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:17.079 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2022-07-03T10:13:17.113 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-07-03T10:13:17.544 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:17.843 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:17.880 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2022-07-03T10:13:18.321 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2022-07-03T10:13:18.602 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:18.604 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2022-07-03T10:13:18.664 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 73 kB/s | 2.4 kB 00:00 2022-07-03T10:13:18.852 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2022-07-03T10:13:19.196 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2022-07-03T10:13:19.531 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2022-07-03T10:13:20.322 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2022-07-03T10:13:20.323 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2022-07-03T10:13:20.324 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2022-07-03T10:13:20.324 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2022-07-03T10:13:20.324 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2022-07-03T10:13:20.325 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 1:1.24.2-4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 8.1 M 2022-07-03T10:13:20.325 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 43-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 494 k 2022-07-03T10:13:20.326 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.4.4-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 209 k 2022-07-03T10:13:20.326 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 149 k 2022-07-03T10:13:20.326 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.6.1-2.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 6.7 M 2022-07-03T10:13:20.326 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-07-03T10:13:20.327 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-3.module+el8.6.0+15011+418a9f05 rhel-8-for-x86_64-appstream-rpms 49 k 2022-07-03T10:13:20.327 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2022-07-03T10:13:20.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-07-03T10:13:20.327 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-07-03T10:13:20.327 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-07-03T10:13:20.328 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2022-07-03T10:13:20.328 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2022-07-03T10:13:20.328 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:20.328 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2022-07-03T10:13:20.329 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2022-07-03T10:13:20.329 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:20.329 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2022-07-03T10:13:20.329 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================================ 2022-07-03T10:13:20.329 INFO:teuthology.orchestra.run.smithi188.stdout:Install 10 Packages 2022-07-03T10:13:20.330 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:20.331 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 18 M 2022-07-03T10:13:20.331 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 64 M 2022-07-03T10:13:20.331 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2022-07-03T10:13:20.415 INFO:teuthology.orchestra.run.smithi188.stdout:(1/10): python3-podman-4.0.0-1.module+el8.6.0+1 1.6 MB/s | 149 kB 00:00 2022-07-03T10:13:20.440 INFO:teuthology.orchestra.run.smithi188.stdout:(2/10): udica-0.2.6-3.module+el8.6.0+15011+418a 432 kB/s | 49 kB 00:00 2022-07-03T10:13:20.465 INFO:teuthology.orchestra.run.smithi188.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 675 kB/s | 94 kB 00:00 2022-07-03T10:13:20.566 INFO:teuthology.orchestra.run.smithi188.stdout:(4/10): toolbox-0.0.99.3-0.4.module+el8.6.0+148 23 MB/s | 2.2 MB 00:00 2022-07-03T10:13:20.608 INFO:teuthology.orchestra.run.smithi188.stdout:(5/10): crun-1.4.4-1.module+el8.6.0+14877+f643d 4.9 MB/s | 209 kB 00:00 2022-07-03T10:13:20.725 INFO:teuthology.orchestra.run.smithi188.stdout:(6/10): skopeo-1.6.1-2.module+el8.6.0+14877+f64 23 MB/s | 6.7 MB 00:00 2022-07-03T10:13:20.758 INFO:teuthology.orchestra.run.smithi188.stdout:(7/10): cockpit-podman-43-1.module+el8.6.0+1487 3.2 MB/s | 494 kB 00:00 2022-07-03T10:13:20.814 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-07-03T10:13:20.816 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================================ 2022-07-03T10:13:20.816 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-07-03T10:13:20.816 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================================ 2022-07-03T10:13:20.817 INFO:teuthology.orchestra.run.smithi161.stdout:Installing group/module packages: 2022-07-03T10:13:20.817 INFO:teuthology.orchestra.run.smithi161.stdout: buildah x86_64 1:1.24.2-4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 8.1 M 2022-07-03T10:13:20.817 INFO:teuthology.orchestra.run.smithi161.stdout: cockpit-podman noarch 43-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 494 k 2022-07-03T10:13:20.817 INFO:teuthology.orchestra.run.smithi161.stdout: crun x86_64 1.4.4-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 209 k 2022-07-03T10:13:20.818 INFO:teuthology.orchestra.run.smithi161.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 149 k 2022-07-03T10:13:20.818 INFO:teuthology.orchestra.run.smithi161.stdout: skopeo x86_64 2:1.6.1-2.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 6.7 M 2022-07-03T10:13:20.818 INFO:teuthology.orchestra.run.smithi161.stdout: toolbox x86_64 0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-07-03T10:13:20.818 INFO:teuthology.orchestra.run.smithi161.stdout: udica noarch 0.2.6-3.module+el8.6.0+15011+418a9f05 rhel-8-for-x86_64-appstream-rpms 49 k 2022-07-03T10:13:20.819 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-07-03T10:13:20.819 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-07-03T10:13:20.819 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-07-03T10:13:20.819 INFO:teuthology.orchestra.run.smithi161.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-07-03T10:13:20.819 INFO:teuthology.orchestra.run.smithi161.stdout:Installing module profiles: 2022-07-03T10:13:20.820 INFO:teuthology.orchestra.run.smithi161.stdout: container-tools/common 2022-07-03T10:13:20.820 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:20.820 INFO:teuthology.orchestra.run.smithi161.stdout:Enabling module streams: 2022-07-03T10:13:20.821 INFO:teuthology.orchestra.run.smithi161.stdout: container-tools rhel8 2022-07-03T10:13:20.821 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:20.821 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-07-03T10:13:20.821 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================================ 2022-07-03T10:13:20.822 INFO:teuthology.orchestra.run.smithi161.stdout:Install 10 Packages 2022-07-03T10:13:20.822 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:20.823 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 18 M 2022-07-03T10:13:20.823 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 64 M 2022-07-03T10:13:20.823 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-07-03T10:13:20.850 INFO:teuthology.orchestra.run.smithi188.stdout:(8/10): buildah-1.24.2-4.module+el8.6.0+14877+f 19 MB/s | 8.1 MB 00:00 2022-07-03T10:13:20.884 INFO:teuthology.orchestra.run.smithi188.stdout:(9/10): python3-pytoml-0.1.14-5.git7dea353.el8. 156 kB/s | 25 kB 00:00 2022-07-03T10:13:20.898 INFO:teuthology.orchestra.run.smithi161.stdout:(1/10): python3-podman-4.0.0-1.module+el8.6.0+1 1.8 MB/s | 149 kB 00:00 2022-07-03T10:13:20.917 INFO:teuthology.orchestra.run.smithi188.stdout:(10/10): yajl-2.1.0-10.el8.x86_64.rpm 257 kB/s | 41 kB 00:00 2022-07-03T10:13:20.918 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:13:20.918 INFO:teuthology.orchestra.run.smithi188.stdout:Total 30 MB/s | 18 MB 00:00 2022-07-03T10:13:20.923 INFO:teuthology.orchestra.run.smithi161.stdout:(2/10): udica-0.2.6-3.module+el8.6.0+15011+418a 471 kB/s | 49 kB 00:00 2022-07-03T10:13:20.948 INFO:teuthology.orchestra.run.smithi161.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 725 kB/s | 94 kB 00:00 2022-07-03T10:13:21.041 INFO:teuthology.orchestra.run.smithi161.stdout:(4/10): toolbox-0.0.99.3-0.4.module+el8.6.0+148 25 MB/s | 2.2 MB 00:00 2022-07-03T10:13:21.083 INFO:teuthology.orchestra.run.smithi161.stdout:(5/10): crun-1.4.4-1.module+el8.6.0+14877+f643d 4.9 MB/s | 209 kB 00:00 2022-07-03T10:13:21.118 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2022-07-03T10:13:21.135 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2022-07-03T10:13:21.136 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2022-07-03T10:13:21.183 INFO:teuthology.orchestra.run.smithi161.stdout:(6/10): skopeo-1.6.1-2.module+el8.6.0+14877+f64 26 MB/s | 6.7 MB 00:00 2022-07-03T10:13:21.249 INFO:teuthology.orchestra.run.smithi161.stdout:(7/10): cockpit-podman-43-1.module+el8.6.0+1487 2.9 MB/s | 494 kB 00:00 2022-07-03T10:13:21.314 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2022-07-03T10:13:21.318 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2022-07-03T10:13:21.350 INFO:teuthology.orchestra.run.smithi161.stdout:(8/10): buildah-1.24.2-4.module+el8.6.0+14877+f 18 MB/s | 8.1 MB 00:00 2022-07-03T10:13:21.375 INFO:teuthology.orchestra.run.smithi161.stdout:(9/10): python3-pytoml-0.1.14-5.git7dea353.el8. 129 kB/s | 25 kB 00:00 2022-07-03T10:13:21.394 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2022-07-03T10:13:21.395 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2022-07-03T10:13:21.396 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2022-07-03T10:13:21.396 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2022-07-03T10:13:21.397 INFO:teuthology.orchestra.run.smithi101.stdout:Installing group/module packages: 2022-07-03T10:13:21.397 INFO:teuthology.orchestra.run.smithi101.stdout: buildah x86_64 1:1.24.2-4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 8.1 M 2022-07-03T10:13:21.397 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman noarch 43-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 494 k 2022-07-03T10:13:21.398 INFO:teuthology.orchestra.run.smithi101.stdout: crun x86_64 1.4.4-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 209 k 2022-07-03T10:13:21.398 INFO:teuthology.orchestra.run.smithi101.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 149 k 2022-07-03T10:13:21.398 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo x86_64 2:1.6.1-2.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 6.7 M 2022-07-03T10:13:21.398 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox x86_64 0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-07-03T10:13:21.399 INFO:teuthology.orchestra.run.smithi101.stdout: udica noarch 0.2.6-3.module+el8.6.0+15011+418a9f05 rhel-8-for-x86_64-appstream-rpms 49 k 2022-07-03T10:13:21.399 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2022-07-03T10:13:21.399 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-07-03T10:13:21.399 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-07-03T10:13:21.399 INFO:teuthology.orchestra.run.smithi101.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-07-03T10:13:21.400 INFO:teuthology.orchestra.run.smithi101.stdout:Installing module profiles: 2022-07-03T10:13:21.400 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools/common 2022-07-03T10:13:21.400 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:21.401 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling module streams: 2022-07-03T10:13:21.401 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools rhel8 2022-07-03T10:13:21.401 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:21.402 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2022-07-03T10:13:21.402 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2022-07-03T10:13:21.402 INFO:teuthology.orchestra.run.smithi101.stdout:Install 10 Packages 2022-07-03T10:13:21.402 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:21.403 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 18 M 2022-07-03T10:13:21.403 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 64 M 2022-07-03T10:13:21.404 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2022-07-03T10:13:21.408 INFO:teuthology.orchestra.run.smithi161.stdout:(10/10): yajl-2.1.0-10.el8.x86_64.rpm 257 kB/s | 41 kB 00:00 2022-07-03T10:13:21.409 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:13:21.410 INFO:teuthology.orchestra.run.smithi161.stdout:Total 30 MB/s | 18 MB 00:00 2022-07-03T10:13:21.475 INFO:teuthology.orchestra.run.smithi101.stdout:(1/10): python3-podman-4.0.0-1.module+el8.6.0+1 1.9 MB/s | 149 kB 00:00 2022-07-03T10:13:21.500 INFO:teuthology.orchestra.run.smithi101.stdout:(2/10): udica-0.2.6-3.module+el8.6.0+15011+418a 487 kB/s | 49 kB 00:00 2022-07-03T10:13:21.526 INFO:teuthology.orchestra.run.smithi101.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 745 kB/s | 94 kB 00:00 2022-07-03T10:13:21.611 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-07-03T10:13:21.629 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-07-03T10:13:21.630 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-07-03T10:13:21.635 INFO:teuthology.orchestra.run.smithi101.stdout:(4/10): toolbox-0.0.99.3-0.4.module+el8.6.0+148 20 MB/s | 2.2 MB 00:00 2022-07-03T10:13:21.685 INFO:teuthology.orchestra.run.smithi101.stdout:(5/10): crun-1.4.4-1.module+el8.6.0+14877+f643d 4.1 MB/s | 209 kB 00:00 2022-07-03T10:13:21.735 INFO:teuthology.orchestra.run.smithi101.stdout:(6/10): cockpit-podman-43-1.module+el8.6.0+1487 9.7 MB/s | 494 kB 00:00 2022-07-03T10:13:21.777 INFO:teuthology.orchestra.run.smithi101.stdout:(7/10): python3-pytoml-0.1.14-5.git7dea353.el8. 598 kB/s | 25 kB 00:00 2022-07-03T10:13:21.797 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2022-07-03T10:13:21.808 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-07-03T10:13:21.812 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-07-03T10:13:21.819 INFO:teuthology.orchestra.run.smithi101.stdout:(8/10): yajl-2.1.0-10.el8.x86_64.rpm 988 kB/s | 41 kB 00:00 2022-07-03T10:13:21.919 INFO:teuthology.orchestra.run.smithi101.stdout:(9/10): skopeo-1.6.1-2.module+el8.6.0+14877+f64 16 MB/s | 6.7 MB 00:00 2022-07-03T10:13:21.947 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2022-07-03T10:13:22.028 INFO:teuthology.orchestra.run.smithi101.stdout:(10/10): buildah-1.24.2-4.module+el8.6.0+14877+ 15 MB/s | 8.1 MB 00:00 2022-07-03T10:13:22.029 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:13:22.029 INFO:teuthology.orchestra.run.smithi101.stdout:Total 28 MB/s | 18 MB 00:00 2022-07-03T10:13:22.127 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/10 2022-07-03T10:13:22.316 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-07-03T10:13:22.338 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:22.495 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2022-07-03T10:13:22.540 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 4/10 2022-07-03T10:13:22.552 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2022-07-03T10:13:22.576 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2022-07-03T10:13:22.576 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2022-07-03T10:13:22.720 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 5/10 2022-07-03T10:13:22.725 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/10 2022-07-03T10:13:22.979 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:23.073 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2022-07-03T10:13:23.078 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2022-07-03T10:13:23.116 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 6/10 2022-07-03T10:13:23.180 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 4/10 2022-07-03T10:13:23.409 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 5/10 2022-07-03T10:13:23.595 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2022-07-03T10:13:23.746 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/10 2022-07-03T10:13:23.779 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 6/10 2022-07-03T10:13:23.883 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 7/10 2022-07-03T10:13:23.947 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/10 2022-07-03T10:13:24.186 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:24.390 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 4/10 2022-07-03T10:13:24.578 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 5/10 2022-07-03T10:13:24.583 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 7/10 2022-07-03T10:13:24.749 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 8/10 2022-07-03T10:13:24.935 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 9/10 2022-07-03T10:13:24.987 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 6/10 2022-07-03T10:13:25.120 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:25.386 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:25.387 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 1/10 2022-07-03T10:13:25.387 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2/10 2022-07-03T10:13:25.388 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:25.388 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 4/10 2022-07-03T10:13:25.388 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 5/10 2022-07-03T10:13:25.388 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 6/10 2022-07-03T10:13:25.389 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 7/10 2022-07-03T10:13:25.389 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 8/10 2022-07-03T10:13:25.389 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 9/10 2022-07-03T10:13:25.465 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 8/10 2022-07-03T10:13:25.703 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 9/10 2022-07-03T10:13:25.816 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 7/10 2022-07-03T10:13:25.900 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:26.168 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:26.169 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 1/10 2022-07-03T10:13:26.169 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2/10 2022-07-03T10:13:26.169 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:26.169 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 4/10 2022-07-03T10:13:26.170 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 5/10 2022-07-03T10:13:26.170 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 6/10 2022-07-03T10:13:26.170 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 7/10 2022-07-03T10:13:26.170 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 8/10 2022-07-03T10:13:26.170 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 9/10 2022-07-03T10:13:26.781 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 8/10 2022-07-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 9/10 2022-07-03T10:13:27.166 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:27.652 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 10/10 2022-07-03T10:13:27.652 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2 1/10 2022-07-03T10:13:27.652 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2/10 2022-07-03T10:13:27.652 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 3/10 2022-07-03T10:13:27.653 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x 4/10 2022-07-03T10:13:27.653 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86 5/10 2022-07-03T10:13:27.653 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6 6/10 2022-07-03T10:13:27.653 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 7/10 2022-07-03T10:13:27.654 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6. 8/10 2022-07-03T10:13:27.654 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 9/10 2022-07-03T10:13:28.456 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2022-07-03T10:13:28.456 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2022-07-03T10:13:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:28.632 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2022-07-03T10:13:28.632 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:28.633 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:28.633 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:28.633 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-07-03T10:13:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-07-03T10:13:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:28.634 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2022-07-03T10:13:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-10.el8.x86_64 2022-07-03T10:13:28.635 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:28.635 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2022-07-03T10:13:28.796 DEBUG:teuthology.parallel:result is None 2022-07-03T10:13:30.466 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2022-07-03T10:13:30.467 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2022-07-03T10:13:30.639 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:30.639 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-07-03T10:13:30.639 INFO:teuthology.orchestra.run.smithi161.stdout: buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:30.640 INFO:teuthology.orchestra.run.smithi161.stdout: cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:30.640 INFO:teuthology.orchestra.run.smithi161.stdout: crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:30.640 INFO:teuthology.orchestra.run.smithi161.stdout: python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:30.640 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-07-03T10:13:30.641 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-07-03T10:13:30.641 INFO:teuthology.orchestra.run.smithi161.stdout: skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:30.641 INFO:teuthology.orchestra.run.smithi161.stdout: toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:30.641 INFO:teuthology.orchestra.run.smithi161.stdout: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2022-07-03T10:13:30.641 INFO:teuthology.orchestra.run.smithi161.stdout: yajl-2.1.0-10.el8.x86_64 2022-07-03T10:13:30.642 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:13:30.642 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-07-03T10:13:30.787 DEBUG:teuthology.parallel:result is None 2022-07-03T10:13:31.491 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 10/10 2022-07-03T10:13:31.492 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2022-07-03T10:13:31.683 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:31.683 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2022-07-03T10:13:31.684 INFO:teuthology.orchestra.run.smithi101.stdout: buildah-1:1.24.2-4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:31.684 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman-43-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:31.684 INFO:teuthology.orchestra.run.smithi101.stdout: crun-1.4.4-1.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:31.684 INFO:teuthology.orchestra.run.smithi101.stdout: python3-podman-4.0.0-1.module+el8.6.0+14877+f643d2d6.noarch 2022-07-03T10:13:31.684 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-07-03T10:13:31.685 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-07-03T10:13:31.685 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo-2:1.6.1-2.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:31.685 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox-0.0.99.3-0.4.module+el8.6.0+14877+f643d2d6.x86_64 2022-07-03T10:13:31.685 INFO:teuthology.orchestra.run.smithi101.stdout: udica-0.2.6-3.module+el8.6.0+15011+418a9f05.noarch 2022-07-03T10:13:31.686 INFO:teuthology.orchestra.run.smithi101.stdout: yajl-2.1.0-10.el8.x86_64 2022-07-03T10:13:31.686 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:13:31.686 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2022-07-03T10:13:32.022 DEBUG:teuthology.parallel:result is None 2022-07-03T10:13:32.022 INFO:teuthology.run_tasks:Running task install... 2022-07-03T10:13:32.031 DEBUG:teuthology.task.install:project ceph 2022-07-03T10:13:32.032 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d'}} 2022-07-03T10:13:32.032 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d'} 2022-07-03T10:13:32.032 INFO:teuthology.task.install:Using flavor: default 2022-07-03T10:13:32.040 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-03T10:13:32.040 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-07-03T10:13:32.041 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2022-07-03T10:13:32.188 INFO:teuthology.orchestra.run.smithi101.stdout:RedHatEnterprise 2022-07-03T10:13:32.193 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-07-03T10:13:32.194 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d', 'tag': None, 'wait_for_package': False} 2022-07-03T10:13:32.195 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2022-07-03T10:13:32.249 INFO:teuthology.orchestra.run.smithi161.stdout:RedHatEnterprise 2022-07-03T10:13:32.250 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.251 DEBUG:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2022-07-03T10:13:32.251 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d', 'tag': None, 'wait_for_package': False} 2022-07-03T10:13:32.252 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-07-03T10:13:32.296 INFO:teuthology.orchestra.run.smithi161.stdout:RedHatEnterprise 2022-07-03T10:13:32.297 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:13:32.307 INFO:teuthology.orchestra.run.smithi188.stdout:RedHatEnterprise 2022-07-03T10:13:32.307 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.308 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d', 'tag': None, 'wait_for_package': False} 2022-07-03T10:13:32.308 DEBUG:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2022-07-03T10:13:32.347 INFO:teuthology.orchestra.run.smithi101.stdout:RedHatEnterprise 2022-07-03T10:13:32.353 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:13:32.359 INFO:teuthology.orchestra.run.smithi188.stdout:RedHatEnterprise 2022-07-03T10:13:32.361 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:13:32.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:13:32.458 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/ 2022-07-03T10:13:32.458 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13322.g61829bae 2022-07-03T10:13:32.515 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/ 2022-07-03T10:13:32.516 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13322.g61829bae 2022-07-03T10:13:32.526 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/ 2022-07-03T10:13:32.527 INFO:teuthology.task.install.rpm:Package version is 17.0.0-13322.g61829bae 2022-07-03T10:13:32.568 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-07-03T10:13:32.609 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:13:32.610 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-07-03T10:13:32.645 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-07-03T10:13:32.646 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/61829bae536fb4126cfaa87ca353b9576e018e7d/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-07-03T10:13:32.670 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-07-03T10:13:32.671 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:13:32.672 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-07-03T10:13:32.676 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20220702-1631/61829bae536fb4126cfaa87ca353b9576e018e7d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-07-03T10:13:32.676 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:13:32.677 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-07-03T10:13:32.708 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-07-03T10:13:32.709 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/61829bae536fb4126cfaa87ca353b9576e018e7d/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-07-03T10:13:32.718 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-07-03T10:13:32.764 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-07-03T10:13:32.764 DEBUG:teuthology.orchestra.run.smithi101:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/61829bae536fb4126cfaa87ca353b9576e018e7d/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-07-03T10:13:32.778 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-07-03T10:13:32.800 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-07-03T10:13:32.853 DEBUG:teuthology.orchestra.run.smithi101:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-07-03T10:13:32.857 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-07-03T10:13:32.867 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2022-07-03T10:13:32.868 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2022-07-03T10:13:32.924 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2022-07-03T10:13:32.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2022-07-03T10:13:32.972 DEBUG:teuthology.orchestra.run.smithi101:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-07-03T10:13:33.059 INFO:teuthology.orchestra.run.smithi101.stdout:check_obsoletes = 1 2022-07-03T10:13:33.064 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum clean all 2022-07-03T10:13:33.244 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:33.297 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:33.509 INFO:teuthology.orchestra.run.smithi188.stdout:43 files removed 2022-07-03T10:13:33.549 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:33.557 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-07-03T10:13:33.827 INFO:teuthology.orchestra.run.smithi101.stdout:43 files removed 2022-07-03T10:13:33.887 INFO:teuthology.orchestra.run.smithi188.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:33.897 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-07-03T10:13:34.333 INFO:teuthology.orchestra.run.smithi161.stdout:43 files removed 2022-07-03T10:13:34.381 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:34.384 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-07-03T10:13:34.436 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 326 kB/s | 80 kB 00:00 2022-07-03T10:13:34.695 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 40 kB/s | 9.1 kB 00:00 2022-07-03T10:13:34.717 INFO:teuthology.orchestra.run.smithi161.stdout:Updating Subscription Management repositories. 2022-07-03T10:13:34.946 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 6.5 kB/s | 1.5 kB 00:00 2022-07-03T10:13:35.028 INFO:teuthology.orchestra.run.smithi101.stdout:ceph packages for x86_64 307 kB/s | 80 kB 00:00 2022-07-03T10:13:35.294 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 303 kB/s | 80 kB 00:00 2022-07-03T10:13:35.295 INFO:teuthology.orchestra.run.smithi101.stdout:ceph noarch packages 39 kB/s | 9.1 kB 00:00 2022-07-03T10:13:35.538 INFO:teuthology.orchestra.run.smithi101.stdout:ceph source packages 6.8 kB/s | 1.5 kB 00:00 2022-07-03T10:13:35.580 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 36 kB/s | 9.1 kB 00:00 2022-07-03T10:13:35.822 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 6.7 kB/s | 1.5 kB 00:00 2022-07-03T10:13:37.379 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-07-03T10:13:37.455 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2022-07-03T10:13:37.865 INFO:teuthology.orchestra.run.smithi161.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:02 2022-07-03T10:13:38.455 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 11 MB/s | 12 MB 00:01 2022-07-03T10:13:38.664 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 10 MB/s | 12 MB 00:01 2022-07-03T10:13:39.181 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 12 MB 00:01 2022-07-03T10:13:41.080 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 420 kB/s | 24 kB 00:00 2022-07-03T10:13:41.823 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 489 kB/s | 24 kB 00:00 2022-07-03T10:13:41.831 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 541 kB/s | 24 kB 00:00 2022-07-03T10:13:41.882 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 45 MB 00:00 2022-07-03T10:13:42.625 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 48 MB 00:00 2022-07-03T10:13:42.808 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 MB/s | 45 MB 00:00 2022-07-03T10:13:49.315 INFO:teuthology.orchestra.run.smithi188.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 MB/s | 48 MB 00:00 2022-07-03T10:13:49.917 INFO:teuthology.orchestra.run.smithi161.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 45 MB 00:00 2022-07-03T10:13:51.092 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 48 MB 00:00 2022-07-03T10:13:59.704 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:13:59.705 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:13:59.718 INFO:teuthology.orchestra.run.smithi188.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-07-03T10:13:59.719 INFO:teuthology.orchestra.run.smithi188.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-07-03T10:13:59.720 INFO:teuthology.orchestra.run.smithi188.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-07-03T10:13:59.722 INFO:teuthology.orchestra.run.smithi188.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-07-03T10:13:59.723 INFO:teuthology.orchestra.run.smithi188.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-07-03T10:13:59.726 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-07-03T10:13:59.727 INFO:teuthology.orchestra.run.smithi188.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-07-03T10:13:59.846 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2022-07-03T10:13:59.856 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================================================= 2022-07-03T10:13:59.857 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2022-07-03T10:13:59.858 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================================================= 2022-07-03T10:13:59.858 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2022-07-03T10:13:59.858 INFO:teuthology.orchestra.run.smithi188.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-07-03T10:13:59.859 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:17.0.0-13322.g61829bae.el8 ceph 6.5 k 2022-07-03T10:13:59.859 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.3 M 2022-07-03T10:13:59.860 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 883 k 2022-07-03T10:13:59.861 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13322.g61829bae.el8 ceph 169 k 2022-07-03T10:13:59.861 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:17.0.0-13322.g61829bae.el8 ceph 1.5 M 2022-07-03T10:13:59.862 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 111 k 2022-07-03T10:13:59.863 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 1.6 M 2022-07-03T10:13:59.863 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 7.4 M 2022-07-03T10:13:59.863 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 47 k 2022-07-03T10:13:59.864 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:13:59.864 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:17.0.0-13322.g61829bae.el8 ceph 53 M 2022-07-03T10:13:59.866 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 259 k 2022-07-03T10:13:59.866 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 79 k 2022-07-03T10:13:59.866 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 272 k 2022-07-03T10:13:59.867 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 14 k 2022-07-03T10:13:59.867 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-07-03T10:13:59.868 INFO:teuthology.orchestra.run.smithi188.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-07-03T10:13:59.868 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-07-03T10:13:59.869 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 30 k 2022-07-03T10:13:59.869 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 796 k 2022-07-03T10:13:59.870 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 125 k 2022-07-03T10:13:59.870 INFO:teuthology.orchestra.run.smithi188.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-07-03T10:13:59.871 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-07-03T10:13:59.871 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel x86_64 1:1.1.1k-6.el8_5 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-07-03T10:13:59.871 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:17.0.0-13322.g61829bae.el8 ceph 178 k 2022-07-03T10:13:59.872 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:17.0.0-13322.g61829bae.el8 ceph 367 k 2022-07-03T10:13:59.872 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 346 k 2022-07-03T10:13:59.873 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 109 k 2022-07-03T10:13:59.873 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 93 k 2022-07-03T10:13:59.874 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.8 M 2022-07-03T10:13:59.874 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 175 k 2022-07-03T10:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-07-03T10:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2022-07-03T10:13:59.876 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.6 M 2022-07-03T10:13:59.876 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:13:59.876 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2022-07-03T10:13:59.877 INFO:teuthology.orchestra.run.smithi188.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-07-03T10:13:59.877 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 M 2022-07-03T10:13:59.878 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 23 k 2022-07-03T10:13:59.878 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:17.0.0-13322.g61829bae.el8 ceph 2.3 M 2022-07-03T10:13:59.879 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 229 k 2022-07-03T10:13:59.879 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:13:59.880 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:13:59.880 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 15 k 2022-07-03T10:13:59.881 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 k 2022-07-03T10:13:59.881 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-07-03T10:13:59.881 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-07-03T10:13:59.882 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-07-03T10:13:59.882 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-07-03T10:13:59.883 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-07-03T10:13:59.883 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:17.0.0-13322.g61829bae.el8 ceph 170 k 2022-07-03T10:13:59.884 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-07-03T10:13:59.884 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 643 k 2022-07-03T10:13:59.885 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-07-03T10:13:59.885 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 171 k 2022-07-03T10:13:59.885 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-07-03T10:13:59.885 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 485 k 2022-07-03T10:13:59.886 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-07-03T10:13:59.886 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.5 M 2022-07-03T10:13:59.886 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-07-03T10:13:59.886 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-07-03T10:13:59.887 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-07-03T10:13:59.887 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-07-03T10:13:59.887 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-07-03T10:13:59.887 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-07-03T10:13:59.888 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-07-03T10:13:59.888 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-07-03T10:13:59.888 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-07-03T10:13:59.888 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-07-03T10:13:59.888 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-07-03T10:13:59.889 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-07-03T10:13:59.889 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-07-03T10:13:59.889 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-07-03T10:13:59.889 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-07-03T10:13:59.890 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-07-03T10:13:59.890 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 45 k 2022-07-03T10:13:59.890 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 106 k 2022-07-03T10:13:59.891 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-07-03T10:13:59.891 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-07-03T10:13:59.891 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-07-03T10:13:59.892 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-07-03T10:13:59.893 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-07-03T10:13:59.893 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-07-03T10:13:59.894 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-07-03T10:13:59.894 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-07-03T10:13:59.895 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-07-03T10:13:59.895 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-07-03T10:13:59.895 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-07-03T10:13:59.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:13:59.896 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-07-03T10:13:59.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-07-03T10:13:59.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-07-03T10:13:59.897 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-07-03T10:13:59.898 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-07-03T10:13:59.898 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-07-03T10:13:59.898 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-07-03T10:13:59.899 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-07-03T10:13:59.899 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-07-03T10:13:59.900 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-07-03T10:13:59.900 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-07-03T10:13:59.900 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-07-03T10:13:59.901 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-07-03T10:13:59.901 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-07-03T10:13:59.901 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-07-03T10:13:59.901 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-07-03T10:13:59.902 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-07-03T10:13:59.902 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-07-03T10:13:59.902 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:13:59.902 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-07-03T10:13:59.902 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-07-03T10:13:59.903 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-07-03T10:13:59.903 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-07-03T10:13:59.903 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-07-03T10:13:59.904 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-07-03T10:13:59.904 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-07-03T10:13:59.904 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-07-03T10:13:59.904 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-07-03T10:13:59.904 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-07-03T10:13:59.905 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-07-03T10:13:59.905 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-07-03T10:13:59.905 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-07-03T10:13:59.905 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-07-03T10:13:59.905 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-07-03T10:13:59.906 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-07-03T10:13:59.906 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-07-03T10:13:59.906 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:59.906 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2022-07-03T10:13:59.907 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================================================= 2022-07-03T10:13:59.907 INFO:teuthology.orchestra.run.smithi188.stdout:Install 122 Packages 2022-07-03T10:13:59.907 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2022-07-03T10:13:59.907 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:13:59.908 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 204 M 2022-07-03T10:13:59.908 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2022-07-03T10:13:59.998 INFO:teuthology.orchestra.run.smithi188.stdout:(1/124): ceph-17.0.0-13322.g61829bae.el8.x86_64 51 kB/s | 6.5 kB 00:00 2022-07-03T10:14:00.222 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:14:00.223 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:14:00.237 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-07-03T10:14:00.238 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-07-03T10:14:00.239 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-07-03T10:14:00.240 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-07-03T10:14:00.242 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-07-03T10:14:00.246 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-07-03T10:14:00.246 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-07-03T10:14:00.248 INFO:teuthology.orchestra.run.smithi188.stdout:(2/124): ceph-fuse-17.0.0-13322.g61829bae.el8.x 3.5 MB/s | 883 kB 00:00 2022-07-03T10:14:00.348 INFO:teuthology.orchestra.run.smithi188.stdout:(3/124): ceph-immutable-object-cache-17.0.0-133 1.7 MB/s | 169 kB 00:00 2022-07-03T10:14:00.367 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2022-07-03T10:14:00.378 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2022-07-03T10:14:00.379 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2022-07-03T10:14:00.379 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2022-07-03T10:14:00.380 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2022-07-03T10:14:00.380 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-07-03T10:14:00.380 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:17.0.0-13322.g61829bae.el8 ceph 6.5 k 2022-07-03T10:14:00.381 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.3 M 2022-07-03T10:14:00.381 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 883 k 2022-07-03T10:14:00.381 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13322.g61829bae.el8 ceph 169 k 2022-07-03T10:14:00.381 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:17.0.0-13322.g61829bae.el8 ceph 1.5 M 2022-07-03T10:14:00.382 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 111 k 2022-07-03T10:14:00.382 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 1.6 M 2022-07-03T10:14:00.382 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 7.4 M 2022-07-03T10:14:00.382 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 47 k 2022-07-03T10:14:00.382 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:14:00.383 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:17.0.0-13322.g61829bae.el8 ceph 53 M 2022-07-03T10:14:00.383 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 259 k 2022-07-03T10:14:00.383 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 79 k 2022-07-03T10:14:00.383 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 272 k 2022-07-03T10:14:00.384 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 14 k 2022-07-03T10:14:00.384 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-07-03T10:14:00.385 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-07-03T10:14:00.385 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-07-03T10:14:00.385 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 30 k 2022-07-03T10:14:00.386 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 796 k 2022-07-03T10:14:00.386 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 125 k 2022-07-03T10:14:00.386 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-07-03T10:14:00.387 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-07-03T10:14:00.387 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:1.1.1k-6.el8_5 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-07-03T10:14:00.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:17.0.0-13322.g61829bae.el8 ceph 178 k 2022-07-03T10:14:00.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:17.0.0-13322.g61829bae.el8 ceph 367 k 2022-07-03T10:14:00.387 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 346 k 2022-07-03T10:14:00.388 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 109 k 2022-07-03T10:14:00.388 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 93 k 2022-07-03T10:14:00.388 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.8 M 2022-07-03T10:14:00.388 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 175 k 2022-07-03T10:14:00.389 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-07-03T10:14:00.389 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2022-07-03T10:14:00.389 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.6 M 2022-07-03T10:14:00.389 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:14:00.390 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2022-07-03T10:14:00.390 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-07-03T10:14:00.390 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 M 2022-07-03T10:14:00.390 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 23 k 2022-07-03T10:14:00.391 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:17.0.0-13322.g61829bae.el8 ceph 2.3 M 2022-07-03T10:14:00.391 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 229 k 2022-07-03T10:14:00.391 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:14:00.391 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:14:00.391 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 15 k 2022-07-03T10:14:00.392 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 k 2022-07-03T10:14:00.392 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-07-03T10:14:00.392 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-07-03T10:14:00.392 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-07-03T10:14:00.393 INFO:teuthology.orchestra.run.smithi161.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-07-03T10:14:00.393 INFO:teuthology.orchestra.run.smithi161.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-07-03T10:14:00.393 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:17.0.0-13322.g61829bae.el8 ceph 170 k 2022-07-03T10:14:00.393 INFO:teuthology.orchestra.run.smithi161.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-07-03T10:14:00.393 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 643 k 2022-07-03T10:14:00.394 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-07-03T10:14:00.394 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 171 k 2022-07-03T10:14:00.394 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-07-03T10:14:00.394 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 485 k 2022-07-03T10:14:00.395 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-07-03T10:14:00.395 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.5 M 2022-07-03T10:14:00.395 INFO:teuthology.orchestra.run.smithi161.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-07-03T10:14:00.396 INFO:teuthology.orchestra.run.smithi161.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-07-03T10:14:00.396 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-07-03T10:14:00.397 INFO:teuthology.orchestra.run.smithi161.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-07-03T10:14:00.397 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-07-03T10:14:00.397 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-07-03T10:14:00.398 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-07-03T10:14:00.398 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-07-03T10:14:00.399 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-07-03T10:14:00.399 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-07-03T10:14:00.400 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-07-03T10:14:00.400 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-07-03T10:14:00.401 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-07-03T10:14:00.401 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-07-03T10:14:00.402 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-07-03T10:14:00.402 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-07-03T10:14:00.402 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 45 k 2022-07-03T10:14:00.402 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 106 k 2022-07-03T10:14:00.403 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-07-03T10:14:00.403 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-07-03T10:14:00.403 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-07-03T10:14:00.403 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-07-03T10:14:00.404 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-07-03T10:14:00.404 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-07-03T10:14:00.404 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-07-03T10:14:00.404 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-07-03T10:14:00.404 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-07-03T10:14:00.405 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-07-03T10:14:00.405 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-07-03T10:14:00.405 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:14:00.405 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-07-03T10:14:00.406 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-07-03T10:14:00.406 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-07-03T10:14:00.406 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-07-03T10:14:00.406 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-07-03T10:14:00.406 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-07-03T10:14:00.407 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-07-03T10:14:00.407 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-07-03T10:14:00.407 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-07-03T10:14:00.407 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-07-03T10:14:00.408 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-07-03T10:14:00.408 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-07-03T10:14:00.408 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-07-03T10:14:00.408 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-07-03T10:14:00.409 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-07-03T10:14:00.409 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-07-03T10:14:00.409 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-07-03T10:14:00.409 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-07-03T10:14:00.410 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:14:00.410 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-07-03T10:14:00.410 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-07-03T10:14:00.410 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-07-03T10:14:00.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-07-03T10:14:00.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-07-03T10:14:00.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-07-03T10:14:00.411 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-07-03T10:14:00.412 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-07-03T10:14:00.412 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-07-03T10:14:00.412 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-07-03T10:14:00.412 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-07-03T10:14:00.413 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-07-03T10:14:00.413 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-07-03T10:14:00.413 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-07-03T10:14:00.413 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-07-03T10:14:00.414 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-07-03T10:14:00.414 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-07-03T10:14:00.414 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:14:00.414 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2022-07-03T10:14:00.414 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================================================= 2022-07-03T10:14:00.415 INFO:teuthology.orchestra.run.smithi161.stdout:Install 122 Packages 2022-07-03T10:14:00.415 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2022-07-03T10:14:00.415 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:14:00.416 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 204 M 2022-07-03T10:14:00.416 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2022-07-03T10:14:00.491 INFO:teuthology.orchestra.run.smithi188.stdout:(4/124): ceph-base-17.0.0-13322.g61829bae.el8.x 8.5 MB/s | 5.3 MB 00:00 2022-07-03T10:14:00.701 INFO:teuthology.orchestra.run.smithi161.stdout:(1/124): ceph-17.0.0-13322.g61829bae.el8.x86_64 21 kB/s | 6.5 kB 00:00 2022-07-03T10:14:00.749 INFO:teuthology.orchestra.run.smithi188.stdout:(5/124): ceph-mgr-17.0.0-13322.g61829bae.el8.x8 6.0 MB/s | 1.5 MB 00:00 2022-07-03T10:14:00.791 INFO:teuthology.orchestra.run.smithi188.stdout:(6/124): ceph-mds-17.0.0-13322.g61829bae.el8.x8 5.2 MB/s | 2.3 MB 00:00 2022-07-03T10:14:01.109 INFO:teuthology.orchestra.run.smithi161.stdout:(2/124): ceph-fuse-17.0.0-13322.g61829bae.el8.x 2.1 MB/s | 883 kB 00:00 2022-07-03T10:14:01.226 INFO:teuthology.orchestra.run.smithi161.stdout:(3/124): ceph-immutable-object-cache-17.0.0-133 1.4 MB/s | 169 kB 00:00 2022-07-03T10:14:01.332 INFO:teuthology.orchestra.run.smithi188.stdout:(7/124): ceph-mon-17.0.0-13322.g61829bae.el8.x8 6.7 MB/s | 3.9 MB 00:00 2022-07-03T10:14:01.927 INFO:teuthology.orchestra.run.smithi161.stdout:(4/124): ceph-mds-17.0.0-13322.g61829bae.el8.x8 3.3 MB/s | 2.3 MB 00:00 2022-07-03T10:14:02.021 INFO:teuthology.orchestra.run.smithi101.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:14:02.022 INFO:teuthology.orchestra.run.smithi101.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-07-03T10:14:02.037 INFO:teuthology.orchestra.run.smithi101.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-07-03T10:14:02.038 INFO:teuthology.orchestra.run.smithi101.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-07-03T10:14:02.039 INFO:teuthology.orchestra.run.smithi101.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-07-03T10:14:02.040 INFO:teuthology.orchestra.run.smithi101.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-07-03T10:14:02.043 INFO:teuthology.orchestra.run.smithi101.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-07-03T10:14:02.046 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-07-03T10:14:02.047 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-07-03T10:14:02.110 INFO:teuthology.orchestra.run.smithi188.stdout:(8/124): ceph-common-17.0.0-13322.g61829bae.el8 11 MB/s | 24 MB 00:02 2022-07-03T10:14:02.172 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2022-07-03T10:14:02.177 INFO:teuthology.orchestra.run.smithi161.stdout:(5/124): ceph-mgr-17.0.0-13322.g61829bae.el8.x8 6.2 MB/s | 1.5 MB 00:00 2022-07-03T10:14:02.184 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================================================= 2022-07-03T10:14:02.184 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2022-07-03T10:14:02.184 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================================================= 2022-07-03T10:14:02.185 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2022-07-03T10:14:02.185 INFO:teuthology.orchestra.run.smithi101.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-07-03T10:14:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: ceph x86_64 2:17.0.0-13322.g61829bae.el8 ceph 6.5 k 2022-07-03T10:14:02.186 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.3 M 2022-07-03T10:14:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 883 k 2022-07-03T10:14:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-13322.g61829bae.el8 ceph 169 k 2022-07-03T10:14:02.187 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr x86_64 2:17.0.0-13322.g61829bae.el8 ceph 1.5 M 2022-07-03T10:14:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 111 k 2022-07-03T10:14:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 1.6 M 2022-07-03T10:14:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 7.4 M 2022-07-03T10:14:02.188 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 47 k 2022-07-03T10:14:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:14:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test x86_64 2:17.0.0-13322.g61829bae.el8 ceph 53 M 2022-07-03T10:14:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 259 k 2022-07-03T10:14:02.189 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 79 k 2022-07-03T10:14:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 272 k 2022-07-03T10:14:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 14 k 2022-07-03T10:14:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-07-03T10:14:02.190 INFO:teuthology.orchestra.run.smithi101.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-07-03T10:14:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-07-03T10:14:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 30 k 2022-07-03T10:14:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 796 k 2022-07-03T10:14:02.191 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel x86_64 2:17.0.0-13322.g61829bae.el8 ceph 125 k 2022-07-03T10:14:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-07-03T10:14:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-07-03T10:14:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel x86_64 1:1.1.1k-6.el8_5 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-07-03T10:14:02.192 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs x86_64 2:17.0.0-13322.g61829bae.el8 ceph 178 k 2022-07-03T10:14:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados x86_64 2:17.0.0-13322.g61829bae.el8 ceph 367 k 2022-07-03T10:14:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 346 k 2022-07-03T10:14:02.193 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw x86_64 2:17.0.0-13322.g61829bae.el8 ceph 109 k 2022-07-03T10:14:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 93 k 2022-07-03T10:14:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.8 M 2022-07-03T10:14:02.194 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 175 k 2022-07-03T10:14:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-07-03T10:14:02.195 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrading: 2022-07-03T10:14:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: librados2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.6 M 2022-07-03T10:14:02.195 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:14:02.196 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2022-07-03T10:14:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-07-03T10:14:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 M 2022-07-03T10:14:02.196 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 23 k 2022-07-03T10:14:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds x86_64 2:17.0.0-13322.g61829bae.el8 ceph 2.3 M 2022-07-03T10:14:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 229 k 2022-07-03T10:14:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon x86_64 2:17.0.0-13322.g61829bae.el8 ceph 3.9 M 2022-07-03T10:14:02.197 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd x86_64 2:17.0.0-13322.g61829bae.el8 ceph 17 M 2022-07-03T10:14:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts noarch 2:17.0.0-13322.g61829bae.el8 ceph-noarch 15 k 2022-07-03T10:14:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux x86_64 2:17.0.0-13322.g61829bae.el8 ceph 24 k 2022-07-03T10:14:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-07-03T10:14:02.198 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-07-03T10:14:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-07-03T10:14:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-07-03T10:14:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-07-03T10:14:02.199 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite x86_64 2:17.0.0-13322.g61829bae.el8 ceph 170 k 2022-07-03T10:14:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-07-03T10:14:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 643 k 2022-07-03T10:14:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-07-03T10:14:02.200 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath x86_64 8.5.0-10.el8 rhel-8-for-x86_64-baseos-rpms 171 k 2022-07-03T10:14:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-07-03T10:14:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 485 k 2022-07-03T10:14:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-07-03T10:14:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2 x86_64 2:17.0.0-13322.g61829bae.el8 ceph 5.5 M 2022-07-03T10:14:02.201 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-07-03T10:14:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-07-03T10:14:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-07-03T10:14:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-07-03T10:14:02.202 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-07-03T10:14:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-07-03T10:14:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-07-03T10:14:02.203 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-07-03T10:14:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-07-03T10:14:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-07-03T10:14:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-07-03T10:14:02.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-07-03T10:14:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-07-03T10:14:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-07-03T10:14:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-07-03T10:14:02.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-07-03T10:14:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse x86_64 2:17.0.0-13322.g61829bae.el8 ceph 45 k 2022-07-03T10:14:02.206 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common x86_64 2:17.0.0-13322.g61829bae.el8 ceph 106 k 2022-07-03T10:14:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-07-03T10:14:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-07-03T10:14:02.207 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-07-03T10:14:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-07-03T10:14:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-07-03T10:14:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-07-03T10:14:02.208 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-07-03T10:14:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-07-03T10:14:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-07-03T10:14:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-07-03T10:14:02.209 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-07-03T10:14:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:14:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-07-03T10:14:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-07-03T10:14:02.210 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-07-03T10:14:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-07-03T10:14:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-07-03T10:14:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-07-03T10:14:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-07-03T10:14:02.211 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-07-03T10:14:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-07-03T10:14:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-07-03T10:14:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-07-03T10:14:02.212 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-07-03T10:14:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-07-03T10:14:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-07-03T10:14:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-07-03T10:14:02.213 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-07-03T10:14:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-07-03T10:14:02.214 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-07-03T10:14:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-07-03T10:14:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-07-03T10:14:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-07-03T10:14:02.215 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-07-03T10:14:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-07-03T10:14:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-07-03T10:14:02.216 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-07-03T10:14:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-07-03T10:14:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-07-03T10:14:02.217 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-07-03T10:14:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-07-03T10:14:02.218 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-07-03T10:14:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-07-03T10:14:02.219 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-07-03T10:14:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-07-03T10:14:02.220 INFO:teuthology.orchestra.run.smithi101.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-07-03T10:14:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-07-03T10:14:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-07-03T10:14:02.221 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:14:02.222 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2022-07-03T10:14:02.222 INFO:teuthology.orchestra.run.smithi101.stdout:======================================================================================================================= 2022-07-03T10:14:02.222 INFO:teuthology.orchestra.run.smithi101.stdout:Install 122 Packages 2022-07-03T10:14:02.223 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrade 2 Packages 2022-07-03T10:14:02.223 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:14:02.224 INFO:teuthology.orchestra.run.smithi188.stdout:(9/124): ceph-selinux-17.0.0-13322.g61829bae.el 326 kB/s | 24 kB 00:00 2022-07-03T10:14:02.225 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 204 M 2022-07-03T10:14:02.225 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2022-07-03T10:14:02.543 INFO:teuthology.orchestra.run.smithi101.stdout:(1/124): ceph-17.0.0-13322.g61829bae.el8.x86_64 19 kB/s | 6.5 kB 00:00 2022-07-03T10:14:02.652 INFO:teuthology.orchestra.run.smithi161.stdout:(6/124): ceph-base-17.0.0-13322.g61829bae.el8.x 2.3 MB/s | 5.3 MB 00:02 2022-07-03T10:14:03.394 INFO:teuthology.orchestra.run.smithi161.stdout:(7/124): ceph-mon-17.0.0-13322.g61829bae.el8.x8 3.2 MB/s | 3.9 MB 00:01 2022-07-03T10:14:03.543 INFO:teuthology.orchestra.run.smithi101.stdout:(2/124): ceph-fuse-17.0.0-13322.g61829bae.el8.x 883 kB/s | 883 kB 00:00 2022-07-03T10:14:03.794 INFO:teuthology.orchestra.run.smithi101.stdout:(3/124): ceph-immutable-object-cache-17.0.0-133 677 kB/s | 169 kB 00:00 2022-07-03T10:14:04.587 INFO:teuthology.orchestra.run.smithi188.stdout:(10/124): ceph-radosgw-17.0.0-13322.g61829bae.e 5.1 MB/s | 17 MB 00:03 2022-07-03T10:14:04.688 INFO:teuthology.orchestra.run.smithi188.stdout:(11/124): cephfs-mirror-17.0.0-13322.g61829bae. 2.7 MB/s | 272 kB 00:00 2022-07-03T10:14:04.788 INFO:teuthology.orchestra.run.smithi188.stdout:(12/124): libcephfs-devel-17.0.0-13322.g61829ba 304 kB/s | 30 kB 00:00 2022-07-03T10:14:05.047 INFO:teuthology.orchestra.run.smithi188.stdout:(13/124): libcephfs2-17.0.0-13322.g61829bae.el8 3.0 MB/s | 796 kB 00:00 2022-07-03T10:14:05.147 INFO:teuthology.orchestra.run.smithi188.stdout:(14/124): libcephsqlite-17.0.0-13322.g61829bae. 1.7 MB/s | 170 kB 00:00 2022-07-03T10:14:05.219 INFO:teuthology.orchestra.run.smithi101.stdout:(4/124): ceph-base-17.0.0-13322.g61829bae.el8.x 1.7 MB/s | 5.3 MB 00:03 2022-07-03T10:14:05.240 INFO:teuthology.orchestra.run.smithi188.stdout:(15/124): librados-devel-17.0.0-13322.g61829bae 1.3 MB/s | 125 kB 00:00 2022-07-03T10:14:05.489 INFO:teuthology.orchestra.run.smithi188.stdout:(16/124): ceph-osd-17.0.0-13322.g61829bae.el8.x 3.6 MB/s | 17 MB 00:04 2022-07-03T10:14:05.514 INFO:teuthology.orchestra.run.smithi188.stdout:(17/124): libradosstriper1-17.0.0-13322.g61829b 1.7 MB/s | 485 kB 00:00 2022-07-03T10:14:05.577 INFO:teuthology.orchestra.run.smithi101.stdout:(5/124): ceph-mds-17.0.0-13322.g61829bae.el8.x8 1.3 MB/s | 2.3 MB 00:01 2022-07-03T10:14:05.606 INFO:teuthology.orchestra.run.smithi188.stdout:(18/124): python3-ceph-argparse-17.0.0-13322.g6 487 kB/s | 45 kB 00:00 2022-07-03T10:14:05.706 INFO:teuthology.orchestra.run.smithi188.stdout:(19/124): python3-ceph-common-17.0.0-13322.g618 1.0 MB/s | 106 kB 00:00 2022-07-03T10:14:05.807 INFO:teuthology.orchestra.run.smithi188.stdout:(20/124): python3-cephfs-17.0.0-13322.g61829bae 1.7 MB/s | 178 kB 00:00 2022-07-03T10:14:06.102 INFO:teuthology.orchestra.run.smithi101.stdout:(6/124): ceph-mgr-17.0.0-13322.g61829bae.el8.x8 1.8 MB/s | 1.5 MB 00:00 2022-07-03T10:14:06.157 INFO:teuthology.orchestra.run.smithi188.stdout:(21/124): python3-rados-17.0.0-13322.g61829bae. 1.0 MB/s | 367 kB 00:00 2022-07-03T10:14:06.324 INFO:teuthology.orchestra.run.smithi188.stdout:(22/124): python3-rbd-17.0.0-13322.g61829bae.el 2.0 MB/s | 346 kB 00:00 2022-07-03T10:14:06.441 INFO:teuthology.orchestra.run.smithi188.stdout:(23/124): python3-rgw-17.0.0-13322.g61829bae.el 937 kB/s | 109 kB 00:00 2022-07-03T10:14:06.533 INFO:teuthology.orchestra.run.smithi188.stdout:(24/124): rbd-fuse-17.0.0-13322.g61829bae.el8.x 1.0 MB/s | 93 kB 00:00 2022-07-03T10:14:06.969 INFO:teuthology.orchestra.run.smithi161.stdout:(8/124): ceph-common-17.0.0-13322.g61829bae.el8 3.6 MB/s | 24 MB 00:06 2022-07-03T10:14:07.052 INFO:teuthology.orchestra.run.smithi161.stdout:(9/124): ceph-selinux-17.0.0-13322.g61829bae.el 292 kB/s | 24 kB 00:00 2022-07-03T10:14:07.058 INFO:teuthology.orchestra.run.smithi188.stdout:(25/124): librgw2-17.0.0-13322.g61829bae.el8.x8 3.5 MB/s | 5.5 MB 00:01 2022-07-03T10:14:07.158 INFO:teuthology.orchestra.run.smithi188.stdout:(26/124): rbd-nbd-17.0.0-13322.g61829bae.el8.x8 1.7 MB/s | 175 kB 00:00 2022-07-03T10:14:07.242 INFO:teuthology.orchestra.run.smithi188.stdout:(27/124): ceph-grafana-dashboards-17.0.0-13322. 275 kB/s | 23 kB 00:00 2022-07-03T10:14:07.317 INFO:teuthology.orchestra.run.smithi188.stdout:(28/124): ceph-mgr-cephadm-17.0.0-13322.g61829b 1.4 MB/s | 111 kB 00:00 2022-07-03T10:14:07.378 INFO:teuthology.orchestra.run.smithi161.stdout:(10/124): ceph-radosgw-17.0.0-13322.g61829bae.e 4.2 MB/s | 17 MB 00:03 2022-07-03T10:14:07.478 INFO:teuthology.orchestra.run.smithi161.stdout:(11/124): cephfs-mirror-17.0.0-13322.g61829bae. 2.7 MB/s | 272 kB 00:00 2022-07-03T10:14:07.542 INFO:teuthology.orchestra.run.smithi188.stdout:(29/124): rbd-mirror-17.0.0-13322.g61829bae.el8 3.7 MB/s | 3.8 MB 00:01 2022-07-03T10:14:07.553 INFO:teuthology.orchestra.run.smithi161.stdout:(12/124): libcephfs-devel-17.0.0-13322.g61829ba 404 kB/s | 30 kB 00:00 2022-07-03T10:14:07.803 INFO:teuthology.orchestra.run.smithi161.stdout:(13/124): libcephfs2-17.0.0-13322.g61829bae.el8 3.1 MB/s | 796 kB 00:00 2022-07-03T10:14:07.843 INFO:teuthology.orchestra.run.smithi188.stdout:(30/124): ceph-mgr-dashboard-17.0.0-13322.g6182 3.1 MB/s | 1.6 MB 00:00 2022-07-03T10:14:07.895 INFO:teuthology.orchestra.run.smithi101.stdout:(7/124): ceph-mon-17.0.0-13322.g61829bae.el8.x8 1.7 MB/s | 3.9 MB 00:02 2022-07-03T10:14:07.896 INFO:teuthology.orchestra.run.smithi161.stdout:(14/124): libcephsqlite-17.0.0-13322.g61829bae. 1.8 MB/s | 170 kB 00:00 2022-07-03T10:14:07.926 INFO:teuthology.orchestra.run.smithi188.stdout:(31/124): ceph-mgr-modules-core-17.0.0-13322.g6 2.7 MB/s | 229 kB 00:00 2022-07-03T10:14:07.995 INFO:teuthology.orchestra.run.smithi161.stdout:(15/124): librados-devel-17.0.0-13322.g61829bae 1.2 MB/s | 125 kB 00:00 2022-07-03T10:14:08.010 INFO:teuthology.orchestra.run.smithi188.stdout:(32/124): ceph-mgr-rook-17.0.0-13322.g61829bae. 564 kB/s | 47 kB 00:00 2022-07-03T10:14:08.093 INFO:teuthology.orchestra.run.smithi188.stdout:(33/124): ceph-prometheus-alerts-17.0.0-13322.g 176 kB/s | 15 kB 00:00 2022-07-03T10:14:08.188 INFO:teuthology.orchestra.run.smithi161.stdout:(16/124): ceph-osd-17.0.0-13322.g61829bae.el8.x 3.1 MB/s | 17 MB 00:05 2022-07-03T10:14:08.230 INFO:teuthology.orchestra.run.smithi161.stdout:(17/124): libradosstriper1-17.0.0-13322.g61829b 2.0 MB/s | 485 kB 00:00 2022-07-03T10:14:08.235 INFO:teuthology.orchestra.run.smithi188.stdout:(34/124): ceph-volume-17.0.0-13322.g61829bae.el 1.8 MB/s | 259 kB 00:00 2022-07-03T10:14:08.288 INFO:teuthology.orchestra.run.smithi161.stdout:(18/124): python3-ceph-argparse-17.0.0-13322.g6 765 kB/s | 45 kB 00:00 2022-07-03T10:14:08.327 INFO:teuthology.orchestra.run.smithi188.stdout:(35/124): cephadm-17.0.0-13322.g61829bae.el8.no 863 kB/s | 79 kB 00:00 2022-07-03T10:14:08.397 INFO:teuthology.orchestra.run.smithi161.stdout:(19/124): python3-ceph-common-17.0.0-13322.g618 977 kB/s | 106 kB 00:00 2022-07-03T10:14:08.419 INFO:teuthology.orchestra.run.smithi188.stdout:(36/124): cephfs-top-17.0.0-13322.g61829bae.el8 157 kB/s | 14 kB 00:00 2022-07-03T10:14:08.480 INFO:teuthology.orchestra.run.smithi161.stdout:(20/124): python3-cephfs-17.0.0-13322.g61829bae 2.1 MB/s | 178 kB 00:00 2022-07-03T10:14:08.630 INFO:teuthology.orchestra.run.smithi161.stdout:(21/124): python3-rados-17.0.0-13322.g61829bae. 2.4 MB/s | 367 kB 00:00 2022-07-03T10:14:08.653 INFO:teuthology.orchestra.run.smithi188.stdout:(37/124): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-07-03T10:14:08.711 INFO:teuthology.orchestra.run.smithi188.stdout:(38/124): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2022-07-03T10:14:08.770 INFO:teuthology.orchestra.run.smithi188.stdout:(39/124): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2022-07-03T10:14:08.781 INFO:teuthology.orchestra.run.smithi161.stdout:(22/124): python3-rbd-17.0.0-13322.g61829bae.el 2.3 MB/s | 346 kB 00:00 2022-07-03T10:14:08.862 INFO:teuthology.orchestra.run.smithi188.stdout:(40/124): python3-asyncssh-2.7.0-2.el8.noarch.r 5.1 MB/s | 476 kB 00:00 2022-07-03T10:14:08.864 INFO:teuthology.orchestra.run.smithi161.stdout:(23/124): python3-rgw-17.0.0-13322.g61829bae.el 1.3 MB/s | 109 kB 00:00 2022-07-03T10:14:08.912 INFO:teuthology.orchestra.run.smithi188.stdout:(41/124): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 876 kB/s | 44 kB 00:00 2022-07-03T10:14:08.948 INFO:teuthology.orchestra.run.smithi161.stdout:(24/124): rbd-fuse-17.0.0-13322.g61829bae.el8.x 1.1 MB/s | 93 kB 00:00 2022-07-03T10:14:08.971 INFO:teuthology.orchestra.run.smithi188.stdout:(42/124): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2022-07-03T10:14:09.021 INFO:teuthology.orchestra.run.smithi188.stdout:(43/124): python3-cachetools-3.1.1-4.el8.noarch 664 kB/s | 33 kB 00:00 2022-07-03T10:14:09.072 INFO:teuthology.orchestra.run.smithi188.stdout:(44/124): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2022-07-03T10:14:09.130 INFO:teuthology.orchestra.run.smithi188.stdout:(45/124): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2022-07-03T10:14:09.230 INFO:teuthology.orchestra.run.smithi188.stdout:(46/124): python3-cherrypy-18.4.0-1.el8.noarch. 3.8 MB/s | 384 kB 00:00 2022-07-03T10:14:09.314 INFO:teuthology.orchestra.run.smithi188.stdout:(47/124): ceph-mgr-diskprediction-local-17.0.0- 4.2 MB/s | 7.4 MB 00:01 2022-07-03T10:14:09.339 INFO:teuthology.orchestra.run.smithi188.stdout:(48/124): python3-google-auth-1.1.1-10.el8.noar 749 kB/s | 81 kB 00:00 2022-07-03T10:14:09.398 INFO:teuthology.orchestra.run.smithi188.stdout:(49/124): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2022-07-03T10:14:09.423 INFO:teuthology.orchestra.run.smithi188.stdout:(50/124): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2022-07-03T10:14:09.515 INFO:teuthology.orchestra.run.smithi188.stdout:(51/124): python3-logutils-0.3.5-11.el8.noarch. 532 kB/s | 49 kB 00:00 2022-07-03T10:14:09.591 INFO:teuthology.orchestra.run.smithi188.stdout:(52/124): python3-kubernetes-11.0.0-6.el8.noarc 6.1 MB/s | 1.2 MB 00:00 2022-07-03T10:14:09.603 INFO:teuthology.orchestra.run.smithi101.stdout:(8/124): ceph-common-17.0.0-13322.g61829bae.el8 3.2 MB/s | 24 MB 00:07 2022-07-03T10:14:09.616 INFO:teuthology.orchestra.run.smithi188.stdout:(53/124): python3-more-itertools-7.2.0-3.el8.no 584 kB/s | 59 kB 00:00 2022-07-03T10:14:09.650 INFO:teuthology.orchestra.run.smithi188.stdout:(54/124): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2022-07-03T10:14:09.661 INFO:teuthology.orchestra.run.smithi101.stdout:(9/124): ceph-selinux-17.0.0-13322.g61829bae.el 421 kB/s | 24 kB 00:00 2022-07-03T10:14:09.673 INFO:teuthology.orchestra.run.smithi161.stdout:(25/124): librgw2-17.0.0-13322.g61829bae.el8.x8 3.7 MB/s | 5.5 MB 00:01 2022-07-03T10:14:09.717 INFO:teuthology.orchestra.run.smithi188.stdout:(55/124): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2022-07-03T10:14:09.742 INFO:teuthology.orchestra.run.smithi188.stdout:(56/124): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2022-07-03T10:14:09.765 INFO:teuthology.orchestra.run.smithi161.stdout:(26/124): rbd-mirror-17.0.0-13322.g61829bae.el8 4.6 MB/s | 3.8 MB 00:00 2022-07-03T10:14:09.775 INFO:teuthology.orchestra.run.smithi188.stdout:(57/124): python3-repoze-lru-0.7-6.el8.noarch.r 577 kB/s | 34 kB 00:00 2022-07-03T10:14:09.791 INFO:teuthology.orchestra.run.smithi161.stdout:(27/124): rbd-nbd-17.0.0-13322.g61829bae.el8.x8 1.5 MB/s | 175 kB 00:00 2022-07-03T10:14:09.801 INFO:teuthology.orchestra.run.smithi188.stdout:(58/124): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2022-07-03T10:14:09.833 INFO:teuthology.orchestra.run.smithi161.stdout:(28/124): ceph-grafana-dashboards-17.0.0-13322. 342 kB/s | 23 kB 00:00 2022-07-03T10:14:09.834 INFO:teuthology.orchestra.run.smithi188.stdout:(59/124): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2022-07-03T10:14:09.858 INFO:teuthology.orchestra.run.smithi161.stdout:(29/124): ceph-mgr-cephadm-17.0.0-13322.g61829b 1.6 MB/s | 111 kB 00:00 2022-07-03T10:14:09.896 INFO:teuthology.orchestra.run.smithi188.stdout:(60/124): python3-simplegeneric-0.8.1-17.el8.no 197 kB/s | 19 kB 00:00 2022-07-03T10:14:10.021 INFO:teuthology.orchestra.run.smithi188.stdout:(61/124): python3-singledispatch-3.4.0.3-18.el8 128 kB/s | 24 kB 00:00 2022-07-03T10:14:10.097 INFO:teuthology.orchestra.run.smithi188.stdout:(62/124): python3-tempora-1.14.1-5.el8.noarch.r 146 kB/s | 29 kB 00:00 2022-07-03T10:14:10.114 INFO:teuthology.orchestra.run.smithi188.stdout:(63/124): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2022-07-03T10:14:10.156 INFO:teuthology.orchestra.run.smithi188.stdout:(64/124): python3-waitress-1.2.1-2.el8.1.noarch 4.0 MB/s | 241 kB 00:00 2022-07-03T10:14:10.181 INFO:teuthology.orchestra.run.smithi188.stdout:(65/124): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2022-07-03T10:14:10.206 INFO:teuthology.orchestra.run.smithi188.stdout:(66/124): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-07-03T10:14:10.208 INFO:teuthology.orchestra.run.smithi161.stdout:(30/124): ceph-mgr-dashboard-17.0.0-13322.g6182 4.4 MB/s | 1.6 MB 00:00 2022-07-03T10:14:10.231 INFO:teuthology.orchestra.run.smithi188.stdout:(67/124): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2022-07-03T10:14:10.265 INFO:teuthology.orchestra.run.smithi188.stdout:(68/124): python3-zc-lockfile-2.0-2.el8.noarch. 391 kB/s | 23 kB 00:00 2022-07-03T10:14:10.292 INFO:teuthology.orchestra.run.smithi161.stdout:(31/124): ceph-mgr-modules-core-17.0.0-13322.g6 2.7 MB/s | 229 kB 00:00 2022-07-03T10:14:10.324 INFO:teuthology.orchestra.run.smithi188.stdout:(69/124): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2022-07-03T10:14:10.382 INFO:teuthology.orchestra.run.smithi188.stdout:(70/124): python3-jinja2-2.10.1-3.el8.noarch.rp 9.1 MB/s | 538 kB 00:00 2022-07-03T10:14:10.404 INFO:teuthology.orchestra.run.smithi161.stdout:(32/124): ceph-mgr-rook-17.0.0-13322.g61829bae. 422 kB/s | 47 kB 00:00 2022-07-03T10:14:10.424 INFO:teuthology.orchestra.run.smithi188.stdout:(71/124): python3-markupsafe-0.23-19.el8.x86_64 936 kB/s | 39 kB 00:00 2022-07-03T10:14:10.474 INFO:teuthology.orchestra.run.smithi188.stdout:(72/124): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2022-07-03T10:14:10.484 INFO:teuthology.orchestra.run.smithi161.stdout:(33/124): ceph-prometheus-alerts-17.0.0-13322.g 183 kB/s | 15 kB 00:00 2022-07-03T10:14:10.516 INFO:teuthology.orchestra.run.smithi188.stdout:(73/124): thrift-0.13.0-2.el8.x86_64.rpm 6.1 MB/s | 1.7 MB 00:00 2022-07-03T10:14:10.567 INFO:teuthology.orchestra.run.smithi161.stdout:(34/124): ceph-volume-17.0.0-13322.g61829bae.el 3.0 MB/s | 259 kB 00:00 2022-07-03T10:14:10.575 INFO:teuthology.orchestra.run.smithi188.stdout:(74/124): python3-prettytable-0.7.2-14.el8.noar 758 kB/s | 44 kB 00:00 2022-07-03T10:14:10.625 INFO:teuthology.orchestra.run.smithi188.stdout:(75/124): python3-werkzeug-0.12.2-4.el8.noarch. 9.0 MB/s | 457 kB 00:00 2022-07-03T10:14:10.643 INFO:teuthology.orchestra.run.smithi161.stdout:(35/124): cephadm-17.0.0-13322.g61829bae.el8.no 1.0 MB/s | 79 kB 00:00 2022-07-03T10:14:10.684 INFO:teuthology.orchestra.run.smithi188.stdout:(76/124): python3-numpy-1.14.3-10.el8.x86_64.rp 18 MB/s | 3.7 MB 00:00 2022-07-03T10:14:10.726 INFO:teuthology.orchestra.run.smithi188.stdout:(77/124): python3-mako-1.0.6-13.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2022-07-03T10:14:10.740 INFO:teuthology.orchestra.run.smithi161.stdout:(36/124): cephfs-top-17.0.0-13322.g61829bae.el8 148 kB/s | 14 kB 00:00 2022-07-03T10:14:10.797 INFO:teuthology.orchestra.run.smithi188.stdout:(78/124): libtool-2.4.6-25.el8.x86_64.rpm 6.4 MB/s | 709 kB 00:00 2022-07-03T10:14:10.843 INFO:teuthology.orchestra.run.smithi188.stdout:(79/124): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.9 MB/s | 225 kB 00:00 2022-07-03T10:14:10.885 INFO:teuthology.orchestra.run.smithi188.stdout:(80/124): python3-jsonpointer-1.10-11.el8.noarc 238 kB/s | 20 kB 00:00 2022-07-03T10:14:10.911 INFO:teuthology.orchestra.run.smithi188.stdout:(81/124): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2022-07-03T10:14:10.961 INFO:teuthology.orchestra.run.smithi188.stdout:(82/124): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.7 MB/s | 126 kB 00:00 2022-07-03T10:14:10.968 INFO:teuthology.orchestra.run.smithi161.stdout:(37/124): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-07-03T10:14:11.027 INFO:teuthology.orchestra.run.smithi161.stdout:(38/124): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2022-07-03T10:14:11.061 INFO:teuthology.orchestra.run.smithi188.stdout:(83/124): openblas-threads-0.3.15-3.el8.x86_64. 28 MB/s | 4.9 MB 00:00 2022-07-03T10:14:11.102 INFO:teuthology.orchestra.run.smithi161.stdout:(39/124): libunwind-1.3.1-3.el8.x86_64.rpm 998 kB/s | 75 kB 00:00 2022-07-03T10:14:11.103 INFO:teuthology.orchestra.run.smithi188.stdout:(84/124): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2022-07-03T10:14:11.178 INFO:teuthology.orchestra.run.smithi188.stdout:(85/124): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2022-07-03T10:14:11.202 INFO:teuthology.orchestra.run.smithi161.stdout:(40/124): python3-asyncssh-2.7.0-2.el8.noarch.r 4.7 MB/s | 476 kB 00:00 2022-07-03T10:14:11.203 INFO:teuthology.orchestra.run.smithi188.stdout:(86/124): python3-pytz-2017.2-9.el8.noarch.rpm 541 kB/s | 54 kB 00:00 2022-07-03T10:14:11.253 INFO:teuthology.orchestra.run.smithi161.stdout:(41/124): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 874 kB/s | 44 kB 00:00 2022-07-03T10:14:11.262 INFO:teuthology.orchestra.run.smithi188.stdout:(87/124): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2022-07-03T10:14:11.304 INFO:teuthology.orchestra.run.smithi188.stdout:(88/124): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2022-07-03T10:14:11.311 INFO:teuthology.orchestra.run.smithi161.stdout:(42/124): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2022-07-03T10:14:11.346 INFO:teuthology.orchestra.run.smithi188.stdout:(89/124): python3-jsonpatch-1.21-2.el8.noarch.r 637 kB/s | 27 kB 00:00 2022-07-03T10:14:11.362 INFO:teuthology.orchestra.run.smithi161.stdout:(43/124): python3-cachetools-3.1.1-4.el8.noarch 663 kB/s | 33 kB 00:00 2022-07-03T10:14:11.388 INFO:teuthology.orchestra.run.smithi188.stdout:(90/124): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2022-07-03T10:14:11.430 INFO:teuthology.orchestra.run.smithi188.stdout:(91/124): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2022-07-03T10:14:11.437 INFO:teuthology.orchestra.run.smithi161.stdout:(44/124): python3-certifi-2018.10.15-7.el8.noar 217 kB/s | 16 kB 00:00 2022-07-03T10:14:11.496 INFO:teuthology.orchestra.run.smithi161.stdout:(45/124): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2022-07-03T10:14:11.573 INFO:teuthology.orchestra.run.smithi188.stdout:(92/124): python3-scipy-1.0.0-21.module+el8.5.0 35 MB/s | 14 MB 00:00 2022-07-03T10:14:11.579 INFO:teuthology.orchestra.run.smithi161.stdout:(46/124): ceph-mgr-diskprediction-local-17.0.0- 4.3 MB/s | 7.4 MB 00:01 2022-07-03T10:14:11.605 INFO:teuthology.orchestra.run.smithi161.stdout:(47/124): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2022-07-03T10:14:11.614 INFO:teuthology.orchestra.run.smithi188.stdout:(93/124): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2022-07-03T10:14:11.656 INFO:teuthology.orchestra.run.smithi188.stdout:(94/124): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2022-07-03T10:14:11.663 INFO:teuthology.orchestra.run.smithi161.stdout:(48/124): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2022-07-03T10:14:11.722 INFO:teuthology.orchestra.run.smithi161.stdout:(49/124): python3-jaraco-functools-2.0-4.el8.no 302 kB/s | 18 kB 00:00 2022-07-03T10:14:11.780 INFO:teuthology.orchestra.run.smithi161.stdout:(50/124): python3-google-auth-1.1.1-10.el8.noar 405 kB/s | 81 kB 00:00 2022-07-03T10:14:11.839 INFO:teuthology.orchestra.run.smithi161.stdout:(51/124): python3-logutils-0.3.5-11.el8.noarch. 838 kB/s | 49 kB 00:00 2022-07-03T10:14:11.897 INFO:teuthology.orchestra.run.smithi161.stdout:(52/124): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2022-07-03T10:14:11.931 INFO:teuthology.orchestra.run.smithi161.stdout:(53/124): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2022-07-03T10:14:11.956 INFO:teuthology.orchestra.run.smithi161.stdout:(54/124): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2022-07-03T10:14:11.990 INFO:teuthology.orchestra.run.smithi161.stdout:(55/124): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2022-07-03T10:14:12.015 INFO:teuthology.orchestra.run.smithi161.stdout:(56/124): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2022-07-03T10:14:12.049 INFO:teuthology.orchestra.run.smithi161.stdout:(57/124): python3-repoze-lru-0.7-6.el8.noarch.r 573 kB/s | 34 kB 00:00 2022-07-03T10:14:12.124 INFO:teuthology.orchestra.run.smithi161.stdout:(58/124): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2022-07-03T10:14:12.128 INFO:teuthology.orchestra.run.smithi101.stdout:(10/124): ceph-osd-17.0.0-13322.g61829bae.el8.x 2.8 MB/s | 17 MB 00:06 2022-07-03T10:14:12.131 INFO:teuthology.orchestra.run.smithi188.stdout:(95/124): ceph-test-17.0.0-13322.g61829bae.el8. 5.3 MB/s | 53 MB 00:09 2022-07-03T10:14:12.150 INFO:teuthology.orchestra.run.smithi161.stdout:(59/124): python3-rsa-4.8-1.el8.noarch.rpm 615 kB/s | 61 kB 00:00 2022-07-03T10:14:12.181 INFO:teuthology.orchestra.run.smithi188.stdout:(96/124): oniguruma-6.8.2-2.el8.x86_64.rpm 357 kB/s | 187 kB 00:00 2022-07-03T10:14:12.183 INFO:teuthology.orchestra.run.smithi161.stdout:(60/124): python3-simplegeneric-0.8.1-17.el8.no 320 kB/s | 19 kB 00:00 2022-07-03T10:14:12.209 INFO:teuthology.orchestra.run.smithi161.stdout:(61/124): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2022-07-03T10:14:12.228 INFO:teuthology.orchestra.run.smithi101.stdout:(11/124): cephfs-mirror-17.0.0-13322.g61829bae. 2.7 MB/s | 272 kB 00:00 2022-07-03T10:14:12.232 INFO:teuthology.orchestra.run.smithi188.stdout:(97/124): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.7 MB/s | 560 kB 00:00 2022-07-03T10:14:12.242 INFO:teuthology.orchestra.run.smithi161.stdout:(62/124): python3-tempora-1.14.1-5.el8.noarch.r 499 kB/s | 29 kB 00:00 2022-07-03T10:14:12.257 INFO:teuthology.orchestra.run.smithi188.stdout:(98/124): python3-oauthlib-2.1.0-1.el8.noarch.r 2.1 MB/s | 155 kB 00:00 2022-07-03T10:14:12.268 INFO:teuthology.orchestra.run.smithi161.stdout:(63/124): python3-trustme-0.6.0-4.el8.noarch.rp 470 kB/s | 27 kB 00:00 2022-07-03T10:14:12.279 INFO:teuthology.orchestra.run.smithi101.stdout:(12/124): libcephfs-devel-17.0.0-13322.g61829ba 611 kB/s | 30 kB 00:00 2022-07-03T10:14:12.301 INFO:teuthology.orchestra.run.smithi161.stdout:(64/124): python3-waitress-1.2.1-2.el8.1.noarch 4.0 MB/s | 241 kB 00:00 2022-07-03T10:14:12.324 INFO:teuthology.orchestra.run.smithi188.stdout:(99/124): python3-babel-2.5.1-7.el8.noarch.rpm 6.4 MB/s | 4.8 MB 00:00 2022-07-03T10:14:12.326 INFO:teuthology.orchestra.run.smithi161.stdout:(65/124): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2022-07-03T10:14:12.360 INFO:teuthology.orchestra.run.smithi161.stdout:(66/124): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2022-07-03T10:14:12.366 INFO:teuthology.orchestra.run.smithi188.stdout:(100/124): python3-pycparser-2.14-14.el8.noarch 816 kB/s | 109 kB 00:00 2022-07-03T10:14:12.385 INFO:teuthology.orchestra.run.smithi161.stdout:(67/124): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2022-07-03T10:14:12.408 INFO:teuthology.orchestra.run.smithi188.stdout:(101/124): python3-cryptography-3.2.1-5.el8.x86 3.7 MB/s | 559 kB 00:00 2022-07-03T10:14:12.433 INFO:teuthology.orchestra.run.smithi188.stdout:(102/124): libverto-devel-0.3.0-5.el8.x86_64.rp 164 kB/s | 18 kB 00:00 2022-07-03T10:14:12.442 INFO:teuthology.orchestra.run.smithi161.stdout:(68/124): python3-zc-lockfile-2.0-2.el8.noarch. 280 kB/s | 23 kB 00:00 2022-07-03T10:14:12.458 INFO:teuthology.orchestra.run.smithi188.stdout:(103/124): libselinux-devel-2.9-5.el8.x86_64.rp 2.1 MB/s | 200 kB 00:00 2022-07-03T10:14:12.471 INFO:teuthology.orchestra.run.smithi101.stdout:(13/124): libcephfs2-17.0.0-13322.g61829bae.el8 4.1 MB/s | 796 kB 00:00 2022-07-03T10:14:12.503 INFO:teuthology.orchestra.run.smithi161.stdout:(69/124): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2022-07-03T10:14:12.509 INFO:teuthology.orchestra.run.smithi188.stdout:(104/124): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.2 MB/s | 220 kB 00:00 2022-07-03T10:14:12.529 INFO:teuthology.orchestra.run.smithi101.stdout:(14/124): libcephsqlite-17.0.0-13322.g61829bae. 2.9 MB/s | 170 kB 00:00 2022-07-03T10:14:12.534 INFO:teuthology.orchestra.run.smithi188.stdout:(105/124): libsepol-devel-2.9-3.el8.x86_64.rpm 868 kB/s | 87 kB 00:00 2022-07-03T10:14:12.559 INFO:teuthology.orchestra.run.smithi188.stdout:(106/124): librabbitmq-0.9.0-3.el8.x86_64.rpm 487 kB/s | 47 kB 00:00 2022-07-03T10:14:12.561 INFO:teuthology.orchestra.run.smithi161.stdout:(70/124): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.4 MB/s | 560 kB 00:00 2022-07-03T10:14:12.588 INFO:teuthology.orchestra.run.smithi101.stdout:(15/124): librados-devel-17.0.0-13322.g61829bae 2.1 MB/s | 125 kB 00:00 2022-07-03T10:14:12.611 INFO:teuthology.orchestra.run.smithi161.stdout:(71/124): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2022-07-03T10:14:12.618 INFO:teuthology.orchestra.run.smithi188.stdout:(107/124): pcre2-devel-10.32-2.el8.x86_64.rpm 5.6 MB/s | 605 kB 00:00 2022-07-03T10:14:12.643 INFO:teuthology.orchestra.run.smithi188.stdout:(108/124): libaio-devel-0.3.112-1.el8.x86_64.rp 172 kB/s | 19 kB 00:00 2022-07-03T10:14:12.653 INFO:teuthology.orchestra.run.smithi161.stdout:(72/124): thrift-0.13.0-2.el8.x86_64.rpm 6.5 MB/s | 1.7 MB 00:00 2022-07-03T10:14:12.669 INFO:teuthology.orchestra.run.smithi188.stdout:(109/124): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.2 MB/s | 238 kB 00:00 2022-07-03T10:14:12.704 INFO:teuthology.orchestra.run.smithi161.stdout:(73/124): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2022-07-03T10:14:12.713 INFO:teuthology.orchestra.run.smithi101.stdout:(16/124): libradosstriper1-17.0.0-13322.g61829b 3.8 MB/s | 485 kB 00:00 2022-07-03T10:14:12.727 INFO:teuthology.orchestra.run.smithi188.stdout:(110/124): libuuid-devel-2.32.1-35.el8.x86_64.r 903 kB/s | 98 kB 00:00 2022-07-03T10:14:12.752 INFO:teuthology.orchestra.run.smithi188.stdout:(111/124): zlib-devel-1.2.11-18.el8_5.x86_64.rp 530 kB/s | 58 kB 00:00 2022-07-03T10:14:12.778 INFO:teuthology.orchestra.run.smithi188.stdout:(112/124): fuse-2.9.7-15.el8.x86_64.rpm 792 kB/s | 83 kB 00:00 2022-07-03T10:14:12.788 INFO:teuthology.orchestra.run.smithi161.stdout:(74/124): python3-cryptography-3.2.1-5.el8.x86_ 4.2 MB/s | 559 kB 00:00 2022-07-03T10:14:12.813 INFO:teuthology.orchestra.run.smithi161.stdout:(75/124): libverto-devel-0.3.0-5.el8.x86_64.rpm 164 kB/s | 18 kB 00:00 2022-07-03T10:14:12.828 INFO:teuthology.orchestra.run.smithi188.stdout:(113/124): python3-jwt-1.6.1-2.el8.noarch.rpm 423 kB/s | 43 kB 00:00 2022-07-03T10:14:12.853 INFO:teuthology.orchestra.run.smithi188.stdout:(114/124): libcom_err-devel-1.45.6-4.el8.x86_64 384 kB/s | 39 kB 00:00 2022-07-03T10:14:12.863 INFO:teuthology.orchestra.run.smithi161.stdout:(76/124): libselinux-devel-2.9-5.el8.x86_64.rpm 2.6 MB/s | 200 kB 00:00 2022-07-03T10:14:12.878 INFO:teuthology.orchestra.run.smithi188.stdout:(115/124): elfutils-libelf-devel-0.186-1.el8.x8 622 kB/s | 60 kB 00:00 2022-07-03T10:14:12.888 INFO:teuthology.orchestra.run.smithi161.stdout:(77/124): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2022-07-03T10:14:12.921 INFO:teuthology.orchestra.run.smithi188.stdout:(116/124): libquadmath-8.5.0-10.el8.x86_64.rpm 1.8 MB/s | 171 kB 00:00 2022-07-03T10:14:12.922 INFO:teuthology.orchestra.run.smithi101.stdout:(17/124): ceph-radosgw-17.0.0-13322.g61829bae.e 3.3 MB/s | 17 MB 00:05 2022-07-03T10:14:12.938 INFO:teuthology.orchestra.run.smithi161.stdout:(78/124): libsepol-devel-2.9-3.el8.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2022-07-03T10:14:12.954 INFO:teuthology.orchestra.run.smithi188.stdout:(117/124): keyutils-libs-devel-1.5.10-9.el8.x86 479 kB/s | 48 kB 00:00 2022-07-03T10:14:12.955 INFO:teuthology.orchestra.run.smithi161.stdout:(79/124): librabbitmq-0.9.0-3.el8.x86_64.rpm 746 kB/s | 47 kB 00:00 2022-07-03T10:14:12.973 INFO:teuthology.orchestra.run.smithi101.stdout:(18/124): python3-ceph-argparse-17.0.0-13322.g6 883 kB/s | 45 kB 00:00 2022-07-03T10:14:12.996 INFO:teuthology.orchestra.run.smithi188.stdout:(118/124): libgfortran-8.5.0-10.el8.x86_64.rpm 5.6 MB/s | 643 kB 00:00 2022-07-03T10:14:12.997 INFO:teuthology.orchestra.run.smithi161.stdout:(80/124): libaio-devel-0.3.112-1.el8.x86_64.rpm 448 kB/s | 19 kB 00:00 2022-07-03T10:14:13.021 INFO:teuthology.orchestra.run.smithi188.stdout:(119/124): xfsprogs-devel-5.0.0-10.el8.x86_64.r 770 kB/s | 72 kB 00:00 2022-07-03T10:14:13.031 INFO:teuthology.orchestra.run.smithi161.stdout:(81/124): pcre2-devel-10.32-2.el8.x86_64.rpm 6.4 MB/s | 605 kB 00:00 2022-07-03T10:14:13.032 INFO:teuthology.orchestra.run.smithi101.stdout:(19/124): python3-ceph-common-17.0.0-13322.g618 1.8 MB/s | 106 kB 00:00 2022-07-03T10:14:13.063 INFO:teuthology.orchestra.run.smithi188.stdout:(120/124): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2022-07-03T10:14:13.073 INFO:teuthology.orchestra.run.smithi161.stdout:(82/124): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2022-07-03T10:14:13.090 INFO:teuthology.orchestra.run.smithi161.stdout:(83/124): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.6 MB/s | 98 kB 00:00 2022-07-03T10:14:13.098 INFO:teuthology.orchestra.run.smithi101.stdout:(20/124): python3-cephfs-17.0.0-13322.g61829bae 2.6 MB/s | 178 kB 00:00 2022-07-03T10:14:13.105 INFO:teuthology.orchestra.run.smithi188.stdout:(121/124): python3-requests-oauthlib-1.0.0-1.el 393 kB/s | 43 kB 00:00 2022-07-03T10:14:13.188 INFO:teuthology.orchestra.run.smithi188.stdout:(122/124): openssl-devel-1.1.1k-6.el8_5.x86_64. 14 MB/s | 2.3 MB 00:00 2022-07-03T10:14:13.199 INFO:teuthology.orchestra.run.smithi101.stdout:(21/124): python3-rados-17.0.0-13322.g61829bae. 3.6 MB/s | 367 kB 00:00 2022-07-03T10:14:13.207 INFO:teuthology.orchestra.run.smithi161.stdout:(84/124): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 432 kB/s | 58 kB 00:00 2022-07-03T10:14:13.299 INFO:teuthology.orchestra.run.smithi101.stdout:(22/124): python3-rbd-17.0.0-13322.g61829bae.el 3.4 MB/s | 346 kB 00:00 2022-07-03T10:14:13.324 INFO:teuthology.orchestra.run.smithi161.stdout:(85/124): fuse-2.9.7-15.el8.x86_64.rpm 360 kB/s | 83 kB 00:00 2022-07-03T10:14:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:(86/124): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2022-07-03T10:14:13.382 INFO:teuthology.orchestra.run.smithi101.stdout:(23/124): python3-rgw-17.0.0-13322.g61829bae.el 1.3 MB/s | 109 kB 00:00 2022-07-03T10:14:13.399 INFO:teuthology.orchestra.run.smithi161.stdout:(87/124): libcom_err-devel-1.45.6-4.el8.x86_64. 515 kB/s | 39 kB 00:00 2022-07-03T10:14:13.458 INFO:teuthology.orchestra.run.smithi101.stdout:(24/124): rbd-fuse-17.0.0-13322.g61829bae.el8.x 1.2 MB/s | 93 kB 00:00 2022-07-03T10:14:13.470 INFO:teuthology.orchestra.run.smithi161.stdout:(88/124): elfutils-libelf-devel-0.186-1.el8.x86 627 kB/s | 60 kB 00:00 2022-07-03T10:14:13.500 INFO:teuthology.orchestra.run.smithi161.stdout:(89/124): libquadmath-8.5.0-10.el8.x86_64.rpm 1.7 MB/s | 171 kB 00:00 2022-07-03T10:14:13.542 INFO:teuthology.orchestra.run.smithi161.stdout:(90/124): keyutils-libs-devel-1.5.10-9.el8.x86_ 672 kB/s | 48 kB 00:00 2022-07-03T10:14:13.575 INFO:teuthology.orchestra.run.smithi161.stdout:(91/124): libgfortran-8.5.0-10.el8.x86_64.rpm 8.9 MB/s | 643 kB 00:00 2022-07-03T10:14:13.601 INFO:teuthology.orchestra.run.smithi161.stdout:(92/124): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2022-07-03T10:14:13.643 INFO:teuthology.orchestra.run.smithi161.stdout:(93/124): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.6 MB/s | 229 kB 00:00 2022-07-03T10:14:13.676 INFO:teuthology.orchestra.run.smithi161.stdout:(94/124): python3-requests-oauthlib-1.0.0-1.el8 568 kB/s | 43 kB 00:00 2022-07-03T10:14:13.735 INFO:teuthology.orchestra.run.smithi161.stdout:(95/124): python3-jinja2-2.10.1-3.el8.noarch.rp 9.0 MB/s | 538 kB 00:00 2022-07-03T10:14:13.815 INFO:teuthology.orchestra.run.smithi161.stdout:(96/124): openssl-devel-1.1.1k-6.el8_5.x86_64.r 13 MB/s | 2.3 MB 00:00 2022-07-03T10:14:13.844 INFO:teuthology.orchestra.run.smithi161.stdout:(97/124): python3-markupsafe-0.23-19.el8.x86_64 357 kB/s | 39 kB 00:00 2022-07-03T10:14:13.895 INFO:teuthology.orchestra.run.smithi161.stdout:(98/124): bison-3.0.4-10.el8.x86_64.rpm 8.5 MB/s | 688 kB 00:00 2022-07-03T10:14:13.922 INFO:teuthology.orchestra.run.smithi188.stdout:(123/124): librbd1-17.0.0-13322.g61829bae.el8.x 4.7 MB/s | 3.9 MB 00:00 2022-07-03T10:14:13.928 INFO:teuthology.orchestra.run.smithi161.stdout:(99/124): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2022-07-03T10:14:13.995 INFO:teuthology.orchestra.run.smithi161.stdout:(100/124): python3-numpy-1.14.3-10.el8.x86_64.r 25 MB/s | 3.7 MB 00:00 2022-07-03T10:14:14.025 INFO:teuthology.orchestra.run.smithi101.stdout:(25/124): librgw2-17.0.0-13322.g61829bae.el8.x8 4.2 MB/s | 5.5 MB 00:01 2022-07-03T10:14:14.030 INFO:teuthology.orchestra.run.smithi188.stdout:(124/124): librados2-17.0.0-13322.g61829bae.el8 3.8 MB/s | 3.6 MB 00:00 2022-07-03T10:14:14.033 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:14:14.034 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 204 MB 00:14 2022-07-03T10:14:14.037 INFO:teuthology.orchestra.run.smithi161.stdout:(101/124): python3-werkzeug-0.12.2-4.el8.noarch 4.1 MB/s | 457 kB 00:00 2022-07-03T10:14:14.071 INFO:teuthology.orchestra.run.smithi161.stdout:(102/124): python3-mako-1.0.6-13.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2022-07-03T10:14:14.083 INFO:teuthology.orchestra.run.smithi101.stdout:(26/124): rbd-nbd-17.0.0-13322.g61829bae.el8.x8 3.0 MB/s | 175 kB 00:00 2022-07-03T10:14:14.129 INFO:teuthology.orchestra.run.smithi161.stdout:(103/124): libtool-2.4.6-25.el8.x86_64.rpm 7.5 MB/s | 709 kB 00:00 2022-07-03T10:14:14.133 INFO:teuthology.orchestra.run.smithi101.stdout:(27/124): ceph-grafana-dashboards-17.0.0-13322. 461 kB/s | 23 kB 00:00 2022-07-03T10:14:14.188 INFO:teuthology.orchestra.run.smithi161.stdout:(104/124): python3-numpy-f2py-1.14.3-10.el8.x86 1.9 MB/s | 225 kB 00:00 2022-07-03T10:14:14.192 INFO:teuthology.orchestra.run.smithi101.stdout:(28/124): ceph-mgr-cephadm-17.0.0-13322.g61829b 1.9 MB/s | 111 kB 00:00 2022-07-03T10:14:14.221 INFO:teuthology.orchestra.run.smithi161.stdout:(105/124): python3-jsonpointer-1.10-11.el8.noar 218 kB/s | 20 kB 00:00 2022-07-03T10:14:14.246 INFO:teuthology.orchestra.run.smithi161.stdout:(106/124): librdkafka-0.11.4-3.el8.x86_64.rpm 6.4 MB/s | 354 kB 00:00 2022-07-03T10:14:14.305 INFO:teuthology.orchestra.run.smithi161.stdout:(107/124): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.3 MB/s | 126 kB 00:00 2022-07-03T10:14:14.384 INFO:teuthology.orchestra.run.smithi101.stdout:(29/124): rbd-mirror-17.0.0-13322.g61829bae.el8 4.1 MB/s | 3.8 MB 00:00 2022-07-03T10:14:14.389 INFO:teuthology.orchestra.run.smithi161.stdout:(108/124): openblas-threads-0.3.15-3.el8.x86_64 29 MB/s | 4.9 MB 00:00 2022-07-03T10:14:14.431 INFO:teuthology.orchestra.run.smithi161.stdout:(109/124): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2022-07-03T10:14:14.535 INFO:teuthology.orchestra.run.smithi101.stdout:(30/124): ceph-mgr-dashboard-17.0.0-13322.g6182 4.8 MB/s | 1.6 MB 00:00 2022-07-03T10:14:14.539 INFO:teuthology.orchestra.run.smithi161.stdout:(110/124): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2022-07-03T10:14:14.564 INFO:teuthology.orchestra.run.smithi161.stdout:(111/124): python3-pytz-2017.2-9.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2022-07-03T10:14:14.618 INFO:teuthology.orchestra.run.smithi101.stdout:(31/124): ceph-mgr-modules-core-17.0.0-13322.g6 2.7 MB/s | 229 kB 00:00 2022-07-03T10:14:14.632 INFO:teuthology.orchestra.run.smithi161.stdout:(112/124): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2022-07-03T10:14:14.668 INFO:teuthology.orchestra.run.smithi101.stdout:(32/124): ceph-mgr-rook-17.0.0-13322.g61829bae. 939 kB/s | 47 kB 00:00 2022-07-03T10:14:14.673 INFO:teuthology.orchestra.run.smithi161.stdout:(113/124): python3-pyasn1-modules-0.3.7-6.el8.n 2.6 MB/s | 110 kB 00:00 2022-07-03T10:14:14.715 INFO:teuthology.orchestra.run.smithi161.stdout:(114/124): python3-jsonpatch-1.21-2.el8.noarch. 638 kB/s | 27 kB 00:00 2022-07-03T10:14:14.719 INFO:teuthology.orchestra.run.smithi101.stdout:(33/124): ceph-prometheus-alerts-17.0.0-13322.g 295 kB/s | 15 kB 00:00 2022-07-03T10:14:14.751 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2022-07-03T10:14:14.766 INFO:teuthology.orchestra.run.smithi161.stdout:(115/124): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2022-07-03T10:14:14.819 INFO:teuthology.orchestra.run.smithi101.stdout:(34/124): ceph-volume-17.0.0-13322.g61829bae.el 2.5 MB/s | 259 kB 00:00 2022-07-03T10:14:14.861 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2022-07-03T10:14:14.862 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2022-07-03T10:14:14.869 INFO:teuthology.orchestra.run.smithi101.stdout:(35/124): cephadm-17.0.0-13322.g61829bae.el8.no 1.6 MB/s | 79 kB 00:00 2022-07-03T10:14:14.911 INFO:teuthology.orchestra.run.smithi101.stdout:(36/124): cephfs-top-17.0.0-13322.g61829bae.el8 347 kB/s | 14 kB 00:00 2022-07-03T10:14:14.958 INFO:teuthology.orchestra.run.smithi161.stdout:(116/124): python3-scipy-1.0.0-21.module+el8.5. 33 MB/s | 14 MB 00:00 2022-07-03T10:14:14.983 INFO:teuthology.orchestra.run.smithi161.stdout:(117/124): flex-2.6.1-9.el8.x86_64.rpm 1.4 MB/s | 320 kB 00:00 2022-07-03T10:14:15.033 INFO:teuthology.orchestra.run.smithi161.stdout:(118/124): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2022-07-03T10:14:15.145 INFO:teuthology.orchestra.run.smithi101.stdout:(37/124): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-07-03T10:14:15.203 INFO:teuthology.orchestra.run.smithi101.stdout:(38/124): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2022-07-03T10:14:15.262 INFO:teuthology.orchestra.run.smithi101.stdout:(39/124): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2022-07-03T10:14:15.371 INFO:teuthology.orchestra.run.smithi101.stdout:(40/124): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2022-07-03T10:14:15.471 INFO:teuthology.orchestra.run.smithi101.stdout:(41/124): ceph-mgr-diskprediction-local-17.0.0- 6.8 MB/s | 7.4 MB 00:01 2022-07-03T10:14:15.496 INFO:teuthology.orchestra.run.smithi101.stdout:(42/124): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2022-07-03T10:14:15.500 INFO:teuthology.orchestra.run.smithi161.stdout:(119/124): ceph-test-17.0.0-13322.g61829bae.el8 6.3 MB/s | 53 MB 00:08 2022-07-03T10:14:15.550 INFO:teuthology.orchestra.run.smithi161.stdout:(120/124): python3-pyOpenSSL-19.0.0-1.el8.noarc 200 kB/s | 103 kB 00:00 2022-07-03T10:14:15.555 INFO:teuthology.orchestra.run.smithi101.stdout:(43/124): python3-cachetools-3.1.1-4.el8.noarch 564 kB/s | 33 kB 00:00 2022-07-03T10:14:15.614 INFO:teuthology.orchestra.run.smithi101.stdout:(44/124): python3-certifi-2018.10.15-7.el8.noar 280 kB/s | 16 kB 00:00 2022-07-03T10:14:15.642 INFO:teuthology.orchestra.run.smithi161.stdout:(121/124): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2022-07-03T10:14:15.664 INFO:teuthology.orchestra.run.smithi101.stdout:(45/124): python3-beautifulsoup4-4.6.3-2.el8.1. 963 kB/s | 185 kB 00:00 2022-07-03T10:14:15.667 INFO:teuthology.orchestra.run.smithi161.stdout:(122/124): oniguruma-6.8.2-2.el8.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2022-07-03T10:14:15.689 INFO:teuthology.orchestra.run.smithi101.stdout:(46/124): python3-cheroot-8.5.2-1.el8.noarch.rp 2.3 MB/s | 173 kB 00:00 2022-07-03T10:14:15.748 INFO:teuthology.orchestra.run.smithi101.stdout:(47/124): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2022-07-03T10:14:15.773 INFO:teuthology.orchestra.run.smithi101.stdout:(48/124): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2022-07-03T10:14:15.807 INFO:teuthology.orchestra.run.smithi101.stdout:(49/124): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2022-07-03T10:14:15.832 INFO:teuthology.orchestra.run.smithi101.stdout:(50/124): python3-jaraco-functools-2.0-4.el8.no 300 kB/s | 18 kB 00:00 2022-07-03T10:14:15.890 INFO:teuthology.orchestra.run.smithi101.stdout:(51/124): python3-logutils-0.3.5-11.el8.noarch. 842 kB/s | 49 kB 00:00 2022-07-03T10:14:15.907 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2022-07-03T10:14:15.911 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2022-07-03T10:14:15.950 INFO:teuthology.orchestra.run.smithi101.stdout:(52/124): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2022-07-03T10:14:15.991 INFO:teuthology.orchestra.run.smithi101.stdout:(53/124): python3-kubernetes-11.0.0-6.el8.noarc 6.3 MB/s | 1.2 MB 00:00 2022-07-03T10:14:16.016 INFO:teuthology.orchestra.run.smithi101.stdout:(54/124): python3-natsort-7.1.1-2.el8.noarch.rp 905 kB/s | 60 kB 00:00 2022-07-03T10:14:16.058 INFO:teuthology.orchestra.run.smithi101.stdout:(55/124): python3-pecan-1.3.2-9.el8.noarch.rpm 4.2 MB/s | 283 kB 00:00 2022-07-03T10:14:16.076 INFO:teuthology.orchestra.run.smithi161.stdout:(123/124): librados2-17.0.0-13322.g61829bae.el8 7.0 MB/s | 3.6 MB 00:00 2022-07-03T10:14:16.084 INFO:teuthology.orchestra.run.smithi101.stdout:(56/124): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2022-07-03T10:14:16.117 INFO:teuthology.orchestra.run.smithi101.stdout:(57/124): python3-repoze-lru-0.7-6.el8.noarch.r 574 kB/s | 34 kB 00:00 2022-07-03T10:14:16.143 INFO:teuthology.orchestra.run.smithi101.stdout:(58/124): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2022-07-03T10:14:16.176 INFO:teuthology.orchestra.run.smithi101.stdout:(59/124): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2022-07-03T10:14:16.202 INFO:teuthology.orchestra.run.smithi101.stdout:(60/124): python3-simplegeneric-0.8.1-17.el8.no 317 kB/s | 19 kB 00:00 2022-07-03T10:14:16.235 INFO:teuthology.orchestra.run.smithi161.stdout:(124/124): librbd1-17.0.0-13322.g61829bae.el8.x 6.5 MB/s | 3.9 MB 00:00 2022-07-03T10:14:16.238 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:14:16.238 INFO:teuthology.orchestra.run.smithi161.stdout:Total 13 MB/s | 204 MB 00:15 2022-07-03T10:14:16.736 INFO:teuthology.orchestra.run.smithi101.stdout:(61/124): ceph-test-17.0.0-13322.g61829bae.el8. 7.5 MB/s | 53 MB 00:07 2022-07-03T10:14:16.752 INFO:teuthology.orchestra.run.smithi101.stdout:(62/124): python3-singledispatch-3.4.0.3-18.el8 41 kB/s | 24 kB 00:00 2022-07-03T10:14:16.769 INFO:teuthology.orchestra.run.smithi101.stdout:(63/124): python3-tempora-1.14.1-5.el8.noarch.r 52 kB/s | 29 kB 00:00 2022-07-03T10:14:16.861 INFO:teuthology.orchestra.run.smithi101.stdout:(64/124): python3-trustme-0.6.0-4.el8.noarch.rp 221 kB/s | 27 kB 00:00 2022-07-03T10:14:16.978 INFO:teuthology.orchestra.run.smithi101.stdout:(65/124): python3-waitress-1.2.1-2.el8.1.noarch 1.1 MB/s | 241 kB 00:00 2022-07-03T10:14:16.980 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2022-07-03T10:14:17.003 INFO:teuthology.orchestra.run.smithi101.stdout:(66/124): python3-webob-1.8.5-1.el8.1.noarch.rp 1.1 MB/s | 251 kB 00:00 2022-07-03T10:14:17.020 INFO:teuthology.orchestra.run.smithi101.stdout:(67/124): python3-websocket-client-0.56.0-5.el8 386 kB/s | 61 kB 00:00 2022-07-03T10:14:17.046 INFO:teuthology.orchestra.run.smithi101.stdout:(68/124): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2022-07-03T10:14:17.071 INFO:teuthology.orchestra.run.smithi101.stdout:(69/124): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2022-07-03T10:14:17.094 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2022-07-03T10:14:17.095 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2022-07-03T10:14:17.105 INFO:teuthology.orchestra.run.smithi101.stdout:(70/124): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2022-07-03T10:14:17.172 INFO:teuthology.orchestra.run.smithi101.stdout:(71/124): python3-jinja2-2.10.1-3.el8.noarch.rp 5.3 MB/s | 538 kB 00:00 2022-07-03T10:14:17.197 INFO:teuthology.orchestra.run.smithi101.stdout:(72/124): python3-markupsafe-0.23-19.el8.x86_64 425 kB/s | 39 kB 00:00 2022-07-03T10:14:17.255 INFO:teuthology.orchestra.run.smithi101.stdout:(73/124): bison-3.0.4-10.el8.x86_64.rpm 8.1 MB/s | 688 kB 00:00 2022-07-03T10:14:17.314 INFO:teuthology.orchestra.run.smithi101.stdout:(74/124): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2022-07-03T10:14:17.339 INFO:teuthology.orchestra.run.smithi101.stdout:(75/124): python3-prettytable-0.7.2-14.el8.noar 528 kB/s | 44 kB 00:00 2022-07-03T10:14:17.406 INFO:teuthology.orchestra.run.smithi101.stdout:(76/124): python3-mako-1.0.6-13.el8.noarch.rpm 2.5 MB/s | 157 kB 00:00 2022-07-03T10:14:17.456 INFO:teuthology.orchestra.run.smithi101.stdout:(77/124): python3-werkzeug-0.12.2-4.el8.noarch. 3.2 MB/s | 457 kB 00:00 2022-07-03T10:14:17.515 INFO:teuthology.orchestra.run.smithi101.stdout:(78/124): libtool-2.4.6-25.el8.x86_64.rpm 6.4 MB/s | 709 kB 00:00 2022-07-03T10:14:17.549 INFO:teuthology.orchestra.run.smithi101.stdout:(79/124): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.5 MB/s | 225 kB 00:00 2022-07-03T10:14:17.582 INFO:teuthology.orchestra.run.smithi101.stdout:(80/124): python3-jsonpointer-1.10-11.el8.noarc 299 kB/s | 20 kB 00:00 2022-07-03T10:14:17.657 INFO:teuthology.orchestra.run.smithi101.stdout:(81/124): python3-numpy-1.14.3-10.el8.x86_64.rp 8.2 MB/s | 3.7 MB 00:00 2022-07-03T10:14:17.691 INFO:teuthology.orchestra.run.smithi101.stdout:(82/124): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2022-07-03T10:14:17.741 INFO:teuthology.orchestra.run.smithi101.stdout:(83/124): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2022-07-03T10:14:17.791 INFO:teuthology.orchestra.run.smithi101.stdout:(84/124): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2022-07-03T10:14:17.842 INFO:teuthology.orchestra.run.smithi101.stdout:(85/124): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2022-07-03T10:14:17.967 INFO:teuthology.orchestra.run.smithi101.stdout:(86/124): openblas-threads-0.3.15-3.el8.x86_64. 13 MB/s | 4.9 MB 00:00 2022-07-03T10:14:18.051 INFO:teuthology.orchestra.run.smithi101.stdout:(87/124): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2022-07-03T10:14:18.101 INFO:teuthology.orchestra.run.smithi101.stdout:(88/124): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2022-07-03T10:14:18.126 INFO:teuthology.orchestra.run.smithi101.stdout:(89/124): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2022-07-03T10:14:18.143 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2022-07-03T10:14:18.147 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2022-07-03T10:14:18.185 INFO:teuthology.orchestra.run.smithi101.stdout:(90/124): python3-jsonpatch-1.21-2.el8.noarch.r 318 kB/s | 27 kB 00:00 2022-07-03T10:14:18.235 INFO:teuthology.orchestra.run.smithi101.stdout:(91/124): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.4 MB/s | 259 kB 00:00 2022-07-03T10:14:18.285 INFO:teuthology.orchestra.run.smithi101.stdout:(92/124): flex-2.6.1-9.el8.x86_64.rpm 3.1 MB/s | 320 kB 00:00 2022-07-03T10:14:18.336 INFO:teuthology.orchestra.run.smithi101.stdout:(93/124): socat-1.7.4.1-1.el8.x86_64.rpm 3.3 MB/s | 323 kB 00:00 2022-07-03T10:14:18.386 INFO:teuthology.orchestra.run.smithi101.stdout:(94/124): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2022-07-03T10:14:18.445 INFO:teuthology.orchestra.run.smithi101.stdout:(95/124): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2022-07-03T10:14:18.504 INFO:teuthology.orchestra.run.smithi101.stdout:(96/124): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.4 MB/s | 560 kB 00:00 2022-07-03T10:14:18.595 INFO:teuthology.orchestra.run.smithi101.stdout:(97/124): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2022-07-03T10:14:18.621 INFO:teuthology.orchestra.run.smithi101.stdout:(98/124): python3-oauthlib-2.1.0-1.el8.noarch.r 1.3 MB/s | 155 kB 00:00 2022-07-03T10:14:18.671 INFO:teuthology.orchestra.run.smithi101.stdout:(99/124): python3-pycparser-2.14-14.el8.noarch. 1.4 MB/s | 109 kB 00:00 2022-07-03T10:14:18.721 INFO:teuthology.orchestra.run.smithi101.stdout:(100/124): python3-cryptography-3.2.1-5.el8.x86 5.5 MB/s | 559 kB 00:00 2022-07-03T10:14:18.738 INFO:teuthology.orchestra.run.smithi101.stdout:(101/124): libverto-devel-0.3.0-5.el8.x86_64.rp 269 kB/s | 18 kB 00:00 2022-07-03T10:14:18.788 INFO:teuthology.orchestra.run.smithi101.stdout:(102/124): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2022-07-03T10:14:18.822 INFO:teuthology.orchestra.run.smithi101.stdout:(103/124): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2022-07-03T10:14:18.883 INFO:teuthology.orchestra.run.smithi101.stdout:(104/124): libsepol-devel-2.9-3.el8.x86_64.rpm 920 kB/s | 87 kB 00:00 2022-07-03T10:14:18.914 INFO:teuthology.orchestra.run.smithi101.stdout:(105/124): librabbitmq-0.9.0-3.el8.x86_64.rpm 511 kB/s | 47 kB 00:00 2022-07-03T10:14:19.090 INFO:teuthology.orchestra.run.smithi101.stdout:(106/124): python3-scipy-1.0.0-21.module+el8.5. 11 MB/s | 14 MB 00:01 2022-07-03T10:14:19.115 INFO:teuthology.orchestra.run.smithi101.stdout:(107/124): libaio-devel-0.3.112-1.el8.x86_64.rp 95 kB/s | 19 kB 00:00 2022-07-03T10:14:19.157 INFO:teuthology.orchestra.run.smithi101.stdout:(108/124): pcre2-devel-10.32-2.el8.x86_64.rpm 2.2 MB/s | 605 kB 00:00 2022-07-03T10:14:19.199 INFO:teuthology.orchestra.run.smithi101.stdout:(109/124): libuuid-devel-2.32.1-35.el8.x86_64.r 1.2 MB/s | 98 kB 00:00 2022-07-03T10:14:19.232 INFO:teuthology.orchestra.run.smithi101.stdout:(110/124): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2022-07-03T10:14:19.258 INFO:teuthology.orchestra.run.smithi101.stdout:(111/124): zlib-devel-1.2.11-18.el8_5.x86_64.rp 628 kB/s | 58 kB 00:00 2022-07-03T10:14:19.308 INFO:teuthology.orchestra.run.smithi101.stdout:(112/124): fuse-2.9.7-15.el8.x86_64.rpm 801 kB/s | 83 kB 00:00 2022-07-03T10:14:19.333 INFO:teuthology.orchestra.run.smithi101.stdout:(113/124): python3-jwt-1.6.1-2.el8.noarch.rpm 424 kB/s | 43 kB 00:00 2022-07-03T10:14:19.358 INFO:teuthology.orchestra.run.smithi101.stdout:(114/124): libcom_err-devel-1.45.6-4.el8.x86_64 401 kB/s | 39 kB 00:00 2022-07-03T10:14:19.408 INFO:teuthology.orchestra.run.smithi101.stdout:(115/124): elfutils-libelf-devel-0.186-1.el8.x8 598 kB/s | 60 kB 00:00 2022-07-03T10:14:19.434 INFO:teuthology.orchestra.run.smithi101.stdout:(116/124): keyutils-libs-devel-1.5.10-9.el8.x86 642 kB/s | 48 kB 00:00 2022-07-03T10:14:19.468 INFO:teuthology.orchestra.run.smithi101.stdout:(117/124): libquadmath-8.5.0-10.el8.x86_64.rpm 1.5 MB/s | 171 kB 00:00 2022-07-03T10:14:19.518 INFO:teuthology.orchestra.run.smithi101.stdout:(118/124): xfsprogs-devel-5.0.0-10.el8.x86_64.r 909 kB/s | 72 kB 00:00 2022-07-03T10:14:19.551 INFO:teuthology.orchestra.run.smithi101.stdout:(119/124): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2022-07-03T10:14:19.593 INFO:teuthology.orchestra.run.smithi101.stdout:(120/124): libgfortran-8.5.0-10.el8.x86_64.rpm 3.4 MB/s | 643 kB 00:00 2022-07-03T10:14:19.627 INFO:teuthology.orchestra.run.smithi101.stdout:(121/124): python3-requests-oauthlib-1.0.0-1.el 393 kB/s | 43 kB 00:00 2022-07-03T10:14:19.719 INFO:teuthology.orchestra.run.smithi101.stdout:(122/124): openssl-devel-1.1.1k-6.el8_5.x86_64. 14 MB/s | 2.3 MB 00:00 2022-07-03T10:14:20.044 INFO:teuthology.orchestra.run.smithi101.stdout:(123/124): librados2-17.0.0-13322.g61829bae.el8 8.1 MB/s | 3.6 MB 00:00 2022-07-03T10:14:20.202 INFO:teuthology.orchestra.run.smithi101.stdout:(124/124): librbd1-17.0.0-13322.g61829bae.el8.x 6.8 MB/s | 3.9 MB 00:00 2022-07-03T10:14:20.205 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2022-07-03T10:14:20.205 INFO:teuthology.orchestra.run.smithi101.stdout:Total 11 MB/s | 204 MB 00:18 2022-07-03T10:14:20.487 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2022-07-03T10:14:20.949 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/126 2022-07-03T10:14:21.105 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/126 2022-07-03T10:14:21.338 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/126 2022-07-03T10:14:21.526 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-07-03T10:14:21.934 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-07-03T10:14:22.142 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:22.571 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:22.735 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2022-07-03T10:14:22.775 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:22.865 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2022-07-03T10:14:22.865 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2022-07-03T10:14:22.897 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:23.117 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:14:23.165 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2022-07-03T10:14:23.216 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:14:23.325 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/126 2022-07-03T10:14:23.406 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 8/126 2022-07-03T10:14:23.536 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/126 2022-07-03T10:14:23.608 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/126 2022-07-03T10:14:23.778 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/126 2022-07-03T10:14:24.158 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/126 2022-07-03T10:14:24.168 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/126 2022-07-03T10:14:24.300 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:24.361 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2022-07-03T10:14:24.366 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2022-07-03T10:14:24.405 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:24.436 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 4/126 2022-07-03T10:14:24.594 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/126 2022-07-03T10:14:24.748 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/126 2022-07-03T10:14:24.848 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 4/126 2022-07-03T10:14:24.891 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:24.997 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:25.119 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:25.158 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:25.234 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:25.235 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:25.358 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 15/126 2022-07-03T10:14:25.485 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:25.491 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 15/126 2022-07-03T10:14:25.579 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:25.616 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 16/126 2022-07-03T10:14:25.842 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 7/126 2022-07-03T10:14:26.019 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/126 2022-07-03T10:14:26.084 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/126 2022-07-03T10:14:26.217 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 16/126 2022-07-03T10:14:26.624 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-07-03T10:14:26.836 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/126 2022-07-03T10:14:27.019 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:27.110 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:27.228 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-07-03T10:14:27.355 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/126 2022-07-03T10:14:27.457 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-3.el8.x86_64 18/126 2022-07-03T10:14:27.558 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/126 2022-07-03T10:14:27.743 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:27.836 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:28.002 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:28.066 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:28.131 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 18/126 2022-07-03T10:14:28.339 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/126 2022-07-03T10:14:28.413 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 19/126 2022-07-03T10:14:28.538 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/126 2022-07-03T10:14:28.581 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/126 2022-07-03T10:14:28.740 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/126 2022-07-03T10:14:28.750 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/126 2022-07-03T10:14:28.885 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 18/126 2022-07-03T10:14:28.969 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/126 2022-07-03T10:14:29.018 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 18/126 2022-07-03T10:14:29.169 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2022-07-03T10:14:29.204 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/126 2022-07-03T10:14:29.359 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/126 2022-07-03T10:14:29.443 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 19/126 2022-07-03T10:14:29.481 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/126 2022-07-03T10:14:29.565 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/126 2022-07-03T10:14:29.655 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 25/126 2022-07-03T10:14:29.796 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/126 2022-07-03T10:14:29.808 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 26/126 2022-07-03T10:14:29.985 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-07-03T10:14:30.009 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 27/126 2022-07-03T10:14:30.061 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 19/126 2022-07-03T10:14:30.276 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 20/126 2022-07-03T10:14:30.509 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/126 2022-07-03T10:14:30.701 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:30.876 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 20/126 2022-07-03T10:14:30.905 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/126 2022-07-03T10:14:31.134 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.15-3.el8.x86_64 21/126 2022-07-03T10:14:31.228 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/126 2022-07-03T10:14:31.260 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 5/126 2022-07-03T10:14:31.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/126 2022-07-03T10:14:31.501 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:31.674 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:14:31.707 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-07-03T10:14:31.814 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 21/126 2022-07-03T10:14:31.893 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:14:32.040 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:14:32.116 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/126 2022-07-03T10:14:32.300 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 23/126 2022-07-03T10:14:32.305 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 8/126 2022-07-03T10:14:32.509 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/126 2022-07-03T10:14:32.595 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/126 2022-07-03T10:14:32.841 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 25/126 2022-07-03T10:14:33.043 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 26/126 2022-07-03T10:14:33.203 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/126 2022-07-03T10:14:33.305 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 27/126 2022-07-03T10:14:33.384 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:33.440 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-07-03T10:14:33.557 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:14:33.559 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 28/126 2022-07-03T10:14:33.758 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/126 2022-07-03T10:14:33.782 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 32/126 2022-07-03T10:14:33.960 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/126 2022-07-03T10:14:33.992 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.8-1.el8.noarch 33/126 2022-07-03T10:14:34.150 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:34.159 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 34/126 2022-07-03T10:14:34.312 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:14:34.364 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 35/126 2022-07-03T10:14:34.484 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:34.560 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 36/126 2022-07-03T10:14:34.621 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 14/126 2022-07-03T10:14:34.775 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libquadmath-8.5.0-10.el8.x86_64 15/126 2022-07-03T10:14:34.776 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 37/126 2022-07-03T10:14:34.973 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libquadmath-8.5.0-10.el8.x86_64 15/126 2022-07-03T10:14:34.992 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 38/126 2022-07-03T10:14:35.117 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libgfortran-8.5.0-10.el8.x86_64 16/126 2022-07-03T10:14:35.126 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 39/126 2022-07-03T10:14:35.273 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 40/126 2022-07-03T10:14:35.303 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 28/126 2022-07-03T10:14:35.496 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-15.el8.x86_64 41/126 2022-07-03T10:14:35.663 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 29/126 2022-07-03T10:14:35.702 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 42/126 2022-07-03T10:14:35.875 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 43/126 2022-07-03T10:14:35.893 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libgfortran-8.5.0-10.el8.x86_64 16/126 2022-07-03T10:14:36.067 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-07-03T10:14:36.199 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 44/126 2022-07-03T10:14:36.452 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 45/126 2022-07-03T10:14:36.624 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/126 2022-07-03T10:14:36.649 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 46/126 2022-07-03T10:14:36.810 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 47/126 2022-07-03T10:14:36.816 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 17/126 2022-07-03T10:14:36.983 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/126 2022-07-03T10:14:37.017 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-0.3.15-3.el8.x86_64 18/126 2022-07-03T10:14:37.134 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 48/126 2022-07-03T10:14:37.286 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/126 2022-07-03T10:14:37.366 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 49/126 2022-07-03T10:14:37.521 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.8-1.el8.noarch 33/126 2022-07-03T10:14:37.695 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 34/126 2022-07-03T10:14:37.728 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 50/126 2022-07-03T10:14:37.934 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 18/126 2022-07-03T10:14:37.967 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 35/126 2022-07-03T10:14:37.992 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 51/126 2022-07-03T10:14:38.148 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/126 2022-07-03T10:14:38.187 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 36/126 2022-07-03T10:14:38.217 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 19/126 2022-07-03T10:14:38.386 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/126 2022-07-03T10:14:38.399 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/126 2022-07-03T10:14:38.428 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 37/126 2022-07-03T10:14:38.633 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 38/126 2022-07-03T10:14:38.634 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 54/126 2022-07-03T10:14:38.666 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/126 2022-07-03T10:14:38.702 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 38/126 2022-07-03T10:14:38.908 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/126 2022-07-03T10:14:38.938 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-3.el8.x86_64 39/126 2022-07-03T10:14:38.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 55/126 2022-07-03T10:14:39.129 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 56/126 2022-07-03T10:14:39.165 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/126 2022-07-03T10:14:39.206 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 40/126 2022-07-03T10:14:39.322 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 57/126 2022-07-03T10:14:39.417 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.1-7.el8.noarch 41/126 2022-07-03T10:14:39.439 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/126 2022-07-03T10:14:39.499 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 58/126 2022-07-03T10:14:39.624 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/126 2022-07-03T10:14:39.638 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 59/126 2022-07-03T10:14:39.691 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 25/126 2022-07-03T10:14:39.718 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 59/126 2022-07-03T10:14:39.816 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/126 2022-07-03T10:14:39.862 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 26/126 2022-07-03T10:14:39.900 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-3.el8.x86_64 60/126 2022-07-03T10:14:40.036 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 44/126 2022-07-03T10:14:40.130 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 27/126 2022-07-03T10:14:40.139 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 61/126 2022-07-03T10:14:40.271 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 45/126 2022-07-03T10:14:40.303 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : automake-1.16.1-7.el8.noarch 62/126 2022-07-03T10:14:40.437 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 46/126 2022-07-03T10:14:40.503 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 63/126 2022-07-03T10:14:40.618 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 47/126 2022-07-03T10:14:40.637 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 64/126 2022-07-03T10:14:40.814 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 65/126 2022-07-03T10:14:40.842 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/126 2022-07-03T10:14:40.987 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 66/126 2022-07-03T10:14:41.092 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 49/126 2022-07-03T10:14:41.156 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 67/126 2022-07-03T10:14:41.278 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 50/126 2022-07-03T10:14:41.296 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/126 2022-07-03T10:14:41.424 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 68/126 2022-07-03T10:14:41.610 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 69/126 2022-07-03T10:14:41.642 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 51/126 2022-07-03T10:14:41.709 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/126 2022-07-03T10:14:41.775 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 70/126 2022-07-03T10:14:41.914 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 71/126 2022-07-03T10:14:41.934 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 52/126 2022-07-03T10:14:42.001 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/126 2022-07-03T10:14:42.094 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 72/126 2022-07-03T10:14:42.160 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 53/126 2022-07-03T10:14:42.237 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-07-03T10:14:42.389 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 54/126 2022-07-03T10:14:42.573 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 73/126 2022-07-03T10:14:42.621 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 55/126 2022-07-03T10:14:42.730 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 74/126 2022-07-03T10:14:42.873 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 75/126 2022-07-03T10:14:42.886 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 56/126 2022-07-03T10:14:43.010 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 76/126 2022-07-03T10:14:43.187 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 77/126 2022-07-03T10:14:43.284 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 57/126 2022-07-03T10:14:43.398 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 78/126 2022-07-03T10:14:43.575 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 79/126 2022-07-03T10:14:43.636 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 58/126 2022-07-03T10:14:43.802 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 80/126 2022-07-03T10:14:43.895 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 59/126 2022-07-03T10:14:43.994 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/126 2022-07-03T10:14:44.159 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 60/126 2022-07-03T10:14:44.167 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 82/126 2022-07-03T10:14:44.365 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 83/126 2022-07-03T10:14:44.473 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 61/126 2022-07-03T10:14:44.487 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 31/126 2022-07-03T10:14:44.826 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 62/126 2022-07-03T10:14:44.853 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 32/126 2022-07-03T10:14:45.057 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 63/126 2022-07-03T10:14:45.073 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rsa-4.8-1.el8.noarch 33/126 2022-07-03T10:14:45.247 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 34/126 2022-07-03T10:14:45.250 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-07-03T10:14:45.403 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 84/126 2022-07-03T10:14:45.514 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 35/126 2022-07-03T10:14:45.519 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 65/126 2022-07-03T10:14:45.673 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/126 2022-07-03T10:14:45.691 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 66/126 2022-07-03T10:14:45.714 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 36/126 2022-07-03T10:14:45.905 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 67/126 2022-07-03T10:14:45.938 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 37/126 2022-07-03T10:14:45.948 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 86/126 2022-07-03T10:14:46.149 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 38/126 2022-07-03T10:14:46.161 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 68/126 2022-07-03T10:14:46.245 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 87/126 2022-07-03T10:14:46.313 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 39/126 2022-07-03T10:14:46.382 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 69/126 2022-07-03T10:14:46.487 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 88/126 2022-07-03T10:14:46.489 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 40/126 2022-07-03T10:14:46.604 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 70/126 2022-07-03T10:14:46.694 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fuse-2.9.7-15.el8.x86_64 41/126 2022-07-03T10:14:46.695 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 89/126 2022-07-03T10:14:46.778 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 71/126 2022-07-03T10:14:46.904 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 90/126 2022-07-03T10:14:46.908 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 42/126 2022-07-03T10:14:47.027 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 72/126 2022-07-03T10:14:47.117 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 43/126 2022-07-03T10:14:47.296 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 73/126 2022-07-03T10:14:47.493 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 44/126 2022-07-03T10:14:47.495 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 74/126 2022-07-03T10:14:47.702 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 75/126 2022-07-03T10:14:47.772 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 45/126 2022-07-03T10:14:47.880 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 76/126 2022-07-03T10:14:47.977 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 46/126 2022-07-03T10:14:48.112 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 77/126 2022-07-03T10:14:48.181 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 47/126 2022-07-03T10:14:48.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 78/126 2022-07-03T10:14:48.471 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 48/126 2022-07-03T10:14:48.596 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 79/126 2022-07-03T10:14:48.714 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 49/126 2022-07-03T10:14:48.855 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-2.6-1.el8.noarch 80/126 2022-07-03T10:14:49.115 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/126 2022-07-03T10:14:49.174 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 50/126 2022-07-03T10:14:49.332 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 82/126 2022-07-03T10:14:49.475 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 51/126 2022-07-03T10:14:49.583 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 83/126 2022-07-03T10:14:49.690 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/126 2022-07-03T10:14:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/126 2022-07-03T10:14:50.228 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 54/126 2022-07-03T10:14:50.521 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:50.594 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 55/126 2022-07-03T10:14:50.728 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:50.738 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 84/126 2022-07-03T10:14:50.842 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 56/126 2022-07-03T10:14:51.053 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 57/126 2022-07-03T10:14:51.086 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/126 2022-07-03T10:14:51.289 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 58/126 2022-07-03T10:14:51.295 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:51.381 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 86/126 2022-07-03T10:14:51.465 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 59/126 2022-07-03T10:14:51.570 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:14:51.589 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 59/126 2022-07-03T10:14:51.642 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:14:51.692 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 87/126 2022-07-03T10:14:51.752 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:14:51.813 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : jq-1.6-3.el8.x86_64 60/126 2022-07-03T10:14:51.948 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 88/126 2022-07-03T10:14:52.142 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 61/126 2022-07-03T10:14:52.231 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 89/126 2022-07-03T10:14:52.332 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : automake-1.16.1-7.el8.noarch 62/126 2022-07-03T10:14:52.458 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 90/126 2022-07-03T10:14:52.525 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 63/126 2022-07-03T10:14:52.701 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 64/126 2022-07-03T10:14:52.961 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 65/126 2022-07-03T10:14:53.125 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 66/126 2022-07-03T10:14:53.325 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 67/126 2022-07-03T10:14:53.591 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 68/126 2022-07-03T10:14:53.815 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 69/126 2022-07-03T10:14:54.012 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 70/126 2022-07-03T10:14:54.187 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 71/126 2022-07-03T10:14:54.408 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 72/126 2022-07-03T10:14:54.657 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 73/126 2022-07-03T10:14:54.880 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 74/126 2022-07-03T10:14:55.060 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 75/126 2022-07-03T10:14:55.213 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 76/126 2022-07-03T10:14:55.463 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 77/126 2022-07-03T10:14:55.671 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 78/126 2022-07-03T10:14:55.888 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 79/126 2022-07-03T10:14:56.126 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:56.172 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-portend-2.6-1.el8.noarch 80/126 2022-07-03T10:14:56.390 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:56.407 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/126 2022-07-03T10:14:56.625 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 82/126 2022-07-03T10:14:56.918 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 83/126 2022-07-03T10:14:56.957 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:14:57.257 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:14:57.350 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:14:57.480 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:14:58.360 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 84/126 2022-07-03T10:14:58.663 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 85/126 2022-07-03T10:14:58.905 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 86/126 2022-07-03T10:14:59.283 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 87/126 2022-07-03T10:14:59.560 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 88/126 2022-07-03T10:14:59.821 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 89/126 2022-07-03T10:14:59.998 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 90/126 2022-07-03T10:15:05.423 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:15:05.647 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:15:06.390 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 91/126 2022-07-03T10:15:06.673 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:15:06.837 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 92/126 2022-07-03T10:15:06.963 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:15:10.355 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:15:10.355 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2022-07-03T10:15:10.356 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2022-07-03T10:15:10.356 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2022-07-03T10:15:10.356 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2022-07-03T10:15:10.356 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2022-07-03T10:15:10.357 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2022-07-03T10:15:10.357 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2022-07-03T10:15:10.357 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2022-07-03T10:15:10.358 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2022-07-03T10:15:10.358 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:15:10.698 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:12.211 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:12.398 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:12.621 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:12.764 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:13.156 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:13.356 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:13.374 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:15.021 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:15.192 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:15.311 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 99/126 2022-07-03T10:15:15.468 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 100/126 2022-07-03T10:15:15.630 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:15.833 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:16.126 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:15:16.126 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2022-07-03T10:15:16.127 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2022-07-03T10:15:16.127 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2022-07-03T10:15:16.127 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /run 2022-07-03T10:15:16.127 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2022-07-03T10:15:16.128 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2022-07-03T10:15:16.128 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2022-07-03T10:15:16.128 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2022-07-03T10:15:16.128 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /dev 2022-07-03T10:15:16.128 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:15:16.156 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:16.494 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:17.115 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:17.288 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:17.376 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:17.743 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 104/126 2022-07-03T10:15:17.913 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:17.948 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:18.007 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:18.121 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:18.132 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:18.207 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:18.432 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:18.644 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:19.030 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:19.211 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:19.228 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:19.679 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 107/126 2022-07-03T10:15:19.877 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:19.953 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:20.104 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:20.148 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:20.312 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:20.696 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:20.793 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:20.829 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:20.935 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:21.100 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 99/126 2022-07-03T10:15:21.279 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 100/126 2022-07-03T10:15:21.485 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:21.688 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:22.023 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:22.983 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:23.194 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:23.285 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:23.677 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 104/126 2022-07-03T10:15:23.876 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:23.911 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:24.126 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:24.138 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:25.708 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 107/126 2022-07-03T10:15:25.831 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:25.932 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:26.008 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:26.154 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 112/126 2022-07-03T10:15:26.193 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:26.237 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:26.294 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libtool-2.4.6-25.el8.x86_64 113/126 2022-07-03T10:15:26.410 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:26.795 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:26.947 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 113/126 2022-07-03T10:15:26.959 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:27.213 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openssl-devel-1:1.1.1k-6.el8_5.x86_64 114/126 2022-07-03T10:15:27.436 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 115/126 2022-07-03T10:15:27.599 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 116/126 2022-07-03T10:15:27.731 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 117/126 2022-07-03T10:15:27.881 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 118/126 2022-07-03T10:15:28.035 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 119/126 2022-07-03T10:15:28.239 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 120/126 2022-07-03T10:15:28.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 121/126 2022-07-03T10:15:28.572 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 122/126 2022-07-03T10:15:28.707 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 93/126 2022-07-03T10:15:28.708 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /sys 2022-07-03T10:15:28.708 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /proc 2022-07-03T10:15:28.709 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2022-07-03T10:15:28.709 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /run 2022-07-03T10:15:28.709 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /mnt 2022-07-03T10:15:28.710 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /var/tmp 2022-07-03T10:15:28.710 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /home 2022-07-03T10:15:28.710 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /tmp 2022-07-03T10:15:28.711 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2022-07-03T10:15:28.711 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:15:28.768 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flex-2.6.1-9.el8.x86_64 123/126 2022-07-03T10:15:28.847 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 123/126 2022-07-03T10:15:29.035 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : bison-3.0.4-10.el8.x86_64 124/126 2022-07-03T10:15:29.049 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 124/126 2022-07-03T10:15:29.049 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:29.111 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:29.238 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:29.239 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:30.049 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:30.050 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 1/126 2022-07-03T10:15:30.050 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2/126 2022-07-03T10:15:30.050 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 3/126 2022-07-03T10:15:30.051 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 4/126 2022-07-03T10:15:30.051 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13322.g6182 5/126 2022-07-03T10:15:30.051 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:15:30.051 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:15:30.052 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 8/126 2022-07-03T10:15:30.052 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 9/126 2022-07-03T10:15:30.052 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:15:30.052 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 11/126 2022-07-03T10:15:30.053 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 12/126 2022-07-03T10:15:30.053 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:15:30.053 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 14/126 2022-07-03T10:15:30.053 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 15/126 2022-07-03T10:15:30.054 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 16/126 2022-07-03T10:15:30.054 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 17/126 2022-07-03T10:15:30.054 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 18/126 2022-07-03T10:15:30.054 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 19/126 2022-07-03T10:15:30.054 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 20/126 2022-07-03T10:15:30.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 21/126 2022-07-03T10:15:30.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 22/126 2022-07-03T10:15:30.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 23/126 2022-07-03T10:15:30.055 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 24/126 2022-07-03T10:15:30.056 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 25/126 2022-07-03T10:15:30.056 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 26/126 2022-07-03T10:15:30.056 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 27/126 2022-07-03T10:15:30.056 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 28/126 2022-07-03T10:15:30.056 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 29/126 2022-07-03T10:15:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 30/126 2022-07-03T10:15:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 31/126 2022-07-03T10:15:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 32/126 2022-07-03T10:15:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 33/126 2022-07-03T10:15:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 34/126 2022-07-03T10:15:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 35/126 2022-07-03T10:15:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 36/126 2022-07-03T10:15:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 37/126 2022-07-03T10:15:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 38/126 2022-07-03T10:15:30.059 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/126 2022-07-03T10:15:30.059 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/126 2022-07-03T10:15:30.059 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/126 2022-07-03T10:15:30.059 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/126 2022-07-03T10:15:30.059 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/126 2022-07-03T10:15:30.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/126 2022-07-03T10:15:30.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/126 2022-07-03T10:15:30.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/126 2022-07-03T10:15:30.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/126 2022-07-03T10:15:30.060 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/126 2022-07-03T10:15:30.061 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/126 2022-07-03T10:15:30.061 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/126 2022-07-03T10:15:30.061 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/126 2022-07-03T10:15:30.061 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/126 2022-07-03T10:15:30.062 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/126 2022-07-03T10:15:30.062 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/126 2022-07-03T10:15:30.062 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/126 2022-07-03T10:15:30.062 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/126 2022-07-03T10:15:30.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/126 2022-07-03T10:15:30.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/126 2022-07-03T10:15:30.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/126 2022-07-03T10:15:30.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/126 2022-07-03T10:15:30.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/126 2022-07-03T10:15:30.064 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/126 2022-07-03T10:15:30.064 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/126 2022-07-03T10:15:30.064 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-07-03T10:15:30.064 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/126 2022-07-03T10:15:30.065 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/126 2022-07-03T10:15:30.065 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/126 2022-07-03T10:15:30.065 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/126 2022-07-03T10:15:30.065 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/126 2022-07-03T10:15:30.066 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/126 2022-07-03T10:15:30.066 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/126 2022-07-03T10:15:30.066 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 72/126 2022-07-03T10:15:30.067 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 73/126 2022-07-03T10:15:30.067 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : bison-3.0.4-10.el8.x86_64 74/126 2022-07-03T10:15:30.067 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/126 2022-07-03T10:15:30.067 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 76/126 2022-07-03T10:15:30.068 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/126 2022-07-03T10:15:30.068 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 78/126 2022-07-03T10:15:30.068 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/126 2022-07-03T10:15:30.068 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 80/126 2022-07-03T10:15:30.068 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/126 2022-07-03T10:15:30.069 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 82/126 2022-07-03T10:15:30.069 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 83/126 2022-07-03T10:15:30.069 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/126 2022-07-03T10:15:30.069 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 85/126 2022-07-03T10:15:30.069 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-3.el8.x86_64 86/126 2022-07-03T10:15:30.070 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 87/126 2022-07-03T10:15:30.070 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 88/126 2022-07-03T10:15:30.070 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : automake-1.16.1-7.el8.noarch 89/126 2022-07-03T10:15:30.070 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 90/126 2022-07-03T10:15:30.071 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 91/126 2022-07-03T10:15:30.071 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 92/126 2022-07-03T10:15:30.071 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flex-2.6.1-9.el8.x86_64 93/126 2022-07-03T10:15:30.071 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 94/126 2022-07-03T10:15:30.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 95/126 2022-07-03T10:15:30.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 96/126 2022-07-03T10:15:30.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 97/126 2022-07-03T10:15:30.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 98/126 2022-07-03T10:15:30.072 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/126 2022-07-03T10:15:30.073 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 100/126 2022-07-03T10:15:30.073 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 101/126 2022-07-03T10:15:30.073 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 102/126 2022-07-03T10:15:30.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 103/126 2022-07-03T10:15:30.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 104/126 2022-07-03T10:15:30.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/126 2022-07-03T10:15:30.074 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 106/126 2022-07-03T10:15:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 107/126 2022-07-03T10:15:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/126 2022-07-03T10:15:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 109/126 2022-07-03T10:15:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 110/126 2022-07-03T10:15:30.075 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 111/126 2022-07-03T10:15:30.076 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 112/126 2022-07-03T10:15:30.076 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 113/126 2022-07-03T10:15:30.076 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 114/126 2022-07-03T10:15:30.076 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 115/126 2022-07-03T10:15:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 116/126 2022-07-03T10:15:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 117/126 2022-07-03T10:15:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 118/126 2022-07-03T10:15:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 119/126 2022-07-03T10:15:30.077 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 120/126 2022-07-03T10:15:30.078 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 121/126 2022-07-03T10:15:30.078 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openssl-devel-1:1.1.1k-6.el8_5.x86_64 122/126 2022-07-03T10:15:30.078 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 123/126 2022-07-03T10:15:30.078 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/126 2022-07-03T10:15:30.078 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 125/126 2022-07-03T10:15:30.905 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 94/126 2022-07-03T10:15:31.086 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:31.402 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 95/126 2022-07-03T10:15:31.560 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:31.952 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:32.063 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 96/126 2022-07-03T10:15:32.244 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:32.314 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 97/126 2022-07-03T10:15:32.835 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 112/126 2022-07-03T10:15:33.124 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:1.1.1k-6.el8_5.x86_64 113/126 2022-07-03T10:15:33.384 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 114/126 2022-07-03T10:15:33.619 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 115/126 2022-07-03T10:15:33.774 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/126 2022-07-03T10:15:33.786 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/126 2022-07-03T10:15:33.996 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 117/126 2022-07-03T10:15:34.162 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 118/126 2022-07-03T10:15:34.235 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:34.235 INFO:teuthology.orchestra.run.smithi188.stdout:Installed products updated. 2022-07-03T10:15:34.383 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 119/126 2022-07-03T10:15:34.430 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:15:34.430 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2022-07-03T10:15:34.431 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.431 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.431 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2022-07-03T10:15:34.431 INFO:teuthology.orchestra.run.smithi188.stdout: automake-1.16.1-7.el8.noarch 2022-07-03T10:15:34.432 INFO:teuthology.orchestra.run.smithi188.stdout: bison-3.0.4-10.el8.x86_64 2022-07-03T10:15:34.432 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.432 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.433 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.433 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.434 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.435 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.435 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.435 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.435 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.436 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.436 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.436 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.436 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.437 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.437 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.437 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.437 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.437 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.438 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.438 INFO:teuthology.orchestra.run.smithi188.stdout: cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:34.438 INFO:teuthology.orchestra.run.smithi188.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-07-03T10:15:34.438 INFO:teuthology.orchestra.run.smithi188.stdout: flex-2.6.1-9.el8.x86_64 2022-07-03T10:15:34.439 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.7-15.el8.x86_64 2022-07-03T10:15:34.439 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-07-03T10:15:34.439 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-3.el8.x86_64 2022-07-03T10:15:34.439 INFO:teuthology.orchestra.run.smithi188.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-07-03T10:15:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-07-03T10:15:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-07-03T10:15:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.440 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.441 INFO:teuthology.orchestra.run.smithi188.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-07-03T10:15:34.441 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-07-03T10:15:34.441 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2022-07-03T10:15:34.441 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-07-03T10:15:34.442 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-07-03T10:15:34.442 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.442 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.442 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-07-03T10:15:34.443 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.443 INFO:teuthology.orchestra.run.smithi188.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-07-03T10:15:34.443 INFO:teuthology.orchestra.run.smithi188.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-07-03T10:15:34.443 INFO:teuthology.orchestra.run.smithi188.stdout: libtool-2.4.6-25.el8.x86_64 2022-07-03T10:15:34.443 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2022-07-03T10:15:34.444 INFO:teuthology.orchestra.run.smithi188.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-07-03T10:15:34.444 INFO:teuthology.orchestra.run.smithi188.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-07-03T10:15:34.445 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-07-03T10:15:34.445 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-07-03T10:15:34.445 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-3.el8.x86_64 2022-07-03T10:15:34.445 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-07-03T10:15:34.446 INFO:teuthology.orchestra.run.smithi188.stdout: openssl-devel-1:1.1.1k-6.el8_5.x86_64 2022-07-03T10:15:34.446 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-07-03T10:15:34.446 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-07-03T10:15:34.446 INFO:teuthology.orchestra.run.smithi188.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-07-03T10:15:34.447 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-07-03T10:15:34.447 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch 2022-07-03T10:15:34.447 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-07-03T10:15:34.447 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-07-03T10:15:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-07-03T10:15:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.448 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-07-03T10:15:34.449 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-07-03T10:15:34.449 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-07-03T10:15:34.449 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-07-03T10:15:34.449 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-07-03T10:15:34.449 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-07-03T10:15:34.450 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2022-07-03T10:15:34.450 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-07-03T10:15:34.450 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-07-03T10:15:34.450 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-07-03T10:15:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-07-03T10:15:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-07-03T10:15:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-07-03T10:15:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-07-03T10:15:34.451 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-13.el8.noarch 2022-07-03T10:15:34.452 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-07-03T10:15:34.452 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-07-03T10:15:34.452 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-07-03T10:15:34.452 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:34.453 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:34.453 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-07-03T10:15:34.453 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-07-03T10:15:34.453 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2022-07-03T10:15:34.453 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-07-03T10:15:34.454 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-07-03T10:15:34.454 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-07-03T10:15:34.454 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-07-03T10:15:34.454 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2022-07-03T10:15:34.455 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-9.el8.noarch 2022-07-03T10:15:34.455 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.455 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.455 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-07-03T10:15:34.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-07-03T10:15:34.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2022-07-03T10:15:34.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.8-1.el8.noarch 2022-07-03T10:15:34.456 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-07-03T10:15:34.457 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-07-03T10:15:34.457 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-07-03T10:15:34.457 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-07-03T10:15:34.457 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-07-03T10:15:34.458 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-07-03T10:15:34.458 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-07-03T10:15:34.458 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-07-03T10:15:34.458 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-07-03T10:15:34.459 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-07-03T10:15:34.459 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-07-03T10:15:34.459 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.459 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.459 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:34.460 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 2022-07-03T10:15:34.460 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2022-07-03T10:15:34.460 INFO:teuthology.orchestra.run.smithi188.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-07-03T10:15:34.460 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-07-03T10:15:34.461 INFO:teuthology.orchestra.run.smithi188.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-07-03T10:15:34.461 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:15:34.462 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2022-07-03T10:15:34.583 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 120/126 2022-07-03T10:15:34.627 DEBUG:teuthology.parallel:result is None 2022-07-03T10:15:34.821 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 121/126 2022-07-03T10:15:35.058 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.1-9.el8.x86_64 122/126 2022-07-03T10:15:35.141 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 122/126 2022-07-03T10:15:35.349 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.0.4-10.el8.x86_64 123/126 2022-07-03T10:15:35.362 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 123/126 2022-07-03T10:15:35.379 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:35.510 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 124/126 2022-07-03T10:15:35.511 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:35.539 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 98/126 2022-07-03T10:15:35.685 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 99/126 2022-07-03T10:15:35.761 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:35.762 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:35.870 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 100/126 2022-07-03T10:15:36.042 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:36.339 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 101/126 2022-07-03T10:15:36.573 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:36.581 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:36.581 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 1/126 2022-07-03T10:15:36.581 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2/126 2022-07-03T10:15:36.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 3/126 2022-07-03T10:15:36.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 4/126 2022-07-03T10:15:36.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13322.g6182 5/126 2022-07-03T10:15:36.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:15:36.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:15:36.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 8/126 2022-07-03T10:15:36.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 9/126 2022-07-03T10:15:36.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:15:36.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 11/126 2022-07-03T10:15:36.584 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 12/126 2022-07-03T10:15:36.584 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:15:36.584 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 14/126 2022-07-03T10:15:36.584 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 15/126 2022-07-03T10:15:36.585 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 16/126 2022-07-03T10:15:36.585 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 17/126 2022-07-03T10:15:36.585 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 18/126 2022-07-03T10:15:36.585 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 19/126 2022-07-03T10:15:36.585 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 20/126 2022-07-03T10:15:36.586 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 21/126 2022-07-03T10:15:36.586 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 22/126 2022-07-03T10:15:36.586 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 23/126 2022-07-03T10:15:36.586 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 24/126 2022-07-03T10:15:36.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 25/126 2022-07-03T10:15:36.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 26/126 2022-07-03T10:15:36.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 27/126 2022-07-03T10:15:36.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 28/126 2022-07-03T10:15:36.587 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 29/126 2022-07-03T10:15:36.588 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 30/126 2022-07-03T10:15:36.588 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 31/126 2022-07-03T10:15:36.588 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 32/126 2022-07-03T10:15:36.588 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 33/126 2022-07-03T10:15:36.589 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 34/126 2022-07-03T10:15:36.589 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 35/126 2022-07-03T10:15:36.589 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 36/126 2022-07-03T10:15:36.589 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 37/126 2022-07-03T10:15:36.589 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 38/126 2022-07-03T10:15:36.590 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/126 2022-07-03T10:15:36.590 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/126 2022-07-03T10:15:36.590 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/126 2022-07-03T10:15:36.590 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/126 2022-07-03T10:15:36.591 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/126 2022-07-03T10:15:36.591 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/126 2022-07-03T10:15:36.591 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/126 2022-07-03T10:15:36.591 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/126 2022-07-03T10:15:36.591 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/126 2022-07-03T10:15:36.592 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/126 2022-07-03T10:15:36.592 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/126 2022-07-03T10:15:36.592 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/126 2022-07-03T10:15:36.592 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/126 2022-07-03T10:15:36.592 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/126 2022-07-03T10:15:36.593 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/126 2022-07-03T10:15:36.593 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/126 2022-07-03T10:15:36.593 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/126 2022-07-03T10:15:36.593 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/126 2022-07-03T10:15:36.594 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/126 2022-07-03T10:15:36.594 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/126 2022-07-03T10:15:36.594 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/126 2022-07-03T10:15:36.594 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/126 2022-07-03T10:15:36.594 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/126 2022-07-03T10:15:36.595 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/126 2022-07-03T10:15:36.595 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/126 2022-07-03T10:15:36.595 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-07-03T10:15:36.595 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/126 2022-07-03T10:15:36.596 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/126 2022-07-03T10:15:36.596 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/126 2022-07-03T10:15:36.596 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/126 2022-07-03T10:15:36.596 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/126 2022-07-03T10:15:36.597 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/126 2022-07-03T10:15:36.597 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/126 2022-07-03T10:15:36.597 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 72/126 2022-07-03T10:15:36.597 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 73/126 2022-07-03T10:15:36.598 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 74/126 2022-07-03T10:15:36.598 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 75/126 2022-07-03T10:15:36.598 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 76/126 2022-07-03T10:15:36.598 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 77/126 2022-07-03T10:15:36.599 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 78/126 2022-07-03T10:15:36.599 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 79/126 2022-07-03T10:15:36.599 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 80/126 2022-07-03T10:15:36.599 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/126 2022-07-03T10:15:36.599 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/126 2022-07-03T10:15:36.600 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 83/126 2022-07-03T10:15:36.600 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 84/126 2022-07-03T10:15:36.600 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/126 2022-07-03T10:15:36.600 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 86/126 2022-07-03T10:15:36.601 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 87/126 2022-07-03T10:15:36.601 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 88/126 2022-07-03T10:15:36.601 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 89/126 2022-07-03T10:15:36.601 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 90/126 2022-07-03T10:15:36.601 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/126 2022-07-03T10:15:36.602 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 92/126 2022-07-03T10:15:36.602 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 93/126 2022-07-03T10:15:36.602 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 94/126 2022-07-03T10:15:36.602 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 95/126 2022-07-03T10:15:36.603 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:1.1.1k-6.el8_5.x86_64 96/126 2022-07-03T10:15:36.603 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 97/126 2022-07-03T10:15:36.603 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 98/126 2022-07-03T10:15:36.603 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.0.4-10.el8.x86_64 99/126 2022-07-03T10:15:36.603 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 100/126 2022-07-03T10:15:36.604 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/126 2022-07-03T10:15:36.604 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 102/126 2022-07-03T10:15:36.604 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 103/126 2022-07-03T10:15:36.604 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 104/126 2022-07-03T10:15:36.605 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 105/126 2022-07-03T10:15:36.605 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 106/126 2022-07-03T10:15:36.605 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 107/126 2022-07-03T10:15:36.605 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 108/126 2022-07-03T10:15:36.605 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 109/126 2022-07-03T10:15:36.606 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 110/126 2022-07-03T10:15:36.606 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-3.el8.x86_64 111/126 2022-07-03T10:15:36.606 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 112/126 2022-07-03T10:15:36.606 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 113/126 2022-07-03T10:15:36.607 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.1-7.el8.noarch 114/126 2022-07-03T10:15:36.607 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 115/126 2022-07-03T10:15:36.607 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 116/126 2022-07-03T10:15:36.607 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 117/126 2022-07-03T10:15:36.608 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.1-9.el8.x86_64 118/126 2022-07-03T10:15:36.608 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 119/126 2022-07-03T10:15:36.608 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 120/126 2022-07-03T10:15:36.608 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 121/126 2022-07-03T10:15:36.608 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 122/126 2022-07-03T10:15:36.609 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 123/126 2022-07-03T10:15:36.609 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/126 2022-07-03T10:15:36.609 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 125/126 2022-07-03T10:15:37.785 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 102/126 2022-07-03T10:15:37.968 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:38.128 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 103/126 2022-07-03T10:15:38.604 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 104/126 2022-07-03T10:15:38.816 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:38.912 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 105/126 2022-07-03T10:15:39.075 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:39.129 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 106/126 2022-07-03T10:15:40.813 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:40.814 INFO:teuthology.orchestra.run.smithi161.stdout:Installed products updated. 2022-07-03T10:15:40.949 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 107/126 2022-07-03T10:15:41.009 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:15:41.009 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2022-07-03T10:15:41.009 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.010 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.010 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2022-07-03T10:15:41.010 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.1-7.el8.noarch 2022-07-03T10:15:41.011 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.0.4-10.el8.x86_64 2022-07-03T10:15:41.011 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.011 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.011 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.012 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.012 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.012 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.012 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.012 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.013 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.013 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.013 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.013 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.014 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.014 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.014 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.015 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:41.016 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-07-03T10:15:41.017 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.1-9.el8.x86_64 2022-07-03T10:15:41.017 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.7-15.el8.x86_64 2022-07-03T10:15:41.017 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-07-03T10:15:41.017 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-3.el8.x86_64 2022-07-03T10:15:41.018 INFO:teuthology.orchestra.run.smithi161.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-07-03T10:15:41.018 INFO:teuthology.orchestra.run.smithi161.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-07-03T10:15:41.018 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-07-03T10:15:41.018 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.019 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.019 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.019 INFO:teuthology.orchestra.run.smithi161.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-07-03T10:15:41.019 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-07-03T10:15:41.019 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.2-3.el8.x86_64 2022-07-03T10:15:41.020 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-07-03T10:15:41.020 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-07-03T10:15:41.020 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.021 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.021 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-07-03T10:15:41.021 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.021 INFO:teuthology.orchestra.run.smithi161.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-07-03T10:15:41.022 INFO:teuthology.orchestra.run.smithi161.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-07-03T10:15:41.022 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-25.el8.x86_64 2022-07-03T10:15:41.022 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.3.1-3.el8.x86_64 2022-07-03T10:15:41.023 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-07-03T10:15:41.023 INFO:teuthology.orchestra.run.smithi161.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-07-03T10:15:41.023 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-07-03T10:15:41.023 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-07-03T10:15:41.024 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.15-3.el8.x86_64 2022-07-03T10:15:41.024 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-07-03T10:15:41.024 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:1.1.1k-6.el8_5.x86_64 2022-07-03T10:15:41.024 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-07-03T10:15:41.024 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-07-03T10:15:41.025 INFO:teuthology.orchestra.run.smithi161.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-07-03T10:15:41.025 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-07-03T10:15:41.025 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.5.1-7.el8.noarch 2022-07-03T10:15:41.025 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-07-03T10:15:41.026 INFO:teuthology.orchestra.run.smithi161.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-07-03T10:15:41.026 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-07-03T10:15:41.026 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.026 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.027 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.027 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-07-03T10:15:41.027 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-07-03T10:15:41.027 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-07-03T10:15:41.028 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-07-03T10:15:41.028 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-07-03T10:15:41.028 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-07-03T10:15:41.028 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-6.2-6.el8.noarch 2022-07-03T10:15:41.028 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-07-03T10:15:41.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-07-03T10:15:41.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-07-03T10:15:41.029 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-07-03T10:15:41.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-07-03T10:15:41.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-07-03T10:15:41.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-07-03T10:15:41.030 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.0.6-13.el8.noarch 2022-07-03T10:15:41.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-07-03T10:15:41.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-07-03T10:15:41.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-07-03T10:15:41.031 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-07-03T10:15:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-07-03T10:15:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-2.6-1.el8.noarch 2022-07-03T10:15:41.032 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-07-03T10:15:41.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-07-03T10:15:41.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-07-03T10:15:41.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-07-03T10:15:41.033 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.14-14.el8.noarch 2022-07-03T10:15:41.034 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2017.2-9.el8.noarch 2022-07-03T10:15:41.034 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.034 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.034 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-07-03T10:15:41.035 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-07-03T10:15:41.035 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.035 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.4.1-12.el8.noarch 2022-07-03T10:15:41.035 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.8-1.el8.noarch 2022-07-03T10:15:41.035 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-07-03T10:15:41.036 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-07-03T10:15:41.036 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-07-03T10:15:41.036 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-07-03T10:15:41.037 INFO:teuthology.orchestra.run.smithi161.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-07-03T10:15:41.037 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-07-03T10:15:41.037 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-07-03T10:15:41.038 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-07-03T10:15:41.038 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-07-03T10:15:41.038 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-07-03T10:15:41.038 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-07-03T10:15:41.038 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.039 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.039 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:41.039 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-1.el8.x86_64 2022-07-03T10:15:41.040 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.13.0-2.el8.x86_64 2022-07-03T10:15:41.040 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-07-03T10:15:41.040 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-07-03T10:15:41.041 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-07-03T10:15:41.041 INFO:teuthology.orchestra.run.smithi161.stdout: 2022-07-03T10:15:41.041 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2022-07-03T10:15:41.168 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:41.210 DEBUG:teuthology.parallel:result is None 2022-07-03T10:15:41.315 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 108/126 2022-07-03T10:15:41.492 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:41.597 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-13322.g6182 109/126 2022-07-03T10:15:41.759 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:42.269 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 110/126 2022-07-03T10:15:42.442 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:48.327 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 111/126 2022-07-03T10:15:48.712 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 112/126 2022-07-03T10:15:48.891 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libtool-2.4.6-25.el8.x86_64 113/126 2022-07-03T10:15:50.092 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 113/126 2022-07-03T10:15:50.381 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openssl-devel-1:1.1.1k-6.el8_5.x86_64 114/126 2022-07-03T10:15:50.670 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 115/126 2022-07-03T10:15:50.847 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 116/126 2022-07-03T10:15:51.021 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 117/126 2022-07-03T10:15:51.188 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 118/126 2022-07-03T10:15:51.360 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 119/126 2022-07-03T10:15:51.561 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 120/126 2022-07-03T10:15:51.734 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 121/126 2022-07-03T10:15:51.910 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 122/126 2022-07-03T10:15:52.132 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flex-2.6.1-9.el8.x86_64 123/126 2022-07-03T10:15:52.273 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 123/126 2022-07-03T10:15:52.474 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : bison-3.0.4-10.el8.x86_64 124/126 2022-07-03T10:15:52.528 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 124/126 2022-07-03T10:15:52.528 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:52.790 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 125/126 2022-07-03T10:15:52.791 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:53.503 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:53.504 INFO:teuthology.orchestra.run.smithi101.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-07-03T10:15:53.504 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:15:54.226 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-2:17.0.0-13322.g61829bae.el8.x86_64 1/126 2022-07-03T10:15:54.227 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2/126 2022-07-03T10:15:54.227 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 3/126 2022-07-03T10:15:54.227 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 4/126 2022-07-03T10:15:54.228 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-13322.g6182 5/126 2022-07-03T10:15:54.228 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 6/126 2022-07-03T10:15:54.228 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 7/126 2022-07-03T10:15:54.228 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 8/126 2022-07-03T10:15:54.229 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 9/126 2022-07-03T10:15:54.229 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 10/126 2022-07-03T10:15:54.229 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 11/126 2022-07-03T10:15:54.229 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 12/126 2022-07-03T10:15:54.230 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_6 13/126 2022-07-03T10:15:54.230 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86 14/126 2022-07-03T10:15:54.230 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 15/126 2022-07-03T10:15:54.230 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_6 16/126 2022-07-03T10:15:54.230 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados-devel-2:17.0.0-13322.g61829bae.el8.x86_ 17/126 2022-07-03T10:15:54.231 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libradosstriper1-2:17.0.0-13322.g61829bae.el8.x8 18/126 2022-07-03T10:15:54.231 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 19/126 2022-07-03T10:15:54.231 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-argparse-2:17.0.0-13322.g61829bae.e 20/126 2022-07-03T10:15:54.231 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-common-2:17.0.0-13322.g61829bae.el8 21/126 2022-07-03T10:15:54.231 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_ 22/126 2022-07-03T10:15:54.232 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rados-2:17.0.0-13322.g61829bae.el8.x86_6 23/126 2022-07-03T10:15:54.232 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 24/126 2022-07-03T10:15:54.232 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 25/126 2022-07-03T10:15:54.232 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 26/126 2022-07-03T10:15:54.233 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 27/126 2022-07-03T10:15:54.233 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 28/126 2022-07-03T10:15:54.244 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-13322.g61829bae 29/126 2022-07-03T10:15:54.245 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.no 30/126 2022-07-03T10:15:54.246 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8. 31/126 2022-07-03T10:15:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-13322.g61 32/126 2022-07-03T10:15:54.247 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.e 33/126 2022-07-03T10:15:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarc 34/126 2022-07-03T10:15:54.248 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-13322.g61829bae. 35/126 2022-07-03T10:15:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 36/126 2022-07-03T10:15:54.249 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephadm-2:17.0.0-13322.g61829bae.el8.noarch 37/126 2022-07-03T10:15:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 38/126 2022-07-03T10:15:54.250 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/126 2022-07-03T10:15:54.251 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/126 2022-07-03T10:15:54.251 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/126 2022-07-03T10:15:54.251 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/126 2022-07-03T10:15:54.252 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/126 2022-07-03T10:15:54.252 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/126 2022-07-03T10:15:54.253 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/126 2022-07-03T10:15:54.254 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/126 2022-07-03T10:15:54.254 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/126 2022-07-03T10:15:54.255 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/126 2022-07-03T10:15:54.255 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/126 2022-07-03T10:15:54.255 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/126 2022-07-03T10:15:54.255 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/126 2022-07-03T10:15:54.256 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/126 2022-07-03T10:15:54.256 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/126 2022-07-03T10:15:54.257 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/126 2022-07-03T10:15:54.257 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/126 2022-07-03T10:15:54.257 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/126 2022-07-03T10:15:54.257 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/126 2022-07-03T10:15:54.258 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/126 2022-07-03T10:15:54.258 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/126 2022-07-03T10:15:54.258 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/126 2022-07-03T10:15:54.258 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/126 2022-07-03T10:15:54.258 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/126 2022-07-03T10:15:54.259 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/126 2022-07-03T10:15:54.259 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/126 2022-07-03T10:15:54.259 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/126 2022-07-03T10:15:54.259 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/126 2022-07-03T10:15:54.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/126 2022-07-03T10:15:54.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/126 2022-07-03T10:15:54.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/126 2022-07-03T10:15:54.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/126 2022-07-03T10:15:54.260 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/126 2022-07-03T10:15:54.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 72/126 2022-07-03T10:15:54.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 73/126 2022-07-03T10:15:54.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : bison-3.0.4-10.el8.x86_64 74/126 2022-07-03T10:15:54.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/126 2022-07-03T10:15:54.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 76/126 2022-07-03T10:15:54.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/126 2022-07-03T10:15:54.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 78/126 2022-07-03T10:15:54.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/126 2022-07-03T10:15:54.263 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 80/126 2022-07-03T10:15:54.263 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/126 2022-07-03T10:15:54.263 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 82/126 2022-07-03T10:15:54.263 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 83/126 2022-07-03T10:15:54.263 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 84/126 2022-07-03T10:15:54.264 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 85/126 2022-07-03T10:15:54.264 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : jq-1.6-3.el8.x86_64 86/126 2022-07-03T10:15:54.264 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 87/126 2022-07-03T10:15:54.264 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 88/126 2022-07-03T10:15:54.264 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : automake-1.16.1-7.el8.noarch 89/126 2022-07-03T10:15:54.265 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 90/126 2022-07-03T10:15:54.265 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 91/126 2022-07-03T10:15:54.265 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 92/126 2022-07-03T10:15:54.265 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flex-2.6.1-9.el8.x86_64 93/126 2022-07-03T10:15:54.266 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 94/126 2022-07-03T10:15:54.266 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 95/126 2022-07-03T10:15:54.266 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 96/126 2022-07-03T10:15:54.266 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 97/126 2022-07-03T10:15:54.267 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 98/126 2022-07-03T10:15:54.267 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/126 2022-07-03T10:15:54.267 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 100/126 2022-07-03T10:15:54.268 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 101/126 2022-07-03T10:15:54.268 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 102/126 2022-07-03T10:15:54.268 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 103/126 2022-07-03T10:15:54.268 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 104/126 2022-07-03T10:15:54.268 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/126 2022-07-03T10:15:54.269 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 106/126 2022-07-03T10:15:54.269 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 107/126 2022-07-03T10:15:54.269 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/126 2022-07-03T10:15:54.269 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 109/126 2022-07-03T10:15:54.270 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 110/126 2022-07-03T10:15:54.270 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 111/126 2022-07-03T10:15:54.270 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 112/126 2022-07-03T10:15:54.270 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 113/126 2022-07-03T10:15:54.271 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 114/126 2022-07-03T10:15:54.271 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 115/126 2022-07-03T10:15:54.271 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libquadmath-8.5.0-10.el8.x86_64 116/126 2022-07-03T10:15:54.271 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 117/126 2022-07-03T10:15:54.271 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libgfortran-8.5.0-10.el8.x86_64 118/126 2022-07-03T10:15:54.272 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 119/126 2022-07-03T10:15:54.272 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 120/126 2022-07-03T10:15:54.272 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 121/126 2022-07-03T10:15:54.272 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openssl-devel-1:1.1.1k-6.el8_5.x86_64 122/126 2022-07-03T10:15:54.272 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:17.0.0-13322.g61829bae.el8.x86_64 123/126 2022-07-03T10:15:54.273 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/126 2022-07-03T10:15:54.273 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 125/126 2022-07-03T10:15:58.894 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/126 2022-07-03T10:15:58.895 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2022-07-03T10:15:59.110 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:15:59.110 INFO:teuthology.orchestra.run.smithi101.stdout:Upgraded: 2022-07-03T10:15:59.111 INFO:teuthology.orchestra.run.smithi101.stdout: librados2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.111 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.111 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2022-07-03T10:15:59.111 INFO:teuthology.orchestra.run.smithi101.stdout: automake-1.16.1-7.el8.noarch 2022-07-03T10:15:59.112 INFO:teuthology.orchestra.run.smithi101.stdout: bison-3.0.4-10.el8.x86_64 2022-07-03T10:15:59.112 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.112 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.112 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.113 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.113 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.113 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.114 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.114 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.114 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.114 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.115 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.115 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.115 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.115 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.115 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.116 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.116 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.116 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.116 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.117 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.117 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.117 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.117 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top-2:17.0.0-13322.g61829bae.el8.noarch 2022-07-03T10:15:59.117 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-07-03T10:15:59.118 INFO:teuthology.orchestra.run.smithi101.stdout: flex-2.6.1-9.el8.x86_64 2022-07-03T10:15:59.118 INFO:teuthology.orchestra.run.smithi101.stdout: fuse-2.9.7-15.el8.x86_64 2022-07-03T10:15:59.118 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-07-03T10:15:59.118 INFO:teuthology.orchestra.run.smithi101.stdout: jq-1.6-3.el8.x86_64 2022-07-03T10:15:59.119 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-07-03T10:15:59.119 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-07-03T10:15:59.119 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-07-03T10:15:59.119 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.119 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.120 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.120 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-07-03T10:15:59.120 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran-8.5.0-10.el8.x86_64 2022-07-03T10:15:59.120 INFO:teuthology.orchestra.run.smithi101.stdout: liboath-2.6.2-3.el8.x86_64 2022-07-03T10:15:59.121 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath-8.5.0-10.el8.x86_64 2022-07-03T10:15:59.121 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-07-03T10:15:59.121 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.121 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.121 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-07-03T10:15:59.122 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.122 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-07-03T10:15:59.122 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-07-03T10:15:59.123 INFO:teuthology.orchestra.run.smithi101.stdout: libtool-2.4.6-25.el8.x86_64 2022-07-03T10:15:59.123 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind-1.3.1-3.el8.x86_64 2022-07-03T10:15:59.123 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-07-03T10:15:59.124 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-07-03T10:15:59.124 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-07-03T10:15:59.125 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-07-03T10:15:59.125 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-0.3.15-3.el8.x86_64 2022-07-03T10:15:59.125 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-07-03T10:15:59.126 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel-1:1.1.1k-6.el8_5.x86_64 2022-07-03T10:15:59.126 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-07-03T10:15:59.127 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-07-03T10:15:59.127 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-07-03T10:15:59.128 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-07-03T10:15:59.128 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel-2.5.1-7.el8.noarch 2022-07-03T10:15:59.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-07-03T10:15:59.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-07-03T10:15:59.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-07-03T10:15:59.129 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-07-03T10:15:59.130 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-07-03T10:15:59.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-07-03T10:15:59.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-07-03T10:15:59.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-07-03T10:15:59.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-07-03T10:15:59.131 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-6.2-6.el8.noarch 2022-07-03T10:15:59.132 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-07-03T10:15:59.132 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-07-03T10:15:59.132 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-07-03T10:15:59.132 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-07-03T10:15:59.133 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-07-03T10:15:59.133 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-07-03T10:15:59.133 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-07-03T10:15:59.134 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako-1.0.6-13.el8.noarch 2022-07-03T10:15:59.134 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-07-03T10:15:59.134 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-07-03T10:15:59.134 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-07-03T10:15:59.135 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:59.135 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-07-03T10:15:59.135 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-07-03T10:15:59.135 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-07-03T10:15:59.135 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend-2.6-1.el8.noarch 2022-07-03T10:15:59.136 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-07-03T10:15:59.136 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-07-03T10:15:59.136 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-07-03T10:15:59.136 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-07-03T10:15:59.137 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser-2.14-14.el8.noarch 2022-07-03T10:15:59.137 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz-2017.2-9.el8.noarch 2022-07-03T10:15:59.137 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.137 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.137 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-07-03T10:15:59.138 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-07-03T10:15:59.138 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.138 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes-2.4.1-12.el8.noarch 2022-07-03T10:15:59.138 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa-4.8-1.el8.noarch 2022-07-03T10:15:59.139 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-07-03T10:15:59.139 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-07-03T10:15:59.139 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-07-03T10:15:59.139 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-07-03T10:15:59.140 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-07-03T10:15:59.140 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-07-03T10:15:59.140 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-07-03T10:15:59.140 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-07-03T10:15:59.140 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-07-03T10:15:59.141 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-07-03T10:15:59.141 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-07-03T10:15:59.141 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.141 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.141 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd-2:17.0.0-13322.g61829bae.el8.x86_64 2022-07-03T10:15:59.142 INFO:teuthology.orchestra.run.smithi101.stdout: socat-1.7.4.1-1.el8.x86_64 2022-07-03T10:15:59.142 INFO:teuthology.orchestra.run.smithi101.stdout: thrift-0.13.0-2.el8.x86_64 2022-07-03T10:15:59.142 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-07-03T10:15:59.142 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-07-03T10:15:59.143 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-07-03T10:15:59.143 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:15:59.144 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2022-07-03T10:15:59.584 DEBUG:teuthology.parallel:result is None 2022-07-03T10:15:59.586 DEBUG:teuthology.orchestra.run.smithi101:> sudo lsb_release -is 2022-07-03T10:15:59.761 INFO:teuthology.orchestra.run.smithi101.stdout:RedHatEnterprise 2022-07-03T10:15:59.766 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:15:59.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:15:59.924 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-07-03T10:15:59.994 INFO:teuthology.orchestra.run.smithi101.stdout:17.0.0-13322.g61829bae.el8 2022-07-03T10:15:59.995 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13322.g61829bae.el8 2022-07-03T10:15:59.995 INFO:teuthology.task.install:The correct ceph version 17.0.0-13322.g61829bae is installed. 2022-07-03T10:15:59.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo lsb_release -is 2022-07-03T10:16:00.061 INFO:teuthology.orchestra.run.smithi161.stdout:RedHatEnterprise 2022-07-03T10:16:00.062 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:16:00.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:16:00.212 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-07-03T10:16:00.237 INFO:teuthology.orchestra.run.smithi161.stdout:17.0.0-13322.g61829bae.el8 2022-07-03T10:16:00.238 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13322.g61829bae.el8 2022-07-03T10:16:00.238 INFO:teuthology.task.install:The correct ceph version 17.0.0-13322.g61829bae is installed. 2022-07-03T10:16:00.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo lsb_release -is 2022-07-03T10:16:00.305 INFO:teuthology.orchestra.run.smithi188.stdout:RedHatEnterprise 2022-07-03T10:16:00.307 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-07-03T10:16:00.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:16:00.460 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-07-03T10:16:00.485 INFO:teuthology.orchestra.run.smithi188.stdout:17.0.0-13322.g61829bae.el8 2022-07-03T10:16:00.485 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-13322.g61829bae.el8 2022-07-03T10:16:00.486 INFO:teuthology.task.install:The correct ceph version 17.0.0-13322.g61829bae is installed. 2022-07-03T10:16:00.487 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-03T10:16:00.487 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:16:00.488 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-03T10:16:00.578 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:16:00.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-03T10:16:00.612 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:16:00.612 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-03T10:16:00.642 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-03T10:16:00.643 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:16:00.644 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/daemon-helper 2022-07-03T10:16:00.737 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-03T10:16:00.832 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:16:00.833 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2022-07-03T10:16:00.866 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-03T10:16:00.937 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:16:00.938 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2022-07-03T10:16:00.970 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-03T10:16:01.041 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-03T10:16:01.095 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:16:01.095 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-03T10:16:01.187 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-03T10:16:01.278 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:16:01.279 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-03T10:16:01.313 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-03T10:16:01.385 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:16:01.386 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-03T10:16:01.419 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-03T10:16:01.490 INFO:teuthology.run_tasks:Running task cephadm... 2022-07-03T10:16:01.574 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '61829bae536fb4126cfaa87ca353b9576e018e7d', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2022-07-03T10:16:01.575 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:16:01.577 INFO:tasks.cephadm:Cluster fsid is 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:16:01.577 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-07-03T10:16:01.577 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.101', 'mon.b': '172.21.15.161', 'mon.c': '172.21.15.188'} 2022-07-03T10:16:01.578 INFO:tasks.cephadm:First mon is mon.a on smithi101 2022-07-03T10:16:01.578 INFO:tasks.cephadm:First mgr is x 2022-07-03T10:16:01.579 INFO:tasks.cephadm:Normalizing hostnames... 2022-07-03T10:16:01.579 DEBUG:teuthology.orchestra.run.smithi101:> sudo hostname $(hostname -s) 2022-07-03T10:16:01.671 DEBUG:teuthology.orchestra.run.smithi161:> sudo hostname $(hostname -s) 2022-07-03T10:16:01.704 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2022-07-03T10:16:01.738 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref 61829bae536fb4126cfaa87ca353b9576e018e7d)... 2022-07-03T10:16:01.739 DEBUG:teuthology.orchestra.run.smithi101:> git archive --remote=git://git.ceph.com/ceph-ci.git 61829bae536fb4126cfaa87ca353b9576e018e7d src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.433 DEBUG:teuthology.orchestra.run.smithi161:> git archive --remote=git://git.ceph.com/ceph-ci.git 61829bae536fb4126cfaa87ca353b9576e018e7d src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.502 DEBUG:teuthology.orchestra.run.smithi188:> git archive --remote=git://git.ceph.com/ceph-ci.git 61829bae536fb4126cfaa87ca353b9576e018e7d src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.563 DEBUG:teuthology.orchestra.run.smithi101:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.631 DEBUG:teuthology.orchestra.run.smithi161:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.654 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-07-03T10:16:02.693 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d on all hosts... 2022-07-03T10:16:02.694 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d pull 2022-07-03T10:16:02.699 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d pull 2022-07-03T10:16:02.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d pull 2022-07-03T10:16:03.997 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d... 2022-07-03T10:16:05.621 INFO:teuthology.orchestra.run.smithi161.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d... 2022-07-03T10:16:08.195 INFO:teuthology.orchestra.run.smithi101.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d... 2022-07-03T10:16:35.489 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2022-07-03T10:16:35.490 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev)", 2022-07-03T10:16:35.490 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "ac818e5c233e1ff0edf6cf606e6ce3e1e74a3258e074af97ac42583e373ee541", 2022-07-03T10:16:35.490 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2022-07-03T10:16:35.491 INFO:teuthology.orchestra.run.smithi188.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d" 2022-07-03T10:16:35.491 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2022-07-03T10:16:35.491 INFO:teuthology.orchestra.run.smithi188.stdout:} 2022-07-03T10:16:49.342 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:16:49.343 INFO:teuthology.orchestra.run.smithi101.stdout: "ceph_version": "ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev)", 2022-07-03T10:16:49.343 INFO:teuthology.orchestra.run.smithi101.stdout: "image_id": "ac818e5c233e1ff0edf6cf606e6ce3e1e74a3258e074af97ac42583e373ee541", 2022-07-03T10:16:49.344 INFO:teuthology.orchestra.run.smithi101.stdout: "repo_digests": [ 2022-07-03T10:16:49.344 INFO:teuthology.orchestra.run.smithi101.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d" 2022-07-03T10:16:49.344 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2022-07-03T10:16:49.344 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:27:16.289 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2022-07-03T10:27:16.291 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev)", 2022-07-03T10:27:16.291 INFO:teuthology.orchestra.run.smithi161.stdout: "image_id": "ac818e5c233e1ff0edf6cf606e6ce3e1e74a3258e074af97ac42583e373ee541", 2022-07-03T10:27:16.292 INFO:teuthology.orchestra.run.smithi161.stdout: "repo_digests": [ 2022-07-03T10:27:16.292 INFO:teuthology.orchestra.run.smithi161.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d" 2022-07-03T10:27:16.292 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2022-07-03T10:27:16.293 INFO:teuthology.orchestra.run.smithi161.stdout:} 2022-07-03T10:27:16.307 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /etc/ceph 2022-07-03T10:27:16.427 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph 2022-07-03T10:27:16.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2022-07-03T10:27:16.516 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 777 /etc/ceph 2022-07-03T10:27:16.605 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /etc/ceph 2022-07-03T10:27:16.633 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2022-07-03T10:27:16.665 INFO:tasks.cephadm:Writing seed config... 2022-07-03T10:27:16.667 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-07-03T10:27:16.668 INFO:tasks.cephadm: override: [client] debug client = 20 2022-07-03T10:27:16.668 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-07-03T10:27:16.669 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-07-03T10:27:16.669 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-07-03T10:27:16.669 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2022-07-03T10:27:16.669 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-07-03T10:27:16.670 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-07-03T10:27:16.670 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2022-07-03T10:27:16.670 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2022-07-03T10:27:16.671 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2022-07-03T10:27:16.671 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2022-07-03T10:27:16.672 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-07-03T10:27:16.672 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-07-03T10:27:16.673 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-07-03T10:27:16.673 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-07-03T10:27:16.673 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-07-03T10:27:16.674 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-07-03T10:27:16.674 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-07-03T10:27:16.674 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-07-03T10:27:16.675 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-07-03T10:27:16.675 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-07-03T10:27:16.675 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-07-03T10:27:16.676 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-07-03T10:27:16.676 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-07-03T10:27:16.677 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-07-03T10:27:16.678 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2022-07-03T10:27:16.678 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-07-03T10:27:16.679 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2022-07-03T10:27:16.679 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2022-07-03T10:27:16.680 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-07-03T10:27:16.680 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-07-03T10:27:16.681 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2022-07-03T10:27:16.681 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2022-07-03T10:27:16.681 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-07-03T10:27:16.682 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-07-03T10:27:16.682 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-07-03T10:27:16.682 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-07-03T10:27:16.682 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2022-07-03T10:27:16.683 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-07-03T10:27:16.683 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-07-03T10:27:16.683 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-07-03T10:27:16.684 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2022-07-03T10:27:16.685 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:27:16.685 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-07-03T10:27:16.743 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 23a851ae-fab9-11ec-842c-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-07-03T10:27:16.744 DEBUG:teuthology.orchestra.run.smithi101:mon.a> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mon.a.service 2022-07-03T10:27:16.751 DEBUG:teuthology.orchestra.run.smithi101:mgr.x> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mgr.x.service 2022-07-03T10:27:16.804 INFO:tasks.cephadm:Bootstrapping... 2022-07-03T10:27:16.804 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d -v bootstrap --fsid 23a851ae-fab9-11ec-842c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.101 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-07-03T10:27:16.838 INFO:journalctl@ceph.mon.a.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:27:16.887 INFO:journalctl@ceph.mgr.x.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:27:17.295 INFO:teuthology.orchestra.run.smithi101.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2022-07-03T10:27:17.296 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying podman|docker is present... 2022-07-03T10:27:17.492 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying lvm2 is present... 2022-07-03T10:27:17.492 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying time synchronization is in place... 2022-07-03T10:27:17.561 INFO:teuthology.orchestra.run.smithi101.stdout:Unit chronyd.service is enabled and running 2022-07-03T10:27:17.561 INFO:teuthology.orchestra.run.smithi101.stdout:Repeating the final host check... 2022-07-03T10:27:17.755 INFO:teuthology.orchestra.run.smithi101.stdout:podman (/bin/podman) version 4.0.2 is present 2022-07-03T10:27:17.756 INFO:teuthology.orchestra.run.smithi101.stdout:systemctl is present 2022-07-03T10:27:17.756 INFO:teuthology.orchestra.run.smithi101.stdout:lvcreate is present 2022-07-03T10:27:17.821 INFO:teuthology.orchestra.run.smithi101.stdout:Unit chronyd.service is enabled and running 2022-07-03T10:27:17.822 INFO:teuthology.orchestra.run.smithi101.stdout:Host looks OK 2022-07-03T10:27:17.822 INFO:teuthology.orchestra.run.smithi101.stdout:Cluster fsid: 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:27:17.823 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying IP 172.21.15.101 port 3300 ... 2022-07-03T10:27:17.823 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying IP 172.21.15.101 port 6789 ... 2022-07-03T10:27:17.852 INFO:teuthology.orchestra.run.smithi101.stdout:Mon IP `172.21.15.101` is in CIDR network `172.21.0.0/20` 2022-07-03T10:27:17.853 INFO:teuthology.orchestra.run.smithi101.stdout:Mon IP `172.21.15.101` is in CIDR network `172.21.0.0/20` 2022-07-03T10:27:17.854 INFO:teuthology.orchestra.run.smithi101.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-07-03T10:27:17.854 INFO:teuthology.orchestra.run.smithi101.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d... 2022-07-03T10:27:19.356 INFO:teuthology.orchestra.run.smithi101.stdout:Ceph version: ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev) 2022-07-03T10:27:19.357 INFO:teuthology.orchestra.run.smithi101.stdout:Extracting ceph user uid/gid from container image... 2022-07-03T10:27:20.156 INFO:teuthology.orchestra.run.smithi101.stdout:Creating initial keys... 2022-07-03T10:27:22.555 INFO:teuthology.orchestra.run.smithi101.stdout:Creating initial monmap... 2022-07-03T10:27:23.356 INFO:teuthology.orchestra.run.smithi101.stdout:Creating mon... 2022-07-03T10:27:26.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:25 smithi101 systemd[1]: Starting Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:27:26.666 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:26 smithi101 podman[84599]: 2022-07-03T10:27:26.727 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:26 smithi101 bash[84599]: 05eeab579b0a05278aaaa9b4ee0292d6269177fa54bbccdeae2170e25e3af627 2022-07-03T10:27:26.727 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:26 smithi101 ceph-mon[84639]: mkfs 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:27:26.784 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mon to start... 2022-07-03T10:27:26.785 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mon... 2022-07-03T10:27:26.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:26 smithi101 ceph-mon[84639]: mon.a is new leader, mons a in quorum (ranks 0) 2022-07-03T10:27:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:26 smithi101 systemd[1]: Started Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:27:27.792 INFO:teuthology.orchestra.run.smithi101.stdout:mon is available 2022-07-03T10:27:27.792 INFO:teuthology.orchestra.run.smithi101.stdout:Assimilating anything we can from ceph.conf... 2022-07-03T10:27:27.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: mon.a is new leader, mons a in quorum (ranks 0) 2022-07-03T10:27:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: monmap e1: 1 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0]} 2022-07-03T10:27:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: fsmap 2022-07-03T10:27:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: osdmap e1: 0 total, 0 up, 0 in 2022-07-03T10:27:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: mgrmap e1: no daemons active 2022-07-03T10:27:27.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:27 smithi101 ceph-mon[84639]: from='client.? 172.21.15.101:0/842913286' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-07-03T10:27:28.793 INFO:teuthology.orchestra.run.smithi101.stdout:Generating new minimal ceph.conf... 2022-07-03T10:27:28.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:28 smithi101 ceph-mon[84639]: from='client.? 172.21.15.101:0/2648391413' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-07-03T10:27:28.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:28 smithi101 ceph-mon[84639]: from='client.? 172.21.15.101:0/2648391413' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-07-03T10:27:29.798 INFO:teuthology.orchestra.run.smithi101.stdout:Restarting the monitor... 2022-07-03T10:27:29.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:29 smithi101 ceph-mon[84639]: from='client.? 172.21.15.101:0/2012583995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:27:29.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:29 smithi101 systemd[1]: Stopping Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:27:30.607 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mon-a[84615]: 2022-07-03T10:27:30.313+0000 7fea06d1d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2022-07-03T10:27:30.607 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mon-a[84615]: 2022-07-03T10:27:30.313+0000 7fea06d1d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-07-03T10:27:30.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 bash[84976]: ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mon-a 2022-07-03T10:27:30.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 systemd[1]: ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mon.a.service: Succeeded. 2022-07-03T10:27:30.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 systemd[1]: Stopped Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:27:30.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:30 smithi101 systemd[1]: Starting Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:27:31.601 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 podman[85086]: 2022-07-03T10:27:31.602 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 bash[85086]: 89cf619fbd64128444e160f4d1931bc6eb05b5ceff2023f86ad3fecf7eddfbb3 2022-07-03T10:27:31.678 INFO:teuthology.orchestra.run.smithi101.stdout:Setting mon public_network to 172.21.0.0/20 2022-07-03T10:27:31.939 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: set uid:gid to 167:167 (ceph:ceph) 2022-07-03T10:27:31.939 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev), process ceph-mon, pid 7 2022-07-03T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: pidfile_write: ignore empty --pid-file 2022-07-03T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: load: jerasure load: lrc 2022-07-03T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: RocksDB version: 6.15.5 2022-07-03T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-07-03T10:27:31.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Compile date Jul 2 2022 2022-07-03T10:27:31.941 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: DB SUMMARY 2022-07-03T10:27:31.941 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: DB Session ID: 8BJDEK43UHYCLLNH8GHI 2022-07-03T10:27:31.941 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: CURRENT file: CURRENT 2022-07-03T10:27:31.941 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: IDENTITY file: IDENTITY 2022-07-03T10:27:31.941 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2022-07-03T10:27:31.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2022-07-03T10:27:31.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 96135 ; 2022-07-03T10:27:31.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.error_if_exists: 0 2022-07-03T10:27:31.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.create_if_missing: 0 2022-07-03T10:27:31.943 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.paranoid_checks: 1 2022-07-03T10:27:31.943 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-07-03T10:27:31.943 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.env: 0x5633d3ae1860 2022-07-03T10:27:31.943 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.fs: Posix File System 2022-07-03T10:27:31.943 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.info_log: 0x5633d435be80 2022-07-03T10:27:31.944 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_file_opening_threads: 16 2022-07-03T10:27:31.944 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.statistics: (nil) 2022-07-03T10:27:31.944 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.use_fsync: 0 2022-07-03T10:27:31.944 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_log_file_size: 0 2022-07-03T10:27:31.945 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-07-03T10:27:31.945 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.log_file_time_to_roll: 0 2022-07-03T10:27:31.945 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.keep_log_file_num: 1000 2022-07-03T10:27:31.945 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.recycle_log_file_num: 0 2022-07-03T10:27:31.945 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_fallocate: 1 2022-07-03T10:27:31.946 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_mmap_reads: 0 2022-07-03T10:27:31.946 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_mmap_writes: 0 2022-07-03T10:27:31.946 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.use_direct_reads: 0 2022-07-03T10:27:31.946 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-07-03T10:27:31.947 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.create_missing_column_families: 0 2022-07-03T10:27:31.947 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.db_log_dir: 2022-07-03T10:27:31.947 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-07-03T10:27:31.947 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.table_cache_numshardbits: 6 2022-07-03T10:27:31.947 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.WAL_ttl_seconds: 0 2022-07-03T10:27:31.948 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.WAL_size_limit_MB: 0 2022-07-03T10:27:31.948 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-07-03T10:27:31.948 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-07-03T10:27:31.948 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.is_fd_close_on_exec: 1 2022-07-03T10:27:31.948 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.advise_random_on_open: 1 2022-07-03T10:27:31.949 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.db_write_buffer_size: 0 2022-07-03T10:27:31.949 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.write_buffer_manager: 0x5633d4448960 2022-07-03T10:27:31.949 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-07-03T10:27:31.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-07-03T10:27:31.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-07-03T10:27:31.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.use_adaptive_mutex: 0 2022-07-03T10:27:31.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.rate_limiter: (nil) 2022-07-03T10:27:31.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-07-03T10:27:31.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.wal_recovery_mode: 2 2022-07-03T10:27:31.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.enable_thread_tracking: 0 2022-07-03T10:27:31.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.enable_pipelined_write: 0 2022-07-03T10:27:31.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.unordered_write: 0 2022-07-03T10:27:31.952 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-07-03T10:27:31.952 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-07-03T10:27:31.952 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-07-03T10:27:31.953 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-07-03T10:27:31.953 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.row_cache: None 2022-07-03T10:27:31.953 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.wal_filter: None 2022-07-03T10:27:31.953 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-07-03T10:27:31.953 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_ingest_behind: 0 2022-07-03T10:27:31.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.preserve_deletes: 0 2022-07-03T10:27:31.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.two_write_queues: 0 2022-07-03T10:27:31.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.manual_wal_flush: 0 2022-07-03T10:27:31.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.atomic_flush: 0 2022-07-03T10:27:31.955 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-07-03T10:27:31.955 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.persist_stats_to_disk: 0 2022-07-03T10:27:31.955 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.write_dbid_to_manifest: 0 2022-07-03T10:27:31.955 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.log_readahead_size: 0 2022-07-03T10:27:31.955 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-07-03T10:27:31.956 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.best_efforts_recovery: 0 2022-07-03T10:27:31.956 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-07-03T10:27:31.956 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-07-03T10:27:31.956 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.allow_data_in_errors: 0 2022-07-03T10:27:31.956 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.db_host_id: __hostname__ 2022-07-03T10:27:31.957 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_background_jobs: 2 2022-07-03T10:27:31.957 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_background_compactions: -1 2022-07-03T10:27:31.957 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_subcompactions: 1 2022-07-03T10:27:31.957 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-07-03T10:27:31.957 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-07-03T10:27:31.958 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.delayed_write_rate : 16777216 2022-07-03T10:27:31.958 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_total_wal_size: 0 2022-07-03T10:27:31.958 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-07-03T10:27:31.958 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.stats_dump_period_sec: 600 2022-07-03T10:27:31.959 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.stats_persist_period_sec: 600 2022-07-03T10:27:31.959 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-07-03T10:27:31.959 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_open_files: -1 2022-07-03T10:27:31.959 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bytes_per_sync: 0 2022-07-03T10:27:31.959 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.wal_bytes_per_sync: 0 2022-07-03T10:27:31.960 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.strict_bytes_per_sync: 0 2022-07-03T10:27:31.960 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_readahead_size: 0 2022-07-03T10:27:31.960 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_background_flushes: -1 2022-07-03T10:27:31.960 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Compression algorithms supported: 2022-07-03T10:27:31.961 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-07-03T10:27:31.961 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kZSTD supported: 0 2022-07-03T10:27:31.961 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kXpressCompression supported: 0 2022-07-03T10:27:31.961 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kLZ4HCCompression supported: 1 2022-07-03T10:27:31.961 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kLZ4Compression supported: 1 2022-07-03T10:27:31.962 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kBZip2Compression supported: 0 2022-07-03T10:27:31.962 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kZlibCompression supported: 1 2022-07-03T10:27:31.962 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: kSnappyCompression supported: 1 2022-07-03T10:27:31.962 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Fast CRC32 supported: Supported on x86 2022-07-03T10:27:31.963 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-07-03T10:27:31.963 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-07-03T10:27:31.963 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-07-03T10:27:31.963 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.merge_operator: 2022-07-03T10:27:31.963 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_filter: None 2022-07-03T10:27:31.964 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_filter_factory: None 2022-07-03T10:27:31.964 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.sst_partitioner_factory: None 2022-07-03T10:27:31.964 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.memtable_factory: SkipListFactory 2022-07-03T10:27:31.964 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.table_factory: BlockBasedTable 2022-07-03T10:27:31.964 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5633d43599f8) 2022-07-03T10:27:31.965 INFO:journalctl@ceph.mon.a.smithi101.stdout: cache_index_and_filter_blocks: 1 2022-07-03T10:27:31.965 INFO:journalctl@ceph.mon.a.smithi101.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-07-03T10:27:31.965 INFO:journalctl@ceph.mon.a.smithi101.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-07-03T10:27:31.965 INFO:journalctl@ceph.mon.a.smithi101.stdout: pin_top_level_index_and_filter: 1 2022-07-03T10:27:31.965 INFO:journalctl@ceph.mon.a.smithi101.stdout: index_type: 0 2022-07-03T10:27:31.966 INFO:journalctl@ceph.mon.a.smithi101.stdout: data_block_index_type: 0 2022-07-03T10:27:31.966 INFO:journalctl@ceph.mon.a.smithi101.stdout: index_shortening: 1 2022-07-03T10:27:31.966 INFO:journalctl@ceph.mon.a.smithi101.stdout: data_block_hash_table_util_ratio: 0.750000 2022-07-03T10:27:31.966 INFO:journalctl@ceph.mon.a.smithi101.stdout: hash_index_allow_collision: 1 2022-07-03T10:27:31.967 INFO:journalctl@ceph.mon.a.smithi101.stdout: checksum: 1 2022-07-03T10:27:31.967 INFO:journalctl@ceph.mon.a.smithi101.stdout: no_block_cache: 0 2022-07-03T10:27:31.967 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_cache: 0x5633d43c49b0 2022-07-03T10:27:31.967 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_cache_name: BinnedLRUCache 2022-07-03T10:27:31.967 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_cache_options: 2022-07-03T10:27:31.968 INFO:journalctl@ceph.mon.a.smithi101.stdout: capacity : 536870912 2022-07-03T10:27:31.968 INFO:journalctl@ceph.mon.a.smithi101.stdout: num_shard_bits : 4 2022-07-03T10:27:31.968 INFO:journalctl@ceph.mon.a.smithi101.stdout: strict_capacity_limit : 0 2022-07-03T10:27:31.968 INFO:journalctl@ceph.mon.a.smithi101.stdout: high_pri_pool_ratio: 0.000 2022-07-03T10:27:31.968 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_cache_compressed: (nil) 2022-07-03T10:27:31.969 INFO:journalctl@ceph.mon.a.smithi101.stdout: persistent_cache: (nil) 2022-07-03T10:27:31.969 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_size: 4096 2022-07-03T10:27:31.969 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_size_deviation: 10 2022-07-03T10:27:31.969 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_restart_interval: 16 2022-07-03T10:27:31.970 INFO:journalctl@ceph.mon.a.smithi101.stdout: index_block_restart_interval: 1 2022-07-03T10:27:31.970 INFO:journalctl@ceph.mon.a.smithi101.stdout: metadata_block_size: 4096 2022-07-03T10:27:31.970 INFO:journalctl@ceph.mon.a.smithi101.stdout: partition_filters: 0 2022-07-03T10:27:31.970 INFO:journalctl@ceph.mon.a.smithi101.stdout: use_delta_encoding: 1 2022-07-03T10:27:31.971 INFO:journalctl@ceph.mon.a.smithi101.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-07-03T10:27:31.971 INFO:journalctl@ceph.mon.a.smithi101.stdout: whole_key_filtering: 1 2022-07-03T10:27:31.971 INFO:journalctl@ceph.mon.a.smithi101.stdout: verify_compression: 0 2022-07-03T10:27:31.971 INFO:journalctl@ceph.mon.a.smithi101.stdout: read_amp_bytes_per_bit: 0 2022-07-03T10:27:31.972 INFO:journalctl@ceph.mon.a.smithi101.stdout: format_version: 4 2022-07-03T10:27:31.972 INFO:journalctl@ceph.mon.a.smithi101.stdout: enable_index_compression: 1 2022-07-03T10:27:31.972 INFO:journalctl@ceph.mon.a.smithi101.stdout: block_align: 0 2022-07-03T10:27:31.972 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.write_buffer_size: 33554432 2022-07-03T10:27:31.972 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_write_buffer_number: 2 2022-07-03T10:27:31.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression: NoCompression 2022-07-03T10:27:31.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression: Disabled 2022-07-03T10:27:31.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.prefix_extractor: nullptr 2022-07-03T10:27:31.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-07-03T10:27:31.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.num_levels: 7 2022-07-03T10:27:31.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-07-03T10:27:31.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-07-03T10:27:31.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-07-03T10:27:31.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-07-03T10:27:31.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-07-03T10:27:31.975 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-07-03T10:27:31.975 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-07-03T10:27:31.975 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-07-03T10:27:31.975 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-07-03T10:27:31.975 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-07-03T10:27:31.976 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.window_bits: -14 2022-07-03T10:27:31.976 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.level: 32767 2022-07-03T10:27:31.976 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.strategy: 0 2022-07-03T10:27:31.976 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-07-03T10:27:31.976 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-07-03T10:27:31.977 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-07-03T10:27:31.977 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compression_opts.enabled: false 2022-07-03T10:27:31.977 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-07-03T10:27:31.977 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-07-03T10:27:31.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-07-03T10:27:31.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.target_file_size_base: 67108864 2022-07-03T10:27:31.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.target_file_size_multiplier: 1 2022-07-03T10:27:31.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-07-03T10:27:31.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-07-03T10:27:31.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-07-03T10:27:31.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-07-03T10:27:31.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-07-03T10:27:31.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-07-03T10:27:31.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-07-03T10:27:31.980 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-07-03T10:27:31.980 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-07-03T10:27:31.980 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-07-03T10:27:31.980 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-07-03T10:27:31.980 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-07-03T10:27:31.981 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.arena_block_size: 4194304 2022-07-03T10:27:31.981 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-07-03T10:27:31.981 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-07-03T10:27:31.981 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-07-03T10:27:31.982 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.disable_auto_compactions: 0 2022-07-03T10:27:31.982 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-07-03T10:27:31.982 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-07-03T10:27:31.982 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-07-03T10:27:31.982 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-07-03T10:27:31.983 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-07-03T10:27:31.983 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-07-03T10:27:31.983 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-07-03T10:27:31.983 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-07-03T10:27:31.983 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-07-03T10:27:31.984 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-07-03T10:27:31.984 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.table_properties_collectors: 2022-07-03T10:27:31.984 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.inplace_update_support: 0 2022-07-03T10:27:31.984 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.inplace_update_num_locks: 10000 2022-07-03T10:27:31.984 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-07-03T10:27:31.985 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-07-03T10:27:31.985 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.memtable_huge_page_size: 0 2022-07-03T10:27:31.985 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.bloom_locality: 0 2022-07-03T10:27:31.985 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.max_successive_merges: 0 2022-07-03T10:27:31.985 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.optimize_filters_for_hits: 0 2022-07-03T10:27:31.986 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.paranoid_file_checks: 0 2022-07-03T10:27:31.986 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.force_consistency_checks: 1 2022-07-03T10:27:31.986 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.report_bg_io_stats: 0 2022-07-03T10:27:31.986 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.ttl: 2592000 2022-07-03T10:27:31.987 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.periodic_compaction_seconds: 0 2022-07-03T10:27:31.987 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.enable_blob_files: false 2022-07-03T10:27:31.987 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.min_blob_size: 0 2022-07-03T10:27:31.987 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.blob_file_size: 268435456 2022-07-03T10:27:31.987 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.blob_compression_type: NoCompression 2022-07-03T10:27:31.988 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.enable_blob_garbage_collection: false 2022-07-03T10:27:31.988 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-07-03T10:27:31.988 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-07-03T10:27:31.988 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2022-07-03T10:27:31.988 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2022-07-03T10:27:31.989 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844051613785, "job": 1, "event": "recovery_started", "wal_files": [10]} 2022-07-03T10:27:31.989 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2022-07-03T10:27:31.989 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-07-03T10:27:31.989 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844051615562, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 91477, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 89570, "index_size": 267, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13275, "raw_average_key_size": 49, "raw_value_size": 82289, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1656844051, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "34d7b518-c94c-40c6-9ca0-d6d04d3f1dcd", "db_session_id": "8BJDEK43UHYCLLNH8GHI"}} 2022-07-03T10:27:31.990 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2022-07-03T10:27:31.990 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844051616431, "job": 1, "event": "recovery_finished"} 2022-07-03T10:27:31.990 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-07-03T10:27:31.990 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5633d43aa700 2022-07-03T10:27:31.990 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: DB pointer 0x5633d43ba000 2022-07-03T10:27:31.991 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-07-03T10:27:31.991 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-07-03T10:27:31.991 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** DB Stats ** 2022-07-03T10:27:31.991 INFO:journalctl@ceph.mon.a.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:27:31.991 INFO:journalctl@ceph.mon.a.smithi101.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-07-03T10:27:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-07-03T10:27:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-07-03T10:27:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-07-03T10:27:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-07-03T10:27:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-07-03T10:27:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** Compaction Stats [default] ** 2022-07-03T10:27:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:27:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:27:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout: L0 2/0 91.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout: Sum 2/0 91.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:31.995 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** Compaction Stats [default] ** 2022-07-03T10:27:31.995 INFO:journalctl@ceph.mon.a.smithi101.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:27:31.995 INFO:journalctl@ceph.mon.a.smithi101.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:27:31.995 INFO:journalctl@ceph.mon.a.smithi101.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.995 INFO:journalctl@ceph.mon.a.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:27:31.996 INFO:journalctl@ceph.mon.a.smithi101.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-07-03T10:27:31.996 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-07-03T10:27:31.996 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-07-03T10:27:31.996 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-07-03T10:27:31.996 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(Keys): cumulative 0, interval 0 2022-07-03T10:27:31.997 INFO:journalctl@ceph.mon.a.smithi101.stdout: Cumulative compaction: 0.00 GB write, 9.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:27:31.997 INFO:journalctl@ceph.mon.a.smithi101.stdout: Interval compaction: 0.00 GB write, 9.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:27:31.997 INFO:journalctl@ceph.mon.a.smithi101.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-07-03T10:27:31.997 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:31.997 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** File Read Latency Histogram By Level [default] ** 2022-07-03T10:27:31.998 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:31.998 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** Compaction Stats [default] ** 2022-07-03T10:27:31.998 INFO:journalctl@ceph.mon.a.smithi101.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:27:31.998 INFO:journalctl@ceph.mon.a.smithi101.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:27:31.999 INFO:journalctl@ceph.mon.a.smithi101.stdout: L0 2/0 91.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.999 INFO:journalctl@ceph.mon.a.smithi101.stdout: Sum 2/0 91.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:31.999 INFO:journalctl@ceph.mon.a.smithi101.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-07-03T10:27:31.999 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:32.000 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** Compaction Stats [default] ** 2022-07-03T10:27:32.000 INFO:journalctl@ceph.mon.a.smithi101.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:27:32.000 INFO:journalctl@ceph.mon.a.smithi101.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:27:32.000 INFO:journalctl@ceph.mon.a.smithi101.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.8 0.00 0.00 1 0.001 0 0 2022-07-03T10:27:32.000 INFO:journalctl@ceph.mon.a.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:27:32.001 INFO:journalctl@ceph.mon.a.smithi101.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-07-03T10:27:32.001 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-07-03T10:27:32.001 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-07-03T10:27:32.001 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-07-03T10:27:32.001 INFO:journalctl@ceph.mon.a.smithi101.stdout: AddFile(Keys): cumulative 0, interval 0 2022-07-03T10:27:32.002 INFO:journalctl@ceph.mon.a.smithi101.stdout: Cumulative compaction: 0.00 GB write, 8.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:27:32.002 INFO:journalctl@ceph.mon.a.smithi101.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:27:32.002 INFO:journalctl@ceph.mon.a.smithi101.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-07-03T10:27:32.003 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:32.003 INFO:journalctl@ceph.mon.a.smithi101.stdout: ** File Read Latency Histogram By Level [default] ** 2022-07-03T10:27:32.003 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: starting mon.a rank 0 at public addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] at bind addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:27:32.003 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???) e1 preinit fsid 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:27:32.004 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).mds e1 new map 2022-07-03T10:27:32.004 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).mds e1 print_map 2022-07-03T10:27:32.004 INFO:journalctl@ceph.mon.a.smithi101.stdout: e1 2022-07-03T10:27:32.004 INFO:journalctl@ceph.mon.a.smithi101.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-07-03T10:27:32.005 INFO:journalctl@ceph.mon.a.smithi101.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-07-03T10:27:32.005 INFO:journalctl@ceph.mon.a.smithi101.stdout: legacy client fscid: -1 2022-07-03T10:27:32.005 INFO:journalctl@ceph.mon.a.smithi101.stdout: 2022-07-03T10:27:32.005 INFO:journalctl@ceph.mon.a.smithi101.stdout: No filesystems configured 2022-07-03T10:27:32.005 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-07-03T10:27:32.006 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:27:32.006 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:27:32.006 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:27:32.006 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-07-03T10:27:32.006 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expand map: {default=false} 2022-07-03T10:27:32.007 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta from 'false' to 'false' 2022-07-03T10:27:32.007 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expanded map: {default=false} 2022-07-03T10:27:32.007 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expand map: {default=info} 2022-07-03T10:27:32.007 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta from 'info' to 'info' 2022-07-03T10:27:32.007 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expanded map: {default=info} 2022-07-03T10:27:32.008 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expand map: {default=daemon} 2022-07-03T10:27:32.008 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta from 'daemon' to 'daemon' 2022-07-03T10:27:32.008 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expanded map: {default=daemon} 2022-07-03T10:27:32.008 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expand map: {default=debug} 2022-07-03T10:27:32.008 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta from 'debug' to 'debug' 2022-07-03T10:27:32.009 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: expand_channel_meta expanded map: {default=debug} 2022-07-03T10:27:32.009 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 systemd[1]: Started Ceph mon.a for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:27:32.009 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mon.a is new leader, mons a in quorum (ranks 0) 2022-07-03T10:27:32.009 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: monmap e1: 1 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0]} 2022-07-03T10:27:32.010 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: fsmap 2022-07-03T10:27:32.010 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: osdmap e1: 0 total, 0 up, 0 in 2022-07-03T10:27:32.010 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:31 smithi101 ceph-mon[85126]: mgrmap e1: no daemons active 2022-07-03T10:27:32.689 INFO:teuthology.orchestra.run.smithi101.stdout:Wrote config to /etc/ceph/ceph.conf 2022-07-03T10:27:32.690 INFO:teuthology.orchestra.run.smithi101.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-07-03T10:27:32.690 INFO:teuthology.orchestra.run.smithi101.stdout:Creating mgr... 2022-07-03T10:27:32.690 INFO:teuthology.orchestra.run.smithi101.stdout:Verifying port 9283 ... 2022-07-03T10:27:33.380 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:33 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/317015135' entity='client.admin' 2022-07-03T10:27:33.743 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:33 smithi101 systemd[1]: Starting Ceph mgr.x for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:27:34.207 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:33 smithi101 podman[85363]: 2022-07-03T10:27:34.367 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mgr to start... 2022-07-03T10:27:34.367 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mgr... 2022-07-03T10:27:34.492 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:34 smithi101 bash[85363]: a4a271435dab3aa8b7a3d57c79e1d65ae9d8ddc44621701f55f34db5934077e4 2022-07-03T10:27:34.493 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:34 smithi101 systemd[1]: Started Ceph mgr.x for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:27:34.493 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:34 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:34.430+0000 7fbc2d080080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-03T10:27:34.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:34 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:34.873+0000 7fbc2d080080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-03T10:27:35.411 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/53400832' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-07-03T10:27:35.412 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.125+0000 7fbc2d080080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-03T10:27:35.413 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.283+0000 7fbc2d080080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-03T10:27:35.477 INFO:teuthology.orchestra.run.smithi101.stdout:mgr not available, waiting (1/15)... 2022-07-03T10:27:35.686 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.409+0000 7fbc2d080080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-03T10:27:35.686 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.566+0000 7fbc2d080080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-03T10:27:35.687 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.683+0000 7fbc2d080080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-03T10:27:35.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.861+0000 7fbc2d080080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-03T10:27:36.248 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:35 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:35.993+0000 7fbc2d080080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-03T10:27:36.551 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:36 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:36.246+0000 7fbc2d080080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-03T10:27:36.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:36 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:36.549+0000 7fbc2d080080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:27:37.719 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:37 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:37.418+0000 7fbc2d080080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-03T10:27:37.719 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:37 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:37.563+0000 7fbc2d080080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-03T10:27:37.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:37 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:37.809+0000 7fbc2d080080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-03T10:27:37.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:37 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:37.960+0000 7fbc2d080080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-03T10:27:38.482 INFO:teuthology.orchestra.run.smithi101.stdout:mgr not available, waiting (2/15)... 2022-07-03T10:27:38.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:38 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3741233348' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-07-03T10:27:38.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:38 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:38.569+0000 7fbc2d080080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:27:39.447 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:39 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:39.154+0000 7fbc2d080080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-03T10:27:39.448 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:39 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:39.299+0000 7fbc2d080080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-03T10:27:39.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:39 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:39.445+0000 7fbc2d080080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-03T10:27:40.734 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:40 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:40.319+0000 7fbc2d080080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-03T10:27:40.734 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:40 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:40.481+0000 7fbc2d080080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-03T10:27:40.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:40 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:40.956+0000 7fbc2d080080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-03T10:27:41.485 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:41 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/166245196' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-07-03T10:27:41.489 INFO:teuthology.orchestra.run.smithi101.stdout:mgr not available, waiting (3/15)... 2022-07-03T10:27:42.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:42 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:42.415+0000 7fbc2d080080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-03T10:27:43.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:43.342+0000 7fbc2d080080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-03T10:27:43.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: Activating manager daemon x 2022-07-03T10:27:43.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: mgrmap e2: x(active, starting, since 0.0061545s) 2022-07-03T10:27:43.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-07-03T10:27:43.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-07-03T10:27:43.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-07-03T10:27:43.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:27:43.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-07-03T10:27:43.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: Manager daemon x is now available 2022-07-03T10:27:43.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:27:43.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:27:43.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:43 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' 2022-07-03T10:27:44.501 INFO:teuthology.orchestra.run.smithi101.stdout:mgr not available, waiting (4/15)... 2022-07-03T10:27:44.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:44 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' 2022-07-03T10:27:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:44 smithi101 ceph-mon[85126]: from='mgr.14100 172.21.15.101:0/2089470654' entity='mgr.x' 2022-07-03T10:27:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:44 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1467521793' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-07-03T10:27:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:44 smithi101 ceph-mon[85126]: mgrmap e3: x(active, since 1.01004s) 2022-07-03T10:27:46.829 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:46 smithi101 ceph-mon[85126]: mgrmap e4: x(active, since 2s) 2022-07-03T10:27:47.709 INFO:teuthology.orchestra.run.smithi101.stdout:mgr is available 2022-07-03T10:27:47.940 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:47 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3355144180' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-07-03T10:27:48.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:48 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1809355840' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-07-03T10:27:48.811 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling cephadm module... 2022-07-03T10:27:49.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:49 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/981357568' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-07-03T10:27:49.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:49 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: ignoring --setuser ceph since I am not root 2022-07-03T10:27:49.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:49 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: ignoring --setgroup ceph since I am not root 2022-07-03T10:27:50.365 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:50.011+0000 7f7ea735a080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-03T10:27:50.636 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:50.430+0000 7f7ea735a080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-03T10:27:50.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/981357568' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-07-03T10:27:50.954 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-mon[85126]: mgrmap e5: x(active, since 6s) 2022-07-03T10:27:50.955 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:50.688+0000 7f7ea735a080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-03T10:27:50.955 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:50.839+0000 7f7ea735a080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-03T10:27:51.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:50 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:50.967+0000 7f7ea735a080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-03T10:27:51.243 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:51.136+0000 7f7ea735a080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-03T10:27:51.320 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for the mgr to restart... 2022-07-03T10:27:51.321 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mgr epoch 5... 2022-07-03T10:27:51.592 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:51.257+0000 7f7ea735a080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-03T10:27:51.592 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:51.452+0000 7f7ea735a080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-03T10:27:51.885 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1964113946' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-07-03T10:27:51.886 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:51.589+0000 7f7ea735a080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-03T10:27:52.194 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:51 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:51.883+0000 7f7ea735a080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-03T10:27:53.392 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:52 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:52.192+0000 7f7ea735a080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:27:53.393 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:53 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:53.018+0000 7f7ea735a080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-03T10:27:53.393 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:53 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:53.156+0000 7f7ea735a080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-03T10:27:53.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:53 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:53.390+0000 7f7ea735a080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-03T10:27:53.743 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:53 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:53.535+0000 7f7ea735a080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-03T10:27:54.492 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:54 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:54.094+0000 7f7ea735a080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:27:54.970 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:54 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:54.678+0000 7f7ea735a080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-03T10:27:54.970 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:54 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:54.823+0000 7f7ea735a080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-03T10:27:55.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:54 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:54.968+0000 7f7ea735a080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-03T10:27:56.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:55 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:55.838+0000 7f7ea735a080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-03T10:27:56.243 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:56 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:56.000+0000 7f7ea735a080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-03T10:27:56.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:56 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:56.438+0000 7f7ea735a080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-03T10:27:58.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:57 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:57.847+0000 7f7ea735a080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-03T10:27:59.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:58 smithi101 ceph-mon[85126]: Active manager daemon x restarted 2022-07-03T10:27:59.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:58 smithi101 ceph-mon[85126]: Activating manager daemon x 2022-07-03T10:27:59.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:58 smithi101 ceph-mon[85126]: osdmap e2: 0 total, 0 up, 0 in 2022-07-03T10:27:59.243 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:27:58 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:27:58.762+0000 7f7ea735a080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-03T10:28:00.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: mgrmap e6: x(active, starting, since 0.0530851s) 2022-07-03T10:28:00.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:28:00.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-07-03T10:28:00.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-07-03T10:28:00.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-07-03T10:28:00.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-07-03T10:28:00.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: Manager daemon x is now available 2022-07-03T10:28:00.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:00.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:00.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:00.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:00.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:00.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:28:00.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:27:59 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:28:00.341 INFO:teuthology.orchestra.run.smithi101.stdout:mgr epoch 5 is available 2022-07-03T10:28:00.342 INFO:teuthology.orchestra.run.smithi101.stdout:Setting orchestrator backend to cephadm... 2022-07-03T10:28:01.280 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: [03/Jul/2022:10:27:59] ENGINE Bus STARTING 2022-07-03T10:28:01.281 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: mgrmap e7: x(active, since 1.06021s) 2022-07-03T10:28:01.281 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: from='client.14126 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-07-03T10:28:01.281 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: from='client.14126 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-07-03T10:28:01.282 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: [03/Jul/2022:10:28:00] ENGINE Serving on https://172.21.15.101:7150 2022-07-03T10:28:01.282 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: [03/Jul/2022:10:28:00] ENGINE Bus STARTED 2022-07-03T10:28:01.282 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:01.282 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:00 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:02.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:02 smithi101 ceph-mon[85126]: mgrmap e8: x(active, since 2s) 2022-07-03T10:28:02.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:02 smithi101 ceph-mon[85126]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:02.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:02 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:02.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:02 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:02.749 INFO:teuthology.orchestra.run.smithi101.stdout:Generating ssh key... 2022-07-03T10:28:03.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-mon[85126]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:04.229 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: Generating public/private rsa key pair. 2022-07-03T10:28:04.230 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: Your identification has been saved in /tmp/tmpev48lik0/key. 2022-07-03T10:28:04.230 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: Your public key has been saved in /tmp/tmpev48lik0/key.pub. 2022-07-03T10:28:04.230 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: The key fingerprint is: 2022-07-03T10:28:04.231 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: SHA256:AMqYf7p/aEmG3qNMvRtWbKHw1iPRFewHvV9msoPC62w ceph-23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:28:04.231 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: The key's randomart image is: 2022-07-03T10:28:04.231 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: +---[RSA 3072]----+ 2022-07-03T10:28:04.231 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | . .oo | 2022-07-03T10:28:04.231 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | + . o .o . | 2022-07-03T10:28:04.232 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: |o + . +. . . | 2022-07-03T10:28:04.232 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | . o = o. o . + | 2022-07-03T10:28:04.232 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | ..* *.S. o * | 2022-07-03T10:28:04.232 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | .=o+ .o . + | 2022-07-03T10:28:04.232 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | .o++o o . | 2022-07-03T10:28:04.233 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | o.o*o..E | 2022-07-03T10:28:04.233 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: | ++++ oo | 2022-07-03T10:28:04.233 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: +----[SHA256]-----+ 2022-07-03T10:28:05.028 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:04 smithi101 ceph-mon[85126]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:05.028 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:04 smithi101 ceph-mon[85126]: Generating ssh key... 2022-07-03T10:28:05.028 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:04 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:05.029 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:04 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:05.353 INFO:teuthology.orchestra.run.smithi101.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-07-03T10:28:05.354 INFO:teuthology.orchestra.run.smithi101.stdout:Adding key to root@localhost authorized_keys... 2022-07-03T10:28:05.355 INFO:teuthology.orchestra.run.smithi101.stdout:Adding host smithi101... 2022-07-03T10:28:06.186 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:05 smithi101 ceph-mon[85126]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:07.811 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:07 smithi101 ceph-mon[85126]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi101", "addr": "172.21.15.101", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:08.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:08 smithi101 ceph-mon[85126]: Deploying cephadm binary to smithi101 2022-07-03T10:28:09.676 INFO:teuthology.orchestra.run.smithi101.stdout:Deploying unmanaged mon service... 2022-07-03T10:28:10.655 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:10 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:10.655 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:10 smithi101 ceph-mon[85126]: Added host smithi101 2022-07-03T10:28:10.656 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:10 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:10.883 INFO:teuthology.orchestra.run.smithi101.stdout:Deploying unmanaged mgr service... 2022-07-03T10:28:11.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:11 smithi101 ceph-mon[85126]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:11.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:11 smithi101 ceph-mon[85126]: Saving service mon spec with placement count:5 2022-07-03T10:28:11.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:11 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:12.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:12 smithi101 ceph-mon[85126]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:12.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:12 smithi101 ceph-mon[85126]: Saving service mgr spec with placement count:2 2022-07-03T10:28:12.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:12 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:12.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:12 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:14.403 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:14 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1120905069' entity='client.admin' 2022-07-03T10:28:14.404 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:14 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:14.721 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling the dashboard module... 2022-07-03T10:28:15.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:15 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1047701256' entity='client.admin' 2022-07-03T10:28:16.617 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:16 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1644071349' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-07-03T10:28:16.617 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:16 smithi101 ceph-mon[85126]: from='mgr.14122 172.21.15.101:0/3167011840' entity='mgr.x' 2022-07-03T10:28:16.881 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:16 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: ignoring --setuser ceph since I am not root 2022-07-03T10:28:16.881 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:16 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: ignoring --setgroup ceph since I am not root 2022-07-03T10:28:16.882 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:16 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:16.810+0000 7fe12de08080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-03T10:28:17.522 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1644071349' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-07-03T10:28:17.522 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-mon[85126]: mgrmap e9: x(active, since 17s) 2022-07-03T10:28:17.523 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:17.253+0000 7fe12de08080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-03T10:28:17.523 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:17.519+0000 7fe12de08080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-03T10:28:17.804 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:17.677+0000 7fe12de08080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-03T10:28:18.110 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:17.802+0000 7fe12de08080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-03T10:28:18.110 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:17 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:17.983+0000 7fe12de08080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-03T10:28:18.263 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for the mgr to restart... 2022-07-03T10:28:18.264 INFO:teuthology.orchestra.run.smithi101.stdout:Waiting for mgr epoch 9... 2022-07-03T10:28:18.371 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:18 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:18.107+0000 7fe12de08080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-03T10:28:18.372 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:18 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:18.302+0000 7fe12de08080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-03T10:28:18.628 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:18 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4208029977' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-07-03T10:28:18.628 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:18 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:18.449+0000 7fe12de08080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-03T10:28:18.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:18 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:18.728+0000 7fe12de08080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-03T10:28:19.492 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:19 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:19.041+0000 7fe12de08080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:28:20.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:19 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:19.871+0000 7fe12de08080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-03T10:28:20.243 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:20 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:20.008+0000 7fe12de08080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-03T10:28:20.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:20 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:20.245+0000 7fe12de08080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-03T10:28:20.743 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:20 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:20.389+0000 7fe12de08080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-03T10:28:21.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:20 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:20.946+0000 7fe12de08080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:28:21.821 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:21 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:21.530+0000 7fe12de08080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-03T10:28:21.822 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:21 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:21.675+0000 7fe12de08080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-03T10:28:22.242 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:21 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:21.819+0000 7fe12de08080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-03T10:28:22.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:22 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:22.692+0000 7fe12de08080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-03T10:28:22.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:22 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:22.853+0000 7fe12de08080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-03T10:28:23.742 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:23 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:23.294+0000 7fe12de08080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-03T10:28:24.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:24 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:24.705+0000 7fe12de08080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-03T10:28:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:25 smithi101 ceph-mon[85126]: Active manager daemon x restarted 2022-07-03T10:28:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:25 smithi101 ceph-mon[85126]: Activating manager daemon x 2022-07-03T10:28:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:25 smithi101 ceph-mon[85126]: osdmap e3: 0 total, 0 up, 0 in 2022-07-03T10:28:25.994 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:28:25 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:28:25.626+0000 7fe12de08080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-03T10:28:27.235 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: mgrmap e10: x(active, starting, since 0.0555777s) 2022-07-03T10:28:27.236 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:28:27.236 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-07-03T10:28:27.236 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-07-03T10:28:27.237 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-07-03T10:28:27.237 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-07-03T10:28:27.237 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: Manager daemon x is now available 2022-07-03T10:28:27.237 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:27.237 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:27.238 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:28:27.238 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:28:27.297 INFO:teuthology.orchestra.run.smithi101.stdout:mgr epoch 9 is available 2022-07-03T10:28:27.297 INFO:teuthology.orchestra.run.smithi101.stdout:Generating a dashboard self-signed certificate... 2022-07-03T10:28:28.159 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:27 smithi101 ceph-mon[85126]: mgrmap e11: x(active, since 1.06621s) 2022-07-03T10:28:28.160 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:27 smithi101 ceph-mon[85126]: [03/Jul/2022:10:28:26] ENGINE Bus STARTING 2022-07-03T10:28:28.160 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:27 smithi101 ceph-mon[85126]: [03/Jul/2022:10:28:27] ENGINE Serving on https://172.21.15.101:7150 2022-07-03T10:28:28.160 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:27 smithi101 ceph-mon[85126]: [03/Jul/2022:10:28:27] ENGINE Bus STARTED 2022-07-03T10:28:28.160 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:28.900 INFO:teuthology.orchestra.run.smithi101.stdout:Creating initial admin user... 2022-07-03T10:28:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:29 smithi101 ceph-mon[85126]: mgrmap e12: x(active, since 2s) 2022-07-03T10:28:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:29 smithi101 ceph-mon[85126]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:29.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:30.309 INFO:teuthology.orchestra.run.smithi101.stdout:Fetching dashboard port number... 2022-07-03T10:28:31.175 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:30 smithi101 ceph-mon[85126]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:31.176 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:30 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:31.176 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:30 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:31.681 INFO:teuthology.orchestra.run.smithi101.stdout:Ceph Dashboard is now available at: 2022-07-03T10:28:31.681 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:31.682 INFO:teuthology.orchestra.run.smithi101.stdout: URL: https://smithi101.front.sepia.ceph.com:8443/ 2022-07-03T10:28:31.682 INFO:teuthology.orchestra.run.smithi101.stdout: User: admin 2022-07-03T10:28:31.682 INFO:teuthology.orchestra.run.smithi101.stdout: Password: hs8fnde6k9 2022-07-03T10:28:31.682 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:31.683 INFO:teuthology.orchestra.run.smithi101.stdout:Saving cluster configuration to /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config directory 2022-07-03T10:28:31.683 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling autotune for osd_memory_target 2022-07-03T10:28:32.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:31 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3679508653' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-07-03T10:28:33.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:33 smithi101 ceph-mon[85126]: mgrmap e13: x(active, since 6s) 2022-07-03T10:28:34.388 INFO:teuthology.orchestra.run.smithi101.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2022-07-03T10:28:34.388 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.388 INFO:teuthology.orchestra.run.smithi101.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-07-03T10:28:34.388 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.389 INFO:teuthology.orchestra.run.smithi101.stdout:Or, if you are only running a single cluster on this host: 2022-07-03T10:28:34.389 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.389 INFO:teuthology.orchestra.run.smithi101.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2022-07-03T10:28:34.389 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.389 INFO:teuthology.orchestra.run.smithi101.stdout:Please consider enabling telemetry to help improve Ceph: 2022-07-03T10:28:34.390 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.390 INFO:teuthology.orchestra.run.smithi101.stdout: ceph telemetry on 2022-07-03T10:28:34.390 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.390 INFO:teuthology.orchestra.run.smithi101.stdout:For more information see: 2022-07-03T10:28:34.391 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.391 INFO:teuthology.orchestra.run.smithi101.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2022-07-03T10:28:34.391 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:34.391 INFO:teuthology.orchestra.run.smithi101.stdout:Bootstrap complete. 2022-07-03T10:28:34.467 INFO:tasks.cephadm:Fetching config... 2022-07-03T10:28:34.468 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:28:34.468 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-07-03T10:28:34.568 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-07-03T10:28:34.569 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:28:34.569 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-07-03T10:28:34.627 INFO:tasks.cephadm:Fetching mon keyring... 2022-07-03T10:28:34.627 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:28:34.628 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-07-03T10:28:34.723 INFO:tasks.cephadm:Fetching pub ssh key... 2022-07-03T10:28:34.723 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:28:34.723 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-07-03T10:28:34.783 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-07-03T10:28:34.784 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-07-03T10:28:34.905 INFO:teuthology.orchestra.run.smithi101.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:28:34.942 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-07-03T10:28:34.988 INFO:teuthology.orchestra.run.smithi161.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:28:35.000 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-07-03T10:28:35.045 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDigfG8bpt2rU5Pafx+aq7PhJraX2jmZuMcqXQbyoDlL/kPsK78cS1N9hnST+QQlpQ6UE6rSjFi0JQoC+1ixI2acqZHz1mBdPdJbb8zjFjGvFYJaYoozi3RhRphs2PSZldrKzJTcqodWGQgJe77bx8cboBeVdr5BHgJkdF20OBNdcCFXyQYJ8PuzWvCZvDRa8my1orNLAxO7KCqDiiOsM8iVsKZ/70g8Kz0AUSYiDqrkzPgL/9vG6lm/uwad9mUHvIl/IxgA4ZBcLjYuFoBTuZDLbNhPAFyNefQ4ETJXoUjlpbb2ngGnKcfITW9yF+mGOr7ui9BMon0rHYPb4ZdYM1J7AKrHUCgRf64VKrHBo97za7+274RVCOV/DeYRcjUvJY02R8gt1kL6+tG5gQY+ukteJ2flpL7ddRQV0ksWP4LrAKBBsppK9ql8ixiUFi9drYBh6YJiPCQ2ScO1YMbocps41alLExr4NZXvhQn65HcwCo2fVrhrvkskeXG6DOHts= ceph-23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:28:35.058 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-07-03T10:28:35.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:34 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1765229922' entity='client.admin' 2022-07-03T10:28:35.955 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:37.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 -- 172.21.15.101:0/3996808990 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 msgr2=0x7f712c0fd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:37.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 --2- 172.21.15.101:0/3996808990 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c0fd2c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f711c009800 tx=0x7f711c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:37.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 -- 172.21.15.101:0/3996808990 shutdown_connections 2022-07-03T10:28:37.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 --2- 172.21.15.101:0/3996808990 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c0fd2c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:37.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 -- 172.21.15.101:0/3996808990 >> 172.21.15.101:0/3996808990 conn(0x7f712c0f8610 msgr2=0x7f712c0faa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:37.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.513+0000 7f712b59e700 1 -- 172.21.15.101:0/3996808990 shutdown_connections 2022-07-03T10:28:37.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.514+0000 7f712b59e700 1 -- 172.21.15.101:0/3996808990 wait complete. 2022-07-03T10:28:37.520 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.514+0000 7f712b59e700 1 Processor -- start 2022-07-03T10:28:37.520 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712b59e700 1 -- start start 2022-07-03T10:28:37.520 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712b59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:37.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712b59e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c10f980 con 0x7f712c0fce90 2022-07-03T10:28:37.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712a59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:37.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712a59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:48204/0 (socket says 172.21.15.101:48204) 2022-07-03T10:28:37.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.515+0000 7f712a59c700 1 -- 172.21.15.101:0/2854643134 learned_addr learned my addr 172.21.15.101:0/2854643134 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:37.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f712a59c700 1 -- 172.21.15.101:0/2854643134 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711c009440 con 0x7f712c0fce90 2022-07-03T10:28:37.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f712a59c700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f711c000c00 tx=0x7f711c0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:37.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f711c01c070 con 0x7f712c0fce90 2022-07-03T10:28:37.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f711c00bd00 con 0x7f712c0fce90 2022-07-03T10:28:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f711c021c90 con 0x7f712c0fce90 2022-07-03T10:28:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.516+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c10fb80 con 0x7f712c0fce90 2022-07-03T10:28:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.517+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c110800 con 0x7f712c0fce90 2022-07-03T10:28:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.517+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f711c028030 con 0x7f712c0fce90 2022-07-03T10:28:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.517+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c05fb50 con 0x7f712c0fce90 2022-07-03T10:28:37.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.518+0000 7f711affd700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 0x7f7114037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:37.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.518+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f711c047c50 con 0x7f712c0fce90 2022-07-03T10:28:37.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.518+0000 7f7129d9b700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 0x7f7114037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:37.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.519+0000 7f7129d9b700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 0x7f7114037140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71200097c0 tx=0x7f7120006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:37.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.523+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f711c005ab0 con 0x7f712c0fce90 2022-07-03T10:28:37.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.708+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f712c04c470 con 0x7f712c0fce90 2022-07-03T10:28:37.715 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.713+0000 7f711affd700 1 -- 172.21.15.101:0/2854643134 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f711c026070 con 0x7f712c0fce90 2022-07-03T10:28:37.718 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 msgr2=0x7f7114037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:37.718 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 0x7f7114037140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71200097c0 tx=0x7f7120006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:37.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 msgr2=0x7f712c10f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:37.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f711c000c00 tx=0x7f711c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:37.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 shutdown_connections 2022-07-03T10:28:37.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7114034c90 0x7f7114037140 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:37.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 --2- 172.21.15.101:0/2854643134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f712c0fce90 0x7f712c10f440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:37.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 >> 172.21.15.101:0/2854643134 conn(0x7f712c0f8610 msgr2=0x7f712c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:37.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 shutdown_connections 2022-07-03T10:28:37.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:37.716+0000 7f712b59e700 1 -- 172.21.15.101:0/2854643134 wait complete. 2022-07-03T10:28:38.093 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-07-03T10:28:38.093 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-07-03T10:28:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi101", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:28:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2854643134' entity='client.admin' 2022-07-03T10:28:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:38.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:38.977 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:40.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 -- 172.21.15.101:0/3514050134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 msgr2=0x7ff7e40fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:40.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 --2- 172.21.15.101:0/3514050134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e40fd4d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009800 tx=0x7ff7e00091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:40.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 -- 172.21.15.101:0/3514050134 shutdown_connections 2022-07-03T10:28:40.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 --2- 172.21.15.101:0/3514050134 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e40fd4d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:40.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 -- 172.21.15.101:0/3514050134 >> 172.21.15.101:0/3514050134 conn(0x7ff7e40f87b0 msgr2=0x7ff7e40fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:40.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 -- 172.21.15.101:0/3514050134 shutdown_connections 2022-07-03T10:28:40.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.473+0000 7ff7eb622700 1 -- 172.21.15.101:0/3514050134 wait complete. 2022-07-03T10:28:40.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.474+0000 7ff7eb622700 1 Processor -- start 2022-07-03T10:28:40.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.474+0000 7ff7eb622700 1 -- start start 2022-07-03T10:28:40.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7eb622700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:40.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7eb622700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e4070870 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7e93be700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:40.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7e93be700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42084/0 (socket says 172.21.15.101:42084) 2022-07-03T10:28:40.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7e93be700 1 -- 172.21.15.101:0/428120295 learned_addr learned my addr 172.21.15.101:0/428120295 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:40.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.475+0000 7ff7e93be700 1 -- 172.21.15.101:0/428120295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0009440 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7e93be700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0000c00 tx=0x7ff7e00091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:40.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7e001c070 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff7e000bd00 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4070a70 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7e0021c90 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.476+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e4070f20 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.477+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7ff7e0028030 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.477+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e404c470 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.478+0000 7ff7d9ffb700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 0x7ff7d0037190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:40.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.478+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff7e0047be0 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.478+0000 7ff7e8bbd700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 0x7ff7d0037190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:40.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.479+0000 7ff7e8bbd700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 0x7ff7d0037190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40097c0 tx=0x7ff7d4006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:40.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.483+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7ff7e0021470 con 0x7ff7e40fd0c0 2022-07-03T10:28:40.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.669+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7e40711f0 con 0x7ff7d0034ce0 2022-07-03T10:28:40.674 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.673+0000 7ff7d9ffb700 1 -- 172.21.15.101:0/428120295 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff7e40711f0 con 0x7ff7d0034ce0 2022-07-03T10:28:40.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 msgr2=0x7ff7d0037190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:40.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 0x7ff7d0037190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff7d40097c0 tx=0x7ff7d4006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:40.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 msgr2=0x7ff7e4072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:40.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0000c00 tx=0x7ff7e00091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:40.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 shutdown_connections 2022-07-03T10:28:40.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7d0034ce0 0x7ff7d0037190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:40.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 --2- 172.21.15.101:0/428120295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd0c0 0x7ff7e4072410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:40.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 >> 172.21.15.101:0/428120295 conn(0x7ff7e40f87b0 msgr2=0x7ff7e40f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:40.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 shutdown_connections 2022-07-03T10:28:40.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:40.675+0000 7ff7eb622700 1 -- 172.21.15.101:0/428120295 wait complete. 2022-07-03T10:28:41.151 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi161 2022-07-03T10:28:41.151 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:28:41.152 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.conf 2022-07-03T10:28:41.173 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:28:41.173 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:28:41.233 INFO:tasks.cephadm:Adding host smithi161 to orchestrator... 2022-07-03T10:28:41.233 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch host add smithi161 2022-07-03T10:28:41.909 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:41.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:41.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:41.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:28:41.911 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:28:41.911 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:28:41.911 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:41 smithi101 ceph-mon[85126]: Updating smithi101:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:28:42.147 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:42.758 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:42 smithi101 ceph-mon[85126]: Updating smithi101:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:28:43.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/3823360198 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 msgr2=0x7fdb38065610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:43.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/3823360198 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38065610 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009800 tx=0x7fdb2c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:43.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/3823360198 shutdown_connections 2022-07-03T10:28:43.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/3823360198 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38065610 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:43.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/3823360198 >> 172.21.15.101:0/3823360198 conn(0x7fdb38072f40 msgr2=0x7fdb38073340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:43.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/3823360198 shutdown_connections 2022-07-03T10:28:43.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.687+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/3823360198 wait complete. 2022-07-03T10:28:43.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.688+0000 7fdb3eb4e700 1 Processor -- start 2022-07-03T10:28:43.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.688+0000 7fdb3eb4e700 1 -- start start 2022-07-03T10:28:43.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3eb4e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:43.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3eb4e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3811a120 con 0x7fdb38101f50 2022-07-03T10:28:43.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3c8ea700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:43.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3c8ea700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42102/0 (socket says 172.21.15.101:42102) 2022-07-03T10:28:43.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3c8ea700 1 -- 172.21.15.101:0/1990497162 learned_addr learned my addr 172.21.15.101:0/1990497162 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:43.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.689+0000 7fdb3c8ea700 1 -- 172.21.15.101:0/1990497162 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c009440 con 0x7fdb38101f50 2022-07-03T10:28:43.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb3c8ea700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0051f0 tx=0x7fdb2c0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:43.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c01c070 con 0x7fdb38101f50 2022-07-03T10:28:43.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdb2c00bd50 con 0x7fdb38101f50 2022-07-03T10:28:43.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3811a320 con 0x7fdb38101f50 2022-07-03T10:28:43.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb2c021ca0 con 0x7fdb38101f50 2022-07-03T10:28:43.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.690+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3811a750 con 0x7fdb38101f50 2022-07-03T10:28:43.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.691+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3805fb50 con 0x7fdb38101f50 2022-07-03T10:28:43.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.692+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fdb2c028030 con 0x7fdb38101f50 2022-07-03T10:28:43.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.692+0000 7fdb357fa700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 0x7fdb28037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:43.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.692+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdb2c047c50 con 0x7fdb38101f50 2022-07-03T10:28:43.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.693+0000 7fdb37fff700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 0x7fdb28037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:43.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.693+0000 7fdb37fff700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 0x7fdb28037140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdb240097c0 tx=0x7fdb24006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:43.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.698+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fdb2c02a980 con 0x7fdb38101f50 2022-07-03T10:28:43.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:43.914+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb3804c4a0 con 0x7fdb28034c90 2022-07-03T10:28:43.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:43 smithi101 ceph-mon[85126]: Updating smithi101:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:28:45.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:45 smithi101 ceph-mon[85126]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi161", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:45.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:45.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:45.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:45.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:45 smithi101 ceph-mon[85126]: Deploying cephadm binary to smithi161 2022-07-03T10:28:45.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.648+0000 7fdb357fa700 1 -- 172.21.15.101:0/1990497162 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdb3804c4a0 con 0x7fdb28034c90 2022-07-03T10:28:45.651 INFO:teuthology.orchestra.run.smithi101.stdout:Added host 'smithi161' with addr '172.21.15.161' 2022-07-03T10:28:45.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.650+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 msgr2=0x7fdb28037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:45.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.650+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 0x7fdb28037140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdb240097c0 tx=0x7fdb24006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:45.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.650+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 msgr2=0x7fdb38119be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:45.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.650+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0051f0 tx=0x7fdb2c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:45.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 shutdown_connections 2022-07-03T10:28:45.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdb28034c90 0x7fdb28037140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:45.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 --2- 172.21.15.101:0/1990497162 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdb38101f50 0x7fdb38119be0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:45.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 >> 172.21.15.101:0/1990497162 conn(0x7fdb38072f40 msgr2=0x7fdb380fad40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:45.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 shutdown_connections 2022-07-03T10:28:45.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:45.651+0000 7fdb3eb4e700 1 -- 172.21.15.101:0/1990497162 wait complete. 2022-07-03T10:28:46.039 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch host ls --format=json 2022-07-03T10:28:46.920 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:46.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:28:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:28:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:47.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:47 smithi101 ceph-mon[85126]: Added host smithi161 2022-07-03T10:28:47.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:47 smithi101 ceph-mon[85126]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:47.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:47.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:48.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 -- 172.21.15.101:0/3207489228 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 msgr2=0x7f957c1034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:48.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 --2- 172.21.15.101:0/3207489228 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1034e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f956c009800 tx=0x7f956c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:48.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 -- 172.21.15.101:0/3207489228 shutdown_connections 2022-07-03T10:28:48.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 --2- 172.21.15.101:0/3207489228 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1034e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:48.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 -- 172.21.15.101:0/3207489228 >> 172.21.15.101:0/3207489228 conn(0x7f957c072f40 msgr2=0x7f957c073340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:48.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 -- 172.21.15.101:0/3207489228 shutdown_connections 2022-07-03T10:28:48.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.417+0000 7f9581c59700 1 -- 172.21.15.101:0/3207489228 wait complete. 2022-07-03T10:28:48.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.418+0000 7f9581c59700 1 Processor -- start 2022-07-03T10:28:48.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.418+0000 7f9581c59700 1 -- start start 2022-07-03T10:28:48.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.419+0000 7f9581c59700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:48.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.419+0000 7f9581c59700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c115c10 con 0x7f957c103110 2022-07-03T10:28:48.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.419+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:48.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.419+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:48380/0 (socket says 172.21.15.101:48380) 2022-07-03T10:28:48.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.419+0000 7f957b7fe700 1 -- 172.21.15.101:0/533391666 learned_addr learned my addr 172.21.15.101:0/533391666 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:48.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f957b7fe700 1 -- 172.21.15.101:0/533391666 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c009440 con 0x7f957c103110 2022-07-03T10:28:48.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f957b7fe700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f956c005220 tx=0x7f956c0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:48.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f956c01c070 con 0x7f957c103110 2022-07-03T10:28:48.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f956c00bd50 con 0x7f957c103110 2022-07-03T10:28:48.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c115e10 con 0x7f957c103110 2022-07-03T10:28:48.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f956c021c60 con 0x7f957c103110 2022-07-03T10:28:48.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.420+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c116240 con 0x7f957c103110 2022-07-03T10:28:48.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.421+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c05fb50 con 0x7f957c103110 2022-07-03T10:28:48.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.421+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f956c028030 con 0x7f957c103110 2022-07-03T10:28:48.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.422+0000 7f9563fff700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 0x7f9564037190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:48.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.422+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f956c047bc0 con 0x7f957c103110 2022-07-03T10:28:48.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.422+0000 7f957affd700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 0x7f9564037190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:48.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.423+0000 7f957affd700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 0x7f9564037190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95700097c0 tx=0x7f9570006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:48.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.427+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f956c005ab0 con 0x7f957c103110 2022-07-03T10:28:48.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.615+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f957c110da0 con 0x7f9564034ce0 2022-07-03T10:28:48.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.616+0000 7f9563fff700 1 -- 172.21.15.101:0/533391666 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f957c110da0 con 0x7f9564034ce0 2022-07-03T10:28:48.618 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:48.618 INFO:teuthology.orchestra.run.smithi101.stdout:[{"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}] 2022-07-03T10:28:48.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 msgr2=0x7f9564037190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:48.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 0x7f9564037190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f95700097c0 tx=0x7f9570006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:48.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 msgr2=0x7f957c1156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:48.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f956c005220 tx=0x7f956c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:48.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 shutdown_connections 2022-07-03T10:28:48.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9564034ce0 0x7f9564037190 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:48.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 --2- 172.21.15.101:0/533391666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f957c103110 0x7f957c1156d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:48.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 >> 172.21.15.101:0/533391666 conn(0x7f957c072f40 msgr2=0x7f957c104710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:48.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 shutdown_connections 2022-07-03T10:28:48.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:48.619+0000 7f9581c59700 1 -- 172.21.15.101:0/533391666 wait complete. 2022-07-03T10:28:48.973 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:48 smithi101 ceph-mon[85126]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:48.974 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:48 smithi101 ceph-mon[85126]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:28:48.994 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2022-07-03T10:28:48.994 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:28:48.995 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2022-07-03T10:28:49.015 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:28:49.016 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:28:49.075 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2022-07-03T10:28:49.075 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch host add smithi188 2022-07-03T10:28:49.951 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:51.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 -- 172.21.15.101:0/1556048419 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 msgr2=0x7f0dc81034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:51.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/1556048419 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81034e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009800 tx=0x7f0dc40091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:51.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 -- 172.21.15.101:0/1556048419 shutdown_connections 2022-07-03T10:28:51.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/1556048419 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81034e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:51.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 -- 172.21.15.101:0/1556048419 >> 172.21.15.101:0/1556048419 conn(0x7f0dc8072f40 msgr2=0x7f0dc8073340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:51.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 -- 172.21.15.101:0/1556048419 shutdown_connections 2022-07-03T10:28:51.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.437+0000 7f0dcf52e700 1 -- 172.21.15.101:0/1556048419 wait complete. 2022-07-03T10:28:51.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.438+0000 7f0dcf52e700 1 Processor -- start 2022-07-03T10:28:51.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.438+0000 7f0dcf52e700 1 -- start start 2022-07-03T10:28:51.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.438+0000 7f0dcf52e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:51.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.438+0000 7f0dcf52e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc8115bf0 con 0x7f0dc8103110 2022-07-03T10:28:51.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.439+0000 7f0dcd2ca700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:51.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.439+0000 7f0dcd2ca700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:48390/0 (socket says 172.21.15.101:48390) 2022-07-03T10:28:51.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.439+0000 7f0dcd2ca700 1 -- 172.21.15.101:0/3174746235 learned_addr learned my addr 172.21.15.101:0/3174746235 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:51.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.439+0000 7f0dcd2ca700 1 -- 172.21.15.101:0/3174746235 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4009440 con 0x7f0dc8103110 2022-07-03T10:28:51.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dcd2ca700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40051f0 tx=0x7f0dc40091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:51.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc401c070 con 0x7f0dc8103110 2022-07-03T10:28:51.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0dc400bd50 con 0x7f0dc8103110 2022-07-03T10:28:51.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc8115df0 con 0x7f0dc8103110 2022-07-03T10:28:51.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc4021ca0 con 0x7f0dc8103110 2022-07-03T10:28:51.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.440+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc81162a0 con 0x7f0dc8103110 2022-07-03T10:28:51.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.441+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f0dc4028030 con 0x7f0dc8103110 2022-07-03T10:28:51.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.441+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc805fb50 con 0x7f0dc8103110 2022-07-03T10:28:51.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.442+0000 7f0dbdffb700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 0x7f0db4037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:51.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.442+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0dc4047d80 con 0x7f0dc8103110 2022-07-03T10:28:51.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.442+0000 7f0dccac9700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 0x7f0db4037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:51.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.442+0000 7f0dccac9700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 0x7f0db4037140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0db80097c0 tx=0x7f0db8006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:51.448 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.446+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f0dc402a590 con 0x7f0dc8103110 2022-07-03T10:28:51.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:51.640+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dc8100260 con 0x7f0db4034c90 2022-07-03T10:28:51.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:51.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:51.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:51.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:51.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:51.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:28:51.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:52.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:52 smithi101 ceph-mon[85126]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:28:53.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.574+0000 7f0dbdffb700 1 -- 172.21.15.101:0/3174746235 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f0dc8100260 con 0x7f0db4034c90 2022-07-03T10:28:53.575 INFO:teuthology.orchestra.run.smithi101.stdout:Added host 'smithi188' with addr '172.21.15.188' 2022-07-03T10:28:53.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 msgr2=0x7f0db4037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:53.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 0x7f0db4037140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0db80097c0 tx=0x7f0db8006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:53.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 msgr2=0x7f0dc81156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:53.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0dc40051f0 tx=0x7f0dc40091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:53.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 shutdown_connections 2022-07-03T10:28:53.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0db4034c90 0x7f0db4037140 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:53.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 --2- 172.21.15.101:0/3174746235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0dc8103110 0x7f0dc81156b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:53.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.576+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 >> 172.21.15.101:0/3174746235 conn(0x7f0dc8072f40 msgr2=0x7f0dc8104710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:53.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.577+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 shutdown_connections 2022-07-03T10:28:53.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:53.577+0000 7f0dcf52e700 1 -- 172.21.15.101:0/3174746235 wait complete. 2022-07-03T10:28:53.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:53 smithi101 ceph-mon[85126]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:53.942 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:53 smithi101 ceph-mon[85126]: Deploying cephadm binary to smithi188 2022-07-03T10:28:54.031 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch host ls --format=json 2022-07-03T10:28:54.926 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:54.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: Added host smithi188 2022-07-03T10:28:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:28:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:28:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:28:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:28:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:55.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:55.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:28:55.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:28:55.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:28:55.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:55.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:55.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:56.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 -- 172.21.15.101:0/4140070540 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 msgr2=0x7f98d8103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:56.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 --2- 172.21.15.101:0/4140070540 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8103490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009800 tx=0x7f98c80091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:56.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 -- 172.21.15.101:0/4140070540 shutdown_connections 2022-07-03T10:28:56.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 --2- 172.21.15.101:0/4140070540 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8103490 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:56.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 -- 172.21.15.101:0/4140070540 >> 172.21.15.101:0/4140070540 conn(0x7f98d80f8ca0 msgr2=0x7f98d80fb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:56.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 -- 172.21.15.101:0/4140070540 shutdown_connections 2022-07-03T10:28:56.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.325+0000 7f98de227700 1 -- 172.21.15.101:0/4140070540 wait complete. 2022-07-03T10:28:56.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.326+0000 7f98de227700 1 Processor -- start 2022-07-03T10:28:56.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.326+0000 7f98de227700 1 -- start start 2022-07-03T10:28:56.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98de227700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:56.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98de227700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8070d60 con 0x7f98d81030c0 2022-07-03T10:28:56.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98d77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:56.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98d77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:48414/0 (socket says 172.21.15.101:48414) 2022-07-03T10:28:56.336 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98d77fe700 1 -- 172.21.15.101:0/2667223278 learned_addr learned my addr 172.21.15.101:0/2667223278 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:56.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.327+0000 7f98d77fe700 1 -- 172.21.15.101:0/2667223278 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8009440 con 0x7f98d81030c0 2022-07-03T10:28:56.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98d77fe700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005220 tx=0x7f98c80091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:56.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c801c070 con 0x7f98d81030c0 2022-07-03T10:28:56.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f98c800bd50 con 0x7f98d81030c0 2022-07-03T10:28:56.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d8070f60 con 0x7f98d81030c0 2022-07-03T10:28:56.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c8021c60 con 0x7f98d81030c0 2022-07-03T10:28:56.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.328+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d8071390 con 0x7f98d81030c0 2022-07-03T10:28:56.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.329+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f98c8028030 con 0x7f98d81030c0 2022-07-03T10:28:56.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.329+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d805fb50 con 0x7f98d81030c0 2022-07-03T10:28:56.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.330+0000 7f98bffff700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 0x7f98c0037190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:56.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.330+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f98c8047c00 con 0x7f98d81030c0 2022-07-03T10:28:56.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.330+0000 7f98d6ffd700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 0x7f98c0037190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:56.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.331+0000 7f98d6ffd700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 0x7f98c0037190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097c0 tx=0x7f98cc006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:56.342 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.335+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f98c802a590 con 0x7f98d81030c0 2022-07-03T10:28:56.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.524+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98d8071f60 con 0x7f98c0034ce0 2022-07-03T10:28:56.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.525+0000 7f98bffff700 1 -- 172.21.15.101:0/2667223278 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f98d8071f60 con 0x7f98c0034ce0 2022-07-03T10:28:56.528 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:28:56.528 INFO:teuthology.orchestra.run.smithi101.stdout:[{"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}, {"addr": "172.21.15.161", "hostname": "smithi161", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2022-07-03T10:28:56.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 msgr2=0x7f98c0037190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:56.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 0x7f98c0037190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097c0 tx=0x7f98cc006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:56.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 msgr2=0x7f98d8070820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:56.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98c8005220 tx=0x7f98c80091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:56.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 shutdown_connections 2022-07-03T10:28:56.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f98c0034ce0 0x7f98c0037190 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:56.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 --2- 172.21.15.101:0/2667223278 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f98d81030c0 0x7f98d8070820 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:56.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 >> 172.21.15.101:0/2667223278 conn(0x7f98d80f8ca0 msgr2=0x7f98d80fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:56.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 shutdown_connections 2022-07-03T10:28:56.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:56.528+0000 7f98de227700 1 -- 172.21.15.101:0/2667223278 wait complete. 2022-07-03T10:28:57.001 INFO:tasks.cephadm:Setting crush tunables to default 2022-07-03T10:28:57.002 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd crush tunables default 2022-07-03T10:28:57.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:56 smithi101 ceph-mon[85126]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:57.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:57.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:56 smithi101 ceph-mon[85126]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:28:57.880 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:28:59.174 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:28:59.174 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:59.174 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:59.175 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:59.175 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:59.175 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:28:59.175 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:28:59.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 -- 172.21.15.101:0/3443941372 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 msgr2=0x7f26c4103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:59.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 --2- 172.21.15.101:0/3443941372 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4103490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009800 tx=0x7f26b40091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:59.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 -- 172.21.15.101:0/3443941372 shutdown_connections 2022-07-03T10:28:59.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 --2- 172.21.15.101:0/3443941372 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4103490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:59.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 -- 172.21.15.101:0/3443941372 >> 172.21.15.101:0/3443941372 conn(0x7f26c40f8ca0 msgr2=0x7f26c40fb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:59.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 -- 172.21.15.101:0/3443941372 shutdown_connections 2022-07-03T10:28:59.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.386+0000 7f26c98a2700 1 -- 172.21.15.101:0/3443941372 wait complete. 2022-07-03T10:28:59.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.387+0000 7f26c98a2700 1 Processor -- start 2022-07-03T10:28:59.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.387+0000 7f26c98a2700 1 -- start start 2022-07-03T10:28:59.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c98a2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:59.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c98a2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c410bc30 con 0x7f26c41030c0 2022-07-03T10:28:59.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c2ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:59.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c2ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:54184/0 (socket says 172.21.15.101:54184) 2022-07-03T10:28:59.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c2ffd700 1 -- 172.21.15.101:0/1523588512 learned_addr learned my addr 172.21.15.101:0/1523588512 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:28:59.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.388+0000 7f26c2ffd700 1 -- 172.21.15.101:0/1523588512 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4009440 con 0x7f26c41030c0 2022-07-03T10:28:59.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26c2ffd700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26b4005220 tx=0x7f26b40091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:59.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26b401c070 con 0x7f26c41030c0 2022-07-03T10:28:59.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f26b400bd50 con 0x7f26c41030c0 2022-07-03T10:28:59.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4108010 con 0x7f26c41030c0 2022-07-03T10:28:59.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26b4021c60 con 0x7f26c41030c0 2022-07-03T10:28:59.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.389+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c4108520 con 0x7f26c41030c0 2022-07-03T10:28:59.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.390+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f26b4028030 con 0x7f26c41030c0 2022-07-03T10:28:59.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.390+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c405fb50 con 0x7f26c41030c0 2022-07-03T10:28:59.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.391+0000 7f26ab7fe700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 0x7f26ac037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:28:59.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.391+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f26b4047bc0 con 0x7f26c41030c0 2022-07-03T10:28:59.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.391+0000 7f26c27fc700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 0x7f26ac037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:28:59.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.392+0000 7f26c27fc700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 0x7f26ac037140 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26b80097c0 tx=0x7f26b8006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:28:59.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.396+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f26b4005ab0 con 0x7f26c41030c0 2022-07-03T10:28:59.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.589+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f26c4107270 con 0x7f26c41030c0 2022-07-03T10:28:59.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.921+0000 7f26ab7fe700 1 -- 172.21.15.101:0/1523588512 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f26b4026030 con 0x7f26c41030c0 2022-07-03T10:28:59.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 msgr2=0x7f26ac037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:59.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 0x7f26ac037140 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26b80097c0 tx=0x7f26b8006d00 comp rx=0 tx=0).stop 2022-07-03T10:28:59.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 msgr2=0x7f26c4107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:28:59.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f26b4005220 tx=0x7f26b40091e0 comp rx=0 tx=0).stop 2022-07-03T10:28:59.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 shutdown_connections 2022-07-03T10:28:59.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f26ac034c90 0x7f26ac037140 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:59.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 --2- 172.21.15.101:0/1523588512 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26c41030c0 0x7f26c4107ad0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:28:59.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 >> 172.21.15.101:0/1523588512 conn(0x7f26c40f8ca0 msgr2=0x7f26c40721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:28:59.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 shutdown_connections 2022-07-03T10:28:59.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:28:59.924+0000 7f26c98a2700 1 -- 172.21.15.101:0/1523588512 wait complete. 2022-07-03T10:28:59.929 INFO:teuthology.orchestra.run.smithi101.stderr:adjusted tunables profile to default 2022-07-03T10:29:00.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:28:59 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-07-03T10:29:00.353 INFO:tasks.cephadm:Adding mon.a on smithi101 2022-07-03T10:29:00.354 INFO:tasks.cephadm:Adding mon.b on smithi161 2022-07-03T10:29:00.354 INFO:tasks.cephadm:Adding mon.c on smithi188 2022-07-03T10:29:00.354 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch apply mon '3;smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c' 2022-07-03T10:29:00.982 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2022-07-03T10:29:01.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:00 smithi101 ceph-mon[85126]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:01.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:00 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-07-03T10:29:01.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:00 smithi101 ceph-mon[85126]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:01.276 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2022-07-03T10:29:02.299 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 -- 172.21.15.188:0/3469192350 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 msgr2=0x7f31800fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 --2- 172.21.15.188:0/3469192350 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f31800fc3c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3174009800 tx=0x7f31740091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 -- 172.21.15.188:0/3469192350 shutdown_connections 2022-07-03T10:29:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 --2- 172.21.15.188:0/3469192350 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f31800fc3c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 -- 172.21.15.188:0/3469192350 >> 172.21.15.188:0/3469192350 conn(0x7f31800f76c0 msgr2=0x7f31800f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 -- 172.21.15.188:0/3469192350 shutdown_connections 2022-07-03T10:29:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 -- 172.21.15.188:0/3469192350 wait complete. 2022-07-03T10:29:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.297+0000 7f3186349700 1 Processor -- start 2022-07-03T10:29:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f3186349700 1 -- start start 2022-07-03T10:29:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f3186349700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f3186349700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010fb60 con 0x7f31800fbfb0 2022-07-03T10:29:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f317ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f317ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.188:59802/0 (socket says 172.21.15.188:59802) 2022-07-03T10:29:02.303 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.298+0000 7f317ffff700 1 -- 172.21.15.188:0/3785218135 learned_addr learned my addr 172.21.15.188:0/3785218135 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:29:02.304 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f317ffff700 1 -- 172.21.15.188:0/3785218135 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3174009440 con 0x7f31800fbfb0 2022-07-03T10:29:02.304 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f317ffff700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3174000c00 tx=0x7f31740091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f317401c070 con 0x7f31800fbfb0 2022-07-03T10:29:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318010fd60 con 0x7f31800fbfb0 2022-07-03T10:29:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f317400bd00 con 0x7f31800fbfb0 2022-07-03T10:29:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3174021c90 con 0x7f31800fbfb0 2022-07-03T10:29:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.299+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31801109e0 con 0x7f31800fbfb0 2022-07-03T10:29:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.300+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318005fb50 con 0x7f31800fbfb0 2022-07-03T10:29:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.300+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f3174028030 con 0x7f31800fbfb0 2022-07-03T10:29:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.300+0000 7f317cff9700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 0x7f3168037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.301+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3174047c30 con 0x7f31800fbfb0 2022-07-03T10:29:02.307 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.301+0000 7f317f7fe700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 0x7f3168037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:02.307 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.302+0000 7f317f7fe700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 0x7f3168037140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31700097c0 tx=0x7f3170006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:02.307 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.306+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f3174005ab0 con 0x7f31800fbfb0 2022-07-03T10:29:02.491 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.489+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f3180110ce0 con 0x7f3168034c90 2022-07-03T10:29:02.498 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.497+0000 7f317cff9700 1 -- 172.21.15.188:0/3785218135 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3180110ce0 con 0x7f3168034c90 2022-07-03T10:29:02.498 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2022-07-03T10:29:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 msgr2=0x7f3168037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 0x7f3168037140 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31700097c0 tx=0x7f3170006d00 comp rx=0 tx=0).stop 2022-07-03T10:29:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 msgr2=0x7f318010f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3174000c00 tx=0x7f31740091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 shutdown_connections 2022-07-03T10:29:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3168034c90 0x7f3168037140 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 --2- 172.21.15.188:0/3785218135 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f31800fbfb0 0x7f318010f620 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.498+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 >> 172.21.15.188:0/3785218135 conn(0x7f31800f76c0 msgr2=0x7f3180065ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:02.502 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.499+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 shutdown_connections 2022-07-03T10:29:02.502 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:02.499+0000 7f3186349700 1 -- 172.21.15.188:0/3785218135 wait complete. 2022-07-03T10:29:02.914 DEBUG:teuthology.orchestra.run.smithi161:mon.b> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mon.b.service 2022-07-03T10:29:02.917 DEBUG:teuthology.orchestra.run.smithi188:mon.c> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mon.c.service 2022-07-03T10:29:02.920 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-07-03T10:29:02.920 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph mon dump -f json 2022-07-03T10:29:02.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:29:02.951 INFO:journalctl@ceph.mon.b.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:29:03.581 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:03.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:03.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:03.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: Saving service mon spec with placement smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c;count:3 2022-07-03T10:29:03.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:03.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:03.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:03.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:03.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:03 smithi101 ceph-mon[85126]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:04.589 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 -- 172.21.15.188:0/2141074241 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 msgr2=0x7fd6fc0fe740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:04.589 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 --2- 172.21.15.188:0/2141074241 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fe740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009800 tx=0x7fd6ec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 -- 172.21.15.188:0/2141074241 shutdown_connections 2022-07-03T10:29:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 --2- 172.21.15.188:0/2141074241 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fe740 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 -- 172.21.15.188:0/2141074241 >> 172.21.15.188:0/2141074241 conn(0x7fd6fc0f7c90 msgr2=0x7fd6fc0fa0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 -- 172.21.15.188:0/2141074241 shutdown_connections 2022-07-03T10:29:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.586+0000 7fd70158e700 1 -- 172.21.15.188:0/2141074241 wait complete. 2022-07-03T10:29:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.587+0000 7fd70158e700 1 Processor -- start 2022-07-03T10:29:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.587+0000 7fd70158e700 1 -- start start 2022-07-03T10:29:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.587+0000 7fd70158e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.587+0000 7fd70158e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0fc590 con 0x7fd6fc0fe370 2022-07-03T10:29:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6faffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6faffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.188:59808/0 (socket says 172.21.15.188:59808) 2022-07-03T10:29:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6faffd700 1 -- 172.21.15.188:0/2963981387 learned_addr learned my addr 172.21.15.188:0/2963981387 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:29:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6faffd700 1 -- 172.21.15.188:0/2963981387 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec009440 con 0x7fd6fc0fe370 2022-07-03T10:29:04.593 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6faffd700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec0051f0 tx=0x7fd6ec0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:04.594 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec01c070 con 0x7fd6fc0fe370 2022-07-03T10:29:04.594 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0fc790 con 0x7fd6fc0fe370 2022-07-03T10:29:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd6ec00bd50 con 0x7fd6fc0fe370 2022-07-03T10:29:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec021ca0 con 0x7fd6fc0fe370 2022-07-03T10:29:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.588+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc0fcbc0 con 0x7fd6fc0fe370 2022-07-03T10:29:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.589+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fd6ec028030 con 0x7fd6fc0fe370 2022-07-03T10:29:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.589+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc05fb50 con 0x7fd6fc0fe370 2022-07-03T10:29:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.589+0000 7fd6e37fe700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 0x7fd6e4037140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.589+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd6ec047c40 con 0x7fd6fc0fe370 2022-07-03T10:29:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.590+0000 7fd6fa7fc700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 0x7fd6e4037140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.591+0000 7fd6fa7fc700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 0x7fd6e4037140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00097c0 tx=0x7fd6f0006d00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:04.597 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.595+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd6ec02a590 con 0x7fd6fc0fe370 2022-07-03T10:29:04.842 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.841+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6fc107400 con 0x7fd6fc0fe370 2022-07-03T10:29:04.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.842+0000 7fd6e37fe700 1 -- 172.21.15.188:0/2963981387 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+714 (secure 0 0 0) 0x7fd6ec026020 con 0x7fd6fc0fe370 2022-07-03T10:29:04.845 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:29:04.845 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","modified":"2022-07-03T10:27:22.976016Z","created":"2022-07-03T10:27:22.976016Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:3300","nonce":0},{"type":"v1","addr":"172.21.15.101:6789","nonce":0}]},"addr":"172.21.15.101:6789/0","public_addr":"172.21.15.101:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-07-03T10:29:04.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 msgr2=0x7fd6e4037140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:04.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 0x7fd6e4037140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00097c0 tx=0x7fd6f0006d00 comp rx=0 tx=0).stop 2022-07-03T10:29:04.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 msgr2=0x7fd6fc0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec0051f0 tx=0x7fd6ec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 shutdown_connections 2022-07-03T10:29:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6e4034c90 0x7fd6e4037140 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:04.850 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 --2- 172.21.15.188:0/2963981387 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6fc0fe370 0x7fd6fc0fc050 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:04.850 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 >> 172.21.15.188:0/2963981387 conn(0x7fd6fc0f7c90 msgr2=0x7fd6fc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:04.851 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 shutdown_connections 2022-07-03T10:29:04.851 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:04.844+0000 7fd70158e700 1 -- 172.21.15.188:0/2963981387 wait complete. 2022-07-03T10:29:04.852 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2022-07-03T10:29:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:29:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:04.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:04.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:04 smithi101 ceph-mon[85126]: Deploying daemon mon.b on smithi161 2022-07-03T10:29:05.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:05 smithi101 ceph-mon[85126]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:05.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:05 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/2963981387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:05 smithi161 systemd[1]: Starting Ceph mon.b for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:29:06.318 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-07-03T10:29:06.319 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph mon dump -f json 2022-07-03T10:29:06.449 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 podman[91539]: 2022-07-03T10:29:06.450 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 bash[91539]: bf56df17d9f5bde0a8313c39a4d4d15025779181aee76d08541a3db734681601 2022-07-03T10:29:06.451 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: set uid:gid to 167:167 (ceph:ceph) 2022-07-03T10:29:06.451 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: ceph version 17.0.0-13322-g61829bae (61829bae536fb4126cfaa87ca353b9576e018e7d) quincy (dev), process ceph-mon, pid 7 2022-07-03T10:29:06.451 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pidfile_write: ignore empty --pid-file 2022-07-03T10:29:06.452 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: load: jerasure load: lrc 2022-07-03T10:29:06.452 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: RocksDB version: 6.15.5 2022-07-03T10:29:06.453 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-07-03T10:29:06.453 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Compile date Jul 2 2022 2022-07-03T10:29:06.454 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: DB SUMMARY 2022-07-03T10:29:06.454 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: DB Session ID: VE4UCLK8ZHY2074PQW6A 2022-07-03T10:29:06.455 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: CURRENT file: CURRENT 2022-07-03T10:29:06.455 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: IDENTITY file: IDENTITY 2022-07-03T10:29:06.455 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-07-03T10:29:06.456 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-07-03T10:29:06.456 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2022-07-03T10:29:06.456 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.error_if_exists: 0 2022-07-03T10:29:06.457 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.create_if_missing: 0 2022-07-03T10:29:06.457 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.paranoid_checks: 1 2022-07-03T10:29:06.457 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-07-03T10:29:06.458 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.env: 0x55f99bbd3860 2022-07-03T10:29:06.458 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.fs: Posix File System 2022-07-03T10:29:06.459 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.info_log: 0x55f99d39be80 2022-07-03T10:29:06.459 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_file_opening_threads: 16 2022-07-03T10:29:06.459 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.statistics: (nil) 2022-07-03T10:29:06.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.use_fsync: 0 2022-07-03T10:29:06.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_log_file_size: 0 2022-07-03T10:29:06.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-07-03T10:29:06.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.log_file_time_to_roll: 0 2022-07-03T10:29:06.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.keep_log_file_num: 1000 2022-07-03T10:29:06.461 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.recycle_log_file_num: 0 2022-07-03T10:29:06.461 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_fallocate: 1 2022-07-03T10:29:06.461 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_mmap_reads: 0 2022-07-03T10:29:06.461 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_mmap_writes: 0 2022-07-03T10:29:06.461 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.use_direct_reads: 0 2022-07-03T10:29:06.462 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-07-03T10:29:06.462 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.create_missing_column_families: 0 2022-07-03T10:29:06.462 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.db_log_dir: 2022-07-03T10:29:06.462 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-07-03T10:29:06.463 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.table_cache_numshardbits: 6 2022-07-03T10:29:06.463 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.WAL_ttl_seconds: 0 2022-07-03T10:29:06.463 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.WAL_size_limit_MB: 0 2022-07-03T10:29:06.463 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-07-03T10:29:06.463 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-07-03T10:29:06.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.is_fd_close_on_exec: 1 2022-07-03T10:29:06.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.advise_random_on_open: 1 2022-07-03T10:29:06.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.db_write_buffer_size: 0 2022-07-03T10:29:06.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.write_buffer_manager: 0x55f99d488960 2022-07-03T10:29:06.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-07-03T10:29:06.465 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-07-03T10:29:06.465 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-07-03T10:29:06.465 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.use_adaptive_mutex: 0 2022-07-03T10:29:06.465 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.rate_limiter: (nil) 2022-07-03T10:29:06.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-07-03T10:29:06.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.wal_recovery_mode: 2 2022-07-03T10:29:06.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.enable_thread_tracking: 0 2022-07-03T10:29:06.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.enable_pipelined_write: 0 2022-07-03T10:29:06.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.unordered_write: 0 2022-07-03T10:29:06.467 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-07-03T10:29:06.467 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-07-03T10:29:06.467 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-07-03T10:29:06.467 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-07-03T10:29:06.467 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.row_cache: None 2022-07-03T10:29:06.468 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.wal_filter: None 2022-07-03T10:29:06.468 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-07-03T10:29:06.468 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_ingest_behind: 0 2022-07-03T10:29:06.468 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.preserve_deletes: 0 2022-07-03T10:29:06.468 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.two_write_queues: 0 2022-07-03T10:29:06.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.manual_wal_flush: 0 2022-07-03T10:29:06.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.atomic_flush: 0 2022-07-03T10:29:06.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-07-03T10:29:06.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.persist_stats_to_disk: 0 2022-07-03T10:29:06.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.write_dbid_to_manifest: 0 2022-07-03T10:29:06.470 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.log_readahead_size: 0 2022-07-03T10:29:06.470 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-07-03T10:29:06.470 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.best_efforts_recovery: 0 2022-07-03T10:29:06.470 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-07-03T10:29:06.471 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-07-03T10:29:06.471 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.allow_data_in_errors: 0 2022-07-03T10:29:06.471 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.db_host_id: __hostname__ 2022-07-03T10:29:06.473 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_background_jobs: 2 2022-07-03T10:29:06.473 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_background_compactions: -1 2022-07-03T10:29:06.473 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_subcompactions: 1 2022-07-03T10:29:06.473 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-07-03T10:29:06.474 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-07-03T10:29:06.474 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.delayed_write_rate : 16777216 2022-07-03T10:29:06.474 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_total_wal_size: 0 2022-07-03T10:29:06.474 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-07-03T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.stats_dump_period_sec: 600 2022-07-03T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.stats_persist_period_sec: 600 2022-07-03T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-07-03T10:29:06.475 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_open_files: -1 2022-07-03T10:29:06.476 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bytes_per_sync: 0 2022-07-03T10:29:06.476 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.wal_bytes_per_sync: 0 2022-07-03T10:29:06.476 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.strict_bytes_per_sync: 0 2022-07-03T10:29:06.476 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_readahead_size: 0 2022-07-03T10:29:06.476 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_background_flushes: -1 2022-07-03T10:29:06.477 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Compression algorithms supported: 2022-07-03T10:29:06.477 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-07-03T10:29:06.477 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kZSTD supported: 0 2022-07-03T10:29:06.477 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kXpressCompression supported: 0 2022-07-03T10:29:06.477 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kLZ4HCCompression supported: 1 2022-07-03T10:29:06.478 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kLZ4Compression supported: 1 2022-07-03T10:29:06.478 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kBZip2Compression supported: 0 2022-07-03T10:29:06.478 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kZlibCompression supported: 1 2022-07-03T10:29:06.478 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: kSnappyCompression supported: 1 2022-07-03T10:29:06.479 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Fast CRC32 supported: Supported on x86 2022-07-03T10:29:06.479 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2022-07-03T10:29:06.479 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-07-03T10:29:06.479 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-07-03T10:29:06.479 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.merge_operator: 2022-07-03T10:29:06.480 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_filter: None 2022-07-03T10:29:06.480 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_filter_factory: None 2022-07-03T10:29:06.480 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.sst_partitioner_factory: None 2022-07-03T10:29:06.480 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.memtable_factory: SkipListFactory 2022-07-03T10:29:06.480 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.table_factory: BlockBasedTable 2022-07-03T10:29:06.481 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f99d3999f8) 2022-07-03T10:29:06.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: cache_index_and_filter_blocks: 1 2022-07-03T10:29:06.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-07-03T10:29:06.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-07-03T10:29:06.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: pin_top_level_index_and_filter: 1 2022-07-03T10:29:06.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_type: 0 2022-07-03T10:29:06.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: data_block_index_type: 0 2022-07-03T10:29:06.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_shortening: 1 2022-07-03T10:29:06.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: data_block_hash_table_util_ratio: 0.750000 2022-07-03T10:29:06.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: hash_index_allow_collision: 1 2022-07-03T10:29:06.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: checksum: 1 2022-07-03T10:29:06.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: no_block_cache: 0 2022-07-03T10:29:06.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache: 0x55f99d4049b0 2022-07-03T10:29:06.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_name: BinnedLRUCache 2022-07-03T10:29:06.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_options: 2022-07-03T10:29:06.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: capacity : 536870912 2022-07-03T10:29:06.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: num_shard_bits : 4 2022-07-03T10:29:06.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: strict_capacity_limit : 0 2022-07-03T10:29:06.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: high_pri_pool_ratio: 0.000 2022-07-03T10:29:06.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_cache_compressed: (nil) 2022-07-03T10:29:06.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: persistent_cache: (nil) 2022-07-03T10:29:06.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_size: 4096 2022-07-03T10:29:06.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_size_deviation: 10 2022-07-03T10:29:06.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_restart_interval: 16 2022-07-03T10:29:06.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: index_block_restart_interval: 1 2022-07-03T10:29:06.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: metadata_block_size: 4096 2022-07-03T10:29:06.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: partition_filters: 0 2022-07-03T10:29:06.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: use_delta_encoding: 1 2022-07-03T10:29:06.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-07-03T10:29:06.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: whole_key_filtering: 1 2022-07-03T10:29:06.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: verify_compression: 0 2022-07-03T10:29:06.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: read_amp_bytes_per_bit: 0 2022-07-03T10:29:06.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: format_version: 4 2022-07-03T10:29:06.488 INFO:journalctl@ceph.mon.b.smithi161.stdout: enable_index_compression: 1 2022-07-03T10:29:06.488 INFO:journalctl@ceph.mon.b.smithi161.stdout: block_align: 0 2022-07-03T10:29:06.488 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.write_buffer_size: 33554432 2022-07-03T10:29:06.488 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_write_buffer_number: 2 2022-07-03T10:29:06.488 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression: NoCompression 2022-07-03T10:29:06.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression: Disabled 2022-07-03T10:29:06.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.prefix_extractor: nullptr 2022-07-03T10:29:06.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-07-03T10:29:06.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.num_levels: 7 2022-07-03T10:29:06.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-07-03T10:29:06.490 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-07-03T10:29:06.490 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-07-03T10:29:06.490 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-07-03T10:29:06.490 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-07-03T10:29:06.491 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-07-03T10:29:06.491 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-07-03T10:29:06.491 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-07-03T10:29:06.491 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-07-03T10:29:06.491 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-07-03T10:29:06.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.window_bits: -14 2022-07-03T10:29:06.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.level: 32767 2022-07-03T10:29:06.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.strategy: 0 2022-07-03T10:29:06.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-07-03T10:29:06.492 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-07-03T10:29:06.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-07-03T10:29:06.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compression_opts.enabled: false 2022-07-03T10:29:06.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-07-03T10:29:06.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-07-03T10:29:06.493 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-07-03T10:29:06.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.target_file_size_base: 67108864 2022-07-03T10:29:06.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.target_file_size_multiplier: 1 2022-07-03T10:29:06.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-07-03T10:29:06.494 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-07-03T10:29:06.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-07-03T10:29:06.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-07-03T10:29:06.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-07-03T10:29:06.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-07-03T10:29:06.495 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-07-03T10:29:06.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-07-03T10:29:06.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-07-03T10:29:06.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-07-03T10:29:06.496 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-07-03T10:29:06.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-07-03T10:29:06.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.arena_block_size: 4194304 2022-07-03T10:29:06.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-07-03T10:29:06.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-07-03T10:29:06.497 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-07-03T10:29:06.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.disable_auto_compactions: 0 2022-07-03T10:29:06.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-07-03T10:29:06.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-07-03T10:29:06.498 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-07-03T10:29:06.499 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-07-03T10:29:06.499 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-07-03T10:29:06.499 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-07-03T10:29:06.499 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-07-03T10:29:06.499 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-07-03T10:29:06.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-07-03T10:29:06.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-07-03T10:29:06.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.table_properties_collectors: 2022-07-03T10:29:06.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.inplace_update_support: 0 2022-07-03T10:29:06.500 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.inplace_update_num_locks: 10000 2022-07-03T10:29:06.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-07-03T10:29:06.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-07-03T10:29:06.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.memtable_huge_page_size: 0 2022-07-03T10:29:06.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.bloom_locality: 0 2022-07-03T10:29:06.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.max_successive_merges: 0 2022-07-03T10:29:06.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.optimize_filters_for_hits: 0 2022-07-03T10:29:06.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.paranoid_file_checks: 0 2022-07-03T10:29:06.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.force_consistency_checks: 1 2022-07-03T10:29:06.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.report_bg_io_stats: 0 2022-07-03T10:29:06.502 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.ttl: 2592000 2022-07-03T10:29:06.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.periodic_compaction_seconds: 0 2022-07-03T10:29:06.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.enable_blob_files: false 2022-07-03T10:29:06.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.min_blob_size: 0 2022-07-03T10:29:06.503 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.blob_file_size: 268435456 2022-07-03T10:29:06.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.blob_compression_type: NoCompression 2022-07-03T10:29:06.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.enable_blob_garbage_collection: false 2022-07-03T10:29:06.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-07-03T10:29:06.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-07-03T10:29:06.504 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2022-07-03T10:29:06.505 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2022-07-03T10:29:06.505 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844146424636, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-07-03T10:29:06.505 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2022-07-03T10:29:06.505 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-07-03T10:29:06.506 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844146424903, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1656844146, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "75a59ff5-5afb-44f5-8872-5e133936530a", "db_session_id": "VE4UCLK8ZHY2074PQW6A"}} 2022-07-03T10:29:06.506 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2022-07-03T10:29:06.506 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: EVENT_LOG_v1 {"time_micros": 1656844146426028, "job": 1, "event": "recovery_finished"} 2022-07-03T10:29:06.506 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-07-03T10:29:06.506 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55f99d3ea700 2022-07-03T10:29:06.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: DB pointer 0x55f99d3fa000 2022-07-03T10:29:06.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b does not exist in monmap, will attempt to join an existing cluster 2022-07-03T10:29:06.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-07-03T10:29:06.507 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-07-03T10:29:06.508 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** DB Stats ** 2022-07-03T10:29:06.508 INFO:journalctl@ceph.mon.b.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:29:06.508 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-07-03T10:29:06.508 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-07-03T10:29:06.508 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-07-03T10:29:06.509 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-07-03T10:29:06.509 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-07-03T10:29:06.509 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-07-03T10:29:06.509 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.509 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2022-07-03T10:29:06.510 INFO:journalctl@ceph.mon.b.smithi161.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:29:06.510 INFO:journalctl@ceph.mon.b.smithi161.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:29:06.510 INFO:journalctl@ceph.mon.b.smithi161.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.510 INFO:journalctl@ceph.mon.b.smithi161.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.511 INFO:journalctl@ceph.mon.b.smithi161.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.511 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.511 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2022-07-03T10:29:06.511 INFO:journalctl@ceph.mon.b.smithi161.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:29:06.511 INFO:journalctl@ceph.mon.b.smithi161.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:29:06.512 INFO:journalctl@ceph.mon.b.smithi161.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.512 INFO:journalctl@ceph.mon.b.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:29:06.512 INFO:journalctl@ceph.mon.b.smithi161.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-07-03T10:29:06.512 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-07-03T10:29:06.512 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-07-03T10:29:06.513 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-07-03T10:29:06.513 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Keys): cumulative 0, interval 0 2022-07-03T10:29:06.513 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:29:06.513 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:29:06.513 INFO:journalctl@ceph.mon.b.smithi161.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-07-03T10:29:06.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** File Read Latency Histogram By Level [default] ** 2022-07-03T10:29:06.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2022-07-03T10:29:06.514 INFO:journalctl@ceph.mon.b.smithi161.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:29:06.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:29:06.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.515 INFO:journalctl@ceph.mon.b.smithi161.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** Compaction Stats [default] ** 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2022-07-03T10:29:06.516 INFO:journalctl@ceph.mon.b.smithi161.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-07-03T10:29:06.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-07-03T10:29:06.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-07-03T10:29:06.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-07-03T10:29:06.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-07-03T10:29:06.517 INFO:journalctl@ceph.mon.b.smithi161.stdout: AddFile(Keys): cumulative 0, interval 0 2022-07-03T10:29:06.518 INFO:journalctl@ceph.mon.b.smithi161.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:29:06.518 INFO:journalctl@ceph.mon.b.smithi161.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-07-03T10:29:06.518 INFO:journalctl@ceph.mon.b.smithi161.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-07-03T10:29:06.518 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.519 INFO:journalctl@ceph.mon.b.smithi161.stdout: ** File Read Latency Histogram By Level [default] ** 2022-07-03T10:29:06.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: using public_addr v2:172.21.15.161:0/0 -> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2022-07-03T10:29:06.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: starting mon.b rank -1 at public addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] at bind addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:29:06.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(???) e0 preinit fsid 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:29:06.519 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 systemd[1]: Started Ceph mon.b for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:29:06.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).mds e1 new map 2022-07-03T10:29:06.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).mds e1 print_map 2022-07-03T10:29:06.520 INFO:journalctl@ceph.mon.b.smithi161.stdout: e1 2022-07-03T10:29:06.520 INFO:journalctl@ceph.mon.b.smithi161.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-07-03T10:29:06.520 INFO:journalctl@ceph.mon.b.smithi161.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-07-03T10:29:06.521 INFO:journalctl@ceph.mon.b.smithi161.stdout: legacy client fscid: -1 2022-07-03T10:29:06.521 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:29:06.521 INFO:journalctl@ceph.mon.b.smithi161.stdout: No filesystems configured 2022-07-03T10:29:06.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-07-03T10:29:06.521 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-07-03T10:29:06.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-07-03T10:29:06.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-07-03T10:29:06.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-07-03T10:29:06.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-07-03T10:29:06.522 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-07-03T10:29:06.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:29:06.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:29:06.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-07-03T10:29:06.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.523 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:29:06.524 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:06.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Deploying cephadm binary to smithi188 2022-07-03T10:29:06.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.525 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Added host smithi188 2022-07-03T10:29:06.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:06.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:06.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:06.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:29:06.526 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:29:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:29:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:29:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-07-03T10:29:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-07-03T10:29:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Saving service mon spec with placement smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c;count:3 2022-07-03T10:29:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:29:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: Deploying daemon mon.b on smithi161 2022-07-03T10:29:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/2963981387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-07-03T10:29:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:06 smithi161 ceph-mon[91579]: mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2022-07-03T10:29:06.831 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:11.550 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 -- 172.21.15.188:0/3418773489 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7c8003440 msgr2=0x7fd7c80058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:11.550 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 --2- 172.21.15.188:0/3418773489 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7c8003440 0x7fd7c80058c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc070d80 tx=0x7fd7d80145b0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.551 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 -- 172.21.15.188:0/3418773489 shutdown_connections 2022-07-03T10:29:11.551 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 --2- 172.21.15.188:0/3418773489 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7c8003440 0x7fd7c80058c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.551 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 -- 172.21.15.188:0/3418773489 >> 172.21.15.188:0/3418773489 conn(0x7fd7dc072f40 msgr2=0x7fd7dc073340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:11.551 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 -- 172.21.15.188:0/3418773489 shutdown_connections 2022-07-03T10:29:11.552 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.545+0000 7fd7e4764700 1 -- 172.21.15.188:0/3418773489 wait complete. 2022-07-03T10:29:11.552 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 Processor -- start 2022-07-03T10:29:11.552 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 -- start start 2022-07-03T10:29:11.553 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:11.553 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 0x7fd7dc117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:11.553 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc10f100 con 0x7fd7dc10ec00 2022-07-03T10:29:11.554 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e4764700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc10f240 con 0x7fd7c8003440 2022-07-03T10:29:11.554 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e2500700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:11.554 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e2500700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.188:35400/0 (socket says 172.21.15.188:35400) 2022-07-03T10:29:11.555 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e2500700 1 -- 172.21.15.188:0/4028442403 learned_addr learned my addr 172.21.15.188:0/4028442403 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:29:11.555 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.546+0000 7fd7e1cff700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 0x7fd7dc117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:11.555 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e2500700 1 -- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 msgr2=0x7fd7dc10e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-07-03T10:29:11.555 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e2500700 1 -- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 msgr2=0x7fd7dc10e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-07-03T10:29:11.556 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e2500700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-07-03T10:29:11.556 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e2500700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-07-03T10:29:11.556 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e1cff700 1 -- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 msgr2=0x7fd7dc10e6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-07-03T10:29:11.556 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e1cff700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.557 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e1cff700 1 -- 172.21.15.188:0/4028442403 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8014850 con 0x7fd7dc10ec00 2022-07-03T10:29:11.557 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e1cff700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 0x7fd7dc117550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b560 tx=0x7fd7cc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:11.557 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd7cc0117e0 con 0x7fd7dc10ec00 2022-07-03T10:29:11.557 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc117a90 con 0x7fd7dc10ec00 2022-07-03T10:29:11.557 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd7cc011df0 con 0x7fd7dc10ec00 2022-07-03T10:29:11.558 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc117fc0 con 0x7fd7dc10ec00 2022-07-03T10:29:11.558 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.547+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd7cc010e80 con 0x7fd7dc10ec00 2022-07-03T10:29:11.559 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.549+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fd7cc00f050 con 0x7fd7dc10ec00 2022-07-03T10:29:11.560 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.549+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc05fb50 con 0x7fd7dc10ec00 2022-07-03T10:29:11.560 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.549+0000 7fd7d2ffd700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 0x7fd7c00372d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:11.560 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.549+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd7cc0461e0 con 0x7fd7dc10ec00 2022-07-03T10:29:11.560 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.549+0000 7fd7e2500700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 0x7fd7c00372d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:11.561 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.550+0000 7fd7e2500700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 0x7fd7c00372d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d8021040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:11.561 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.555+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd7cc00f4c0 con 0x7fd7dc10ec00 2022-07-03T10:29:11.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.840+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7dc103c10 con 0x7fd7dc10ec00 2022-07-03T10:29:11.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.841+0000 7fd7d2ffd700 1 -- 172.21.15.188:0/4028442403 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+984 (secure 0 0 0) 0x7fd7cc015020 con 0x7fd7dc10ec00 2022-07-03T10:29:11.844 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:29:11.844 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","modified":"2022-07-03T10:29:06.453318Z","created":"2022-07-03T10:27:22.976016Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:3300","nonce":0},{"type":"v1","addr":"172.21.15.101:6789","nonce":0}]},"addr":"172.21.15.101:6789/0","public_addr":"172.21.15.101:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-07-03T10:29:11.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.844+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 msgr2=0x7fd7c00372d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:11.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.844+0000 7fd7e4764700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 0x7fd7c00372d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d8021040 comp rx=0 tx=0).stop 2022-07-03T10:29:11.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.844+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 msgr2=0x7fd7dc117550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:11.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.844+0000 7fd7e4764700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 0x7fd7dc117550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b560 tx=0x7fd7cc009060 comp rx=0 tx=0).stop 2022-07-03T10:29:11.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 shutdown_connections 2022-07-03T10:29:11.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7c0034e20 0x7fd7c00372d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7c8003440 0x7fd7dc10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 --2- 172.21.15.188:0/4028442403 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7dc10ec00 0x7fd7dc117550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:11.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 >> 172.21.15.188:0/4028442403 conn(0x7fd7dc072f40 msgr2=0x7fd7dc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:11.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 shutdown_connections 2022-07-03T10:29:11.849 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:11.845+0000 7fd7e4764700 1 -- 172.21.15.188:0/4028442403 wait complete. 2022-07-03T10:29:11.850 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2022-07-03T10:29:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: mon.a calling monitor election 2022-07-03T10:29:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: mon.b calling monitor election 2022-07-03T10:29:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-07-03T10:29:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: monmap e2: 2 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} 2022-07-03T10:29:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: fsmap 2022-07-03T10:29:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:11.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: mgrmap e13: x(active, since 45s) 2022-07-03T10:29:11.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: overall HEALTH_OK 2022-07-03T10:29:11.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:11.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:11.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:11.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:11.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:11.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: mon.a calling monitor election 2022-07-03T10:29:11.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: mon.b calling monitor election 2022-07-03T10:29:11.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:11.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:11.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-07-03T10:29:11.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: monmap e2: 2 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} 2022-07-03T10:29:11.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: fsmap 2022-07-03T10:29:11.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:11.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: mgrmap e13: x(active, since 45s) 2022-07-03T10:29:11.997 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: overall HEALTH_OK 2022-07-03T10:29:11.997 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:11.997 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:11.997 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:11.998 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:12 smithi161 ceph-mon[91579]: Deploying daemon mon.c on smithi188 2022-07-03T10:29:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:12 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/4028442403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:12.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:12 smithi101 ceph-mon[85126]: Deploying daemon mon.c on smithi188 2022-07-03T10:29:12.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:12 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/4028442403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:13.183 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-07-03T10:29:13.184 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph mon dump -f json 2022-07-03T10:29:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:13 smithi161 ceph-mon[91579]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:13.910 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:29:13.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:13 smithi101 ceph-mon[85126]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.659 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.829 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:29:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:14.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Deploying cephadm binary to smithi188 2022-07-03T10:29:14.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Added host smithi188 2022-07-03T10:29:14.831 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:14.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:14.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:14.832 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:29:14.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:14.833 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:14.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:29:14.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:29:14.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:29:14.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-07-03T10:29:14.837 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1523588512' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-07-03T10:29:14.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:14.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.838 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:14.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Saving service mon spec with placement smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c;count:3 2022-07-03T10:29:14.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:14.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:14.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:14.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:14.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:14.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:14.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:29:14.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.841 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:14.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:14.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Deploying daemon mon.b on smithi161 2022-07-03T10:29:14.842 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/2963981387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:14.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.843 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:14.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mon.a calling monitor election 2022-07-03T10:29:14.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.844 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.845 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mon.b calling monitor election 2022-07-03T10:29:14.845 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.845 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.845 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:14.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-07-03T10:29:14.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: monmap e2: 2 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0]} 2022-07-03T10:29:14.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: fsmap 2022-07-03T10:29:14.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:14.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mgrmap e13: x(active, since 45s) 2022-07-03T10:29:14.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: overall HEALTH_OK 2022-07-03T10:29:14.847 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.848 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:14.848 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:14.848 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:14.848 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: Deploying daemon mon.c on smithi188 2022-07-03T10:29:14.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/4028442403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:14.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:14.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-07-03T10:29:14.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 ceph-mon[92425]: mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2022-07-03T10:29:14.849 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 bash[92356]: 0212006b729dcdb57cf6b234718aafa12391117781ca765dc5b8bd1ff4c01628 2022-07-03T10:29:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:14 smithi188 systemd[1]: Started Ceph mon.c for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:29:19.714 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 -- 172.21.15.188:0/926027494 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 msgr2=0x7fcc84005940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:19.715 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/926027494 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc84005940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0fc370 tx=0x7fcc90012450 comp rx=0 tx=0).stop 2022-07-03T10:29:19.715 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 -- 172.21.15.188:0/926027494 shutdown_connections 2022-07-03T10:29:19.716 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/926027494 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc84007f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:19.716 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/926027494 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc84005940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:19.716 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 -- 172.21.15.188:0/926027494 >> 172.21.15.188:0/926027494 conn(0x7fcc9c072330 msgr2=0x7fcc9c072730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:19.716 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 -- 172.21.15.188:0/926027494 shutdown_connections 2022-07-03T10:29:19.716 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.712+0000 7fcca1bb4700 1 -- 172.21.15.188:0/926027494 wait complete. 2022-07-03T10:29:19.720 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 Processor -- start 2022-07-03T10:29:19.720 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 -- start start 2022-07-03T10:29:19.720 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc9c113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:19.720 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:19.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:19.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c10e830 con 0x7fcc84003500 2022-07-03T10:29:19.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c10e9a0 con 0x7fcc9c102d60 2022-07-03T10:29:19.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.713+0000 7fcca1bb4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c10eb60 con 0x7fcc9c10ddf0 2022-07-03T10:29:19.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:19.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40378/0 (socket says 172.21.15.188:40378) 2022-07-03T10:29:19.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 -- 172.21.15.188:0/402229249 learned_addr learned my addr 172.21.15.188:0/402229249 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:29:19.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc9c113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:19.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9affd700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:19.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 msgr2=0x7fcc9c10e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2022-07-03T10:29:19.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 msgr2=0x7fcc9c10e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-07-03T10:29:19.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-07-03T10:29:19.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9bfff700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-07-03T10:29:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9affd700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 msgr2=0x7fcc9c1142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-07-03T10:29:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9affd700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 msgr2=0x7fcc9c1142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-07-03T10:29:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9affd700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-07-03T10:29:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9affd700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-07-03T10:29:19.724 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 msgr2=0x7fcc9c10e260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-07-03T10:29:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 msgr2=0x7fcc9c1142a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-07-03T10:29:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:19.725 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.714+0000 7fcc9b7fe700 1 -- 172.21.15.188:0/402229249 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900126b0 con 0x7fcc84003500 2022-07-03T10:29:19.726 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcc9b7fe700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc9c113d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc90020040 tx=0x7fcc90012450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:19.726 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90003b80 con 0x7fcc84003500 2022-07-03T10:29:19.726 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c11ab70 con 0x7fcc84003500 2022-07-03T10:29:19.726 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcc90016030 con 0x7fcc84003500 2022-07-03T10:29:19.726 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90004d20 con 0x7fcc84003500 2022-07-03T10:29:19.727 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.715+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c11b070 con 0x7fcc84003500 2022-07-03T10:29:19.727 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.716+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fcc90036070 con 0x7fcc84003500 2022-07-03T10:29:19.727 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.716+0000 7fcc83fff700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 0x7fcc880374b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:19.727 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.716+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcc900497b0 con 0x7fcc84003500 2022-07-03T10:29:19.728 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.716+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c04ccb0 con 0x7fcc84003500 2022-07-03T10:29:19.728 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.716+0000 7fcc9affd700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 0x7fcc880374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:19.728 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.717+0000 7fcc9affd700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 0x7fcc880374b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c005d00 tx=0x7fcc8c004040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:19.729 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.726+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fcc9002a080 con 0x7fcc84003500 2022-07-03T10:29:19.997 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.994+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc9c10f350 con 0x7fcc84003500 2022-07-03T10:29:19.998 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.995+0000 7fcc83fff700 1 -- 172.21.15.188:0/402229249 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1254 (secure 0 0 0) 0x7fcc90022020 con 0x7fcc84003500 2022-07-03T10:29:19.998 INFO:teuthology.orchestra.run.smithi188.stdout: 2022-07-03T10:29:19.998 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","modified":"2022-07-03T10:29:14.664388Z","created":"2022-07-03T10:27:22.976016Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:3300","nonce":0},{"type":"v1","addr":"172.21.15.101:6789","nonce":0}]},"addr":"172.21.15.101:6789/0","public_addr":"172.21.15.101:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2022-07-03T10:29:20.000 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 msgr2=0x7fcc880374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:20.000 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 0x7fcc880374b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c005d00 tx=0x7fcc8c004040 comp rx=0 tx=0).stop 2022-07-03T10:29:20.000 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 msgr2=0x7fcc9c113d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:20.000 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc9c113d60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc90020040 tx=0x7fcc90012450 comp rx=0 tx=0).stop 2022-07-03T10:29:20.001 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 shutdown_connections 2022-07-03T10:29:20.001 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcc88035000 0x7fcc880374b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:20.001 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcc84003500 0x7fcc9c113d60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:20.001 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc9c10ddf0 0x7fcc9c10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:20.002 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 --2- 172.21.15.188:0/402229249 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc9c102d60 0x7fcc9c1142a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:20.002 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 >> 172.21.15.188:0/402229249 conn(0x7fcc9c072330 msgr2=0x7fcc9c065df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:20.002 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.997+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 shutdown_connections 2022-07-03T10:29:20.002 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:19.998+0000 7fcca1bb4700 1 -- 172.21.15.188:0/402229249 wait complete. 2022-07-03T10:29:20.003 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2022-07-03T10:29:20.022 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:20.022 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: mon.a calling monitor election 2022-07-03T10:29:20.022 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:20.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: mon.b calling monitor election 2022-07-03T10:29:20.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.023 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: mon.c calling monitor election 2022-07-03T10:29:20.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-07-03T10:29:20.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: monmap e3: 3 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} 2022-07-03T10:29:20.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: fsmap 2022-07-03T10:29:20.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:20.026 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: mgrmap e13: x(active, since 53s) 2022-07-03T10:29:20.026 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: overall HEALTH_OK 2022-07-03T10:29:20.026 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.026 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:20.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:20.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:20.072 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:20.072 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: mon.a calling monitor election 2022-07-03T10:29:20.072 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:20.073 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.073 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: mon.b calling monitor election 2022-07-03T10:29:20.073 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.073 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.074 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.074 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: mon.c calling monitor election 2022-07-03T10:29:20.074 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.074 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.075 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.075 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.075 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-07-03T10:29:20.075 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: monmap e3: 3 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} 2022-07-03T10:29:20.075 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: fsmap 2022-07-03T10:29:20.076 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:20.076 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: mgrmap e13: x(active, since 53s) 2022-07-03T10:29:20.076 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: overall HEALTH_OK 2022-07-03T10:29:20.076 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.077 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.077 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:20.077 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:20.077 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:29:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: mon.a calling monitor election 2022-07-03T10:29:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:29:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: mon.b calling monitor election 2022-07-03T10:29:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: mon.c calling monitor election 2022-07-03T10:29:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-07-03T10:29:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: monmap e3: 3 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} 2022-07-03T10:29:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: fsmap 2022-07-03T10:29:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: osdmap e4: 0 total, 0 up, 0 in 2022-07-03T10:29:20.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: mgrmap e13: x(active, since 53s) 2022-07-03T10:29:20.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: overall HEALTH_OK 2022-07-03T10:29:20.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:20.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:20.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:20.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:20.403 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-07-03T10:29:20.404 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph config generate-minimal-conf 2022-07-03T10:29:20.918 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:20 smithi101 ceph-mon[85126]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:29:20.918 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:20 smithi101 ceph-mon[85126]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:20.919 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:20 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/402229249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:20.919 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:20 smithi101 ceph-mon[85126]: Updating smithi101:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:20 smithi161 ceph-mon[91579]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:29:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:20 smithi161 ceph-mon[91579]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:20 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/402229249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:20 smithi161 ceph-mon[91579]: Updating smithi101:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:20 smithi188 ceph-mon[92425]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:29:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:20 smithi188 ceph-mon[92425]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:20 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/402229249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-07-03T10:29:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:20 smithi188 ceph-mon[92425]: Updating smithi101:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:21.334 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:29:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:29:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:29:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:22.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:22.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:22.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:29:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:29:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:22.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:22.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:22.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:22.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 -- 172.21.15.101:0/432514063 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 msgr2=0x7fae40065610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:22.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 --2- 172.21.15.101:0/432514063 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40065610 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fae30009800 tx=0x7fae300091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:22.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 -- 172.21.15.101:0/432514063 shutdown_connections 2022-07-03T10:29:22.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 --2- 172.21.15.101:0/432514063 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40065610 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:22.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 -- 172.21.15.101:0/432514063 >> 172.21.15.101:0/432514063 conn(0x7fae40072f40 msgr2=0x7fae40073340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:22.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 -- 172.21.15.101:0/432514063 shutdown_connections 2022-07-03T10:29:22.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.862+0000 7fae44fc5700 1 -- 172.21.15.101:0/432514063 wait complete. 2022-07-03T10:29:22.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.863+0000 7fae44fc5700 1 Processor -- start 2022-07-03T10:29:22.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.863+0000 7fae44fc5700 1 -- start start 2022-07-03T10:29:22.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:22.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4011a3e0 0x7fae40113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:22.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 0x7fae401146d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:22.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4011aad0 con 0x7fae40101f50 2022-07-03T10:29:22.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40120cb0 con 0x7fae4011a3e0 2022-07-03T10:29:22.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae44fc5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40120e40 con 0x7fae40114260 2022-07-03T10:29:22.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae3e59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:22.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae3e59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:37730/0 (socket says 172.21.15.101:37730) 2022-07-03T10:29:22.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae3e59c700 1 -- 172.21.15.101:0/2944081283 learned_addr learned my addr 172.21.15.101:0/2944081283 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:29:22.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae3ed9d700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 0x7fae401146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:22.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.864+0000 7fae3dd9b700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4011a3e0 0x7fae40113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:22.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 -- 172.21.15.101:0/2944081283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 msgr2=0x7fae401146d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:22.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 0x7fae401146d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:22.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 -- 172.21.15.101:0/2944081283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4011a3e0 msgr2=0x7fae40113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:22.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4011a3e0 0x7fae40113d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:22.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 -- 172.21.15.101:0/2944081283 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae30009440 con 0x7fae40101f50 2022-07-03T10:29:22.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3ed9d700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 0x7fae401146d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:29:22.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae3e59c700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae30006010 tx=0x7fae300091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:22.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.865+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3001d070 con 0x7fae40101f50 2022-07-03T10:29:22.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.866+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fae300050f0 con 0x7fae40101f50 2022-07-03T10:29:22.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.866+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae401210c0 con 0x7fae40101f50 2022-07-03T10:29:22.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.866+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae30022b80 con 0x7fae40101f50 2022-07-03T10:29:22.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.866+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae401215c0 con 0x7fae40101f50 2022-07-03T10:29:22.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.867+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fae30029030 con 0x7fae40101f50 2022-07-03T10:29:22.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.867+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4005fb50 con 0x7fae40101f50 2022-07-03T10:29:22.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.868+0000 7fae2effd700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 0x7fae28037610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:22.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.868+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fae30047da0 con 0x7fae40101f50 2022-07-03T10:29:22.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.868+0000 7fae3dd9b700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 0x7fae28037610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:22.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.869+0000 7fae3dd9b700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 0x7fae28037610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae34009ad0 tx=0x7fae34009060 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:22.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:22.873+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fae300213d0 con 0x7fae40101f50 2022-07-03T10:29:23.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.089+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fae4010d890 con 0x7fae40101f50 2022-07-03T10:29:23.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.090+0000 7fae2effd700 1 -- 172.21.15.101:0/2944081283 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7fae30027070 con 0x7fae40101f50 2022-07-03T10:29:23.092 INFO:teuthology.orchestra.run.smithi101.stdout:# minimal ceph.conf for 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:29:23.092 INFO:teuthology.orchestra.run.smithi101.stdout:[global] 2022-07-03T10:29:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: fsid = 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:29:23.093 INFO:teuthology.orchestra.run.smithi101.stdout: mon_host = [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2022-07-03T10:29:23.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 msgr2=0x7fae28037610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:23.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 0x7fae28037610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae34009ad0 tx=0x7fae34009060 comp rx=0 tx=0).stop 2022-07-03T10:29:23.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 msgr2=0x7fae40119ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:23.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae30006010 tx=0x7fae300091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:23.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 shutdown_connections 2022-07-03T10:29:23.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae28035160 0x7fae28037610 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:23.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae40101f50 0x7fae40119ea0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:23.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae40114260 0x7fae401146d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:23.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 --2- 172.21.15.101:0/2944081283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae4011a3e0 0x7fae40113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:23.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 >> 172.21.15.101:0/2944081283 conn(0x7fae40072f40 msgr2=0x7fae400fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:23.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.092+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 shutdown_connections 2022-07-03T10:29:23.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:23.093+0000 7fae44fc5700 1 -- 172.21.15.101:0/2944081283 wait complete. 2022-07-03T10:29:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:23 smithi188 ceph-mon[92425]: Reconfiguring mon.a (unknown last config time)... 2022-07-03T10:29:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:23 smithi188 ceph-mon[92425]: Reconfiguring daemon mon.a on smithi101 2022-07-03T10:29:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:23 smithi188 ceph-mon[92425]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:23 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2944081283' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:23.465 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:23 smithi101 ceph-mon[85126]: Reconfiguring mon.a (unknown last config time)... 2022-07-03T10:29:23.465 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:23 smithi101 ceph-mon[85126]: Reconfiguring daemon mon.a on smithi101 2022-07-03T10:29:23.466 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:23 smithi101 ceph-mon[85126]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:23.466 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:23 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2944081283' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:23.569 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:23 smithi161 ceph-mon[91579]: Reconfiguring mon.a (unknown last config time)... 2022-07-03T10:29:23.570 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:23 smithi161 ceph-mon[91579]: Reconfiguring daemon mon.a on smithi101 2022-07-03T10:29:23.570 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:23 smithi161 ceph-mon[91579]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:23.570 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:23 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2944081283' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:23.725 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-07-03T10:29:23.725 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:29:23.726 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.conf 2022-07-03T10:29:23.816 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:29:23.816 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:23.905 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:29:23.906 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.conf 2022-07-03T10:29:23.934 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:29:23.935 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:24.002 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:29:24.002 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2022-07-03T10:29:24.035 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:29:24.035 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:29:24.104 INFO:tasks.cephadm:Adding mgr.x on smithi101 2022-07-03T10:29:24.105 INFO:tasks.cephadm:Adding mgr.y on smithi161 2022-07-03T10:29:24.105 INFO:tasks.cephadm:Adding mgr.z on smithi188 2022-07-03T10:29:24.106 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch apply mgr '3;smithi101=x;smithi161=y;smithi188=z' 2022-07-03T10:29:24.737 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:29:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.797 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: Reconfiguring mon.b (monmap changed)... 2022-07-03T10:29:24.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:24.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:24.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:24.799 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:24 smithi188 ceph-mon[92425]: Reconfiguring daemon mon.b on smithi161 2022-07-03T10:29:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: Reconfiguring mon.b (monmap changed)... 2022-07-03T10:29:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:24 smithi161 ceph-mon[91579]: Reconfiguring daemon mon.b on smithi161 2022-07-03T10:29:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: Reconfiguring mon.b (monmap changed)... 2022-07-03T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:24.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:24 smithi101 ceph-mon[85126]: Reconfiguring daemon mon.b on smithi161 2022-07-03T10:29:25.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:25.874 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:25.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:25.875 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:25.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-07-03T10:29:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-07-03T10:29:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.121 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 -- 172.21.15.188:0/219651049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 msgr2=0x7f3b240fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:26.122 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 --2- 172.21.15.188:0/219651049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b240fd620 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009880 tx=0x7f3b140091e0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.122 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 -- 172.21.15.188:0/219651049 shutdown_connections 2022-07-03T10:29:26.122 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 --2- 172.21.15.188:0/219651049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b240fd620 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.122 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 --2- 172.21.15.188:0/219651049 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3b240fc2e0 0x7f3b240fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.123 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 -- 172.21.15.188:0/219651049 >> 172.21.15.188:0/219651049 conn(0x7f3b240f7a50 msgr2=0x7f3b240f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:26.123 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 -- 172.21.15.188:0/219651049 shutdown_connections 2022-07-03T10:29:26.123 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.119+0000 7f3b290ea700 1 -- 172.21.15.188:0/219651049 wait complete. 2022-07-03T10:29:26.126 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 Processor -- start 2022-07-03T10:29:26.126 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 -- start start 2022-07-03T10:29:26.126 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3b240fc2e0 0x7f3b24066680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:26.127 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b24112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:26.127 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:26.127 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24115de0 con 0x7f3b240fc2e0 2022-07-03T10:29:26.127 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24115f50 con 0x7f3b240fcc30 2022-07-03T10:29:26.128 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b290ea700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24116110 con 0x7f3b24113200 2022-07-03T10:29:26.128 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:26.128 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40394/0 (socket says 172.21.15.188:40394) 2022-07-03T10:29:26.128 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 -- 172.21.15.188:0/2301345527 learned_addr learned my addr 172.21.15.188:0/2301345527 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:29:26.128 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2259c700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b24112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:26.129 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 -- 172.21.15.188:0/2301345527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 msgr2=0x7f3b24112c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:26.129 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b24112c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.129 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 -- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3b240fc2e0 msgr2=0x7f3b24066680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:26.129 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3b240fc2e0 0x7f3b24066680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.130 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.120+0000 7f3b2359e700 1 -- 172.21.15.188:0/2301345527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b14009440 con 0x7f3b24113200 2022-07-03T10:29:26.130 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b2359e700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00bc30 tx=0x7f3b1c009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:26.130 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00c900 con 0x7f3b24113200 2022-07-03T10:29:26.130 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3b1c014440 con 0x7f3b24113200 2022-07-03T10:29:26.131 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c007650 con 0x7f3b24113200 2022-07-03T10:29:26.131 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b241163f0 con 0x7f3b24113200 2022-07-03T10:29:26.131 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b24116980 con 0x7f3b24113200 2022-07-03T10:29:26.131 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.121+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f3b1c01b030 con 0x7f3b24113200 2022-07-03T10:29:26.131 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.122+0000 7f3b0b7fe700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 0x7f3b0c0374b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:26.132 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.122+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3b1c0460d0 con 0x7f3b24113200 2022-07-03T10:29:26.132 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.122+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24104c30 con 0x7f3b24113200 2022-07-03T10:29:26.132 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.122+0000 7f3b22d9d700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 0x7f3b0c0374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:26.132 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.123+0000 7f3b22d9d700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 0x7f3b0c0374b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009b00 tx=0x7f3b18005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:26.133 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.127+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f3b1c019070 con 0x7f3b24113200 2022-07-03T10:29:26.317 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.314+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi101=x;smithi161=y;smithi188=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b24101fb0 con 0x7f3b0c035000 2022-07-03T10:29:26.324 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.322+0000 7f3b0b7fe700 1 -- 172.21.15.188:0/2301345527 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3b24101fb0 con 0x7f3b0c035000 2022-07-03T10:29:26.324 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2022-07-03T10:29:26.326 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 msgr2=0x7f3b0c0374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:26.327 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 0x7f3b0c0374b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009b00 tx=0x7f3b18005ef0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.327 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 msgr2=0x7f3b24115640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:26.327 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00bc30 tx=0x7f3b1c009820 comp rx=0 tx=0).stop 2022-07-03T10:29:26.327 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 shutdown_connections 2022-07-03T10:29:26.328 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3b0c035000 0x7f3b0c0374b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.328 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3b240fc2e0 0x7f3b24066680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.328 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b24113200 0x7f3b24115640 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.328 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 --2- 172.21.15.188:0/2301345527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3b240fcc30 0x7f3b24112c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:26.328 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 >> 172.21.15.188:0/2301345527 conn(0x7f3b240f7a50 msgr2=0x7f3b24100830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:26.329 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 shutdown_connections 2022-07-03T10:29:26.329 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:29:26.324+0000 7f3b290ea700 1 -- 172.21.15.188:0/2301345527 wait complete. 2022-07-03T10:29:26.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: Reconfiguring mon.c (monmap changed)... 2022-07-03T10:29:26.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: Reconfiguring daemon mon.c on smithi188 2022-07-03T10:29:26.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:26.753 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:29:26.753 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:29:26.753 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:26.754 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:26.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:26.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:26.755 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.757 DEBUG:teuthology.orchestra.run.smithi161:mgr.y> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mgr.y.service 2022-07-03T10:29:26.760 DEBUG:teuthology.orchestra.run.smithi188:mgr.z> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mgr.z.service 2022-07-03T10:29:26.762 INFO:tasks.cephadm:Deploying OSDs... 2022-07-03T10:29:26.763 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:29:26.763 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/scratch_devs of=/dev/stdout 2022-07-03T10:29:26.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:29:26.791 INFO:journalctl@ceph.mgr.z.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:29:26.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-03T10:29:26.824 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-03T10:29:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: Reconfiguring mon.c (monmap changed)... 2022-07-03T10:29:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: Reconfiguring daemon mon.c on smithi188 2022-07-03T10:29:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:29:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:29:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.884 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-03T10:29:26.884 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:26.884 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 452 Links: 1 2022-07-03T10:29:26.885 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:26.885 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:26.885 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2022-07-03 10:28:36.276434905 +0000 2022-07-03T10:29:26.885 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2022-07-03 10:12:21.920521261 +0000 2022-07-03T10:29:26.886 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2022-07-03 10:12:21.920521261 +0000 2022-07-03T10:29:26.886 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2022-07-03 10:12:21.884522046 +0000 2022-07-03T10:29:26.907 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: Reconfiguring mon.c (monmap changed)... 2022-07-03T10:29:26.907 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: Reconfiguring daemon mon.c on smithi188 2022-07-03T10:29:26.907 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:26.907 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:29:26.908 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:29:26.908 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.908 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.908 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.909 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:26.909 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.909 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:26.909 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:26.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:26.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:26.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:26.910 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2022-07-03T10:29:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2022-07-03T10:29:26.915 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.00062286 s, 822 kB/s 2022-07-03T10:29:26.925 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-03T10:29:26.983 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-03T10:29:26.984 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:26.984 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 458 Links: 1 2022-07-03T10:29:26.984 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:26.985 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:26.985 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2022-07-03 10:28:36.276434905 +0000 2022-07-03T10:29:26.985 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2022-07-03 10:12:22.931499216 +0000 2022-07-03T10:29:26.985 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2022-07-03 10:12:22.931499216 +0000 2022-07-03T10:29:26.985 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2022-07-03 10:12:22.895500001 +0000 2022-07-03T10:29:27.013 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2022-07-03T10:29:27.014 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2022-07-03T10:29:27.014 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000654312 s, 783 kB/s 2022-07-03T10:29:27.023 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-03T10:29:27.082 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-03T10:29:27.083 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.083 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 464 Links: 1 2022-07-03T10:29:27.083 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.084 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.084 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2022-07-03 10:28:36.276434905 +0000 2022-07-03T10:29:27.085 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2022-07-03 10:12:23.940477215 +0000 2022-07-03T10:29:27.085 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2022-07-03 10:12:23.940477215 +0000 2022-07-03T10:29:27.086 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2022-07-03 10:12:23.903478021 +0000 2022-07-03T10:29:27.111 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2022-07-03T10:29:27.112 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2022-07-03T10:29:27.112 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000887268 s, 577 kB/s 2022-07-03T10:29:27.121 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-03T10:29:27.179 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-03T10:29:27.180 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.180 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 5h/5d Inode: 470 Links: 1 2022-07-03T10:29:27.180 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.180 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.181 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2022-07-03 10:28:36.276434905 +0000 2022-07-03T10:29:27.181 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2022-07-03 10:12:24.940455410 +0000 2022-07-03T10:29:27.181 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2022-07-03 10:12:24.940455410 +0000 2022-07-03T10:29:27.181 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2022-07-03 10:12:24.906456151 +0000 2022-07-03T10:29:27.209 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2022-07-03T10:29:27.209 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2022-07-03T10:29:27.209 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.00063077 s, 812 kB/s 2022-07-03T10:29:27.219 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2022-07-03T10:29:27.219 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2022-07-03T10:29:27.237 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-03T10:29:27.238 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-03T10:29:27.295 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-03T10:29:27.295 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.296 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 348288 Links: 1 2022-07-03T10:29:27.296 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.296 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.296 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-07-03 10:29:05.903204499 +0000 2022-07-03T10:29:27.297 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-07-03 10:08:53.764399215 +0000 2022-07-03T10:29:27.297 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-07-03 10:08:53.764399215 +0000 2022-07-03T10:29:27.297 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-07-03T10:29:27.306 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-07-03T10:29:27.306 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-07-03T10:29:27.306 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000213857 s, 2.4 MB/s 2022-07-03T10:29:27.309 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-03T10:29:27.366 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-03T10:29:27.366 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.366 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 341970 Links: 1 2022-07-03T10:29:27.367 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.367 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.367 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-07-03 10:29:05.904204477 +0000 2022-07-03T10:29:27.367 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-07-03 10:08:54.342387402 +0000 2022-07-03T10:29:27.368 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-07-03 10:08:54.342387402 +0000 2022-07-03T10:29:27.368 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-07-03T10:29:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-07-03T10:29:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-07-03T10:29:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000223586 s, 2.3 MB/s 2022-07-03T10:29:27.379 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-03T10:29:27.437 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-03T10:29:27.438 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.438 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 346496 Links: 1 2022-07-03T10:29:27.438 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.438 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.439 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-07-03 10:29:05.904204477 +0000 2022-07-03T10:29:27.439 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-07-03 10:08:54.971374546 +0000 2022-07-03T10:29:27.439 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-07-03 10:08:54.971374546 +0000 2022-07-03T10:29:27.439 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-07-03T10:29:27.449 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-07-03T10:29:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-07-03T10:29:27.450 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000186472 s, 2.7 MB/s 2022-07-03T10:29:27.453 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-03T10:29:27.509 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-03T10:29:27.510 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.510 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 6h/6d Inode: 351396 Links: 1 2022-07-03T10:29:27.510 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.511 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.511 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2022-07-03 10:29:05.904204477 +0000 2022-07-03T10:29:27.511 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2022-07-03 10:08:55.540362916 +0000 2022-07-03T10:29:27.511 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2022-07-03 10:08:55.540362916 +0000 2022-07-03T10:29:27.511 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2022-07-03T10:29:27.519 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2022-07-03T10:29:27.520 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2022-07-03T10:29:27.520 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000226186 s, 2.3 MB/s 2022-07-03T10:29:27.522 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2022-07-03T10:29:27.523 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2022-07-03T10:29:27.543 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-03T10:29:27.543 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-03T10:29:27.601 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-03T10:29:27.602 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.602 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 347328 Links: 1 2022-07-03T10:29:27.602 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.603 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.603 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2022-07-03 10:29:14.053869025 +0000 2022-07-03T10:29:27.603 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2022-07-03 10:08:52.199136382 +0000 2022-07-03T10:29:27.603 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2022-07-03 10:08:52.199136382 +0000 2022-07-03T10:29:27.603 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2022-07-03T10:29:27.612 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2022-07-03T10:29:27.613 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2022-07-03T10:29:27.613 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000242189 s, 2.1 MB/s 2022-07-03T10:29:27.616 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-03T10:29:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-03T10:29:27.674 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.675 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 346251 Links: 1 2022-07-03T10:29:27.675 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.675 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.675 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2022-07-03 10:29:14.053869025 +0000 2022-07-03T10:29:27.676 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2022-07-03 10:08:52.745126899 +0000 2022-07-03T10:29:27.676 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2022-07-03 10:08:52.745126899 +0000 2022-07-03T10:29:27.676 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2022-07-03T10:29:27.685 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2022-07-03T10:29:27.685 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2022-07-03T10:29:27.685 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000247191 s, 2.1 MB/s 2022-07-03T10:29:27.688 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-03T10:29:27.746 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-03T10:29:27.747 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.747 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 346335 Links: 1 2022-07-03T10:29:27.747 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.747 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.747 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2022-07-03 10:29:14.053869025 +0000 2022-07-03T10:29:27.748 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2022-07-03 10:08:53.292117399 +0000 2022-07-03T10:29:27.748 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2022-07-03 10:08:53.292117399 +0000 2022-07-03T10:29:27.748 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2022-07-03T10:29:27.757 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2022-07-03T10:29:27.757 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2022-07-03T10:29:27.757 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000187 s, 2.7 MB/s 2022-07-03T10:29:27.759 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:27 smithi161 ceph-mon[91579]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:27.759 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:27 smithi161 ceph-mon[91579]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi101=x;smithi161=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:27.759 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:27 smithi161 ceph-mon[91579]: Saving service mgr spec with placement smithi101=x;smithi161=y;smithi188=z;count:3 2022-07-03T10:29:27.759 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:27 smithi161 ceph-mon[91579]: Deploying daemon mgr.y on smithi161 2022-07-03T10:29:27.760 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-03T10:29:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-03T10:29:27.818 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-03T10:29:27.818 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 350329 Links: 1 2022-07-03T10:29:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:29:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2022-07-03T10:29:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2022-07-03 10:29:14.053869025 +0000 2022-07-03T10:29:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2022-07-03 10:08:53.843107830 +0000 2022-07-03T10:29:27.820 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2022-07-03 10:08:53.843107830 +0000 2022-07-03T10:29:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2022-07-03T10:29:27.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:27 smithi188 ceph-mon[92425]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:27.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:27 smithi188 ceph-mon[92425]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi101=x;smithi161=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:27.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:27 smithi188 ceph-mon[92425]: Saving service mgr spec with placement smithi101=x;smithi161=y;smithi188=z;count:3 2022-07-03T10:29:27.826 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:27 smithi188 ceph-mon[92425]: Deploying daemon mgr.y on smithi161 2022-07-03T10:29:27.828 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2022-07-03T10:29:27.828 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2022-07-03T10:29:27.829 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000220734 s, 2.3 MB/s 2022-07-03T10:29:27.831 INFO:tasks.cephadm:Deploying osd.0 on smithi101 with /dev/vg_nvme/lv_4... 2022-07-03T10:29:27.832 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-07-03T10:29:27.913 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:27 smithi101 ceph-mon[85126]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:27.913 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:27 smithi101 ceph-mon[85126]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi101=x;smithi161=y;smithi188=z", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:27.913 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:27 smithi101 ceph-mon[85126]: Saving service mgr spec with placement smithi101=x;smithi161=y;smithi188=z;count:3 2022-07-03T10:29:27.914 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:27 smithi101 ceph-mon[85126]: Deploying daemon mgr.y on smithi161 2022-07-03T10:29:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:27 smithi161 systemd[1]: Starting Ceph mgr.y for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:29:28.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 podman[92855]: 2022-07-03T10:29:28.714 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:29:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 bash[92855]: 66d37cba56fdd8aab4224a75497841741787eedefeb632dcd3c1313e3821db95 2022-07-03T10:29:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 ceph-mgr[92895]: -- 172.21.15.161:0/3646211165 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565524a4340 con 0x556552479000 2022-07-03T10:29:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 systemd[1]: Started Ceph mgr.y for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:29:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:28.631+0000 7f72e826d080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-03T10:29:29.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:28.921+0000 7f72e826d080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-03T10:29:29.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.083+0000 7f72e826d080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-03T10:29:29.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.186+0000 7f72e826d080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-03T10:29:29.450 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.265+0000 7f72e826d080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-03T10:29:29.450 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.371+0000 7f72e826d080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-03T10:29:29.450 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.447+0000 7f72e826d080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-03T10:29:29.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.574+0000 7f72e826d080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-03T10:29:29.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.662+0000 7f72e826d080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-03T10:29:29.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:29.828+0000 7f72e826d080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-03T10:29:29.832 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:29.832 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.832 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.833 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:29.833 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:29.833 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:29.833 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:29.834 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:29 smithi161 ceph-mon[91579]: Deploying daemon mgr.z on smithi188 2022-07-03T10:29:29.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:29.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:29.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:29.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:29.880 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:29.881 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:29 smithi188 ceph-mon[92425]: Deploying daemon mgr.z on smithi188 2022-07-03T10:29:29.896 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:29.897 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.897 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:29.897 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:29.898 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-07-03T10:29:29.898 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:29.898 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:29.898 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:29 smithi101 ceph-mon[85126]: Deploying daemon mgr.z on smithi188 2022-07-03T10:29:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:30.040+0000 7f72e826d080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:29:30.164 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 systemd[1]: Starting Ceph mgr.z for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:29:30.663 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-07-03T10:29:30.664 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-07-03T10:29:30.664 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2022-07-03T10:29:30.664 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2022-07-03T10:29:30.664 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0439095 s, 239 MB/s 2022-07-03T10:29:30.665 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2022-07-03T10:29:30.665 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 podman[94021]: 2022-07-03T10:29:30.665 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 bash[94021]: 5f6cd708a843e0b47b9b8e32a7f8c0ec147072f6822702d0cd238a99a3727187 2022-07-03T10:29:30.666 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 ceph-mgr[94061]: -- 172.21.15.188:0/1221348110 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2f5376340 con 0x55b2f534b000 2022-07-03T10:29:30.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:30.590+0000 7f72e826d080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-03T10:29:30.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:30.676+0000 7f72e826d080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-03T10:29:31.051 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 systemd[1]: Started Ceph mgr.z for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:29:31.051 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:30 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:30.735+0000 7f92230b2080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-07-03T10:29:31.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:30.852+0000 7f72e826d080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-03T10:29:31.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:30.946+0000 7f72e826d080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-03T10:29:31.190 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_4 2022-07-03T10:29:31.333 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.049+0000 7f92230b2080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-07-03T10:29:31.334 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.220+0000 7f92230b2080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-07-03T10:29:31.607 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.329+0000 7f92230b2080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-07-03T10:29:31.607 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.412+0000 7f92230b2080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-07-03T10:29:31.608 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.525+0000 7f92230b2080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-07-03T10:29:31.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:31.326+0000 7f72e826d080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:29:31.913 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.604+0000 7f92230b2080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-07-03T10:29:31.914 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.744+0000 7f92230b2080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-07-03T10:29:31.914 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:31.839+0000 7f92230b2080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-07-03T10:29:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:31.748+0000 7f72e826d080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-03T10:29:31.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:31.846+0000 7f72e826d080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-03T10:29:31.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:31.944+0000 7f72e826d080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-03T10:29:31.948 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:31.948 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:31.948 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.948 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:31.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:31.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:31.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:32.129 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:29:32.414 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:32 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:32.029+0000 7f92230b2080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-07-03T10:29:32.414 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:32 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:32.255+0000 7f92230b2080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:29:32.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:32.554+0000 7f72e826d080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-03T10:29:32.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:32.648+0000 7f72e826d080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-03T10:29:33.067 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:32 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:32.808+0000 7f92230b2080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-07-03T10:29:33.067 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:32 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:32.894+0000 7f92230b2080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-07-03T10:29:33.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:32.897+0000 7f72e826d080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-03T10:29:33.364 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:33.064+0000 7f92230b2080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-07-03T10:29:33.364 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:33.152+0000 7f92230b2080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-07-03T10:29:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: Reconfiguring mgr.x (unknown last config time)... 2022-07-03T10:29:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-mon[91579]: Reconfiguring daemon mgr.x on smithi101 2022-07-03T10:29:33.664 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:33.511+0000 7f92230b2080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-07-03T10:29:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: Reconfiguring mgr.x (unknown last config time)... 2022-07-03T10:29:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-mon[92425]: Reconfiguring daemon mgr.x on smithi101 2022-07-03T10:29:33.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: Reconfiguring mgr.x (unknown last config time)... 2022-07-03T10:29:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-07-03T10:29:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-07-03T10:29:33.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:33.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:33 smithi101 ceph-mon[85126]: Reconfiguring daemon mgr.x on smithi101 2022-07-03T10:29:33.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 -- 172.21.15.101:0/98774734 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580ff0d0 msgr2=0x7f9358105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:33.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 --2- 172.21.15.101:0/98774734 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580ff0d0 0x7f9358105990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9350009ca0 tx=0x7f9350009520 comp rx=0 tx=0).stop 2022-07-03T10:29:33.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 -- 172.21.15.101:0/98774734 shutdown_connections 2022-07-03T10:29:33.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 --2- 172.21.15.101:0/98774734 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580ff0d0 0x7f9358105990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:33.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 --2- 172.21.15.101:0/98774734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fd4c0 0x7f93580fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:33.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 --2- 172.21.15.101:0/98774734 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580fe740 0x7f93580feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:33.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.768+0000 7f935759e700 1 -- 172.21.15.101:0/98774734 >> 172.21.15.101:0/98774734 conn(0x7f93580f8c30 msgr2=0x7f93580fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:33.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.769+0000 7f935759e700 1 -- 172.21.15.101:0/98774734 shutdown_connections 2022-07-03T10:29:33.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.769+0000 7f935759e700 1 -- 172.21.15.101:0/98774734 wait complete. 2022-07-03T10:29:33.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 Processor -- start 2022-07-03T10:29:33.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 -- start start 2022-07-03T10:29:33.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580fd4c0 0x7f93581103d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:33.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:33.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580ff0d0 0x7f9358117980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:33.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358117f00 con 0x7f93580fd4c0 2022-07-03T10:29:33.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358118040 con 0x7f93580ff0d0 2022-07-03T10:29:33.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935759e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9358118200 con 0x7f93580fe740 2022-07-03T10:29:33.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f935659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580fd4c0 0x7f93581103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:33.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.770+0000 7f9355d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:33.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9356d9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580ff0d0 0x7f9358117980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:33.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:41846/0 (socket says 172.21.15.101:41846) 2022-07-03T10:29:33.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 -- 172.21.15.101:0/3492956521 learned_addr learned my addr 172.21.15.101:0/3492956521 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:29:33.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 -- 172.21.15.101:0/3492956521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580ff0d0 msgr2=0x7f9358117980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:33.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580ff0d0 0x7f9358117980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:33.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 -- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580fd4c0 msgr2=0x7f93581103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:33.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580fd4c0 0x7f93581103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:33.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 -- 172.21.15.101:0/3492956521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c009410 con 0x7f93580fe740 2022-07-03T10:29:33.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9355d9b700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f934c00ea80 tx=0x7f934c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:33.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.771+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c009690 con 0x7f93580fe740 2022-07-03T10:29:33.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.772+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93500097e0 con 0x7f93580fe740 2022-07-03T10:29:33.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.772+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f934c00c8f0 con 0x7f93580fe740 2022-07-03T10:29:33.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.772+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c003a50 con 0x7f93580fe740 2022-07-03T10:29:33.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.772+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93581188e0 con 0x7f93580fe740 2022-07-03T10:29:33.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.773+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f934c00c040 con 0x7f93580fe740 2022-07-03T10:29:33.780 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.773+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9358108cb0 con 0x7f93580fe740 2022-07-03T10:29:33.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.774+0000 7f9346ffd700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 0x7f9340037380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:29:33.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.774+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f934c014070 con 0x7f93580fe740 2022-07-03T10:29:33.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.774+0000 7f935659c700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 0x7f9340037380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:29:33.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.774+0000 7f935659c700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 0x7f9340037380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93580fe5a0 tx=0x7f9348005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:29:33.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.782+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f934c00cdb0 con 0x7f93580fe740 2022-07-03T10:29:33.993 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:33.992+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9358071e70 con 0x7f9340034ed0 2022-07-03T10:29:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:33.829+0000 7f72e826d080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-03T10:29:34.163 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:33.907+0000 7f92230b2080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-07-03T10:29:34.164 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:33 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:33.991+0000 7f92230b2080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-07-03T10:29:34.164 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:34.075+0000 7f92230b2080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-07-03T10:29:34.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:29:34.425+0000 7f72e826d080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-03T10:29:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:29:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:29:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.659 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:29:34.659 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:29:34.659 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:34.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:29:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:29:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:34.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:34.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:34.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:34.914 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:34.656+0000 7f92230b2080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-07-03T10:29:34.914 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:34.750+0000 7f92230b2080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-07-03T10:29:35.373 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:34 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:34.997+0000 7f92230b2080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-07-03T10:29:35.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:35.383+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mgrmap(e 14) v1 ==== 83495+0+0 (secure 0 0 0) 0x7f934c0213f0 con 0x7f93580fe740 2022-07-03T10:29:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-07-03T10:29:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: Standby manager daemon y started 2022-07-03T10:29:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-07-03T10:29:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:35 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-07-03T10:29:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: Standby manager daemon y started 2022-07-03T10:29:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-07-03T10:29:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:35.676 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:35.676 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:29:35.676 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-07-03T10:29:35.677 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: Standby manager daemon y started 2022-07-03T10:29:35.677 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:35.677 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-07-03T10:29:35.677 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:35 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:36.385 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:35 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:35.925+0000 7f92230b2080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-07-03T10:29:36.664 INFO:journalctl@ceph.mgr.z.smithi188.stdout:Jul 03 10:29:36 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-z[94037]: 2022-07-03T10:29:36.515+0000 7f92230b2080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-07-03T10:29:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:36 smithi188 ceph-mon[92425]: mgrmap e14: x(active, since 69s), standbys: y 2022-07-03T10:29:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-07-03T10:29:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:36.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:36 smithi101 ceph-mon[85126]: mgrmap e14: x(active, since 69s), standbys: y 2022-07-03T10:29:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-07-03T10:29:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:36 smithi161 ceph-mon[91579]: mgrmap e14: x(active, since 69s), standbys: y 2022-07-03T10:29:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-07-03T10:29:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:37.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:37.396+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mon.2 v2:172.21.15.188:3300/0 8 ==== mgrmap(e 15) v1 ==== 125092+0+0 (secure 0 0 0) 0x7f934c004020 con 0x7f93580fe740 2022-07-03T10:29:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: Standby manager daemon z started 2022-07-03T10:29:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-07-03T10:29:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-07-03T10:29:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]: dispatch 2022-07-03T10:29:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]': finished 2022-07-03T10:29:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: osdmap e5: 1 total, 0 up, 1 in 2022-07-03T10:29:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:29:37.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:37.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: Standby manager daemon z started 2022-07-03T10:29:37.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-07-03T10:29:37.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:37.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-07-03T10:29:37.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:37.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]: dispatch 2022-07-03T10:29:37.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]': finished 2022-07-03T10:29:37.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: osdmap e5: 1 total, 0 up, 1 in 2022-07-03T10:29:37.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:29:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: Standby manager daemon z started 2022-07-03T10:29:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-07-03T10:29:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-07-03T10:29:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-07-03T10:29:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='mgr.? 172.21.15.188:0/4036426647' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-07-03T10:29:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]: dispatch 2022-07-03T10:29:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2876146895' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6b433d6-6618-4afa-9a41-a4dbde9dfd3d"}]': finished 2022-07-03T10:29:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: osdmap e5: 1 total, 0 up, 1 in 2022-07-03T10:29:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:29:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:38 smithi188 ceph-mon[92425]: mgrmap e15: x(active, since 71s), standbys: y, z 2022-07-03T10:29:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-07-03T10:29:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:38 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3548189897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:29:38.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:38 smithi101 ceph-mon[85126]: mgrmap e15: x(active, since 71s), standbys: y, z 2022-07-03T10:29:38.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-07-03T10:29:38.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:38 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3548189897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:29:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:38 smithi161 ceph-mon[91579]: mgrmap e15: x(active, since 71s), standbys: y, z 2022-07-03T10:29:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-07-03T10:29:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:38 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3548189897' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:29:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:39 smithi188 ceph-mon[92425]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:39.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:39 smithi101 ceph-mon[85126]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:39 smithi161 ceph-mon[91579]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:41 smithi188 ceph-mon[92425]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:41.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:41 smithi101 ceph-mon[85126]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:41.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:41 smithi161 ceph-mon[91579]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:42.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:42 smithi101 ceph-mon[85126]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:42 smithi161 ceph-mon[91579]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:42 smithi188 ceph-mon[92425]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:45.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:44 smithi101 ceph-mon[85126]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:44 smithi161 ceph-mon[91579]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:44 smithi188 ceph-mon[92425]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:46.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-07-03T10:29:46.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:46.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:45 smithi101 ceph-mon[85126]: Deploying daemon osd.0 on smithi101 2022-07-03T10:29:46.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-07-03T10:29:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:45 smithi161 ceph-mon[91579]: Deploying daemon osd.0 on smithi101 2022-07-03T10:29:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-07-03T10:29:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:45 smithi188 ceph-mon[92425]: Deploying daemon osd.0 on smithi101 2022-07-03T10:29:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:47.162 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:46 smithi101 ceph-mon[85126]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:46 smithi161 ceph-mon[91579]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:46 smithi188 ceph-mon[92425]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:49.054 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:48 smithi101 ceph-mon[85126]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:48 smithi161 ceph-mon[91579]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:48 smithi188 ceph-mon[92425]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:51.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:50 smithi101 ceph-mon[85126]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:51.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:50 smithi161 ceph-mon[91579]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:50 smithi188 ceph-mon[92425]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:53.723 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:53.723 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.724 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:53.724 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:53.724 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:53.724 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:55 smithi161 ceph-mon[91579]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:55 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:55 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:55 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:55 smithi188 ceph-mon[92425]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:55 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:55 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:55 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:55 smithi101 ceph-mon[85126]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:55 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.356 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.354+0000 7f9346ffd700 1 -- 172.21.15.101:0/3492956521 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9358071e70 con 0x7f9340034ed0 2022-07-03T10:29:56.357 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 0 on host 'smithi101' 2022-07-03T10:29:56.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.357+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 msgr2=0x7f9340037380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:56.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.357+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 0x7f9340037380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93580fe5a0 tx=0x7f9348005ef0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.357+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 msgr2=0x7f9358110910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:29:56.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.357+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f934c00ea80 tx=0x7f934c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 shutdown_connections 2022-07-03T10:29:56.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9340034ed0 0x7f9340037380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93580fd4c0 0x7f93581103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93580ff0d0 0x7f9358117980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 --2- 172.21.15.101:0/3492956521 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93580fe740 0x7f9358110910 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:29:56.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 >> 172.21.15.101:0/3492956521 conn(0x7f93580f8c30 msgr2=0x7f9358070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:29:56.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 shutdown_connections 2022-07-03T10:29:56.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:29:56.358+0000 7f935759e700 1 -- 172.21.15.101:0/3492956521 wait complete. 2022-07-03T10:29:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.726 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:29:56.726 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.726 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:29:56.726 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:29:56.727 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:29:56.727 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:29:56.756 DEBUG:teuthology.orchestra.run.smithi101:osd.0> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.0.service 2022-07-03T10:29:56.762 INFO:tasks.cephadm:Deploying osd.1 on smithi161 with /dev/vg_nvme/lv_4... 2022-07-03T10:29:56.762 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-07-03T10:29:56.847 INFO:journalctl@ceph.osd.0.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:29:57.361 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:29:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:57 smithi161 ceph-mon[91579]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:57 smithi188 ceph-mon[92425]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:57.708 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:57 smithi101 ceph-mon[85126]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-07-03T10:29:58.980 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-07-03T10:29:58.981 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2022-07-03T10:29:58.981 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-07-03T10:29:58.981 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0157366 s, 666 MB/s 2022-07-03T10:29:58.981 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2022-07-03T10:29:59.393 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi161:vg_nvme/lv_4 2022-07-03T10:29:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:29:59 smithi161 ceph-mon[91579]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:29:59 smithi188 ceph-mon[92425]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:59.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:29:59 smithi101 ceph-mon[85126]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:29:59.989 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:30:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:00 smithi161 ceph-mon[91579]: overall HEALTH_OK 2022-07-03T10:30:00.636 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:00 smithi101 ceph-mon[85126]: overall HEALTH_OK 2022-07-03T10:30:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:00 smithi188 ceph-mon[92425]: overall HEALTH_OK 2022-07-03T10:30:01.269 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.260+0000 7f24e807f700 1 -- 172.21.15.161:0/823318539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 msgr2=0x7f24e00fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:01.269 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.260+0000 7f24e807f700 1 --2- 172.21.15.161:0/823318539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fcd10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009900 tx=0x7f24dc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.269 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 -- 172.21.15.161:0/823318539 shutdown_connections 2022-07-03T10:30:01.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 --2- 172.21.15.161:0/823318539 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e00fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.270 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 --2- 172.21.15.161:0/823318539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.271 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 --2- 172.21.15.161:0/823318539 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fcd10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.271 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 -- 172.21.15.161:0/823318539 >> 172.21.15.161:0/823318539 conn(0x7f24e00f7480 msgr2=0x7f24e00f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:30:01.272 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 -- 172.21.15.161:0/823318539 shutdown_connections 2022-07-03T10:30:01.272 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 -- 172.21.15.161:0/823318539 wait complete. 2022-07-03T10:30:01.273 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 Processor -- start 2022-07-03T10:30:01.273 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 -- start start 2022-07-03T10:30:01.274 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:01.274 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.261+0000 7f24e807f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0113410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:01.275 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e0115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:01.275 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0116120 con 0x7f24e00fdb70 2022-07-03T10:30:01.275 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0116290 con 0x7f24e00fc8a0 2022-07-03T10:30:01.276 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0116450 con 0x7f24e00fd250 2022-07-03T10:30:01.276 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:01.277 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48268/0 (socket says 172.21.15.161:48268) 2022-07-03T10:30:01.277 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 -- 172.21.15.161:0/4014542698 learned_addr learned my addr 172.21.15.161:0/4014542698 (peer_addr_for_me v2:172.21.15.161:0/0) 2022-07-03T10:30:01.278 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e561a700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0113410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:01.278 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e661c700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e0115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:01.278 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 -- 172.21.15.161:0/4014542698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 msgr2=0x7f24e0113410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:01.278 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0113410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.279 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 -- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 msgr2=0x7f24e0115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:01.279 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e0115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:01.279 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 -- 172.21.15.161:0/4014542698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc009440 con 0x7f24e00fc8a0 2022-07-03T10:30:01.280 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e561a700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0113410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:30:01.280 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e661c700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e0115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:30:01.280 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e5e1b700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d7e0 tx=0x7f24d000b3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:01.280 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0004020 con 0x7f24e00fc8a0 2022-07-03T10:30:01.280 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e0116730 con 0x7f24e00fc8a0 2022-07-03T10:30:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f24d0004180 con 0x7f24e00fc8a0 2022-07-03T10:30:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d00039d0 con 0x7f24e00fc8a0 2022-07-03T10:30:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.262+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e0116cc0 con 0x7f24e00fc8a0 2022-07-03T10:30:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.263+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e0107cd0 con 0x7f24e00fc8a0 2022-07-03T10:30:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.264+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) v1 ==== 125092+0+0 (secure 0 0 0) 0x7f24d0022070 con 0x7f24e00fc8a0 2022-07-03T10:30:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.265+0000 7f24d67fc700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 0x7f24cc099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.265+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1266+0+0 (secure 0 0 0) 0x7f24d0016030 con 0x7f24e00fc8a0 2022-07-03T10:30:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.266+0000 7f24e561a700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 0x7f24cc099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.266+0000 7f24e561a700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 0x7f24cc099180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00b5c0 tx=0x7f24dc01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:01.283 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.269+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f24d008cbc0 con 0x7f24e00fc8a0 2022-07-03T10:30:01.452 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:01.451+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f24e005e960 con 0x7f24cc096cd0 2022-07-03T10:30:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:01.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:01.745 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:01 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-0[96390]: 2022-07-03T10:30:01.605+0000 7fd3d1c3d440 -1 osd.0 0 log_to_monitors true 2022-07-03T10:30:02.578 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi101 2022-07-03T10:30:02.578 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi101 to 17271M 2022-07-03T10:30:02.578 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:02.578 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:02.579 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:02.579 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:02.579 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:02.580 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:02 smithi161 ceph-mon[91579]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi101 2022-07-03T10:30:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi101 to 17271M 2022-07-03T10:30:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:02 smithi188 ceph-mon[92425]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:02.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi101 2022-07-03T10:30:02.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi101 to 17271M 2022-07-03T10:30:02.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:02.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:02.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:02.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:02.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:02.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:02 smithi101 ceph-mon[85126]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-07-03T10:30:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-07-03T10:30:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: osdmap e6: 1 total, 0 up, 1 in 2022-07-03T10:30:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:03 smithi188 ceph-mon[92425]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:03.743 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-0[96390]: 2022-07-03T10:30:03.400+0000 7fd3c57e2700 -1 osd.0 0 waiting for initial osdmap 2022-07-03T10:30:03.743 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-0[96390]: 2022-07-03T10:30:03.405+0000 7fd3c217c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:30:03.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:03.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-07-03T10:30:03.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: osdmap e6: 1 total, 0 up, 1 in 2022-07-03T10:30:03.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:03.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:03.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:03 smithi101 ceph-mon[85126]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-07-03T10:30:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: osdmap e6: 1 total, 0 up, 1 in 2022-07-03T10:30:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: from='osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:03 smithi161 ceph-mon[91579]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:30:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:30:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: osdmap e7: 1 total, 0 up, 1 in 2022-07-03T10:30:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/2621584934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] boot 2022-07-03T10:30:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]': finished 2022-07-03T10:30:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: osdmap e8: 2 total, 1 up, 2 in 2022-07-03T10:30:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:04 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/1725400411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:04.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:30:04.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: osdmap e7: 1 total, 0 up, 1 in 2022-07-03T10:30:04.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/2621584934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] boot 2022-07-03T10:30:04.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]': finished 2022-07-03T10:30:04.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: osdmap e8: 2 total, 1 up, 2 in 2022-07-03T10:30:04.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:04.748 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:04 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/1725400411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:30:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: osdmap e7: 1 total, 0 up, 1 in 2022-07-03T10:30:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/2621584934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]: dispatch 2022-07-03T10:30:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: osd.0 [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] boot 2022-07-03T10:30:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59dc2862-3622-479a-a620-87a0f8de138b"}]': finished 2022-07-03T10:30:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: osdmap e8: 2 total, 1 up, 2 in 2022-07-03T10:30:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:30:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:04 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/1725400411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:05 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:30:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:05 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:30:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:05 smithi188 ceph-mon[92425]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:05.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:05 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:30:05.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:05 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:30:05.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:05 smithi101 ceph-mon[85126]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:05 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:30:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:05 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:30:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:05 smithi161 ceph-mon[91579]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-07-03T10:30:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:06 smithi188 ceph-mon[92425]: osdmap e9: 2 total, 1 up, 2 in 2022-07-03T10:30:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:06 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:06 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:06.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:06 smithi101 ceph-mon[85126]: osdmap e9: 2 total, 1 up, 2 in 2022-07-03T10:30:06.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:06 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:06.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:06 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:06 smithi161 ceph-mon[91579]: osdmap e9: 2 total, 1 up, 2 in 2022-07-03T10:30:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:07 smithi188 ceph-mon[92425]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:07.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:07 smithi101 ceph-mon[85126]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:07.789 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:07 smithi161 ceph-mon[91579]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:09.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:09 smithi101 ceph-mon[85126]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:09.869 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:09 smithi161 ceph-mon[91579]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:09 smithi188 ceph-mon[92425]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-07-03T10:30:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:10.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-07-03T10:30:10.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-07-03T10:30:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:11 smithi161 ceph-mon[91579]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:11 smithi161 ceph-mon[91579]: Deploying daemon osd.1 on smithi161 2022-07-03T10:30:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:11.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:11 smithi101 ceph-mon[85126]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:11.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:11 smithi101 ceph-mon[85126]: Deploying daemon osd.1 on smithi161 2022-07-03T10:30:11.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:11 smithi188 ceph-mon[92425]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:11 smithi188 ceph-mon[92425]: Deploying daemon osd.1 on smithi161 2022-07-03T10:30:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:12.968 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:12 smithi161 ceph-mon[91579]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:12.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:12 smithi101 ceph-mon[85126]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:12 smithi188 ceph-mon[92425]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:15.567 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:15.568 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.568 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.568 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:15.568 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:15.569 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:15.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:15.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:15.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:15.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:15.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:16 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:16 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:16.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:16 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:17.223 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.220+0000 7f24d67fc700 1 -- 172.21.15.161:0/4014542698 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f24e005e960 con 0x7f24cc096cd0 2022-07-03T10:30:17.223 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 1 on host 'smithi161' 2022-07-03T10:30:17.225 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 msgr2=0x7f24cc099180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:17.225 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 0x7f24cc099180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00b5c0 tx=0x7f24dc01a040 comp rx=0 tx=0).stop 2022-07-03T10:30:17.225 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 msgr2=0x7f24e00fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d7e0 tx=0x7f24d000b3a0 comp rx=0 tx=0).stop 2022-07-03T10:30:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 shutdown_connections 2022-07-03T10:30:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f24cc096cd0 0x7f24cc099180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24e00fdb70 0x7f24e0115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:17.227 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24e00fc8a0 0x7f24e00fc260 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:17.227 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 --2- 172.21.15.161:0/4014542698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f24e00fd250 0x7f24e0113410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:17.227 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 >> 172.21.15.161:0/4014542698 conn(0x7f24e00f7480 msgr2=0x7f24e0100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:30:17.227 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 shutdown_connections 2022-07-03T10:30:17.228 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:30:17.223+0000 7f24e807f700 1 -- 172.21.15.161:0/4014542698 wait complete. 2022-07-03T10:30:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:17 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:17.650 DEBUG:teuthology.orchestra.run.smithi161:osd.1> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.1.service 2022-07-03T10:30:17.652 INFO:tasks.cephadm:Deploying osd.2 on smithi188 with /dev/vg_nvme/lv_4... 2022-07-03T10:30:17.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-07-03T10:30:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:17 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:17.681 INFO:journalctl@ceph.osd.1.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:30:17.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:17.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:17.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:17.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:17 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:18.258 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:30:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:19 smithi188 ceph-mon[92425]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:19.736 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-07-03T10:30:19.736 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-07-03T10:30:19.736 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2022-07-03T10:30:19.736 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2022-07-03T10:30:19.737 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110861 s, 946 MB/s 2022-07-03T10:30:19.737 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2022-07-03T10:30:19.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:19 smithi101 ceph-mon[85126]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:19.766 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:19 smithi161 ceph-mon[91579]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:20.389 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2022-07-03T10:30:21.022 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:30:21.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:21.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi161 2022-07-03T10:30:21.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:21.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi161 to 17168M 2022-07-03T10:30:21.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:21.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:21.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T10:30:21.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:21.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 17168M 2022-07-03T10:30:21.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T10:30:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 17168M 2022-07-03T10:30:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:22.158 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 -- 172.21.15.188:0/1066771821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fc5d0 msgr2=0x7f8c700fc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:22.158 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 --2- 172.21.15.188:0/1066771821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fc5d0 0x7f8c700fc9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009900 tx=0x7f8c6c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.159 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 -- 172.21.15.188:0/1066771821 shutdown_connections 2022-07-03T10:30:22.159 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 --2- 172.21.15.188:0/1066771821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 0x7f8c70104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.159 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 --2- 172.21.15.188:0/1066771821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fc5d0 0x7f8c700fc9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.160 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 --2- 172.21.15.188:0/1066771821 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8c700fd850 0x7f8c700fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.160 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.153+0000 7f8c777cc700 1 -- 172.21.15.188:0/1066771821 >> 172.21.15.188:0/1066771821 conn(0x7f8c700f7d40 msgr2=0x7f8c700fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:30:22.160 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- 172.21.15.188:0/1066771821 shutdown_connections 2022-07-03T10:30:22.161 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- 172.21.15.188:0/1066771821 wait complete. 2022-07-03T10:30:22.161 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 Processor -- start 2022-07-03T10:30:22.162 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- start start 2022-07-03T10:30:22.162 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8c700fc5d0 0x7f8c70066980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:22.162 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:22.163 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 0x7f8c70115a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:22.163 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c701161d0 con 0x7f8c700fc5d0 2022-07-03T10:30:22.164 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70116340 con 0x7f8c700fe1e0 2022-07-03T10:30:22.164 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c777cc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70116500 con 0x7f8c700fd850 2022-07-03T10:30:22.164 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c74d67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:22.165 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c74d67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40446/0 (socket says 172.21.15.188:40446) 2022-07-03T10:30:22.165 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.154+0000 7f8c74d67700 1 -- 172.21.15.188:0/2011790187 learned_addr learned my addr 172.21.15.188:0/2011790187 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:30:22.166 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c75d69700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 0x7f8c70115a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:22.166 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 -- 172.21.15.188:0/2011790187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 msgr2=0x7f8c70115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:22.167 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 0x7f8c70115a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.167 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 -- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8c700fc5d0 msgr2=0x7f8c70066980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:22.167 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8c700fc5d0 0x7f8c70066980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:22.167 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 -- 172.21.15.188:0/2011790187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c009440 con 0x7f8c700fd850 2022-07-03T10:30:22.168 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c74d67700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000b5e0 tx=0x7f8c60009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:22.168 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60011630 con 0x7f8c700fd850 2022-07-03T10:30:22.168 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8c60011c40 con 0x7f8c700fd850 2022-07-03T10:30:22.168 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c701167e0 con 0x7f8c700fd850 2022-07-03T10:30:22.168 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60010cf0 con 0x7f8c700fd850 2022-07-03T10:30:22.169 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.155+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c70116dc0 con 0x7f8c700fd850 2022-07-03T10:30:22.170 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.156+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c70107da0 con 0x7f8c700fd850 2022-07-03T10:30:22.170 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.157+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 15) v1 ==== 125092+0+0 (secure 0 0 0) 0x7f8c60021070 con 0x7f8c700fd850 2022-07-03T10:30:22.170 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.157+0000 7f8c65ffb700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 0x7f8c5c099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:22.171 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.158+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1878+0+0 (secure 0 0 0) 0x7f8c600bd8e0 con 0x7f8c700fd850 2022-07-03T10:30:22.171 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.158+0000 7f8c75568700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 0x7f8c5c099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:22.171 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.159+0000 7f8c75568700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 0x7f8c5c099280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009fd0 tx=0x7f8c6c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:22.171 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.162+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f8c6008c440 con 0x7f8c700fd850 2022-07-03T10:30:22.346 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:22.344+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c7005e960 con 0x7f8c5c096dd0 2022-07-03T10:30:22.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:22 smithi101 ceph-mon[85126]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:22.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:22 smithi101 ceph-mon[85126]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:22.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:22.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:22.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:22 smithi161 ceph-mon[91579]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:22 smithi161 ceph-mon[91579]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:23.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:23.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:22 smithi188 ceph-mon[92425]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:22 smithi188 ceph-mon[92425]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:23.696 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-1[96064]: 2022-07-03T10:30:23.383+0000 7fa14abeb440 -1 osd.1 0 log_to_monitors true 2022-07-03T10:30:23.792 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:23 smithi188 ceph-mon[92425]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:23.792 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:23 smithi188 ceph-mon[92425]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:23.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:23 smithi101 ceph-mon[85126]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:23.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:23 smithi101 ceph-mon[85126]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:23 smithi161 ceph-mon[91579]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:23 smithi161 ceph-mon[91579]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-07-03T10:30:24.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-07-03T10:30:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: osdmap e10: 2 total, 1 up, 2 in 2022-07-03T10:30:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/3511162050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:24.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:30:24.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]': finished 2022-07-03T10:30:24.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: osdmap e11: 3 total, 1 up, 3 in 2022-07-03T10:30:24.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:24.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:24.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.130 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-1[96064]: 2022-07-03T10:30:24.658+0000 7fa13e790700 -1 osd.1 0 waiting for initial osdmap 2022-07-03T10:30:25.130 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-1[96064]: 2022-07-03T10:30:24.664+0000 7fa139927700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:30:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-07-03T10:30:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: osdmap e10: 2 total, 1 up, 2 in 2022-07-03T10:30:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/3511162050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:30:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]': finished 2022-07-03T10:30:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: osdmap e11: 3 total, 1 up, 3 in 2022-07-03T10:30:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:25.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-07-03T10:30:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: osdmap e10: 2 total, 1 up, 2 in 2022-07-03T10:30:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:30:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-07-03T10:30:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/3511162050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]: dispatch 2022-07-03T10:30:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:30:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "acec9c8a-8e42-45f1-83a0-f598007cc2e1"}]': finished 2022-07-03T10:30:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: osdmap e11: 3 total, 1 up, 3 in 2022-07-03T10:30:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/3933654014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] boot 2022-07-03T10:30:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: osdmap e12: 3 total, 2 up, 3 in 2022-07-03T10:30:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/3933654014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] boot 2022-07-03T10:30:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: osdmap e12: 3 total, 2 up, 3 in 2022-07-03T10:30:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/3933654014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: osd.1 [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] boot 2022-07-03T10:30:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: osdmap e12: 3 total, 2 up, 3 in 2022-07-03T10:30:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:30:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:26.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:26 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:30:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:26 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:30:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:26 smithi101 ceph-mon[85126]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:30:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:30:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:26 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:30:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:26 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:30:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:26 smithi161 ceph-mon[91579]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:30:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:30:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:26 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:30:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:26 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:30:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:26 smithi188 ceph-mon[92425]: pgmap v62: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:30:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:30:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:27 smithi101 ceph-mon[85126]: osdmap e13: 3 total, 2 up, 3 in 2022-07-03T10:30:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:27 smithi161 ceph-mon[91579]: osdmap e13: 3 total, 2 up, 3 in 2022-07-03T10:30:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:27 smithi188 ceph-mon[92425]: osdmap e13: 3 total, 2 up, 3 in 2022-07-03T10:30:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:28 smithi188 ceph-mon[92425]: pgmap v64: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:28.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:28 smithi101 ceph-mon[85126]: pgmap v64: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:28 smithi161 ceph-mon[91579]: pgmap v64: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:30 smithi188 ceph-mon[92425]: pgmap v65: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:30 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:31.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:30 smithi101 ceph-mon[85126]: pgmap v65: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:31.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:30 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:30 smithi161 ceph-mon[91579]: pgmap v65: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:30 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:32.190 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-07-03T10:30:32.190 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:32.190 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:31 smithi188 ceph-mon[92425]: Deploying daemon osd.2 on smithi188 2022-07-03T10:30:32.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-07-03T10:30:32.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:32.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:31 smithi101 ceph-mon[85126]: Deploying daemon osd.2 on smithi188 2022-07-03T10:30:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-07-03T10:30:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:31 smithi161 ceph-mon[91579]: Deploying daemon osd.2 on smithi188 2022-07-03T10:30:33.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:32 smithi188 ceph-mon[92425]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:33.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:32 smithi101 ceph-mon[85126]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:32 smithi161 ceph-mon[91579]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:35.110 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:34 smithi188 ceph-mon[92425]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:35.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:34 smithi101 ceph-mon[85126]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:34 smithi161 ceph-mon[91579]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:36.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:36.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:36.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:36.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:36.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:37.134 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:37 smithi188 ceph-mon[92425]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:37 smithi161 ceph-mon[91579]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:37.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:37 smithi101 ceph-mon[85126]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:38.056 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.053+0000 7f8c65ffb700 1 -- 172.21.15.188:0/2011790187 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c7005e960 con 0x7f8c5c096dd0 2022-07-03T10:30:38.057 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 2 on host 'smithi188' 2022-07-03T10:30:38.059 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.055+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 msgr2=0x7f8c5c099280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:38.059 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.055+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 0x7f8c5c099280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009fd0 tx=0x7f8c6c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:30:38.060 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.055+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 msgr2=0x7f8c70113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:38.060 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.055+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000b5e0 tx=0x7f8c60009120 comp rx=0 tx=0).stop 2022-07-03T10:30:38.060 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 shutdown_connections 2022-07-03T10:30:38.061 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8c5c096dd0 0x7f8c5c099280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:38.061 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8c700fc5d0 0x7f8c70066980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:38.062 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c700fe1e0 0x7f8c70115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:38.062 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 --2- 172.21.15.188:0/2011790187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c700fd850 0x7f8c70113400 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:38.062 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 >> 172.21.15.188:0/2011790187 conn(0x7f8c700f7d40 msgr2=0x7f8c700fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:30:38.063 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 shutdown_connections 2022-07-03T10:30:38.063 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:30:38.056+0000 7f8c777cc700 1 -- 172.21.15.188:0/2011790187 wait complete. 2022-07-03T10:30:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.471 DEBUG:teuthology.orchestra.run.smithi188:osd.2> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.2.service 2022-07-03T10:30:38.474 INFO:tasks.cephadm:Deploying osd.3 on smithi101 with /dev/vg_nvme/lv_3... 2022-07-03T10:30:38.475 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-07-03T10:30:38.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:30:38.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:38.498 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.498 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:30:38.498 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:38.501 INFO:journalctl@ceph.osd.2.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:30:39.288 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:39 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:30:39.020+0000 7fcd3ae12440 -1 osd.2 0 log_to_monitors true 2022-07-03T10:30:39.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:39 smithi188 ceph-mon[92425]: pgmap v69: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:39.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:39 smithi188 ceph-mon[92425]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:39.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:39 smithi188 ceph-mon[92425]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:39.354 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:30:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:39 smithi161 ceph-mon[91579]: pgmap v69: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:39 smithi161 ceph-mon[91579]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:39 smithi161 ceph-mon[91579]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:39.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:39 smithi101 ceph-mon[85126]: pgmap v69: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:39.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:39 smithi101 ceph-mon[85126]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:39.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:39 smithi101 ceph-mon[85126]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-07-03T10:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-mon[92425]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-07-03T10:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-mon[92425]: osdmap e14: 3 total, 2 up, 3 in 2022-07-03T10:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-mon[92425]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-mon[92425]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:40.415 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:30:40.136+0000 7fcd2e9b7700 -1 osd.2 0 waiting for initial osdmap 2022-07-03T10:30:40.415 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:40 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:30:40.140+0000 7fcd27349700 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:30:40.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:40 smithi101 ceph-mon[85126]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-07-03T10:30:40.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:40 smithi101 ceph-mon[85126]: osdmap e14: 3 total, 2 up, 3 in 2022-07-03T10:30:40.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:40 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:40.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:40 smithi101 ceph-mon[85126]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:40.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:40 smithi101 ceph-mon[85126]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:40 smithi161 ceph-mon[91579]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-07-03T10:30:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:40 smithi161 ceph-mon[91579]: osdmap e14: 3 total, 2 up, 3 in 2022-07-03T10:30:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:40 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:40 smithi161 ceph-mon[91579]: from='osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:40 smithi161 ceph-mon[91579]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:30:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:30:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: osdmap e15: 3 total, 2 up, 3 in 2022-07-03T10:30:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.459 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-07-03T10:30:41.460 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-07-03T10:30:41.460 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2022-07-03T10:30:41.460 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2022-07-03T10:30:41.460 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0739736 s, 142 MB/s 2022-07-03T10:30:41.460 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2022-07-03T10:30:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:30:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: osdmap e15: 3 total, 2 up, 3 in 2022-07-03T10:30:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:41.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:30:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: osdmap e15: 3 total, 2 up, 3 in 2022-07-03T10:30:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:30:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:41.830 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_3 2022-07-03T10:30:42.724 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:30:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:42 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T10:30:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:42 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 17168M 2022-07-03T10:30:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:42 smithi188 ceph-mon[92425]: osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] boot 2022-07-03T10:30:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:42 smithi188 ceph-mon[92425]: osdmap e16: 3 total, 3 up, 3 in 2022-07-03T10:30:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:42 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:42.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:42 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi188 2022-07-03T10:30:42.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:42 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi188 to 17168M 2022-07-03T10:30:42.978 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:42 smithi101 ceph-mon[85126]: osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] boot 2022-07-03T10:30:42.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:42 smithi101 ceph-mon[85126]: osdmap e16: 3 total, 3 up, 3 in 2022-07-03T10:30:42.979 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:42 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:42 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T10:30:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:42 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 17168M 2022-07-03T10:30:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:42 smithi161 ceph-mon[91579]: osd.2 [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] boot 2022-07-03T10:30:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:42 smithi161 ceph-mon[91579]: osdmap e16: 3 total, 3 up, 3 in 2022-07-03T10:30:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:42 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:30:43.925 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:30:43.925 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:30:43.926 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:43.926 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:43.926 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:43.927 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: osdmap e17: 3 total, 3 up, 3 in 2022-07-03T10:30:43.927 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:30:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:30:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: osdmap e17: 3 total, 3 up, 3 in 2022-07-03T10:30:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:44.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 -- 172.21.15.101:0/2174710029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 msgr2=0x7f66400feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:44.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 --2- 172.21.15.101:0/2174710029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f66400feb90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6634009860 tx=0x7f6634005ef0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 -- 172.21.15.101:0/2174710029 shutdown_connections 2022-07-03T10:30:44.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 --2- 172.21.15.101:0/2174710029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 0x7f6640105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 --2- 172.21.15.101:0/2174710029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f66400fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 --2- 172.21.15.101:0/2174710029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f66400feb90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 -- 172.21.15.101:0/2174710029 >> 172.21.15.101:0/2174710029 conn(0x7f66400f8c30 msgr2=0x7f66400fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:30:44.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 -- 172.21.15.101:0/2174710029 shutdown_connections 2022-07-03T10:30:44.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.144+0000 7f663f59e700 1 -- 172.21.15.101:0/2174710029 wait complete. 2022-07-03T10:30:44.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.145+0000 7f663f59e700 1 Processor -- start 2022-07-03T10:30:44.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 -- start start 2022-07-03T10:30:44.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:44.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f6640070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:44.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 0x7f6640070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:44.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640071480 con 0x7f66400ff0d0 2022-07-03T10:30:44.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66400715f0 con 0x7f66400fd4c0 2022-07-03T10:30:44.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663f59e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401185a0 con 0x7f66400fe740 2022-07-03T10:30:44.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663e59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:44.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663e59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:56396/0 (socket says 172.21.15.101:56396) 2022-07-03T10:30:44.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.146+0000 7f663e59c700 1 -- 172.21.15.101:0/1212985830 learned_addr learned my addr 172.21.15.101:0/1212985830 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:30:44.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663ed9d700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 0x7f6640070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:44.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663dd9b700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f6640070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:44.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 -- 172.21.15.101:0/1212985830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 msgr2=0x7f6640070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:44.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f6640070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 -- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 msgr2=0x7f6640070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:30:44.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 0x7f6640070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:30:44.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 -- 172.21.15.101:0/1212985830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634009410 con 0x7f66400fd4c0 2022-07-03T10:30:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663dd9b700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f6640070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:30:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.147+0000 7f663e59c700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f663000ea50 tx=0x7f6630005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.148+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630004020 con 0x7f66400fd4c0 2022-07-03T10:30:44.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.148+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f663000cc70 con 0x7f66400fd4c0 2022-07-03T10:30:44.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.148+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6640118800 con 0x7f66400fd4c0 2022-07-03T10:30:44.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.148+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66300039d0 con 0x7f66400fd4c0 2022-07-03T10:30:44.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.148+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6640118d60 con 0x7f66400fd4c0 2022-07-03T10:30:44.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.149+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6640108cb0 con 0x7f66400fd4c0 2022-07-03T10:30:44.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.151+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 15) v1 ==== 125092+0+0 (secure 0 0 0) 0x7f663000c040 con 0x7f66400fd4c0 2022-07-03T10:30:44.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.153+0000 7f662effd700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 0x7f66280992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:30:44.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.153+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3233+0+0 (secure 0 0 0) 0x7f6630014070 con 0x7f66400fd4c0 2022-07-03T10:30:44.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.153+0000 7f663dd9b700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 0x7f66280992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:30:44.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.154+0000 7f663dd9b700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 0x7f66280992d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6634009ed0 tx=0x7f6634005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:30:44.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.154+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f663004f100 con 0x7f66400fd4c0 2022-07-03T10:30:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:30:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:30:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-07-03T10:30:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: osdmap e17: 3 total, 3 up, 3 in 2022-07-03T10:30:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-07-03T10:30:44.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:44.353+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f664005e960 con 0x7f6628096e20 2022-07-03T10:30:44.696 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:44 smithi101 sudo[98760]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:44.696 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:44 smithi101 sudo[98760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:44.697 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:44 smithi101 sudo[98760]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:44.697 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:44 smithi101 sudo[98760]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:44.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: osdmap e18: 3 total, 3 up, 3 in 2022-07-03T10:30:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:44.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:44.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:44.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:44.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: osdmap e18: 3 total, 3 up, 3 in 2022-07-03T10:30:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-07-03T10:30:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: osdmap e18: 3 total, 3 up, 3 in 2022-07-03T10:30:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:30:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:30:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:30:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:45.701 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:45 smithi101 sudo[98808]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:45.701 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:45 smithi101 sudo[98808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:45.702 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:45 smithi101 sudo[98808]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:45.702 INFO:journalctl@ceph.osd.0.smithi101.stdout:Jul 03 10:30:45 smithi101 sudo[98808]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:45.973 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:45 smithi161 ceph-mon[91579]: osdmap e19: 3 total, 3 up, 3 in 2022-07-03T10:30:45.973 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:45.974 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:45.974 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97810]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:45.974 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:45.975 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97810]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:45.975 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97810]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:45.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:45 smithi101 ceph-mon[85126]: osdmap e19: 3 total, 3 up, 3 in 2022-07-03T10:30:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:45 smithi188 ceph-mon[92425]: osdmap e19: 3 total, 3 up, 3 in 2022-07-03T10:30:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:46.380 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97813]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:46.380 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97813]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:46.380 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97813]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:46.380 INFO:journalctl@ceph.osd.1.smithi161.stdout:Jul 03 10:30:45 smithi161 sudo[97813]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:46.495 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98957]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:46.496 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:46.496 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98957]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:46.496 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98957]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:46.914 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98960]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:46.914 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98960]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:46.914 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98960]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:46.914 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:30:46 smithi188 sudo[98960]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:47.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:47 smithi101 sudo[98939]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:47 smithi101 sudo[98939]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:47 smithi101 sudo[98939]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:47 smithi101 sudo[98939]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:48.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:30:48.053+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f66300bd650 con 0x7f66400fd4c0 2022-07-03T10:30:48.307 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.307 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.307 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:48.308 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97816]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:48.308 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97816]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:48.309 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97816]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:48.309 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97816]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:48.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 sudo[98963]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:48.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 sudo[98963]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:48.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 sudo[98963]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:48.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 sudo[98963]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:48.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:30:48.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:48 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97819]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97819]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:48 smithi161 sudo[97819]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:49.042 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 sudo[98963]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-07-03T10:30:49.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 sudo[98963]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:49.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 sudo[98963]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:49.043 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:48 smithi188 sudo[98963]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1593825692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-07-03T10:30:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]': finished 2022-07-03T10:30:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: osdmap e20: 4 total, 3 up, 4 in 2022-07-03T10:30:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:30:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:49 smithi161 ceph-mon[91579]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 sudo[98966]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-07-03T10:30:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 sudo[98966]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-07-03T10:30:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 sudo[98966]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-07-03T10:30:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 sudo[98966]: pam_unix(sudo:session): session closed for user root 2022-07-03T10:30:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1593825692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-07-03T10:30:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]': finished 2022-07-03T10:30:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: osdmap e20: 4 total, 3 up, 4 in 2022-07-03T10:30:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:30:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:49 smithi188 ceph-mon[92425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1593825692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]: dispatch 2022-07-03T10:30:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-07-03T10:30:49.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:49.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14eb692b-ed90-438b-9b81-5b71756db9c5"}]': finished 2022-07-03T10:30:49.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: osdmap e20: 4 total, 3 up, 4 in 2022-07-03T10:30:49.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:30:49.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-07-03T10:30:49.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:30:49.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:30:49.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:30:49.497 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:49 smithi101 ceph-mon[85126]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-07-03T10:30:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:50 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4192062333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:50 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4192062333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4192062333' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:30:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:51 smithi161 ceph-mon[91579]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:51 smithi188 ceph-mon[92425]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:51.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:51 smithi101 ceph-mon[85126]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:30:53.359 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:53 smithi101 ceph-mon[85126]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2022-07-03T10:30:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:53 smithi161 ceph-mon[91579]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2022-07-03T10:30:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:53 smithi188 ceph-mon[92425]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2022-07-03T10:30:55.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:55 smithi101 ceph-mon[85126]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2022-07-03T10:30:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:55 smithi161 ceph-mon[91579]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2022-07-03T10:30:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:55 smithi188 ceph-mon[92425]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2022-07-03T10:30:57.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:57 smithi101 ceph-mon[85126]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:30:57.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-07-03T10:30:57.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:57 smithi161 ceph-mon[91579]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:30:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-07-03T10:30:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:57 smithi188 ceph-mon[92425]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:30:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-07-03T10:30:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:30:58.350 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:58 smithi101 ceph-mon[85126]: Deploying daemon osd.3 on smithi101 2022-07-03T10:30:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:58 smithi161 ceph-mon[91579]: Deploying daemon osd.3 on smithi101 2022-07-03T10:30:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:58 smithi188 ceph-mon[92425]: Deploying daemon osd.3 on smithi101 2022-07-03T10:30:59.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:30:59 smithi101 ceph-mon[85126]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:30:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:30:59 smithi161 ceph-mon[91579]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:30:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:30:59 smithi188 ceph-mon[92425]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2022-07-03T10:31:01.349 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:01 smithi101 ceph-mon[85126]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2022-07-03T10:31:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:01 smithi161 ceph-mon[91579]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2022-07-03T10:31:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:01 smithi188 ceph-mon[92425]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2022-07-03T10:31:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:03 smithi161 ceph-mon[91579]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2022-07-03T10:31:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:03 smithi188 ceph-mon[92425]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2022-07-03T10:31:03.475 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:03 smithi101 ceph-mon[85126]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2022-07-03T10:31:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:04.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:04.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:04 smithi101 ceph-mon[85126]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:04 smithi161 ceph-mon[91579]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:04 smithi188 ceph-mon[92425]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:07.192 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:06 smithi101 ceph-mon[85126]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:06 smithi161 ceph-mon[91579]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:06 smithi188 ceph-mon[92425]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:07.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.602+0000 7f662effd700 1 -- 172.21.15.101:0/1212985830 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f664005e960 con 0x7f6628096e20 2022-07-03T10:31:07.605 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 3 on host 'smithi101' 2022-07-03T10:31:07.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 msgr2=0x7f66280992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:07.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 0x7f66280992d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6634009ed0 tx=0x7f6634005040 comp rx=0 tx=0).stop 2022-07-03T10:31:07.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 msgr2=0x7f6640072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:07.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f663000ea50 tx=0x7f6630005ef0 comp rx=0 tx=0).stop 2022-07-03T10:31:07.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 shutdown_connections 2022-07-03T10:31:07.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6628096e20 0x7f66280992d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:07.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f66400ff0d0 0x7f6640070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:07.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66400fd4c0 0x7f6640072230 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:07.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 --2- 172.21.15.101:0/1212985830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f66400fe740 0x7f6640070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:07.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 >> 172.21.15.101:0/1212985830 conn(0x7f66400f8c30 msgr2=0x7f6640101970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:07.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.605+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 shutdown_connections 2022-07-03T10:31:07.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:07.606+0000 7f663f59e700 1 -- 172.21.15.101:0/1212985830 wait complete. 2022-07-03T10:31:08.062 DEBUG:teuthology.orchestra.run.smithi101:osd.3> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.3.service 2022-07-03T10:31:08.067 INFO:tasks.cephadm:Deploying osd.4 on smithi161 with /dev/vg_nvme/lv_3... 2022-07-03T10:31:08.068 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-07-03T10:31:08.155 INFO:journalctl@ceph.osd.3.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:31:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:08.669 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:31:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:08 smithi161 ceph-mon[91579]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:08 smithi161 ceph-mon[91579]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:08 smithi161 ceph-mon[91579]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:09.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:08 smithi101 ceph-mon[85126]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:09.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:08 smithi101 ceph-mon[85126]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:09.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:08 smithi101 ceph-mon[85126]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:08 smithi188 ceph-mon[92425]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:08 smithi188 ceph-mon[92425]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:08 smithi188 ceph-mon[92425]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-07-03T10:31:10.151 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-07-03T10:31:10.152 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-07-03T10:31:10.152 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2022-07-03T10:31:10.152 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-07-03T10:31:10.152 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111067 s, 944 MB/s 2022-07-03T10:31:10.152 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2022-07-03T10:31:10.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-mon[85126]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-07-03T10:31:10.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-mon[85126]: osdmap e21: 4 total, 3 up, 4 in 2022-07-03T10:31:10.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:10.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-mon[85126]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-mon[85126]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.246 INFO:journalctl@ceph.osd.3.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-3[101040]: 2022-07-03T10:31:09.986+0000 7fa4c7611700 -1 osd.3 0 waiting for initial osdmap 2022-07-03T10:31:10.246 INFO:journalctl@ceph.osd.3.smithi101.stdout:Jul 03 10:31:09 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-3[101040]: 2022-07-03T10:31:09.997+0000 7fa4bffa3700 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:31:10.284 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:09 smithi161 ceph-mon[91579]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-07-03T10:31:10.285 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:09 smithi161 ceph-mon[91579]: osdmap e21: 4 total, 3 up, 4 in 2022-07-03T10:31:10.285 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:09 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:10.286 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:09 smithi161 ceph-mon[91579]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.286 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:09 smithi161 ceph-mon[91579]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:09 smithi188 ceph-mon[92425]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-07-03T10:31:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:09 smithi188 ceph-mon[92425]: osdmap e21: 4 total, 3 up, 4 in 2022-07-03T10:31:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:09 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:09 smithi188 ceph-mon[92425]: from='osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:09 smithi188 ceph-mon[92425]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:31:10.499 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi161:vg_nvme/lv_3 2022-07-03T10:31:11.098 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:31:11.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:10 smithi101 ceph-mon[85126]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:11.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:10 smithi101 ceph-mon[85126]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:31:11.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:10 smithi101 ceph-mon[85126]: osdmap e22: 4 total, 3 up, 4 in 2022-07-03T10:31:11.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:11.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:11.292 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:10 smithi161 ceph-mon[91579]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:11.292 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:10 smithi161 ceph-mon[91579]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:31:11.293 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:10 smithi161 ceph-mon[91579]: osdmap e22: 4 total, 3 up, 4 in 2022-07-03T10:31:11.293 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:11.293 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:10 smithi188 ceph-mon[92425]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2022-07-03T10:31:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:10 smithi188 ceph-mon[92425]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:31:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:10 smithi188 ceph-mon[92425]: osdmap e22: 4 total, 3 up, 4 in 2022-07-03T10:31:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:12.240 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:31:12.241 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:31:12.241 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] boot 2022-07-03T10:31:12.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: osdmap e23: 4 total, 4 up, 4 in 2022-07-03T10:31:12.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:12.242 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.243 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.244 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:31:12.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:31:12.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] boot 2022-07-03T10:31:12.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: osdmap e23: 4 total, 4 up, 4 in 2022-07-03T10:31:12.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:12.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.249 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.391 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 -- 172.21.15.161:0/2006531909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c100a70 msgr2=0x7f471c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:12.392 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 --2- 172.21.15.161:0/2006531909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c100a70 0x7f471c102e70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f470c009900 tx=0x7f470c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.392 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 -- 172.21.15.161:0/2006531909 shutdown_connections 2022-07-03T10:31:12.392 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 --2- 172.21.15.161:0/2006531909 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c0673b0 0x7f471c1004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.392 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 --2- 172.21.15.161:0/2006531909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.392 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 --2- 172.21.15.161:0/2006531909 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c100a70 0x7f471c102e70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.393 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 -- 172.21.15.161:0/2006531909 >> 172.21.15.161:0/2006531909 conn(0x7f471c0f7ed0 msgr2=0x7f471c0fa340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:12.393 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 -- 172.21.15.161:0/2006531909 shutdown_connections 2022-07-03T10:31:12.393 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.386+0000 7f4720a5f700 1 -- 172.21.15.161:0/2006531909 wait complete. 2022-07-03T10:31:12.394 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 Processor -- start 2022-07-03T10:31:12.394 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 -- start start 2022-07-03T10:31:12.394 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:12.394 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 0x7f471c10ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:12.394 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:12.395 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c116350 con 0x7f471c100a70 2022-07-03T10:31:12.395 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c116490 con 0x7f471c0673b0 2022-07-03T10:31:12.395 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4720a5f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c116650 con 0x7f471c103500 2022-07-03T10:31:12.395 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f471a59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:12.395 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f471a59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48346/0 (socket says 172.21.15.161:48346) 2022-07-03T10:31:12.396 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f471a59c700 1 -- 172.21.15.161:0/4019577549 learned_addr learned my addr 172.21.15.161:0/4019577549 (peer_addr_for_me v2:172.21.15.161:0/0) 2022-07-03T10:31:12.396 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f4719d9b700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 0x7f471c10ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:12.396 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.387+0000 7f471ad9d700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:12.396 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 -- 172.21.15.161:0/4019577549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 msgr2=0x7f471c115dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:12.397 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c115dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.397 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 -- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 msgr2=0x7f471c10ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:12.397 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 0x7f471c10ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:12.397 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 -- 172.21.15.161:0/4019577549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c009440 con 0x7f471c0673b0 2022-07-03T10:31:12.397 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471ad9d700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:31:12.398 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4719d9b700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 0x7f471c10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:31:12.398 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f471a59c700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f470400ca80 tx=0x7f470400a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:12.398 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704007970 con 0x7f471c0673b0 2022-07-03T10:31:12.398 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4704018480 con 0x7f471c0673b0 2022-07-03T10:31:12.399 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c116930 con 0x7f471c0673b0 2022-07-03T10:31:12.399 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470400f6c0 con 0x7f471c0673b0 2022-07-03T10:31:12.399 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.388+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c116f10 con 0x7f471c0673b0 2022-07-03T10:31:12.400 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.389+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c067d20 con 0x7f471c0673b0 2022-07-03T10:31:12.401 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.390+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f470401f060 con 0x7f471c0673b0 2022-07-03T10:31:12.401 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.391+0000 7f4712ffd700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 0x7f4708099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:12.401 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.391+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 3694+0+0 (secure 0 0 0) 0x7f4704013070 con 0x7f471c0673b0 2022-07-03T10:31:12.402 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.391+0000 7f4719d9b700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 0x7f4708099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:12.402 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.392+0000 7f4719d9b700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 0x7f4708099250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f470c006010 tx=0x7f470c01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:12.402 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.395+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f470408d3d0 con 0x7f471c0673b0 2022-07-03T10:31:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:31:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:31:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: osd.3 [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] boot 2022-07-03T10:31:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: osdmap e23: 4 total, 4 up, 4 in 2022-07-03T10:31:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:31:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:12.579 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:12.577+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f471c05e960 con 0x7f4708096da0 2022-07-03T10:31:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi101 2022-07-03T10:31:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi101 to 8635M 2022-07-03T10:31:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: osdmap e24: 4 total, 4 up, 4 in 2022-07-03T10:31:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:12 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi101 2022-07-03T10:31:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi101 to 8635M 2022-07-03T10:31:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: osdmap e24: 4 total, 4 up, 4 in 2022-07-03T10:31:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:13.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:12 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:13.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi101 2022-07-03T10:31:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi101 to 8635M 2022-07-03T10:31:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: osdmap e24: 4 total, 4 up, 4 in 2022-07-03T10:31:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:12 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:13 smithi161 ceph-mon[91579]: osdmap e25: 4 total, 4 up, 4 in 2022-07-03T10:31:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:14 smithi188 ceph-mon[92425]: osdmap e25: 4 total, 4 up, 4 in 2022-07-03T10:31:14.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:13 smithi101 ceph-mon[85126]: osdmap e25: 4 total, 4 up, 4 in 2022-07-03T10:31:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: osdmap e26: 4 total, 4 up, 4 in 2022-07-03T10:31:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/1126555089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]': finished 2022-07-03T10:31:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: osdmap e27: 5 total, 4 up, 5 in 2022-07-03T10:31:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: osdmap e26: 4 total, 4 up, 4 in 2022-07-03T10:31:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/1126555089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]': finished 2022-07-03T10:31:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: osdmap e27: 5 total, 4 up, 5 in 2022-07-03T10:31:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:15.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: osdmap e26: 4 total, 4 up, 4 in 2022-07-03T10:31:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/1126555089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]: dispatch 2022-07-03T10:31:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4de30afa-ff39-4be7-9258-990f21fe1fc9"}]': finished 2022-07-03T10:31:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: osdmap e27: 5 total, 4 up, 5 in 2022-07-03T10:31:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:16 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/178961763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:16 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/178961763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:16.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:16 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/178961763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:17 smithi161 ceph-mon[91579]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:17 smithi188 ceph-mon[92425]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:17.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:17 smithi101 ceph-mon[85126]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:19 smithi161 ceph-mon[91579]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-07-03T10:31:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:19 smithi188 ceph-mon[92425]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-07-03T10:31:19.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:19 smithi101 ceph-mon[85126]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2022-07-03T10:31:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:21 smithi161 ceph-mon[91579]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-07-03T10:31:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:21 smithi188 ceph-mon[92425]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-07-03T10:31:21.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:21 smithi101 ceph-mon[85126]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2022-07-03T10:31:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-07-03T10:31:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:22 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:22 smithi161 ceph-mon[91579]: Deploying daemon osd.4 on smithi161 2022-07-03T10:31:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-07-03T10:31:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:22 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:22 smithi188 ceph-mon[92425]: Deploying daemon osd.4 on smithi161 2022-07-03T10:31:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-07-03T10:31:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:22 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:22.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:22 smithi101 ceph-mon[85126]: Deploying daemon osd.4 on smithi161 2022-07-03T10:31:23.340 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:23 smithi161 ceph-mon[91579]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-07-03T10:31:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:23 smithi188 ceph-mon[92425]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-07-03T10:31:23.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:23 smithi101 ceph-mon[85126]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-07-03T10:31:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:25 smithi161 ceph-mon[91579]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2022-07-03T10:31:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:25 smithi188 ceph-mon[92425]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2022-07-03T10:31:25.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:25 smithi101 ceph-mon[85126]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2022-07-03T10:31:27.248 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-07-03T10:31:27.248 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.248 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.248 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:27.249 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:27.249 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:27.249 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:31:27.249 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:31:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-07-03T10:31:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:31:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:31:27.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-07-03T10:31:27.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:27.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:27.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:27.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:27.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:31:27.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:31:28.932 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.930+0000 7f4712ffd700 1 -- 172.21.15.161:0/4019577549 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f471c05e960 con 0x7f4708096da0 2022-07-03T10:31:28.932 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 4 on host 'smithi161' 2022-07-03T10:31:28.934 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 msgr2=0x7f4708099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:28.934 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 0x7f4708099250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f470c006010 tx=0x7f470c01a040 comp rx=0 tx=0).stop 2022-07-03T10:31:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 msgr2=0x7f471c10e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f470400ca80 tx=0x7f470400a300 comp rx=0 tx=0).stop 2022-07-03T10:31:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 shutdown_connections 2022-07-03T10:31:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4708096da0 0x7f4708099250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f471c103500 0x7f471c115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f471c0673b0 0x7f471c10e820 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 --2- 172.21.15.161:0/4019577549 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f471c100a70 0x7f471c10ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.932+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 >> 172.21.15.161:0/4019577549 conn(0x7f471c0f7ed0 msgr2=0x7f471c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:28.936 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.933+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 shutdown_connections 2022-07-03T10:31:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:31:28.933+0000 7f4720a5f700 1 -- 172.21.15.161:0/4019577549 wait complete. 2022-07-03T10:31:29.363 DEBUG:teuthology.orchestra.run.smithi161:osd.4> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.4.service 2022-07-03T10:31:29.366 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2022-07-03T10:31:29.367 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-07-03T10:31:29.395 INFO:journalctl@ceph.osd.4.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:29.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:29.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:29.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:29.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:29.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:29.967 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:31:30.130 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:31:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:31:29.863+0000 7fddcba16440 -1 osd.4 0 log_to_monitors true 2022-07-03T10:31:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:30 smithi188 ceph-mon[92425]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:30 smithi188 ceph-mon[92425]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:30.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:30 smithi161 ceph-mon[91579]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:30.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:30 smithi161 ceph-mon[91579]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:30.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:30 smithi101 ceph-mon[85126]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:30.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:30 smithi101 ceph-mon[85126]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-07-03T10:31:31.438 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-07-03T10:31:31.439 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-07-03T10:31:31.439 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2022-07-03T10:31:31.440 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2022-07-03T10:31:31.440 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110752 s, 947 MB/s 2022-07-03T10:31:31.441 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2022-07-03T10:31:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-07-03T10:31:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: osdmap e28: 5 total, 4 up, 5 in 2022-07-03T10:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:31 smithi188 ceph-mon[92425]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.739 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:31.739 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-07-03T10:31:31.740 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: osdmap e28: 5 total, 4 up, 5 in 2022-07-03T10:31:31.740 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:31.740 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.740 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-mon[91579]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.741 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:31:31.412+0000 7fddbf5bb700 -1 osd.4 0 waiting for initial osdmap 2022-07-03T10:31:31.741 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:31:31.418+0000 7fddbb754700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:31:31.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:31.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-07-03T10:31:31.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: osdmap e28: 5 total, 4 up, 5 in 2022-07-03T10:31:31.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:31.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: from='osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:31 smithi101 ceph-mon[85126]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:31:31.884 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2022-07-03T10:31:32.480 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:31:32.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:31:32.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: osdmap e29: 5 total, 4 up, 5 in 2022-07-03T10:31:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:31:32.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: osdmap e29: 5 total, 4 up, 5 in 2022-07-03T10:31:32.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:31:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: osdmap e29: 5 total, 4 up, 5 in 2022-07-03T10:31:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:33.633 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 -- 172.21.15.188:0/1706546334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fd850 msgr2=0x7f8b2c0fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:33.634 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/1706546334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c0fdca0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009900 tx=0x7f8b180091e0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.634 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 -- 172.21.15.188:0/1706546334 shutdown_connections 2022-07-03T10:31:33.634 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/1706546334 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8b2c0fe1e0 0x7f8b2c104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.634 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/1706546334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c0fc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.635 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/1706546334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c0fdca0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.635 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 -- 172.21.15.188:0/1706546334 >> 172.21.15.188:0/1706546334 conn(0x7f8b2c0f7d40 msgr2=0x7f8b2c0fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:33.635 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 -- 172.21.15.188:0/1706546334 shutdown_connections 2022-07-03T10:31:33.635 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.628+0000 7f8b32bd1700 1 -- 172.21.15.188:0/1706546334 wait complete. 2022-07-03T10:31:33.636 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 Processor -- start 2022-07-03T10:31:33.636 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 -- start start 2022-07-03T10:31:33.636 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:33.636 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:33.637 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8b2c0fe1e0 0x7f8b2c114a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:33.637 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c115000 con 0x7f8b2c0fe1e0 2022-07-03T10:31:33.637 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c115140 con 0x7f8b2c0fd850 2022-07-03T10:31:33.637 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b32bd1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c115300 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.637 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.629+0000 7f8b3096d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:33.638 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40534/0 (socket says 172.21.15.188:40534) 2022-07-03T10:31:33.638 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 -- 172.21.15.188:0/3898779237 learned_addr learned my addr 172.21.15.188:0/3898779237 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:31:33.638 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b2bfff700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:33.638 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 -- 172.21.15.188:0/3898779237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fd850 msgr2=0x7f8b2c10da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:33.639 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c10da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.639 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 -- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8b2c0fe1e0 msgr2=0x7f8b2c114a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:33.639 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8b2c0fe1e0 0x7f8b2c114a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:33.639 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 -- 172.21.15.188:0/3898779237 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18009440 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.640 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b3096d700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000ea50 tx=0x7f8b20005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:33.640 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20009500 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.640 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8b2000c9d0 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.640 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c066030 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.641 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20003b90 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.641 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.630+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c066610 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.641 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.631+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2c107d00 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.632+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f8b2000c040 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.632+0000 7f8b297fa700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 0x7f8b1c099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:33.642 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.633+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4097+0+0 (secure 0 0 0) 0x7f8b20014070 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.633+0000 7f8b2bfff700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 0x7f8b1c099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.633+0000 7f8b2bfff700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 0x7f8b1c099250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b180059f0 tx=0x7f8b180091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:33.643 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.637+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f8b2008cf60 con 0x7f8b2c0fc5d0 2022-07-03T10:31:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:31:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:31:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T10:31:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 8584M 2022-07-03T10:31:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] boot 2022-07-03T10:31:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: osdmap e30: 5 total, 5 up, 5 in 2022-07-03T10:31:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:31:33.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:31:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:33.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi161 2022-07-03T10:31:33.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi161 to 8584M 2022-07-03T10:31:33.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:33.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] boot 2022-07-03T10:31:33.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: osdmap e30: 5 total, 5 up, 5 in 2022-07-03T10:31:33.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:33.822 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:33.820+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b2c066a40 con 0x7f8b1c096da0 2022-07-03T10:31:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:31:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2022-07-03T10:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T10:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 8584M 2022-07-03T10:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: osd.4 [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] boot 2022-07-03T10:31:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: osdmap e30: 5 total, 5 up, 5 in 2022-07-03T10:31:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:31:34.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:34 smithi188 ceph-mon[92425]: osdmap e31: 5 total, 5 up, 5 in 2022-07-03T10:31:34.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:34.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:34.695 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:34.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:34 smithi101 ceph-mon[85126]: osdmap e31: 5 total, 5 up, 5 in 2022-07-03T10:31:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:34.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:34 smithi161 ceph-mon[91579]: osdmap e31: 5 total, 5 up, 5 in 2022-07-03T10:31:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:35.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:35 smithi101 ceph-mon[85126]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:35.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:35 smithi101 ceph-mon[85126]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:35 smithi161 ceph-mon[91579]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:35 smithi161 ceph-mon[91579]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:35 smithi188 ceph-mon[92425]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:35 smithi188 ceph-mon[92425]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:36.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/2881227988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:36 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:36 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]': finished 2022-07-03T10:31:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:36 smithi101 ceph-mon[85126]: osdmap e32: 6 total, 5 up, 6 in 2022-07-03T10:31:36.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/2881227988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:36 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:36 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]': finished 2022-07-03T10:31:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:36 smithi161 ceph-mon[91579]: osdmap e32: 6 total, 5 up, 6 in 2022-07-03T10:31:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/2881227988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:36 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]: dispatch 2022-07-03T10:31:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:36 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93593083-57f3-4a9c-a08d-c32067460386"}]': finished 2022-07-03T10:31:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:36 smithi188 ceph-mon[92425]: osdmap e32: 6 total, 5 up, 6 in 2022-07-03T10:31:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:37.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:37 smithi101 ceph-mon[85126]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:37.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:37 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/72743082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:37 smithi161 ceph-mon[91579]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:37 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/72743082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:37 smithi188 ceph-mon[92425]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:37 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/72743082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:31:39.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:39 smithi101 ceph-mon[85126]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:39.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:39 smithi188 ceph-mon[92425]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:39 smithi161 ceph-mon[91579]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:41 smithi188 ceph-mon[92425]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:41.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:41 smithi101 ceph-mon[85126]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:41 smithi161 ceph-mon[91579]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 27 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:43.648 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:43 smithi188 ceph-mon[92425]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:43.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-07-03T10:31:43.649 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:43.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:43 smithi101 ceph-mon[85126]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:43.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-07-03T10:31:43.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:43 smithi161 ceph-mon[91579]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-07-03T10:31:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:44.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:44 smithi101 ceph-mon[85126]: Deploying daemon osd.5 on smithi188 2022-07-03T10:31:44.846 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:44 smithi188 ceph-mon[92425]: Deploying daemon osd.5 on smithi188 2022-07-03T10:31:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:44 smithi161 ceph-mon[91579]: Deploying daemon osd.5 on smithi188 2022-07-03T10:31:45.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:45 smithi101 ceph-mon[85126]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:45 smithi161 ceph-mon[91579]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:45 smithi188 ceph-mon[92425]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:47.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:47.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:49 smithi188 ceph-mon[92425]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.682 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2022-07-03T10:31:49.683 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.681+0000 7f8b297fa700 1 -- 172.21.15.188:0/3898779237 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8b2c066a40 con 0x7f8b1c096da0 2022-07-03T10:31:49.684 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 msgr2=0x7f8b1c099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 0x7f8b1c099250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b180059f0 tx=0x7f8b180091e0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 msgr2=0x7f8b2c10d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000ea50 tx=0x7f8b20005ef0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.685 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 shutdown_connections 2022-07-03T10:31:49.686 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8b1c096da0 0x7f8b1c099250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.686 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8b2c0fe1e0 0x7f8b2c114a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.686 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8b2c0fc5d0 0x7f8b2c10d4d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.686 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 --2- 172.21.15.188:0/3898779237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c0fd850 0x7f8b2c10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:49.687 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 >> 172.21.15.188:0/3898779237 conn(0x7f8b2c0f7d40 msgr2=0x7f8b2c100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:49.687 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 shutdown_connections 2022-07-03T10:31:49.687 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:31:49.683+0000 7f8b32bd1700 1 -- 172.21.15.188:0/3898779237 wait complete. 2022-07-03T10:31:49.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:49 smithi101 ceph-mon[85126]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:49.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:49 smithi161 ceph-mon[91579]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:50.045 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.5.service 2022-07-03T10:31:50.048 INFO:tasks.cephadm:Deploying osd.6 on smithi101 with /dev/vg_nvme/lv_2... 2022-07-03T10:31:50.049 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-07-03T10:31:50.075 INFO:journalctl@ceph.osd.5.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:31:50.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:50.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:50.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:50.889 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:50.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:50.932 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:31:50.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:50.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:50.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:50.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:50.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:31:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:31:51.155 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 03 10:31:50 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-5[101800]: 2022-07-03T10:31:50.955+0000 7f674210f440 -1 osd.5 0 log_to_monitors true 2022-07-03T10:31:51.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:51 smithi101 ceph-mon[85126]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:51.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:51 smithi101 ceph-mon[85126]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:51.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:51 smithi101 ceph-mon[85126]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:51 smithi161 ceph-mon[91579]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:51 smithi161 ceph-mon[91579]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:51 smithi161 ceph-mon[91579]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:51 smithi188 ceph-mon[92425]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:51 smithi188 ceph-mon[92425]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:51 smithi188 ceph-mon[92425]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-07-03T10:31:52.908 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-07-03T10:31:52.909 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-07-03T10:31:52.909 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2022-07-03T10:31:52.909 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2022-07-03T10:31:52.910 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0435231 s, 241 MB/s 2022-07-03T10:31:52.910 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2022-07-03T10:31:52.913 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-5[101800]: 2022-07-03T10:31:52.702+0000 7f6735cb4700 -1 osd.5 0 waiting for initial osdmap 2022-07-03T10:31:52.914 INFO:journalctl@ceph.osd.5.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-5[101800]: 2022-07-03T10:31:52.709+0000 7f672e646700 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:31:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-07-03T10:31:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: osdmap e33: 6 total, 5 up, 6 in 2022-07-03T10:31:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:52 smithi188 ceph-mon[92425]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:52.949 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-07-03T10:31:52.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: osdmap e33: 6 total, 5 up, 6 in 2022-07-03T10:31:52.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:52.950 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:52.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:52.951 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:52 smithi101 ceph-mon[85126]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-07-03T10:31:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: osdmap e33: 6 total, 5 up, 6 in 2022-07-03T10:31:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: from='osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:31:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:52 smithi161 ceph-mon[91579]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2022-07-03T10:31:53.407 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_2 2022-07-03T10:31:53.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:31:53.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: osdmap e34: 6 total, 5 up, 6 in 2022-07-03T10:31:53.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:53.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:53.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi188 2022-07-03T10:31:53.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:53.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:53.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:53.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:53.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi188 to 8584M 2022-07-03T10:31:53.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:53.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:53.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] boot 2022-07-03T10:31:53.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: osdmap e35: 6 total, 6 up, 6 in 2022-07-03T10:31:53.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:31:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: osdmap e34: 6 total, 5 up, 6 in 2022-07-03T10:31:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T10:31:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 8584M 2022-07-03T10:31:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] boot 2022-07-03T10:31:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: osdmap e35: 6 total, 6 up, 6 in 2022-07-03T10:31:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:31:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: osdmap e34: 6 total, 5 up, 6 in 2022-07-03T10:31:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T10:31:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:31:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 8584M 2022-07-03T10:31:54.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:31:54.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: osd.5 [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] boot 2022-07-03T10:31:54.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: osdmap e35: 6 total, 6 up, 6 in 2022-07-03T10:31:54.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:31:54.291 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:31:54.933 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:54 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:31:54.933 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:54 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:31:54.933 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:54 smithi101 ceph-mon[85126]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:31:54.934 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:54 smithi101 ceph-mon[85126]: osdmap e36: 6 total, 6 up, 6 in 2022-07-03T10:31:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:54 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:31:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:54 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:31:55.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:54 smithi161 ceph-mon[91579]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:31:55.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:54 smithi161 ceph-mon[91579]: osdmap e36: 6 total, 6 up, 6 in 2022-07-03T10:31:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:54 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:31:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:54 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:31:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:54 smithi188 ceph-mon[92425]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:31:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:54 smithi188 ceph-mon[92425]: osdmap e36: 6 total, 6 up, 6 in 2022-07-03T10:31:55.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 -- 172.21.15.101:0/702684801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fd4c0 msgr2=0x7f9cb80fd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:55.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/702684801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb80fd8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009900 tx=0x7f9cb40091e0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 -- 172.21.15.101:0/702684801 shutdown_connections 2022-07-03T10:31:55.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/702684801 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80ff0d0 0x7f9cb8105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/702684801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb80fd8d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/702684801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80fe740 0x7f9cb80feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 -- 172.21.15.101:0/702684801 >> 172.21.15.101:0/702684801 conn(0x7f9cb80f8c30 msgr2=0x7f9cb80fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:31:55.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 -- 172.21.15.101:0/702684801 shutdown_connections 2022-07-03T10:31:55.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.781+0000 7f9cbe05e700 1 -- 172.21.15.101:0/702684801 wait complete. 2022-07-03T10:31:55.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.782+0000 7f9cbe05e700 1 Processor -- start 2022-07-03T10:31:55.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.782+0000 7f9cbe05e700 1 -- start start 2022-07-03T10:31:55.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb81106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:55.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:55.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80ff0d0 0x7f9cb8117c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:55.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb81181f0 con 0x7f9cb80fd4c0 2022-07-03T10:31:55.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8118330 con 0x7f9cb80fe740 2022-07-03T10:31:55.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbe05e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb81184f0 con 0x7f9cb80ff0d0 2022-07-03T10:31:55.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbc85b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:55.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbd05c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb81106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:55.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbc85b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:44490/0 (socket says 172.21.15.101:44490) 2022-07-03T10:31:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbc85b700 1 -- 172.21.15.101:0/1515162427 learned_addr learned my addr 172.21.15.101:0/1515162427 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:31:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.783+0000 7f9cbd85d700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80ff0d0 0x7f9cb8117c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 -- 172.21.15.101:0/1515162427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80ff0d0 msgr2=0x7f9cb8117c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80ff0d0 0x7f9cb8117c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 -- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80fd4c0 msgr2=0x7f9cb81106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:31:55.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb81106c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:31:55.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 -- 172.21.15.101:0/1515162427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4009440 con 0x7f9cb80fe740 2022-07-03T10:31:55.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbc85b700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800ea80 tx=0x7f9ca8005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:55.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca800c7c0 con 0x7f9cb80fe740 2022-07-03T10:31:55.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9ca800cdd0 con 0x7f9cb80fe740 2022-07-03T10:31:55.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb81187d0 con 0x7f9cb80fe740 2022-07-03T10:31:55.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.784+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8010df0 con 0x7f9cb80fe740 2022-07-03T10:31:55.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.785+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb8118d60 con 0x7f9cb80fe740 2022-07-03T10:31:55.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.786+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb8108c50 con 0x7f9cb80fe740 2022-07-03T10:31:55.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.787+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f9ca800c040 con 0x7f9cb80fe740 2022-07-03T10:31:55.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.788+0000 7f9cadffb700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 0x7f9ca40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:31:55.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.789+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4558+0+0 (secure 0 0 0) 0x7f9ca8014070 con 0x7f9cb80fe740 2022-07-03T10:31:55.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.789+0000 7f9cbd05c700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 0x7f9ca40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:31:55.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.789+0000 7f9cbd05c700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 0x7f9ca40991d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40097d0 tx=0x7f9cb40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:31:55.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.791+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f9ca808cbd0 con 0x7f9cb80fe740 2022-07-03T10:31:55.991 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:31:55.989+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cb805e960 con 0x7f9ca4096d20 2022-07-03T10:31:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:56 smithi161 ceph-mon[91579]: osdmap e37: 6 total, 6 up, 6 in 2022-07-03T10:31:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:56 smithi188 ceph-mon[92425]: osdmap e37: 6 total, 6 up, 6 in 2022-07-03T10:31:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:56.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:56 smithi101 ceph-mon[85126]: osdmap e37: 6 total, 6 up, 6 in 2022-07-03T10:31:56.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:31:56.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:31:56.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:31:57.350 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:57 smithi101 ceph-mon[85126]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:57.351 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:57 smithi101 ceph-mon[85126]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:57.351 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:57 smithi101 ceph-mon[85126]: osdmap e38: 6 total, 6 up, 6 in 2022-07-03T10:31:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:57 smithi161 ceph-mon[91579]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:57 smithi161 ceph-mon[91579]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:57 smithi161 ceph-mon[91579]: osdmap e38: 6 total, 6 up, 6 in 2022-07-03T10:31:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:57 smithi188 ceph-mon[92425]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:57 smithi188 ceph-mon[92425]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:31:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:57 smithi188 ceph-mon[92425]: osdmap e38: 6 total, 6 up, 6 in 2022-07-03T10:31:58.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:58 smithi101 ceph-mon[85126]: osdmap e39: 6 total, 6 up, 6 in 2022-07-03T10:31:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:58 smithi161 ceph-mon[91579]: osdmap e39: 6 total, 6 up, 6 in 2022-07-03T10:31:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:58 smithi188 ceph-mon[92425]: osdmap e39: 6 total, 6 up, 6 in 2022-07-03T10:31:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:31:59 smithi161 ceph-mon[91579]: pgmap v135: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:31:59 smithi188 ceph-mon[92425]: pgmap v135: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:31:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:31:59 smithi101 ceph-mon[85126]: pgmap v135: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:32:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:00 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3543241680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:00 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:00 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]': finished 2022-07-03T10:32:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:00 smithi161 ceph-mon[91579]: osdmap e40: 7 total, 6 up, 7 in 2022-07-03T10:32:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:00 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3543241680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:00 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:00 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]': finished 2022-07-03T10:32:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:00 smithi188 ceph-mon[92425]: osdmap e40: 7 total, 6 up, 7 in 2022-07-03T10:32:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:00.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:00 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3543241680' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:00 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]: dispatch 2022-07-03T10:32:00.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:00 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3e65880-acae-4f11-9494-023c542b5e8c"}]': finished 2022-07-03T10:32:00.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:00 smithi101 ceph-mon[85126]: osdmap e40: 7 total, 6 up, 7 in 2022-07-03T10:32:00.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:01 smithi161 ceph-mon[91579]: pgmap v137: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:32:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:01 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3369597557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:01 smithi188 ceph-mon[92425]: pgmap v137: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:32:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:01 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3369597557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:01.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:01 smithi101 ceph-mon[85126]: pgmap v137: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:32:01.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:01 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3369597557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:03 smithi161 ceph-mon[91579]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:03 smithi188 ceph-mon[92425]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:03.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:03 smithi101 ceph-mon[85126]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:05 smithi161 ceph-mon[91579]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:05 smithi188 ceph-mon[92425]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:05.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:05 smithi101 ceph-mon[85126]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:07 smithi161 ceph-mon[91579]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:07.390 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:07 smithi101 ceph-mon[85126]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:07 smithi188 ceph-mon[92425]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-07-03T10:32:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:08 smithi161 ceph-mon[91579]: Deploying daemon osd.6 on smithi101 2022-07-03T10:32:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-07-03T10:32:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:08 smithi188 ceph-mon[92425]: Deploying daemon osd.6 on smithi101 2022-07-03T10:32:08.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-07-03T10:32:08.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:08.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:08 smithi101 ceph-mon[85126]: Deploying daemon osd.6 on smithi101 2022-07-03T10:32:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:09 smithi161 ceph-mon[91579]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:09 smithi188 ceph-mon[92425]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:09.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:09 smithi101 ceph-mon[85126]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:11.310 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:11 smithi101 ceph-mon[85126]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:11 smithi161 ceph-mon[91579]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:11 smithi188 ceph-mon[92425]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:13 smithi161 ceph-mon[91579]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:13 smithi188 ceph-mon[92425]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:13 smithi101 ceph-mon[85126]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:15.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:15.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:15.707 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:15.707 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:15.707 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:17 smithi161 ceph-mon[91579]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:17 smithi188 ceph-mon[92425]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:17 smithi101 ceph-mon[85126]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:18.236 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 6 on host 'smithi101' 2022-07-03T10:32:18.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.234+0000 7f9cadffb700 1 -- 172.21.15.101:0/1515162427 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9cb805e960 con 0x7f9ca4096d20 2022-07-03T10:32:18.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.236+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 msgr2=0x7f9ca40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:18.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.236+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 0x7f9ca40991d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40097d0 tx=0x7f9cb40091e0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 msgr2=0x7f9cb8110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:18.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800ea80 tx=0x7f9ca8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 shutdown_connections 2022-07-03T10:32:18.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9ca4096d20 0x7f9ca40991d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9cb80fd4c0 0x7f9cb81106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9cb80ff0d0 0x7f9cb8117c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 --2- 172.21.15.101:0/1515162427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9cb80fe740 0x7f9cb8110c00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:18.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 >> 172.21.15.101:0/1515162427 conn(0x7f9cb80f8c30 msgr2=0x7f9cb8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:32:18.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 shutdown_connections 2022-07-03T10:32:18.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:32:18.237+0000 7f9cbe05e700 1 -- 172.21.15.101:0/1515162427 wait complete. 2022-07-03T10:32:18.622 DEBUG:teuthology.orchestra.run.smithi101:osd.6> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.6.service 2022-07-03T10:32:18.629 INFO:tasks.cephadm:Deploying osd.7 on smithi161 with /dev/vg_nvme/lv_2... 2022-07-03T10:32:18.629 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-07-03T10:32:18.717 INFO:journalctl@ceph.osd.6.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:32:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:19.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:19.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:19.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:19.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:19.241 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:32:19.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:19.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:19.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:19.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:19.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:20.752 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-07-03T10:32:20.752 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-07-03T10:32:20.752 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2022-07-03T10:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-07-03T10:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011451 s, 916 MB/s 2022-07-03T10:32:20.753 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2022-07-03T10:32:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:20 smithi161 ceph-mon[91579]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:21.159 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi161:vg_nvme/lv_2 2022-07-03T10:32:21.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:20 smithi101 ceph-mon[85126]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:20 smithi188 ceph-mon[92425]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:21.755 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:32:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 -- 172.21.15.161:0/1097709121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 msgr2=0x7fd810105c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 --2- 172.21.15.161:0/1097709121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd810105c80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd808009cd0 tx=0x7fd808009520 comp rx=0 tx=0).stop 2022-07-03T10:32:22.946 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 -- 172.21.15.161:0/1097709121 shutdown_connections 2022-07-03T10:32:22.946 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 --2- 172.21.15.161:0/1097709121 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd810105c80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 --2- 172.21.15.161:0/1097709121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 0x7fd8100fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:22.947 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 --2- 172.21.15.161:0/1097709121 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8100fc310 0x7fd8100fea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 -- 172.21.15.161:0/1097709121 >> 172.21.15.161:0/1097709121 conn(0x7fd8100f7480 msgr2=0x7fd8100f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:32:22.948 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 -- 172.21.15.161:0/1097709121 shutdown_connections 2022-07-03T10:32:22.949 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.937+0000 7fd815a80700 1 -- 172.21.15.161:0/1097709121 wait complete. 2022-07-03T10:32:22.949 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 Processor -- start 2022-07-03T10:32:22.949 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 -- start start 2022-07-03T10:32:22.949 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 0x7fd81010e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:22.950 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8100fc310 0x7fd81010e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:22.950 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810115f40 con 0x7fd8100fc310 2022-07-03T10:32:22.951 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810116080 con 0x7fd8100ff0c0 2022-07-03T10:32:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd815a80700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810116240 con 0x7fd8100fb930 2022-07-03T10:32:22.952 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48452/0 (socket says 172.21.15.161:48452) 2022-07-03T10:32:22.953 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 -- 172.21.15.161:0/583267519 learned_addr learned my addr 172.21.15.161:0/583267519 (peer_addr_for_me v2:172.21.15.161:0/0) 2022-07-03T10:32:22.954 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80f7fe700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 0x7fd81010e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:22.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 -- 172.21.15.161:0/583267519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 msgr2=0x7fd81010e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:22.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 0x7fd81010e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:22.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 -- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8100fc310 msgr2=0x7fd81010e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:22.955 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.938+0000 7fd80ffff700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8100fc310 0x7fd81010e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd80ffff700 1 -- 172.21.15.161:0/583267519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080097e0 con 0x7fd8100ff0c0 2022-07-03T10:32:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd80ffff700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8080097a0 tx=0x7fd808009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:32:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80801d070 con 0x7fd8100ff0c0 2022-07-03T10:32:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8101164c0 con 0x7fd8100ff0c0 2022-07-03T10:32:22.956 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd808005b00 con 0x7fd8100ff0c0 2022-07-03T10:32:22.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808017810 con 0x7fd8100ff0c0 2022-07-03T10:32:22.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.939+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd810116970 con 0x7fd8100ff0c0 2022-07-03T10:32:22.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.940+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd810108eb0 con 0x7fd8100ff0c0 2022-07-03T10:32:22.957 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.941+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd808028030 con 0x7fd8100ff0c0 2022-07-03T10:32:22.958 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.941+0000 7fd807fff700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 0x7fd7fc0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:22.958 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.941+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4644+0+0 (secure 0 0 0) 0x7fd8080c0340 con 0x7fd8100ff0c0 2022-07-03T10:32:22.958 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.941+0000 7fd80f7fe700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 0x7fd7fc0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:22.958 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.942+0000 7fd80f7fe700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 0x7fd7fc0991d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd810065bf0 tx=0x7fd7f8005ec0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:32:22.959 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:22.945+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd80808e470 con 0x7fd8100ff0c0 2022-07-03T10:32:23.128 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:23.125+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81005e960 con 0x7fd7fc096d20 2022-07-03T10:32:23.743 INFO:journalctl@ceph.osd.6.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-6[105684]: 2022-07-03T10:32:23.305+0000 7f1ef9b15440 -1 osd.6 0 log_to_monitors true 2022-07-03T10:32:23.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:23.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi101 2022-07-03T10:32:23.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi101 to 5757M 2022-07-03T10:32:23.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.748 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.748 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:23.748 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:23.749 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:23.749 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:23 smithi101 ceph-mon[85126]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-07-03T10:32:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi101 2022-07-03T10:32:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi101 to 5757M 2022-07-03T10:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:23.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:23 smithi161 ceph-mon[91579]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-07-03T10:32:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi101 2022-07-03T10:32:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:23.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi101 to 5757M 2022-07-03T10:32:23.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:23.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:23.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:23.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:23.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:23 smithi188 ceph-mon[92425]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-07-03T10:32:24.820 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:24 smithi161 ceph-mon[91579]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:24.820 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:24 smithi161 ceph-mon[91579]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-07-03T10:32:24.821 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:24 smithi161 ceph-mon[91579]: osdmap e41: 7 total, 6 up, 7 in 2022-07-03T10:32:24.821 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:24 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:24.821 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:24 smithi161 ceph-mon[91579]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:32:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:24 smithi188 ceph-mon[92425]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:24 smithi188 ceph-mon[92425]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-07-03T10:32:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:24 smithi188 ceph-mon[92425]: osdmap e41: 7 total, 6 up, 7 in 2022-07-03T10:32:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:24 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:24 smithi188 ceph-mon[92425]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:32:24.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:24 smithi101 ceph-mon[85126]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:24 smithi101 ceph-mon[85126]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-07-03T10:32:24.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:24 smithi101 ceph-mon[85126]: osdmap e41: 7 total, 6 up, 7 in 2022-07-03T10:32:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:24 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:24.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:24 smithi101 ceph-mon[85126]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:32:25.492 INFO:journalctl@ceph.osd.6.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-6[105684]: 2022-07-03T10:32:25.088+0000 7f1eed6ba700 -1 osd.6 0 waiting for initial osdmap 2022-07-03T10:32:25.493 INFO:journalctl@ceph.osd.6.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-6[105684]: 2022-07-03T10:32:25.104+0000 7f1ee8851700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:32:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:32:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: osdmap e42: 7 total, 6 up, 7 in 2022-07-03T10:32:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/2789768575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] boot 2022-07-03T10:32:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]': finished 2022-07-03T10:32:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: osdmap e43: 8 total, 7 up, 8 in 2022-07-03T10:32:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:32:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: osdmap e42: 7 total, 6 up, 7 in 2022-07-03T10:32:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/2789768575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] boot 2022-07-03T10:32:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]': finished 2022-07-03T10:32:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: osdmap e43: 8 total, 7 up, 8 in 2022-07-03T10:32:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:32:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: osdmap e42: 7 total, 6 up, 7 in 2022-07-03T10:32:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/2789768575' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]: dispatch 2022-07-03T10:32:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: osd.6 [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] boot 2022-07-03T10:32:25.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed7c95a0-e815-4121-8542-947140c403f7"}]': finished 2022-07-03T10:32:25.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: osdmap e43: 8 total, 7 up, 8 in 2022-07-03T10:32:25.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:32:25.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:26 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:32:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:26 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:32:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:26 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/2077387241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:32:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:32:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:26 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:32:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:26 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:32:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:26 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/2077387241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:32:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:32:26.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:26 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:32:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:26 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:32:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:26 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/2077387241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:26.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:32:26.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:32:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:27 smithi161 ceph-mon[91579]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:27 smithi161 ceph-mon[91579]: osdmap e44: 8 total, 7 up, 8 in 2022-07-03T10:32:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:27 smithi188 ceph-mon[92425]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:27 smithi188 ceph-mon[92425]: osdmap e44: 8 total, 7 up, 8 in 2022-07-03T10:32:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:27.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:27 smithi101 ceph-mon[85126]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-07-03T10:32:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:27 smithi101 ceph-mon[85126]: osdmap e44: 8 total, 7 up, 8 in 2022-07-03T10:32:27.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:29 smithi161 ceph-mon[91579]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:29 smithi188 ceph-mon[92425]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:29.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:29 smithi101 ceph-mon[85126]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:31 smithi161 ceph-mon[91579]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:31 smithi188 ceph-mon[92425]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:31.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:31 smithi101 ceph-mon[85126]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:32.872 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-07-03T10:32:32.873 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-07-03T10:32:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:32.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-07-03T10:32:32.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:33.611 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:33 smithi161 ceph-mon[91579]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:33.612 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:33 smithi161 ceph-mon[91579]: Deploying daemon osd.7 on smithi161 2022-07-03T10:32:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:33 smithi188 ceph-mon[92425]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:33 smithi188 ceph-mon[92425]: Deploying daemon osd.7 on smithi161 2022-07-03T10:32:33.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:33 smithi101 ceph-mon[85126]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:33.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:33 smithi101 ceph-mon[85126]: Deploying daemon osd.7 on smithi161 2022-07-03T10:32:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:35 smithi161 ceph-mon[91579]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:35 smithi188 ceph-mon[92425]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:35.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:35 smithi101 ceph-mon[85126]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:37 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:37 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:37.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:37.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:37.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:37.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:37.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:37 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:39.477 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.475+0000 7fd807fff700 1 -- 172.21.15.161:0/583267519 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd81005e960 con 0x7fd7fc096d20 2022-07-03T10:32:39.485 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 7 on host 'smithi161' 2022-07-03T10:32:39.486 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 msgr2=0x7fd7fc0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:39.486 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 0x7fd7fc0991d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd810065bf0 tx=0x7fd7f8005ec0 comp rx=0 tx=0).stop 2022-07-03T10:32:39.487 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 msgr2=0x7fd8101159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:39.487 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8080097a0 tx=0x7fd808009520 comp rx=0 tx=0).stop 2022-07-03T10:32:39.487 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 shutdown_connections 2022-07-03T10:32:39.487 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd7fc096d20 0x7fd7fc0991d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:39.488 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8100ff0c0 0x7fd8101159c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:39.488 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8100fb930 0x7fd81010e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:39.488 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 --2- 172.21.15.161:0/583267519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8100fc310 0x7fd81010e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:39.488 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 >> 172.21.15.161:0/583267519 conn(0x7fd8100f7480 msgr2=0x7fd810101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:32:39.489 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 shutdown_connections 2022-07-03T10:32:39.489 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:32:39.477+0000 7fd815a80700 1 -- 172.21.15.161:0/583267519 wait complete. 2022-07-03T10:32:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:39 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:39 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.922 DEBUG:teuthology.orchestra.run.smithi161:osd.7> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.7.service 2022-07-03T10:32:39.924 INFO:tasks.cephadm:Deploying osd.8 on smithi188 with /dev/vg_nvme/lv_2... 2022-07-03T10:32:39.925 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-07-03T10:32:39.955 INFO:journalctl@ceph.osd.7.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:32:39.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:39.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:39.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:39.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:39.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:39.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:39 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:40.535 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:32:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:41 smithi161 ceph-mon[91579]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:41 smithi188 ceph-mon[92425]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:41.955 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-07-03T10:32:41.956 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-07-03T10:32:41.956 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2022-07-03T10:32:41.956 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2022-07-03T10:32:41.956 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109525 s, 957 MB/s 2022-07-03T10:32:41.957 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2022-07-03T10:32:41.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:41 smithi101 ceph-mon[85126]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:42.347 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2022-07-03T10:32:42.943 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:32:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T10:32:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 5722M 2022-07-03T10:32:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.890 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:43.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T10:32:43.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 5722M 2022-07-03T10:32:43.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.892 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:43.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi161 2022-07-03T10:32:43.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:32:43.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi161 to 5722M 2022-07-03T10:32:43.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:43.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:44.093 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 -- 172.21.15.188:0/2894791868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740fe910 msgr2=0x7f5d740fed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:44.094 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 --2- 172.21.15.188:0/2894791868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740fe910 0x7f5d740fed60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009900 tx=0x7f5d680091e0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.094 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 -- 172.21.15.188:0/2894791868 shutdown_connections 2022-07-03T10:32:44.094 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 --2- 172.21.15.188:0/2894791868 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740ff2a0 0x7f5d74105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 --2- 172.21.15.188:0/2894791868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d740fdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 --2- 172.21.15.188:0/2894791868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740fe910 0x7f5d740fed60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.088+0000 7f5d7884f700 1 -- 172.21.15.188:0/2894791868 >> 172.21.15.188:0/2894791868 conn(0x7f5d740f8e00 msgr2=0x7f5d740fb250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:32:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- 172.21.15.188:0/2894791868 shutdown_connections 2022-07-03T10:32:44.095 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- 172.21.15.188:0/2894791868 wait complete. 2022-07-03T10:32:44.096 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 Processor -- start 2022-07-03T10:32:44.096 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- start start 2022-07-03T10:32:44.096 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d7410e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:44.096 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740fe910 0x7f5d7410eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:44.097 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:44.097 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d741161a0 con 0x7f5d740fe910 2022-07-03T10:32:44.097 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d741162e0 con 0x7f5d740fd690 2022-07-03T10:32:44.097 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7884f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d741164a0 con 0x7f5d740ff2a0 2022-07-03T10:32:44.097 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d72d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:44.098 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d72d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40646/0 (socket says 172.21.15.188:40646) 2022-07-03T10:32:44.098 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d72d9d700 1 -- 172.21.15.188:0/1661341174 learned_addr learned my addr 172.21.15.188:0/1661341174 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:32:44.098 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.089+0000 7f5d7259c700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d7410e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:44.099 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d71d9b700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740fe910 0x7f5d7410eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:44.099 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 -- 172.21.15.188:0/1661341174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 msgr2=0x7f5d7410e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:44.099 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d7410e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.099 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 -- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740fe910 msgr2=0x7f5d7410eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:44.100 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740fe910 0x7f5d7410eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:44.100 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 -- 172.21.15.188:0/1661341174 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68009440 con 0x7f5d740ff2a0 2022-07-03T10:32:44.101 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d7259c700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d7410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:32:44.101 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d72d9d700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00bc30 tx=0x7f5d6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:32:44.101 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c0092e0 con 0x7f5d740ff2a0 2022-07-03T10:32:44.102 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d74116720 con 0x7f5d740ff2a0 2022-07-03T10:32:44.102 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5d6c00cc50 con 0x7f5d740ff2a0 2022-07-03T10:32:44.103 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c007d50 con 0x7f5d740ff2a0 2022-07-03T10:32:44.103 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.090+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d74116cb0 con 0x7f5d740ff2a0 2022-07-03T10:32:44.103 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.091+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74108ea0 con 0x7f5d740ff2a0 2022-07-03T10:32:44.104 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.092+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f5d6c007400 con 0x7f5d740ff2a0 2022-07-03T10:32:44.104 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.092+0000 7f5d62ffd700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 0x7f5d5c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:32:44.105 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.093+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5076+0+0 (secure 0 0 0) 0x7f5d6c0be3a0 con 0x7f5d740ff2a0 2022-07-03T10:32:44.107 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.093+0000 7f5d7259c700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 0x7f5d5c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:32:44.107 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.093+0000 7f5d7259c700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 0x7f5d5c0991d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5d740fe770 tx=0x7f5d64005ed0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:32:44.108 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.096+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f5d6c04b100 con 0x7f5d740ff2a0 2022-07-03T10:32:44.280 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:44.279+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d7405e960 con 0x7f5d5c096d20 2022-07-03T10:32:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:44.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:32:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:32:44.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:45 smithi161 ceph-mon[91579]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:45 smithi161 ceph-mon[91579]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:45 smithi161 ceph-mon[91579]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:45 smithi161 ceph-mon[91579]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:45.632 INFO:journalctl@ceph.osd.7.smithi161.stdout:Jul 03 10:32:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-7[105250]: 2022-07-03T10:32:45.370+0000 7fda3fc46440 -1 osd.7 0 log_to_monitors true 2022-07-03T10:32:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:45 smithi188 ceph-mon[92425]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:45 smithi188 ceph-mon[92425]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:45 smithi188 ceph-mon[92425]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:45 smithi188 ceph-mon[92425]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:45 smithi101 ceph-mon[85126]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:45 smithi101 ceph-mon[85126]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:32:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:45 smithi101 ceph-mon[85126]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:45.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:45 smithi101 ceph-mon[85126]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-07-03T10:32:46.879 INFO:journalctl@ceph.osd.7.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-7[105250]: 2022-07-03T10:32:46.516+0000 7fda337eb700 -1 osd.7 0 waiting for initial osdmap 2022-07-03T10:32:46.880 INFO:journalctl@ceph.osd.7.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-7[105250]: 2022-07-03T10:32:46.528+0000 7fda2e982700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:32:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-07-03T10:32:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: osdmap e45: 8 total, 7 up, 8 in 2022-07-03T10:32:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/715248727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:32:46.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]': finished 2022-07-03T10:32:46.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: osdmap e46: 9 total, 7 up, 9 in 2022-07-03T10:32:46.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:46.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:46 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-07-03T10:32:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: osdmap e45: 8 total, 7 up, 8 in 2022-07-03T10:32:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/715248727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:32:46.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]': finished 2022-07-03T10:32:46.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: osdmap e46: 9 total, 7 up, 9 in 2022-07-03T10:32:46.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:46.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:46 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-07-03T10:32:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: osdmap e45: 8 total, 7 up, 8 in 2022-07-03T10:32:46.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:32:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/715248727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]: dispatch 2022-07-03T10:32:46.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:32:46.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f"}]': finished 2022-07-03T10:32:46.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: osdmap e46: 9 total, 7 up, 9 in 2022-07-03T10:32:46.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:46.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:46.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/148618939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] boot 2022-07-03T10:32:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: osdmap e47: 9 total, 8 up, 9 in 2022-07-03T10:32:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/148618939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] boot 2022-07-03T10:32:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: osdmap e47: 9 total, 8 up, 9 in 2022-07-03T10:32:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:47.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2022-07-03T10:32:47.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/148618939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:32:47.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: osd.7 [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] boot 2022-07-03T10:32:47.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: osdmap e47: 9 total, 8 up, 9 in 2022-07-03T10:32:47.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:32:47.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:49 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:32:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:49 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:32:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:49 smithi161 ceph-mon[91579]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:49 smithi161 ceph-mon[91579]: osdmap e48: 9 total, 8 up, 9 in 2022-07-03T10:32:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:49 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:32:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:49 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:32:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:49 smithi188 ceph-mon[92425]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:49 smithi188 ceph-mon[92425]: osdmap e48: 9 total, 8 up, 9 in 2022-07-03T10:32:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:49 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:32:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:49 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:32:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:49 smithi101 ceph-mon[85126]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:49 smithi101 ceph-mon[85126]: osdmap e48: 9 total, 8 up, 9 in 2022-07-03T10:32:49.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:50 smithi161 ceph-mon[91579]: osdmap e49: 9 total, 8 up, 9 in 2022-07-03T10:32:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:50 smithi188 ceph-mon[92425]: osdmap e49: 9 total, 8 up, 9 in 2022-07-03T10:32:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:50.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:50 smithi101 ceph-mon[85126]: osdmap e49: 9 total, 8 up, 9 in 2022-07-03T10:32:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:51.355 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:51 smithi188 ceph-mon[92425]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:51.356 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:51 smithi188 ceph-mon[92425]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-07-03T10:32:51.356 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:51 smithi188 ceph-mon[92425]: osdmap e50: 9 total, 8 up, 9 in 2022-07-03T10:32:51.356 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:51 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:51 smithi161 ceph-mon[91579]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:51 smithi161 ceph-mon[91579]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-07-03T10:32:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:51 smithi161 ceph-mon[91579]: osdmap e50: 9 total, 8 up, 9 in 2022-07-03T10:32:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:51 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:51.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:51 smithi101 ceph-mon[85126]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:51.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:51 smithi101 ceph-mon[85126]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-07-03T10:32:51.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:51 smithi101 ceph-mon[85126]: osdmap e50: 9 total, 8 up, 9 in 2022-07-03T10:32:51.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:52 smithi161 ceph-mon[91579]: osdmap e51: 9 total, 8 up, 9 in 2022-07-03T10:32:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:52 smithi188 ceph-mon[92425]: osdmap e51: 9 total, 8 up, 9 in 2022-07-03T10:32:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:52.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:52 smithi101 ceph-mon[85126]: osdmap e51: 9 total, 8 up, 9 in 2022-07-03T10:32:52.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:32:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:53 smithi161 ceph-mon[91579]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-07-03T10:32:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:53 smithi188 ceph-mon[92425]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-07-03T10:32:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:53.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:53 smithi101 ceph-mon[85126]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:53.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-07-03T10:32:53.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:54 smithi188 ceph-mon[92425]: Deploying daemon osd.8 on smithi188 2022-07-03T10:32:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:54 smithi161 ceph-mon[91579]: Deploying daemon osd.8 on smithi188 2022-07-03T10:32:54.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:54 smithi101 ceph-mon[85126]: Deploying daemon osd.8 on smithi188 2022-07-03T10:32:55.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:55 smithi188 ceph-mon[92425]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:55.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:55 smithi188 ceph-mon[92425]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-07-03T10:32:55.324 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:55 smithi188 ceph-mon[92425]: Cluster is now healthy 2022-07-03T10:32:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:55 smithi161 ceph-mon[91579]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:55 smithi161 ceph-mon[91579]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-07-03T10:32:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:55 smithi161 ceph-mon[91579]: Cluster is now healthy 2022-07-03T10:32:55.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:55 smithi101 ceph-mon[85126]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:55.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:55 smithi101 ceph-mon[85126]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-07-03T10:32:55.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:55 smithi101 ceph-mon[85126]: Cluster is now healthy 2022-07-03T10:32:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:57.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:57.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:32:57.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:32:57.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:32:57.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:32:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:32:59 smithi188 ceph-mon[92425]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:32:59 smithi101 ceph-mon[85126]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:32:59 smithi161 ceph-mon[91579]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:32:59.866 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 8 on host 'smithi188' 2022-07-03T10:32:59.866 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.863+0000 7f5d62ffd700 1 -- 172.21.15.188:0/1661341174 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5d7405e960 con 0x7f5d5c096d20 2022-07-03T10:32:59.868 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 msgr2=0x7f5d5c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:59.869 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 0x7f5d5c0991d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5d740fe770 tx=0x7f5d64005ed0 comp rx=0 tx=0).stop 2022-07-03T10:32:59.869 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 msgr2=0x7f5d74115b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:32:59.869 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c00bc30 tx=0x7f5d6c009840 comp rx=0 tx=0).stop 2022-07-03T10:32:59.870 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 shutdown_connections 2022-07-03T10:32:59.870 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5d5c096d20 0x7f5d5c0991d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:59.871 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d740ff2a0 0x7f5d74115b90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:59.871 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d740fd690 0x7f5d7410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:59.871 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 --2- 172.21.15.188:0/1661341174 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5d740fe910 0x7f5d7410eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:32:59.872 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 >> 172.21.15.188:0/1661341174 conn(0x7f5d740f8e00 msgr2=0x7f5d740fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:32:59.872 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 shutdown_connections 2022-07-03T10:32:59.872 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:32:59.865+0000 7f5d7884f700 1 -- 172.21.15.188:0/1661341174 wait complete. 2022-07-03T10:33:00.308 DEBUG:teuthology.orchestra.run.smithi188:osd.8> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.8.service 2022-07-03T10:33:00.312 INFO:tasks.cephadm:Deploying osd.9 on smithi101 with /dev/vg_nvme/lv_1... 2022-07-03T10:33:00.313 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-07-03T10:33:00.345 INFO:journalctl@ceph.osd.8.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:33:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:00 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:00.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:00.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:00.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:00 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:00 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:01.198 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:33:01.397 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 03 10:33:01 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-8[106410]: 2022-07-03T10:33:01.017+0000 7f6f7011d440 -1 osd.8 0 log_to_monitors true 2022-07-03T10:33:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:01 smithi188 ceph-mon[92425]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:01 smithi188 ceph-mon[92425]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:01 smithi188 ceph-mon[92425]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:01 smithi101 ceph-mon[85126]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:01 smithi101 ceph-mon[85126]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:01.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:01 smithi101 ceph-mon[85126]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:01 smithi161 ceph-mon[91579]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:01 smithi161 ceph-mon[91579]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:01 smithi161 ceph-mon[91579]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-07-03T10:33:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-07-03T10:33:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: osdmap e52: 9 total, 8 up, 9 in 2022-07-03T10:33:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T10:33:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 5722M 2022-07-03T10:33:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:02.921 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-8[106410]: 2022-07-03T10:33:02.877+0000 7f6f63cc2700 -1 osd.8 0 waiting for initial osdmap 2022-07-03T10:33:02.921 INFO:journalctl@ceph.osd.8.smithi188.stdout:Jul 03 10:33:02 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-8[106410]: 2022-07-03T10:33:02.888+0000 7f6f5c654700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:33:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-07-03T10:33:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: osdmap e52: 9 total, 8 up, 9 in 2022-07-03T10:33:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T10:33:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 5722M 2022-07-03T10:33:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.159 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-07-03T10:33:03.160 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-07-03T10:33:03.160 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2022-07-03T10:33:03.160 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2022-07-03T10:33:03.160 INFO:teuthology.orchestra.run.smithi101.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0589858 s, 178 MB/s 2022-07-03T10:33:03.161 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2022-07-03T10:33:03.201 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-07-03T10:33:03.201 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: osdmap e52: 9 total, 8 up, 9 in 2022-07-03T10:33:03.202 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:03.202 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:03.202 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:33:03.203 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:33:03.203 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi188 2022-07-03T10:33:03.203 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.203 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.203 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.204 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.204 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:03.204 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi188 to 5722M 2022-07-03T10:33:03.204 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.205 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:03.577 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_1 2022-07-03T10:33:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:33:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: osdmap e53: 9 total, 8 up, 9 in 2022-07-03T10:33:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] boot 2022-07-03T10:33:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: osdmap e54: 9 total, 9 up, 9 in 2022-07-03T10:33:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:33:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: osdmap e53: 9 total, 8 up, 9 in 2022-07-03T10:33:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] boot 2022-07-03T10:33:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: osdmap e54: 9 total, 9 up, 9 in 2022-07-03T10:33:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:33:04.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: osdmap e53: 9 total, 8 up, 9 in 2022-07-03T10:33:04.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: osd.8 [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] boot 2022-07-03T10:33:04.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: osdmap e54: 9 total, 9 up, 9 in 2022-07-03T10:33:04.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:33:04.463 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:33:04.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:04 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:33:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:04 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:33:04.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:04 smithi101 ceph-mon[85126]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:04.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:04 smithi101 ceph-mon[85126]: osdmap e55: 9 total, 9 up, 9 in 2022-07-03T10:33:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:04 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:33:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:04 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:33:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:04 smithi188 ceph-mon[92425]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:04 smithi188 ceph-mon[92425]: osdmap e55: 9 total, 9 up, 9 in 2022-07-03T10:33:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:04 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:04 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:04 smithi161 ceph-mon[91579]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:04 smithi161 ceph-mon[91579]: osdmap e55: 9 total, 9 up, 9 in 2022-07-03T10:33:05.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.880+0000 7fa763b51700 1 -- 172.21.15.101:0/3015413041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fe8e0 msgr2=0x7fa75c0fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:05.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.880+0000 7fa763b51700 1 --2- 172.21.15.101:0/3015413041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c0fed30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009860 tx=0x7fa74c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 -- 172.21.15.101:0/3015413041 shutdown_connections 2022-07-03T10:33:05.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 --2- 172.21.15.101:0/3015413041 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0ff270 0x7fa75c105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 --2- 172.21.15.101:0/3015413041 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0fd660 0x7fa75c0fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 --2- 172.21.15.101:0/3015413041 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c0fed30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 -- 172.21.15.101:0/3015413041 >> 172.21.15.101:0/3015413041 conn(0x7fa75c0f8dd0 msgr2=0x7fa75c0fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:33:05.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 -- 172.21.15.101:0/3015413041 shutdown_connections 2022-07-03T10:33:05.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.881+0000 7fa763b51700 1 -- 172.21.15.101:0/3015413041 wait complete. 2022-07-03T10:33:05.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 Processor -- start 2022-07-03T10:33:05.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 -- start start 2022-07-03T10:33:05.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fd660 0x7fa75c072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:05.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:05.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0ff270 0x7fa75c070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:05.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c071480 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c0715f0 con 0x7fa75c0fd660 2022-07-03T10:33:05.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.882+0000 7fa763b51700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c118700 con 0x7fa75c0ff270 2022-07-03T10:33:05.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:05.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7618ed700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fd660 0x7fa75c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:05.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52302/0 (socket says 172.21.15.101:52302) 2022-07-03T10:33:05.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 -- 172.21.15.101:0/705762080 learned_addr learned my addr 172.21.15.101:0/705762080 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:33:05.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7620ee700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0ff270 0x7fa75c070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:05.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 -- 172.21.15.101:0/705762080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0ff270 msgr2=0x7fa75c070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:05.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0ff270 0x7fa75c070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 -- 172.21.15.101:0/705762080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fd660 msgr2=0x7fa75c072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:05.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fd660 0x7fa75c072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:05.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 -- 172.21.15.101:0/705762080 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa758009440 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.883+0000 7fa7610ec700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009830 tx=0x7fa74c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:05.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.884+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c01d070 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.884+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa754009700 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.884+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa74c0057e0 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.884+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c017b50 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.884+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c009410 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.885+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c108e00 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.887+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa74c028020 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.888+0000 7fa7527fc700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 0x7fa7480991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:05.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.888+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa74c0c0600 con 0x7fa75c0fe8e0 2022-07-03T10:33:05.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.888+0000 7fa7618ed700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 0x7fa7480991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:05.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.889+0000 7fa7618ed700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 0x7fa7480991d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0fe740 tx=0x7fa7580091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:05.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:05.893+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa74c08e190 con 0x7fa75c0fe8e0 2022-07-03T10:33:06.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:06.087+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa75c05e960 con 0x7fa748096d20 2022-07-03T10:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:06 smithi161 ceph-mon[91579]: osdmap e56: 9 total, 9 up, 9 in 2022-07-03T10:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:06 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:06 smithi188 ceph-mon[92425]: osdmap e56: 9 total, 9 up, 9 in 2022-07-03T10:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:06 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:06 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:06 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:06.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:06 smithi101 ceph-mon[85126]: osdmap e56: 9 total, 9 up, 9 in 2022-07-03T10:33:06.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:06 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:06.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:06 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:06.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:06 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:07 smithi161 ceph-mon[91579]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:07 smithi161 ceph-mon[91579]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:07 smithi161 ceph-mon[91579]: osdmap e57: 9 total, 9 up, 9 in 2022-07-03T10:33:07.408 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:07 smithi101 ceph-mon[85126]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:07.408 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:07 smithi101 ceph-mon[85126]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:07.408 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:07 smithi101 ceph-mon[85126]: osdmap e57: 9 total, 9 up, 9 in 2022-07-03T10:33:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:07 smithi188 ceph-mon[92425]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 51 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:07 smithi188 ceph-mon[92425]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:07 smithi188 ceph-mon[92425]: osdmap e57: 9 total, 9 up, 9 in 2022-07-03T10:33:08.283 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:08 smithi101 ceph-mon[85126]: osdmap e58: 9 total, 9 up, 9 in 2022-07-03T10:33:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:08 smithi161 ceph-mon[91579]: osdmap e58: 9 total, 9 up, 9 in 2022-07-03T10:33:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:08 smithi188 ceph-mon[92425]: osdmap e58: 9 total, 9 up, 9 in 2022-07-03T10:33:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:09 smithi161 ceph-mon[91579]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:33:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:09 smithi188 ceph-mon[92425]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:33:09.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:09 smithi101 ceph-mon[85126]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-07-03T10:33:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:10 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]: dispatch 2022-07-03T10:33:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:10 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]': finished 2022-07-03T10:33:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:10 smithi161 ceph-mon[91579]: osdmap e59: 10 total, 9 up, 10 in 2022-07-03T10:33:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:10 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1749505762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:10 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]: dispatch 2022-07-03T10:33:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:10 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]': finished 2022-07-03T10:33:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:10 smithi188 ceph-mon[92425]: osdmap e59: 10 total, 9 up, 10 in 2022-07-03T10:33:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:10 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1749505762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:10.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:10 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]: dispatch 2022-07-03T10:33:10.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:10 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1663925567' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "241d0da9-abc5-4259-8e73-454c3a885ebf"}]': finished 2022-07-03T10:33:10.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:10 smithi101 ceph-mon[85126]: osdmap e59: 10 total, 9 up, 10 in 2022-07-03T10:33:10.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:10.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:10 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1749505762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:11 smithi161 ceph-mon[91579]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:33:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:11 smithi188 ceph-mon[92425]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:33:11.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:11 smithi101 ceph-mon[85126]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2022-07-03T10:33:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:13 smithi161 ceph-mon[91579]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:33:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:13 smithi188 ceph-mon[92425]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:33:13.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:13 smithi101 ceph-mon[85126]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:33:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:15 smithi161 ceph-mon[91579]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-07-03T10:33:15.648 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:15 smithi101 ceph-mon[85126]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-07-03T10:33:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:15 smithi188 ceph-mon[92425]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-07-03T10:33:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:17 smithi161 ceph-mon[91579]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2022-07-03T10:33:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:17 smithi188 ceph-mon[92425]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2022-07-03T10:33:17.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:17 smithi101 ceph-mon[85126]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2022-07-03T10:33:18.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-07-03T10:33:18.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:18 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-07-03T10:33:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:18 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-07-03T10:33:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:18 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:19 smithi161 ceph-mon[91579]: Deploying daemon osd.9 on smithi101 2022-07-03T10:33:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:19 smithi161 ceph-mon[91579]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:19 smithi188 ceph-mon[92425]: Deploying daemon osd.9 on smithi101 2022-07-03T10:33:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:19 smithi188 ceph-mon[92425]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:19.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:19 smithi101 ceph-mon[85126]: Deploying daemon osd.9 on smithi101 2022-07-03T10:33:19.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:19 smithi101 ceph-mon[85126]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:21.532 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:21 smithi101 ceph-mon[85126]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:21 smithi161 ceph-mon[91579]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:21 smithi188 ceph-mon[92425]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:23 smithi161 ceph-mon[91579]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:23 smithi188 ceph-mon[92425]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:23.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:23 smithi101 ceph-mon[85126]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:25 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:25.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:25.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:25 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:25 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:26.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:33:26.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:33:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:33:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:33:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:33:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:33:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:27 smithi188 ceph-mon[92425]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:27.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:27 smithi101 ceph-mon[85126]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:27 smithi161 ceph-mon[91579]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:28.654 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 9 on host 'smithi101' 2022-07-03T10:33:28.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.651+0000 7fa7527fc700 1 -- 172.21.15.101:0/705762080 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa75c05e960 con 0x7fa748096d20 2022-07-03T10:33:28.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.654+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 msgr2=0x7fa7480991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:28.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.654+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 0x7fa7480991d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0fe740 tx=0x7fa7580091e0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 msgr2=0x7fa75c070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:28.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009830 tx=0x7fa74c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 shutdown_connections 2022-07-03T10:33:28.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa748096d20 0x7fa7480991d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa75c0ff270 0x7fa75c070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa75c0fd660 0x7fa75c072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 --2- 172.21.15.101:0/705762080 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa75c0fe8e0 0x7fa75c070870 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:28.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 >> 172.21.15.101:0/705762080 conn(0x7fa75c0f8dd0 msgr2=0x7fa75c101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:33:28.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 shutdown_connections 2022-07-03T10:33:28.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:33:28.655+0000 7fa763b51700 1 -- 172.21.15.101:0/705762080 wait complete. 2022-07-03T10:33:29.206 DEBUG:teuthology.orchestra.run.smithi101:osd.9> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.9.service 2022-07-03T10:33:29.212 INFO:tasks.cephadm:Deploying osd.10 on smithi161 with /dev/vg_nvme/lv_1... 2022-07-03T10:33:29.213 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-07-03T10:33:29.304 INFO:journalctl@ceph.osd.9.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:33:29.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:29.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:29 smithi101 ceph-mon[85126]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:29 smithi161 ceph-mon[91579]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:29 smithi188 ceph-mon[92425]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-07-03T10:33:29.815 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:33:30.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-mon[85126]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-07-03T10:33:30.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-mon[85126]: osdmap e60: 10 total, 9 up, 10 in 2022-07-03T10:33:30.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:30.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-mon[85126]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:30.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-mon[85126]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:30.995 INFO:journalctl@ceph.osd.9.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-9[110345]: 2022-07-03T10:33:30.668+0000 7fe7d264b700 -1 osd.9 0 waiting for initial osdmap 2022-07-03T10:33:30.995 INFO:journalctl@ceph.osd.9.smithi101.stdout:Jul 03 10:33:30 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-9[110345]: 2022-07-03T10:33:30.689+0000 7fe7cafdd700 -1 osd.9 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:33:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:30 smithi161 ceph-mon[91579]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-07-03T10:33:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:30 smithi161 ceph-mon[91579]: osdmap e60: 10 total, 9 up, 10 in 2022-07-03T10:33:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:30 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:30 smithi161 ceph-mon[91579]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:30 smithi161 ceph-mon[91579]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:30 smithi188 ceph-mon[92425]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-07-03T10:33:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:30 smithi188 ceph-mon[92425]: osdmap e60: 10 total, 9 up, 10 in 2022-07-03T10:33:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:30 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:30 smithi188 ceph-mon[92425]: from='osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:30 smithi188 ceph-mon[92425]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2022-07-03T10:33:31.308 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-07-03T10:33:31.309 INFO:teuthology.orchestra.run.smithi161.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-07-03T10:33:31.309 INFO:teuthology.orchestra.run.smithi161.stderr: stderr: 10+0 records in 2022-07-03T10:33:31.309 INFO:teuthology.orchestra.run.smithi161.stderr:10+0 records out 2022-07-03T10:33:31.309 INFO:teuthology.orchestra.run.smithi161.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0144496 s, 726 MB/s 2022-07-03T10:33:31.310 INFO:teuthology.orchestra.run.smithi161.stderr:--> Zapping successful for: 2022-07-03T10:33:31.745 DEBUG:teuthology.orchestra.run.smithi161:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi161:vg_nvme/lv_1 2022-07-03T10:33:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:31 smithi101 ceph-mon[85126]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:31 smithi101 ceph-mon[85126]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:33:31.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:31 smithi101 ceph-mon[85126]: osdmap e61: 10 total, 9 up, 10 in 2022-07-03T10:33:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:31.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:31 smithi161 ceph-mon[91579]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:31 smithi161 ceph-mon[91579]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:33:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:31 smithi161 ceph-mon[91579]: osdmap e61: 10 total, 9 up, 10 in 2022-07-03T10:33:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:31 smithi188 ceph-mon[92425]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:31 smithi188 ceph-mon[92425]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2022-07-03T10:33:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:31 smithi188 ceph-mon[92425]: osdmap e61: 10 total, 9 up, 10 in 2022-07-03T10:33:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.344 INFO:teuthology.orchestra.run.smithi161.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.b/config 2022-07-03T10:33:32.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:32 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:33:32.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:32 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:33:32.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:32 smithi161 ceph-mon[91579]: osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] boot 2022-07-03T10:33:32.838 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:32 smithi161 ceph-mon[91579]: osdmap e62: 10 total, 10 up, 10 in 2022-07-03T10:33:32.838 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:32 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:32.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:32 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:33:32.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:32 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:33:32.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:32 smithi101 ceph-mon[85126]: osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] boot 2022-07-03T10:33:32.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:32 smithi101 ceph-mon[85126]: osdmap e62: 10 total, 10 up, 10 in 2022-07-03T10:33:32.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:32 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:32 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:33:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:32 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:33:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:32 smithi188 ceph-mon[92425]: osd.9 [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] boot 2022-07-03T10:33:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:32 smithi188 ceph-mon[92425]: osdmap e62: 10 total, 10 up, 10 in 2022-07-03T10:33:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:32 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:33:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- 172.21.15.161:0/4169504467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 msgr2=0x7f77a80fc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/4169504467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a80fc9e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7798009900 tx=0x7f77980091e0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.503 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- 172.21.15.161:0/4169504467 shutdown_connections 2022-07-03T10:33:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/4169504467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fe1e0 0x7f77a8104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/4169504467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a80fc9e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/4169504467 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fd850 0x7f77a80fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.504 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- 172.21.15.161:0/4169504467 >> 172.21.15.161:0/4169504467 conn(0x7f77a80f7d40 msgr2=0x7f77a80fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:33:33.505 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- 172.21.15.161:0/4169504467 shutdown_connections 2022-07-03T10:33:33.505 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- 172.21.15.161:0/4169504467 wait complete. 2022-07-03T10:33:33.505 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 Processor -- start 2022-07-03T10:33:33.505 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.498+0000 7f77ac8d8700 1 -- start start 2022-07-03T10:33:33.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:33.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 0x7f77a8113300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:33.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 0x7f77a8115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:33.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8116040 con 0x7f77a80fe1e0 2022-07-03T10:33:33.506 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a81161b0 con 0x7f77a80fc5d0 2022-07-03T10:33:33.507 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8116370 con 0x7f77a80fd850 2022-07-03T10:33:33.507 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:33.507 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:48600/0 (socket says 172.21.15.161:48600) 2022-07-03T10:33:33.507 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 -- 172.21.15.161:0/340900148 learned_addr learned my addr 172.21.15.161:0/340900148 (peer_addr_for_me v2:172.21.15.161:0/0) 2022-07-03T10:33:33.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a5d9b700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 0x7f77a8113300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:33.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a6d9d700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 0x7f77a8115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:33.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 -- 172.21.15.161:0/340900148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 msgr2=0x7f77a8113300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:33.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 0x7f77a8113300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.508 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 -- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 msgr2=0x7f77a8115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:33.509 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 0x7f77a8115930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:33.509 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 -- 172.21.15.161:0/340900148 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009440 con 0x7f77a80fc5d0 2022-07-03T10:33:33.509 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a5d9b700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 0x7f77a8113300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:33:33.509 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a6d9d700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 0x7f77a8115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:33:33.510 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77a659c700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77980097d0 tx=0x7f77980091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:33.510 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779801d070 con 0x7f77a80fc5d0 2022-07-03T10:33:33.510 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7798005500 con 0x7f77a80fc5d0 2022-07-03T10:33:33.510 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a81165f0 con 0x7f77a80fc5d0 2022-07-03T10:33:33.511 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7798022dc0 con 0x7f77a80fc5d0 2022-07-03T10:33:33.511 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.499+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a8116a90 con 0x7f77a80fc5d0 2022-07-03T10:33:33.511 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.501+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a8107d00 con 0x7f77a80fc5d0 2022-07-03T10:33:33.512 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.501+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f7798029030 con 0x7f77a80fc5d0 2022-07-03T10:33:33.512 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.502+0000 7f7796ffd700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 0x7f77900991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:33.513 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.502+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f77980c06e0 con 0x7f77a80fc5d0 2022-07-03T10:33:33.513 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.502+0000 7f77a5d9b700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 0x7f77900991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:33.513 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.503+0000 7f77a5d9b700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 0x7f77900991d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f779c00ac10 tx=0x7f779c005f30 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:33.514 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.506+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f779808e1c0 con 0x7f77a80fc5d0 2022-07-03T10:33:33.692 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:33.690+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f77a805e960 con 0x7f7790096d20 2022-07-03T10:33:33.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:33.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: osdmap e63: 10 total, 10 up, 10 in 2022-07-03T10:33:33.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:33.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:33.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:33.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:33.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:33.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:33.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:33.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:33 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: osdmap e63: 10 total, 10 up, 10 in 2022-07-03T10:33:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:33 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2022-07-03T10:33:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: osdmap e63: 10 total, 10 up, 10 in 2022-07-03T10:33:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:34.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:33 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:34.828 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi101 2022-07-03T10:33:34.828 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi101 to 4317M 2022-07-03T10:33:34.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:34.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:34.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:34.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:34.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi101 2022-07-03T10:33:34.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi101 to 4317M 2022-07-03T10:33:34.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:34.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:34.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:34.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi101 2022-07-03T10:33:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi101 to 4317M 2022-07-03T10:33:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi161:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:35.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:35 smithi101 ceph-mon[85126]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:35 smithi161 ceph-mon[91579]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:35 smithi188 ceph-mon[92425]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:36.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/4019676130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:36.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:36.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]': finished 2022-07-03T10:33:36.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: osdmap e64: 11 total, 10 up, 11 in 2022-07-03T10:33:36.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:36.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.161:0/3812615415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/4019676130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]': finished 2022-07-03T10:33:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: osdmap e64: 11 total, 10 up, 11 in 2022-07-03T10:33:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.161:0/3812615415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/4019676130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]: dispatch 2022-07-03T10:33:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bee2129f-d332-4a19-b6c8-3e308c9e262a"}]': finished 2022-07-03T10:33:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: osdmap e64: 11 total, 10 up, 11 in 2022-07-03T10:33:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.161:0/3812615415' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:37.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:37 smithi101 ceph-mon[85126]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:37 smithi161 ceph-mon[91579]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:37 smithi188 ceph-mon[92425]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:39 smithi161 ceph-mon[91579]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:39 smithi188 ceph-mon[92425]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:39.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:39 smithi101 ceph-mon[85126]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:41.318 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:41 smithi161 ceph-mon[91579]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:41 smithi188 ceph-mon[92425]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:41.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:41 smithi101 ceph-mon[85126]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:43 smithi161 ceph-mon[91579]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-07-03T10:33:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:43 smithi161 ceph-mon[91579]: Deploying daemon osd.10 on smithi161 2022-07-03T10:33:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:43 smithi188 ceph-mon[92425]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-07-03T10:33:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:43 smithi188 ceph-mon[92425]: Deploying daemon osd.10 on smithi161 2022-07-03T10:33:43.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:43 smithi101 ceph-mon[85126]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:43.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-07-03T10:33:43.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:43.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:43 smithi101 ceph-mon[85126]: Deploying daemon osd.10 on smithi161 2022-07-03T10:33:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:45 smithi161 ceph-mon[91579]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:45 smithi188 ceph-mon[92425]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:45.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:45 smithi101 ceph-mon[85126]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:47.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:47.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:47.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:47.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:47.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:49 smithi188 ceph-mon[92425]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:49.455 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:49 smithi161 ceph-mon[91579]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:49.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:49 smithi101 ceph-mon[85126]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:49.874 INFO:teuthology.orchestra.run.smithi161.stdout:Created osd(s) 10 on host 'smithi161' 2022-07-03T10:33:49.874 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.872+0000 7f7796ffd700 1 -- 172.21.15.161:0/340900148 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f77a805e960 con 0x7f7790096d20 2022-07-03T10:33:49.876 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 msgr2=0x7f77900991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:49.876 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 0x7f77900991d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f779c00ac10 tx=0x7f779c005f30 comp rx=0 tx=0).stop 2022-07-03T10:33:49.876 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 msgr2=0x7f77a8066980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:49.877 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77980097d0 tx=0x7f77980091e0 comp rx=0 tx=0).stop 2022-07-03T10:33:49.877 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 shutdown_connections 2022-07-03T10:33:49.877 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7790096d20 0x7f77900991d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:49.877 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77a80fe1e0 0x7f77a8115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:49.878 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77a80fc5d0 0x7f77a8066980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:49.878 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 --2- 172.21.15.161:0/340900148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77a80fd850 0x7f77a8113300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:49.878 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 >> 172.21.15.161:0/340900148 conn(0x7f77a80f7d40 msgr2=0x7f77a80f9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:33:49.878 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 shutdown_connections 2022-07-03T10:33:49.878 INFO:teuthology.orchestra.run.smithi161.stderr:2022-07-03T10:33:49.875+0000 7f77ac8d8700 1 -- 172.21.15.161:0/340900148 wait complete. 2022-07-03T10:33:50.216 DEBUG:teuthology.orchestra.run.smithi161:osd.10> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.10.service 2022-07-03T10:33:50.219 INFO:tasks.cephadm:Deploying osd.11 on smithi188 with /dev/vg_nvme/lv_1... 2022-07-03T10:33:50.220 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-07-03T10:33:50.252 INFO:journalctl@ceph.osd.10.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:33:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:50.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:50.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:50.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:50.751 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:33:50.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:50.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:50.753 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:33:50.824 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:33:51.034 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:33:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:33:50.965+0000 7f2c50af6440 -1 osd.10 0 log_to_monitors true 2022-07-03T10:33:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:51 smithi188 ceph-mon[92425]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:51 smithi188 ceph-mon[92425]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:51 smithi188 ceph-mon[92425]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:51 smithi101 ceph-mon[85126]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:51 smithi101 ceph-mon[85126]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:51.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:51 smithi101 ceph-mon[85126]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:51 smithi161 ceph-mon[91579]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:51 smithi161 ceph-mon[91579]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:51 smithi161 ceph-mon[91579]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-07-03T10:33:52.258 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-07-03T10:33:52.259 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-07-03T10:33:52.259 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2022-07-03T10:33:52.259 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2022-07-03T10:33:52.259 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109478 s, 958 MB/s 2022-07-03T10:33:52.260 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2022-07-03T10:33:52.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2022-07-03T10:33:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-07-03T10:33:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: osdmap e65: 11 total, 10 up, 11 in 2022-07-03T10:33:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.129 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:33:52.882+0000 7f2c4469b700 -1 osd.10 0 waiting for initial osdmap 2022-07-03T10:33:53.130 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:33:52.896+0000 7f2c3e830700 -1 osd.10 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:33:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-07-03T10:33:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: osdmap e65: 11 total, 10 up, 11 in 2022-07-03T10:33:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:53.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:53.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-07-03T10:33:53.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: osdmap e65: 11 total, 10 up, 11 in 2022-07-03T10:33:53.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:53.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:53.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]: dispatch 2022-07-03T10:33:53.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 58 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:53.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:33:53.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:33:53.254 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:33:54.026 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T10:33:54.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 4292M 2022-07-03T10:33:54.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:33:54.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: osdmap e66: 11 total, 10 up, 11 in 2022-07-03T10:33:54.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.029 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:53 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi161 2022-07-03T10:33:54.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi161 to 4292M 2022-07-03T10:33:54.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:33:54.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: osdmap e66: 11 total, 10 up, 11 in 2022-07-03T10:33:54.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:53 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T10:33:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 4292M 2022-07-03T10:33:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi161", "root=default"]}]': finished 2022-07-03T10:33:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: osdmap e66: 11 total, 10 up, 11 in 2022-07-03T10:33:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:53 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:54.393 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/3015203067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 msgr2=0x7f3e740fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:54.394 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/3015203067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e740fdca0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009900 tx=0x7f3e640091e0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.394 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/3015203067 shutdown_connections 2022-07-03T10:33:54.395 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/3015203067 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fc5d0 0x7f3e740fc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.395 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/3015203067 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fe1e0 0x7f3e74104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.396 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/3015203067 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e740fdca0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.396 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/3015203067 >> 172.21.15.188:0/3015203067 conn(0x7f3e740f7d40 msgr2=0x7f3e740fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:33:54.396 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/3015203067 shutdown_connections 2022-07-03T10:33:54.397 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/3015203067 wait complete. 2022-07-03T10:33:54.397 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 Processor -- start 2022-07-03T10:33:54.398 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.388+0000 7f3e7bcf0700 1 -- start start 2022-07-03T10:33:54.398 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 0x7f3e74066960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:54.398 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:54.399 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fe1e0 0x7f3e74115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:54.399 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74116190 con 0x7f3e740fe1e0 2022-07-03T10:33:54.400 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74116300 con 0x7f3e740fc5d0 2022-07-03T10:33:54.400 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e741164c0 con 0x7f3e740fd850 2022-07-03T10:33:54.401 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:54.401 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40800/0 (socket says 172.21.15.188:40800) 2022-07-03T10:33:54.401 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 -- 172.21.15.188:0/2967500636 learned_addr learned my addr 172.21.15.188:0/2967500636 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:33:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e79a8c700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 0x7f3e74066960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 -- 172.21.15.188:0/2967500636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 msgr2=0x7f3e74066960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:54.403 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 0x7f3e74066960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.403 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7a28d700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fe1e0 0x7f3e74115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:54.403 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 -- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fe1e0 msgr2=0x7f3e74115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:33:54.404 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fe1e0 0x7f3e74115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:33:54.404 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 -- 172.21.15.188:0/2967500636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64009440 con 0x7f3e740fd850 2022-07-03T10:33:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e79a8c700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 0x7f3e74066960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:33:54.405 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7928b700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e640097d0 tx=0x7f3e640091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6401d070 con 0x7f3e740fd850 2022-07-03T10:33:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e74116740 con 0x7f3e740fd850 2022-07-03T10:33:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3e64005500 con 0x7f3e740fd850 2022-07-03T10:33:54.406 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6400f460 con 0x7f3e740fd850 2022-07-03T10:33:54.407 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.389+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e74116c40 con 0x7f3e740fd850 2022-07-03T10:33:54.407 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.390+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e74107de0 con 0x7f3e740fd850 2022-07-03T10:33:54.407 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.391+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f3e64029030 con 0x7f3e740fd850 2022-07-03T10:33:54.408 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.392+0000 7f3e6a7fc700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 0x7f3e600991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:33:54.408 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.392+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f3e640c0790 con 0x7f3e740fd850 2022-07-03T10:33:54.409 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.392+0000 7f3e79a8c700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 0x7f3e600991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:33:54.409 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.393+0000 7f3e79a8c700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 0x7f3e600991d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3e740fd6b0 tx=0x7f3e70008040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:33:54.409 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.396+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f3e6408e0a0 con 0x7f3e740fd850 2022-07-03T10:33:54.580 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:33:54.577+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e7405e960 con 0x7f3e60096d20 2022-07-03T10:33:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:33:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:33:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] boot 2022-07-03T10:33:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: osdmap e67: 11 total, 11 up, 11 in 2022-07-03T10:33:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:54 smithi188 ceph-mon[92425]: osdmap e68: 11 total, 11 up, 11 in 2022-07-03T10:33:55.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:33:55.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:33:55.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] boot 2022-07-03T10:33:55.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: osdmap e67: 11 total, 11 up, 11 in 2022-07-03T10:33:55.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:55.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:55.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:55.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:55.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:55.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:55.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:54 smithi101 ceph-mon[85126]: osdmap e68: 11 total, 11 up, 11 in 2022-07-03T10:33:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:33:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:33:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: osd.10 [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] boot 2022-07-03T10:33:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: osdmap e67: 11 total, 11 up, 11 in 2022-07-03T10:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2022-07-03T10:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-07-03T10:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-07-03T10:33:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:33:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:54 smithi161 ceph-mon[91579]: osdmap e68: 11 total, 11 up, 11 in 2022-07-03T10:33:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:56 smithi188 ceph-mon[92425]: osdmap e69: 11 total, 11 up, 11 in 2022-07-03T10:33:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:56 smithi188 ceph-mon[92425]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:33:57.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:56 smithi101 ceph-mon[85126]: osdmap e69: 11 total, 11 up, 11 in 2022-07-03T10:33:57.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:56 smithi101 ceph-mon[85126]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:33:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:56 smithi161 ceph-mon[91579]: osdmap e69: 11 total, 11 up, 11 in 2022-07-03T10:33:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:56 smithi161 ceph-mon[91579]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:33:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/535832514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: osdmap e70: 11 total, 11 up, 11 in 2022-07-03T10:33:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]': finished 2022-07-03T10:33:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: osdmap e71: 12 total, 11 up, 12 in 2022-07-03T10:33:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:57 smithi188 ceph-mon[92425]: from='client.? 172.21.15.188:0/2338603844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:58.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/535832514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: osdmap e70: 11 total, 11 up, 11 in 2022-07-03T10:33:58.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]': finished 2022-07-03T10:33:58.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: osdmap e71: 12 total, 11 up, 12 in 2022-07-03T10:33:58.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:58.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:57 smithi101 ceph-mon[85126]: from='client.? 172.21.15.188:0/2338603844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/535832514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: osdmap e70: 11 total, 11 up, 11 in 2022-07-03T10:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]: dispatch 2022-07-03T10:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "818815ae-5ae2-4fa9-a4bd-237e7f420337"}]': finished 2022-07-03T10:33:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: osdmap e71: 12 total, 11 up, 12 in 2022-07-03T10:33:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:57 smithi161 ceph-mon[91579]: from='client.? 172.21.15.188:0/2338603844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-07-03T10:33:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:58 smithi188 ceph-mon[92425]: osdmap e72: 12 total, 11 up, 12 in 2022-07-03T10:33:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:33:58 smithi188 ceph-mon[92425]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail; 190 KiB/s, 0 objects/s recovering 2022-07-03T10:33:59.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:58 smithi101 ceph-mon[85126]: osdmap e72: 12 total, 11 up, 12 in 2022-07-03T10:33:59.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:59.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:33:58 smithi101 ceph-mon[85126]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail; 190 KiB/s, 0 objects/s recovering 2022-07-03T10:33:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:58 smithi161 ceph-mon[91579]: osdmap e72: 12 total, 11 up, 12 in 2022-07-03T10:33:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:33:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:33:58 smithi161 ceph-mon[91579]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 64 MiB used, 983 GiB / 983 GiB avail; 190 KiB/s, 0 objects/s recovering 2022-07-03T10:34:01.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:00 smithi101 ceph-mon[85126]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2022-07-03T10:34:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:00 smithi161 ceph-mon[91579]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2022-07-03T10:34:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:00 smithi188 ceph-mon[92425]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2022-07-03T10:34:03.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:02 smithi101 ceph-mon[85126]: pgmap v230: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:34:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:02 smithi161 ceph-mon[91579]: pgmap v230: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:34:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:02 smithi188 ceph-mon[92425]: pgmap v230: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-07-03T10:34:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-07-03T10:34:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:04 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:04 smithi161 ceph-mon[91579]: Deploying daemon osd.11 on smithi188 2022-07-03T10:34:04.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-07-03T10:34:04.408 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:04 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:04.409 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:04 smithi188 ceph-mon[92425]: Deploying daemon osd.11 on smithi188 2022-07-03T10:34:04.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-07-03T10:34:04.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:04 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:04.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:04 smithi101 ceph-mon[85126]: Deploying daemon osd.11 on smithi188 2022-07-03T10:34:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:05 smithi161 ceph-mon[91579]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2022-07-03T10:34:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:05 smithi188 ceph-mon[92425]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2022-07-03T10:34:05.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:05 smithi101 ceph-mon[85126]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2022-07-03T10:34:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:07 smithi161 ceph-mon[91579]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2022-07-03T10:34:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:07 smithi188 ceph-mon[92425]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2022-07-03T10:34:07.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:07 smithi101 ceph-mon[85126]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2022-07-03T10:34:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:08 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:08.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:08.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:08.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:08.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:08.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:08 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:08 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:09.785 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:09 smithi188 ceph-mon[92425]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:09.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:09 smithi101 ceph-mon[85126]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:09 smithi161 ceph-mon[91579]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:10.718 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 11 on host 'smithi188' 2022-07-03T10:34:10.719 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.716+0000 7f3e6a7fc700 1 -- 172.21.15.188:0/2967500636 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3e7405e960 con 0x7f3e60096d20 2022-07-03T10:34:10.720 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 msgr2=0x7f3e600991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:10.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 0x7f3e600991d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3e740fd6b0 tx=0x7f3e70008040 comp rx=0 tx=0).stop 2022-07-03T10:34:10.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 msgr2=0x7f3e74113480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:10.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e640097d0 tx=0x7f3e640091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:10.721 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 shutdown_connections 2022-07-03T10:34:10.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3e60096d20 0x7f3e600991d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:10.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3e740fe1e0 0x7f3e74115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:10.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3e740fc5d0 0x7f3e74066960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:10.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 --2- 172.21.15.188:0/2967500636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e740fd850 0x7f3e74113480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:10.722 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 >> 172.21.15.188:0/2967500636 conn(0x7f3e740f7d40 msgr2=0x7f3e74100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:10.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 shutdown_connections 2022-07-03T10:34:10.723 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:10.718+0000 7f3e7bcf0700 1 -- 172.21.15.188:0/2967500636 wait complete. 2022-07-03T10:34:11.119 DEBUG:teuthology.orchestra.run.smithi188:osd.11> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@osd.11.service 2022-07-03T10:34:11.121 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2022-07-03T10:34:11.122 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd stat -f json 2022-07-03T10:34:11.150 INFO:journalctl@ceph.osd.11.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:34:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:11.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:11.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:11.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:11 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:11 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:11.914 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 03 10:34:11 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-11[111038]: 2022-07-03T10:34:11.630+0000 7f7e89039440 -1 osd.11 0 log_to_monitors true 2022-07-03T10:34:12.009 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:34:12.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:12 smithi101 ceph-mon[85126]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:12.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:12 smithi101 ceph-mon[85126]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:12 smithi161 ceph-mon[91579]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:12 smithi161 ceph-mon[91579]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:12 smithi188 ceph-mon[92425]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:12 smithi188 ceph-mon[92425]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-07-03T10:34:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-07-03T10:34:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: osdmap e73: 12 total, 11 up, 12 in 2022-07-03T10:34:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:13 smithi101 ceph-mon[85126]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 -- 172.21.15.101:0/2500224237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140ff270 msgr2=0x7fb614105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:13.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 --2- 172.21.15.101:0/2500224237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140ff270 0x7fb614105b30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009bf0 tx=0x7fb60c009460 comp rx=0 tx=0).stop 2022-07-03T10:34:13.497 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 -- 172.21.15.101:0/2500224237 shutdown_connections 2022-07-03T10:34:13.497 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 --2- 172.21.15.101:0/2500224237 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fd660 0x7fb6140fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.497 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 --2- 172.21.15.101:0/2500224237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140ff270 0x7fb614105b30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 --2- 172.21.15.101:0/2500224237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140fe8e0 0x7fb6140fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 -- 172.21.15.101:0/2500224237 >> 172.21.15.101:0/2500224237 conn(0x7fb6140f8dd0 msgr2=0x7fb6140fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:13.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 -- 172.21.15.101:0/2500224237 shutdown_connections 2022-07-03T10:34:13.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.494+0000 7fb61acd4700 1 -- 172.21.15.101:0/2500224237 wait complete. 2022-07-03T10:34:13.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.495+0000 7fb61acd4700 1 Processor -- start 2022-07-03T10:34:13.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.495+0000 7fb61acd4700 1 -- start start 2022-07-03T10:34:13.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:13.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fe8e0 0x7fb61410e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:13.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140ff270 0x7fb6141159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:13.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614115f50 con 0x7fb6140fe8e0 2022-07-03T10:34:13.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614116090 con 0x7fb6140ff270 2022-07-03T10:34:13.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb61acd4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614116250 con 0x7fb6140fd660 2022-07-03T10:34:13.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb613fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fe8e0 0x7fb61410e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:13.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb618a70700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:13.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb618a70700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53238/0 (socket says 172.21.15.101:53238) 2022-07-03T10:34:13.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb618a70700 1 -- 172.21.15.101:0/3859026052 learned_addr learned my addr 172.21.15.101:0/3859026052 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:13.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.496+0000 7fb619271700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140ff270 0x7fb6141159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:13.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 -- 172.21.15.101:0/3859026052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140ff270 msgr2=0x7fb6141159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:13.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140ff270 0x7fb6141159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 -- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fe8e0 msgr2=0x7fb61410e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:13.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fe8e0 0x7fb61410e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 -- 172.21.15.101:0/3859026052 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb608009440 con 0x7fb6140fd660 2022-07-03T10:34:13.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb618a70700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb60800ebc0 tx=0x7fb6080091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:13.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60800c950 con 0x7fb6140fd660 2022-07-03T10:34:13.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c009700 con 0x7fb6140fd660 2022-07-03T10:34:13.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb608010430 con 0x7fb6140fd660 2022-07-03T10:34:13.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb608018690 con 0x7fb6140fd660 2022-07-03T10:34:13.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.497+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb614070c70 con 0x7fb6140fd660 2022-07-03T10:34:13.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.498+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb614108ec0 con 0x7fb6140fd660 2022-07-03T10:34:13.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.500+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fb60800c040 con 0x7fb6140fd660 2022-07-03T10:34:13.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.501+0000 7fb6117fa700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 0x7fb6040991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:13.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.501+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6843+0+0 (secure 0 0 0) 0x7fb608014070 con 0x7fb6140fd660 2022-07-03T10:34:13.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.502+0000 7fb613fff700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 0x7fb6040991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:13.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.502+0000 7fb613fff700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 0x7fb6040991d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb600009d00 tx=0x7fb600005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:13.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.505+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fb60808cf40 con 0x7fb6140fd660 2022-07-03T10:34:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-07-03T10:34:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: osdmap e73: 12 total, 11 up, 12 in 2022-07-03T10:34:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:13 smithi161 ceph-mon[91579]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.664 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-11[111038]: 2022-07-03T10:34:13.237+0000 7f7e7cbde700 -1 osd.11 0 waiting for initial osdmap 2022-07-03T10:34:13.665 INFO:journalctl@ceph.osd.11.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-11[111038]: 2022-07-03T10:34:13.257+0000 7f7e77d75700 -1 osd.11 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-03T10:34:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-07-03T10:34:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: osdmap e73: 12 total, 11 up, 12 in 2022-07-03T10:34:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: from='osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:13 smithi188 ceph-mon[92425]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2022-07-03T10:34:13.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.694+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb614104a60 con 0x7fb6140fd660 2022-07-03T10:34:13.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.694+0000 7fb6117fa700 1 -- 172.21.15.101:0/3859026052 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7fb608090560 con 0x7fb6140fd660 2022-07-03T10:34:13.697 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:13.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 msgr2=0x7fb6040991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:13.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 0x7fb6040991d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb600009d00 tx=0x7fb600005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 msgr2=0x7fb61410e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:13.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb60800ebc0 tx=0x7fb6080091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 shutdown_connections 2022-07-03T10:34:13.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb604096d20 0x7fb6040991d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6140ff270 0x7fb6141159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6140fd660 0x7fb61410e420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 --2- 172.21.15.101:0/3859026052 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb6140fe8e0 0x7fb61410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:13.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 >> 172.21.15.101:0/3859026052 conn(0x7fb6140f8dd0 msgr2=0x7fb614101d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:13.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 shutdown_connections 2022-07-03T10:34:13.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:13.697+0000 7fb61acd4700 1 -- 172.21.15.101:0/3859026052 wait complete. 2022-07-03T10:34:14.087 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1656844433,"num_in_osds":12,"osd_in_since":1656844436,"num_remapped_pgs":0} 2022-07-03T10:34:14.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:34:14.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: osdmap e74: 12 total, 11 up, 12 in 2022-07-03T10:34:14.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: Detected new or changed devices on smithi188 2022-07-03T10:34:14.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3859026052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:14.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: Adjusting osd_memory_target on smithi188 to 4292M 2022-07-03T10:34:14.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.496 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:34:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: osdmap e74: 12 total, 11 up, 12 in 2022-07-03T10:34:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T10:34:14.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3859026052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:14.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 4292M 2022-07-03T10:34:14.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2022-07-03T10:34:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: osdmap e74: 12 total, 11 up, 12 in 2022-07-03T10:34:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T10:34:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3859026052' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:34:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 4292M 2022-07-03T10:34:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:15.089 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd stat -f json 2022-07-03T10:34:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: purged_snaps scrub starts 2022-07-03T10:34:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: purged_snaps scrub ok 2022-07-03T10:34:15.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] boot 2022-07-03T10:34:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: osdmap e75: 12 total, 12 up, 12 in 2022-07-03T10:34:15.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:15 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: purged_snaps scrub starts 2022-07-03T10:34:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: purged_snaps scrub ok 2022-07-03T10:34:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] boot 2022-07-03T10:34:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: osdmap e75: 12 total, 12 up, 12 in 2022-07-03T10:34:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:15 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: purged_snaps scrub starts 2022-07-03T10:34:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: purged_snaps scrub ok 2022-07-03T10:34:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-07-03T10:34:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: osd.11 [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] boot 2022-07-03T10:34:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: osdmap e75: 12 total, 12 up, 12 in 2022-07-03T10:34:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:15 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:34:15.978 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:34:16.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:16 smithi101 ceph-mon[85126]: osdmap e76: 12 total, 12 up, 12 in 2022-07-03T10:34:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:16 smithi161 ceph-mon[91579]: osdmap e76: 12 total, 12 up, 12 in 2022-07-03T10:34:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:16 smithi188 ceph-mon[92425]: osdmap e76: 12 total, 12 up, 12 in 2022-07-03T10:34:17.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.475+0000 7f779c3c0700 1 -- 172.21.15.101:0/1618105069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940ff270 msgr2=0x7f7794105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:17.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.475+0000 7f779c3c0700 1 --2- 172.21.15.101:0/1618105069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940ff270 0x7f7794105b30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f778c009ca0 tx=0x7f778c009520 comp rx=0 tx=0).stop 2022-07-03T10:34:17.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 -- 172.21.15.101:0/1618105069 shutdown_connections 2022-07-03T10:34:17.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 --2- 172.21.15.101:0/1618105069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940ff270 0x7f7794105b30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 --2- 172.21.15.101:0/1618105069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940fd660 0x7f77940fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 --2- 172.21.15.101:0/1618105069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 0x7f77940fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 -- 172.21.15.101:0/1618105069 >> 172.21.15.101:0/1618105069 conn(0x7f77940f8dd0 msgr2=0x7f77940fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:17.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 -- 172.21.15.101:0/1618105069 shutdown_connections 2022-07-03T10:34:17.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.476+0000 7f779c3c0700 1 -- 172.21.15.101:0/1618105069 wait complete. 2022-07-03T10:34:17.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 Processor -- start 2022-07-03T10:34:17.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 -- start start 2022-07-03T10:34:17.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:17.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 0x7f7794110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:17.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940ff270 0x7f7794117b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:17.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794118080 con 0x7f77940fd660 2022-07-03T10:34:17.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77941181c0 con 0x7f77940ff270 2022-07-03T10:34:17.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779c3c0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794118380 con 0x7f77940fe8e0 2022-07-03T10:34:17.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.477+0000 7f779a15c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:17.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:38730/0 (socket says 172.21.15.101:38730) 2022-07-03T10:34:17.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 -- 172.21.15.101:0/4284264531 learned_addr learned my addr 172.21.15.101:0/4284264531 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:17.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779995b700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 0x7f7794110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:17.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a95d700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940ff270 0x7f7794117b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:17.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 -- 172.21.15.101:0/4284264531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 msgr2=0x7f7794110a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:17.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 0x7f7794110a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 -- 172.21.15.101:0/4284264531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940ff270 msgr2=0x7f7794117b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:17.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940ff270 0x7f7794117b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 -- 172.21.15.101:0/4284264531 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0097e0 con 0x7f77940fd660 2022-07-03T10:34:17.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.478+0000 7f779a15c700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f779000ea50 tx=0x7f7790005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:17.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.479+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790004020 con 0x7f77940fd660 2022-07-03T10:34:17.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.479+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f779000cc70 con 0x7f77940fd660 2022-07-03T10:34:17.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.479+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77900039d0 con 0x7f77940fd660 2022-07-03T10:34:17.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.479+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794118660 con 0x7f77940fd660 2022-07-03T10:34:17.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.479+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7794118ba0 con 0x7f77940fd660 2022-07-03T10:34:17.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.480+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794108ec0 con 0x7f77940fd660 2022-07-03T10:34:17.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.482+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f779000c040 con 0x7f77940fd660 2022-07-03T10:34:17.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.483+0000 7f778affd700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 0x7f7780099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:17.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.483+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f7790014070 con 0x7f77940fd660 2022-07-03T10:34:17.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.484+0000 7f779995b700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 0x7f7780099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:17.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.484+0000 7f779995b700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 0x7f7780099250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7784009d00 tx=0x7f7784005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:17.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.487+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f779008ccb0 con 0x7f77940fd660 2022-07-03T10:34:17.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:17 smithi101 ceph-mon[85126]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 71 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:17 smithi161 ceph-mon[91579]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 71 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:17 smithi188 ceph-mon[92425]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 71 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:17.675 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.673+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7794119240 con 0x7f77940fd660 2022-07-03T10:34:17.676 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.673+0000 7f778affd700 1 -- 172.21.15.101:0/4284264531 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7f77900902d0 con 0x7f77940fd660 2022-07-03T10:34:17.677 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:17.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.675+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 msgr2=0x7f7780099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:17.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.675+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 0x7f7780099250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7784009d00 tx=0x7f7784005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 msgr2=0x7f7794110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:17.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f779000ea50 tx=0x7f7790005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 shutdown_connections 2022-07-03T10:34:17.681 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7780096da0 0x7f7780099250 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.681 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f77940fd660 0x7f7794110550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77940ff270 0x7f7794117b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 --2- 172.21.15.101:0/4284264531 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77940fe8e0 0x7f7794110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:17.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 >> 172.21.15.101:0/4284264531 conn(0x7f77940f8dd0 msgr2=0x7f77940fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:17.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 shutdown_connections 2022-07-03T10:34:17.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:17.676+0000 7f779c3c0700 1 -- 172.21.15.101:0/4284264531 wait complete. 2022-07-03T10:34:18.153 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1656844454,"num_in_osds":12,"osd_in_since":1656844436,"num_remapped_pgs":0} 2022-07-03T10:34:18.154 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd dump --format=json 2022-07-03T10:34:18.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:18 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4284264531' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:18 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4284264531' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:18 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4284264531' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-07-03T10:34:19.037 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:34:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:19 smithi161 ceph-mon[91579]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:19 smithi188 ceph-mon[92425]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:19.672 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:19 smithi101 ceph-mon[85126]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:20.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 -- 172.21.15.101:0/2882464313 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a0065eb0 msgr2=0x7f12a0066320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:20.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/2882464313 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a0065eb0 0x7f12a0066320 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f128c009930 tx=0x7f128c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 -- 172.21.15.101:0/2882464313 shutdown_connections 2022-07-03T10:34:20.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/2882464313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a010a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/2882464313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a01036e0 0x7f12a0103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/2882464313 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a0065eb0 0x7f12a0066320 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 -- 172.21.15.101:0/2882464313 >> 172.21.15.101:0/2882464313 conn(0x7f12a0074450 msgr2=0x7f12a0074850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:20.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 -- 172.21.15.101:0/2882464313 shutdown_connections 2022-07-03T10:34:20.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.433+0000 7f12a6ff6700 1 -- 172.21.15.101:0/2882464313 wait complete. 2022-07-03T10:34:20.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.434+0000 7f12a6ff6700 1 Processor -- start 2022-07-03T10:34:20.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.434+0000 7f12a6ff6700 1 -- start start 2022-07-03T10:34:20.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.434+0000 7f12a6ff6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 0x7f12a0116760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:20.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a6ff6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:20.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a6ff6700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 0x7f12a0110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:20.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a6ff6700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0110de0 con 0x7f12a01036e0 2022-07-03T10:34:20.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a6ff6700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0110f50 con 0x7f12a0065eb0 2022-07-03T10:34:20.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a6ff6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a0111110 con 0x7f12a0066860 2022-07-03T10:34:20.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a5593700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 0x7f12a0110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:20.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:20.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a4d92700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 0x7f12a0116760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:20.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a5593700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 0x7f12a0110810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60674/0 (socket says 172.21.15.101:60674) 2022-07-03T10:34:20.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:40552/0 (socket says 172.21.15.101:40552) 2022-07-03T10:34:20.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a4d92700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 0x7f12a0116760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:38144/0 (socket says 172.21.15.101:38144) 2022-07-03T10:34:20.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f12a5593700 1 -- 172.21.15.101:0/3020260029 learned_addr learned my addr 172.21.15.101:0/3020260029 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:20.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 -- 172.21.15.101:0/3020260029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 msgr2=0x7f12a0116760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:20.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 0x7f12a0116760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 -- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 msgr2=0x7f12a0110810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:20.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 0x7f12a0110810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.435+0000 7f129ffff700 1 -- 172.21.15.101:0/3020260029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294009410 con 0x7f12a0066860 2022-07-03T10:34:20.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f129ffff700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f128c006010 tx=0x7f128c0091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:20.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c01d070 con 0x7f12a0066860 2022-07-03T10:34:20.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c009440 con 0x7f12a0066860 2022-07-03T10:34:20.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a011d540 con 0x7f12a0066860 2022-07-03T10:34:20.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f128c00bbd0 con 0x7f12a0066860 2022-07-03T10:34:20.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.436+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c022960 con 0x7f12a0066860 2022-07-03T10:34:20.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.437+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a004ccb0 con 0x7f12a0066860 2022-07-03T10:34:20.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.439+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f128c029030 con 0x7f12a0066860 2022-07-03T10:34:20.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.440+0000 7f129d7fa700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 0x7f12900991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:20.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.440+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f128c0c0390 con 0x7f12a0066860 2022-07-03T10:34:20.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.441+0000 7f12a4d92700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 0x7f12900991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:20.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.441+0000 7f12a4d92700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 0x7f12900991d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e90 tx=0x7f1294005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:20.448 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.445+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f128c05a100 con 0x7f12a0066860 2022-07-03T10:34:20.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.631+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12a011daf0 con 0x7f12a0066860 2022-07-03T10:34:20.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.633+0000 7f129d7fa700 1 -- 172.21.15.101:0/3020260029 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18195 (secure 0 0 0) 0x7f128c08dab0 con 0x7f12a0066860 2022-07-03T10:34:20.635 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:20.636 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":76,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:34:15.242218+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:34:20.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 msgr2=0x7f12900991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:20.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 0x7f12900991d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e90 tx=0x7f1294005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 msgr2=0x7f12a0116ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:20.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f128c006010 tx=0x7f128c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 shutdown_connections 2022-07-03T10:34:20.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1290096d20 0x7f12900991d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12a01036e0 0x7f12a0110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12a0065eb0 0x7f12a0116760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 --2- 172.21.15.101:0/3020260029 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12a0066860 0x7f12a0116ca0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:20.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.635+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 >> 172.21.15.101:0/3020260029 conn(0x7f12a0074450 msgr2=0x7f12a0108840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:20.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.636+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 shutdown_connections 2022-07-03T10:34:20.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:20.636+0000 7f12a6ff6700 1 -- 172.21.15.101:0/3020260029 wait complete. 2022-07-03T10:34:21.113 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-07-03T10:34:21.115 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd pool get .mgr pg_num 2022-07-03T10:34:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:21 smithi161 ceph-mon[91579]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:21 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3020260029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:21 smithi188 ceph-mon[92425]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:21 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3020260029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:21.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:21 smithi101 ceph-mon[85126]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:21.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:21 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3020260029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:21.999 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:34:23.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.403+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2459568083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe46410e6f0 msgr2=0x7fe464110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:23.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.403+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2459568083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe46410e6f0 0x7fe464110ad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009ca0 tx=0x7fe45c009520 comp rx=0 tx=0).stop 2022-07-03T10:34:23.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2459568083 shutdown_connections 2022-07-03T10:34:23.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2459568083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe46410e6f0 0x7fe464110ad0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2459568083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4640708c0 0x7fe464070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2459568083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640711d0 0x7fe46410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2459568083 >> 172.21.15.101:0/2459568083 conn(0x7fe4640f9220 msgr2=0x7fe4640fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:23.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2459568083 shutdown_connections 2022-07-03T10:34:23.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.404+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2459568083 wait complete. 2022-07-03T10:34:23.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 Processor -- start 2022-07-03T10:34:23.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 -- start start 2022-07-03T10:34:23.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:23.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 0x7fe46411aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:23.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe46410e6f0 0x7fe464114a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:23.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe464115030 con 0x7fe4640711d0 2022-07-03T10:34:23.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641151a0 con 0x7fe4640708c0 2022-07-03T10:34:23.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.405+0000 7fe46b3f4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe464115360 con 0x7fe46410e6f0 2022-07-03T10:34:23.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe46898f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 0x7fe46411aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:23.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:23.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:38166/0 (socket says 172.21.15.101:38166) 2022-07-03T10:34:23.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe46898f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 0x7fe46411aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60688/0 (socket says 172.21.15.101:60688) 2022-07-03T10:34:23.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 -- 172.21.15.101:0/2801659958 learned_addr learned my addr 172.21.15.101:0/2801659958 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:23.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469991700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe46410e6f0 0x7fe464114a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:23.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 -- 172.21.15.101:0/2801659958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe46410e6f0 msgr2=0x7fe464114a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:23.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe46410e6f0 0x7fe464114a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 -- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 msgr2=0x7fe46411aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:23.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 0x7fe46411aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 -- 172.21.15.101:0/2801659958 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0097e0 con 0x7fe4640708c0 2022-07-03T10:34:23.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.406+0000 7fe469190700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe46000ea80 tx=0x7fe460005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:23.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.407+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000c740 con 0x7fe4640708c0 2022-07-03T10:34:23.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.407+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe464115640 con 0x7fe4640708c0 2022-07-03T10:34:23.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.407+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe460010430 con 0x7fe4640708c0 2022-07-03T10:34:23.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.407+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460018420 con 0x7fe4640708c0 2022-07-03T10:34:23.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.407+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe464121970 con 0x7fe4640708c0 2022-07-03T10:34:23.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.408+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46404ccb0 con 0x7fe4640708c0 2022-07-03T10:34:23.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.410+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fe46000c040 con 0x7fe4640708c0 2022-07-03T10:34:23.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.411+0000 7fe459ffb700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 0x7fe4500991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:23.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.411+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe460014070 con 0x7fe4640708c0 2022-07-03T10:34:23.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.411+0000 7fe46898f700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 0x7fe4500991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:23.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.412+0000 7fe46898f700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 0x7fe4500991d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4641161c0 tx=0x7fe454005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:23.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.415+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe46004f100 con 0x7fe4640708c0 2022-07-03T10:34:23.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:23 smithi101 ceph-mon[85126]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:23.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.599+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe464113aa0 con 0x7fe4640708c0 2022-07-03T10:34:23.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.600+0000 7fe459ffb700 1 -- 172.21.15.101:0/2801659958 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe46008cb80 con 0x7fe4640708c0 2022-07-03T10:34:23.603 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:34:23.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 msgr2=0x7fe4500991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:23.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 0x7fe4500991d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4641161c0 tx=0x7fe454005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 msgr2=0x7fe46411a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:23.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe46000ea80 tx=0x7fe460005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 shutdown_connections 2022-07-03T10:34:23.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe450096d20 0x7fe4500991d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe46410e6f0 0x7fe464114a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4640708c0 0x7fe46411a9b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 --2- 172.21.15.101:0/2801659958 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4640711d0 0x7fe46411aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:23.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 >> 172.21.15.101:0/2801659958 conn(0x7fe4640f9220 msgr2=0x7fe464110840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:23.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 shutdown_connections 2022-07-03T10:34:23.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:23.603+0000 7fe46b3f4700 1 -- 172.21.15.101:0/2801659958 wait complete. 2022-07-03T10:34:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:23 smithi161 ceph-mon[91579]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:23 smithi188 ceph-mon[92425]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:24.082 INFO:tasks.cephadm:Adding mds.a on smithi101 2022-07-03T10:34:24.082 INFO:tasks.cephadm:Adding mds.d on smithi101 2022-07-03T10:34:24.083 INFO:tasks.cephadm:Adding mds.g on smithi101 2022-07-03T10:34:24.083 INFO:tasks.cephadm:Adding mds.j on smithi101 2022-07-03T10:34:24.083 INFO:tasks.cephadm:Adding mds.b on smithi161 2022-07-03T10:34:24.084 INFO:tasks.cephadm:Adding mds.e on smithi161 2022-07-03T10:34:24.084 INFO:tasks.cephadm:Adding mds.h on smithi161 2022-07-03T10:34:24.084 INFO:tasks.cephadm:Adding mds.k on smithi161 2022-07-03T10:34:24.085 INFO:tasks.cephadm:Adding mds.c on smithi188 2022-07-03T10:34:24.085 INFO:tasks.cephadm:Adding mds.f on smithi188 2022-07-03T10:34:24.085 INFO:tasks.cephadm:Adding mds.i on smithi188 2022-07-03T10:34:24.086 INFO:tasks.cephadm:Adding mds.l on smithi188 2022-07-03T10:34:24.086 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph orch apply mds all '12;smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l' 2022-07-03T10:34:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:24 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2801659958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:24.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:24 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2801659958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:24 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2801659958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:24.697 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.c/config 2022-07-03T10:34:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:25 smithi161 ceph-mon[91579]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:25.636 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:25 smithi188 ceph-mon[92425]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:25.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:25 smithi101 ceph-mon[85126]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:25.837 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 -- 172.21.15.188:0/2156626152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fd850 msgr2=0x7f5c580fdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:25.838 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2156626152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fd850 0x7f5c580fdca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5c54009900 tx=0x7f5c540091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.838 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 -- 172.21.15.188:0/2156626152 shutdown_connections 2022-07-03T10:34:25.838 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2156626152 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fe1e0 0x7f5c58104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.839 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2156626152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fc5d0 0x7f5c580fc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.839 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.831+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2156626152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fd850 0x7f5c580fdca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.839 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.832+0000 7f5c5f882700 1 -- 172.21.15.188:0/2156626152 >> 172.21.15.188:0/2156626152 conn(0x7f5c580f7d40 msgr2=0x7f5c580fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:25.839 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.832+0000 7f5c5f882700 1 -- 172.21.15.188:0/2156626152 shutdown_connections 2022-07-03T10:34:25.839 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.832+0000 7f5c5f882700 1 -- 172.21.15.188:0/2156626152 wait complete. 2022-07-03T10:34:25.840 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.832+0000 7f5c5f882700 1 Processor -- start 2022-07-03T10:34:25.840 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.832+0000 7f5c5f882700 1 -- start start 2022-07-03T10:34:25.840 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:25.840 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fd850 0x7f5c581133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:25.841 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fe1e0 0x7f5c581159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:25.841 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58116170 con 0x7f5c580fd850 2022-07-03T10:34:25.841 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581162e0 con 0x7f5c580fe1e0 2022-07-03T10:34:25.841 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581164a0 con 0x7f5c580fc5d0 2022-07-03T10:34:25.842 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:25.842 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40936/0 (socket says 172.21.15.188:40936) 2022-07-03T10:34:25.842 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 -- 172.21.15.188:0/2366773669 learned_addr learned my addr 172.21.15.188:0/2366773669 (peer_addr_for_me v2:172.21.15.188:0/0) 2022-07-03T10:34:25.842 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5de1f700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fe1e0 0x7f5c581159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:25.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5ce1d700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fd850 0x7f5c581133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:25.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 -- 172.21.15.188:0/2366773669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fe1e0 msgr2=0x7f5c581159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:25.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fe1e0 0x7f5c581159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 -- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fd850 msgr2=0x7f5c581133d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:25.843 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fd850 0x7f5c581133d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:25.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 -- 172.21.15.188:0/2366773669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c54009440 con 0x7f5c580fc5d0 2022-07-03T10:34:25.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5d61e700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00c9d0 tx=0x7f5c4c00a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:25.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00ce60 con 0x7f5c580fc5d0 2022-07-03T10:34:25.844 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c58116780 con 0x7f5c580fc5d0 2022-07-03T10:34:25.845 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5c4c007900 con 0x7f5c580fc5d0 2022-07-03T10:34:25.845 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.833+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c003a50 con 0x7f5c580fc5d0 2022-07-03T10:34:25.845 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.834+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c58116d60 con 0x7f5c580fc5d0 2022-07-03T10:34:25.845 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.835+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58107d70 con 0x7f5c580fc5d0 2022-07-03T10:34:25.845 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.835+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f5c4c01e060 con 0x7f5c580fc5d0 2022-07-03T10:34:25.846 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.836+0000 7f5c49ffb700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 0x7f5c440991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:25.846 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.836+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5c4c013070 con 0x7f5c580fc5d0 2022-07-03T10:34:25.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.837+0000 7f5c5ce1d700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 0x7f5c440991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:25.847 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.837+0000 7f5c5ce1d700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 0x7f5c440991d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400b5c0 tx=0x7f5c5401a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:25.848 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:25.841+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f5c4c08c990 con 0x7f5c580fc5d0 2022-07-03T10:34:26.024 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.022+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5805ea60 con 0x7f5c44096d20 2022-07-03T10:34:26.033 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.032+0000 7f5c49ffb700 1 -- 172.21.15.188:0/2366773669 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5c5805ea60 con 0x7f5c44096d20 2022-07-03T10:34:26.033 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mds.all update... 2022-07-03T10:34:26.035 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 msgr2=0x7f5c440991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:26.035 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 0x7f5c440991d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400b5c0 tx=0x7f5c5401a040 comp rx=0 tx=0).stop 2022-07-03T10:34:26.035 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 msgr2=0x7f5c58066960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:26.036 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00c9d0 tx=0x7f5c4c00a2e0 comp rx=0 tx=0).stop 2022-07-03T10:34:26.036 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 shutdown_connections 2022-07-03T10:34:26.036 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5c44096d20 0x7f5c440991d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:26.036 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c580fe1e0 0x7f5c581159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:26.037 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c580fc5d0 0x7f5c58066960 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:26.037 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 --2- 172.21.15.188:0/2366773669 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5c580fd850 0x7f5c581133d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:26.037 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 >> 172.21.15.188:0/2366773669 conn(0x7f5c580f7d40 msgr2=0x7f5c580fa070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:26.037 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 shutdown_connections 2022-07-03T10:34:26.037 INFO:teuthology.orchestra.run.smithi188.stderr:2022-07-03T10:34:26.034+0000 7f5c5f882700 1 -- 172.21.15.188:0/2366773669 wait complete. 2022-07-03T10:34:26.518 DEBUG:teuthology.orchestra.run.smithi101:mds.a> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.a.service 2022-07-03T10:34:26.525 DEBUG:teuthology.orchestra.run.smithi101:mds.d> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.d.service 2022-07-03T10:34:26.573 DEBUG:teuthology.orchestra.run.smithi101:mds.g> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.g.service 2022-07-03T10:34:26.579 DEBUG:teuthology.orchestra.run.smithi101:mds.j> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.j.service 2022-07-03T10:34:26.627 DEBUG:teuthology.orchestra.run.smithi161:mds.b> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.b.service 2022-07-03T10:34:26.631 DEBUG:teuthology.orchestra.run.smithi161:mds.e> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.e.service 2022-07-03T10:34:26.633 INFO:journalctl@ceph.mds.a.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:34:26.658 INFO:journalctl@ceph.mds.b.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:34:26.659 DEBUG:teuthology.orchestra.run.smithi161:mds.h> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.h.service 2022-07-03T10:34:26.680 INFO:journalctl@ceph.mds.g.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:34:26.682 INFO:journalctl@ceph.mds.d.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:34:26.684 INFO:journalctl@ceph.mds.e.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:34:26.686 DEBUG:teuthology.orchestra.run.smithi161:mds.k> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.k.service 2022-07-03T10:34:26.711 INFO:journalctl@ceph.mds.h.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:34:26.712 DEBUG:teuthology.orchestra.run.smithi188:mds.c> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.c.service 2022-07-03T10:34:26.715 DEBUG:teuthology.orchestra.run.smithi188:mds.f> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.f.service 2022-07-03T10:34:26.727 INFO:journalctl@ceph.mds.j.smithi101.stdout:-- Logs begin at Sun 2022-07-03 10:01:41 UTC. -- 2022-07-03T10:34:26.737 INFO:journalctl@ceph.mds.k.smithi161.stdout:-- Logs begin at Sun 2022-07-03 09:57:51 UTC. -- 2022-07-03T10:34:26.742 INFO:journalctl@ceph.mds.c.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:34:26.743 DEBUG:teuthology.orchestra.run.smithi188:mds.i> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.i.service 2022-07-03T10:34:26.768 INFO:journalctl@ceph.mds.f.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:34:26.769 DEBUG:teuthology.orchestra.run.smithi188:mds.l> sudo journalctl -f -n 0 -u ceph-23a851ae-fab9-11ec-842c-001a4aab830c@mds.l.service 2022-07-03T10:34:26.795 INFO:journalctl@ceph.mds.i.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:34:26.796 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-07-03T10:34:26.796 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:34:26.821 INFO:journalctl@ceph.mds.l.smithi188.stdout:-- Logs begin at Sun 2022-07-03 09:57:39 UTC. -- 2022-07-03T10:34:27.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 -- 172.21.15.101:0/2187641561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fd5b0 msgr2=0x7fae500fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 --2- 172.21.15.101:0/2187641561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fd5b0 0x7fae500fd9c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae40009860 tx=0x7fae40005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 -- 172.21.15.101:0/2187641561 shutdown_connections 2022-07-03T10:34:27.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 --2- 172.21.15.101:0/2187641561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 --2- 172.21.15.101:0/2187641561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fd5b0 0x7fae500fd9c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 --2- 172.21.15.101:0/2187641561 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fe830 0x7fae500fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.088+0000 7fae555ff700 1 -- 172.21.15.101:0/2187641561 >> 172.21.15.101:0/2187641561 conn(0x7fae500f8d00 msgr2=0x7fae500fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:27.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.089+0000 7fae555ff700 1 -- 172.21.15.101:0/2187641561 shutdown_connections 2022-07-03T10:34:27.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.089+0000 7fae555ff700 1 -- 172.21.15.101:0/2187641561 wait complete. 2022-07-03T10:34:27.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.089+0000 7fae555ff700 1 Processor -- start 2022-07-03T10:34:27.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 -- start start 2022-07-03T10:34:27.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fe830 0x7fae50070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50071480 con 0x7fae500fd5b0 2022-07-03T10:34:27.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae500715f0 con 0x7fae500fe830 2022-07-03T10:34:27.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae555ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50118670 con 0x7fae500ff1c0 2022-07-03T10:34:27.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae4effd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae4effd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60710/0 (socket says 172.21.15.101:60710) 2022-07-03T10:34:27.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae4effd700 1 -- 172.21.15.101:0/3380876077 learned_addr learned my addr 172.21.15.101:0/3380876077 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:27.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae4e7fc700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fe830 0x7fae50070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.090+0000 7fae4f7fe700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 -- 172.21.15.101:0/3380876077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 msgr2=0x7fae50070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 -- 172.21.15.101:0/3380876077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fe830 msgr2=0x7fae50070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fe830 0x7fae50070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 -- 172.21.15.101:0/3380876077 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44009440 con 0x7fae500fd5b0 2022-07-03T10:34:27.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4f7fe700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:27.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae4effd700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fae40009830 tx=0x7fae40005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:27.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.091+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4001d070 con 0x7fae500fd5b0 2022-07-03T10:34:27.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.092+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fae40005590 con 0x7fae500fd5b0 2022-07-03T10:34:27.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.092+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae40009410 con 0x7fae500fd5b0 2022-07-03T10:34:27.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.092+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40017bc0 con 0x7fae500fd5b0 2022-07-03T10:34:27.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.092+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae50118b80 con 0x7fae500fd5b0 2022-07-03T10:34:27.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.093+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae50108d40 con 0x7fae500fd5b0 2022-07-03T10:34:27.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.094+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fae40028020 con 0x7fae500fd5b0 2022-07-03T10:34:27.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.096+0000 7fae377fe700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 0x7fae380991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.096+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fae400c0a00 con 0x7fae500fd5b0 2022-07-03T10:34:27.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.096+0000 7fae4e7fc700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 0x7fae380991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.097+0000 7fae4e7fc700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 0x7fae380991d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae44005fd0 tx=0x7fae440091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:27.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.100+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fae4008e090 con 0x7fae500fd5b0 2022-07-03T10:34:27.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:27.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='client.24428 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:34:27.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: Saving service mds.all spec with placement smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2022-07-03T10:34:27.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:27.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:27.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:34:27.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:34:27.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:27.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:27.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.249 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:27 smithi101 ceph-mon[85126]: Deploying daemon mds.a on smithi101 2022-07-03T10:34:27.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.301+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae50104560 con 0x7fae500fd5b0 2022-07-03T10:34:27.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.302+0000 7fae377fe700 1 -- 172.21.15.101:0/3380876077 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18195 (secure 0 0 0) 0x7fae400916b0 con 0x7fae500fd5b0 2022-07-03T10:34:27.305 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:27.307 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":76,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:34:15.242218+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:34:27.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.305+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 msgr2=0x7fae380991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.305+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 0x7fae380991d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae44005fd0 tx=0x7fae440091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.305+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 msgr2=0x7fae50072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.305+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fae40009830 tx=0x7fae40005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 shutdown_connections 2022-07-03T10:34:27.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae38096d20 0x7fae380991d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae500fd5b0 0x7fae50072230 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae500ff1c0 0x7fae50070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 --2- 172.21.15.101:0/3380876077 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae500fe830 0x7fae50070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 >> 172.21.15.101:0/3380876077 conn(0x7fae500f8d00 msgr2=0x7fae500fb130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:27.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 shutdown_connections 2022-07-03T10:34:27.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.306+0000 7fae555ff700 1 -- 172.21.15.101:0/3380876077 wait complete. 2022-07-03T10:34:27.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-07-03T10:34:27.338 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:34:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='client.24428 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:34:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: Saving service mds.all spec with placement smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2022-07-03T10:34:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:34:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:34:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:27 smithi161 ceph-mon[91579]: Deploying daemon mds.a on smithi101 2022-07-03T10:34:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='client.24428 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:34:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: Saving service mds.all spec with placement smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2022-07-03T10:34:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:34:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:34:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:27 smithi188 ceph-mon[92425]: Deploying daemon mds.a on smithi101 2022-07-03T10:34:27.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 -- 172.21.15.101:0/1127849611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0fe670 msgr2=0x7f57cc0feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 --2- 172.21.15.101:0/1127849611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0fe670 0x7f57cc0feac0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57c0009900 tx=0x7f57c00091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 -- 172.21.15.101:0/1127849611 shutdown_connections 2022-07-03T10:34:27.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 --2- 172.21.15.101:0/1127849611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0ff000 0x7f57cc1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 --2- 172.21.15.101:0/1127849611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc0fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 --2- 172.21.15.101:0/1127849611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0fe670 0x7f57cc0feac0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 -- 172.21.15.101:0/1127849611 >> 172.21.15.101:0/1127849611 conn(0x7f57cc0f8b40 msgr2=0x7f57cc0fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:27.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 -- 172.21.15.101:0/1127849611 shutdown_connections 2022-07-03T10:34:27.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.584+0000 7f57cbfff700 1 -- 172.21.15.101:0/1127849611 wait complete. 2022-07-03T10:34:27.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.585+0000 7f57cbfff700 1 Processor -- start 2022-07-03T10:34:27.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.585+0000 7f57cbfff700 1 -- start start 2022-07-03T10:34:27.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.585+0000 7f57cbfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cbfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 0x7f57cc070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cbfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cbfff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc071480 con 0x7f57cc0ff000 2022-07-03T10:34:27.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cbfff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc0715f0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cbfff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc118480 con 0x7f57cc0fe670 2022-07-03T10:34:27.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cb7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cb7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60716/0 (socket says 172.21.15.101:60716) 2022-07-03T10:34:27.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cb7fe700 1 -- 172.21.15.101:0/3045047293 learned_addr learned my addr 172.21.15.101:0/3045047293 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:27.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57ca7fc700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 0x7f57cc070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 -- 172.21.15.101:0/3045047293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 msgr2=0x7f57cc070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 0x7f57cc070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 -- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 msgr2=0x7f57cc070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57caffd700 1 -- 172.21.15.101:0/3045047293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4009700 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.586+0000 7f57cb7fe700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:27.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57ca7fc700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 0x7f57cc070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:27.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57caffd700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc072230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00bfd0 tx=0x7f57bc00b3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:27.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00f6a0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f57bc00fcb0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c0009440 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc010db0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.587+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc118ab0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.589+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc108be0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.591+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f57bc022070 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.593+0000 7f57b37fe700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:27.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.593+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f57bc0bf2f0 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.593+0000 7f57ca7fc700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:27.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.594+0000 7f57ca7fc700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f57c00056f0 tx=0x7f57c00091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:27.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.597+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f57bc08ca70 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.812+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57cc108d70 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.813+0000 7f57b37fe700 1 -- 172.21.15.101:0/3045047293 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57cc108d70 con 0x7f57cc0fd3f0 2022-07-03T10:34:27.816 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:34:27.819 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 msgr2=0x7f57b40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f57c00056f0 tx=0x7f57c00091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 msgr2=0x7f57cc072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:27.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc072230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00bfd0 tx=0x7f57bc00b3a0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 shutdown_connections 2022-07-03T10:34:27.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57cc0ff000 0x7f57cc070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57cc0fd3f0 0x7f57cc072230 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 --2- 172.21.15.101:0/3045047293 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57cc0fe670 0x7f57cc070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:27.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 >> 172.21.15.101:0/3045047293 conn(0x7f57cc0f8b40 msgr2=0x7f57cc0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:27.823 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 shutdown_connections 2022-07-03T10:34:27.823 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:27.816+0000 7f57cbfff700 1 -- 172.21.15.101:0/3045047293 wait complete. 2022-07-03T10:34:27.853 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:28.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 -- 172.21.15.101:0/3723231298 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 msgr2=0x7fd06c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:28.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 --2- 172.21.15.101:0/3723231298 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1058a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd064009ca0 tx=0x7fd064009520 comp rx=0 tx=0).stop 2022-07-03T10:34:28.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 -- 172.21.15.101:0/3723231298 shutdown_connections 2022-07-03T10:34:28.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 --2- 172.21.15.101:0/3723231298 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1058a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 --2- 172.21.15.101:0/3723231298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c0fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 --2- 172.21.15.101:0/3723231298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fe650 0x7fd06c0feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.096+0000 7fd06bfff700 1 -- 172.21.15.101:0/3723231298 >> 172.21.15.101:0/3723231298 conn(0x7fd06c0f8b20 msgr2=0x7fd06c0faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:28.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.097+0000 7fd06bfff700 1 -- 172.21.15.101:0/3723231298 shutdown_connections 2022-07-03T10:34:28.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.097+0000 7fd06bfff700 1 -- 172.21.15.101:0/3723231298 wait complete. 2022-07-03T10:34:28.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.098+0000 7fd06bfff700 1 Processor -- start 2022-07-03T10:34:28.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.098+0000 7fd06bfff700 1 -- start start 2022-07-03T10:34:28.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:28.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fe650 0x7fd06c110840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:28.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1178b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:28.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c117e30 con 0x7fd06c0fefe0 2022-07-03T10:34:28.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c117f70 con 0x7fd06c0fe650 2022-07-03T10:34:28.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06bfff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c118130 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06b7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1178b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:28.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06affd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:28.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06affd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:40632/0 (socket says 172.21.15.101:40632) 2022-07-03T10:34:28.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06affd700 1 -- 172.21.15.101:0/4291294753 learned_addr learned my addr 172.21.15.101:0/4291294753 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:28.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.099+0000 7fd06a7fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fe650 0x7fd06c110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:28.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 -- 172.21.15.101:0/4291294753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fe650 msgr2=0x7fd06c110840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:28.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fe650 0x7fd06c110840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 -- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 msgr2=0x7fd06c1178b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:28.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1178b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 -- 172.21.15.101:0/4291294753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0640097e0 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06b7fe700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1178b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:28.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06affd700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00ea50 tx=0x7fd05c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:28.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c004020 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd05c00cc70 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c118410 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.100+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c003970 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.112 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.101+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c1189a0 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.112 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.102+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c108bf0 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.105+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd05c00c040 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.106+0000 7fd0537fe700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 0x7fd0540991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:28.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.106+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd05c014070 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.107+0000 7fd06a7fc700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 0x7fd0540991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:28.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.107+0000 7fd06a7fc700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 0x7fd0540991d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd060009ce0 tx=0x7fd060005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:28.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.111+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd05c08cc80 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.205 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:28 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3380876077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:28.205 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:28 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3045047293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:28.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.376+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd06c119040 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.377+0000 7fd0537fe700 1 -- 172.21.15.101:0/4291294753 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fd05c0902a0 con 0x7fd06c0fd3d0 2022-07-03T10:34:28.385 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:28.386 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-07-03T10:34:28.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.379+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 msgr2=0x7fd0540991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:28.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.379+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 0x7fd0540991d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd060009ce0 tx=0x7fd060005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 msgr2=0x7fd06c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:28.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00ea50 tx=0x7fd05c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 shutdown_connections 2022-07-03T10:34:28.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd054096d20 0x7fd0540991d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd06c0fefe0 0x7fd06c1178b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd06c0fd3d0 0x7fd06c110300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 --2- 172.21.15.101:0/4291294753 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd06c0fe650 0x7fd06c110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:28.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 >> 172.21.15.101:0/4291294753 conn(0x7fd06c0f8b20 msgr2=0x7fd06c0fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:28.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 shutdown_connections 2022-07-03T10:34:28.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:28.380+0000 7fd06bfff700 1 -- 172.21.15.101:0/4291294753 wait complete. 2022-07-03T10:34:28.391 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 1 2022-07-03T10:34:28.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:28 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3380876077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:28.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:28 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3045047293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:28 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3380876077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:28 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3045047293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:34:29.198 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:29 smithi101 ceph-mon[85126]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:29.198 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:29 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4291294753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:29 smithi188 ceph-mon[92425]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:29 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4291294753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:29.493 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:29 smithi101 systemd[1]: Starting Ceph mds.a for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:29 smithi161 ceph-mon[91579]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:29 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4291294753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:30.027 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:29 smithi101 podman[113694]: 2022-07-03T10:34:30.390 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:30 smithi101 bash[113694]: 96a23032e64b92d7cef729c40411b95dcfd164ea7c5bc2691af4b75479519fce 2022-07-03T10:34:30.390 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:30 smithi101 ceph-mds[113734]: -- 172.21.15.101:0/945905747 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594f98e4340 con 0x5594f98b8c00 2022-07-03T10:34:30.391 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:30 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-a[113710]: starting mds.a at 2022-07-03T10:34:30.391 INFO:journalctl@ceph.mds.a.smithi101.stdout:Jul 03 10:34:30 smithi101 systemd[1]: Started Ceph mds.a for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:30.412 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:30.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 -- 172.21.15.101:0/1245416693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 msgr2=0x7fe720107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:30.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 --2- 172.21.15.101:0/1245416693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720107b00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe718009bf0 tx=0x7fe718009460 comp rx=0 tx=0).stop 2022-07-03T10:34:30.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 -- 172.21.15.101:0/1245416693 shutdown_connections 2022-07-03T10:34:30.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 --2- 172.21.15.101:0/1245416693 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720107b00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 --2- 172.21.15.101:0/1245416693 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 0x7fe7200ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 --2- 172.21.15.101:0/1245416693 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe720100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 -- 172.21.15.101:0/1245416693 >> 172.21.15.101:0/1245416693 conn(0x7fe7200faef0 msgr2=0x7fe7200fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:30.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 -- 172.21.15.101:0/1245416693 shutdown_connections 2022-07-03T10:34:30.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.645+0000 7fe72739d700 1 -- 172.21.15.101:0/1245416693 wait complete. 2022-07-03T10:34:30.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 Processor -- start 2022-07-03T10:34:30.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 -- start start 2022-07-03T10:34:30.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 0x7fe720110380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:30.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:30.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720117930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:30.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe720117eb0 con 0x7fe7201008b0 2022-07-03T10:34:30.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe720117ff0 con 0x7fe7200ff630 2022-07-03T10:34:30.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.646+0000 7fe72739d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7201181b0 con 0x7fe720101240 2022-07-03T10:34:30.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:30.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:57790/0 (socket says 172.21.15.101:57790) 2022-07-03T10:34:30.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 -- 172.21.15.101:0/3498116214 learned_addr learned my addr 172.21.15.101:0/3498116214 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:30.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe726b9c700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720117930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:30.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe72639b700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 0x7fe720110380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:30.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 -- 172.21.15.101:0/3498116214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 msgr2=0x7fe720117930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:30.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720117930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 -- 172.21.15.101:0/3498116214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 msgr2=0x7fe720110380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:30.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 0x7fe720110380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.647+0000 7fe725b9a700 1 -- 172.21.15.101:0/3498116214 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe718009700 con 0x7fe7201008b0 2022-07-03T10:34:30.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe725b9a700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe71000ea50 tx=0x7fe7100091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:30.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710005020 con 0x7fe7201008b0 2022-07-03T10:34:30.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe71000c7c0 con 0x7fe7201008b0 2022-07-03T10:34:30.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720118490 con 0x7fe7201008b0 2022-07-03T10:34:30.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710004b00 con 0x7fe7201008b0 2022-07-03T10:34:30.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.648+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe720118a20 con 0x7fe7201008b0 2022-07-03T10:34:30.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.649+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72010adf0 con 0x7fe7201008b0 2022-07-03T10:34:30.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.651+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fe71000c040 con 0x7fe7201008b0 2022-07-03T10:34:30.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.652+0000 7fe716ffd700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 0x7fe70c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:30.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.652+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe710014070 con 0x7fe7201008b0 2022-07-03T10:34:30.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.652+0000 7fe72639b700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 0x7fe70c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:30.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.653+0000 7fe72639b700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 0x7fe70c0991d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe720100710 tx=0x7fe71c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:30.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.656+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe71008d180 con 0x7fe7201008b0 2022-07-03T10:34:30.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.899+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7201190c0 con 0x7fe7201008b0 2022-07-03T10:34:30.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.899+0000 7fe716ffd700 1 -- 172.21.15.101:0/3498116214 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7fe7100907a0 con 0x7fe7201008b0 2022-07-03T10:34:30.902 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:30.902 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-07-03T10:34:30.904 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 msgr2=0x7fe70c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:30.904 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 0x7fe70c0991d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe720100710 tx=0x7fe71c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 msgr2=0x7fe7201108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:30.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe71000ea50 tx=0x7fe7100091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 shutdown_connections 2022-07-03T10:34:30.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe70c096d20 0x7fe70c0991d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe720101240 0x7fe720117930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe7200ff630 0x7fe720110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 --2- 172.21.15.101:0/3498116214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7201008b0 0x7fe7201108c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:30.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 >> 172.21.15.101:0/3498116214 conn(0x7fe7200faef0 msgr2=0x7fe7200fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:30.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 shutdown_connections 2022-07-03T10:34:30.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:30.902+0000 7fe72739d700 1 -- 172.21.15.101:0/3498116214 wait complete. 2022-07-03T10:34:30.907 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 2 2022-07-03T10:34:31.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:31.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.101:6834/369400191,v1:172.21.15.101:6835/369400191] up:boot 2022-07-03T10:34:31.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: fsmap 1 up:standby 2022-07-03T10:34:31.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-07-03T10:34:31.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:31.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:31.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:31.247 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: Deploying daemon mds.d on smithi101 2022-07-03T10:34:31.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:31 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3498116214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.101:6834/369400191,v1:172.21.15.101:6835/369400191] up:boot 2022-07-03T10:34:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: fsmap 1 up:standby 2022-07-03T10:34:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-07-03T10:34:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: Deploying daemon mds.d on smithi101 2022-07-03T10:34:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:31 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3498116214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.101:6834/369400191,v1:172.21.15.101:6835/369400191] up:boot 2022-07-03T10:34:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: fsmap 1 up:standby 2022-07-03T10:34:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-07-03T10:34:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: Deploying daemon mds.d on smithi101 2022-07-03T10:34:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:31 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3498116214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:32.932 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:32.992 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:32 smithi101 systemd[1]: Starting Ceph mds.d for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:33.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.195+0000 7faf799ff700 1 -- 172.21.15.101:0/2633499142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74103220 msgr2=0x7faf741056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:33.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.195+0000 7faf799ff700 1 --2- 172.21.15.101:0/2633499142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74103220 0x7faf741056a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009ca0 tx=0x7faf6c009520 comp rx=0 tx=0).stop 2022-07-03T10:34:33.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 -- 172.21.15.101:0/2633499142 shutdown_connections 2022-07-03T10:34:33.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 --2- 172.21.15.101:0/2633499142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74103220 0x7faf741056a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 --2- 172.21.15.101:0/2633499142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74067690 0x7faf74100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 --2- 172.21.15.101:0/2633499142 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 0x7faf74102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 -- 172.21.15.101:0/2633499142 >> 172.21.15.101:0/2633499142 conn(0x7faf740f7c60 msgr2=0x7faf740fa070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:33.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.196+0000 7faf799ff700 1 -- 172.21.15.101:0/2633499142 shutdown_connections 2022-07-03T10:34:33.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.197+0000 7faf799ff700 1 -- 172.21.15.101:0/2633499142 wait complete. 2022-07-03T10:34:33.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 Processor -- start 2022-07-03T10:34:33.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 -- start start 2022-07-03T10:34:33.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74067690 0x7faf740fdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:33.213 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 0x7faf740fc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:33.213 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:33.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740fcd20 con 0x7faf74100750 2022-07-03T10:34:33.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740fce90 con 0x7faf74103220 2022-07-03T10:34:33.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.198+0000 7faf799ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf741183b0 con 0x7faf74067690 2022-07-03T10:34:33.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf727fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 0x7faf740fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:33.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:33.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55830/0 (socket says 172.21.15.101:55830) 2022-07-03T10:34:33.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 -- 172.21.15.101:0/2478842101 learned_addr learned my addr 172.21.15.101:0/2478842101 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:33.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf72ffd700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74067690 0x7faf740fdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:33.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 -- 172.21.15.101:0/2478842101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74067690 msgr2=0x7faf740fdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:33.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74067690 0x7faf740fdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 -- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 msgr2=0x7faf740fc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:33.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 0x7faf740fc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.199+0000 7faf737fe700 1 -- 172.21.15.101:0/2478842101 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64009410 con 0x7faf74103220 2022-07-03T10:34:33.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf737fe700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf6c004bf0 tx=0x7faf6c009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:33.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c01d070 con 0x7faf74103220 2022-07-03T10:34:33.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faf6c004e10 con 0x7faf74103220 2022-07-03T10:34:33.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c00f6d0 con 0x7faf74103220 2022-07-03T10:34:33.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf68009410 con 0x7faf74103220 2022-07-03T10:34:33.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.200+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c0097e0 con 0x7faf74103220 2022-07-03T10:34:33.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.202+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf74108da0 con 0x7faf74103220 2022-07-03T10:34:33.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.203+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7faf6c029030 con 0x7faf74103220 2022-07-03T10:34:33.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.204+0000 7faf5b7fe700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 0x7faf5c099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:33.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.204+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7faf6c022e40 con 0x7faf74103220 2022-07-03T10:34:33.225 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.205+0000 7faf72ffd700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 0x7faf5c099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:33.225 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.205+0000 7faf72ffd700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 0x7faf5c099250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faf64009e90 tx=0x7faf64005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:33.226 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.208+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7faf6c08e700 con 0x7faf74103220 2022-07-03T10:34:33.306 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:33 smithi101 ceph-mon[85126]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:33 smithi161 ceph-mon[91579]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:33 smithi188 ceph-mon[92425]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:33.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.445+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7405f1d0 con 0x7faf74103220 2022-07-03T10:34:33.449 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.446+0000 7faf5b7fe700 1 -- 172.21.15.101:0/2478842101 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7faf6c027020 con 0x7faf74103220 2022-07-03T10:34:33.449 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:33.450 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-07-03T10:34:33.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 msgr2=0x7faf5c099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:33.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 0x7faf5c099250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faf64009e90 tx=0x7faf64005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 msgr2=0x7faf740fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf6c004bf0 tx=0x7faf6c009520 comp rx=0 tx=0).stop 2022-07-03T10:34:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 shutdown_connections 2022-07-03T10:34:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faf5c096da0 0x7faf5c099250 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf74103220 0x7faf740fc650 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf74067690 0x7faf740fdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 --2- 172.21.15.101:0/2478842101 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf74100750 0x7faf740fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:33.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 >> 172.21.15.101:0/2478842101 conn(0x7faf740f7c60 msgr2=0x7faf740fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:33.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 shutdown_connections 2022-07-03T10:34:33.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:33.450+0000 7faf799ff700 1 -- 172.21.15.101:0/2478842101 wait complete. 2022-07-03T10:34:33.455 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 2 2022-07-03T10:34:33.593 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:33 smithi101 podman[114117]: 2022-07-03T10:34:33.917 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:33 smithi101 bash[114117]: a4baaa09326fe81cf04c2c227d64ac7283d751d0014e2983eb104a46eceecba7 2022-07-03T10:34:33.918 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:33 smithi101 ceph-mds[114179]: -- 172.21.15.101:0/2977470691 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a16298a340 con 0x55a16295f000 2022-07-03T10:34:33.918 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:33 smithi101 systemd[1]: Started Ceph mds.d for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:33.918 INFO:journalctl@ceph.mds.d.smithi101.stdout:Jul 03 10:34:33 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-d[114153]: starting mds.d at 2022-07-03T10:34:34.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2478842101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:34.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:34.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:34.246 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:34 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2478842101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:34 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2478842101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:34 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:35.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:35 smithi101 ceph-mon[85126]: Deploying daemon mds.g on smithi101 2022-07-03T10:34:35.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:35 smithi101 ceph-mon[85126]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:35.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:35 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.101:6836/3291034276,v1:172.21.15.101:6837/3291034276] up:boot 2022-07-03T10:34:35.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:35 smithi101 ceph-mon[85126]: fsmap 2 up:standby 2022-07-03T10:34:35.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:35 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-07-03T10:34:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:35 smithi188 ceph-mon[92425]: Deploying daemon mds.g on smithi101 2022-07-03T10:34:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:35 smithi188 ceph-mon[92425]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:35 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.101:6836/3291034276,v1:172.21.15.101:6837/3291034276] up:boot 2022-07-03T10:34:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:35 smithi188 ceph-mon[92425]: fsmap 2 up:standby 2022-07-03T10:34:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:35 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-07-03T10:34:35.481 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:35 smithi161 ceph-mon[91579]: Deploying daemon mds.g on smithi101 2022-07-03T10:34:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:35 smithi161 ceph-mon[91579]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:35 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.101:6836/3291034276,v1:172.21.15.101:6837/3291034276] up:boot 2022-07-03T10:34:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:35 smithi161 ceph-mon[91579]: fsmap 2 up:standby 2022-07-03T10:34:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:35 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-07-03T10:34:35.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.738+0000 7fa238bff700 1 -- 172.21.15.101:0/304287566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 msgr2=0x7fa2340fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:35.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 --2- 172.21.15.101:0/304287566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa2340fec80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa228009860 tx=0x7fa228005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 -- 172.21.15.101:0/304287566 shutdown_connections 2022-07-03T10:34:35.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 --2- 172.21.15.101:0/304287566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340ff1c0 0x7fa234105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 --2- 172.21.15.101:0/304287566 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340fd5b0 0x7fa2340fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 --2- 172.21.15.101:0/304287566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa2340fec80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 -- 172.21.15.101:0/304287566 >> 172.21.15.101:0/304287566 conn(0x7fa2340f8d00 msgr2=0x7fa2340fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:35.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 -- 172.21.15.101:0/304287566 shutdown_connections 2022-07-03T10:34:35.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.739+0000 7fa238bff700 1 -- 172.21.15.101:0/304287566 wait complete. 2022-07-03T10:34:35.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 Processor -- start 2022-07-03T10:34:35.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 -- start start 2022-07-03T10:34:35.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:35.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa234110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:35.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340ff1c0 0x7fa234117dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:35.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341183d0 con 0x7fa2340fe830 2022-07-03T10:34:35.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234118510 con 0x7fa2340ff1c0 2022-07-03T10:34:35.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.740+0000 7fa238bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341186d0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:35.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa231d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa234110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:35.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52110/0 (socket says 172.21.15.101:52110) 2022-07-03T10:34:35.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 -- 172.21.15.101:0/1747494438 learned_addr learned my addr 172.21.15.101:0/1747494438 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:35.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa232d9d700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340ff1c0 0x7fa234117dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:35.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 -- 172.21.15.101:0/1747494438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340ff1c0 msgr2=0x7fa234117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340ff1c0 0x7fa234117dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 -- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 msgr2=0x7fa234110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa234110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:35.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 -- 172.21.15.101:0/1747494438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009410 con 0x7fa2340fd5b0 2022-07-03T10:34:35.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa231d9b700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa234110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:35.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.741+0000 7fa23259c700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa224009fd0 tx=0x7fa224009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:35.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.742+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22400be60 con 0x7fa2340fd5b0 2022-07-03T10:34:35.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.742+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2341189b0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.742+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa224011ae0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.742+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224010b80 con 0x7fa2340fd5b0 2022-07-03T10:34:35.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.742+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa234118f40 con 0x7fa2340fd5b0 2022-07-03T10:34:35.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.743+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234108de0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.745+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa224021070 con 0x7fa2340fd5b0 2022-07-03T10:34:35.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.746+0000 7fa222ffd700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 0x7fa21c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:35.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.746+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fa2240bebf0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.746+0000 7fa231d9b700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 0x7fa21c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:35.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.747+0000 7fa231d9b700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 0x7fa21c0991d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa228009f80 tx=0x7fa228005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:35.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.750+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa22408c370 con 0x7fa2340fd5b0 2022-07-03T10:34:35.994 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.992+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa23410ace0 con 0x7fa2340fd5b0 2022-07-03T10:34:35.995 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.993+0000 7fa222ffd700 1 -- 172.21.15.101:0/1747494438 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7fa224015800 con 0x7fa2340fd5b0 2022-07-03T10:34:35.996 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:35.996 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-07-03T10:34:35.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.995+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 msgr2=0x7fa21c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:35.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.995+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 0x7fa21c0991d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa228009f80 tx=0x7fa228005040 comp rx=0 tx=0).stop 2022-07-03T10:34:35.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.995+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 msgr2=0x7fa234110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:35.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.995+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa224009fd0 tx=0x7fa224009060 comp rx=0 tx=0).stop 2022-07-03T10:34:35.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 shutdown_connections 2022-07-03T10:34:35.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa21c096d20 0x7fa21c0991d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:36.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2340ff1c0 0x7fa234117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:36.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2340fd5b0 0x7fa234110810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:36.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 --2- 172.21.15.101:0/1747494438 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2340fe830 0x7fa234110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:36.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 >> 172.21.15.101:0/1747494438 conn(0x7fa2340f8d00 msgr2=0x7fa2341042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:36.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 shutdown_connections 2022-07-03T10:34:36.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:35.996+0000 7fa238bff700 1 -- 172.21.15.101:0/1747494438 wait complete. 2022-07-03T10:34:36.002 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 3 2022-07-03T10:34:36.340 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1747494438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:36.342 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:36 smithi101 systemd[1]: Starting Ceph mds.g for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1747494438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1747494438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:37.156 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:36 smithi101 podman[114540]: 2022-07-03T10:34:37.409 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:37 smithi101 ceph-mon[85126]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:37.410 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:37 smithi101 bash[114540]: 22942aba4c00c633a37ffd043d7889f30a4df798b623a87beb3bb13de28b3116 2022-07-03T10:34:37.410 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:37 smithi101 ceph-mds[114580]: -- 172.21.15.101:0/385755515 <== mon.0 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca2c2fa340 con 0x55ca2c2ce800 2022-07-03T10:34:37.410 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:37 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-g[114556]: starting mds.g at 2022-07-03T10:34:37.411 INFO:journalctl@ceph.mds.g.smithi101.stdout:Jul 03 10:34:37 smithi101 systemd[1]: Started Ceph mds.g for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:37 smithi188 ceph-mon[92425]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:37 smithi161 ceph-mon[91579]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:38.027 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:38.273 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2288031350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 msgr2=0x7f11a40fd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:38.273 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2288031350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40fd7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1198009860 tx=0x7f1198005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.274 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2288031350 shutdown_connections 2022-07-03T10:34:38.274 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2288031350 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a40fc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.275 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2288031350 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 0x7f11a4104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.275 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2288031350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40fd7d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.275 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2288031350 >> 172.21.15.101:0/2288031350 conn(0x7f11a40f7c20 msgr2=0x7f11a40fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:38.275 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2288031350 shutdown_connections 2022-07-03T10:34:38.276 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.271+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2288031350 wait complete. 2022-07-03T10:34:38.278 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.272+0000 7f11aa3ff700 1 Processor -- start 2022-07-03T10:34:38.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 -- start start 2022-07-03T10:34:38.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:38.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40feaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:38.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 0x7f11a40ff030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:38.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40ff700 con 0x7f11a40fc490 2022-07-03T10:34:38.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a40ff870 con 0x7f11a40fdf00 2022-07-03T10:34:38.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11aa3ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41186c0 con 0x7f11a40fcde0 2022-07-03T10:34:38.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11a3fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:38.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11a3fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:59214/0 (socket says 172.21.15.101:59214) 2022-07-03T10:34:38.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.273+0000 7f11a3fff700 1 -- 172.21.15.101:0/2404858673 learned_addr learned my addr 172.21.15.101:0/2404858673 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:38.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a37fe700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40feaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:38.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a899c700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 0x7f11a40ff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:38.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 -- 172.21.15.101:0/2404858673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 msgr2=0x7f11a40feaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:38.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40feaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 -- 172.21.15.101:0/2404858673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 msgr2=0x7f11a40ff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:38.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 0x7f11a40ff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.283 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 -- 172.21.15.101:0/2404858673 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1194009410 con 0x7f11a40fc490 2022-07-03T10:34:38.283 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.274+0000 7f11a3fff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f119400ea80 tx=0x7f1194005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:38.283 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.275+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194009690 con 0x7f11a40fc490 2022-07-03T10:34:38.283 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.275+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c0097e0 con 0x7f11a40fc490 2022-07-03T10:34:38.284 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.275+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f119400c970 con 0x7f11a40fc490 2022-07-03T10:34:38.284 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.275+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1194003a90 con 0x7f11a40fc490 2022-07-03T10:34:38.284 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.275+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1198009410 con 0x7f11a40fc490 2022-07-03T10:34:38.284 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.276+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a4107c50 con 0x7f11a40fc490 2022-07-03T10:34:38.286 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.278+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f119400c040 con 0x7f11a40fc490 2022-07-03T10:34:38.286 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.279+0000 7f11a0ff9700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 0x7f118c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:38.286 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.279+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1194014070 con 0x7f11a40fc490 2022-07-03T10:34:38.287 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.279+0000 7f11a37fe700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 0x7f118c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:38.287 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.280+0000 7f11a37fe700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 0x7f118c0991d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1198009830 tx=0x7f1198005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:38.288 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.283+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f119408d020 con 0x7f11a40fc490 2022-07-03T10:34:38.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:38.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: Deploying daemon mds.j on smithi101 2022-07-03T10:34:38.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.101:6838/3965232315,v1:172.21.15.101:6839/3965232315] up:boot 2022-07-03T10:34:38.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: fsmap 3 up:standby 2022-07-03T10:34:38.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:38 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-07-03T10:34:38.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.526+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a4119340 con 0x7f11a40fc490 2022-07-03T10:34:38.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.527+0000 7f11a0ff9700 1 -- 172.21.15.101:0/2404858673 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f1194090640 con 0x7f11a40fc490 2022-07-03T10:34:38.531 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:38.531 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-07-03T10:34:38.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 msgr2=0x7f118c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:38.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 0x7f118c0991d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1198009830 tx=0x7f1198005040 comp rx=0 tx=0).stop 2022-07-03T10:34:38.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 msgr2=0x7f11a41004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:38.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f119400ea80 tx=0x7f1194005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 shutdown_connections 2022-07-03T10:34:38.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f118c096d20 0x7f118c0991d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f11a40fc490 0x7f11a41004b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11a40fdf00 0x7f11a40ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 --2- 172.21.15.101:0/2404858673 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f11a40fcde0 0x7f11a40feaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:38.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 >> 172.21.15.101:0/2404858673 conn(0x7f11a40f7c20 msgr2=0x7f11a4065710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:38.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 shutdown_connections 2022-07-03T10:34:38.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:38.530+0000 7f11aa3ff700 1 -- 172.21.15.101:0/2404858673 wait complete. 2022-07-03T10:34:38.536 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 4 2022-07-03T10:34:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: Deploying daemon mds.j on smithi101 2022-07-03T10:34:38.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.101:6838/3965232315,v1:172.21.15.101:6839/3965232315] up:boot 2022-07-03T10:34:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: fsmap 3 up:standby 2022-07-03T10:34:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:38 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-07-03T10:34:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: Deploying daemon mds.j on smithi101 2022-07-03T10:34:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.101:6838/3965232315,v1:172.21.15.101:6839/3965232315] up:boot 2022-07-03T10:34:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: fsmap 3 up:standby 2022-07-03T10:34:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:38 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-07-03T10:34:39.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:39 smithi101 ceph-mon[85126]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:39.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:39 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2404858673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:39 smithi161 ceph-mon[91579]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:39 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2404858673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:39 smithi188 ceph-mon[92425]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:39 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2404858673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:40.242 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:39 smithi101 systemd[1]: Starting Ceph mds.j for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:40.563 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:40.742 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:40 smithi101 podman[114939]: 2022-07-03T10:34:40.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.812+0000 7fd166bff700 1 -- 172.21.15.101:0/3761145489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1600723f0 msgr2=0x7fd1600708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:40.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.812+0000 7fd166bff700 1 --2- 172.21.15.101:0/3761145489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1600723f0 0x7fd1600708d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd154009900 tx=0x7fd1540091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 -- 172.21.15.101:0/3761145489 shutdown_connections 2022-07-03T10:34:40.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 --2- 172.21.15.101:0/3761145489 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 --2- 172.21.15.101:0/3761145489 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1600723f0 0x7fd1600708d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.816 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 --2- 172.21.15.101:0/3761145489 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd160070e10 0x7fd160071280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.816 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 -- 172.21.15.101:0/3761145489 >> 172.21.15.101:0/3761145489 conn(0x7fd1600f8d60 msgr2=0x7fd1600fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:40.816 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 -- 172.21.15.101:0/3761145489 shutdown_connections 2022-07-03T10:34:40.817 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.813+0000 7fd166bff700 1 -- 172.21.15.101:0/3761145489 wait complete. 2022-07-03T10:34:40.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.814+0000 7fd166bff700 1 Processor -- start 2022-07-03T10:34:40.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.814+0000 7fd166bff700 1 -- start start 2022-07-03T10:34:40.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:40.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1600723f0 0x7fd160110d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:40.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160117d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:40.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601182f0 con 0x7fd160105520 2022-07-03T10:34:40.823 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160118430 con 0x7fd1600723f0 2022-07-03T10:34:40.823 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd166bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601185f0 con 0x7fd160070e10 2022-07-03T10:34:40.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd16499b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:40.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd16499b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:54396/0 (socket says 172.21.15.101:54396) 2022-07-03T10:34:40.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd16499b700 1 -- 172.21.15.101:0/3182973602 learned_addr learned my addr 172.21.15.101:0/3182973602 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:40.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd15ffff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1600723f0 0x7fd160110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:40.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.815+0000 7fd16519c700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160117d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:40.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 -- 172.21.15.101:0/3182973602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1600723f0 msgr2=0x7fd160110d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:40.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1600723f0 0x7fd160110d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 -- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 msgr2=0x7fd160117d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:40.826 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160117d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:40.826 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 -- 172.21.15.101:0/3182973602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154009440 con 0x7fd160070e10 2022-07-03T10:34:40.826 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16519c700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160117d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:40.827 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.816+0000 7fd16499b700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd154005cf0 tx=0x7fd1540091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:40.827 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.817+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15401d070 con 0x7fd160070e10 2022-07-03T10:34:40.827 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.817+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd15400b9c0 con 0x7fd160070e10 2022-07-03T10:34:40.827 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.817+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd160118870 con 0x7fd160070e10 2022-07-03T10:34:40.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.817+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154022b70 con 0x7fd160070e10 2022-07-03T10:34:40.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.817+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd160118d70 con 0x7fd160070e10 2022-07-03T10:34:40.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.818+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16010afa0 con 0x7fd160070e10 2022-07-03T10:34:40.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.820+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd154029030 con 0x7fd160070e10 2022-07-03T10:34:40.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.822+0000 7fd15d7fa700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 0x7fd150099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:40.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.822+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd1540c0930 con 0x7fd160070e10 2022-07-03T10:34:40.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.822+0000 7fd15ffff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 0x7fd150099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:40.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.823+0000 7fd15ffff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 0x7fd150099250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009d00 tx=0x7fd14c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:40.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:40.823+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd15408dfc0 con 0x7fd160070e10 2022-07-03T10:34:41.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.087+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd160103b30 con 0x7fd160070e10 2022-07-03T10:34:41.090 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.088+0000 7fd15d7fa700 1 -- 172.21.15.101:0/3182973602 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fd1540915e0 con 0x7fd160070e10 2022-07-03T10:34:41.090 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:41.090 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-07-03T10:34:41.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 msgr2=0x7fd150099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:41.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 0x7fd150099250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009d00 tx=0x7fd14c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 msgr2=0x7fd1601107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:41.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd154005cf0 tx=0x7fd1540091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 shutdown_connections 2022-07-03T10:34:41.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd150096da0 0x7fd150099250 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd160105520 0x7fd160117d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd160070e10 0x7fd1601107c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 --2- 172.21.15.101:0/3182973602 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd1600723f0 0x7fd160110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:41.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 >> 172.21.15.101:0/3182973602 conn(0x7fd1600f8d60 msgr2=0x7fd1600fb140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:41.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 shutdown_connections 2022-07-03T10:34:41.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:41.091+0000 7fd166bff700 1 -- 172.21.15.101:0/3182973602 wait complete. 2022-07-03T10:34:41.096 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 4 2022-07-03T10:34:41.117 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:40 smithi101 bash[114939]: 757dd143ed7b0107bd40b86a695eaed3badb433366f73201e80a1008edfd9102 2022-07-03T10:34:41.117 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:40 smithi101 ceph-mds[115002]: -- 172.21.15.101:0/1548306411 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ecce97e340 con 0x55ecce952c00 2022-07-03T10:34:41.117 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:40 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-j[114964]: starting mds.j at 2022-07-03T10:34:41.118 INFO:journalctl@ceph.mds.j.smithi101.stdout:Jul 03 10:34:40 smithi101 systemd[1]: Started Ceph mds.j for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:41.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:41 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3182973602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:41.606 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:41.607 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.607 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.607 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:41.608 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:41.608 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:41.608 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:41 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3182973602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:41 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3182973602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:42.630 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 systemd[1]: Starting Ceph mds.b for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-mon[91579]: Deploying daemon mds.b on smithi161 2022-07-03T10:34:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.101:6840/2161782899,v1:172.21.15.101:6841/2161782899] up:boot 2022-07-03T10:34:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-mon[91579]: fsmap 4 up:standby 2022-07-03T10:34:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-07-03T10:34:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:42 smithi188 ceph-mon[92425]: Deploying daemon mds.b on smithi161 2022-07-03T10:34:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:42 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.101:6840/2161782899,v1:172.21.15.101:6841/2161782899] up:boot 2022-07-03T10:34:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:42 smithi188 ceph-mon[92425]: fsmap 4 up:standby 2022-07-03T10:34:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:42 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-07-03T10:34:42.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:42 smithi101 ceph-mon[85126]: Deploying daemon mds.b on smithi161 2022-07-03T10:34:42.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:42 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.101:6840/2161782899,v1:172.21.15.101:6841/2161782899] up:boot 2022-07-03T10:34:42.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:42 smithi101 ceph-mon[85126]: fsmap 4 up:standby 2022-07-03T10:34:42.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:42 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-07-03T10:34:42.977 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 podman[112157]: 2022-07-03T10:34:42.978 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 bash[112157]: e9e9f7d1cb36a48f268a5471c8a27199f99fa37a4b603071bc784e7338997931 2022-07-03T10:34:42.978 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-mds[112197]: -- 172.21.15.161:0/4192445698 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55abc9204340 con 0x55abc91d8800 2022-07-03T10:34:42.978 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-b[112173]: starting mds.b at 2022-07-03T10:34:43.122 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:43.352 INFO:journalctl@ceph.mds.b.smithi161.stdout:Jul 03 10:34:42 smithi161 systemd[1]: Started Ceph mds.b for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:43.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 -- 172.21.15.101:0/3307967491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fe660 msgr2=0x7fd2900feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:43.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3307967491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fe660 0x7fd2900feab0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009900 tx=0x7fd27c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 -- 172.21.15.101:0/3307967491 shutdown_connections 2022-07-03T10:34:43.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3307967491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900fd3e0 0x7fd2900fd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3307967491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900feff0 0x7fd2901058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3307967491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fe660 0x7fd2900feab0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 -- 172.21.15.101:0/3307967491 >> 172.21.15.101:0/3307967491 conn(0x7fd2900f8b30 msgr2=0x7fd2900fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:43.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 -- 172.21.15.101:0/3307967491 shutdown_connections 2022-07-03T10:34:43.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.359+0000 7fd295b9d700 1 -- 172.21.15.101:0/3307967491 wait complete. 2022-07-03T10:34:43.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.360+0000 7fd295b9d700 1 Processor -- start 2022-07-03T10:34:43.364 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.360+0000 7fd295b9d700 1 -- start start 2022-07-03T10:34:43.366 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fd3e0 0x7fd29010e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:43.366 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:43.366 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900feff0 0x7fd290115950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:43.367 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290115ed0 con 0x7fd2900feff0 2022-07-03T10:34:43.367 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290116010 con 0x7fd2900fe660 2022-07-03T10:34:43.367 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd295b9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2901161d0 con 0x7fd2900fd3e0 2022-07-03T10:34:43.367 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:43.368 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd294b9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fd3e0 0x7fd29010e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:43.368 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55386/0 (socket says 172.21.15.101:55386) 2022-07-03T10:34:43.368 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd29539c700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900feff0 0x7fd290115950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:43.368 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 -- 172.21.15.101:0/3250583395 learned_addr learned my addr 172.21.15.101:0/3250583395 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:43.368 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 -- 172.21.15.101:0/3250583395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fd3e0 msgr2=0x7fd29010e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:43.369 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fd3e0 0x7fd29010e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.369 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 -- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900feff0 msgr2=0x7fd290115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:43.369 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900feff0 0x7fd290115950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.369 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.361+0000 7fd28ffff700 1 -- 172.21.15.101:0/3250583395 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284009410 con 0x7fd2900fe660 2022-07-03T10:34:43.370 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd28ffff700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005d00 tx=0x7fd27c0091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:43.370 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c01d070 con 0x7fd2900fe660 2022-07-03T10:34:43.370 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd27c00b9c0 con 0x7fd2900fe660 2022-07-03T10:34:43.370 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c022b80 con 0x7fd2900fe660 2022-07-03T10:34:43.371 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c009440 con 0x7fd2900fe660 2022-07-03T10:34:43.371 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.362+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd290116760 con 0x7fd2900fe660 2022-07-03T10:34:43.371 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.363+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd290108be0 con 0x7fd2900fe660 2022-07-03T10:34:43.371 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.364+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd27c029030 con 0x7fd2900fe660 2022-07-03T10:34:43.372 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.366+0000 7fd28d7fa700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 0x7fd280099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:43.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.366+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd27c0c0660 con 0x7fd2900fe660 2022-07-03T10:34:43.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.366+0000 7fd294b9b700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 0x7fd280099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:43.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.367+0000 7fd294b9b700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 0x7fd280099250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd2900fe4c0 tx=0x7fd284005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:43.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.370+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd27c05a100 con 0x7fd2900fe660 2022-07-03T10:34:43.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:43.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:43.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:43.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:43 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:43.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.601+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd29005f1d0 con 0x7fd2900fe660 2022-07-03T10:34:43.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.602+0000 7fd28d7fa700 1 -- 172.21.15.101:0/3250583395 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4297 (secure 0 0 0) 0x7fd27c08dcf0 con 0x7fd2900fe660 2022-07-03T10:34:43.603 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:43.604 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2022-07-03T10:34:43.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 msgr2=0x7fd280099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:43.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 0x7fd280099250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd2900fe4c0 tx=0x7fd284005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 msgr2=0x7fd29010e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:43.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd27c005d00 tx=0x7fd27c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 shutdown_connections 2022-07-03T10:34:43.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd280096da0 0x7fd280099250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2900feff0 0x7fd290115950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2900fd3e0 0x7fd29010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 --2- 172.21.15.101:0/3250583395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd2900fe660 0x7fd29010e8e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:43.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 >> 172.21.15.101:0/3250583395 conn(0x7fd2900f8b30 msgr2=0x7fd2900faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:43.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 shutdown_connections 2022-07-03T10:34:43.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:43.605+0000 7fd295b9d700 1 -- 172.21.15.101:0/3250583395 wait complete. 2022-07-03T10:34:43.609 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 6 2022-07-03T10:34:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:43 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:43 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mon[91579]: Deploying daemon mds.e on smithi161 2022-07-03T10:34:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.161:6832/2910285108,v1:172.21.15.161:6833/2910285108] up:boot 2022-07-03T10:34:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mon[91579]: fsmap 5 up:standby 2022-07-03T10:34:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-07-03T10:34:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3250583395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:44.634 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 systemd[1]: Starting Ceph mds.e for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:44 smithi188 ceph-mon[92425]: Deploying daemon mds.e on smithi161 2022-07-03T10:34:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.161:6832/2910285108,v1:172.21.15.161:6833/2910285108] up:boot 2022-07-03T10:34:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:44 smithi188 ceph-mon[92425]: fsmap 5 up:standby 2022-07-03T10:34:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:44 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-07-03T10:34:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:44 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3250583395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:44.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:44 smithi101 ceph-mon[85126]: Deploying daemon mds.e on smithi161 2022-07-03T10:34:44.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:44 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.161:6832/2910285108,v1:172.21.15.161:6833/2910285108] up:boot 2022-07-03T10:34:44.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:44 smithi101 ceph-mon[85126]: fsmap 5 up:standby 2022-07-03T10:34:44.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:44 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-07-03T10:34:44.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:44 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3250583395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:44.999 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 podman[112509]: 2022-07-03T10:34:44.999 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 bash[112509]: 1cf8271c00708fea52b3988541bf724ab59f4807756a54706c366042a8748839 2022-07-03T10:34:45.000 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-mds[112550]: -- 172.21.15.161:0/2869916296 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c456da340 con 0x561c456af000 2022-07-03T10:34:45.000 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-e[112525]: starting mds.e at 2022-07-03T10:34:45.365 INFO:journalctl@ceph.mds.e.smithi161.stdout:Jul 03 10:34:44 smithi161 systemd[1]: Started Ceph mds.e for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:45 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:45.639 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:45 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:45.721 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:45.721 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.721 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:45.722 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:45.722 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:45.722 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:45 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:45.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.870+0000 7ff7e8fff700 1 -- 172.21.15.101:0/3596923338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40fe830 msgr2=0x7ff7e40fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:45.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.870+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/3596923338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40fe830 0x7ff7e40fec80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009900 tx=0x7ff7d80091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.870+0000 7ff7e8fff700 1 -- 172.21.15.101:0/3596923338 shutdown_connections 2022-07-03T10:34:45.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.870+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/3596923338 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e40fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.870+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/3596923338 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40ff1c0 0x7ff7e4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.871+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/3596923338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40fe830 0x7ff7e40fec80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.871+0000 7ff7e8fff700 1 -- 172.21.15.101:0/3596923338 >> 172.21.15.101:0/3596923338 conn(0x7ff7e40f8d00 msgr2=0x7ff7e40fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:45.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.871+0000 7ff7e8fff700 1 -- 172.21.15.101:0/3596923338 shutdown_connections 2022-07-03T10:34:45.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.871+0000 7ff7e8fff700 1 -- 172.21.15.101:0/3596923338 wait complete. 2022-07-03T10:34:45.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.871+0000 7ff7e8fff700 1 Processor -- start 2022-07-03T10:34:45.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 -- start start 2022-07-03T10:34:45.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:45.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fe830 0x7ff7e4110ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:45.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40ff1c0 0x7ff7e4117d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:45.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e4118290 con 0x7ff7e40fe830 2022-07-03T10:34:45.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e41183d0 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e8fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e4118590 con 0x7ff7e40ff1c0 2022-07-03T10:34:45.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:45.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55402/0 (socket says 172.21.15.101:55402) 2022-07-03T10:34:45.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e259c700 1 -- 172.21.15.101:0/2944879723 learned_addr learned my addr 172.21.15.101:0/2944879723 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:45.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e1d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fe830 0x7ff7e4110ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:45.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.872+0000 7ff7e2d9d700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40ff1c0 0x7ff7e4117d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:45.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 -- 172.21.15.101:0/2944879723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40ff1c0 msgr2=0x7ff7e4117d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:45.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40ff1c0 0x7ff7e4117d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 -- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fe830 msgr2=0x7ff7e4110ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:45.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fe830 0x7ff7e4110ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:45.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 -- 172.21.15.101:0/2944879723 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8009440 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e259c700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400ea50 tx=0x7ff7d4005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:45.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4004020 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e4118870 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff7d400cc70 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40039d0 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.873+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e4118e00 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.874+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e4108dc0 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.876+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7ff7d400c040 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.877+0000 7ff7d2ffd700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 0x7ff7cc0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:45.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.877+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff7d4014070 con 0x7ff7e40fd5b0 2022-07-03T10:34:45.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.878+0000 7ff7e1d9b700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 0x7ff7cc0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:45.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.878+0000 7ff7e1d9b700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 0x7ff7cc0991d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8005d00 tx=0x7ff7d80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:45.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:45.881+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7ff7d408cc90 con 0x7ff7e40fd5b0 2022-07-03T10:34:46.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.112+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7e4108f50 con 0x7ff7e40fd5b0 2022-07-03T10:34:46.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.113+0000 7ff7d2ffd700 1 -- 172.21.15.101:0/2944879723 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7ff7e4108f50 con 0x7ff7e40fd5b0 2022-07-03T10:34:46.115 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:46.115 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2022-07-03T10:34:46.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 msgr2=0x7ff7cc0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:46.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 0x7ff7cc0991d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8005d00 tx=0x7ff7d80091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 msgr2=0x7ff7e4110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:46.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400ea50 tx=0x7ff7d4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 shutdown_connections 2022-07-03T10:34:46.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff7cc096d20 0x7ff7cc0991d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7e40ff1c0 0x7ff7e4117d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7e40fd5b0 0x7ff7e4110760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 --2- 172.21.15.101:0/2944879723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7e40fe830 0x7ff7e4110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:46.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.115+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 >> 172.21.15.101:0/2944879723 conn(0x7ff7e40f8d00 msgr2=0x7ff7e40fb030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:46.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.116+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 shutdown_connections 2022-07-03T10:34:46.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:46.116+0000 7ff7e8fff700 1 -- 172.21.15.101:0/2944879723 wait complete. 2022-07-03T10:34:46.121 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 7 2022-07-03T10:34:46.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:46 smithi101 ceph-mon[85126]: Deploying daemon mds.h on smithi161 2022-07-03T10:34:46.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:46 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.161:6834/429921526,v1:172.21.15.161:6835/429921526] up:boot 2022-07-03T10:34:46.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:46 smithi101 ceph-mon[85126]: fsmap 6 up:standby 2022-07-03T10:34:46.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:46 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-07-03T10:34:46.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:46 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2944879723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:46.630 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:46 smithi161 systemd[1]: Starting Ceph mds.h for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:46 smithi161 ceph-mon[91579]: Deploying daemon mds.h on smithi161 2022-07-03T10:34:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:46 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.161:6834/429921526,v1:172.21.15.161:6835/429921526] up:boot 2022-07-03T10:34:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:46 smithi161 ceph-mon[91579]: fsmap 6 up:standby 2022-07-03T10:34:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:46 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-07-03T10:34:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:46 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2944879723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:46 smithi188 ceph-mon[92425]: Deploying daemon mds.h on smithi161 2022-07-03T10:34:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:46 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.161:6834/429921526,v1:172.21.15.161:6835/429921526] up:boot 2022-07-03T10:34:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:46 smithi188 ceph-mon[92425]: fsmap 6 up:standby 2022-07-03T10:34:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:46 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-07-03T10:34:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:46 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2944879723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:47.099 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:46 smithi161 podman[112861]: 2022-07-03T10:34:47.100 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:46 smithi161 bash[112861]: 7dc2d86a028593977f8d8882a4992e1089093e94d7f11bd75a55d4cfbb827d94 2022-07-03T10:34:47.100 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mds[112901]: -- 172.21.15.161:0/2061883141 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55802fe56340 con 0x55802fe2ac00 2022-07-03T10:34:47.100 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-h[112877]: starting mds.h at 2022-07-03T10:34:47.100 INFO:journalctl@ceph.mds.h.smithi161.stdout:Jul 03 10:34:47 smithi161 systemd[1]: Started Ceph mds.h for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:47 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:47 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:47.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:47.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:47.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:47 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:48.145 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:48.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.376+0000 7f7b6db9d700 1 -- 172.21.15.101:0/951233857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b680674d0 msgr2=0x7f7b68100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:48.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.376+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/951233857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b680674d0 0x7f7b68100090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009900 tx=0x7f7b5c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 -- 172.21.15.101:0/951233857 shutdown_connections 2022-07-03T10:34:48.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/951233857 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b68103070 0x7f7b681054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/951233857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b680674d0 0x7f7b68100090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/951233857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 0x7f7b681029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 -- 172.21.15.101:0/951233857 >> 172.21.15.101:0/951233857 conn(0x7f7b680f7a60 msgr2=0x7f7b680f9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:48.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 -- 172.21.15.101:0/951233857 shutdown_connections 2022-07-03T10:34:48.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.377+0000 7f7b6db9d700 1 -- 172.21.15.101:0/951233857 wait complete. 2022-07-03T10:34:48.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 Processor -- start 2022-07-03T10:34:48.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 -- start start 2022-07-03T10:34:48.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 0x7f7b680fd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:48.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 0x7f7b680fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:48.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 0x7f7b680fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:48.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680fcba0 con 0x7f7b680674d0 2022-07-03T10:34:48.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680fcd10 con 0x7f7b68103070 2022-07-03T10:34:48.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6db9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b68118540 con 0x7f7b681005d0 2022-07-03T10:34:48.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.378+0000 7f7b6cb9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 0x7f7b680fd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:48.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 0x7f7b680fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:48.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6cb9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 0x7f7b680fd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60642/0 (socket says 172.21.15.101:60642) 2022-07-03T10:34:48.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6cb9b700 1 -- 172.21.15.101:0/2438046488 learned_addr learned my addr 172.21.15.101:0/2438046488 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:48.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b67fff700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 0x7f7b680fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:48.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 -- 172.21.15.101:0/2438046488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 msgr2=0x7f7b680fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:48.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 0x7f7b680fbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 -- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 msgr2=0x7f7b680fd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:48.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 0x7f7b680fd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 -- 172.21.15.101:0/2438046488 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c009440 con 0x7f7b68103070 2022-07-03T10:34:48.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.379+0000 7f7b6d39c700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 0x7f7b680fc4d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000eab0 tx=0x7f7b60009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:48.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.380+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60009850 con 0x7f7b68103070 2022-07-03T10:34:48.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.380+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7b6000c9d0 con 0x7f7b68103070 2022-07-03T10:34:48.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.380+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b681187a0 con 0x7f7b68103070 2022-07-03T10:34:48.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.380+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60004020 con 0x7f7b68103070 2022-07-03T10:34:48.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.380+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68118d30 con 0x7f7b68103070 2022-07-03T10:34:48.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.381+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b68108be0 con 0x7f7b68103070 2022-07-03T10:34:48.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.383+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f7b6001e020 con 0x7f7b68103070 2022-07-03T10:34:48.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.384+0000 7f7b657fa700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 0x7f7b58099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:48.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.384+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f7b60014070 con 0x7f7b68103070 2022-07-03T10:34:48.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.384+0000 7f7b6cb9b700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 0x7f7b58099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:48.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.385+0000 7f7b6cb9b700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 0x7f7b58099250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0092d0 tx=0x7f7b5c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:48.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.390+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f7b6008d110 con 0x7f7b68103070 2022-07-03T10:34:48.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:48 smithi101 ceph-mon[85126]: Deploying daemon mds.k on smithi161 2022-07-03T10:34:48.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:48 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.161:6836/1658187203,v1:172.21.15.161:6837/1658187203] up:boot 2022-07-03T10:34:48.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:48 smithi101 ceph-mon[85126]: fsmap 7 up:standby 2022-07-03T10:34:48.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:48 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-07-03T10:34:48.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.622+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b68108d70 con 0x7f7b68103070 2022-07-03T10:34:48.629 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.622+0000 7f7b657fa700 1 -- 172.21.15.101:0/2438046488 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5814 (secure 0 0 0) 0x7f7b68108d70 con 0x7f7b68103070 2022-07-03T10:34:48.629 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:48.630 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-07-03T10:34:48.631 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 msgr2=0x7f7b58099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:48.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 0x7f7b58099250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0092d0 tx=0x7f7b5c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 msgr2=0x7f7b680fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:48.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 0x7f7b680fc4d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000eab0 tx=0x7f7b60009460 comp rx=0 tx=0).stop 2022-07-03T10:34:48.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 shutdown_connections 2022-07-03T10:34:48.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7b58096da0 0x7f7b58099250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7b680674d0 0x7f7b680fd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7b68103070 0x7f7b680fc4d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 --2- 172.21.15.101:0/2438046488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b681005d0 0x7f7b680fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:48.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 >> 172.21.15.101:0/2438046488 conn(0x7f7b680f7a60 msgr2=0x7f7b680fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:48.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 shutdown_connections 2022-07-03T10:34:48.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:48.625+0000 7f7b6db9d700 1 -- 172.21.15.101:0/2438046488 wait complete. 2022-07-03T10:34:48.634 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 8 2022-07-03T10:34:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:48 smithi161 ceph-mon[91579]: Deploying daemon mds.k on smithi161 2022-07-03T10:34:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:48 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.161:6836/1658187203,v1:172.21.15.161:6837/1658187203] up:boot 2022-07-03T10:34:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:48 smithi161 ceph-mon[91579]: fsmap 7 up:standby 2022-07-03T10:34:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:48 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-07-03T10:34:48.645 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:48 smithi161 systemd[1]: Starting Ceph mds.k for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:48 smithi188 ceph-mon[92425]: Deploying daemon mds.k on smithi161 2022-07-03T10:34:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:48 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.161:6836/1658187203,v1:172.21.15.161:6837/1658187203] up:boot 2022-07-03T10:34:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:48 smithi188 ceph-mon[92425]: fsmap 7 up:standby 2022-07-03T10:34:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:48 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-07-03T10:34:49.215 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:48 smithi161 podman[113211]: 2022-07-03T10:34:49.216 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:49 smithi161 bash[113211]: f43dafd10881fd424c2c688bd3d4fab53cce232d153161aa804d7c5d6766e7f5 2022-07-03T10:34:49.630 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mds[113252]: -- 172.21.15.161:0/961514902 <== mon.1 v2:172.21.15.161:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56534d950340 con 0x56534d924800 2022-07-03T10:34:49.630 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-k[113227]: starting mds.k at 2022-07-03T10:34:49.631 INFO:journalctl@ceph.mds.k.smithi161.stdout:Jul 03 10:34:49 smithi161 systemd[1]: Started Ceph mds.k for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2438046488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:49 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2438046488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:49 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:49.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:49.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2438046488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:49.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:49.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:49.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:49.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:49 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:50.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:50 smithi188 ceph-mon[92425]: Deploying daemon mds.c on smithi188 2022-07-03T10:34:50.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:50 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.161:6838/2119471810,v1:172.21.15.161:6839/2119471810] up:boot 2022-07-03T10:34:50.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:50 smithi188 ceph-mon[92425]: fsmap 8 up:standby 2022-07-03T10:34:50.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:50 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-07-03T10:34:50.655 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:50.737 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:50 smithi101 ceph-mon[85126]: Deploying daemon mds.c on smithi188 2022-07-03T10:34:50.737 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:50 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.161:6838/2119471810,v1:172.21.15.161:6839/2119471810] up:boot 2022-07-03T10:34:50.737 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:50 smithi101 ceph-mon[85126]: fsmap 8 up:standby 2022-07-03T10:34:50.738 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:50 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-07-03T10:34:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:50 smithi161 ceph-mon[91579]: Deploying daemon mds.c on smithi188 2022-07-03T10:34:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:50 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.161:6838/2119471810,v1:172.21.15.161:6839/2119471810] up:boot 2022-07-03T10:34:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:50 smithi161 ceph-mon[91579]: fsmap 8 up:standby 2022-07-03T10:34:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:50 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-07-03T10:34:50.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 -- 172.21.15.101:0/1843507809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 msgr2=0x7f82680fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:50.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 --2- 172.21.15.101:0/1843507809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f82680fd9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8258009900 tx=0x7f82580091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 -- 172.21.15.101:0/1843507809 shutdown_connections 2022-07-03T10:34:50.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 --2- 172.21.15.101:0/1843507809 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680ff1c0 0x7f8268105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 --2- 172.21.15.101:0/1843507809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f82680fd9c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 --2- 172.21.15.101:0/1843507809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680fe830 0x7f82680fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 -- 172.21.15.101:0/1843507809 >> 172.21.15.101:0/1843507809 conn(0x7f82680f8d00 msgr2=0x7f82680fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:50.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 -- 172.21.15.101:0/1843507809 shutdown_connections 2022-07-03T10:34:50.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.887+0000 7f826d7ff700 1 -- 172.21.15.101:0/1843507809 wait complete. 2022-07-03T10:34:50.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.888+0000 7f826d7ff700 1 Processor -- start 2022-07-03T10:34:50.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.888+0000 7f826d7ff700 1 -- start start 2022-07-03T10:34:50.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:50.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:50.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680ff1c0 0x7f8268115b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:50.902 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82681160d0 con 0x7f82680fe830 2022-07-03T10:34:50.902 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8268116210 con 0x7f82680fd5b0 2022-07-03T10:34:50.903 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f826d7ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82681163d0 con 0x7f82680ff1c0 2022-07-03T10:34:50.904 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f82667fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:50.904 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f8266ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:50.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f82667fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:60652/0 (socket says 172.21.15.101:60652) 2022-07-03T10:34:50.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f8266ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55504/0 (socket says 172.21.15.101:55504) 2022-07-03T10:34:50.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f82667fc700 1 -- 172.21.15.101:0/880533302 learned_addr learned my addr 172.21.15.101:0/880533302 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:50.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.889+0000 7f82677fe700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680ff1c0 0x7f8268115b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:50.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 -- 172.21.15.101:0/880533302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680ff1c0 msgr2=0x7f8268115b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:50.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680ff1c0 0x7f8268115b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 -- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 msgr2=0x7f826810eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:50.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:50.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 -- 172.21.15.101:0/880533302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260009700 con 0x7f82680fd5b0 2022-07-03T10:34:50.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f82667fc700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:50.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f8266ffd700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8258006010 tx=0x7f82580091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:50.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825801d070 con 0x7f82680fd5b0 2022-07-03T10:34:50.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f825800b9c0 con 0x7f82680fd5b0 2022-07-03T10:34:50.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8258009440 con 0x7f82680fd5b0 2022-07-03T10:34:50.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258022ac0 con 0x7f82680fd5b0 2022-07-03T10:34:50.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.890+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8268070a60 con 0x7f82680fd5b0 2022-07-03T10:34:50.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.891+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8268108d60 con 0x7f82680fd5b0 2022-07-03T10:34:50.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.893+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f8258029030 con 0x7f82680fd5b0 2022-07-03T10:34:50.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.894+0000 7f824f7fe700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 0x7f82500991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:50.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.895+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f82580c0e90 con 0x7f82680fd5b0 2022-07-03T10:34:50.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.895+0000 7f82667fc700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 0x7f82500991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:50.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.895+0000 7f82667fc700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 0x7f82500991d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f825c011640 tx=0x7f825c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:50.915 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:50 smithi188 systemd[1]: Starting Ceph mds.c for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:50.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:50.897+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f825808e520 con 0x7f82680fd5b0 2022-07-03T10:34:51.130 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.128+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8268108ef0 con 0x7f82680fd5b0 2022-07-03T10:34:51.131 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.130+0000 7f824f7fe700 1 -- 172.21.15.101:0/880533302 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6574 (secure 0 0 0) 0x7f8268108ef0 con 0x7f82680fd5b0 2022-07-03T10:34:51.132 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:51.132 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2022-07-03T10:34:51.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.132+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 msgr2=0x7f82500991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:51.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.132+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 0x7f82500991d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f825c011640 tx=0x7f825c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.132+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 msgr2=0x7f826810e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:51.136 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.132+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8258006010 tx=0x7f82580091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.136 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 shutdown_connections 2022-07-03T10:34:51.136 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8250096d20 0x7f82500991d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82680ff1c0 0x7f8268115b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82680fd5b0 0x7f826810e5a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 --2- 172.21.15.101:0/880533302 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f82680fe830 0x7f826810eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:51.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 >> 172.21.15.101:0/880533302 conn(0x7f82680f8d00 msgr2=0x7f8268101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:51.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 shutdown_connections 2022-07-03T10:34:51.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:51.133+0000 7f826d7ff700 1 -- 172.21.15.101:0/880533302 wait complete. 2022-07-03T10:34:51.138 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 9 2022-07-03T10:34:51.358 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:51 smithi188 podman[113527]: 2022-07-03T10:34:51.359 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:51 smithi188 bash[113527]: 6bb8865a5c8f77fe55c4b66cacb719ef0ff58eb51b71458eb69e8c68745f2f0a 2022-07-03T10:34:51.359 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:51 smithi188 ceph-mds[113567]: -- 172.21.15.188:0/2185299275 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e4dda8340 con 0x559e4dd7c800 2022-07-03T10:34:51.359 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:51 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-c[113543]: starting mds.c at 2022-07-03T10:34:51.664 INFO:journalctl@ceph.mds.c.smithi188.stdout:Jul 03 10:34:51 smithi188 systemd[1]: Started Ceph mds.c for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:51 smithi188 ceph-mon[92425]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:51 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/880533302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:51 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:51 smithi101 ceph-mon[85126]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:51 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/880533302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:51.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:51 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:51 smithi161 ceph-mon[91579]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:51 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/880533302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:51 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6832/1718258929,v1:172.21.15.188:6833/1718258929] up:boot 2022-07-03T10:34:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: fsmap 9 up:standby 2022-07-03T10:34:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-07-03T10:34:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:52 smithi188 ceph-mon[92425]: Deploying daemon mds.f on smithi188 2022-07-03T10:34:52.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:52.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:52.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.188:6832/1718258929,v1:172.21.15.188:6833/1718258929] up:boot 2022-07-03T10:34:52.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: fsmap 9 up:standby 2022-07-03T10:34:52.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-07-03T10:34:52.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:52.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:52.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:52 smithi101 ceph-mon[85126]: Deploying daemon mds.f on smithi188 2022-07-03T10:34:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6832/1718258929,v1:172.21.15.188:6833/1718258929] up:boot 2022-07-03T10:34:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: fsmap 9 up:standby 2022-07-03T10:34:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-07-03T10:34:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:52 smithi161 ceph-mon[91579]: Deploying daemon mds.f on smithi188 2022-07-03T10:34:53.152 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:52 smithi188 systemd[1]: Starting Ceph mds.f for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:53.163 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:53.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 -- 172.21.15.101:0/2694203429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 msgr2=0x7f53700fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:53.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 --2- 172.21.15.101:0/2694203429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f53700fd9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5360009900 tx=0x7f53600091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 -- 172.21.15.101:0/2694203429 shutdown_connections 2022-07-03T10:34:53.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 --2- 172.21.15.101:0/2694203429 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700ff1c0 0x7f5370105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 --2- 172.21.15.101:0/2694203429 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f53700fd9c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 --2- 172.21.15.101:0/2694203429 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700fe830 0x7f53700fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 -- 172.21.15.101:0/2694203429 >> 172.21.15.101:0/2694203429 conn(0x7f53700f8d00 msgr2=0x7f53700fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:53.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 -- 172.21.15.101:0/2694203429 shutdown_connections 2022-07-03T10:34:53.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.398+0000 7f5374bff700 1 -- 172.21.15.101:0/2694203429 wait complete. 2022-07-03T10:34:53.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 Processor -- start 2022-07-03T10:34:53.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 -- start start 2022-07-03T10:34:53.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:53.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700fe830 0x7f5370070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:53.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700ff1c0 0x7f5370070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:53.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5370071480 con 0x7f53700ff1c0 2022-07-03T10:34:53.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53700715f0 con 0x7f53700fe830 2022-07-03T10:34:53.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.399+0000 7f5374bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53701186b0 con 0x7f53700fd5b0 2022-07-03T10:34:53.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:53.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:50268/0 (socket says 172.21.15.101:50268) 2022-07-03T10:34:53.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 -- 172.21.15.101:0/1635721892 learned_addr learned my addr 172.21.15.101:0/1635721892 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:53.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536dd9b700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700fe830 0x7f5370070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:53.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536ed9d700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700ff1c0 0x7f5370070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:53.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 -- 172.21.15.101:0/1635721892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700fe830 msgr2=0x7f5370070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:53.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700fe830 0x7f5370070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 -- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700ff1c0 msgr2=0x7f5370070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:53.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700ff1c0 0x7f5370070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 -- 172.21.15.101:0/1635721892 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009410 con 0x7f53700fd5b0 2022-07-03T10:34:53.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.400+0000 7f536e59c700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5360005e10 tx=0x7f53600091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:53.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.401+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536001d070 con 0x7f53700fd5b0 2022-07-03T10:34:53.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.401+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f536000baa0 con 0x7f53700fd5b0 2022-07-03T10:34:53.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.401+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5360009440 con 0x7f53700fd5b0 2022-07-03T10:34:53.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.401+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360022ad0 con 0x7f53700fd5b0 2022-07-03T10:34:53.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.401+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5370118c50 con 0x7f53700fd5b0 2022-07-03T10:34:53.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.402+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5370108db0 con 0x7f53700fd5b0 2022-07-03T10:34:53.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.404+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f5360029030 con 0x7f53700fd5b0 2022-07-03T10:34:53.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.405+0000 7f535effd700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 0x7f5358099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:53.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.405+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f53600c0800 con 0x7f53700fd5b0 2022-07-03T10:34:53.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.406+0000 7f536dd9b700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 0x7f5358099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:53.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.406+0000 7f536dd9b700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 0x7f5358099250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5364009e90 tx=0x7f5364005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:53.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.408+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f536008df80 con 0x7f53700fd5b0 2022-07-03T10:34:53.414 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:53 smithi188 podman[113878]: 2022-07-03T10:34:53.414 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:53 smithi188 ceph-mds[113919]: -- 172.21.15.188:0/2189909887 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9c826a340 con 0x55b9c823e800 2022-07-03T10:34:53.415 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:53 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-f[113894]: starting mds.f at 2022-07-03T10:34:53.415 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:53 smithi188 bash[113878]: 5f0b76866bbdbf860ef96d58cfe967c8a2bca5d58091a8dc1f9cae6baf7710a2 2022-07-03T10:34:53.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.646+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5370108f40 con 0x7f53700fd5b0 2022-07-03T10:34:53.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.647+0000 7f535effd700 1 -- 172.21.15.101:0/1635721892 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7336 (secure 0 0 0) 0x7f5370108f40 con 0x7f53700fd5b0 2022-07-03T10:34:53.649 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:53.649 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2022-07-03T10:34:53.652 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 msgr2=0x7f5358099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:53.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 0x7f5358099250 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5364009e90 tx=0x7f5364005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 msgr2=0x7f5370072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:53.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5360005e10 tx=0x7f53600091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 shutdown_connections 2022-07-03T10:34:53.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5358096da0 0x7f5358099250 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53700ff1c0 0x7f5370070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f53700fd5b0 0x7f5370072230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 --2- 172.21.15.101:0/1635721892 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53700fe830 0x7f5370070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:53.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 >> 172.21.15.101:0/1635721892 conn(0x7f53700f8d00 msgr2=0x7f53700fb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:53.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 shutdown_connections 2022-07-03T10:34:53.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:53.650+0000 7f5374bff700 1 -- 172.21.15.101:0/1635721892 wait complete. 2022-07-03T10:34:53.658 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 10 2022-07-03T10:34:53.675 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:53 smithi101 ceph-mon[85126]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:53 smithi161 ceph-mon[91579]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:53 smithi188 ceph-mon[92425]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:53.914 INFO:journalctl@ceph.mds.f.smithi188.stdout:Jul 03 10:34:53 smithi188 systemd[1]: Started Ceph mds.f for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:54.749 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:54.750 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:54.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:54.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: Deploying daemon mds.i on smithi188 2022-07-03T10:34:54.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1635721892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:54.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6834/1026299276,v1:172.21.15.188:6835/1026299276] up:boot 2022-07-03T10:34:54.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: fsmap 10 up:standby 2022-07-03T10:34:54.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:54 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-07-03T10:34:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: Deploying daemon mds.i on smithi188 2022-07-03T10:34:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1635721892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6834/1026299276,v1:172.21.15.188:6835/1026299276] up:boot 2022-07-03T10:34:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: fsmap 10 up:standby 2022-07-03T10:34:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:54 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-07-03T10:34:54.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:54.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: Deploying daemon mds.i on smithi188 2022-07-03T10:34:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1635721892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:54.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.188:6834/1026299276,v1:172.21.15.188:6835/1026299276] up:boot 2022-07-03T10:34:54.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: fsmap 10 up:standby 2022-07-03T10:34:54.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:54 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-07-03T10:34:55.164 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:54 smithi188 systemd[1]: Starting Ceph mds.i for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:55.520 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:55 smithi188 podman[114227]: 2022-07-03T10:34:55.680 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:55 smithi161 ceph-mon[91579]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:55.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1209643744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 msgr2=0x7fed880fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:55.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1209643744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed880fec80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009900 tx=0x7fed7c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1209643744 shutdown_connections 2022-07-03T10:34:55.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1209643744 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880fd5b0 0x7fed880fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1209643744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880ff1c0 0x7fed88105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1209643744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed880fec80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1209643744 >> 172.21.15.101:0/1209643744 conn(0x7fed880f8d00 msgr2=0x7fed880fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:55.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1209643744 shutdown_connections 2022-07-03T10:34:55.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.911+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1209643744 wait complete. 2022-07-03T10:34:55.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 Processor -- start 2022-07-03T10:34:55.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 -- start start 2022-07-03T10:34:55.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 0x7fed88110720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:55.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:55.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880ff1c0 0x7fed88117cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:55.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88118250 con 0x7fed880ff1c0 2022-07-03T10:34:55.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88118390 con 0x7fed880fd5b0 2022-07-03T10:34:55.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.912+0000 7fed8d9ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88118550 con 0x7fed880fe830 2022-07-03T10:34:55.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:55.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:50286/0 (socket says 172.21.15.101:50286) 2022-07-03T10:34:55.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 -- 172.21.15.101:0/1456549628 learned_addr learned my addr 172.21.15.101:0/1456549628 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:55.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed877fe700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880ff1c0 0x7fed88117cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:55.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed86ffd700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 0x7fed88110720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:55.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 -- 172.21.15.101:0/1456549628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 msgr2=0x7fed88110720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:55.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 0x7fed88110720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 -- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880ff1c0 msgr2=0x7fed88117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880ff1c0 0x7fed88117cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 -- 172.21.15.101:0/1456549628 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c009440 con 0x7fed880fe830 2022-07-03T10:34:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed86ffd700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 0x7fed88110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:34:55.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.913+0000 7fed867fc700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0059f0 tx=0x7fed7c0091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:55.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.914+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c01d070 con 0x7fed880fe830 2022-07-03T10:34:55.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.914+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fed7c00baa0 con 0x7fed880fe830 2022-07-03T10:34:55.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.914+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed881187d0 con 0x7fed880fe830 2022-07-03T10:34:55.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.914+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c022ca0 con 0x7fed880fe830 2022-07-03T10:34:55.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.914+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed88118cd0 con 0x7fed880fe830 2022-07-03T10:34:55.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.915+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88108da0 con 0x7fed880fe830 2022-07-03T10:34:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:55 smithi188 ceph-mon[92425]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:55.927 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:55 smithi188 ceph-mds[114267]: -- 172.21.15.188:0/1211097528 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561071ef8340 con 0x561071ecc800 2022-07-03T10:34:55.927 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:55 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-i[114243]: starting mds.i at 2022-07-03T10:34:55.928 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:55 smithi188 bash[114227]: 92ed3cdb4f484dce97013803bc46fcabfc111275d560f4abca956d27d9fb7d23 2022-07-03T10:34:55.928 INFO:journalctl@ceph.mds.i.smithi188.stdout:Jul 03 10:34:55 smithi188 systemd[1]: Started Ceph mds.i for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:55.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.917+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fed7c029030 con 0x7fed880fe830 2022-07-03T10:34:55.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.918+0000 7fed6f7fe700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 0x7fed70099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:55.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.918+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fed7c0c0ba0 con 0x7fed880fe830 2022-07-03T10:34:55.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.918+0000 7fed86ffd700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 0x7fed70099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:55.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.919+0000 7fed86ffd700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 0x7fed70099250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fed880fe690 tx=0x7fed78005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:55.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:55.922+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fed7c08e270 con 0x7fed880fe830 2022-07-03T10:34:55.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:55 smithi101 ceph-mon[85126]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:56.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.158+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8810aca0 con 0x7fed880fe830 2022-07-03T10:34:56.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.159+0000 7fed6f7fe700 1 -- 172.21.15.101:0/1456549628 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8097 (secure 0 0 0) 0x7fed7c091890 con 0x7fed880fe830 2022-07-03T10:34:56.162 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:56.163 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2022-07-03T10:34:56.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 msgr2=0x7fed70099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:56.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 0x7fed70099250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fed880fe690 tx=0x7fed78005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 msgr2=0x7fed88110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:56.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0059f0 tx=0x7fed7c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 shutdown_connections 2022-07-03T10:34:56.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fed70096da0 0x7fed70099250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fed880ff1c0 0x7fed88117cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed880fd5b0 0x7fed88110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 --2- 172.21.15.101:0/1456549628 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed880fe830 0x7fed88110c60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:56.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.161+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 >> 172.21.15.101:0/1456549628 conn(0x7fed880f8d00 msgr2=0x7fed88104250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:56.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.162+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 shutdown_connections 2022-07-03T10:34:56.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:56.162+0000 7fed8d9ff700 1 -- 172.21.15.101:0/1456549628 wait complete. 2022-07-03T10:34:56.168 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 11 2022-07-03T10:34:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: Deploying daemon mds.l on smithi188 2022-07-03T10:34:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1456549628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6836/639939509,v1:172.21.15.188:6837/639939509] up:boot 2022-07-03T10:34:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: fsmap 11 up:standby 2022-07-03T10:34:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:56 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-07-03T10:34:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: Deploying daemon mds.l on smithi188 2022-07-03T10:34:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1456549628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6836/639939509,v1:172.21.15.188:6837/639939509] up:boot 2022-07-03T10:34:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: fsmap 11 up:standby 2022-07-03T10:34:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:56 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-07-03T10:34:56.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:56.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-07-03T10:34:56.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-07-03T10:34:56.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:56.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: Deploying daemon mds.l on smithi188 2022-07-03T10:34:56.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1456549628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:56.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.188:6836/639939509,v1:172.21.15.188:6837/639939509] up:boot 2022-07-03T10:34:56.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: fsmap 11 up:standby 2022-07-03T10:34:56.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:56 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-07-03T10:34:57.414 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 systemd[1]: Starting Ceph mds.l for 23a851ae-fab9-11ec-842c-001a4aab830c... 2022-07-03T10:34:57.769 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 podman[114580]: 2022-07-03T10:34:57.770 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:57 smithi188 ceph-mon[92425]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:57 smithi161 ceph-mon[91579]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:57.928 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:57 smithi101 ceph-mon[85126]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:58.164 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 ceph-mds[114621]: -- 172.21.15.188:0/3766303149 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5581af20a340 con 0x5581af1df000 2022-07-03T10:34:58.164 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-l[114596]: starting mds.l at 2022-07-03T10:34:58.164 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 bash[114580]: 43e917df20a6f685a9d5409a748fb0c88f8603f3555b552d1291d3f802a0a18e 2022-07-03T10:34:58.165 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:34:57 smithi188 systemd[1]: Started Ceph mds.l for 23a851ae-fab9-11ec-842c-001a4aab830c. 2022-07-03T10:34:58.191 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:34:58.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 -- 172.21.15.101:0/1004768904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4072430 msgr2=0x7fa8b4070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/1004768904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4072430 0x7fa8b4070910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009900 tx=0x7fa8b00091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 -- 172.21.15.101:0/1004768904 shutdown_connections 2022-07-03T10:34:58.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/1004768904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 0x7fa8b410bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/1004768904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4072430 0x7fa8b4070910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/1004768904 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4070e50 0x7fa8b40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 -- 172.21.15.101:0/1004768904 >> 172.21.15.101:0/1004768904 conn(0x7fa8b40f8b70 msgr2=0x7fa8b40fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:58.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 -- 172.21.15.101:0/1004768904 shutdown_connections 2022-07-03T10:34:58.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.423+0000 7fa8bb39d700 1 -- 172.21.15.101:0/1004768904 wait complete. 2022-07-03T10:34:58.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.424+0000 7fa8bb39d700 1 Processor -- start 2022-07-03T10:34:58.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.424+0000 7fa8bb39d700 1 -- start start 2022-07-03T10:34:58.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 0x7fa8b4114f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 0x7fa8b411bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411c530 con 0x7fa8b4072430 2022-07-03T10:34:58.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411c670 con 0x7fa8b4100f00 2022-07-03T10:34:58.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bb39d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b411c830 con 0x7fa8b4070e50 2022-07-03T10:34:58.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8ba39b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8ba39b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59514/0 (socket says 172.21.15.101:59514) 2022-07-03T10:34:58.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8ba39b700 1 -- 172.21.15.101:0/3517626564 learned_addr learned my addr 172.21.15.101:0/3517626564 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:58.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8bab9c700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 0x7fa8b411bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.425+0000 7fa8b9b9a700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 0x7fa8b4114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 -- 172.21.15.101:0/3517626564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 msgr2=0x7fa8b411bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 0x7fa8b411bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 -- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 msgr2=0x7fa8b4114f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 0x7fa8b4114f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 -- 172.21.15.101:0/3517626564 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4009410 con 0x7fa8b4070e50 2022-07-03T10:34:58.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8b9b9a700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 0x7fa8b4114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:34:58.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8ba39b700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0005d00 tx=0x7fa8b00091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:58.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b001d070 con 0x7fa8b4070e50 2022-07-03T10:34:58.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa8b000b9c0 con 0x7fa8b4070e50 2022-07-03T10:34:58.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.426+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0022cd0 con 0x7fa8b4070e50 2022-07-03T10:34:58.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.427+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b0009440 con 0x7fa8b4070e50 2022-07-03T10:34:58.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.427+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b411cd70 con 0x7fa8b4070e50 2022-07-03T10:34:58.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.429+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c0051a0 con 0x7fa8b4070e50 2022-07-03T10:34:58.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.431+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa8b0029030 con 0x7fa8b4070e50 2022-07-03T10:34:58.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.432+0000 7fa8aaffd700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 0x7fa8a0099320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.432+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fa8b00c07c0 con 0x7fa8b4070e50 2022-07-03T10:34:58.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.432+0000 7fa8b9b9a700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 0x7fa8a0099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.433+0000 7fa8b9b9a700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 0x7fa8a0099320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009dd0 tx=0x7fa8a4005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:58.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.436+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa8b008dee0 con 0x7fa8b4070e50 2022-07-03T10:34:58.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.677+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa89c005df0 con 0x7fa8b4070e50 2022-07-03T10:34:58.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.678+0000 7fa8aaffd700 1 -- 172.21.15.101:0/3517626564 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9613 (secure 0 0 0) 0x7fa8b0091500 con 0x7fa8b4070e50 2022-07-03T10:34:58.681 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:58.681 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24557,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24569,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2022-07-03T10:34:58.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 msgr2=0x7fa8a0099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 0x7fa8a0099320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009dd0 tx=0x7fa8a4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 msgr2=0x7fa8b4114a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0005d00 tx=0x7fa8b00091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 shutdown_connections 2022-07-03T10:34:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa8a0096e70 0x7fa8a0099320 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8b4100f00 0x7fa8b411bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8b4070e50 0x7fa8b4114a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 --2- 172.21.15.101:0/3517626564 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8b4072430 0x7fa8b4114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 >> 172.21.15.101:0/3517626564 conn(0x7fa8b40f8b70 msgr2=0x7fa8b40faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:58.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 shutdown_connections 2022-07-03T10:34:58.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.681+0000 7fa8bb39d700 1 -- 172.21.15.101:0/3517626564 wait complete. 2022-07-03T10:34:58.687 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 13 2022-07-03T10:34:58.716 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:34:58.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/88476680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8070e50 msgr2=0x7f5bf80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/88476680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8070e50 0x7f5bf80712c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009900 tx=0x7f5bec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/88476680 shutdown_connections 2022-07-03T10:34:58.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/88476680 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8072430 0x7f5bf8070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/88476680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf810bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/88476680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8070e50 0x7f5bf80712c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/88476680 >> 172.21.15.101:0/88476680 conn(0x7f5bf80f8b20 msgr2=0x7f5bf80faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:58.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/88476680 shutdown_connections 2022-07-03T10:34:58.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.957+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/88476680 wait complete. 2022-07-03T10:34:58.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.958+0000 7f5bfcf9d700 1 Processor -- start 2022-07-03T10:34:58.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.958+0000 7f5bfcf9d700 1 -- start start 2022-07-03T10:34:58.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.958+0000 7f5bfcf9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8070e50 0x7f5bf81149a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bfcf9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bfcf9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bfcf9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811c4d0 con 0x7f5bf8070e50 2022-07-03T10:34:58.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bfcf9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811c610 con 0x7f5bf8072430 2022-07-03T10:34:58.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bfcf9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf811c7d0 con 0x7f5bf8100ed0 2022-07-03T10:34:58.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf6ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8070e50 0x7f5bf81149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59528/0 (socket says 172.21.15.101:59528) 2022-07-03T10:34:58.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf6ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34162/0 (socket says 172.21.15.101:34162) 2022-07-03T10:34:58.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 -- 172.21.15.101:0/3264433363 learned_addr learned my addr 172.21.15.101:0/3264433363 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:58.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 -- 172.21.15.101:0/3264433363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 msgr2=0x7f5bf8114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 -- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8070e50 msgr2=0x7f5bf81149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:58.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8070e50 0x7f5bf81149a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:58.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf7fff700 1 -- 172.21.15.101:0/3264433363 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec009440 con 0x7f5bf8100ed0 2022-07-03T10:34:58.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.959+0000 7f5bf6ffd700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:58.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bf7fff700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000eb50 tx=0x7f5bf0009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:58.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0004a50 con 0x7f5bf8100ed0 2022-07-03T10:34:58.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5bf000ce80 con 0x7f5bf8100ed0 2022-07-03T10:34:58.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf811cab0 con 0x7f5bf8100ed0 2022-07-03T10:34:58.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0005730 con 0x7f5bf8100ed0 2022-07-03T10:34:58.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.960+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf811cff0 con 0x7f5bf8100ed0 2022-07-03T10:34:58.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.962+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf810f1f0 con 0x7f5bf8100ed0 2022-07-03T10:34:58.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.963+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f5bf000c040 con 0x7f5bf8100ed0 2022-07-03T10:34:58.983 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.964+0000 7f5bdffff700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 0x7f5be00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:58.984 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.964+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5bf0014070 con 0x7f5bf8100ed0 2022-07-03T10:34:58.984 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.965+0000 7f5bf77fe700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 0x7f5be00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:58.984 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.965+0000 7f5bf77fe700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 0x7f5be00991d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5be800b840 tx=0x7f5be800b3a0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:58.985 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:58.969+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f5bf008d410 con 0x7f5bf8100ed0 2022-07-03T10:34:58.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:58.992 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:58.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:58.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:58.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:58.993 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:58.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:boot 2022-07-03T10:34:58.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: fsmap 12 up:standby 2022-07-03T10:34:58.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-07-03T10:34:58.994 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:58 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3517626564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:boot 2022-07-03T10:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: fsmap 12 up:standby 2022-07-03T10:34:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-07-03T10:34:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:58 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3517626564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:34:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:34:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:34:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:34:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:boot 2022-07-03T10:34:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: fsmap 12 up:standby 2022-07-03T10:34:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-07-03T10:34:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:58 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3517626564' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.172+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bf805f160 con 0x7f5bf8100ed0 2022-07-03T10:34:59.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.173+0000 7f5bdffff700 1 -- 172.21.15.101:0/3264433363 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18195 (secure 0 0 0) 0x7f5bf0090a30 con 0x7f5bf8100ed0 2022-07-03T10:34:59.176 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:34:59.177 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":76,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:34:15.242218+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:34:59.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 msgr2=0x7f5be00991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 0x7f5be00991d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5be800b840 tx=0x7f5be800b3a0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 msgr2=0x7f5bf811bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bf000eb50 tx=0x7f5bf0009460 comp rx=0 tx=0).stop 2022-07-03T10:34:59.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 shutdown_connections 2022-07-03T10:34:59.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5be0096d20 0x7f5be00991d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5bf8070e50 0x7f5bf81149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5bf8100ed0 0x7f5bf811bf50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 --2- 172.21.15.101:0/3264433363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5bf8072430 0x7f5bf8114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 >> 172.21.15.101:0/3264433363 conn(0x7f5bf80f8b20 msgr2=0x7f5bf80faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:59.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 shutdown_connections 2022-07-03T10:34:59.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.176+0000 7f5bfcf9d700 1 -- 172.21.15.101:0/3264433363 wait complete. 2022-07-03T10:34:59.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-07-03T10:34:59.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:34:59.451 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 -- 172.21.15.101:0/3855938678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fe670 msgr2=0x7f32680feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 --2- 172.21.15.101:0/3855938678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fe670 0x7f32680feac0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f325c009900 tx=0x7f325c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 -- 172.21.15.101:0/3855938678 shutdown_connections 2022-07-03T10:34:59.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 --2- 172.21.15.101:0/3855938678 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 0x7f32681058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 --2- 172.21.15.101:0/3855938678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fd3f0 0x7f32680fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 --2- 172.21.15.101:0/3855938678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fe670 0x7f32680feac0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.443+0000 7f3267fff700 1 -- 172.21.15.101:0/3855938678 >> 172.21.15.101:0/3855938678 conn(0x7f32680f8b40 msgr2=0x7f32680fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:59.456 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.444+0000 7f3267fff700 1 -- 172.21.15.101:0/3855938678 shutdown_connections 2022-07-03T10:34:59.456 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.444+0000 7f3267fff700 1 -- 172.21.15.101:0/3855938678 wait complete. 2022-07-03T10:34:59.457 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.444+0000 7f3267fff700 1 Processor -- start 2022-07-03T10:34:59.457 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 -- start start 2022-07-03T10:34:59.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 0x7f3268117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268118190 con 0x7f32680ff000 2022-07-03T10:34:59.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32681182d0 con 0x7f32680fe670 2022-07-03T10:34:59.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3267fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268118490 con 0x7f32680fd3f0 2022-07-03T10:34:59.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3266ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f32667fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3266ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59532/0 (socket says 172.21.15.101:59532) 2022-07-03T10:34:59.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f3266ffd700 1 -- 172.21.15.101:0/251864109 learned_addr learned my addr 172.21.15.101:0/251864109 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:59.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f32667fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34186/0 (socket says 172.21.15.101:34186) 2022-07-03T10:34:59.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.445+0000 7f32677fe700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 0x7f3268117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 -- 172.21.15.101:0/251864109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 msgr2=0x7f3268110ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 -- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 msgr2=0x7f3268117c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 0x7f3268117c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 -- 172.21.15.101:0/251864109 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260009700 con 0x7f32680fd3f0 2022-07-03T10:34:59.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f32667fc700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:59.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3266ffd700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f325800e840 tx=0x7f3258005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:59.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258004030 con 0x7f32680fd3f0 2022-07-03T10:34:59.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c009440 con 0x7f32680fd3f0 2022-07-03T10:34:59.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3258004190 con 0x7f32680fd3f0 2022-07-03T10:34:59.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.446+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3258003a20 con 0x7f32680fd3f0 2022-07-03T10:34:59.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.447+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3268118b70 con 0x7f32680fd3f0 2022-07-03T10:34:59.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.448+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3268108bb0 con 0x7f32680fd3f0 2022-07-03T10:34:59.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.449+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f325800c040 con 0x7f32680fd3f0 2022-07-03T10:34:59.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.451+0000 7f324f7fe700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 0x7f32500991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.451+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f3258014070 con 0x7f32680fd3f0 2022-07-03T10:34:59.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.451+0000 7f32667fc700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 0x7f32500991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.452+0000 7f32667fc700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 0x7f32500991d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f325c009f70 tx=0x7f325c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:59.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.453+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f325808d2e0 con 0x7f32680fd3f0 2022-07-03T10:34:59.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.657+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f326810aab0 con 0x7f32680fd3f0 2022-07-03T10:34:59.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.657+0000 7f324f7fe700 1 -- 172.21.15.101:0/251864109 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3258090900 con 0x7f32680fd3f0 2022-07-03T10:34:59.663 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:34:59.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 msgr2=0x7f32500991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 0x7f32500991d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f325c009f70 tx=0x7f325c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 msgr2=0x7f3268110660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f325800e840 tx=0x7f3258005ef0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 shutdown_connections 2022-07-03T10:34:59.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3250096d20 0x7f32500991d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f32680ff000 0x7f3268117c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f32680fd3f0 0x7f3268110660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 --2- 172.21.15.101:0/251864109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32680fe670 0x7f3268110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 >> 172.21.15.101:0/251864109 conn(0x7f32680f8b40 msgr2=0x7f3268104120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:59.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 shutdown_connections 2022-07-03T10:34:59.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.660+0000 7f3267fff700 1 -- 172.21.15.101:0/251864109 wait complete. 2022-07-03T10:34:59.690 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-03T10:34:59.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 -- 172.21.15.101:0/3589166052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 msgr2=0x7f86440fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 --2- 172.21.15.101:0/3589166052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f86440fd9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8634009900 tx=0x7f86340091e0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 -- 172.21.15.101:0/3589166052 shutdown_connections 2022-07-03T10:34:59.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 --2- 172.21.15.101:0/3589166052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440ff1e0 0x7f8644105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 --2- 172.21.15.101:0/3589166052 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f86440fd9e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 --2- 172.21.15.101:0/3589166052 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440fe850 0x7f86440feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 -- 172.21.15.101:0/3589166052 >> 172.21.15.101:0/3589166052 conn(0x7f86440f8d20 msgr2=0x7f86440fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:34:59.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 -- 172.21.15.101:0/3589166052 shutdown_connections 2022-07-03T10:34:59.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.939+0000 7f86491ff700 1 -- 172.21.15.101:0/3589166052 wait complete. 2022-07-03T10:34:59.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.940+0000 7f86491ff700 1 Processor -- start 2022-07-03T10:34:59.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.940+0000 7f86491ff700 1 -- start start 2022-07-03T10:34:59.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440ff1e0 0x7f8644117db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644118330 con 0x7f86440ff1e0 2022-07-03T10:34:59.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644118470 con 0x7f86440fe850 2022-07-03T10:34:59.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f86491ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644118630 con 0x7f86440fd5d0 2022-07-03T10:34:59.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f8642d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f864259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f8642d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59548/0 (socket says 172.21.15.101:59548) 2022-07-03T10:34:59.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f864259c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34204/0 (socket says 172.21.15.101:34204) 2022-07-03T10:34:59.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f8642d9d700 1 -- 172.21.15.101:0/4007460939 learned_addr learned my addr 172.21.15.101:0/4007460939 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:34:59.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f864359e700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440ff1e0 0x7f8644117db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.941+0000 7f864259c700 1 -- 172.21.15.101:0/4007460939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 msgr2=0x7f8644110800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f864259c700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f864259c700 1 -- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440ff1e0 msgr2=0x7f8644117db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:34:59.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f864259c700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440ff1e0 0x7f8644117db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:34:59.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f864259c700 1 -- 172.21.15.101:0/4007460939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634009440 con 0x7f86440fe850 2022-07-03T10:34:59.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f8642d9d700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:34:59.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f864259c700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f863800e7d0 tx=0x7f8638005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:59.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638010710 con 0x7f86440fe850 2022-07-03T10:34:59.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8638010d20 con 0x7f86440fe850 2022-07-03T10:34:59.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8644118910 con 0x7f86440fe850 2022-07-03T10:34:59.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638018df0 con 0x7f86440fe850 2022-07-03T10:34:59.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.942+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8644118ef0 con 0x7f86440fe850 2022-07-03T10:34:59.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.943+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644108dd0 con 0x7f86440fe850 2022-07-03T10:34:59.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.945+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f863800c040 con 0x7f86440fe850 2022-07-03T10:34:59.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.946+0000 7f862b7fe700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 0x7f862c099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:34:59.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.946+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f8638014070 con 0x7f86440fe850 2022-07-03T10:34:59.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.947+0000 7f8642d9d700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 0x7f862c099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:34:59.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.947+0000 7f8642d9d700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 0x7f862c099250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86340056f0 tx=0x7f86340091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:34:59.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:34:59.948+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f863808cc40 con 0x7f86440fe850 2022-07-03T10:34:59.995 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:59 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3264433363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:34:59.996 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:34:59 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/251864109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:35:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:59 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3264433363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:34:59 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/251864109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:35:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:59 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3264433363' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:34:59 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/251864109' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:35:00.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.173+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f86441194a0 con 0x7f86440fe850 2022-07-03T10:35:00.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.173+0000 7f862b7fe700 1 -- 172.21.15.101:0/4007460939 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8638090260 con 0x7f86440fe850 2022-07-03T10:35:00.177 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:00.177 INFO:teuthology.orchestra.run.smithi101.stdout:[ 2022-07-03T10:35:00.177 INFO:teuthology.orchestra.run.smithi101.stdout: { 2022-07-03T10:35:00.178 INFO:teuthology.orchestra.run.smithi101.stdout: "poolnum": 1, 2022-07-03T10:35:00.178 INFO:teuthology.orchestra.run.smithi101.stdout: "poolname": ".mgr" 2022-07-03T10:35:00.178 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:35:00.179 INFO:teuthology.orchestra.run.smithi101.stdout:] 2022-07-03T10:35:00.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.176+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 msgr2=0x7f862c099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:00.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.176+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 0x7f862c099250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86340056f0 tx=0x7f86340091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.176+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 msgr2=0x7f8644110d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:00.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.176+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f863800e7d0 tx=0x7f8638005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 shutdown_connections 2022-07-03T10:35:00.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f862c096da0 0x7f862c099250 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86440ff1e0 0x7f8644117db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86440fd5d0 0x7f8644110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 --2- 172.21.15.101:0/4007460939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86440fe850 0x7f8644110d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 >> 172.21.15.101:0/4007460939 conn(0x7f86440f8d20 msgr2=0x7f86440fb130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:00.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 shutdown_connections 2022-07-03T10:35:00.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.177+0000 7f86491ff700 1 -- 172.21.15.101:0/4007460939 wait complete. 2022-07-03T10:35:00.209 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-03T10:35:00.210 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-07-03T10:35:00.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 -- 172.21.15.101:0/3648616475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0ff1f0 msgr2=0x7f745c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:00.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 --2- 172.21.15.101:0/3648616475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0ff1f0 0x7f745c105ab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7454009ca0 tx=0x7f7454009520 comp rx=0 tx=0).stop 2022-07-03T10:35:00.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 -- 172.21.15.101:0/3648616475 shutdown_connections 2022-07-03T10:35:00.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 --2- 172.21.15.101:0/3648616475 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0fd5e0 0x7f745c0fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 --2- 172.21.15.101:0/3648616475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0ff1f0 0x7f745c105ab0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 --2- 172.21.15.101:0/3648616475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 0x7f745c0fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 -- 172.21.15.101:0/3648616475 >> 172.21.15.101:0/3648616475 conn(0x7f745c0f8d50 msgr2=0x7f745c0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:00.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 -- 172.21.15.101:0/3648616475 shutdown_connections 2022-07-03T10:35:00.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.459+0000 7f7463dff700 1 -- 172.21.15.101:0/3648616475 wait complete. 2022-07-03T10:35:00.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.460+0000 7f7463dff700 1 Processor -- start 2022-07-03T10:35:00.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.460+0000 7f7463dff700 1 -- start start 2022-07-03T10:35:00.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:00.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 0x7f745c10e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:00.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0ff1f0 0x7f745c1158f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:00.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c115e70 con 0x7f745c0ff1f0 2022-07-03T10:35:00.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c115fb0 con 0x7f745c0fd5e0 2022-07-03T10:35:00.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7463dff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c116170 con 0x7f745c0fe860 2022-07-03T10:35:00.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7461b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:00.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7461b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34216/0 (socket says 172.21.15.101:34216) 2022-07-03T10:35:00.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f7461b9b700 1 -- 172.21.15.101:0/147738815 learned_addr learned my addr 172.21.15.101:0/147738815 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:00.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.461+0000 7f746139a700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 0x7f745c10e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:00.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f746239c700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0ff1f0 0x7f745c1158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:00.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 -- 172.21.15.101:0/147738815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 msgr2=0x7f745c10e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:00.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 0x7f745c10e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 -- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0ff1f0 msgr2=0x7f745c1158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:00.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0ff1f0 0x7f745c1158f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:00.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 -- 172.21.15.101:0/147738815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7458009410 con 0x7f745c0fd5e0 2022-07-03T10:35:00.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7461b9b700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f745800ea00 tx=0x7f7458005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:00.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458009690 con 0x7f745c0fd5e0 2022-07-03T10:35:00.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f745800ce80 con 0x7f745c0fd5e0 2022-07-03T10:35:00.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74540097e0 con 0x7f745c0fd5e0 2022-07-03T10:35:00.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.462+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458018420 con 0x7f745c0fd5e0 2022-07-03T10:35:00.475 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.463+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c1168a0 con 0x7f745c0fd5e0 2022-07-03T10:35:00.475 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.464+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c108df0 con 0x7f745c0fd5e0 2022-07-03T10:35:00.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.465+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f745800c040 con 0x7f745c0fd5e0 2022-07-03T10:35:00.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.466+0000 7f74527fc700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 0x7f7448099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:00.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.467+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f7458014070 con 0x7f745c0fd5e0 2022-07-03T10:35:00.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.467+0000 7f746139a700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 0x7f7448099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:00.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.467+0000 7f746139a700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 0x7f7448099250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f744c009d00 tx=0x7f744c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:00.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.470+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f745808cf80 con 0x7f745c0fd5e0 2022-07-03T10:35:00.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:00.680+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f745c108f80 con 0x7f745c0fd5e0 2022-07-03T10:35:01.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.620+0000 7f74527fc700 1 -- 172.21.15.101:0/147738815 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f745c108f80 con 0x7f745c0fd5e0 2022-07-03T10:35:01.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.626+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 msgr2=0x7f7448099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:01.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.626+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 0x7f7448099250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f744c009d00 tx=0x7f744c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.626+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 msgr2=0x7f745c10e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:01.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.626+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f745800ea00 tx=0x7f7458005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 shutdown_connections 2022-07-03T10:35:01.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7448096da0 0x7f7448099250 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f745c0ff1f0 0x7f745c1158f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745c0fd5e0 0x7f745c10e340 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 --2- 172.21.15.101:0/147738815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f745c0fe860 0x7f745c10e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 >> 172.21.15.101:0/147738815 conn(0x7f745c0f8d50 msgr2=0x7f745c0faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:01.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 shutdown_connections 2022-07-03T10:35:01.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.627+0000 7f7463dff700 1 -- 172.21.15.101:0/147738815 wait complete. 2022-07-03T10:35:01.639 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'cephfs_metadata' created 2022-07-03T10:35:01.663 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-07-03T10:35:01.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:01 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4007460939' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-07-03T10:35:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:01 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:01 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4007460939' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-07-03T10:35:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:01 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.904 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1441720994 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c066420 msgr2=0x7fcf2c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:01.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1441720994 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c066420 0x7fcf2c0fffb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009900 tx=0x7fcf1c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1441720994 shutdown_connections 2022-07-03T10:35:01.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1441720994 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c066420 0x7fcf2c0fffb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.905 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1441720994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c103070 0x7fcf2c1054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1441720994 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c100580 0x7fcf2c1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1441720994 >> 172.21.15.101:0/1441720994 conn(0x7fcf2c0f7a40 msgr2=0x7fcf2c0f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:01.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1441720994 shutdown_connections 2022-07-03T10:35:01.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.902+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1441720994 wait complete. 2022-07-03T10:35:01.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.903+0000 7fcf2bfff700 1 Processor -- start 2022-07-03T10:35:01.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 -- start start 2022-07-03T10:35:01.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 0x7fcf2c10e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:01.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:01.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c103070 0x7fcf2c1159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:01.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c115f50 con 0x7fcf2c103070 2022-07-03T10:35:01.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c116090 con 0x7fcf2c066420 2022-07-03T10:35:01.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.904+0000 7fcf2bfff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c116250 con 0x7fcf2c100580 2022-07-03T10:35:01.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:01.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2affd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 0x7fcf2c10e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:01.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59568/0 (socket says 172.21.15.101:59568) 2022-07-03T10:35:01.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 -- 172.21.15.101:0/1402482129 learned_addr learned my addr 172.21.15.101:0/1402482129 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:01.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2b7fe700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c103070 0x7fcf2c1159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:01.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 -- 172.21.15.101:0/1402482129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 msgr2=0x7fcf2c10e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:01.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 0x7fcf2c10e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 -- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c103070 msgr2=0x7fcf2c1159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:01.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c103070 0x7fcf2c1159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:01.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 -- 172.21.15.101:0/1402482129 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf20009410 con 0x7fcf2c100580 2022-07-03T10:35:01.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2affd700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 0x7fcf2c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:01.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.905+0000 7fcf2a7fc700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000ea50 tx=0x7fcf20005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:01.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.906+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20009670 con 0x7fcf2c100580 2022-07-03T10:35:01.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.906+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c009440 con 0x7fcf2c100580 2022-07-03T10:35:01.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.906+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcf2000c970 con 0x7fcf2c100580 2022-07-03T10:35:01.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.906+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20003af0 con 0x7fcf2c100580 2022-07-03T10:35:01.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.906+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c116980 con 0x7fcf2c100580 2022-07-03T10:35:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:01 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4007460939' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-07-03T10:35:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:01 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:01.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.907+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c108bd0 con 0x7fcf2c100580 2022-07-03T10:35:01.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.909+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fcf2000c040 con 0x7fcf2c100580 2022-07-03T10:35:01.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.910+0000 7fcf137fe700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 0x7fcf140991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:01.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.911+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fcf20014070 con 0x7fcf2c100580 2022-07-03T10:35:01.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.911+0000 7fcf2affd700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 0x7fcf140991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:01.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.912+0000 7fcf2affd700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 0x7fcf140991d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097d0 tx=0x7fcf1c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:01.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:01.915+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fcf2008cef0 con 0x7fcf2c100580 2022-07-03T10:35:02.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.144+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcf2c108d60 con 0x7fcf2c100580 2022-07-03T10:35:02.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.852+0000 7fcf137fe700 1 -- 172.21.15.101:0/1402482129 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcf2c108d60 con 0x7fcf2c100580 2022-07-03T10:35:02.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.868+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 msgr2=0x7fcf140991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:02.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.868+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 0x7fcf140991d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0097d0 tx=0x7fcf1c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.868+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 msgr2=0x7fcf2c10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:02.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.868+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000ea50 tx=0x7fcf20005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 shutdown_connections 2022-07-03T10:35:02.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcf14096d20 0x7fcf140991d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf2c103070 0x7fcf2c1159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcf2c066420 0x7fcf2c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.878 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 --2- 172.21.15.101:0/1402482129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf2c100580 0x7fcf2c10e960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:02.878 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.874+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 >> 172.21.15.101:0/1402482129 conn(0x7fcf2c0f7a40 msgr2=0x7fcf2c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:02.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.877+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 shutdown_connections 2022-07-03T10:35:02.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:02.880+0000 7fcf2bfff700 1 -- 172.21.15.101:0/1402482129 wait complete. 2022-07-03T10:35:02.886 INFO:teuthology.orchestra.run.smithi101.stderr:pool 'cephfs_data' created 2022-07-03T10:35:02.913 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-03T10:35:03.093 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:03.094 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/147738815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.094 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.094 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.094 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.095 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.095 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-07-03T10:35:03.095 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: osdmap e77: 12 total, 12 up, 12 in 2022-07-03T10:35:03.096 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.096 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.096 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:02 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/147738815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-07-03T10:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: osdmap e77: 12 total, 12 up, 12 in 2022-07-03T10:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:02 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 -- 172.21.15.101:0/4206106129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40fe880 msgr2=0x7fd6c40fecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/4206106129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40fe880 0x7fd6c40fecd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009900 tx=0x7fd6b80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 -- 172.21.15.101:0/4206106129 shutdown_connections 2022-07-03T10:35:03.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/4206106129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40ff210 0x7fd6c4105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/4206106129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c40fda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/4206106129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40fe880 0x7fd6c40fecd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 -- 172.21.15.101:0/4206106129 >> 172.21.15.101:0/4206106129 conn(0x7fd6c40f8db0 msgr2=0x7fd6c40fb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:03.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 -- 172.21.15.101:0/4206106129 shutdown_connections 2022-07-03T10:35:03.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.159+0000 7fd6c9bff700 1 -- 172.21.15.101:0/4206106129 wait complete. 2022-07-03T10:35:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/147738815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-07-03T10:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-07-03T10:35:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: osdmap e77: 12 total, 12 up, 12 in 2022-07-03T10:35:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:02 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:03.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.160+0000 7fd6c9bff700 1 Processor -- start 2022-07-03T10:35:03.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.160+0000 7fd6c9bff700 1 -- start start 2022-07-03T10:35:03.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 0x7fd6c4110d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 0x7fd6c4117df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4118370 con 0x7fd6c40fe880 2022-07-03T10:35:03.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41184b0 con 0x7fd6c40ff210 2022-07-03T10:35:03.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c9bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4118670 con 0x7fd6c40fd600 2022-07-03T10:35:03.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59582/0 (socket says 172.21.15.101:59582) 2022-07-03T10:35:03.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c37fe700 1 -- 172.21.15.101:0/1609073950 learned_addr learned my addr 172.21.15.101:0/1609073950 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:03.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c3fff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 0x7fd6c4117df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.161+0000 7fd6c2ffd700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 0x7fd6c4110d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 -- 172.21.15.101:0/1609073950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 msgr2=0x7fd6c4117df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 0x7fd6c4117df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 -- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 msgr2=0x7fd6c4110d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 0x7fd6c4110d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 -- 172.21.15.101:0/1609073950 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b8009440 con 0x7fd6c40fd600 2022-07-03T10:35:03.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c2ffd700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 0x7fd6c4110d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:35:03.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c3fff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 0x7fd6c4117df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:03.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c37fe700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400ea50 tx=0x7fd6b4005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:03.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4004020 con 0x7fd6c40fd600 2022-07-03T10:35:03.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd6b400cc70 con 0x7fd6c40fd600 2022-07-03T10:35:03.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c4118950 con 0x7fd6c40fd600 2022-07-03T10:35:03.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40039d0 con 0x7fd6c40fd600 2022-07-03T10:35:03.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.162+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c4118ee0 con 0x7fd6c40fd600 2022-07-03T10:35:03.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.163+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4108df0 con 0x7fd6c40fd600 2022-07-03T10:35:03.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.165+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd6b400c040 con 0x7fd6c40fd600 2022-07-03T10:35:03.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.166+0000 7fd6abfff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 0x7fd6ac0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.166+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7875+0+0 (secure 0 0 0) 0x7fd6b4014070 con 0x7fd6c40fd600 2022-07-03T10:35:03.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.167+0000 7fd6c2ffd700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 0x7fd6ac0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.177 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.167+0000 7fd6c2ffd700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 0x7fd6ac0991d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009f70 tx=0x7fd6b80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:03.177 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.170+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd6b408cd00 con 0x7fd6c40fd600 2022-07-03T10:35:03.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.424+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd6c4119550 con 0x7fd6c40fd600 2022-07-03T10:35:03.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.436+0000 7fd6abfff700 1 -- 172.21.15.101:0/1609073950 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd6b4090320 con 0x7fd6c40fd600 2022-07-03T10:35:03.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.452+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 msgr2=0x7fd6ac0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.452+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 0x7fd6ac0991d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009f70 tx=0x7fd6b80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.452+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 msgr2=0x7fd6c4110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.452+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400ea50 tx=0x7fd6b4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 shutdown_connections 2022-07-03T10:35:03.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd6ac096d20 0x7fd6ac0991d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6c40ff210 0x7fd6c4117df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6c40fd600 0x7fd6c4110840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 --2- 172.21.15.101:0/1609073950 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6c40fe880 0x7fd6c4110d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.455+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 >> 172.21.15.101:0/1609073950 conn(0x7fd6c40f8db0 msgr2=0x7fd6c40fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:03.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.462+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 shutdown_connections 2022-07-03T10:35:03.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.463+0000 7fd6c9bff700 1 -- 172.21.15.101:0/1609073950 wait complete. 2022-07-03T10:35:03.474 INFO:teuthology.orchestra.run.smithi101.stderr:new fs with metadata pool 2 and data pool 3 2022-07-03T10:35:03.502 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:35:03.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 -- 172.21.15.101:0/4106326687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840fd5d0 msgr2=0x7f10840fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 --2- 172.21.15.101:0/4106326687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840fd5d0 0x7f10840fd9e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1078009900 tx=0x7f10780091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 -- 172.21.15.101:0/4106326687 shutdown_connections 2022-07-03T10:35:03.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 --2- 172.21.15.101:0/4106326687 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840ff1e0 0x7f1084105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 --2- 172.21.15.101:0/4106326687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840fd5d0 0x7f10840fd9e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 --2- 172.21.15.101:0/4106326687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f10840feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 -- 172.21.15.101:0/4106326687 >> 172.21.15.101:0/4106326687 conn(0x7f10840f8d20 msgr2=0x7f10840fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:03.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 -- 172.21.15.101:0/4106326687 shutdown_connections 2022-07-03T10:35:03.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.746+0000 7f108afff700 1 -- 172.21.15.101:0/4106326687 wait complete. 2022-07-03T10:35:03.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 Processor -- start 2022-07-03T10:35:03.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 -- start start 2022-07-03T10:35:03.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f1084070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840ff1e0 0x7f1084070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084071480 con 0x7f10840fd5d0 2022-07-03T10:35:03.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10840715f0 con 0x7f10840fe850 2022-07-03T10:35:03.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.748+0000 7f108afff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10841186e0 con 0x7f10840ff1e0 2022-07-03T10:35:03.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:58534/0 (socket says 172.21.15.101:58534) 2022-07-03T10:35:03.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 -- 172.21.15.101:0/218790787 learned_addr learned my addr 172.21.15.101:0/218790787 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:03.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f108959c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840ff1e0 0x7f1084070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1083fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f1084070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 -- 172.21.15.101:0/218790787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840ff1e0 msgr2=0x7f1084070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840ff1e0 0x7f1084070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 -- 172.21.15.101:0/218790787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 msgr2=0x7f1084070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f1084070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 -- 172.21.15.101:0/218790787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009440 con 0x7f10840fd5d0 2022-07-03T10:35:03.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1083fff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f1084070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:03.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.749+0000 7f1088d9b700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1078005e10 tx=0x7f10780091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:03.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.750+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107801d070 con 0x7f10840fd5d0 2022-07-03T10:35:03.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.750+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10841188e0 con 0x7f10840fd5d0 2022-07-03T10:35:03.761 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.750+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f107800baa0 con 0x7f10840fd5d0 2022-07-03T10:35:03.761 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.750+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078022ad0 con 0x7f10840fd5d0 2022-07-03T10:35:03.761 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.750+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1084118dc0 con 0x7f10840fd5d0 2022-07-03T10:35:03.762 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.751+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1084108db0 con 0x7f10840fd5d0 2022-07-03T10:35:03.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.753+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f1078029030 con 0x7f10840fd5d0 2022-07-03T10:35:03.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.754+0000 7f10817fa700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 0x7f10740991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:03.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.754+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f10780c0b70 con 0x7f10840fd5d0 2022-07-03T10:35:03.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.755+0000 7f1083fff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 0x7f10740991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:03.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.755+0000 7f1083fff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 0x7f10740991d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1070011560 tx=0x7f1070005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:03.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.758+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f107808deb0 con 0x7f10840fd5d0 2022-07-03T10:35:03.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.959+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1084108f40 con 0x7f10840fd5d0 2022-07-03T10:35:03.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.961+0000 7f10817fa700 1 -- 172.21.15.101:0/218790787 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+21433 (secure 0 0 0) 0x7f1084108f40 con 0x7f10840fd5d0 2022-07-03T10:35:03.963 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:03.963 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":79,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:03.426687+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:35:03.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 msgr2=0x7f10740991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 0x7f10740991d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1070011560 tx=0x7f1070005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 msgr2=0x7f1084072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:03.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1078005e10 tx=0x7f10780091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 shutdown_connections 2022-07-03T10:35:03.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1074096d20 0x7f10740991d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f10840fd5d0 0x7f1084072230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10840ff1e0 0x7f1084070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 --2- 172.21.15.101:0/218790787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840fe850 0x7f1084070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:03.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 >> 172.21.15.101:0/218790787 conn(0x7f10840f8d20 msgr2=0x7f1084101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:03.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 shutdown_connections 2022-07-03T10:35:03.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:03.963+0000 7f108afff700 1 -- 172.21.15.101:0/218790787 wait complete. 2022-07-03T10:35:03.994 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:35:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: pgmap v265: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1402482129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:35:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:35:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-07-03T10:35:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: osdmap e78: 12 total, 12 up, 12 in 2022-07-03T10:35:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:35:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1609073950' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-07-03T10:35:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: osdmap e79: 12 total, 12 up, 12 in 2022-07-03T10:35:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-07-03T10:35:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-07-03T10:35:04.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:03 smithi161 ceph-mon[91579]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-07-03T10:35:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: pgmap v265: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1402482129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:35:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:35:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-07-03T10:35:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: osdmap e78: 12 total, 12 up, 12 in 2022-07-03T10:35:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:35:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1609073950' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-07-03T10:35:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: osdmap e79: 12 total, 12 up, 12 in 2022-07-03T10:35:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-07-03T10:35:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-07-03T10:35:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:03 smithi188 ceph-mon[92425]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-07-03T10:35:04.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.229+0000 7f845179d700 1 -- 172.21.15.101:0/3556261682 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fd400 msgr2=0x7f844c0fd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.229+0000 7f845179d700 1 --2- 172.21.15.101:0/3556261682 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fd400 0x7f844c0fd810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f843c009900 tx=0x7f843c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 -- 172.21.15.101:0/3556261682 shutdown_connections 2022-07-03T10:35:04.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 --2- 172.21.15.101:0/3556261682 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fd400 0x7f844c0fd810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 --2- 172.21.15.101:0/3556261682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0ff010 0x7f844c1058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 --2- 172.21.15.101:0/3556261682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0fe680 0x7f844c0fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 -- 172.21.15.101:0/3556261682 >> 172.21.15.101:0/3556261682 conn(0x7f844c0f8b70 msgr2=0x7f844c0fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:04.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 -- 172.21.15.101:0/3556261682 shutdown_connections 2022-07-03T10:35:04.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.230+0000 7f845179d700 1 -- 172.21.15.101:0/3556261682 wait complete. 2022-07-03T10:35:04.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 Processor -- start 2022-07-03T10:35:04.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 -- start start 2022-07-03T10:35:04.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fe680 0x7f844c110b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 0x7f844c117bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c118170 con 0x7f844c0fe680 2022-07-03T10:35:04.242 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c1182b0 con 0x7f844c0fd400 2022-07-03T10:35:04.242 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.231+0000 7f845179d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c118470 con 0x7f844c0ff010 2022-07-03T10:35:04.242 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f8450f9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 0x7f844c117bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.242 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.243 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f8450f9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 0x7f844c117bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59620/0 (socket says 172.21.15.101:59620) 2022-07-03T10:35:04.243 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f8450f9c700 1 -- 172.21.15.101:0/664900548 learned_addr learned my addr 172.21.15.101:0/664900548 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:04.243 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34314/0 (socket says 172.21.15.101:34314) 2022-07-03T10:35:04.243 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844b7fe700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fe680 0x7f844c110b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.244 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 -- 172.21.15.101:0/664900548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 msgr2=0x7f844c117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.244 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 0x7f844c117bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.244 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 -- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fe680 msgr2=0x7f844c110b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fe680 0x7f844c110b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 -- 172.21.15.101:0/664900548 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444009700 con 0x7f844c0fd400 2022-07-03T10:35:04.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.232+0000 7f844bfff700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f843c006010 tx=0x7f843c0091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:04.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.233+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c01d070 con 0x7f844c0fd400 2022-07-03T10:35:04.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.233+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c009440 con 0x7f844c0fd400 2022-07-03T10:35:04.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.233+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f843c00b9c0 con 0x7f844c0fd400 2022-07-03T10:35:04.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.233+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c022be0 con 0x7f844c0fd400 2022-07-03T10:35:04.248 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.233+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c118a60 con 0x7f844c0fd400 2022-07-03T10:35:04.248 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.234+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c108c00 con 0x7f844c0fd400 2022-07-03T10:35:04.249 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.238+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f843c029030 con 0x7f844c0fd400 2022-07-03T10:35:04.249 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.239+0000 7f8448ff9700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 0x7f8434099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.239+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f843c0c0810 con 0x7f844c0fd400 2022-07-03T10:35:04.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.240+0000 7f844b7fe700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 0x7f8434099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.240+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f843c08db50 con 0x7f844c0fd400 2022-07-03T10:35:04.251 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: pgmap v265: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2022-07-03T10:35:04.251 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1402482129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.251 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-07-03T10:35:04.252 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:35:04.252 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:35:04.252 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-07-03T10:35:04.252 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: osdmap e78: 12 total, 12 up, 12 in 2022-07-03T10:35:04.253 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:35:04.253 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1609073950' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.253 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-07-03T10:35:04.253 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-07-03T10:35:04.254 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: osdmap e79: 12 total, 12 up, 12 in 2022-07-03T10:35:04.254 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-07-03T10:35:04.254 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-07-03T10:35:04.254 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:03 smithi101 ceph-mon[85126]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-07-03T10:35:04.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.241+0000 7f844b7fe700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 0x7f8434099250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8440009c20 tx=0x7f8440005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:04.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.450+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f844c119420 con 0x7f844c0fd400 2022-07-03T10:35:04.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.452+0000 7f8448ff9700 1 -- 172.21.15.101:0/664900548 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21420 (secure 0 0 0) 0x7f843c027090 con 0x7f844c0fd400 2022-07-03T10:35:04.455 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:04.456 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":80,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:04.429266+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:35:04.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 msgr2=0x7f8434099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 0x7f8434099250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8440009c20 tx=0x7f8440005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 msgr2=0x7f844c110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f843c006010 tx=0x7f843c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 shutdown_connections 2022-07-03T10:35:04.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8434096da0 0x7f8434099250 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f844c0ff010 0x7f844c117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f844c0fd400 0x7f844c110640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 --2- 172.21.15.101:0/664900548 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f844c0fe680 0x7f844c110b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 >> 172.21.15.101:0/664900548 conn(0x7f844c0f8b70 msgr2=0x7f844c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:04.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 shutdown_connections 2022-07-03T10:35:04.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.454+0000 7f845179d700 1 -- 172.21.15.101:0/664900548 wait complete. 2022-07-03T10:35:04.487 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-03T10:35:04.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.728+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/819975840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4100580 msgr2=0x7fa0e41029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.728+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/819975840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4100580 0x7fa0e41029e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009900 tx=0x7fa0d00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/819975840 shutdown_connections 2022-07-03T10:35:04.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/819975840 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 0x7fa0e40fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/819975840 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4103070 0x7fa0e41054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/819975840 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4100580 0x7fa0e41029e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/819975840 >> 172.21.15.101:0/819975840 conn(0x7fa0e40f7a40 msgr2=0x7fa0e40f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:04.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/819975840 shutdown_connections 2022-07-03T10:35:04.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.729+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/819975840 wait complete. 2022-07-03T10:35:04.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.730+0000 7fa0e9f9d700 1 Processor -- start 2022-07-03T10:35:04.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.730+0000 7fa0e9f9d700 1 -- start start 2022-07-03T10:35:04.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 0x7fa0e4110650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4100580 0x7fa0e4110b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4118180 con 0x7fa0e4066420 2022-07-03T10:35:04.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41182c0 con 0x7fa0e4100580 2022-07-03T10:35:04.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e9f9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4118480 con 0x7fa0e4103070 2022-07-03T10:35:04.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e979c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e979c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59634/0 (socket says 172.21.15.101:59634) 2022-07-03T10:35:04.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e979c700 1 -- 172.21.15.101:0/995754820 learned_addr learned my addr 172.21.15.101:0/995754820 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:04.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e8f9b700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 0x7fa0e4110650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.731+0000 7fa0e3fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4100580 0x7fa0e4110b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 -- 172.21.15.101:0/995754820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4100580 msgr2=0x7fa0e4110b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4100580 0x7fa0e4110b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 -- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 msgr2=0x7fa0e4110650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:04.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 0x7fa0e4110650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:04.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 -- 172.21.15.101:0/995754820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0009440 con 0x7fa0e4103070 2022-07-03T10:35:04.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e979c700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00eb80 tx=0x7fa0dc009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:04.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc004a50 con 0x7fa0e4103070 2022-07-03T10:35:04.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e4118760 con 0x7fa0e4103070 2022-07-03T10:35:04.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa0dc00ce80 con 0x7fa0e4103070 2022-07-03T10:35:04.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.732+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0056d0 con 0x7fa0e4103070 2022-07-03T10:35:04.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.733+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4118cf0 con 0x7fa0e4103070 2022-07-03T10:35:04.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.734+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4108bf0 con 0x7fa0e4103070 2022-07-03T10:35:04.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.735+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa0dc00c040 con 0x7fa0e4103070 2022-07-03T10:35:04.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.737+0000 7fa0e17fa700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 0x7fa0d40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:04.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.737+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa0dc014070 con 0x7fa0e4103070 2022-07-03T10:35:04.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.737+0000 7fa0e8f9b700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 0x7fa0d40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:04.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.738+0000 7fa0e8f9b700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 0x7fa0d40991d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40667f0 tx=0x7fa0d8005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:04.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:04.741+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa0dc08d440 con 0x7fa0e4103070 2022-07-03T10:35:05.006 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.004+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa0e4108d80 con 0x7fa0e4103070 2022-07-03T10:35:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: pgmap v268: 97 pgs: 11 creating+peering, 62 unknown, 24 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2022-07-03T10:35:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/218790787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: osdmap e80: 12 total, 12 up, 12 in 2022-07-03T10:35:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:35:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:04 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/664900548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: pgmap v268: 97 pgs: 11 creating+peering, 62 unknown, 24 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2022-07-03T10:35:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/218790787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: osdmap e80: 12 total, 12 up, 12 in 2022-07-03T10:35:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:35:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:04 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/664900548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: pgmap v268: 97 pgs: 11 creating+peering, 62 unknown, 24 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2022-07-03T10:35:05.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/218790787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: osdmap e80: 12 total, 12 up, 12 in 2022-07-03T10:35:05.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:35:05.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:05.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:04 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/664900548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:05.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.864+0000 7fa0e17fa700 1 -- 172.21.15.101:0/995754820 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa0e4108d80 con 0x7fa0e4103070 2022-07-03T10:35:05.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 msgr2=0x7fa0d40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:05.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 0x7fa0d40991d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40667f0 tx=0x7fa0d8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:05.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 msgr2=0x7fa0e4117c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:05.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00eb80 tx=0x7fa0dc009460 comp rx=0 tx=0).stop 2022-07-03T10:35:05.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 shutdown_connections 2022-07-03T10:35:05.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa0d4096d20 0x7fa0d40991d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:05.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa0e4066420 0x7fa0e4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:05.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa0e4103070 0x7fa0e4117c00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:05.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 --2- 172.21.15.101:0/995754820 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0e4100580 0x7fa0e4110b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:05.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.866+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 >> 172.21.15.101:0/995754820 conn(0x7fa0e40f7a40 msgr2=0x7fa0e40f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:05.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.867+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 shutdown_connections 2022-07-03T10:35:05.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:05.867+0000 7fa0e9f9d700 1 -- 172.21.15.101:0/995754820 wait complete. 2022-07-03T10:35:05.902 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2022-07-03T10:35:05.903 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2022-07-03T10:35:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/995754820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:05 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.145 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 -- 172.21.15.101:0/2025709394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c103240 msgr2=0x7f685c1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:06.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 --2- 172.21.15.101:0/2025709394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c103240 0x7f685c1056c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6854009ca0 tx=0x7f6854009520 comp rx=0 tx=0).stop 2022-07-03T10:35:06.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 -- 172.21.15.101:0/2025709394 shutdown_connections 2022-07-03T10:35:06.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 --2- 172.21.15.101:0/2025709394 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c067620 0x7f685c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 --2- 172.21.15.101:0/2025709394 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c103240 0x7f685c1056c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 --2- 172.21.15.101:0/2025709394 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c1007b0 0x7f685c102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 -- 172.21.15.101:0/2025709394 >> 172.21.15.101:0/2025709394 conn(0x7f685c0f7c50 msgr2=0x7f685c0fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:06.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 -- 172.21.15.101:0/2025709394 shutdown_connections 2022-07-03T10:35:06.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.143+0000 7f6862fff700 1 -- 172.21.15.101:0/2025709394 wait complete. 2022-07-03T10:35:06.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.144+0000 7f6862fff700 1 Processor -- start 2022-07-03T10:35:06.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.144+0000 7f6862fff700 1 -- start start 2022-07-03T10:35:06.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:06.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:06.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c103240 0x7f685c1158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:06.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c115e50 con 0x7f685c103240 2022-07-03T10:35:06.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c115f90 con 0x7f685c1007b0 2022-07-03T10:35:06.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6862fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c116150 con 0x7f685c067620 2022-07-03T10:35:06.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6860d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:06.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f685bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:06.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6860d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59654/0 (socket says 172.21.15.101:59654) 2022-07-03T10:35:06.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f685bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34336/0 (socket says 172.21.15.101:34336) 2022-07-03T10:35:06.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f6860d9b700 1 -- 172.21.15.101:0/3754662567 learned_addr learned my addr 172.21.15.101:0/3754662567 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:06.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.145+0000 7f686159c700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c103240 0x7f685c1158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:06.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 -- 172.21.15.101:0/3754662567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 msgr2=0x7f685c10e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:06.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 -- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c103240 msgr2=0x7f685c1158d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:06.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c103240 0x7f685c1158d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 -- 172.21.15.101:0/3754662567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68540097e0 con 0x7f685c067620 2022-07-03T10:35:06.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f685bfff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:06.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6860d9b700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f685000ea80 tx=0x7f6850005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:06.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685000c740 con 0x7f685c067620 2022-07-03T10:35:06.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c116430 con 0x7f685c067620 2022-07-03T10:35:06.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f685000cd50 con 0x7f685c067620 2022-07-03T10:35:06.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.146+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6850010d40 con 0x7f685c067620 2022-07-03T10:35:06.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.147+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c116a10 con 0x7f685c067620 2022-07-03T10:35:06.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.147+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c108e00 con 0x7f685c067620 2022-07-03T10:35:06.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.149+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f685000c040 con 0x7f685c067620 2022-07-03T10:35:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/995754820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:05 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.150+0000 7f68597fa700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 0x7f684c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:06.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.150+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f6850014070 con 0x7f685c067620 2022-07-03T10:35:06.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.151+0000 7f685bfff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 0x7f684c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:06.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.151+0000 7f685bfff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 0x7f684c0991d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f685c067a50 tx=0x7f6848005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:06.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.154+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f685004f100 con 0x7f685c067620 2022-07-03T10:35:06.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/995754820' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-07-03T10:35:06.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:05 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:06.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.407+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f685c0f9640 con 0x7f685c067620 2022-07-03T10:35:06.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.874+0000 7f68597fa700 1 -- 172.21.15.101:0/3754662567 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7f685008cb80 con 0x7f685c067620 2022-07-03T10:35:06.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 msgr2=0x7f684c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:06.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 0x7f684c0991d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f685c067a50 tx=0x7f6848005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 msgr2=0x7f685c10e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:06.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f685000ea80 tx=0x7f6850005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 shutdown_connections 2022-07-03T10:35:06.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f684c096d20 0x7f684c0991d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f685c103240 0x7f685c1158d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f685c067620 0x7f685c10e320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 --2- 172.21.15.101:0/3754662567 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f685c1007b0 0x7f685c10e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:06.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 >> 172.21.15.101:0/3754662567 conn(0x7f685c0f7c50 msgr2=0x7f685c0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:06.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 shutdown_connections 2022-07-03T10:35:06.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:06.877+0000 7f6862fff700 1 -- 172.21.15.101:0/3754662567 wait complete. 2022-07-03T10:35:06.906 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-07-03T10:35:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:06 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-07-03T10:35:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:06 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:06 smithi161 ceph-mon[91579]: pgmap v270: 97 pgs: 11 creating+peering, 33 unknown, 53 active+clean; 578 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2022-07-03T10:35:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:06 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3754662567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:06 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 -- 172.21.15.101:0/2434879273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984100670 msgr2=0x7f7984102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:07.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 --2- 172.21.15.101:0/2434879273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984100670 0x7f7984102a50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7974009900 tx=0x7f79740091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 -- 172.21.15.101:0/2434879273 shutdown_connections 2022-07-03T10:35:07.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 --2- 172.21.15.101:0/2434879273 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79841030c0 0x7f7984105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 --2- 172.21.15.101:0/2434879273 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7984067480 0x7f7984100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 --2- 172.21.15.101:0/2434879273 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984100670 0x7f7984102a50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.144+0000 7f798a39d700 1 -- 172.21.15.101:0/2434879273 >> 172.21.15.101:0/2434879273 conn(0x7f79840f7ae0 msgr2=0x7f79840f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:07.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.145+0000 7f798a39d700 1 -- 172.21.15.101:0/2434879273 shutdown_connections 2022-07-03T10:35:07.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.145+0000 7f798a39d700 1 -- 172.21.15.101:0/2434879273 wait complete. 2022-07-03T10:35:07.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 Processor -- start 2022-07-03T10:35:07.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 -- start start 2022-07-03T10:35:07.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 0x7f798410e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:07.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:07.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79841030c0 0x7f7984115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:07.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984115fc0 con 0x7f7984067480 2022-07-03T10:35:07.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984116100 con 0x7f7984100670 2022-07-03T10:35:07.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798a39d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841162c0 con 0x7f79841030c0 2022-07-03T10:35:07.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f7988b9a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:07.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.146+0000 7f798939b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 0x7f798410e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:07.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f7989b9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79841030c0 0x7f7984115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:07.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f7988b9a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:34358/0 (socket says 172.21.15.101:34358) 2022-07-03T10:35:07.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f7988b9a700 1 -- 172.21.15.101:0/2774411468 learned_addr learned my addr 172.21.15.101:0/2774411468 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:07.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 -- 172.21.15.101:0/2774411468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79841030c0 msgr2=0x7f7984115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79841030c0 0x7f7984115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 -- 172.21.15.101:0/2774411468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 msgr2=0x7f798410e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 -- 172.21.15.101:0/2774411468 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c009700 con 0x7f7984067480 2022-07-03T10:35:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f7988b9a700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f798939b700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 0x7f798410e490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7980009a90 tx=0x7f7980005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.147+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980015070 con 0x7f7984067480 2022-07-03T10:35:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.148+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f79800107f0 con 0x7f7984067480 2022-07-03T10:35:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.148+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980003b80 con 0x7f7984067480 2022-07-03T10:35:07.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.148+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7974009440 con 0x7f7984067480 2022-07-03T10:35:07.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.148+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79840fc320 con 0x7f7984067480 2022-07-03T10:35:07.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.149+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7984108c00 con 0x7f7984067480 2022-07-03T10:35:07.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.151+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f7980020030 con 0x7f7984067480 2022-07-03T10:35:07.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.152+0000 7f7979ffb700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 0x7f7970099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.152+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f79800c2ff0 con 0x7f7984067480 2022-07-03T10:35:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.152+0000 7f7988b9a700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 0x7f7970099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.153+0000 7f7988b9a700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 0x7f7970099250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79740056f0 tx=0x7f79740091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.156+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f7980090350 con 0x7f7984067480 2022-07-03T10:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:06 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-07-03T10:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:06 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:06 smithi188 ceph-mon[92425]: pgmap v270: 97 pgs: 11 creating+peering, 33 unknown, 53 active+clean; 578 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2022-07-03T10:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:06 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3754662567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:06 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:06 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-07-03T10:35:07.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:06 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:07.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:06 smithi101 ceph-mon[85126]: pgmap v270: 97 pgs: 11 creating+peering, 33 unknown, 53 active+clean; 578 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 7 op/s 2022-07-03T10:35:07.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:06 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3754662567' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:06 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-07-03T10:35:07.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:07.394+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f7984108d90 con 0x7f7984067480 2022-07-03T10:35:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-07-03T10:35:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:07 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-07-03T10:35:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-07-03T10:35:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:07 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-07-03T10:35:08.213 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.211+0000 7f7979ffb700 1 -- 172.21.15.101:0/2774411468 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7984108d90 con 0x7f7984067480 2022-07-03T10:35:08.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 msgr2=0x7f7970099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 0x7f7970099250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79740056f0 tx=0x7f79740091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 msgr2=0x7f798410e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 0x7f798410e490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7980009a90 tx=0x7f7980005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 shutdown_connections 2022-07-03T10:35:08.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7970096da0 0x7f7970099250 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7984067480 0x7f798410e490 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79841030c0 0x7f7984115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 --2- 172.21.15.101:0/2774411468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7984100670 0x7f798410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 >> 172.21.15.101:0/2774411468 conn(0x7f79840f7ae0 msgr2=0x7f79840fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:08.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.214+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 shutdown_connections 2022-07-03T10:35:08.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.215+0000 7f798a39d700 1 -- 172.21.15.101:0/2774411468 wait complete. 2022-07-03T10:35:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-07-03T10:35:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-07-03T10:35:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:08.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:35:08.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:07 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-07-03T10:35:08.248 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2022-07-03T10:35:08.248 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2022-07-03T10:35:08.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.479+0000 7f3feffff700 1 -- 172.21.15.101:0/849738847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 msgr2=0x7f3ff00fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.479+0000 7f3feffff700 1 --2- 172.21.15.101:0/849738847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff00fd800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009900 tx=0x7f3fe00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 -- 172.21.15.101:0/849738847 shutdown_connections 2022-07-03T10:35:08.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 --2- 172.21.15.101:0/849738847 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 0x7f3ff01058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 --2- 172.21.15.101:0/849738847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff00fd800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 --2- 172.21.15.101:0/849738847 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff00feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 -- 172.21.15.101:0/849738847 >> 172.21.15.101:0/849738847 conn(0x7f3ff00f8b40 msgr2=0x7f3ff00fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:08.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 -- 172.21.15.101:0/849738847 shutdown_connections 2022-07-03T10:35:08.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.480+0000 7f3feffff700 1 -- 172.21.15.101:0/849738847 wait complete. 2022-07-03T10:35:08.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 Processor -- start 2022-07-03T10:35:08.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 -- start start 2022-07-03T10:35:08.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff010e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:08.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:08.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 0x7f3ff01158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:08.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0115e60 con 0x7f3ff00fe670 2022-07-03T10:35:08.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0115fa0 con 0x7f3ff00ff000 2022-07-03T10:35:08.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3feffff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0116160 con 0x7f3ff00fd3f0 2022-07-03T10:35:08.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3fee7fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:08.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fef7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 0x7f3ff01158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:08.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.481+0000 7f3fee7fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46098/0 (socket says 172.21.15.101:46098) 2022-07-03T10:35:08.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 -- 172.21.15.101:0/4235820161 learned_addr learned my addr 172.21.15.101:0/4235820161 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:08.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3feeffd700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff010e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:08.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 -- 172.21.15.101:0/4235820161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 msgr2=0x7f3ff010e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff010e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 -- 172.21.15.101:0/4235820161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 msgr2=0x7f3ff01158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 0x7f3ff01158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 -- 172.21.15.101:0/4235820161 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe0009440 con 0x7f3ff00fe670 2022-07-03T10:35:08.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fee7fc700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400ea80 tx=0x7f3fe4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:08.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.482+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400c7c0 con 0x7f3ff00fe670 2022-07-03T10:35:08.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.483+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3fe400cdd0 con 0x7f3ff00fe670 2022-07-03T10:35:08.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.483+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4010df0 con 0x7f3ff00fe670 2022-07-03T10:35:08.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.483+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff0116440 con 0x7f3ff00fe670 2022-07-03T10:35:08.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.483+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff01169d0 con 0x7f3ff00fe670 2022-07-03T10:35:08.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.484+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff0108ba0 con 0x7f3ff00fe670 2022-07-03T10:35:08.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.486+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f3fe400c040 con 0x7f3ff00fe670 2022-07-03T10:35:08.495 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.487+0000 7f3fd77fe700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 0x7f3fd8099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:08.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.487+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f3fe4014070 con 0x7f3ff00fe670 2022-07-03T10:35:08.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.487+0000 7f3feeffd700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 0x7f3fd8099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:08.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.488+0000 7f3feeffd700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 0x7f3fd8099250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00059f0 tx=0x7f3fe00091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:08.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.490+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f3fe404f100 con 0x7f3ff00fe670 2022-07-03T10:35:08.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.707+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ff005e9f0 con 0x7f3fd8096da0 2022-07-03T10:35:08.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.746+0000 7f3fd77fe700 1 -- 172.21.15.101:0/4235820161 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3ff005e9f0 con 0x7f3fd8096da0 2022-07-03T10:35:08.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.748+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 msgr2=0x7f3fd8099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.748+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 0x7f3fd8099250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00059f0 tx=0x7f3fe00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.748+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 msgr2=0x7f3ff010e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:08.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.748+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400ea80 tx=0x7f3fe4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 shutdown_connections 2022-07-03T10:35:08.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3fd8096da0 0x7f3fd8099250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ff00ff000 0x7f3ff01158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ff00fd3f0 0x7f3ff010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 --2- 172.21.15.101:0/4235820161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3ff00fe670 0x7f3ff010e870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:08.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 >> 172.21.15.101:0/4235820161 conn(0x7f3ff00f8b40 msgr2=0x7f3ff00fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:08.753 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 shutdown_connections 2022-07-03T10:35:08.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:08.749+0000 7f3feffff700 1 -- 172.21.15.101:0/4235820161 wait complete. 2022-07-03T10:35:08.778 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2022-07-03T10:35:08.913 INFO:journalctl@ceph.mds.l.smithi188.stdout:Jul 03 10:35:08 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mds-l[114596]: 2022-07-03T10:35:08.644+0000 7f2a3b64c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-07-03T10:35:08.992 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:35:08 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:35:08.714+0000 7fe0ddead700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:35:08.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:35:08 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:35:08.714+0000 7fe0ddead700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:35:08.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:35:08 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:35:08.714+0000 7fe0ddead700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:35:08.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:35:08 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:35:08.714+0000 7fe0ddead700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:35:08.993 INFO:journalctl@ceph.mgr.x.smithi101.stdout:Jul 03 10:35:08 smithi101 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-x[85379]: 2022-07-03T10:35:08.714+0000 7fe0ddead700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:35:09.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 -- 172.21.15.101:0/466452422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564066380 msgr2=0x7fa564100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/466452422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564066380 0x7fa564100050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa560009900 tx=0x7fa5600091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 -- 172.21.15.101:0/466452422 shutdown_connections 2022-07-03T10:35:09.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/466452422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564103060 0x7fa5641054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/466452422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564066380 0x7fa564100050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/466452422 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa5641029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 -- 172.21.15.101:0/466452422 >> 172.21.15.101:0/466452422 conn(0x7fa5640f7a40 msgr2=0x7fa5640f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:09.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 -- 172.21.15.101:0/466452422 shutdown_connections 2022-07-03T10:35:09.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.007+0000 7fa56ad9d700 1 -- 172.21.15.101:0/466452422 wait complete. 2022-07-03T10:35:09.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 Processor -- start 2022-07-03T10:35:09.014 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 -- start start 2022-07-03T10:35:09.014 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564066380 0x7fa56410e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.014 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 0x7fa5641159b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa564115f30 con 0x7fa564100590 2022-07-03T10:35:09.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa564116070 con 0x7fa564066380 2022-07-03T10:35:09.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.008+0000 7fa56ad9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa564116230 con 0x7fa564103060 2022-07-03T10:35:09.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56959a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 0x7fa5641159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56959a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46138/0 (socket says 172.21.15.101:46138) 2022-07-03T10:35:09.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56959a700 1 -- 172.21.15.101:0/3942644683 learned_addr learned my addr 172.21.15.101:0/3942644683 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:09.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa569d9b700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564066380 0x7fa56410e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 -- 172.21.15.101:0/3942644683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564066380 msgr2=0x7fa56410e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564066380 0x7fa56410e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 -- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 msgr2=0x7fa56410e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 -- 172.21.15.101:0/3942644683 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560009440 con 0x7fa564103060 2022-07-03T10:35:09.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56959a700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:09.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.009+0000 7fa56a59c700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 0x7fa5641159b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00eab0 tx=0x7fa55c009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:09.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.010+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009850 con 0x7fa564103060 2022-07-03T10:35:09.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.010+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa564116510 con 0x7fa564103060 2022-07-03T10:35:09.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.010+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa55c00c9d0 con 0x7fa564103060 2022-07-03T10:35:09.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.010+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c004020 con 0x7fa564103060 2022-07-03T10:35:09.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.010+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa564116aa0 con 0x7fa564103060 2022-07-03T10:35:09.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.011+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa564108bc0 con 0x7fa564103060 2022-07-03T10:35:09.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.013+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa55c01e020 con 0x7fa564103060 2022-07-03T10:35:09.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.014+0000 7fa55a7fc700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 0x7fa5500991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.014+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa55c014070 con 0x7fa564103060 2022-07-03T10:35:09.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.014+0000 7fa569d9b700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 0x7fa5500991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.015+0000 7fa569d9b700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 0x7fa5500991d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa560009f70 tx=0x7fa5600091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:09.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.016+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa55c08d190 con 0x7fa564103060 2022-07-03T10:35:09.242 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.240+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa56405e960 con 0x7fa550096d20 2022-07-03T10:35:09.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.244+0000 7fa55a7fc700 1 -- 172.21.15.101:0/3942644683 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fa56405e960 con 0x7fa550096d20 2022-07-03T10:35:09.246 INFO:teuthology.orchestra.run.smithi101.stdout:/volumes/_nogroup/sv_0/315d7180-954f-408e-8dc9-aa9b3be5dd74 2022-07-03T10:35:09.249 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.246+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 msgr2=0x7fa5500991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.249 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.246+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 0x7fa5500991d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa560009f70 tx=0x7fa5600091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 msgr2=0x7fa5641159b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 0x7fa5641159b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00eab0 tx=0x7fa55c009460 comp rx=0 tx=0).stop 2022-07-03T10:35:09.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 shutdown_connections 2022-07-03T10:35:09.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa550096d20 0x7fa5500991d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa564103060 0x7fa5641159b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa564066380 0x7fa56410e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 --2- 172.21.15.101:0/3942644683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa564100590 0x7fa56410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 >> 172.21.15.101:0/3942644683 conn(0x7fa5640f7a40 msgr2=0x7fa5640fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:09.252 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 shutdown_connections 2022-07-03T10:35:09.252 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.247+0000 7fa56ad9d700 1 -- 172.21.15.101:0/3942644683 wait complete. 2022-07-03T10:35:09.275 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2022-07-03T10:35:09.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:09 smithi101 ceph-mon[85126]: pgmap v271: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-07-03T10:35:09.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:09 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-07-03T10:35:09.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:09 smithi101 ceph-mon[85126]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:09.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:09 smithi101 ceph-mon[85126]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.502+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/166719912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 msgr2=0x7f2d98105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.502+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/166719912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98105ab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009ca0 tx=0x7f2d90009520 comp rx=0 tx=0).stop 2022-07-03T10:35:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/166719912 shutdown_connections 2022-07-03T10:35:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/166719912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98105ab0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/166719912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fd5e0 0x7f2d980fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/166719912 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fe860 0x7f2d980fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/166719912 >> 172.21.15.101:0/166719912 conn(0x7f2d980f8d50 msgr2=0x7f2d980fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/166719912 shutdown_connections 2022-07-03T10:35:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.503+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/166719912 wait complete. 2022-07-03T10:35:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 Processor -- start 2022-07-03T10:35:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 -- start start 2022-07-03T10:35:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fe860 0x7f2d9810e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98115890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98115e10 con 0x7f2d980fd5e0 2022-07-03T10:35:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98115f50 con 0x7f2d980ff1f0 2022-07-03T10:35:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9f1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98116110 con 0x7f2d980fe860 2022-07-03T10:35:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.504+0000 7f2d9cf9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46148/0 (socket says 172.21.15.101:46148) 2022-07-03T10:35:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 -- 172.21.15.101:0/2672786242 learned_addr learned my addr 172.21.15.101:0/2672786242 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d97fff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fe860 0x7f2d9810e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9d79c700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98115890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 -- 172.21.15.101:0/2672786242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fe860 msgr2=0x7f2d9810e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fe860 0x7f2d9810e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 -- 172.21.15.101:0/2672786242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 msgr2=0x7f2d98115890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98115890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 -- 172.21.15.101:0/2672786242 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c009410 con 0x7f2d980fd5e0 2022-07-03T10:35:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d9cf9b700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00ea00 tx=0x7f2d8c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.505+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00c7a0 con 0x7f2d980fd5e0 2022-07-03T10:35:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.506+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d84009410 con 0x7f2d980fd5e0 2022-07-03T10:35:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.506+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2d8c00cdb0 con 0x7f2d980fd5e0 2022-07-03T10:35:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.506+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c010e50 con 0x7f2d980fd5e0 2022-07-03T10:35:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.506+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d900097e0 con 0x7f2d980fd5e0 2022-07-03T10:35:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.507+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d98108dd0 con 0x7f2d980fd5e0 2022-07-03T10:35:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.508+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f2d8c00c040 con 0x7f2d980fd5e0 2022-07-03T10:35:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.510+0000 7f2d957fa700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 0x7f2d880991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.510+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f2d8c014070 con 0x7f2d980fd5e0 2022-07-03T10:35:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.510+0000 7f2d97fff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 0x7f2d880991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.511+0000 7f2d97fff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 0x7f2d880991d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009e90 tx=0x7f2d84005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.513+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f2d8c08cef0 con 0x7f2d980fd5e0 2022-07-03T10:35:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:09 smithi161 ceph-mon[91579]: pgmap v271: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-07-03T10:35:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:09 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-07-03T10:35:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:09 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:09 smithi161 ceph-mon[91579]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:09 smithi188 ceph-mon[92425]: pgmap v271: 97 pgs: 2 creating+peering, 95 active+clean; 579 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.3 KiB/s wr, 7 op/s 2022-07-03T10:35:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:09 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2774411468' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-07-03T10:35:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:09 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:35:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:09 smithi188 ceph-mon[92425]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:09.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.730+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d9805e9f0 con 0x7f2d88096d20 2022-07-03T10:35:09.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.752+0000 7f2d957fa700 1 -- 172.21.15.101:0/2672786242 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2d9805e9f0 con 0x7f2d88096d20 2022-07-03T10:35:09.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 msgr2=0x7f2d880991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 0x7f2d880991d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009e90 tx=0x7f2d84005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 msgr2=0x7f2d9810e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:09.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00ea00 tx=0x7f2d8c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 shutdown_connections 2022-07-03T10:35:09.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2d88096d20 0x7f2d880991d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2d980fd5e0 0x7f2d9810e2e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d980ff1f0 0x7f2d98115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 --2- 172.21.15.101:0/2672786242 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d980fe860 0x7f2d9810e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:09.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 >> 172.21.15.101:0/2672786242 conn(0x7f2d980f8d50 msgr2=0x7f2d980fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:09.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 shutdown_connections 2022-07-03T10:35:09.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:09.755+0000 7f2d9f1ff700 1 -- 172.21.15.101:0/2672786242 wait complete. 2022-07-03T10:35:09.784 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2022-07-03T10:35:10.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/2112841876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fd5e0 msgr2=0x7fd8b80fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/2112841876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b80fd9f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009900 tx=0x7fd8a80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/2112841876 shutdown_connections 2022-07-03T10:35:10.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/2112841876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 0x7fd8b8105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/2112841876 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b80fd9f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/2112841876 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fe860 0x7fd8b80fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/2112841876 >> 172.21.15.101:0/2112841876 conn(0x7fd8b80f8d90 msgr2=0x7fd8b80fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:10.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/2112841876 shutdown_connections 2022-07-03T10:35:10.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.013+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/2112841876 wait complete. 2022-07-03T10:35:10.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.014+0000 7fd8bd7ff700 1 Processor -- start 2022-07-03T10:35:10.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 -- start start 2022-07-03T10:35:10.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fe860 0x7fd8b8110d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 0x7fd8b8117d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b8118310 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b8118450 con 0x7fd8b80fe860 2022-07-03T10:35:10.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8bd7ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b8118610 con 0x7fd8b80ff1f0 2022-07-03T10:35:10.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.015+0000 7fd8b6ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fe860 0x7fd8b8110d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b77fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 0x7fd8b8117d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46164/0 (socket says 172.21.15.101:46164) 2022-07-03T10:35:10.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 -- 172.21.15.101:0/4071730336 learned_addr learned my addr 172.21.15.101:0/4071730336 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:10.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 -- 172.21.15.101:0/4071730336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 msgr2=0x7fd8b8117d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 0x7fd8b8117d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 -- 172.21.15.101:0/4071730336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fe860 msgr2=0x7fd8b8110d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fe860 0x7fd8b8110d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 -- 172.21.15.101:0/4071730336 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0009700 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.016+0000 7fd8b6ffd700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a80091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:10.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.017+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a801d070 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.017+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a8009440 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.017+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd8a800b9c0 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.017+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8022b10 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.017+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b8118c00 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.018+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b8108d80 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.020+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd8a8029030 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.021+0000 7fd89f7fe700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 0x7fd8a00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.021+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fd8a80c0b70 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.021+0000 7fd8b67fc700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 0x7fd8a00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.022+0000 7fd8b67fc700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 0x7fd8a00991d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009c70 tx=0x7fd8ac005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:10.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.024+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd8a808df40 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.227+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd8a800c040 con 0x7fd8b80fd5e0 2022-07-03T10:35:10.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.249+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b8071e70 con 0x7fd8a0096d20 2022-07-03T10:35:10.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.253+0000 7fd89f7fe700 1 -- 172.21.15.101:0/4071730336 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fd8b8071e70 con 0x7fd8a0096d20 2022-07-03T10:35:10.255 INFO:teuthology.orchestra.run.smithi101.stdout:/volumes/_nogroup/sv_1/f64dc3a0-3e2a-4139-9ee3-035a6e925e48 2022-07-03T10:35:10.258 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 msgr2=0x7fd8a00991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.258 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 0x7fd8a00991d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009c70 tx=0x7fd8ac005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.259 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 msgr2=0x7fd8b81107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.259 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.259 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 shutdown_connections 2022-07-03T10:35:10.259 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd8a0096d20 0x7fd8a00991d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.260 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd8b80fd5e0 0x7fd8b81107e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.260 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8b80ff1f0 0x7fd8b8117d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.260 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 --2- 172.21.15.101:0/4071730336 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8b80fe860 0x7fd8b8110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.261 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 >> 172.21.15.101:0/4071730336 conn(0x7fd8b80f8d90 msgr2=0x7fd8b8070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:10.261 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 shutdown_connections 2022-07-03T10:35:10.261 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.256+0000 7fd8bd7ff700 1 -- 172.21.15.101:0/4071730336 wait complete. 2022-07-03T10:35:10.284 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:35:10.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:10 smithi101 ceph-mon[85126]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:10.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.509+0000 7feb9759e700 1 -- 172.21.15.101:0/3803205799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980ff010 msgr2=0x7feb981058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.509+0000 7feb9759e700 1 --2- 172.21.15.101:0/3803205799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980ff010 0x7feb981058d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb90009ca0 tx=0x7feb90009520 comp rx=0 tx=0).stop 2022-07-03T10:35:10.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 -- 172.21.15.101:0/3803205799 shutdown_connections 2022-07-03T10:35:10.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 --2- 172.21.15.101:0/3803205799 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980ff010 0x7feb981058d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 --2- 172.21.15.101:0/3803205799 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980fd400 0x7feb980fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 --2- 172.21.15.101:0/3803205799 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb980fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 -- 172.21.15.101:0/3803205799 >> 172.21.15.101:0/3803205799 conn(0x7feb980f8b70 msgr2=0x7feb980fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:10.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 -- 172.21.15.101:0/3803205799 shutdown_connections 2022-07-03T10:35:10.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 -- 172.21.15.101:0/3803205799 wait complete. 2022-07-03T10:35:10.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.510+0000 7feb9759e700 1 Processor -- start 2022-07-03T10:35:10.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 -- start start 2022-07-03T10:35:10.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 0x7feb9810e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980ff010 0x7feb98115670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98115bf0 con 0x7feb980fe680 2022-07-03T10:35:10.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98115d30 con 0x7feb980ff010 2022-07-03T10:35:10.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9759e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb98115ef0 con 0x7feb980fd400 2022-07-03T10:35:10.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb95d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb9659c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 0x7feb9810e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb95d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46186/0 (socket says 172.21.15.101:46186) 2022-07-03T10:35:10.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb95d9b700 1 -- 172.21.15.101:0/177320306 learned_addr learned my addr 172.21.15.101:0/177320306 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:10.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.511+0000 7feb96d9d700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980ff010 0x7feb98115670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 -- 172.21.15.101:0/177320306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980ff010 msgr2=0x7feb98115670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980ff010 0x7feb98115670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 -- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 msgr2=0x7feb9810e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 -- 172.21.15.101:0/177320306 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900097e0 con 0x7feb980fd400 2022-07-03T10:35:10.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb95d9b700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:10.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9659c700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 0x7feb9810e0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb8800e820 tx=0x7feb88005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:10.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88004030 con 0x7feb980fd400 2022-07-03T10:35:10.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7feb88004190 con 0x7feb980fd400 2022-07-03T10:35:10.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb980707b0 con 0x7feb980fd400 2022-07-03T10:35:10.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.512+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88003980 con 0x7feb980fd400 2022-07-03T10:35:10.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.513+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb98070d40 con 0x7feb980fd400 2022-07-03T10:35:10.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.513+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb98108c00 con 0x7feb980fd400 2022-07-03T10:35:10.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.515+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7feb8800c040 con 0x7feb980fd400 2022-07-03T10:35:10.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.517+0000 7feb86ffd700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 0x7feb80099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:10.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.517+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7feb88014070 con 0x7feb980fd400 2022-07-03T10:35:10.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.517+0000 7feb95d9b700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 0x7feb80099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:10.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.518+0000 7feb95d9b700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 0x7feb80099290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c012040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:10.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.519+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7feb8808d1b0 con 0x7feb980fd400 2022-07-03T10:35:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:10 smithi161 ceph-mon[91579]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:10 smithi188 ceph-mon[92425]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:10.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.751+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb98108d90 con 0x7feb980fd400 2022-07-03T10:35:10.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.753+0000 7feb86ffd700 1 -- 172.21.15.101:0/177320306 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7feb98108d90 con 0x7feb980fd400 2022-07-03T10:35:10.761 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:10.761 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:35:10.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 msgr2=0x7feb80099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 0x7feb80099290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0098a0 tx=0x7feb8c012040 comp rx=0 tx=0).stop 2022-07-03T10:35:10.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 msgr2=0x7feb9810e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:10.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 0x7feb9810e0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb8800e820 tx=0x7feb88005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 shutdown_connections 2022-07-03T10:35:10.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feb80096de0 0x7feb80099290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb980ff010 0x7feb98115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb980fd400 0x7feb9810e0c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 --2- 172.21.15.101:0/177320306 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb980fe680 0x7feb9810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:10.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 >> 172.21.15.101:0/177320306 conn(0x7feb980f8b70 msgr2=0x7feb980fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:10.766 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.755+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 shutdown_connections 2022-07-03T10:35:10.766 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:10.756+0000 7feb9759e700 1 -- 172.21.15.101:0/177320306 wait complete. 2022-07-03T10:35:10.766 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:35:10.784 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:35:11.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.008+0000 7f61739ff700 1 -- 172.21.15.101:0/3279533897 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c105570 msgr2=0x7f616c107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.008+0000 7f61739ff700 1 --2- 172.21.15.101:0/3279533897 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c105570 0x7f616c107950 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6164009ca0 tx=0x7f6164009520 comp rx=0 tx=0).stop 2022-07-03T10:35:11.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 -- 172.21.15.101:0/3279533897 shutdown_connections 2022-07-03T10:35:11.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 --2- 172.21.15.101:0/3279533897 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c105570 0x7f616c107950 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 --2- 172.21.15.101:0/3279533897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c0723b0 0x7f616c070890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 --2- 172.21.15.101:0/3279533897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c070dd0 0x7f616c071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 -- 172.21.15.101:0/3279533897 >> 172.21.15.101:0/3279533897 conn(0x7f616c0f8d50 msgr2=0x7f616c0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:11.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 -- 172.21.15.101:0/3279533897 shutdown_connections 2022-07-03T10:35:11.013 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.009+0000 7f61739ff700 1 -- 172.21.15.101:0/3279533897 wait complete. 2022-07-03T10:35:11.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 Processor -- start 2022-07-03T10:35:11.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 -- start start 2022-07-03T10:35:11.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 0x7f616c110580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c105570 0x7f616c117b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.016 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1180b0 con 0x7f616c0723b0 2022-07-03T10:35:11.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1181f0 con 0x7f616c070dd0 2022-07-03T10:35:11.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f61739ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1183b0 con 0x7f616c105570 2022-07-03T10:35:11.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f6170f9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.017 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.010+0000 7f617179b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 0x7f616c110580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f6170f9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46202/0 (socket says 172.21.15.101:46202) 2022-07-03T10:35:11.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f6170f9a700 1 -- 172.21.15.101:0/4015357681 learned_addr learned my addr 172.21.15.101:0/4015357681 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:11.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f6171f9c700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c105570 0x7f616c117b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 -- 172.21.15.101:0/4015357681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c105570 msgr2=0x7f616c117b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c105570 0x7f616c117b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 -- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 msgr2=0x7f616c110ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 -- 172.21.15.101:0/4015357681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640097e0 con 0x7f616c070dd0 2022-07-03T10:35:11.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f6170f9a700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:11.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.011+0000 7f617179b700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 0x7f616c110580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f616800e7d0 tx=0x7f6168005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.012+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168010710 con 0x7f616c070dd0 2022-07-03T10:35:11.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.012+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c118690 con 0x7f616c070dd0 2022-07-03T10:35:11.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.012+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6168010d20 con 0x7f616c070dd0 2022-07-03T10:35:11.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.012+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168018dd0 con 0x7f616c070dd0 2022-07-03T10:35:11.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.012+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c118c20 con 0x7f616c070dd0 2022-07-03T10:35:11.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.013+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c10afc0 con 0x7f616c070dd0 2022-07-03T10:35:11.022 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.015+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f616800c040 con 0x7f616c070dd0 2022-07-03T10:35:11.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.016+0000 7f6161ffb700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 0x7f6158099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.016+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f6168014070 con 0x7f616c070dd0 2022-07-03T10:35:11.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.016+0000 7f6170f9a700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 0x7f6158099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.023 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.017+0000 7f6170f9a700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 0x7f6158099210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f615c011640 tx=0x7f615c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.020+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f616804f100 con 0x7f616c070dd0 2022-07-03T10:35:11.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.209+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f616c10b150 con 0x7f616c070dd0 2022-07-03T10:35:11.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.211+0000 7f6161ffb700 1 -- 172.21.15.101:0/4015357681 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21420 (secure 0 0 0) 0x7f616c10b150 con 0x7f616c070dd0 2022-07-03T10:35:11.213 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:11.214 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":80,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:04.429266+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:35:11.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 msgr2=0x7f6158099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 0x7f6158099210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f615c011640 tx=0x7f615c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 msgr2=0x7f616c110580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 0x7f616c110580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f616800e7d0 tx=0x7f6168005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 shutdown_connections 2022-07-03T10:35:11.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6158096d60 0x7f6158099210 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f616c105570 0x7f616c117b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f616c070dd0 0x7f616c110580 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 --2- 172.21.15.101:0/4015357681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f616c0723b0 0x7f616c110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 >> 172.21.15.101:0/4015357681 conn(0x7f616c0f8d50 msgr2=0x7f616c1076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:11.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 shutdown_connections 2022-07-03T10:35:11.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.213+0000 7f61739ff700 1 -- 172.21.15.101:0/4015357681 wait complete. 2022-07-03T10:35:11.244 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-07-03T10:35:11.245 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-03T10:35:11.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.469+0000 7f40f119d700 1 -- 172.21.15.101:0/2649119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0fd3e0 msgr2=0x7f40ec0fd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.469+0000 7f40f119d700 1 --2- 172.21.15.101:0/2649119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec0fd7f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f40dc009900 tx=0x7f40dc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 -- 172.21.15.101:0/2649119510 shutdown_connections 2022-07-03T10:35:11.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 --2- 172.21.15.101:0/2649119510 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0feff0 0x7f40ec1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 --2- 172.21.15.101:0/2649119510 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec0fd7f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 --2- 172.21.15.101:0/2649119510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec0feab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 -- 172.21.15.101:0/2649119510 >> 172.21.15.101:0/2649119510 conn(0x7f40ec0f8b30 msgr2=0x7f40ec0fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:11.474 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 -- 172.21.15.101:0/2649119510 shutdown_connections 2022-07-03T10:35:11.475 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.470+0000 7f40f119d700 1 -- 172.21.15.101:0/2649119510 wait complete. 2022-07-03T10:35:11.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.471+0000 7f40f119d700 1 Processor -- start 2022-07-03T10:35:11.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.471+0000 7f40f119d700 1 -- start start 2022-07-03T10:35:11.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec10e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 0x7f40ec1159a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec115f20 con 0x7f40ec0fd3e0 2022-07-03T10:35:11.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec116060 con 0x7f40ec0fe660 2022-07-03T10:35:11.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f119d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec116220 con 0x7f40ec0feff0 2022-07-03T10:35:11.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40eb7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40eb7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:60116/0 (socket says 172.21.15.101:60116) 2022-07-03T10:35:11.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40eb7fe700 1 -- 172.21.15.101:0/758204793 learned_addr learned my addr 172.21.15.101:0/758204793 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:11.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40ebfff700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec10e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.472+0000 7f40f099c700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 0x7f40ec1159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 -- 172.21.15.101:0/758204793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 msgr2=0x7f40ec1159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 0x7f40ec1159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 -- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0fd3e0 msgr2=0x7f40ec10e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec10e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 -- 172.21.15.101:0/758204793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4009700 con 0x7f40ec0fe660 2022-07-03T10:35:11.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40f099c700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 0x7f40ec1159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:35:11.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40eb7fe700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f40e000ea80 tx=0x7f40e0005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0009e30 con 0x7f40ec0fe660 2022-07-03T10:35:11.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc009440 con 0x7f40ec0fe660 2022-07-03T10:35:11.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f40e000ce80 con 0x7f40ec0fe660 2022-07-03T10:35:11.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0004ac0 con 0x7f40ec0fe660 2022-07-03T10:35:11.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.473+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec116900 con 0x7f40ec0fe660 2022-07-03T10:35:11.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.474+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec108bb0 con 0x7f40ec0fe660 2022-07-03T10:35:11.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.476+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f40e000c040 con 0x7f40ec0fe660 2022-07-03T10:35:11.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.477+0000 7f40e8ff9700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 0x7f40d4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.477+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f40e0014070 con 0x7f40ec0fe660 2022-07-03T10:35:11.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.478+0000 7f40ebfff700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 0x7f40d4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.478+0000 7f40ebfff700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 0x7f40d4099210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0097d0 tx=0x7f40dc0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.480+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f40e008d1b0 con 0x7f40ec0fe660 2022-07-03T10:35:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: pgmap v272: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2022-07-03T10:35:11.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: mgrmap e17: x(active, since 6m), standbys: y, z 2022-07-03T10:35:11.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/177320306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.495 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:11 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4015357681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: pgmap v272: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2022-07-03T10:35:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: mgrmap e17: x(active, since 6m), standbys: y, z 2022-07-03T10:35:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/177320306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:11 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4015357681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: pgmap v272: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6.8 KiB/s wr, 9 op/s 2022-07-03T10:35:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: mgrmap e17: x(active, since 6m), standbys: y, z 2022-07-03T10:35:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:35:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/177320306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:11 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4015357681' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:11.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.665+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40ec05e960 con 0x7f40d4096d60 2022-07-03T10:35:11.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.678+0000 7f40e8ff9700 1 -- 172.21.15.101:0/758204793 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+334085 (secure 0 0 0) 0x7f40ec05e960 con 0x7f40d4096d60 2022-07-03T10:35:11.686 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:11.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 msgr2=0x7f40d4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 0x7f40d4099210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0097d0 tx=0x7f40dc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 msgr2=0x7f40ec10e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f40e000ea80 tx=0x7f40e0005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 shutdown_connections 2022-07-03T10:35:11.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f40d4096d60 0x7f40d4099210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40ec0fd3e0 0x7f40ec10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40ec0feff0 0x7f40ec1159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 --2- 172.21.15.101:0/758204793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40ec0fe660 0x7f40ec10e930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 >> 172.21.15.101:0/758204793 conn(0x7f40ec0f8b30 msgr2=0x7f40ec101890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:11.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 shutdown_connections 2022-07-03T10:35:11.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.683+0000 7f40f119d700 1 -- 172.21.15.101:0/758204793 wait complete. 2022-07-03T10:35:11.694 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:35:11.712 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2022-07-03T10:35:09.933594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624668,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":141,"num_read_kb":137,"num_write":236,"num_write_kb":2986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":76464,"kb_used_data":10384,"kb_used_omap":0,"kb_used_meta":65984,"kb_avail":1124865360,"statfs":{"total":1151940427776,"available":1151862128640,"internally_reserved":0,"allocated":10633216,"data_stored":6827782,"data_compressed":107352,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":67567616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34300,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":55,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.084932"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461438+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:35:03.461438+0000","last_peered":"2022-07-03T10:35:03.461438+0000","last_clean":"2022-07-03T10:35:03.461438+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:35:03.461438+0000","last_undegraded":"2022-07-03T10:35:03.461438+0000","last_fullsized":"2022-07-03T10:35:03.461438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472013+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:35:03.472013+0000","last_peered":"2022-07-03T10:35:03.472013+0000","last_clean":"2022-07-03T10:35:03.472013+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:35:03.472013+0000","last_undegraded":"2022-07-03T10:35:03.472013+0000","last_fullsized":"2022-07-03T10:35:03.472013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.469723+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:35:03.469723+0000","last_peered":"2022-07-03T10:35:03.469723+0000","last_clean":"2022-07-03T10:35:03.469723+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:35:03.469723+0000","last_undegraded":"2022-07-03T10:35:03.469723+0000","last_fullsized":"2022-07-03T10:35:03.469723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459886+0000","last_change":"2022-07-03T10:35:03.459299+0000","last_active":"2022-07-03T10:35:03.459886+0000","last_peered":"2022-07-03T10:35:03.459886+0000","last_clean":"2022-07-03T10:35:03.459886+0000","last_became_active":"2022-07-03T10:35:03.458602+0000","last_became_peered":"2022-07-03T10:35:03.458602+0000","last_unstale":"2022-07-03T10:35:03.459886+0000","last_undegraded":"2022-07-03T10:35:03.459886+0000","last_fullsized":"2022-07-03T10:35:03.459886+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:02:23.899940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.466024+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:35:03.466024+0000","last_peered":"2022-07-03T10:35:03.466024+0000","last_clean":"2022-07-03T10:35:03.466024+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:35:03.466024+0000","last_undegraded":"2022-07-03T10:35:03.466024+0000","last_fullsized":"2022-07-03T10:35:03.466024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453962+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:35:03.453962+0000","last_peered":"2022-07-03T10:35:03.453962+0000","last_clean":"2022-07-03T10:35:03.453962+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:35:03.453962+0000","last_undegraded":"2022-07-03T10:35:03.453962+0000","last_fullsized":"2022-07-03T10:35:03.453962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455141+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:35:03.455141+0000","last_peered":"2022-07-03T10:35:03.455141+0000","last_clean":"2022-07-03T10:35:03.455141+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:35:03.455141+0000","last_undegraded":"2022-07-03T10:35:03.455141+0000","last_fullsized":"2022-07-03T10:35:03.455141+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443794+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:35:03.443794+0000","last_peered":"2022-07-03T10:35:03.443794+0000","last_clean":"2022-07-03T10:35:03.443794+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:35:03.443794+0000","last_undegraded":"2022-07-03T10:35:03.443794+0000","last_fullsized":"2022-07-03T10:35:03.443794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452820+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:35:03.452820+0000","last_peered":"2022-07-03T10:35:03.452820+0000","last_clean":"2022-07-03T10:35:03.452820+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:35:03.452820+0000","last_undegraded":"2022-07-03T10:35:03.452820+0000","last_fullsized":"2022-07-03T10:35:03.452820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455453+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:35:03.455453+0000","last_peered":"2022-07-03T10:35:03.455453+0000","last_clean":"2022-07-03T10:35:03.455453+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:35:03.455453+0000","last_undegraded":"2022-07-03T10:35:03.455453+0000","last_fullsized":"2022-07-03T10:35:03.455453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.456999+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:35:03.456999+0000","last_peered":"2022-07-03T10:35:03.456999+0000","last_clean":"2022-07-03T10:35:03.456999+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:35:03.456999+0000","last_undegraded":"2022-07-03T10:35:03.456999+0000","last_fullsized":"2022-07-03T10:35:03.456999+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459417+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:35:03.459417+0000","last_peered":"2022-07-03T10:35:03.459417+0000","last_clean":"2022-07-03T10:35:03.459417+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:03.459417+0000","last_undegraded":"2022-07-03T10:35:03.459417+0000","last_fullsized":"2022-07-03T10:35:03.459417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443495+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:35:03.443495+0000","last_peered":"2022-07-03T10:35:03.443495+0000","last_clean":"2022-07-03T10:35:03.443495+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:35:03.443495+0000","last_undegraded":"2022-07-03T10:35:03.443495+0000","last_fullsized":"2022-07-03T10:35:03.443495+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470143+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:35:03.470143+0000","last_peered":"2022-07-03T10:35:03.470143+0000","last_clean":"2022-07-03T10:35:03.470143+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:35:03.470143+0000","last_undegraded":"2022-07-03T10:35:03.470143+0000","last_fullsized":"2022-07-03T10:35:03.470143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471960+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:35:03.471960+0000","last_peered":"2022-07-03T10:35:03.471960+0000","last_clean":"2022-07-03T10:35:03.471960+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:35:03.471960+0000","last_undegraded":"2022-07-03T10:35:03.471960+0000","last_fullsized":"2022-07-03T10:35:03.471960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461440+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:35:03.461440+0000","last_peered":"2022-07-03T10:35:03.461440+0000","last_clean":"2022-07-03T10:35:03.461440+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:35:03.461440+0000","last_undegraded":"2022-07-03T10:35:03.461440+0000","last_fullsized":"2022-07-03T10:35:03.461440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.462784+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:35:03.462784+0000","last_peered":"2022-07-03T10:35:03.462784+0000","last_clean":"2022-07-03T10:35:03.462784+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:35:03.462784+0000","last_undegraded":"2022-07-03T10:35:03.462784+0000","last_fullsized":"2022-07-03T10:35:03.462784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452828+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:35:03.452828+0000","last_peered":"2022-07-03T10:35:03.452828+0000","last_clean":"2022-07-03T10:35:03.452828+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:35:03.452828+0000","last_undegraded":"2022-07-03T10:35:03.452828+0000","last_fullsized":"2022-07-03T10:35:03.452828+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472689+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:35:03.472689+0000","last_peered":"2022-07-03T10:35:03.472689+0000","last_clean":"2022-07-03T10:35:03.472689+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:35:03.472689+0000","last_undegraded":"2022-07-03T10:35:03.472689+0000","last_fullsized":"2022-07-03T10:35:03.472689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.482646+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:04.482646+0000","last_peered":"2022-07-03T10:35:04.482646+0000","last_clean":"2022-07-03T10:35:04.482646+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:04.482646+0000","last_undegraded":"2022-07-03T10:35:04.482646+0000","last_fullsized":"2022-07-03T10:35:04.482646+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464595+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:35:03.464595+0000","last_peered":"2022-07-03T10:35:03.464595+0000","last_clean":"2022-07-03T10:35:03.464595+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:35:03.464595+0000","last_undegraded":"2022-07-03T10:35:03.464595+0000","last_fullsized":"2022-07-03T10:35:03.464595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.463832+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:03.463832+0000","last_peered":"2022-07-03T10:35:03.463832+0000","last_clean":"2022-07-03T10:35:03.463832+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:03.463832+0000","last_undegraded":"2022-07-03T10:35:03.463832+0000","last_fullsized":"2022-07-03T10:35:03.463832+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472286+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:35:03.472286+0000","last_peered":"2022-07-03T10:35:03.472286+0000","last_clean":"2022-07-03T10:35:03.472286+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:35:03.472286+0000","last_undegraded":"2022-07-03T10:35:03.472286+0000","last_fullsized":"2022-07-03T10:35:03.472286+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460634+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:03.460634+0000","last_peered":"2022-07-03T10:35:03.460634+0000","last_clean":"2022-07-03T10:35:03.460634+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:03.460634+0000","last_undegraded":"2022-07-03T10:35:03.460634+0000","last_fullsized":"2022-07-03T10:35:03.460634+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.454988+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:35:03.454988+0000","last_peered":"2022-07-03T10:35:03.454988+0000","last_clean":"2022-07-03T10:35:03.454988+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:35:03.454988+0000","last_undegraded":"2022-07-03T10:35:03.454988+0000","last_fullsized":"2022-07-03T10:35:03.454988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.454239+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:03.454239+0000","last_peered":"2022-07-03T10:35:03.454239+0000","last_clean":"2022-07-03T10:35:03.454239+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:03.454239+0000","last_undegraded":"2022-07-03T10:35:03.454239+0000","last_fullsized":"2022-07-03T10:35:03.454239+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451613+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:35:03.451613+0000","last_peered":"2022-07-03T10:35:03.451613+0000","last_clean":"2022-07-03T10:35:03.451613+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:35:03.451613+0000","last_undegraded":"2022-07-03T10:35:03.451613+0000","last_fullsized":"2022-07-03T10:35:03.451613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449401+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:03.449401+0000","last_peered":"2022-07-03T10:35:03.449401+0000","last_clean":"2022-07-03T10:35:03.449401+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:03.449401+0000","last_undegraded":"2022-07-03T10:35:03.449401+0000","last_fullsized":"2022-07-03T10:35:03.449401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.456790+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:35:03.456790+0000","last_peered":"2022-07-03T10:35:03.456790+0000","last_clean":"2022-07-03T10:35:03.456790+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:35:03.456790+0000","last_undegraded":"2022-07-03T10:35:03.456790+0000","last_fullsized":"2022-07-03T10:35:03.456790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451111+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:03.451111+0000","last_peered":"2022-07-03T10:35:03.451111+0000","last_clean":"2022-07-03T10:35:03.451111+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:03.451111+0000","last_undegraded":"2022-07-03T10:35:03.451111+0000","last_fullsized":"2022-07-03T10:35:03.451111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449346+0000","last_change":"2022-07-03T10:35:03.448987+0000","last_active":"2022-07-03T10:35:03.449346+0000","last_peered":"2022-07-03T10:35:03.449346+0000","last_clean":"2022-07-03T10:35:03.449346+0000","last_became_active":"2022-07-03T10:35:03.448669+0000","last_became_peered":"2022-07-03T10:35:03.448669+0000","last_unstale":"2022-07-03T10:35:03.449346+0000","last_undegraded":"2022-07-03T10:35:03.449346+0000","last_fullsized":"2022-07-03T10:35:03.449346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:32:30.164547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.434059+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:03.434059+0000","last_peered":"2022-07-03T10:35:03.434059+0000","last_clean":"2022-07-03T10:35:03.434059+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:03.434059+0000","last_undegraded":"2022-07-03T10:35:03.434059+0000","last_fullsized":"2022-07-03T10:35:03.434059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.458898+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:35:03.458898+0000","last_peered":"2022-07-03T10:35:03.458898+0000","last_clean":"2022-07-03T10:35:03.458898+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:35:03.458898+0000","last_undegraded":"2022-07-03T10:35:03.458898+0000","last_fullsized":"2022-07-03T10:35:03.458898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443557+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:03.443557+0000","last_peered":"2022-07-03T10:35:03.443557+0000","last_clean":"2022-07-03T10:35:03.443557+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:03.443557+0000","last_undegraded":"2022-07-03T10:35:03.443557+0000","last_fullsized":"2022-07-03T10:35:03.443557+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464856+0000","last_change":"2022-07-03T10:35:03.462976+0000","last_active":"2022-07-03T10:35:03.464856+0000","last_peered":"2022-07-03T10:35:03.464856+0000","last_clean":"2022-07-03T10:35:03.464856+0000","last_became_active":"2022-07-03T10:35:03.462590+0000","last_became_peered":"2022-07-03T10:35:03.462590+0000","last_unstale":"2022-07-03T10:35:03.464856+0000","last_undegraded":"2022-07-03T10:35:03.464856+0000","last_fullsized":"2022-07-03T10:35:03.464856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:51:17.814072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.431704+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:03.431704+0000","last_peered":"2022-07-03T10:35:03.431704+0000","last_clean":"2022-07-03T10:35:03.431704+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:03.431704+0000","last_undegraded":"2022-07-03T10:35:03.431704+0000","last_fullsized":"2022-07-03T10:35:03.431704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451820+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:35:03.451820+0000","last_peered":"2022-07-03T10:35:03.451820+0000","last_clean":"2022-07-03T10:35:03.451820+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:35:03.451820+0000","last_undegraded":"2022-07-03T10:35:03.451820+0000","last_fullsized":"2022-07-03T10:35:03.451820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438569+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:03.438569+0000","last_peered":"2022-07-03T10:35:03.438569+0000","last_clean":"2022-07-03T10:35:03.438569+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:03.438569+0000","last_undegraded":"2022-07-03T10:35:03.438569+0000","last_fullsized":"2022-07-03T10:35:03.438569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470197+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:35:03.470197+0000","last_peered":"2022-07-03T10:35:03.470197+0000","last_clean":"2022-07-03T10:35:03.470197+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:35:03.470197+0000","last_undegraded":"2022-07-03T10:35:03.470197+0000","last_fullsized":"2022-07-03T10:35:03.470197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471231+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:03.471231+0000","last_peered":"2022-07-03T10:35:03.471231+0000","last_clean":"2022-07-03T10:35:03.471231+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:03.471231+0000","last_undegraded":"2022-07-03T10:35:03.471231+0000","last_fullsized":"2022-07-03T10:35:03.471231+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459058+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:35:03.459058+0000","last_peered":"2022-07-03T10:35:03.459058+0000","last_clean":"2022-07-03T10:35:03.459058+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:03.459058+0000","last_undegraded":"2022-07-03T10:35:03.459058+0000","last_fullsized":"2022-07-03T10:35:03.459058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'16","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:09.747061+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:35:09.747061+0000","last_peered":"2022-07-03T10:35:09.747061+0000","last_clean":"2022-07-03T10:35:09.747061+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:35:09.747061+0000","last_undegraded":"2022-07-03T10:35:09.747061+0000","last_fullsized":"2022-07-03T10:35:09.747061+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33395,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":22,"num_write":16,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455885+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:35:03.455885+0000","last_peered":"2022-07-03T10:35:03.455885+0000","last_clean":"2022-07-03T10:35:03.455885+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:35:03.455885+0000","last_undegraded":"2022-07-03T10:35:03.455885+0000","last_fullsized":"2022-07-03T10:35:03.455885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.482073+0000","last_change":"2022-07-03T10:35:02.869468+0000","last_active":"2022-07-03T10:35:04.482073+0000","last_peered":"2022-07-03T10:35:04.482073+0000","last_clean":"2022-07-03T10:35:04.482073+0000","last_became_active":"2022-07-03T10:35:02.869069+0000","last_became_peered":"2022-07-03T10:35:02.869069+0000","last_unstale":"2022-07-03T10:35:04.482073+0000","last_undegraded":"2022-07-03T10:35:04.482073+0000","last_fullsized":"2022-07-03T10:35:04.482073+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:43:52.762318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452376+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:35:03.452376+0000","last_peered":"2022-07-03T10:35:03.452376+0000","last_clean":"2022-07-03T10:35:03.452376+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:35:03.452376+0000","last_undegraded":"2022-07-03T10:35:03.452376+0000","last_fullsized":"2022-07-03T10:35:03.452376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443336+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:03.443336+0000","last_peered":"2022-07-03T10:35:03.443336+0000","last_clean":"2022-07-03T10:35:03.443336+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:03.443336+0000","last_undegraded":"2022-07-03T10:35:03.443336+0000","last_fullsized":"2022-07-03T10:35:03.443336+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449611+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:35:03.449611+0000","last_peered":"2022-07-03T10:35:03.449611+0000","last_clean":"2022-07-03T10:35:03.449611+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:35:03.449611+0000","last_undegraded":"2022-07-03T10:35:03.449611+0000","last_fullsized":"2022-07-03T10:35:03.449611+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438228+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:03.438228+0000","last_peered":"2022-07-03T10:35:03.438228+0000","last_clean":"2022-07-03T10:35:03.438228+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:03.438228+0000","last_undegraded":"2022-07-03T10:35:03.438228+0000","last_fullsized":"2022-07-03T10:35:03.438228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453137+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:35:03.453137+0000","last_peered":"2022-07-03T10:35:03.453137+0000","last_clean":"2022-07-03T10:35:03.453137+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:35:03.453137+0000","last_undegraded":"2022-07-03T10:35:03.453137+0000","last_fullsized":"2022-07-03T10:35:03.453137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472850+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:03.472850+0000","last_peered":"2022-07-03T10:35:03.472850+0000","last_clean":"2022-07-03T10:35:03.472850+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:03.472850+0000","last_undegraded":"2022-07-03T10:35:03.472850+0000","last_fullsized":"2022-07-03T10:35:03.472850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449113+0000","last_change":"2022-07-03T10:35:03.448350+0000","last_active":"2022-07-03T10:35:03.449113+0000","last_peered":"2022-07-03T10:35:03.449113+0000","last_clean":"2022-07-03T10:35:03.449113+0000","last_became_active":"2022-07-03T10:35:03.448015+0000","last_became_peered":"2022-07-03T10:35:03.448015+0000","last_unstale":"2022-07-03T10:35:03.449113+0000","last_undegraded":"2022-07-03T10:35:03.449113+0000","last_fullsized":"2022-07-03T10:35:03.449113+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:52:58.951227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.483163+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:04.483163+0000","last_peered":"2022-07-03T10:35:04.483163+0000","last_clean":"2022-07-03T10:35:04.483163+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:04.483163+0000","last_undegraded":"2022-07-03T10:35:04.483163+0000","last_fullsized":"2022-07-03T10:35:04.483163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451468+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:35:03.451468+0000","last_peered":"2022-07-03T10:35:03.451468+0000","last_clean":"2022-07-03T10:35:03.451468+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:35:03.451468+0000","last_undegraded":"2022-07-03T10:35:03.451468+0000","last_fullsized":"2022-07-03T10:35:03.451468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.446946+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:03.446946+0000","last_peered":"2022-07-03T10:35:03.446946+0000","last_clean":"2022-07-03T10:35:03.446946+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:03.446946+0000","last_undegraded":"2022-07-03T10:35:03.446946+0000","last_fullsized":"2022-07-03T10:35:03.446946+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.432833+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:35:04.432833+0000","last_peered":"2022-07-03T10:35:04.432833+0000","last_clean":"2022-07-03T10:35:04.432833+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:35:04.432833+0000","last_undegraded":"2022-07-03T10:35:04.432833+0000","last_fullsized":"2022-07-03T10:35:04.432833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.440467+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:03.440467+0000","last_peered":"2022-07-03T10:35:03.440467+0000","last_clean":"2022-07-03T10:35:03.440467+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:03.440467+0000","last_undegraded":"2022-07-03T10:35:03.440467+0000","last_fullsized":"2022-07-03T10:35:03.440467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459439+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:35:03.459439+0000","last_peered":"2022-07-03T10:35:03.459439+0000","last_clean":"2022-07-03T10:35:03.459439+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:35:03.459439+0000","last_undegraded":"2022-07-03T10:35:03.459439+0000","last_fullsized":"2022-07-03T10:35:03.459439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.448258+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:35:03.448258+0000","last_peered":"2022-07-03T10:35:03.448258+0000","last_clean":"2022-07-03T10:35:03.448258+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:35:03.448258+0000","last_undegraded":"2022-07-03T10:35:03.448258+0000","last_fullsized":"2022-07-03T10:35:03.448258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.433956+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:03.433956+0000","last_peered":"2022-07-03T10:35:03.433956+0000","last_clean":"2022-07-03T10:35:03.433956+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:03.433956+0000","last_undegraded":"2022-07-03T10:35:03.433956+0000","last_fullsized":"2022-07-03T10:35:03.433956+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457006+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:35:03.457006+0000","last_peered":"2022-07-03T10:35:03.457006+0000","last_clean":"2022-07-03T10:35:03.457006+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:35:03.457006+0000","last_undegraded":"2022-07-03T10:35:03.457006+0000","last_fullsized":"2022-07-03T10:35:03.457006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.439376+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:03.439376+0000","last_peered":"2022-07-03T10:35:03.439376+0000","last_clean":"2022-07-03T10:35:03.439376+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:03.439376+0000","last_undegraded":"2022-07-03T10:35:03.439376+0000","last_fullsized":"2022-07-03T10:35:03.439376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438794+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:03.438794+0000","last_peered":"2022-07-03T10:35:03.438794+0000","last_clean":"2022-07-03T10:35:03.438794+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:03.438794+0000","last_undegraded":"2022-07-03T10:35:03.438794+0000","last_fullsized":"2022-07-03T10:35:03.438794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470153+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:35:03.470153+0000","last_peered":"2022-07-03T10:35:03.470153+0000","last_clean":"2022-07-03T10:35:03.470153+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:35:03.470153+0000","last_undegraded":"2022-07-03T10:35:03.470153+0000","last_fullsized":"2022-07-03T10:35:03.470153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438916+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:03.438916+0000","last_peered":"2022-07-03T10:35:03.438916+0000","last_clean":"2022-07-03T10:35:03.438916+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:03.438916+0000","last_undegraded":"2022-07-03T10:35:03.438916+0000","last_fullsized":"2022-07-03T10:35:03.438916+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438900+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:03.438900+0000","last_peered":"2022-07-03T10:35:03.438900+0000","last_clean":"2022-07-03T10:35:03.438900+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:03.438900+0000","last_undegraded":"2022-07-03T10:35:03.438900+0000","last_fullsized":"2022-07-03T10:35:03.438900+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470304+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:35:03.470304+0000","last_peered":"2022-07-03T10:35:03.470304+0000","last_clean":"2022-07-03T10:35:03.470304+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:35:03.470304+0000","last_undegraded":"2022-07-03T10:35:03.470304+0000","last_fullsized":"2022-07-03T10:35:03.470304+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.444191+0000","last_change":"2022-07-03T10:35:02.876478+0000","last_active":"2022-07-03T10:35:03.444191+0000","last_peered":"2022-07-03T10:35:03.444191+0000","last_clean":"2022-07-03T10:35:03.444191+0000","last_became_active":"2022-07-03T10:35:02.876049+0000","last_became_peered":"2022-07-03T10:35:02.876049+0000","last_unstale":"2022-07-03T10:35:03.444191+0000","last_undegraded":"2022-07-03T10:35:03.444191+0000","last_fullsized":"2022-07-03T10:35:03.444191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:45.607526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.458929+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:35:03.458929+0000","last_peered":"2022-07-03T10:35:03.458929+0000","last_clean":"2022-07-03T10:35:03.458929+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:35:03.458929+0000","last_undegraded":"2022-07-03T10:35:03.458929+0000","last_fullsized":"2022-07-03T10:35:03.458929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452526+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:03.452526+0000","last_peered":"2022-07-03T10:35:03.452526+0000","last_clean":"2022-07-03T10:35:03.452526+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:03.452526+0000","last_undegraded":"2022-07-03T10:35:03.452526+0000","last_fullsized":"2022-07-03T10:35:03.452526+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471996+0000","last_change":"2022-07-03T10:35:03.470001+0000","last_active":"2022-07-03T10:35:03.471996+0000","last_peered":"2022-07-03T10:35:03.471996+0000","last_clean":"2022-07-03T10:35:03.471996+0000","last_became_active":"2022-07-03T10:35:03.461137+0000","last_became_peered":"2022-07-03T10:35:03.461137+0000","last_unstale":"2022-07-03T10:35:03.471996+0000","last_undegraded":"2022-07-03T10:35:03.471996+0000","last_fullsized":"2022-07-03T10:35:03.471996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:45:25.469289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.442789+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:03.442789+0000","last_peered":"2022-07-03T10:35:03.442789+0000","last_clean":"2022-07-03T10:35:03.442789+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:03.442789+0000","last_undegraded":"2022-07-03T10:35:03.442789+0000","last_fullsized":"2022-07-03T10:35:03.442789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455870+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:35:03.455870+0000","last_peered":"2022-07-03T10:35:03.455870+0000","last_clean":"2022-07-03T10:35:03.455870+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:35:03.455870+0000","last_undegraded":"2022-07-03T10:35:03.455870+0000","last_fullsized":"2022-07-03T10:35:03.455870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.432615+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:03.432615+0000","last_peered":"2022-07-03T10:35:03.432615+0000","last_clean":"2022-07-03T10:35:03.432615+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:03.432615+0000","last_undegraded":"2022-07-03T10:35:03.432615+0000","last_fullsized":"2022-07-03T10:35:03.432615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461191+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:35:03.461191+0000","last_peered":"2022-07-03T10:35:03.461191+0000","last_clean":"2022-07-03T10:35:03.461191+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:35:03.461191+0000","last_undegraded":"2022-07-03T10:35:03.461191+0000","last_fullsized":"2022-07-03T10:35:03.461191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453026+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:03.453026+0000","last_peered":"2022-07-03T10:35:03.453026+0000","last_clean":"2022-07-03T10:35:03.453026+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:03.453026+0000","last_undegraded":"2022-07-03T10:35:03.453026+0000","last_fullsized":"2022-07-03T10:35:03.453026+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.465427+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:35:03.465427+0000","last_peered":"2022-07-03T10:35:03.465427+0000","last_clean":"2022-07-03T10:35:03.465427+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:35:03.465427+0000","last_undegraded":"2022-07-03T10:35:03.465427+0000","last_fullsized":"2022-07-03T10:35:03.465427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.483047+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:04.483047+0000","last_peered":"2022-07-03T10:35:04.483047+0000","last_clean":"2022-07-03T10:35:04.483047+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:04.483047+0000","last_undegraded":"2022-07-03T10:35:04.483047+0000","last_fullsized":"2022-07-03T10:35:04.483047+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451094+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:35:03.451094+0000","last_peered":"2022-07-03T10:35:03.451094+0000","last_clean":"2022-07-03T10:35:03.451094+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:35:03.451094+0000","last_undegraded":"2022-07-03T10:35:03.451094+0000","last_fullsized":"2022-07-03T10:35:03.451094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455489+0000","last_change":"2022-07-03T10:35:02.879933+0000","last_active":"2022-07-03T10:35:03.455489+0000","last_peered":"2022-07-03T10:35:03.455489+0000","last_clean":"2022-07-03T10:35:03.455489+0000","last_became_active":"2022-07-03T10:35:02.879537+0000","last_became_peered":"2022-07-03T10:35:02.879537+0000","last_unstale":"2022-07-03T10:35:03.455489+0000","last_undegraded":"2022-07-03T10:35:03.455489+0000","last_fullsized":"2022-07-03T10:35:03.455489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:04:10.041882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455333+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:35:03.455333+0000","last_peered":"2022-07-03T10:35:03.455333+0000","last_clean":"2022-07-03T10:35:03.455333+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:35:03.455333+0000","last_undegraded":"2022-07-03T10:35:03.455333+0000","last_fullsized":"2022-07-03T10:35:03.455333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.481718+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:04.481718+0000","last_peered":"2022-07-03T10:35:04.481718+0000","last_clean":"2022-07-03T10:35:04.481718+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:04.481718+0000","last_undegraded":"2022-07-03T10:35:04.481718+0000","last_fullsized":"2022-07-03T10:35:04.481718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457937+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:35:03.457937+0000","last_peered":"2022-07-03T10:35:03.457937+0000","last_clean":"2022-07-03T10:35:03.457937+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:35:03.457937+0000","last_undegraded":"2022-07-03T10:35:03.457937+0000","last_fullsized":"2022-07-03T10:35:03.457937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.466452+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:03.466452+0000","last_peered":"2022-07-03T10:35:03.466452+0000","last_clean":"2022-07-03T10:35:03.466452+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:03.466452+0000","last_undegraded":"2022-07-03T10:35:03.466452+0000","last_fullsized":"2022-07-03T10:35:03.466452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457040+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:35:03.457040+0000","last_peered":"2022-07-03T10:35:03.457040+0000","last_clean":"2022-07-03T10:35:03.457040+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:35:03.457040+0000","last_undegraded":"2022-07-03T10:35:03.457040+0000","last_fullsized":"2022-07-03T10:35:03.457040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451455+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:35:03.451455+0000","last_peered":"2022-07-03T10:35:03.451455+0000","last_clean":"2022-07-03T10:35:03.451455+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:35:03.451455+0000","last_undegraded":"2022-07-03T10:35:03.451455+0000","last_fullsized":"2022-07-03T10:35:03.451455+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459361+0000","last_change":"2022-07-03T10:35:03.455645+0000","last_active":"2022-07-03T10:35:03.459361+0000","last_peered":"2022-07-03T10:35:03.459361+0000","last_clean":"2022-07-03T10:35:03.459361+0000","last_became_active":"2022-07-03T10:35:03.455338+0000","last_became_peered":"2022-07-03T10:35:03.455338+0000","last_unstale":"2022-07-03T10:35:03.459361+0000","last_undegraded":"2022-07-03T10:35:03.459361+0000","last_fullsized":"2022-07-03T10:35:03.459361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:45:15.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460610+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:35:03.460610+0000","last_peered":"2022-07-03T10:35:03.460610+0000","last_clean":"2022-07-03T10:35:03.460610+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:35:03.460610+0000","last_undegraded":"2022-07-03T10:35:03.460610+0000","last_fullsized":"2022-07-03T10:35:03.460610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.462640+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:35:03.462640+0000","last_peered":"2022-07-03T10:35:03.462640+0000","last_clean":"2022-07-03T10:35:03.462640+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:35:03.462640+0000","last_undegraded":"2022-07-03T10:35:03.462640+0000","last_fullsized":"2022-07-03T10:35:03.462640+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470323+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:35:03.470323+0000","last_peered":"2022-07-03T10:35:03.470323+0000","last_clean":"2022-07-03T10:35:03.470323+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:35:03.470323+0000","last_undegraded":"2022-07-03T10:35:03.470323+0000","last_fullsized":"2022-07-03T10:35:03.470323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455581+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:35:03.455581+0000","last_peered":"2022-07-03T10:35:03.455581+0000","last_clean":"2022-07-03T10:35:03.455581+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:35:03.455581+0000","last_undegraded":"2022-07-03T10:35:03.455581+0000","last_fullsized":"2022-07-03T10:35:03.455581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460129+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:35:03.460129+0000","last_peered":"2022-07-03T10:35:03.460129+0000","last_clean":"2022-07-03T10:35:03.460129+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:35:03.460129+0000","last_undegraded":"2022-07-03T10:35:03.460129+0000","last_fullsized":"2022-07-03T10:35:03.460129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464434+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:35:03.464434+0000","last_peered":"2022-07-03T10:35:03.464434+0000","last_clean":"2022-07-03T10:35:03.464434+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:35:03.464434+0000","last_undegraded":"2022-07-03T10:35:03.464434+0000","last_fullsized":"2022-07-03T10:35:03.464434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451590+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:35:03.451590+0000","last_peered":"2022-07-03T10:35:03.451590+0000","last_clean":"2022-07-03T10:35:03.451590+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:35:03.451590+0000","last_undegraded":"2022-07-03T10:35:03.451590+0000","last_fullsized":"2022-07-03T10:35:03.451590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461326+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:35:03.461326+0000","last_peered":"2022-07-03T10:35:03.461326+0000","last_clean":"2022-07-03T10:35:03.461326+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:35:03.461326+0000","last_undegraded":"2022-07-03T10:35:03.461326+0000","last_fullsized":"2022-07-03T10:35:03.461326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.463003+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:35:03.463003+0000","last_peered":"2022-07-03T10:35:03.463003+0000","last_clean":"2022-07-03T10:35:03.463003+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:35:03.463003+0000","last_undegraded":"2022-07-03T10:35:03.463003+0000","last_fullsized":"2022-07-03T10:35:03.463003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470330+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:35:03.470330+0000","last_peered":"2022-07-03T10:35:03.470330+0000","last_clean":"2022-07-03T10:35:03.470330+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:35:03.470330+0000","last_undegraded":"2022-07-03T10:35:03.470330+0000","last_fullsized":"2022-07-03T10:35:03.470330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460022+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:35:03.460022+0000","last_peered":"2022-07-03T10:35:03.460022+0000","last_clean":"2022-07-03T10:35:03.460022+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:35:03.460022+0000","last_undegraded":"2022-07-03T10:35:03.460022+0000","last_fullsized":"2022-07-03T10:35:03.460022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":34177,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":54,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":38837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547212,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5844,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739308,"statfs":{"total":95995035648,"available":95989051392,"internally_reserved":0,"allocated":864256,"data_stored":419384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":10,"up_from":67,"seq":287762808849,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":954368,"data_stored":1028574,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47899999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57199999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.78900000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.55700000000000005}]},{"osd":9,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.755},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.748}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":9,"up_from":62,"seq":266287972374,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5976,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739176,"statfs":{"total":95995035648,"available":95988916224,"internally_reserved":0,"allocated":868352,"data_stored":419950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72699999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.73499999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.54200000000000004}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.84099999999999997}]},{"osd":6,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.82099999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.77100000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":0.873},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.80100000000000005}]},{"osd":10,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.753}]}]},{"osd":8,"up_from":54,"seq":231928234011,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":872448,"data_stored":419972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46800000000000003}]},{"osd":1,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.505}]},{"osd":2,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.443}]},{"osd":3,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.61599999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995}]},{"osd":5,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.628}]},{"osd":7,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.66400000000000003}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.70399999999999996}]},{"osd":10,"last update":"Sun Jul 3 10:34:56 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.68899999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":7,"up_from":47,"seq":201863462942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":917504,"data_stored":1010318,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.29599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.436}]},{"osd":2,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.46400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.46400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.71499999999999997,"5min":0.745,"15min":0.745},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.48099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.222,"15min":1.222},"last":0.55300000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.47399999999999998,"15min":0.46000000000000002},"min":{"1min":0.36499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.38500000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.51400000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.49199999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96999999999999997,"5min":5.077,"15min":5.077},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.68000000000000005,"15min":0.66400000000000003},"min":{"1min":0.54200000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.0129999999999999,"5min":1.02,"15min":1.02},"last":0.68300000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:35:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.52600000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.623},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57799999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:34:37 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.53000000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6164,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738988,"statfs":{"total":95995035648,"available":95988723712,"internally_reserved":0,"allocated":864256,"data_stored":419384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.40999999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66200000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.95699999999999996}]},{"osd":1,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.65300000000000002,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.69199999999999995,"15min":0.68000000000000005},"min":{"1min":0.54700000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97699999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.626}]},{"osd":2,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47499999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.71599999999999997,"15min":0.70799999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.03,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.56100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68600000000000005,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98699999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71599999999999997,"15min":0.70399999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96899999999999997,"5min":1.022,"15min":1.022},"last":0.72599999999999998}]},{"osd":4,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.753,"15min":0.751},"min":{"1min":0.58599999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.016,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.79800000000000004,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.0580000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.80200000000000005,"15min":0.79300000000000004},"min":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.048,"5min":1.155,"15min":1.155},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.81599999999999995,"15min":0.80700000000000005},"min":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0920000000000001,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.66500000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:34:48 2022","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.84599999999999997,"15min":0.83599999999999997},"min":{"1min":0.68200000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1459999999999999,"5min":1.333,"15min":1.333},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.86499999999999999,"15min":0.85699999999999998},"min":{"1min":0.65700000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.093,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.71899999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:35:04 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.81799999999999995}]},{"osd":9,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.751},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.68899999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.69799999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607609,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6944,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738208,"statfs":{"total":95995035648,"available":95987924992,"internally_reserved":0,"allocated":876544,"data_stored":420932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:40 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.58599999999999997,"15min":0.56999999999999995},"min":{"1min":0.46500000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.49399999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:34:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.57299999999999995,"15min":0.53900000000000003},"min":{"1min":0.52900000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.877,"15min":0.877},"last":0.59799999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:16 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.434}]},{"osd":4,"last update":"Sun Jul 3 10:34:40 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.39800000000000002},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.373},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.38800000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.55200000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.51600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.66700000000000004}]},{"osd":6,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.60399999999999998}]},{"osd":7,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.41799999999999998}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.64700000000000002}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.71499999999999997}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.53800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7000,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738152,"statfs":{"total":95995035648,"available":95987867648,"internally_reserved":0,"allocated":868352,"data_stored":419474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:34:38 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91500000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.56599999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64200000000000002,"15min":0.61799999999999999},"min":{"1min":0.46700000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.022,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72699999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:33:16 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.46600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:38 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68700000000000006,"15min":0.66500000000000004},"min":{"1min":0.57499999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.64000000000000001,"15min":0.624},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.747}]},{"osd":5,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.069,"5min":1.141,"15min":1.141},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.03,"5min":1.256,"15min":1.256},"last":0.73999999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:34:28 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.873},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.76000000000000001,"15min":0.747},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.89200000000000002}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.5,"5min":0.497,"15min":0.497},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85199999999999998},"min":{"1min":0.63500000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.82899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:34:07 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.85999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:34:32 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.94199999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.88600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":2,"up_from":16,"seq":68719476791,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7076,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738076,"statfs":{"total":95995035648,"available":95987789824,"internally_reserved":0,"allocated":880640,"data_stored":420966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.59199999999999997},"min":{"1min":0.47699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81299999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.73199999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.42199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79300000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.42199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84599999999999997,"5min":0.874,"15min":0.874},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:13 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.39500000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64200000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.67500000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Jul 3 10:35:00 2022","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.443,"15min":0.436},"min":{"1min":0.33700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.623}]},{"osd":6,"last update":"Sun Jul 3 10:34:29 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65100000000000002,"15min":0.64400000000000002},"min":{"1min":0.48399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46800000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.74399999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51100000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.876}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":3,"up_from":23,"seq":98784247856,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":872448,"data_stored":419488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.44600000000000001,"15min":0.42399999999999999},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72699999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.47099999999999997,"15min":0.441},"min":{"1min":0.38800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.45300000000000001}]},{"osd":1,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59699999999999998,"15min":0.57699999999999996},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.61899999999999999,"15min":0.58499999999999996},"min":{"1min":0.46700000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.68000000000000005,"15min":0.64600000000000002},"min":{"1min":0.438,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.151,"5min":2.581,"15min":2.581},"last":0.746},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69499999999999995,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1559999999999999,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.82099999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:34:37 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.67100000000000004,"15min":0.63100000000000001},"min":{"1min":0.55200000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.67100000000000004,"15min":0.63900000000000001},"min":{"1min":0.56899999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77800000000000002,"15min":0.73099999999999998},"min":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.125,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78900000000000003,"15min":0.754},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1180000000000001,"5min":2.46,"15min":2.46},"last":0.75800000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:34:26 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.64100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0589999999999999,"5min":1.214,"15min":1.214},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.79400000000000004,"15min":0.77400000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.69199999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.90300000000000002,"15min":0.91100000000000003},"min":{"1min":0.65600000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.159,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88900000000000001,"15min":0.88400000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.80300000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:34:31 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.83499999999999996}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.86199999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6168,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738984,"statfs":{"total":95995035648,"available":95988719616,"internally_reserved":0,"allocated":868352,"data_stored":419406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52500000000000002,"15min":0.48999999999999999},"min":{"1min":0.45300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79800000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.52800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.498}]},{"osd":2,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.55700000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57599999999999996,"15min":0.55000000000000004},"min":{"1min":0.46000000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53200000000000003}]},{"osd":5,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.622},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.58099999999999996,"15min":0.56100000000000005},"min":{"1min":0.51400000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.50600000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.83299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:34:51 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.36799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.70999999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:07 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.46300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.872}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.73899999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.91300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":5,"up_from":35,"seq":150323855400,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":925696,"data_stored":1009934,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70899999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":1,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.42099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78200000000000003,"5min":0.874,"15min":0.874},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53000000000000003,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.315,"15min":0.315},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56499999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.48299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.379},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55600000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39700000000000002}]},{"osd":3,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4129999999999998,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.77600000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.47799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.79200000000000004,"5min":0.874,"15min":0.874},"last":0.62},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53900000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:34:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jul 3 10:34:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63500000000000001}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.753,"15min":0.753},"last":0.48999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.71699999999999997}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":33589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:35:11.723 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-03T10:35:11.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.945+0000 7f2a07fff700 1 -- 172.21.15.101:0/669761898 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000fe830 msgr2=0x7f2a000fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.945+0000 7f2a07fff700 1 --2- 172.21.15.101:0/669761898 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000fe830 0x7f2a000fec80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009900 tx=0x7f29f00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 -- 172.21.15.101:0/669761898 shutdown_connections 2022-07-03T10:35:11.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 --2- 172.21.15.101:0/669761898 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000ff1c0 0x7f2a00105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 --2- 172.21.15.101:0/669761898 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fd5b0 0x7f2a000fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 --2- 172.21.15.101:0/669761898 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000fe830 0x7f2a000fec80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 -- 172.21.15.101:0/669761898 >> 172.21.15.101:0/669761898 conn(0x7f2a000f8d00 msgr2=0x7f2a000fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:11.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 -- 172.21.15.101:0/669761898 shutdown_connections 2022-07-03T10:35:11.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.946+0000 7f2a07fff700 1 -- 172.21.15.101:0/669761898 wait complete. 2022-07-03T10:35:11.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 Processor -- start 2022-07-03T10:35:11.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 -- start start 2022-07-03T10:35:11.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fe830 0x7f2a00070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 0x7f2a00070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001123b0 con 0x7f2a000ff1c0 2022-07-03T10:35:11.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000712f0 con 0x7f2a000fe830 2022-07-03T10:35:11.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a07fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000714b0 con 0x7f2a000fd5b0 2022-07-03T10:35:11.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a05d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.947+0000 7f2a0659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 0x7f2a00070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:44568/0 (socket says 172.21.15.101:44568) 2022-07-03T10:35:11.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 -- 172.21.15.101:0/330223869 learned_addr learned my addr 172.21.15.101:0/330223869 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:11.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a0559a700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fe830 0x7f2a00070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 -- 172.21.15.101:0/330223869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fe830 msgr2=0x7f2a00070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fe830 0x7f2a00070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 -- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 msgr2=0x7f2a00070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:11.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 0x7f2a00070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:11.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 -- 172.21.15.101:0/330223869 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009410 con 0x7f2a000fd5b0 2022-07-03T10:35:11.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a0659c700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 0x7f2a00070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:11.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.948+0000 7f2a05d9b700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00ea00 tx=0x7f29fc005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.949+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc00c820 con 0x7f2a000fd5b0 2022-07-03T10:35:11.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.949+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f29fc00ce30 con 0x7f2a000fd5b0 2022-07-03T10:35:11.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.949+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f0009440 con 0x7f2a000fd5b0 2022-07-03T10:35:11.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.949+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc018420 con 0x7f2a000fd5b0 2022-07-03T10:35:11.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.949+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0011d010 con 0x7f2a000fd5b0 2022-07-03T10:35:11.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.950+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a00108d70 con 0x7f2a000fd5b0 2022-07-03T10:35:11.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.952+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f29fc00c040 con 0x7f2a000fd5b0 2022-07-03T10:35:11.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.953+0000 7f29f67fc700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 0x7f29ec099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:11.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.953+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f29fc014070 con 0x7f2a000fd5b0 2022-07-03T10:35:11.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.953+0000 7f2a0559a700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 0x7f29ec099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:11.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.954+0000 7f2a0559a700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 0x7f29ec099210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009f70 tx=0x7f29f00091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:11.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:11.955+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f29fc08cfc0 con 0x7f2a000fd5b0 2022-07-03T10:35:12.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.141+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a0005f320 con 0x7f29ec096d60 2022-07-03T10:35:12.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.151+0000 7f29f67fc700 1 -- 172.21.15.101:0/330223869 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+334264 (secure 0 0 0) 0x7f2a0005f320 con 0x7f29ec096d60 2022-07-03T10:35:12.159 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:12.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 msgr2=0x7f29ec099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:12.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 0x7f29ec099210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009f70 tx=0x7f29f00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 msgr2=0x7f2a00072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:12.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00ea00 tx=0x7f29fc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 shutdown_connections 2022-07-03T10:35:12.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f29ec096d60 0x7f29ec099210 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2a000ff1c0 0x7f2a00070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a000fd5b0 0x7f2a00072230 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 --2- 172.21.15.101:0/330223869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a000fe830 0x7f2a00070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:12.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 >> 172.21.15.101:0/330223869 conn(0x7f2a000f8d00 msgr2=0x7f2a000fb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:12.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 shutdown_connections 2022-07-03T10:35:12.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:12.156+0000 7f2a07fff700 1 -- 172.21.15.101:0/330223869 wait complete. 2022-07-03T10:35:12.165 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:35:12.185 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2022-07-03T10:35:11.934172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624668,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":141,"num_read_kb":137,"num_write":236,"num_write_kb":2986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":76568,"kb_used_data":10424,"kb_used_omap":0,"kb_used_meta":66048,"kb_avail":1124865256,"statfs":{"total":1151940427776,"available":1151862022144,"internally_reserved":0,"allocated":10674176,"data_stored":6859998,"data_compressed":107352,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":67633152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34300,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":55,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.504101"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461438+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:35:03.461438+0000","last_peered":"2022-07-03T10:35:03.461438+0000","last_clean":"2022-07-03T10:35:03.461438+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:35:03.461438+0000","last_undegraded":"2022-07-03T10:35:03.461438+0000","last_fullsized":"2022-07-03T10:35:03.461438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472013+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:35:03.472013+0000","last_peered":"2022-07-03T10:35:03.472013+0000","last_clean":"2022-07-03T10:35:03.472013+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:35:03.472013+0000","last_undegraded":"2022-07-03T10:35:03.472013+0000","last_fullsized":"2022-07-03T10:35:03.472013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.469723+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:35:03.469723+0000","last_peered":"2022-07-03T10:35:03.469723+0000","last_clean":"2022-07-03T10:35:03.469723+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:35:03.469723+0000","last_undegraded":"2022-07-03T10:35:03.469723+0000","last_fullsized":"2022-07-03T10:35:03.469723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459886+0000","last_change":"2022-07-03T10:35:03.459299+0000","last_active":"2022-07-03T10:35:03.459886+0000","last_peered":"2022-07-03T10:35:03.459886+0000","last_clean":"2022-07-03T10:35:03.459886+0000","last_became_active":"2022-07-03T10:35:03.458602+0000","last_became_peered":"2022-07-03T10:35:03.458602+0000","last_unstale":"2022-07-03T10:35:03.459886+0000","last_undegraded":"2022-07-03T10:35:03.459886+0000","last_fullsized":"2022-07-03T10:35:03.459886+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:02:23.899940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.466024+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:35:03.466024+0000","last_peered":"2022-07-03T10:35:03.466024+0000","last_clean":"2022-07-03T10:35:03.466024+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:35:03.466024+0000","last_undegraded":"2022-07-03T10:35:03.466024+0000","last_fullsized":"2022-07-03T10:35:03.466024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453962+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:35:03.453962+0000","last_peered":"2022-07-03T10:35:03.453962+0000","last_clean":"2022-07-03T10:35:03.453962+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:35:03.453962+0000","last_undegraded":"2022-07-03T10:35:03.453962+0000","last_fullsized":"2022-07-03T10:35:03.453962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455141+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:35:03.455141+0000","last_peered":"2022-07-03T10:35:03.455141+0000","last_clean":"2022-07-03T10:35:03.455141+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:35:03.455141+0000","last_undegraded":"2022-07-03T10:35:03.455141+0000","last_fullsized":"2022-07-03T10:35:03.455141+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443794+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:35:03.443794+0000","last_peered":"2022-07-03T10:35:03.443794+0000","last_clean":"2022-07-03T10:35:03.443794+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:35:03.443794+0000","last_undegraded":"2022-07-03T10:35:03.443794+0000","last_fullsized":"2022-07-03T10:35:03.443794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452820+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:35:03.452820+0000","last_peered":"2022-07-03T10:35:03.452820+0000","last_clean":"2022-07-03T10:35:03.452820+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:35:03.452820+0000","last_undegraded":"2022-07-03T10:35:03.452820+0000","last_fullsized":"2022-07-03T10:35:03.452820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455453+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:35:03.455453+0000","last_peered":"2022-07-03T10:35:03.455453+0000","last_clean":"2022-07-03T10:35:03.455453+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:35:03.455453+0000","last_undegraded":"2022-07-03T10:35:03.455453+0000","last_fullsized":"2022-07-03T10:35:03.455453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.456999+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:35:03.456999+0000","last_peered":"2022-07-03T10:35:03.456999+0000","last_clean":"2022-07-03T10:35:03.456999+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:35:03.456999+0000","last_undegraded":"2022-07-03T10:35:03.456999+0000","last_fullsized":"2022-07-03T10:35:03.456999+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459417+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:35:03.459417+0000","last_peered":"2022-07-03T10:35:03.459417+0000","last_clean":"2022-07-03T10:35:03.459417+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:03.459417+0000","last_undegraded":"2022-07-03T10:35:03.459417+0000","last_fullsized":"2022-07-03T10:35:03.459417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443495+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:35:03.443495+0000","last_peered":"2022-07-03T10:35:03.443495+0000","last_clean":"2022-07-03T10:35:03.443495+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:35:03.443495+0000","last_undegraded":"2022-07-03T10:35:03.443495+0000","last_fullsized":"2022-07-03T10:35:03.443495+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470143+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:35:03.470143+0000","last_peered":"2022-07-03T10:35:03.470143+0000","last_clean":"2022-07-03T10:35:03.470143+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:35:03.470143+0000","last_undegraded":"2022-07-03T10:35:03.470143+0000","last_fullsized":"2022-07-03T10:35:03.470143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471960+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:35:03.471960+0000","last_peered":"2022-07-03T10:35:03.471960+0000","last_clean":"2022-07-03T10:35:03.471960+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:35:03.471960+0000","last_undegraded":"2022-07-03T10:35:03.471960+0000","last_fullsized":"2022-07-03T10:35:03.471960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461440+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:35:03.461440+0000","last_peered":"2022-07-03T10:35:03.461440+0000","last_clean":"2022-07-03T10:35:03.461440+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:35:03.461440+0000","last_undegraded":"2022-07-03T10:35:03.461440+0000","last_fullsized":"2022-07-03T10:35:03.461440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.462784+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:35:03.462784+0000","last_peered":"2022-07-03T10:35:03.462784+0000","last_clean":"2022-07-03T10:35:03.462784+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:35:03.462784+0000","last_undegraded":"2022-07-03T10:35:03.462784+0000","last_fullsized":"2022-07-03T10:35:03.462784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452828+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:35:03.452828+0000","last_peered":"2022-07-03T10:35:03.452828+0000","last_clean":"2022-07-03T10:35:03.452828+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:35:03.452828+0000","last_undegraded":"2022-07-03T10:35:03.452828+0000","last_fullsized":"2022-07-03T10:35:03.452828+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472689+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:35:03.472689+0000","last_peered":"2022-07-03T10:35:03.472689+0000","last_clean":"2022-07-03T10:35:03.472689+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:35:03.472689+0000","last_undegraded":"2022-07-03T10:35:03.472689+0000","last_fullsized":"2022-07-03T10:35:03.472689+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.482646+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:04.482646+0000","last_peered":"2022-07-03T10:35:04.482646+0000","last_clean":"2022-07-03T10:35:04.482646+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:04.482646+0000","last_undegraded":"2022-07-03T10:35:04.482646+0000","last_fullsized":"2022-07-03T10:35:04.482646+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464595+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:35:03.464595+0000","last_peered":"2022-07-03T10:35:03.464595+0000","last_clean":"2022-07-03T10:35:03.464595+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:35:03.464595+0000","last_undegraded":"2022-07-03T10:35:03.464595+0000","last_fullsized":"2022-07-03T10:35:03.464595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.463832+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:03.463832+0000","last_peered":"2022-07-03T10:35:03.463832+0000","last_clean":"2022-07-03T10:35:03.463832+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:03.463832+0000","last_undegraded":"2022-07-03T10:35:03.463832+0000","last_fullsized":"2022-07-03T10:35:03.463832+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472286+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:35:03.472286+0000","last_peered":"2022-07-03T10:35:03.472286+0000","last_clean":"2022-07-03T10:35:03.472286+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:35:03.472286+0000","last_undegraded":"2022-07-03T10:35:03.472286+0000","last_fullsized":"2022-07-03T10:35:03.472286+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460634+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:03.460634+0000","last_peered":"2022-07-03T10:35:03.460634+0000","last_clean":"2022-07-03T10:35:03.460634+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:03.460634+0000","last_undegraded":"2022-07-03T10:35:03.460634+0000","last_fullsized":"2022-07-03T10:35:03.460634+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.454988+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:35:03.454988+0000","last_peered":"2022-07-03T10:35:03.454988+0000","last_clean":"2022-07-03T10:35:03.454988+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:35:03.454988+0000","last_undegraded":"2022-07-03T10:35:03.454988+0000","last_fullsized":"2022-07-03T10:35:03.454988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.454239+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:03.454239+0000","last_peered":"2022-07-03T10:35:03.454239+0000","last_clean":"2022-07-03T10:35:03.454239+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:03.454239+0000","last_undegraded":"2022-07-03T10:35:03.454239+0000","last_fullsized":"2022-07-03T10:35:03.454239+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451613+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:35:03.451613+0000","last_peered":"2022-07-03T10:35:03.451613+0000","last_clean":"2022-07-03T10:35:03.451613+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:35:03.451613+0000","last_undegraded":"2022-07-03T10:35:03.451613+0000","last_fullsized":"2022-07-03T10:35:03.451613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449401+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:03.449401+0000","last_peered":"2022-07-03T10:35:03.449401+0000","last_clean":"2022-07-03T10:35:03.449401+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:03.449401+0000","last_undegraded":"2022-07-03T10:35:03.449401+0000","last_fullsized":"2022-07-03T10:35:03.449401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.456790+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:35:03.456790+0000","last_peered":"2022-07-03T10:35:03.456790+0000","last_clean":"2022-07-03T10:35:03.456790+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:35:03.456790+0000","last_undegraded":"2022-07-03T10:35:03.456790+0000","last_fullsized":"2022-07-03T10:35:03.456790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":12,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451111+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:03.451111+0000","last_peered":"2022-07-03T10:35:03.451111+0000","last_clean":"2022-07-03T10:35:03.451111+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:03.451111+0000","last_undegraded":"2022-07-03T10:35:03.451111+0000","last_fullsized":"2022-07-03T10:35:03.451111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449346+0000","last_change":"2022-07-03T10:35:03.448987+0000","last_active":"2022-07-03T10:35:03.449346+0000","last_peered":"2022-07-03T10:35:03.449346+0000","last_clean":"2022-07-03T10:35:03.449346+0000","last_became_active":"2022-07-03T10:35:03.448669+0000","last_became_peered":"2022-07-03T10:35:03.448669+0000","last_unstale":"2022-07-03T10:35:03.449346+0000","last_undegraded":"2022-07-03T10:35:03.449346+0000","last_fullsized":"2022-07-03T10:35:03.449346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:32:30.164547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.434059+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:03.434059+0000","last_peered":"2022-07-03T10:35:03.434059+0000","last_clean":"2022-07-03T10:35:03.434059+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:03.434059+0000","last_undegraded":"2022-07-03T10:35:03.434059+0000","last_fullsized":"2022-07-03T10:35:03.434059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.458898+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:35:03.458898+0000","last_peered":"2022-07-03T10:35:03.458898+0000","last_clean":"2022-07-03T10:35:03.458898+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:35:03.458898+0000","last_undegraded":"2022-07-03T10:35:03.458898+0000","last_fullsized":"2022-07-03T10:35:03.458898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443557+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:03.443557+0000","last_peered":"2022-07-03T10:35:03.443557+0000","last_clean":"2022-07-03T10:35:03.443557+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:03.443557+0000","last_undegraded":"2022-07-03T10:35:03.443557+0000","last_fullsized":"2022-07-03T10:35:03.443557+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464856+0000","last_change":"2022-07-03T10:35:03.462976+0000","last_active":"2022-07-03T10:35:03.464856+0000","last_peered":"2022-07-03T10:35:03.464856+0000","last_clean":"2022-07-03T10:35:03.464856+0000","last_became_active":"2022-07-03T10:35:03.462590+0000","last_became_peered":"2022-07-03T10:35:03.462590+0000","last_unstale":"2022-07-03T10:35:03.464856+0000","last_undegraded":"2022-07-03T10:35:03.464856+0000","last_fullsized":"2022-07-03T10:35:03.464856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:51:17.814072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.431704+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:03.431704+0000","last_peered":"2022-07-03T10:35:03.431704+0000","last_clean":"2022-07-03T10:35:03.431704+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:03.431704+0000","last_undegraded":"2022-07-03T10:35:03.431704+0000","last_fullsized":"2022-07-03T10:35:03.431704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451820+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:35:03.451820+0000","last_peered":"2022-07-03T10:35:03.451820+0000","last_clean":"2022-07-03T10:35:03.451820+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:35:03.451820+0000","last_undegraded":"2022-07-03T10:35:03.451820+0000","last_fullsized":"2022-07-03T10:35:03.451820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438569+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:03.438569+0000","last_peered":"2022-07-03T10:35:03.438569+0000","last_clean":"2022-07-03T10:35:03.438569+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:03.438569+0000","last_undegraded":"2022-07-03T10:35:03.438569+0000","last_fullsized":"2022-07-03T10:35:03.438569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470197+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:35:03.470197+0000","last_peered":"2022-07-03T10:35:03.470197+0000","last_clean":"2022-07-03T10:35:03.470197+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:35:03.470197+0000","last_undegraded":"2022-07-03T10:35:03.470197+0000","last_fullsized":"2022-07-03T10:35:03.470197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":14,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471231+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:03.471231+0000","last_peered":"2022-07-03T10:35:03.471231+0000","last_clean":"2022-07-03T10:35:03.471231+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:03.471231+0000","last_undegraded":"2022-07-03T10:35:03.471231+0000","last_fullsized":"2022-07-03T10:35:03.471231+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459058+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:35:03.459058+0000","last_peered":"2022-07-03T10:35:03.459058+0000","last_clean":"2022-07-03T10:35:03.459058+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:03.459058+0000","last_undegraded":"2022-07-03T10:35:03.459058+0000","last_fullsized":"2022-07-03T10:35:03.459058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'16","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:09.747061+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:35:09.747061+0000","last_peered":"2022-07-03T10:35:09.747061+0000","last_clean":"2022-07-03T10:35:09.747061+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:35:09.747061+0000","last_undegraded":"2022-07-03T10:35:09.747061+0000","last_fullsized":"2022-07-03T10:35:09.747061+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33395,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":22,"num_write":16,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455885+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:35:03.455885+0000","last_peered":"2022-07-03T10:35:03.455885+0000","last_clean":"2022-07-03T10:35:03.455885+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:35:03.455885+0000","last_undegraded":"2022-07-03T10:35:03.455885+0000","last_fullsized":"2022-07-03T10:35:03.455885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.482073+0000","last_change":"2022-07-03T10:35:02.869468+0000","last_active":"2022-07-03T10:35:04.482073+0000","last_peered":"2022-07-03T10:35:04.482073+0000","last_clean":"2022-07-03T10:35:04.482073+0000","last_became_active":"2022-07-03T10:35:02.869069+0000","last_became_peered":"2022-07-03T10:35:02.869069+0000","last_unstale":"2022-07-03T10:35:04.482073+0000","last_undegraded":"2022-07-03T10:35:04.482073+0000","last_fullsized":"2022-07-03T10:35:04.482073+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:43:52.762318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452376+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:35:03.452376+0000","last_peered":"2022-07-03T10:35:03.452376+0000","last_clean":"2022-07-03T10:35:03.452376+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:35:03.452376+0000","last_undegraded":"2022-07-03T10:35:03.452376+0000","last_fullsized":"2022-07-03T10:35:03.452376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.443336+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:03.443336+0000","last_peered":"2022-07-03T10:35:03.443336+0000","last_clean":"2022-07-03T10:35:03.443336+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:03.443336+0000","last_undegraded":"2022-07-03T10:35:03.443336+0000","last_fullsized":"2022-07-03T10:35:03.443336+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449611+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:35:03.449611+0000","last_peered":"2022-07-03T10:35:03.449611+0000","last_clean":"2022-07-03T10:35:03.449611+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:35:03.449611+0000","last_undegraded":"2022-07-03T10:35:03.449611+0000","last_fullsized":"2022-07-03T10:35:03.449611+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438228+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:03.438228+0000","last_peered":"2022-07-03T10:35:03.438228+0000","last_clean":"2022-07-03T10:35:03.438228+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:03.438228+0000","last_undegraded":"2022-07-03T10:35:03.438228+0000","last_fullsized":"2022-07-03T10:35:03.438228+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453137+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:35:03.453137+0000","last_peered":"2022-07-03T10:35:03.453137+0000","last_clean":"2022-07-03T10:35:03.453137+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:35:03.453137+0000","last_undegraded":"2022-07-03T10:35:03.453137+0000","last_fullsized":"2022-07-03T10:35:03.453137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":19,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.472850+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:03.472850+0000","last_peered":"2022-07-03T10:35:03.472850+0000","last_clean":"2022-07-03T10:35:03.472850+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:03.472850+0000","last_undegraded":"2022-07-03T10:35:03.472850+0000","last_fullsized":"2022-07-03T10:35:03.472850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.449113+0000","last_change":"2022-07-03T10:35:03.448350+0000","last_active":"2022-07-03T10:35:03.449113+0000","last_peered":"2022-07-03T10:35:03.449113+0000","last_clean":"2022-07-03T10:35:03.449113+0000","last_became_active":"2022-07-03T10:35:03.448015+0000","last_became_peered":"2022-07-03T10:35:03.448015+0000","last_unstale":"2022-07-03T10:35:03.449113+0000","last_undegraded":"2022-07-03T10:35:03.449113+0000","last_fullsized":"2022-07-03T10:35:03.449113+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:52:58.951227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.483163+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:04.483163+0000","last_peered":"2022-07-03T10:35:04.483163+0000","last_clean":"2022-07-03T10:35:04.483163+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:04.483163+0000","last_undegraded":"2022-07-03T10:35:04.483163+0000","last_fullsized":"2022-07-03T10:35:04.483163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451468+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:35:03.451468+0000","last_peered":"2022-07-03T10:35:03.451468+0000","last_clean":"2022-07-03T10:35:03.451468+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:35:03.451468+0000","last_undegraded":"2022-07-03T10:35:03.451468+0000","last_fullsized":"2022-07-03T10:35:03.451468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.446946+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:03.446946+0000","last_peered":"2022-07-03T10:35:03.446946+0000","last_clean":"2022-07-03T10:35:03.446946+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:03.446946+0000","last_undegraded":"2022-07-03T10:35:03.446946+0000","last_fullsized":"2022-07-03T10:35:03.446946+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.432833+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:35:04.432833+0000","last_peered":"2022-07-03T10:35:04.432833+0000","last_clean":"2022-07-03T10:35:04.432833+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:35:04.432833+0000","last_undegraded":"2022-07-03T10:35:04.432833+0000","last_fullsized":"2022-07-03T10:35:04.432833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.440467+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:03.440467+0000","last_peered":"2022-07-03T10:35:03.440467+0000","last_clean":"2022-07-03T10:35:03.440467+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:03.440467+0000","last_undegraded":"2022-07-03T10:35:03.440467+0000","last_fullsized":"2022-07-03T10:35:03.440467+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459439+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:35:03.459439+0000","last_peered":"2022-07-03T10:35:03.459439+0000","last_clean":"2022-07-03T10:35:03.459439+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:35:03.459439+0000","last_undegraded":"2022-07-03T10:35:03.459439+0000","last_fullsized":"2022-07-03T10:35:03.459439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.448258+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:35:03.448258+0000","last_peered":"2022-07-03T10:35:03.448258+0000","last_clean":"2022-07-03T10:35:03.448258+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:35:03.448258+0000","last_undegraded":"2022-07-03T10:35:03.448258+0000","last_fullsized":"2022-07-03T10:35:03.448258+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.433956+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:03.433956+0000","last_peered":"2022-07-03T10:35:03.433956+0000","last_clean":"2022-07-03T10:35:03.433956+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:03.433956+0000","last_undegraded":"2022-07-03T10:35:03.433956+0000","last_fullsized":"2022-07-03T10:35:03.433956+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457006+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:35:03.457006+0000","last_peered":"2022-07-03T10:35:03.457006+0000","last_clean":"2022-07-03T10:35:03.457006+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:35:03.457006+0000","last_undegraded":"2022-07-03T10:35:03.457006+0000","last_fullsized":"2022-07-03T10:35:03.457006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.439376+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:03.439376+0000","last_peered":"2022-07-03T10:35:03.439376+0000","last_clean":"2022-07-03T10:35:03.439376+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:03.439376+0000","last_undegraded":"2022-07-03T10:35:03.439376+0000","last_fullsized":"2022-07-03T10:35:03.439376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438794+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:03.438794+0000","last_peered":"2022-07-03T10:35:03.438794+0000","last_clean":"2022-07-03T10:35:03.438794+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:03.438794+0000","last_undegraded":"2022-07-03T10:35:03.438794+0000","last_fullsized":"2022-07-03T10:35:03.438794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470153+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:35:03.470153+0000","last_peered":"2022-07-03T10:35:03.470153+0000","last_clean":"2022-07-03T10:35:03.470153+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:35:03.470153+0000","last_undegraded":"2022-07-03T10:35:03.470153+0000","last_fullsized":"2022-07-03T10:35:03.470153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438916+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:03.438916+0000","last_peered":"2022-07-03T10:35:03.438916+0000","last_clean":"2022-07-03T10:35:03.438916+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:03.438916+0000","last_undegraded":"2022-07-03T10:35:03.438916+0000","last_fullsized":"2022-07-03T10:35:03.438916+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.438900+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:03.438900+0000","last_peered":"2022-07-03T10:35:03.438900+0000","last_clean":"2022-07-03T10:35:03.438900+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:03.438900+0000","last_undegraded":"2022-07-03T10:35:03.438900+0000","last_fullsized":"2022-07-03T10:35:03.438900+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470304+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:35:03.470304+0000","last_peered":"2022-07-03T10:35:03.470304+0000","last_clean":"2022-07-03T10:35:03.470304+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:35:03.470304+0000","last_undegraded":"2022-07-03T10:35:03.470304+0000","last_fullsized":"2022-07-03T10:35:03.470304+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.444191+0000","last_change":"2022-07-03T10:35:02.876478+0000","last_active":"2022-07-03T10:35:03.444191+0000","last_peered":"2022-07-03T10:35:03.444191+0000","last_clean":"2022-07-03T10:35:03.444191+0000","last_became_active":"2022-07-03T10:35:02.876049+0000","last_became_peered":"2022-07-03T10:35:02.876049+0000","last_unstale":"2022-07-03T10:35:03.444191+0000","last_undegraded":"2022-07-03T10:35:03.444191+0000","last_fullsized":"2022-07-03T10:35:03.444191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:45.607526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.458929+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:35:03.458929+0000","last_peered":"2022-07-03T10:35:03.458929+0000","last_clean":"2022-07-03T10:35:03.458929+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:35:03.458929+0000","last_undegraded":"2022-07-03T10:35:03.458929+0000","last_fullsized":"2022-07-03T10:35:03.458929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.452526+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:03.452526+0000","last_peered":"2022-07-03T10:35:03.452526+0000","last_clean":"2022-07-03T10:35:03.452526+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:03.452526+0000","last_undegraded":"2022-07-03T10:35:03.452526+0000","last_fullsized":"2022-07-03T10:35:03.452526+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.471996+0000","last_change":"2022-07-03T10:35:03.470001+0000","last_active":"2022-07-03T10:35:03.471996+0000","last_peered":"2022-07-03T10:35:03.471996+0000","last_clean":"2022-07-03T10:35:03.471996+0000","last_became_active":"2022-07-03T10:35:03.461137+0000","last_became_peered":"2022-07-03T10:35:03.461137+0000","last_unstale":"2022-07-03T10:35:03.471996+0000","last_undegraded":"2022-07-03T10:35:03.471996+0000","last_fullsized":"2022-07-03T10:35:03.471996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:45:25.469289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.442789+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:03.442789+0000","last_peered":"2022-07-03T10:35:03.442789+0000","last_clean":"2022-07-03T10:35:03.442789+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:03.442789+0000","last_undegraded":"2022-07-03T10:35:03.442789+0000","last_fullsized":"2022-07-03T10:35:03.442789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455870+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:35:03.455870+0000","last_peered":"2022-07-03T10:35:03.455870+0000","last_clean":"2022-07-03T10:35:03.455870+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:35:03.455870+0000","last_undegraded":"2022-07-03T10:35:03.455870+0000","last_fullsized":"2022-07-03T10:35:03.455870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.432615+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:03.432615+0000","last_peered":"2022-07-03T10:35:03.432615+0000","last_clean":"2022-07-03T10:35:03.432615+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:03.432615+0000","last_undegraded":"2022-07-03T10:35:03.432615+0000","last_fullsized":"2022-07-03T10:35:03.432615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461191+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:35:03.461191+0000","last_peered":"2022-07-03T10:35:03.461191+0000","last_clean":"2022-07-03T10:35:03.461191+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:35:03.461191+0000","last_undegraded":"2022-07-03T10:35:03.461191+0000","last_fullsized":"2022-07-03T10:35:03.461191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.453026+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:03.453026+0000","last_peered":"2022-07-03T10:35:03.453026+0000","last_clean":"2022-07-03T10:35:03.453026+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:03.453026+0000","last_undegraded":"2022-07-03T10:35:03.453026+0000","last_fullsized":"2022-07-03T10:35:03.453026+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.465427+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:35:03.465427+0000","last_peered":"2022-07-03T10:35:03.465427+0000","last_clean":"2022-07-03T10:35:03.465427+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:35:03.465427+0000","last_undegraded":"2022-07-03T10:35:03.465427+0000","last_fullsized":"2022-07-03T10:35:03.465427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.483047+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:04.483047+0000","last_peered":"2022-07-03T10:35:04.483047+0000","last_clean":"2022-07-03T10:35:04.483047+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:04.483047+0000","last_undegraded":"2022-07-03T10:35:04.483047+0000","last_fullsized":"2022-07-03T10:35:04.483047+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451094+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:35:03.451094+0000","last_peered":"2022-07-03T10:35:03.451094+0000","last_clean":"2022-07-03T10:35:03.451094+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:35:03.451094+0000","last_undegraded":"2022-07-03T10:35:03.451094+0000","last_fullsized":"2022-07-03T10:35:03.451094+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455489+0000","last_change":"2022-07-03T10:35:02.879933+0000","last_active":"2022-07-03T10:35:03.455489+0000","last_peered":"2022-07-03T10:35:03.455489+0000","last_clean":"2022-07-03T10:35:03.455489+0000","last_became_active":"2022-07-03T10:35:02.879537+0000","last_became_peered":"2022-07-03T10:35:02.879537+0000","last_unstale":"2022-07-03T10:35:03.455489+0000","last_undegraded":"2022-07-03T10:35:03.455489+0000","last_fullsized":"2022-07-03T10:35:03.455489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:04:10.041882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455333+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:35:03.455333+0000","last_peered":"2022-07-03T10:35:03.455333+0000","last_clean":"2022-07-03T10:35:03.455333+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:35:03.455333+0000","last_undegraded":"2022-07-03T10:35:03.455333+0000","last_fullsized":"2022-07-03T10:35:03.455333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2022-07-03T10:35:04.481718+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:04.481718+0000","last_peered":"2022-07-03T10:35:04.481718+0000","last_clean":"2022-07-03T10:35:04.481718+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:04.481718+0000","last_undegraded":"2022-07-03T10:35:04.481718+0000","last_fullsized":"2022-07-03T10:35:04.481718+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457937+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:35:03.457937+0000","last_peered":"2022-07-03T10:35:03.457937+0000","last_clean":"2022-07-03T10:35:03.457937+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:35:03.457937+0000","last_undegraded":"2022-07-03T10:35:03.457937+0000","last_fullsized":"2022-07-03T10:35:03.457937+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":13,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.466452+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:03.466452+0000","last_peered":"2022-07-03T10:35:03.466452+0000","last_clean":"2022-07-03T10:35:03.466452+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:03.466452+0000","last_undegraded":"2022-07-03T10:35:03.466452+0000","last_fullsized":"2022-07-03T10:35:03.466452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.457040+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:35:03.457040+0000","last_peered":"2022-07-03T10:35:03.457040+0000","last_clean":"2022-07-03T10:35:03.457040+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:35:03.457040+0000","last_undegraded":"2022-07-03T10:35:03.457040+0000","last_fullsized":"2022-07-03T10:35:03.457040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451455+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:35:03.451455+0000","last_peered":"2022-07-03T10:35:03.451455+0000","last_clean":"2022-07-03T10:35:03.451455+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:35:03.451455+0000","last_undegraded":"2022-07-03T10:35:03.451455+0000","last_fullsized":"2022-07-03T10:35:03.451455+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.459361+0000","last_change":"2022-07-03T10:35:03.455645+0000","last_active":"2022-07-03T10:35:03.459361+0000","last_peered":"2022-07-03T10:35:03.459361+0000","last_clean":"2022-07-03T10:35:03.459361+0000","last_became_active":"2022-07-03T10:35:03.455338+0000","last_became_peered":"2022-07-03T10:35:03.455338+0000","last_unstale":"2022-07-03T10:35:03.459361+0000","last_undegraded":"2022-07-03T10:35:03.459361+0000","last_fullsized":"2022-07-03T10:35:03.459361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:45:15.371858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460610+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:35:03.460610+0000","last_peered":"2022-07-03T10:35:03.460610+0000","last_clean":"2022-07-03T10:35:03.460610+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:35:03.460610+0000","last_undegraded":"2022-07-03T10:35:03.460610+0000","last_fullsized":"2022-07-03T10:35:03.460610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.462640+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:35:03.462640+0000","last_peered":"2022-07-03T10:35:03.462640+0000","last_clean":"2022-07-03T10:35:03.462640+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:35:03.462640+0000","last_undegraded":"2022-07-03T10:35:03.462640+0000","last_fullsized":"2022-07-03T10:35:03.462640+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470323+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:35:03.470323+0000","last_peered":"2022-07-03T10:35:03.470323+0000","last_clean":"2022-07-03T10:35:03.470323+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:35:03.470323+0000","last_undegraded":"2022-07-03T10:35:03.470323+0000","last_fullsized":"2022-07-03T10:35:03.470323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.455581+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:35:03.455581+0000","last_peered":"2022-07-03T10:35:03.455581+0000","last_clean":"2022-07-03T10:35:03.455581+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:35:03.455581+0000","last_undegraded":"2022-07-03T10:35:03.455581+0000","last_fullsized":"2022-07-03T10:35:03.455581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460129+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:35:03.460129+0000","last_peered":"2022-07-03T10:35:03.460129+0000","last_clean":"2022-07-03T10:35:03.460129+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:35:03.460129+0000","last_undegraded":"2022-07-03T10:35:03.460129+0000","last_fullsized":"2022-07-03T10:35:03.460129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.464434+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:35:03.464434+0000","last_peered":"2022-07-03T10:35:03.464434+0000","last_clean":"2022-07-03T10:35:03.464434+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:35:03.464434+0000","last_undegraded":"2022-07-03T10:35:03.464434+0000","last_fullsized":"2022-07-03T10:35:03.464434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.451590+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:35:03.451590+0000","last_peered":"2022-07-03T10:35:03.451590+0000","last_clean":"2022-07-03T10:35:03.451590+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:35:03.451590+0000","last_undegraded":"2022-07-03T10:35:03.451590+0000","last_fullsized":"2022-07-03T10:35:03.451590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.461326+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:35:03.461326+0000","last_peered":"2022-07-03T10:35:03.461326+0000","last_clean":"2022-07-03T10:35:03.461326+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:35:03.461326+0000","last_undegraded":"2022-07-03T10:35:03.461326+0000","last_fullsized":"2022-07-03T10:35:03.461326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.463003+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:35:03.463003+0000","last_peered":"2022-07-03T10:35:03.463003+0000","last_clean":"2022-07-03T10:35:03.463003+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:35:03.463003+0000","last_undegraded":"2022-07-03T10:35:03.463003+0000","last_fullsized":"2022-07-03T10:35:03.463003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.470330+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:35:03.470330+0000","last_peered":"2022-07-03T10:35:03.470330+0000","last_clean":"2022-07-03T10:35:03.470330+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:35:03.470330+0000","last_undegraded":"2022-07-03T10:35:03.470330+0000","last_fullsized":"2022-07-03T10:35:03.470330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2022-07-03T10:35:03.460022+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:35:03.460022+0000","last_peered":"2022-07-03T10:35:03.460022+0000","last_clean":"2022-07-03T10:35:03.460022+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:35:03.460022+0000","last_undegraded":"2022-07-03T10:35:03.460022+0000","last_fullsized":"2022-07-03T10:35:03.460022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":34177,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":15,"num_read_kb":28,"num_write":54,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":70684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547213,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5844,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739308,"statfs":{"total":95995035648,"available":95989051392,"internally_reserved":0,"allocated":864256,"data_stored":419384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]}]},{"osd":10,"up_from":67,"seq":287762808849,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":954368,"data_stored":1028574,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47899999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57199999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.78900000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.55700000000000005}]},{"osd":9,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.755},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.748}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":9,"up_from":62,"seq":266287972374,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5976,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739176,"statfs":{"total":95995035648,"available":95988916224,"internally_reserved":0,"allocated":868352,"data_stored":419950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72699999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.73499999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.8360000000000001,"5min":1.8360000000000001,"15min":1.8360000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.54200000000000004}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.84099999999999997}]},{"osd":6,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.82099999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.77100000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.921,"5min":1.921,"15min":1.921},"last":0.873},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.80100000000000005}]},{"osd":10,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.753}]}]},{"osd":8,"up_from":54,"seq":231928234011,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":852,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":872448,"data_stored":419972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46800000000000003}]},{"osd":1,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.505}]},{"osd":2,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.443}]},{"osd":3,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.61599999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995}]},{"osd":5,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.64100000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.628}]},{"osd":7,"last update":"Sun Jul 3 10:34:05 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.66400000000000003}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.70399999999999996}]},{"osd":10,"last update":"Sun Jul 3 10:34:56 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.68899999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]}]},{"osd":7,"up_from":47,"seq":201863462942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6280,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":917504,"data_stored":1010318,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.29599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.436}]},{"osd":2,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.46400000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.46400000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.71499999999999997,"5min":0.745,"15min":0.745},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.5,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.48099999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":1.222,"15min":1.222},"last":0.55300000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.47399999999999998,"15min":0.46000000000000002},"min":{"1min":0.36499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.38500000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.51400000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.49199999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.88,"15min":0.89800000000000002},"min":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96999999999999997,"5min":5.077,"15min":5.077},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.68000000000000005,"15min":0.66400000000000003},"min":{"1min":0.54200000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.0129999999999999,"5min":1.02,"15min":1.02},"last":0.68300000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:35:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.52600000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.623},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57799999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:34:37 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.53000000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6168,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738984,"statfs":{"total":95995035648,"available":95988719616,"internally_reserved":0,"allocated":868352,"data_stored":419507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.40999999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66200000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.39500000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.65300000000000002,"15min":0.63700000000000001},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.69199999999999995,"15min":0.68000000000000005},"min":{"1min":0.54700000000000004,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97699999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.752}]},{"osd":2,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.47499999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.624},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.71599999999999997,"15min":0.70799999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.03,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68600000000000005,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.98699999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71599999999999997,"15min":0.70399999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.96899999999999997,"5min":1.022,"15min":1.022},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.753,"15min":0.751},"min":{"1min":0.58599999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.016,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.79800000000000004,"15min":0.78900000000000003},"min":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.0580000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.80200000000000005,"15min":0.79300000000000004},"min":{"1min":0.61599999999999999,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.048,"5min":1.155,"15min":1.155},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.81599999999999995,"15min":0.80700000000000005},"min":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0920000000000001,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85699999999999998}]},{"osd":7,"last update":"Sun Jul 3 10:34:48 2022","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.84599999999999997,"15min":0.83599999999999997},"min":{"1min":0.68200000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1459999999999999,"5min":1.333,"15min":1.333},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.86499999999999999,"15min":0.85699999999999998},"min":{"1min":0.65700000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.093,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.84999999999999998}]},{"osd":8,"last update":"Sun Jul 3 10:35:04 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.67700000000000005}]},{"osd":9,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.80700000000000005}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.82099999999999995}]}]},{"osd":1,"up_from":12,"seq":51539607610,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7036,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738116,"statfs":{"total":95995035648,"available":95987830784,"internally_reserved":0,"allocated":905216,"data_stored":452779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:40 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.40699999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.58599999999999997,"15min":0.56999999999999995},"min":{"1min":0.46500000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52600000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:34:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.45000000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.57299999999999995,"15min":0.53900000000000003},"min":{"1min":0.52900000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":0.877,"15min":0.877},"last":0.72299999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:16 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.66900000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:34:40 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.41799999999999998,"15min":0.39800000000000002},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.59399999999999997}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.55200000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.751},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.51600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76000000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:34:30 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65500000000000003,"15min":0.65100000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.71499999999999997}]},{"osd":7,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.376,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.89900000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:35:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.93100000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.83499999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.83499999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.006}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.81699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7000,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738152,"statfs":{"total":95995035648,"available":95987867648,"internally_reserved":0,"allocated":868352,"data_stored":419474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:34:38 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40799999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91500000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.56599999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61699999999999999,"15min":0.62},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93300000000000005,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64200000000000002,"15min":0.61799999999999999},"min":{"1min":0.46700000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.022,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72699999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:33:16 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.46600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:34:38 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68700000000000006,"15min":0.66500000000000004},"min":{"1min":0.57499999999999996,"5min":0.504,"15min":0.504},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.64000000000000001,"15min":0.624},"min":{"1min":0.55800000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.747}]},{"osd":5,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.069,"5min":1.141,"15min":1.141},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.03,"5min":1.256,"15min":1.256},"last":0.73999999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:34:28 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.746},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.873},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.76000000000000001,"15min":0.747},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.89200000000000002}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.5,"5min":0.497,"15min":0.497},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85199999999999998},"min":{"1min":0.63500000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.82899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:34:07 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.85999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:34:32 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.94199999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.88600000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":2,"up_from":16,"seq":68719476791,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7076,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738076,"statfs":{"total":95995035648,"available":95987789824,"internally_reserved":0,"allocated":880640,"data_stored":420966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.57299999999999995},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.59199999999999997},"min":{"1min":0.47699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.81299999999999994,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.73199999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.53600000000000003,"15min":0.52200000000000002},"min":{"1min":0.42199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79300000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.51200000000000001},"min":{"1min":0.42199999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.84599999999999997,"5min":0.874,"15min":0.874},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:34:13 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58499999999999996,"15min":0.57899999999999996},"min":{"1min":0.39500000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64200000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.67500000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.55100000000000005}]},{"osd":5,"last update":"Sun Jul 3 10:35:00 2022","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.443,"15min":0.436},"min":{"1min":0.33700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.623}]},{"osd":6,"last update":"Sun Jul 3 10:34:29 2022","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.65100000000000002,"15min":0.64400000000000002},"min":{"1min":0.48399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46800000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.74399999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.51100000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.876}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.69099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":3,"up_from":23,"seq":98784247857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6372,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":880640,"data_stored":419734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.44600000000000001,"15min":0.42399999999999999},"min":{"1min":0.39700000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72699999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.47099999999999997,"15min":0.441},"min":{"1min":0.38800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.503}]},{"osd":1,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59699999999999998,"15min":0.57699999999999996},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.61899999999999999,"15min":0.58499999999999996},"min":{"1min":0.46700000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:34:14 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.68000000000000005,"15min":0.64600000000000002},"min":{"1min":0.438,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.151,"5min":2.581,"15min":2.581},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69499999999999995,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1559999999999999,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.436}]},{"osd":4,"last update":"Sun Jul 3 10:34:37 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.67100000000000004,"15min":0.63100000000000001},"min":{"1min":0.55200000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.67100000000000004,"15min":0.63900000000000001},"min":{"1min":0.56899999999999995,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.61099999999999999}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77800000000000002,"15min":0.73099999999999998},"min":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.125,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78900000000000003,"15min":0.754},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1180000000000001,"5min":2.46,"15min":2.46},"last":0.52100000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:34:26 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.64100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0589999999999999,"5min":1.214,"15min":1.214},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.79400000000000004,"15min":0.77400000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.57199999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.90300000000000002,"15min":0.91100000000000003},"min":{"1min":0.65600000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.159,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88900000000000001,"15min":0.88400000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.76200000000000001}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:34:31 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.79300000000000004}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.746}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018925,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6168,"kb_used_data":848,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738984,"statfs":{"total":95995035648,"available":95988719616,"internally_reserved":0,"allocated":868352,"data_stored":419406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52500000000000002,"15min":0.48999999999999999},"min":{"1min":0.45300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79800000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.52800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63600000000000001}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.498}]},{"osd":2,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56399999999999995,"15min":0.55700000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57599999999999996,"15min":0.55000000000000004},"min":{"1min":0.46000000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.53200000000000003}]},{"osd":5,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.622},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.58099999999999996,"15min":0.56100000000000005},"min":{"1min":0.51400000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.50600000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.83299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66000000000000003,"15min":0.65000000000000002},"min":{"1min":0.51200000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:34:51 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.36799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.38700000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.70999999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:07 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.46300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.872}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.73899999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.91300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":5,"up_from":35,"seq":150323855400,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":925696,"data_stored":1009934,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70899999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66800000000000004}]},{"osd":1,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.42099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78200000000000003,"5min":0.874,"15min":0.874},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53000000000000003,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.315,"15min":0.315},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56499999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.48299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.379},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55600000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39700000000000002}]},{"osd":3,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4129999999999998,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.77600000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.47799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.79200000000000004,"5min":0.874,"15min":0.874},"last":0.62},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53900000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:34:29 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jul 3 10:34:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.63500000000000001}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.753,"15min":0.753},"last":0.48999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:34:34 2022","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.71699999999999997}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":33395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":33589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:35:12.196 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-07-03T10:35:12.197 INFO:tasks.cephadm:Setting up client nodes... 2022-07-03T10:35:12.197 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-07-03T10:35:12.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:12 smithi101 ceph-mon[85126]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:12.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:12 smithi161 ceph-mon[91579]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:12 smithi188 ceph-mon[92425]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:13.100 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:13.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:13 smithi101 ceph-mon[85126]: pgmap v273: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2022-07-03T10:35:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:13 smithi101 ceph-mon[85126]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:13 smithi161 ceph-mon[91579]: pgmap v273: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2022-07-03T10:35:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:13 smithi161 ceph-mon[91579]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:13 smithi188 ceph-mon[92425]: pgmap v273: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5.6 KiB/s wr, 8 op/s 2022-07-03T10:35:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:13 smithi188 ceph-mon[92425]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:35:14.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 -- 172.21.15.101:0/278915150 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 msgr2=0x7fb160071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:14.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 --2- 172.21.15.101:0/278915150 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160071260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb150009900 tx=0x7fb1500091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 -- 172.21.15.101:0/278915150 shutdown_connections 2022-07-03T10:35:14.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 --2- 172.21.15.101:0/278915150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 --2- 172.21.15.101:0/278915150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb1600708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 --2- 172.21.15.101:0/278915150 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160071260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 -- 172.21.15.101:0/278915150 >> 172.21.15.101:0/278915150 conn(0x7fb1600f8c30 msgr2=0x7fb1600fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:14.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 -- 172.21.15.101:0/278915150 shutdown_connections 2022-07-03T10:35:14.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.731+0000 7fb166a12700 1 -- 172.21.15.101:0/278915150 wait complete. 2022-07-03T10:35:14.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 Processor -- start 2022-07-03T10:35:14.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 -- start start 2022-07-03T10:35:14.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:14.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb160110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:14.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160117c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:14.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1601181c0 con 0x7fb160070df0 2022-07-03T10:35:14.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb160118300 con 0x7fb160105430 2022-07-03T10:35:14.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.732+0000 7fb166a12700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1601184c0 con 0x7fb1600723d0 2022-07-03T10:35:14.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:14.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb16520f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb160110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:14.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46288/0 (socket says 172.21.15.101:46288) 2022-07-03T10:35:14.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb16520f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb160110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:44582/0 (socket says 172.21.15.101:44582) 2022-07-03T10:35:14.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 -- 172.21.15.101:0/6011698 learned_addr learned my addr 172.21.15.101:0/6011698 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:14.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb166211700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160117c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:14.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 -- 172.21.15.101:0/6011698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 msgr2=0x7fb160110bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:14.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb160110bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 -- 172.21.15.101:0/6011698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 msgr2=0x7fb160117c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:14.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160117c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:14.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 -- 172.21.15.101:0/6011698 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150009440 con 0x7fb160070df0 2022-07-03T10:35:14.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb166211700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160117c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:35:14.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.733+0000 7fb165a10700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00ea80 tx=0x7fb15c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:14.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.734+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c009e10 con 0x7fb160070df0 2022-07-03T10:35:14.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.734+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1601187a0 con 0x7fb160070df0 2022-07-03T10:35:14.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.734+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb15c00ce80 con 0x7fb160070df0 2022-07-03T10:35:14.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.734+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c010e70 con 0x7fb160070df0 2022-07-03T10:35:14.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.734+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb160118d30 con 0x7fb160070df0 2022-07-03T10:35:14.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.735+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16010ae80 con 0x7fb160070df0 2022-07-03T10:35:14.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.737+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fb15c00c040 con 0x7fb160070df0 2022-07-03T10:35:14.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.738+0000 7fb1567fc700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 0x7fb14c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:14.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.738+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fb15c014070 con 0x7fb160070df0 2022-07-03T10:35:14.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.739+0000 7fb16520f700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 0x7fb14c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:14.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.739+0000 7fb16520f700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 0x7fb14c099210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb150006010 tx=0x7fb1500091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:14.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:14.743+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fb15c04f100 con 0x7fb160070df0 2022-07-03T10:35:15.003 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.000+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb16005f1d0 con 0x7fb160070df0 2022-07-03T10:35:15.007 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.005+0000 7fb1567fc700 1 -- 172.21.15.101:0/6011698 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb15c08cd00 con 0x7fb160070df0 2022-07-03T10:35:15.008 INFO:teuthology.orchestra.run.smithi101.stdout:[client.0] 2022-07-03T10:35:15.008 INFO:teuthology.orchestra.run.smithi101.stdout: key = AQDjcMFiflgnABAA2vhyK7REdunVL3O3Arsz0g== 2022-07-03T10:35:15.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 msgr2=0x7fb14c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:15.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 0x7fb14c099210 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb150006010 tx=0x7fb1500091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 msgr2=0x7fb160110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:15.011 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00ea80 tx=0x7fb15c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 shutdown_connections 2022-07-03T10:35:15.012 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb14c096d60 0x7fb14c099210 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.013 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb160070df0 0x7fb160110690 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.013 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb160105430 0x7fb160117c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.014 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 --2- 172.21.15.101:0/6011698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1600723d0 0x7fb160110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:15.014 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 >> 172.21.15.101:0/6011698 conn(0x7fb1600f8c30 msgr2=0x7fb1600fb010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:15.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 shutdown_connections 2022-07-03T10:35:15.015 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:15.008+0000 7fb166a12700 1 -- 172.21.15.101:0/6011698 wait complete. 2022-07-03T10:35:15.459 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:15 smithi101 ceph-mon[85126]: pgmap v274: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.6 KiB/s wr, 4 op/s 2022-07-03T10:35:15.460 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:15 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-07-03T10:35:15.460 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:15 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-07-03T10:35:15.480 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:35:15.481 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-07-03T10:35:15.481 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-07-03T10:35:15.607 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-07-03T10:35:15.608 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-07-03T10:35:15.608 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph mgr dump --format=json 2022-07-03T10:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:15 smithi161 ceph-mon[91579]: pgmap v274: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.6 KiB/s wr, 4 op/s 2022-07-03T10:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:15 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-07-03T10:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:15 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-07-03T10:35:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:15 smithi188 ceph-mon[92425]: pgmap v274: 97 pgs: 97 active+clean; 610 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.6 KiB/s wr, 4 op/s 2022-07-03T10:35:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:15 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-07-03T10:35:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:15 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/6011698' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-07-03T10:35:16.489 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:17.435 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:17 smithi101 ceph-mon[85126]: pgmap v275: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2022-07-03T10:35:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:17 smithi161 ceph-mon[91579]: pgmap v275: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2022-07-03T10:35:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:17 smithi188 ceph-mon[92425]: pgmap v275: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2022-07-03T10:35:18.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 -- 172.21.15.101:0/3758020842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103530 msgr2=0x7fa2ec103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:18.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 --2- 172.21.15.101:0/3758020842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103530 0x7fa2ec103900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009860 tx=0x7fa2dc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 -- 172.21.15.101:0/3758020842 shutdown_connections 2022-07-03T10:35:18.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 --2- 172.21.15.101:0/3758020842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103530 0x7fa2ec103900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 --2- 172.21.15.101:0/3758020842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec10a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.139 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 --2- 172.21.15.101:0/3758020842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 -- 172.21.15.101:0/3758020842 >> 172.21.15.101:0/3758020842 conn(0x7fa2ec0f90b0 msgr2=0x7fa2ec0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:18.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 -- 172.21.15.101:0/3758020842 shutdown_connections 2022-07-03T10:35:18.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.135+0000 7fa2f086d700 1 -- 172.21.15.101:0/3758020842 wait complete. 2022-07-03T10:35:18.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.136+0000 7fa2f086d700 1 Processor -- start 2022-07-03T10:35:18.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.136+0000 7fa2f086d700 1 -- start start 2022-07-03T10:35:18.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:18.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 0x7fa2ec070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:18.203 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:18.203 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec11cab0 con 0x7fa2ec103e40 2022-07-03T10:35:18.203 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec11cc20 con 0x7fa2ec103530 2022-07-03T10:35:18.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2f086d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ec11cde0 con 0x7fa2ec0fdc80 2022-07-03T10:35:18.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2e37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:18.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2ebfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:18.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 0x7fa2ec070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2e37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:44592/0 (socket says 172.21.15.101:44592) 2022-07-03T10:35:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2e37fe700 1 -- 172.21.15.101:0/1141156423 learned_addr learned my addr 172.21.15.101:0/1141156423 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 -- 172.21.15.101:0/1141156423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 msgr2=0x7fa2ec070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 -- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 msgr2=0x7fa2ec0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:18.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2eb7fe700 1 -- 172.21.15.101:0/1141156423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40097e0 con 0x7fa2ec103530 2022-07-03T10:35:18.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.137+0000 7fa2ebfff700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:18.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2eb7fe700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 0x7fa2ec070d60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec104230 tx=0x7fa2d4005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:18.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4010710 con 0x7fa2ec103530 2022-07-03T10:35:18.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa2d4010d20 con 0x7fa2ec103530 2022-07-03T10:35:18.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc009410 con 0x7fa2ec103530 2022-07-03T10:35:18.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4018df0 con 0x7fa2ec103530 2022-07-03T10:35:18.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.138+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ec11d4c0 con 0x7fa2ec103530 2022-07-03T10:35:18.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.139+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ec04ccb0 con 0x7fa2ec103530 2022-07-03T10:35:18.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.141+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fa2d400c040 con 0x7fa2ec103530 2022-07-03T10:35:18.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.143+0000 7fa2e8ff9700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 0x7fa2d8099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:18.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.143+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fa2d4014070 con 0x7fa2ec103530 2022-07-03T10:35:18.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.143+0000 7fa2e37fe700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 0x7fa2d8099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:18.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.143+0000 7fa2e37fe700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 0x7fa2d8099210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc005ac0 tx=0x7fa2dc005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:18.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.145+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa2d408d1a0 con 0x7fa2ec103530 2022-07-03T10:35:18.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.381+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa2ec10d2d0 con 0x7fa2ec103530 2022-07-03T10:35:18.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.386+0000 7fa2e8ff9700 1 -- 172.21.15.101:0/1141156423 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+241893 (secure 0 0 0) 0x7fa2d4026090 con 0x7fa2ec103530 2022-07-03T10:35:18.393 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:18.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 msgr2=0x7fa2d8099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:18.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 0x7fa2d8099210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc005ac0 tx=0x7fa2dc005040 comp rx=0 tx=0).stop 2022-07-03T10:35:18.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 msgr2=0x7fa2ec070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:18.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 0x7fa2ec070d60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec104230 tx=0x7fa2d4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 shutdown_connections 2022-07-03T10:35:18.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa2d8096d60 0x7fa2d8099210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa2ec103e40 0x7fa2ec0712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.390+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2ec0fdc80 0x7fa2ec070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.391+0000 7fa2f086d700 1 --2- 172.21.15.101:0/1141156423 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2ec103530 0x7fa2ec070d60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:18.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.391+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 >> 172.21.15.101:0/1141156423 conn(0x7fa2ec0f90b0 msgr2=0x7fa2ec0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:18.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.391+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 shutdown_connections 2022-07-03T10:35:18.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:18.391+0000 7fa2f086d700 1 -- 172.21.15.101:0/1141156423 wait complete. 2022-07-03T10:35:18.868 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":17,"active_gid":14154,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":1865562803},{"type":"v1","addr":"172.21.15.101:6801","nonce":1865562803}]},"active_addr":"172.21.15.101:6801/1865562803","active_change":"2022-07-03T10:28:25.795582+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24112,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"z","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.101:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":4046216538}]},{"addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":3886638657}]},{"addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":2173650926}]},{"addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":3608506438}]},{"addrvec":[{"type":"v2","addr":"172.21.15.101:0","nonce":2131692570}]}]}} 2022-07-03T10:35:18.950 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-07-03T10:35:18.951 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-07-03T10:35:18.951 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd dump --format=json 2022-07-03T10:35:19.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:19 smithi101 ceph-mon[85126]: pgmap v276: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 4 op/s 2022-07-03T10:35:19.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:19 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1141156423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-07-03T10:35:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:19 smithi161 ceph-mon[91579]: pgmap v276: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 4 op/s 2022-07-03T10:35:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:19 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1141156423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-07-03T10:35:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:19 smithi188 ceph-mon[92425]: pgmap v276: 97 pgs: 97 active+clean; 613 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.9 KiB/s wr, 4 op/s 2022-07-03T10:35:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:19 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1141156423' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-07-03T10:35:19.842 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:21.289 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:21 smithi101 ceph-mon[85126]: pgmap v277: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2022-07-03T10:35:21.553 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 -- 172.21.15.101:0/1520069791 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac41036b0 msgr2=0x7f7ac4103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:21.553 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/1520069791 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac41036b0 0x7f7ac4103a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009900 tx=0x7f7ac00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.553 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 -- 172.21.15.101:0/1520069791 shutdown_connections 2022-07-03T10:35:21.554 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/1520069791 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac41036b0 0x7f7ac4103a80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.554 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/1520069791 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac40fddc0 0x7f7ac410a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.554 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/1520069791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac4104050 0x7f7ac40fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.555 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 -- 172.21.15.101:0/1520069791 >> 172.21.15.101:0/1520069791 conn(0x7f7ac40f9220 msgr2=0x7f7ac40fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:21.555 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 -- 172.21.15.101:0/1520069791 shutdown_connections 2022-07-03T10:35:21.555 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.551+0000 7f7acc3fc700 1 -- 172.21.15.101:0/1520069791 wait complete. 2022-07-03T10:35:21.557 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 Processor -- start 2022-07-03T10:35:21.557 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 -- start start 2022-07-03T10:35:21.557 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:21.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac41036b0 0x7f7ac4114b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:21.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac4104050 0x7f7ac410e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:21.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac410eda0 con 0x7f7ac40fddc0 2022-07-03T10:35:21.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac410ef10 con 0x7f7ac4104050 2022-07-03T10:35:21.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.552+0000 7f7acc3fc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac410f0d0 con 0x7f7ac41036b0 2022-07-03T10:35:21.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:21.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47640/0 (socket says 172.21.15.101:47640) 2022-07-03T10:35:21.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 -- 172.21.15.101:0/2930504794 learned_addr learned my addr 172.21.15.101:0/2930504794 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:21.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca999700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac4104050 0x7f7ac410e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:21.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7ac9997700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac41036b0 0x7f7ac4114b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:21.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 -- 172.21.15.101:0/2930504794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac41036b0 msgr2=0x7f7ac4114b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:21.561 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac41036b0 0x7f7ac4114b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.561 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 -- 172.21.15.101:0/2930504794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac4104050 msgr2=0x7f7ac410e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:21.561 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac4104050 0x7f7ac410e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.561 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.553+0000 7f7aca198700 1 -- 172.21.15.101:0/2930504794 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc009700 con 0x7f7ac40fddc0 2022-07-03T10:35:21.562 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7aca198700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0003c70 tx=0x7f7ac00091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:21.562 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac001d070 con 0x7f7ac40fddc0 2022-07-03T10:35:21.562 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7ac000b9c0 con 0x7f7ac40fddc0 2022-07-03T10:35:21.562 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0022b60 con 0x7f7ac40fddc0 2022-07-03T10:35:21.563 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0009440 con 0x7f7ac40fddc0 2022-07-03T10:35:21.563 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.554+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac410f5e0 con 0x7f7ac40fddc0 2022-07-03T10:35:21.563 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.555+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac404ccb0 con 0x7f7ac40fddc0 2022-07-03T10:35:21.564 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.557+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f7ac0029030 con 0x7f7ac40fddc0 2022-07-03T10:35:21.565 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.558+0000 7f7abaffd700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 0x7f7ab0099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:21.565 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.558+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f7ac00c0860 con 0x7f7ac40fddc0 2022-07-03T10:35:21.565 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.558+0000 7f7ac9997700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 0x7f7ab0099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:21.566 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.559+0000 7f7ac9997700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 0x7f7ab0099290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ac410fe90 tx=0x7f7ab4005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:21.566 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.562+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f7ac008dc50 con 0x7f7ac40fddc0 2022-07-03T10:35:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:21 smithi161 ceph-mon[91579]: pgmap v277: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2022-07-03T10:35:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:21 smithi188 ceph-mon[92425]: pgmap v277: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.8 KiB/s wr, 5 op/s 2022-07-03T10:35:21.751 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.749+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ac411b8a0 con 0x7f7ac40fddc0 2022-07-03T10:35:21.752 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.750+0000 7f7abaffd700 1 -- 172.21.15.101:0/2930504794 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21420 (secure 0 0 0) 0x7f7ac0027090 con 0x7f7ac40fddc0 2022-07-03T10:35:21.753 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:21.753 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":80,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:04.429266+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:35:21.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.752+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 msgr2=0x7f7ab0099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:21.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.752+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 0x7f7ab0099290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7ac410fe90 tx=0x7f7ab4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.752+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 msgr2=0x7f7ac4114610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:21.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.752+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0003c70 tx=0x7f7ac00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 shutdown_connections 2022-07-03T10:35:21.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7ab0096de0 0x7f7ab0099290 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7ac40fddc0 0x7f7ac4114610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ac4104050 0x7f7ac410e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 --2- 172.21.15.101:0/2930504794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ac41036b0 0x7f7ac4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:21.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 >> 172.21.15.101:0/2930504794 conn(0x7f7ac40f9220 msgr2=0x7f7ac41013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:21.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 shutdown_connections 2022-07-03T10:35:21.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:21.753+0000 7f7acc3fc700 1 -- 172.21.15.101:0/2930504794 wait complete. 2022-07-03T10:35:22.315 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-07-03T10:35:22.315 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd dump --format=json 2022-07-03T10:35:22.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:22 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2930504794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:22 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2930504794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:22 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2930504794' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:23.204 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:23.521 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:23 smithi101 ceph-mon[85126]: pgmap v278: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:23 smithi161 ceph-mon[91579]: pgmap v278: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:23 smithi188 ceph-mon[92425]: pgmap v278: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:24.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 -- 172.21.15.101:0/1107885827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc1036b0 msgr2=0x7efebc103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:24.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 --2- 172.21.15.101:0/1107885827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc1036b0 0x7efebc103a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efeac009900 tx=0x7efeac0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 -- 172.21.15.101:0/1107885827 shutdown_connections 2022-07-03T10:35:24.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 --2- 172.21.15.101:0/1107885827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc1036b0 0x7efebc103a80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 --2- 172.21.15.101:0/1107885827 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc0fddc0 0x7efebc10a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 --2- 172.21.15.101:0/1107885827 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc0fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.717+0000 7efec16b6700 1 -- 172.21.15.101:0/1107885827 >> 172.21.15.101:0/1107885827 conn(0x7efebc0f9220 msgr2=0x7efebc0fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:24.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.718+0000 7efec16b6700 1 -- 172.21.15.101:0/1107885827 shutdown_connections 2022-07-03T10:35:24.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.718+0000 7efec16b6700 1 -- 172.21.15.101:0/1107885827 wait complete. 2022-07-03T10:35:24.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.719+0000 7efec16b6700 1 Processor -- start 2022-07-03T10:35:24.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.719+0000 7efec16b6700 1 -- start start 2022-07-03T10:35:24.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc0fddc0 0x7efebc114620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:24.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:24.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc10e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:24.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc10ed40 con 0x7efebc0fddc0 2022-07-03T10:35:24.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc10eeb0 con 0x7efebc104050 2022-07-03T10:35:24.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efec16b6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc10f070 con 0x7efebc1036b0 2022-07-03T10:35:24.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efeba7fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:24.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efebb7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc10e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:24.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efeba7fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:49832/0 (socket says 172.21.15.101:49832) 2022-07-03T10:35:24.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efeba7fc700 1 -- 172.21.15.101:0/2096821475 learned_addr learned my addr 172.21.15.101:0/2096821475 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:24.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.720+0000 7efebaffd700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc0fddc0 0x7efebc114620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:24.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 -- 172.21.15.101:0/2096821475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 msgr2=0x7efebc114ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:24.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 -- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc0fddc0 msgr2=0x7efebc114620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:24.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc0fddc0 0x7efebc114620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 -- 172.21.15.101:0/2096821475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac009440 con 0x7efebc104050 2022-07-03T10:35:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efeba7fc700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efebb7fe700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc10e6d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efeb400eab0 tx=0x7efeb4009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.721+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4009850 con 0x7efebc104050 2022-07-03T10:35:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.722+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efeb400c9d0 con 0x7efebc104050 2022-07-03T10:35:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.722+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc10f350 con 0x7efebc104050 2022-07-03T10:35:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.722+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4004020 con 0x7efebc104050 2022-07-03T10:35:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.722+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc11b5e0 con 0x7efebc104050 2022-07-03T10:35:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.723+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc04ccb0 con 0x7efebc104050 2022-07-03T10:35:24.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.725+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7efeb401e020 con 0x7efebc104050 2022-07-03T10:35:24.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.726+0000 7efea37fe700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 0x7efea4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:24.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.726+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7efeb4014070 con 0x7efebc104050 2022-07-03T10:35:24.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.726+0000 7efebaffd700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 0x7efea4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:24.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.727+0000 7efebaffd700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 0x7efea4099210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efeac0097d0 tx=0x7efeac0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:24.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.728+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7efeb408d1c0 con 0x7efebc104050 2022-07-03T10:35:24.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.914+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efebc11bbf0 con 0x7efebc104050 2022-07-03T10:35:24.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.915+0000 7efea37fe700 1 -- 172.21.15.101:0/2096821475 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21420 (secure 0 0 0) 0x7efeb40907e0 con 0x7efebc104050 2022-07-03T10:35:24.917 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:24.917 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":80,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:04.429266+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:35:24.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 msgr2=0x7efea4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:24.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 0x7efea4099210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efeac0097d0 tx=0x7efeac0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 msgr2=0x7efebc10e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:24.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc10e6d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efeb400eab0 tx=0x7efeb4009460 comp rx=0 tx=0).stop 2022-07-03T10:35:24.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 shutdown_connections 2022-07-03T10:35:24.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efea4096d60 0x7efea4099210 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efebc0fddc0 0x7efebc114620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efebc104050 0x7efebc10e6d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 --2- 172.21.15.101:0/2096821475 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efebc1036b0 0x7efebc114ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:24.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 >> 172.21.15.101:0/2096821475 conn(0x7efebc0f9220 msgr2=0x7efebc0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:24.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 shutdown_connections 2022-07-03T10:35:24.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:24.918+0000 7efec16b6700 1 -- 172.21.15.101:0/2096821475 wait complete. 2022-07-03T10:35:25.386 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-07-03T10:35:25.387 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-07-03T10:35:25.387 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-07-03T10:35:25.388 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-07-03T10:35:25.389 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-07-03T10:35:25.389 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-07-03T10:35:25.390 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-07-03T10:35:25.391 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-07-03T10:35:25.392 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2022-07-03T10:35:25.393 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2022-07-03T10:35:25.393 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2022-07-03T10:35:25.394 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2022-07-03T10:35:25.558 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:25 smithi101 ceph-mon[85126]: pgmap v279: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:25.559 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:25 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2096821475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:25 smithi161 ceph-mon[91579]: pgmap v279: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:25 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2096821475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:25 smithi188 ceph-mon[92425]: pgmap v279: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2022-07-03T10:35:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:25 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2096821475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:35:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-07-03T10:35:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2022-07-03T10:35:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-07-03T10:35:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2022-07-03T10:35:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-07-03T10:35:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2022-07-03T10:35:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2022-07-03T10:35:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:35:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:35:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-07-03T10:35:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2022-07-03T10:35:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-07-03T10:35:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2022-07-03T10:35:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-07-03T10:35:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2022-07-03T10:35:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2022-07-03T10:35:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:35:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:35:26.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-07-03T10:35:26.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2022-07-03T10:35:26.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-07-03T10:35:26.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2022-07-03T10:35:26.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2022-07-03T10:35:26.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]: dispatch 2022-07-03T10:35:26.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.746 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2022-07-03T10:35:26.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2022-07-03T10:35:26.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:35:26.747 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:35:27.350 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: pgmap v280: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2022-07-03T10:35:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-07-03T10:35:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2022-07-03T10:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-07-03T10:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2022-07-03T10:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-07-03T10:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2022-07-03T10:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2022-07-03T10:35:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2022-07-03T10:35:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2022-07-03T10:35:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:27 smithi161 ceph-mon[91579]: osdmap e81: 12 total, 12 up, 12 in 2022-07-03T10:35:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: pgmap v280: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2022-07-03T10:35:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-07-03T10:35:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2022-07-03T10:35:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-07-03T10:35:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2022-07-03T10:35:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-07-03T10:35:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2022-07-03T10:35:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2022-07-03T10:35:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2022-07-03T10:35:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2022-07-03T10:35:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:27 smithi188 ceph-mon[92425]: osdmap e81: 12 total, 12 up, 12 in 2022-07-03T10:35:27.703 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: pgmap v280: 97 pgs: 97 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2022-07-03T10:35:27.704 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-07-03T10:35:27.704 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2022-07-03T10:35:27.704 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-07-03T10:35:27.705 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2022-07-03T10:35:27.705 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2022-07-03T10:35:27.705 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 6]}]': finished 2022-07-03T10:35:27.705 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2022-07-03T10:35:27.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2022-07-03T10:35:27.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2022-07-03T10:35:27.706 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:27 smithi101 ceph-mon[85126]: osdmap e81: 12 total, 12 up, 12 in 2022-07-03T10:35:27.907 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:27.933 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:27.937 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:28.326 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:28.327 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:28.436 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:28.591 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:28 smithi101 ceph-mon[85126]: osdmap e82: 12 total, 12 up, 12 in 2022-07-03T10:35:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:28 smithi161 ceph-mon[91579]: osdmap e82: 12 total, 12 up, 12 in 2022-07-03T10:35:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:28 smithi188 ceph-mon[92425]: osdmap e82: 12 total, 12 up, 12 in 2022-07-03T10:35:29.489 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:29 smithi101 ceph-mon[85126]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:29 smithi161 ceph-mon[91579]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:29 smithi188 ceph-mon[92425]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 628 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:29.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 -- 172.21.15.101:0/2035007926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c103530 msgr2=0x7f069c103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 --2- 172.21.15.101:0/2035007926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c103530 0x7f069c103900 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0698009900 tx=0x7f06980091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 -- 172.21.15.101:0/2035007926 shutdown_connections 2022-07-03T10:35:29.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 --2- 172.21.15.101:0/2035007926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c103530 0x7f069c103900 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 --2- 172.21.15.101:0/2035007926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f069c0fdc80 0x7f069c10a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 --2- 172.21.15.101:0/2035007926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f069c103e40 0x7f069c0fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 -- 172.21.15.101:0/2035007926 >> 172.21.15.101:0/2035007926 conn(0x7f069c0f90b0 msgr2=0x7f069c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:29.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 -- 172.21.15.101:0/2035007926 shutdown_connections 2022-07-03T10:35:29.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.910+0000 7f06a23c1700 1 -- 172.21.15.101:0/2035007926 wait complete. 2022-07-03T10:35:29.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.911+0000 7f06a23c1700 1 Processor -- start 2022-07-03T10:35:29.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.911+0000 7f06a23c1700 1 -- start start 2022-07-03T10:35:29.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 0x7f069c070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:29.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f069c103530 0x7f069c070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:29.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f069c103e40 0x7f069c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:29.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c11ca60 con 0x7f069c0fdc80 2022-07-03T10:35:29.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c11cbd0 con 0x7f069c103530 2022-07-03T10:35:29.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a23c1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c11cd90 con 0x7f069c103e40 2022-07-03T10:35:29.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a13bf700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 0x7f069c070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:29.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a13bf700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 0x7f069c070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52702/0 (socket says 172.21.15.101:52702) 2022-07-03T10:35:29.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a13bf700 1 -- 172.21.15.101:0/131091860 learned_addr learned my addr 172.21.15.101:0/131091860 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:29.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a1bc0700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f069c103e40 0x7f069c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:29.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.912+0000 7f06a0bbe700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f069c103530 0x7f069c070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:29.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 -- 172.21.15.101:0/131091860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f069c103e40 msgr2=0x7f069c0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f069c103e40 0x7f069c0712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 -- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f069c103530 msgr2=0x7f069c070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f069c103530 0x7f069c070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698009440 con 0x7f069c0fdc80 2022-07-03T10:35:29.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a13bf700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 0x7f069c070820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0698006010 tx=0x7f06980091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:29.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069801d070 con 0x7f069c0fdc80 2022-07-03T10:35:29.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0698000c90 con 0x7f069c0fdc80 2022-07-03T10:35:29.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c11d010 con 0x7f069c0fdc80 2022-07-03T10:35:29.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.913+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698022a10 con 0x7f069c0fdc80 2022-07-03T10:35:29.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.914+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c11d4b0 con 0x7f069c0fdc80 2022-07-03T10:35:29.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.914+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f069c0fb1a0 con 0x7f069c0fdc80 2022-07-03T10:35:29.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.916+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f0698029030 con 0x7f069c0fdc80 2022-07-03T10:35:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f0691ffb700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0688096d60 0x7f0688099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f0698022e40 con 0x7f069c0fdc80 2022-07-03T10:35:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f06a0bbe700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0688096d60 0x7f0688099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f0691ffb700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] conn(0x7f06880a0910 0x7f06880a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:29.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06880a3470 con 0x7f06880a0910 2022-07-03T10:35:29.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.918+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06980c1960 con 0x7f069c0fdc80 2022-07-03T10:35:29.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.919+0000 7f06a0bbe700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0688096d60 0x7f0688099210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f069c072010 tx=0x7f068c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:29.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.919+0000 7f06a1bc0700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] conn(0x7f06880a0910 0x7f06880a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:29.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.919+0000 7f06a1bc0700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] conn(0x7f06880a0910 0x7f06880a2da0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:29.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.920+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== osd.1 v2:172.21.15.161:6800/967237995 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f06880a3470 con 0x7f06880a0910 2022-07-03T10:35:29.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.937+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 --> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f069c04ccb0 con 0x7f06880a0910 2022-07-03T10:35:29.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.937+0000 7f0691ffb700 1 -- 172.21.15.101:0/131091860 <== osd.1 v2:172.21.15.161:6800/967237995 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f069c04ccb0 con 0x7f06880a0910 2022-07-03T10:35:29.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] conn(0x7f06880a0910 msgr2=0x7f06880a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] conn(0x7f06880a0910 0x7f06880a2da0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0688096d60 msgr2=0x7f0688099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0688096d60 0x7f0688099210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f069c072010 tx=0x7f068c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 msgr2=0x7f069c070820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:29.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a23c1700 1 --2- 172.21.15.101:0/131091860 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f069c0fdc80 0x7f069c070820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0698006010 tx=0x7f06980091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:29.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.938+0000 7f06a1bc0700 1 -- 172.21.15.101:0/131091860 reap_dead start 2022-07-03T10:35:29.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.939+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 shutdown_connections 2022-07-03T10:35:29.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.939+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 >> 172.21.15.101:0/131091860 conn(0x7f069c0f90b0 msgr2=0x7f069c0fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:29.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.939+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 shutdown_connections 2022-07-03T10:35:29.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:29.939+0000 7f06a23c1700 1 -- 172.21.15.101:0/131091860 wait complete. 2022-07-03T10:35:31.439 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:31 smithi101 ceph-mon[85126]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:31 smithi161 ceph-mon[91579]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:31 smithi188 ceph-mon[92425]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:31.835 INFO:teuthology.orchestra.run.smithi101.stdout:51539607614 2022-07-03T10:35:31.835 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-07-03T10:35:31.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 -- 172.21.15.101:0/3885757899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52800fddc0 msgr2=0x7f528010a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 --2- 172.21.15.101:0/3885757899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52800fddc0 0x7f528010a4b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5278009ca0 tx=0x7f5278009520 comp rx=0 tx=0).stop 2022-07-03T10:35:31.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 -- 172.21.15.101:0/3885757899 shutdown_connections 2022-07-03T10:35:31.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 --2- 172.21.15.101:0/3885757899 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f52801036b0 0x7f5280103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 --2- 172.21.15.101:0/3885757899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52800fddc0 0x7f528010a4b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.936+0000 7f5287950700 1 --2- 172.21.15.101:0/3885757899 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5280104050 0x7f52800fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.937+0000 7f5287950700 1 -- 172.21.15.101:0/3885757899 >> 172.21.15.101:0/3885757899 conn(0x7f52800f9220 msgr2=0x7f52800fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:31.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.937+0000 7f5287950700 1 -- 172.21.15.101:0/3885757899 shutdown_connections 2022-07-03T10:35:31.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.937+0000 7f5287950700 1 -- 172.21.15.101:0/3885757899 wait complete. 2022-07-03T10:35:31.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.938+0000 7f5287950700 1 Processor -- start 2022-07-03T10:35:31.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.938+0000 7f5287950700 1 -- start start 2022-07-03T10:35:31.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52800fddc0 0x7f52801142c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:31.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52801036b0 0x7f5280114800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:31.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5280104050 0x7f528010e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:31.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528010e8e0 con 0x7f5280104050 2022-07-03T10:35:31.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528010ea50 con 0x7f52800fddc0 2022-07-03T10:35:31.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5287950700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528010ec10 con 0x7f52801036b0 2022-07-03T10:35:31.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5284eeb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52801036b0 0x7f5280114800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:31.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f52856ec700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52800fddc0 0x7f52801142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:31.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5285eed700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5280104050 0x7f528010e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:31.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5284eeb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52801036b0 0x7f5280114800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48036/0 (socket says 172.21.15.101:48036) 2022-07-03T10:35:31.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.939+0000 7f5284eeb700 1 -- 172.21.15.101:0/431532136 learned_addr learned my addr 172.21.15.101:0/431532136 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:31.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 -- 172.21.15.101:0/431532136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52801036b0 msgr2=0x7f5280114800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f52801036b0 0x7f5280114800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 -- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5280104050 msgr2=0x7f528010e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5280104050 0x7f528010e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c009410 con 0x7f52800fddc0 2022-07-03T10:35:31.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f52856ec700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52800fddc0 0x7f52801142c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f527c00ea80 tx=0x7f527c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:31.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c009650 con 0x7f52800fddc0 2022-07-03T10:35:31.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5270009410 con 0x7f52800fddc0 2022-07-03T10:35:31.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.940+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f527c00ee60 con 0x7f52800fddc0 2022-07-03T10:35:31.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.941+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c003ab0 con 0x7f52800fddc0 2022-07-03T10:35:31.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.941+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52780097e0 con 0x7f52800fddc0 2022-07-03T10:35:31.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.941+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5280102c50 con 0x7f52800fddc0 2022-07-03T10:35:31.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.943+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f527c00c040 con 0x7f52800fddc0 2022-07-03T10:35:31.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5275ffb700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f526c096d60 0x7f526c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:31.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f527c014070 con 0x7f52800fddc0 2022-07-03T10:35:31.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5284eeb700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f526c096d60 0x7f526c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:31.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5275ffb700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] conn(0x7f526c0a07a0 0x7f526c0a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:31.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f526c0a3300 con 0x7f526c0a07a0 2022-07-03T10:35:31.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.945+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f527c0c00b0 con 0x7f52800fddc0 2022-07-03T10:35:31.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.946+0000 7f5285eed700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] conn(0x7f526c0a07a0 0x7f526c0a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:31.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.946+0000 7f5284eeb700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f526c096d60 0x7f526c099210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f528010fb40 tx=0x7f5270005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:31.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.946+0000 7f5285eed700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] conn(0x7f526c0a07a0 0x7f526c0a2c30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:31.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.947+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== osd.9 v2:172.21.15.101:6826/1335772894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f526c0a3300 con 0x7f526c0a07a0 2022-07-03T10:35:31.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.964+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 --> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f528004ccb0 con 0x7f526c0a07a0 2022-07-03T10:35:31.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.965+0000 7f5275ffb700 1 -- 172.21.15.101:0/431532136 <== osd.9 v2:172.21.15.101:6826/1335772894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f528004ccb0 con 0x7f526c0a07a0 2022-07-03T10:35:31.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] conn(0x7f526c0a07a0 msgr2=0x7f526c0a2c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] conn(0x7f526c0a07a0 0x7f526c0a2c30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f526c096d60 msgr2=0x7f526c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f526c096d60 0x7f526c099210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f528010fb40 tx=0x7f5270005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52800fddc0 msgr2=0x7f52801142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:31.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 --2- 172.21.15.101:0/431532136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52800fddc0 0x7f52801142c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f527c00ea80 tx=0x7f527c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:31.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5285eed700 1 -- 172.21.15.101:0/431532136 reap_dead start 2022-07-03T10:35:31.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 shutdown_connections 2022-07-03T10:35:31.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 >> 172.21.15.101:0/431532136 conn(0x7f52800f9220 msgr2=0x7f5280101230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:31.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 shutdown_connections 2022-07-03T10:35:31.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:31.966+0000 7f5287950700 1 -- 172.21.15.101:0/431532136 wait complete. 2022-07-03T10:35:32.443 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:32.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.456+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3537187797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04105720 msgr2=0x7f5e04105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.456+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3537187797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04105720 0x7f5e04105af0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009900 tx=0x7f5df40091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.456+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3537187797 shutdown_connections 2022-07-03T10:35:32.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3537187797 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5e040ffe30 0x7f5e0410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3537187797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04105720 0x7f5e04105af0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3537187797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e041060c0 0x7f5e040ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3537187797 >> 172.21.15.101:0/3537187797 conn(0x7f5e040fb290 msgr2=0x7f5e040fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:32.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3537187797 shutdown_connections 2022-07-03T10:35:32.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.457+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3537187797 wait complete. 2022-07-03T10:35:32.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.458+0000 7f5e08ecc700 1 Processor -- start 2022-07-03T10:35:32.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.458+0000 7f5e08ecc700 1 -- start start 2022-07-03T10:35:32.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5e040ffe30 0x7f5e04116650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:32.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 0x7f5e04116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:32.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e041060c0 0x7f5e04110700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:32.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04110d70 con 0x7f5e040ffe30 2022-07-03T10:35:32.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04110ee0 con 0x7f5e04105720 2022-07-03T10:35:32.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e08ecc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041110a0 con 0x7f5e041060c0 2022-07-03T10:35:32.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5dfbfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 0x7f5e04116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:32.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e037fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5e040ffe30 0x7f5e04116650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:32.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5dfbfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 0x7f5e04116bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:49732/0 (socket says 172.21.15.101:49732) 2022-07-03T10:35:32.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5dfbfff700 1 -- 172.21.15.101:0/3709321735 learned_addr learned my addr 172.21.15.101:0/3709321735 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:32.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.459+0000 7f5e03fff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e041060c0 0x7f5e04110700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:32.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 -- 172.21.15.101:0/3709321735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e041060c0 msgr2=0x7f5e04110700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e041060c0 0x7f5e04110700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 -- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5e040ffe30 msgr2=0x7f5e04116650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5e040ffe30 0x7f5e04116650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4009440 con 0x7f5e04105720 2022-07-03T10:35:32.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5dfbfff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 0x7f5e04116bd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00ea80 tx=0x7f5dec005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:32.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.460+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec00c7c0 con 0x7f5e04105720 2022-07-03T10:35:32.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.461+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e04111380 con 0x7f5e04105720 2022-07-03T10:35:32.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.461+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5dec00cdd0 con 0x7f5e04105720 2022-07-03T10:35:32.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.461+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec010df0 con 0x7f5e04105720 2022-07-03T10:35:32.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.461+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0411d610 con 0x7f5e04105720 2022-07-03T10:35:32.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.462+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e040fd220 con 0x7f5e04105720 2022-07-03T10:35:32.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.465+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f5dec00c040 con 0x7f5e04105720 2022-07-03T10:35:32.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.467+0000 7f5e00ff9700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5df0096d60 0x7f5df0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:32.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.467+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f5dec014070 con 0x7f5e04105720 2022-07-03T10:35:32.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.468+0000 7f5e037fe700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5df0096d60 0x7f5df0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:32.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.468+0000 7f5e00ff9700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] conn(0x7f5df00a0910 0x7f5df00a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:32.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.468+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df00a3470 con 0x7f5df00a0910 2022-07-03T10:35:32.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.468+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dec0c1d70 con 0x7f5e04105720 2022-07-03T10:35:32.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.468+0000 7f5e03fff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] conn(0x7f5df00a0910 0x7f5df00a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:32.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.469+0000 7f5e037fe700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5df0096d60 0x7f5df0099210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009f70 tx=0x7f5df40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:32.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.469+0000 7f5e03fff700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] conn(0x7f5df00a0910 0x7f5df00a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:32.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.470+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== osd.3 v2:172.21.15.101:6810/3606989058 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f5df00a3470 con 0x7f5df00a0910 2022-07-03T10:35:32.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.498+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 --> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e04111510 con 0x7f5df00a0910 2022-07-03T10:35:32.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.499+0000 7f5e00ff9700 1 -- 172.21.15.101:0/3709321735 <== osd.3 v2:172.21.15.101:6810/3606989058 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5e04111510 con 0x7f5df00a0910 2022-07-03T10:35:32.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.499+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] conn(0x7f5df00a0910 msgr2=0x7f5df00a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.499+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] conn(0x7f5df00a0910 0x7f5df00a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.499+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5df0096d60 msgr2=0x7f5df0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.499+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5df0096d60 0x7f5df0099210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009f70 tx=0x7f5df40091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 msgr2=0x7f5e04116bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:32.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 --2- 172.21.15.101:0/3709321735 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e04105720 0x7f5e04116bd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00ea80 tx=0x7f5dec005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:32.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e03fff700 1 -- 172.21.15.101:0/3709321735 reap_dead start 2022-07-03T10:35:32.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 shutdown_connections 2022-07-03T10:35:32.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 >> 172.21.15.101:0/3709321735 conn(0x7f5e040fb290 msgr2=0x7f5e040fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:32.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 shutdown_connections 2022-07-03T10:35:32.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:32.500+0000 7f5e08ecc700 1 -- 172.21.15.101:0/3709321735 wait complete. 2022-07-03T10:35:32.739 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:32.740 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:32.750 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:32.753 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:33.199 INFO:teuthology.orchestra.run.smithi101.stdout:266287972379 2022-07-03T10:35:33.199 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.9 2022-07-03T10:35:33.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:33 smithi101 ceph-mon[85126]: pgmap v285: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:33 smithi161 ceph-mon[91579]: pgmap v285: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:33 smithi188 ceph-mon[92425]: pgmap v285: 97 pgs: 1 peering, 96 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:34.303 INFO:teuthology.orchestra.run.smithi101.stdout:98784247862 2022-07-03T10:35:34.303 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-07-03T10:35:34.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 -- 172.21.15.101:0/2845973462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c103920 msgr2=0x7fa50c103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 --2- 172.21.15.101:0/2845973462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c103920 0x7fa50c103cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009860 tx=0x7fa4fc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 -- 172.21.15.101:0/2845973462 shutdown_connections 2022-07-03T10:35:34.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 --2- 172.21.15.101:0/2845973462 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c0fe040 0x7fa50c10a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 --2- 172.21.15.101:0/2845973462 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c103920 0x7fa50c103cf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 --2- 172.21.15.101:0/2845973462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c1042c0 0x7fa50c0fd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 -- 172.21.15.101:0/2845973462 >> 172.21.15.101:0/2845973462 conn(0x7fa50c075880 msgr2=0x7fa50c075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:34.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.585+0000 7fa512251700 1 -- 172.21.15.101:0/2845973462 shutdown_connections 2022-07-03T10:35:34.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.586+0000 7fa512251700 1 -- 172.21.15.101:0/2845973462 wait complete. 2022-07-03T10:35:34.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.587+0000 7fa512251700 1 Processor -- start 2022-07-03T10:35:34.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.587+0000 7fa512251700 1 -- start start 2022-07-03T10:35:34.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c0fe040 0x7fa50c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c103920 0x7fa50c06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c1042c0 0x7fa50c068a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c069030 con 0x7fa50c103920 2022-07-03T10:35:34.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c0691a0 con 0x7fa50c1042c0 2022-07-03T10:35:34.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa512251700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c069360 con 0x7fa50c0fe040 2022-07-03T10:35:34.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.588+0000 7fa50b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c0fe040 0x7fa50c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c1042c0 0x7fa50c068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c0fe040 0x7fa50c06fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48110/0 (socket says 172.21.15.101:48110) 2022-07-03T10:35:34.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50b7fe700 1 -- 172.21.15.101:0/1654453159 learned_addr learned my addr 172.21.15.101:0/1654453159 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:34.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50affd700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c103920 0x7fa50c06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 -- 172.21.15.101:0/1654453159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c0fe040 msgr2=0x7fa50c06fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa50c0fe040 0x7fa50c06fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 -- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c103920 msgr2=0x7fa50c06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c103920 0x7fa50c06ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc009410 con 0x7fa50c1042c0 2022-07-03T10:35:34.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50affd700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa50c103920 0x7fa50c06ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:35:34.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.589+0000 7fa50bfff700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c1042c0 0x7fa50c068a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa50400eab0 tx=0x7fa504009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.590+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50400caa0 con 0x7fa50c1042c0 2022-07-03T10:35:34.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.590+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa50400cc00 con 0x7fa50c1042c0 2022-07-03T10:35:34.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.590+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c069640 con 0x7fa50c1042c0 2022-07-03T10:35:34.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.590+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504004020 con 0x7fa50c1042c0 2022-07-03T10:35:34.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.590+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c06d770 con 0x7fa50c1042c0 2022-07-03T10:35:34.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.591+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa50c0fceb0 con 0x7fa50c1042c0 2022-07-03T10:35:34.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.593+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fa50401e020 con 0x7fa50c1042c0 2022-07-03T10:35:34.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.595+0000 7fa4f3fff700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa4f4096d60 0x7fa4f4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.596+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa504014070 con 0x7fa50c1042c0 2022-07-03T10:35:34.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.596+0000 7fa50b7fe700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa4f4096d60 0x7fa4f4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.596+0000 7fa4f3fff700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] conn(0x7fa4f40a0910 0x7fa4f40a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.596+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4f40a3470 con 0x7fa4f40a0910 2022-07-03T10:35:34.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.596+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5040c0310 con 0x7fa50c1042c0 2022-07-03T10:35:34.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.597+0000 7fa50affd700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] conn(0x7fa4f40a0910 0x7fa4f40a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.597+0000 7fa50b7fe700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa4f4096d60 0x7fa4f4099210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc000e50 tx=0x7fa4fc005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.597+0000 7fa50affd700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] conn(0x7fa4f40a0910 0x7fa4f40a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.598+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== osd.7 v2:172.21.15.161:6816/328327819 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fa4f40a3470 con 0x7fa4f40a0910 2022-07-03T10:35:34.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 -- 172.21.15.101:0/2687176123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 msgr2=0x7f21b00fd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 --2- 172.21.15.101:0/2687176123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b00fd560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009900 tx=0x7f21a40091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 -- 172.21.15.101:0/2687176123 shutdown_connections 2022-07-03T10:35:34.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 --2- 172.21.15.101:0/2687176123 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f21b00fdc90 0x7f21b010a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 --2- 172.21.15.101:0/2687176123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b0103540 0x7f21b0103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 --2- 172.21.15.101:0/2687176123 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b00fd560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 -- 172.21.15.101:0/2687176123 >> 172.21.15.101:0/2687176123 conn(0x7f21b00f9060 msgr2=0x7f21b00fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:34.613 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 -- 172.21.15.101:0/2687176123 shutdown_connections 2022-07-03T10:35:34.613 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.431+0000 7f21af59e700 1 -- 172.21.15.101:0/2687176123 wait complete. 2022-07-03T10:35:34.613 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.432+0000 7f21af59e700 1 Processor -- start 2022-07-03T10:35:34.613 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 -- start start 2022-07-03T10:35:34.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 0x7f21b01166d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f21b0103540 0x7f21b0116c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b01107a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0110d20 con 0x7f21b0103540 2022-07-03T10:35:34.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0110e90 con 0x7f21b0103e50 2022-07-03T10:35:34.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.433+0000 7f21af59e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0111050 con 0x7f21b00fdc90 2022-07-03T10:35:34.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 0x7f21b01166d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21add9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f21b0103540 0x7f21b0116c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 0x7f21b01166d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48088/0 (socket says 172.21.15.101:48088) 2022-07-03T10:35:34.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 -- 172.21.15.101:0/1509690882 learned_addr learned my addr 172.21.15.101:0/1509690882 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:34.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21aed9d700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b01107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 -- 172.21.15.101:0/1509690882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 msgr2=0x7f21b01107a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b01107a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 -- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f21b0103540 msgr2=0x7f21b0116c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f21b0103540 0x7f21b0116c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21ae59c700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4009440 con 0x7f21b00fdc90 2022-07-03T10:35:34.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.434+0000 7f21aed9d700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21b0103e50 0x7f21b01107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:35:34.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f21ae59c700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 0x7f21b01166d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21a000ea80 tx=0x7f21a0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0009e10 con 0x7f21b00fdc90 2022-07-03T10:35:34.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b0111330 con 0x7f21b00fdc90 2022-07-03T10:35:34.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f21a000ce80 con 0x7f21b00fdc90 2022-07-03T10:35:34.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0010e70 con 0x7f21b00fdc90 2022-07-03T10:35:34.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.435+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b011d700 con 0x7f21b00fdc90 2022-07-03T10:35:34.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.436+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21b010a290 con 0x7f21b00fdc90 2022-07-03T10:35:34.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.438+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f21a000c040 con 0x7f21b00fdc90 2022-07-03T10:35:34.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.440+0000 7f219effd700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2198096d60 0x7f2198099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.440+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f21a0014070 con 0x7f21b00fdc90 2022-07-03T10:35:34.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f21add9b700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2198096d60 0x7f2198099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f219effd700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] conn(0x7f21980a0910 0x7f21980a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:34.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21980a3470 con 0x7f21980a0910 2022-07-03T10:35:34.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21a00bfef0 con 0x7f21b00fdc90 2022-07-03T10:35:34.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f21aed9d700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] conn(0x7f21980a0910 0x7f21980a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:34.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.441+0000 7f21add9b700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2198096d60 0x7f2198099210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21a4006010 tx=0x7f21a40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.442+0000 7f21aed9d700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] conn(0x7f21980a0910 0x7f21980a2da0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:34.622 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.443+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== osd.5 v2:172.21.15.188:6808/1424487492 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f21980a3470 con 0x7f21980a0910 2022-07-03T10:35:34.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.463+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 --> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21b004ccb0 con 0x7f21980a0910 2022-07-03T10:35:34.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.463+0000 7f219effd700 1 -- 172.21.15.101:0/1509690882 <== osd.5 v2:172.21.15.188:6808/1424487492 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f21b004ccb0 con 0x7f21980a0910 2022-07-03T10:35:34.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] conn(0x7f21980a0910 msgr2=0x7f21980a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.625 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] conn(0x7f21980a0910 0x7f21980a2da0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.625 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2198096d60 msgr2=0x7f2198099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2198096d60 0x7f2198099210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21a4006010 tx=0x7f21a40091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 msgr2=0x7f21b01166d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21af59e700 1 --2- 172.21.15.101:0/1509690882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21b00fdc90 0x7f21b01166d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21a000ea80 tx=0x7f21a0005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.464+0000 7f21aed9d700 1 -- 172.21.15.101:0/1509690882 reap_dead start 2022-07-03T10:35:34.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.465+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 shutdown_connections 2022-07-03T10:35:34.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.465+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 >> 172.21.15.101:0/1509690882 conn(0x7f21b00f9060 msgr2=0x7f21b0108830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:34.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.466+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 shutdown_connections 2022-07-03T10:35:34.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.466+0000 7f21af59e700 1 -- 172.21.15.101:0/1509690882 wait complete. 2022-07-03T10:35:34.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.631+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 --> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa50c069830 con 0x7fa4f40a0910 2022-07-03T10:35:34.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.631+0000 7fa4f3fff700 1 -- 172.21.15.101:0/1654453159 <== osd.7 v2:172.21.15.161:6816/328327819 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa50c069830 con 0x7fa4f40a0910 2022-07-03T10:35:34.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] conn(0x7fa4f40a0910 msgr2=0x7fa4f40a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] conn(0x7fa4f40a0910 0x7fa4f40a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:34.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa4f4096d60 msgr2=0x7fa4f4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa4f4096d60 0x7fa4f4099210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc000e50 tx=0x7fa4fc005040 comp rx=0 tx=0).stop 2022-07-03T10:35:34.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c1042c0 msgr2=0x7fa50c068a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:34.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 --2- 172.21.15.101:0/1654453159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa50c1042c0 0x7fa50c068a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa50400eab0 tx=0x7fa504009460 comp rx=0 tx=0).stop 2022-07-03T10:35:34.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa50bfff700 1 -- 172.21.15.101:0/1654453159 reap_dead start 2022-07-03T10:35:34.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 shutdown_connections 2022-07-03T10:35:34.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.632+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 >> 172.21.15.101:0/1654453159 conn(0x7fa50c075880 msgr2=0x7fa50c0fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:34.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.633+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 shutdown_connections 2022-07-03T10:35:34.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:34.633+0000 7fa512251700 1 -- 172.21.15.101:0/1654453159 wait complete. 2022-07-03T10:35:35.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:35 smithi101 ceph-mon[85126]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:35 smithi161 ceph-mon[91579]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:35 smithi188 ceph-mon[92425]: pgmap v286: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2022-07-03T10:35:36.804 INFO:teuthology.orchestra.run.smithi101.stdout:150323855406 2022-07-03T10:35:36.805 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-07-03T10:35:36.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 -- 172.21.15.101:0/3332467441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb0104070 msgr2=0x7fcbb00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 --2- 172.21.15.101:0/3332467441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb0104070 0x7fcbb00fd700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009900 tx=0x7fcba00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 -- 172.21.15.101:0/3332467441 shutdown_connections 2022-07-03T10:35:36.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 --2- 172.21.15.101:0/3332467441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb00fdde0 0x7fcbb010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 --2- 172.21.15.101:0/3332467441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbb01036d0 0x7fcbb0103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 --2- 172.21.15.101:0/3332467441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb0104070 0x7fcbb00fd700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 -- 172.21.15.101:0/3332467441 >> 172.21.15.101:0/3332467441 conn(0x7fcbb00f9200 msgr2=0x7fcbb00fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:36.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 -- 172.21.15.101:0/3332467441 shutdown_connections 2022-07-03T10:35:36.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.853+0000 7fcbb8420700 1 -- 172.21.15.101:0/3332467441 wait complete. 2022-07-03T10:35:36.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.854+0000 7fcbb8420700 1 Processor -- start 2022-07-03T10:35:36.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 -- start start 2022-07-03T10:35:36.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb00fdde0 0x7fcbb0070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:36.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb01036d0 0x7fcbb0070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:36.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbb0104070 0x7fcbb00712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:36.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb011cc20 con 0x7fcbb00fdde0 2022-07-03T10:35:36.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb011cd90 con 0x7fcbb01036d0 2022-07-03T10:35:36.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.855+0000 7fcbb8420700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb011cf50 con 0x7fcbb0104070 2022-07-03T10:35:36.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb61bc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb00fdde0 0x7fcbb0070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:36.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb61bc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb00fdde0 0x7fcbb0070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52812/0 (socket says 172.21.15.101:52812) 2022-07-03T10:35:36.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb61bc700 1 -- 172.21.15.101:0/846106455 learned_addr learned my addr 172.21.15.101:0/846106455 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:36.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb01036d0 0x7fcbb0070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:36.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb69bd700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbb0104070 0x7fcbb00712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:36.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 -- 172.21.15.101:0/846106455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbb0104070 msgr2=0x7fcbb00712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbb0104070 0x7fcbb00712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 -- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb00fdde0 msgr2=0x7fcbb0070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbb00fdde0 0x7fcbb0070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.856+0000 7fcbb59bb700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0009440 con 0x7fcbb01036d0 2022-07-03T10:35:36.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcbb59bb700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb01036d0 0x7fcbb0070d60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcba00092d0 tx=0x7fcba00091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:36.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba001d070 con 0x7fcbb01036d0 2022-07-03T10:35:36.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb011d1d0 con 0x7fcbb01036d0 2022-07-03T10:35:36.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcba000b9c0 con 0x7fcbb01036d0 2022-07-03T10:35:36.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0022a90 con 0x7fcbb01036d0 2022-07-03T10:35:36.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.857+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb011d6d0 con 0x7fcbb01036d0 2022-07-03T10:35:36.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.858+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbb00fb380 con 0x7fcbb01036d0 2022-07-03T10:35:36.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.860+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fcba0029030 con 0x7fcbb01036d0 2022-07-03T10:35:36.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcba6ffd700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcb9c096d60 0x7fcb9c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:36.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fcba00c0e30 con 0x7fcbb01036d0 2022-07-03T10:35:36.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcbb61bc700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcb9c096d60 0x7fcb9c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:36.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcba6ffd700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] conn(0x7fcb9c0a0910 0x7fcb9c0a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:36.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb9c0a3470 con 0x7fcb9c0a0910 2022-07-03T10:35:36.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.862+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba00c1180 con 0x7fcbb01036d0 2022-07-03T10:35:36.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.863+0000 7fcbb69bd700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] conn(0x7fcb9c0a0910 0x7fcb9c0a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:36.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.863+0000 7fcbb61bc700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcb9c096d60 0x7fcb9c099210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009d30 tx=0x7fcbac005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:36.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.863+0000 7fcbb69bd700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] conn(0x7fcb9c0a0910 0x7fcb9c0a2da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:36.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.864+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== osd.10 v2:172.21.15.161:6824/1352813899 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fcb9c0a3470 con 0x7fcb9c0a0910 2022-07-03T10:35:36.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.881+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 --> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcbb004ccb0 con 0x7fcb9c0a0910 2022-07-03T10:35:36.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcba6ffd700 1 -- 172.21.15.101:0/846106455 <== osd.10 v2:172.21.15.161:6824/1352813899 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcbb004ccb0 con 0x7fcb9c0a0910 2022-07-03T10:35:36.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] conn(0x7fcb9c0a0910 msgr2=0x7fcb9c0a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] conn(0x7fcb9c0a0910 0x7fcb9c0a2da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcb9c096d60 msgr2=0x7fcb9c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcb9c096d60 0x7fcb9c099210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009d30 tx=0x7fcbac005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb01036d0 msgr2=0x7fcbb0070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:36.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb8420700 1 --2- 172.21.15.101:0/846106455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbb01036d0 0x7fcbb0070d60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcba00092d0 tx=0x7fcba00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:36.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.882+0000 7fcbb69bd700 1 -- 172.21.15.101:0/846106455 reap_dead start 2022-07-03T10:35:36.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.883+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 shutdown_connections 2022-07-03T10:35:36.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.883+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 >> 172.21.15.101:0/846106455 conn(0x7fcbb00f9200 msgr2=0x7fcbb00fa930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:36.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.883+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 shutdown_connections 2022-07-03T10:35:36.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:36.883+0000 7fcbb8420700 1 -- 172.21.15.101:0/846106455 wait complete. 2022-07-03T10:35:36.979 INFO:teuthology.orchestra.run.smithi101.stdout:201863462948 2022-07-03T10:35:36.979 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-07-03T10:35:37.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 -- 172.21.15.101:0/4286585660 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 msgr2=0x7f514c066320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 --2- 172.21.15.101:0/4286585660 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c066320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f513c009890 tx=0x7f513c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 -- 172.21.15.101:0/4286585660 shutdown_connections 2022-07-03T10:35:37.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 --2- 172.21.15.101:0/4286585660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f514c066860 0x7f514c10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 --2- 172.21.15.101:0/4286585660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1036e0 0x7f514c103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 --2- 172.21.15.101:0/4286585660 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c066320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 -- 172.21.15.101:0/4286585660 >> 172.21.15.101:0/4286585660 conn(0x7f514c074450 msgr2=0x7f514c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:37.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.418+0000 7f5153655700 1 -- 172.21.15.101:0/4286585660 shutdown_connections 2022-07-03T10:35:37.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.419+0000 7f5153655700 1 -- 172.21.15.101:0/4286585660 wait complete. 2022-07-03T10:35:37.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.419+0000 7f5153655700 1 Processor -- start 2022-07-03T10:35:37.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 -- start start 2022-07-03T10:35:37.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c0fc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:37.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f514c066860 0x7f514c0fcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:37.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1036e0 0x7f514c0fd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:37.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c11cbb0 con 0x7f514c065eb0 2022-07-03T10:35:37.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c11cd20 con 0x7f514c1036e0 2022-07-03T10:35:37.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.420+0000 7f5153655700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c11cee0 con 0x7f514c066860 2022-07-03T10:35:37.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c0fc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:37.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c0fc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52836/0 (socket says 172.21.15.101:52836) 2022-07-03T10:35:37.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 -- 172.21.15.101:0/3642275255 learned_addr learned my addr 172.21.15.101:0/3642275255 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:37.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f5150bf0700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f514c066860 0x7f514c0fcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:37.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f5151bf2700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1036e0 0x7f514c0fd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:37.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 -- 172.21.15.101:0/3642275255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f514c066860 msgr2=0x7f514c0fcc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f514c066860 0x7f514c0fcc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 -- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1036e0 msgr2=0x7f514c0fd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f514c1036e0 0x7f514c0fd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.421+0000 7f51513f1700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c009410 con 0x7f514c065eb0 2022-07-03T10:35:37.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f51513f1700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c0fc6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f514800ea50 tx=0x7f51480091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:37.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148005020 con 0x7f514c065eb0 2022-07-03T10:35:37.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f514800c7c0 con 0x7f514c065eb0 2022-07-03T10:35:37.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c11d1c0 con 0x7f514c065eb0 2022-07-03T10:35:37.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148004b00 con 0x7f514c065eb0 2022-07-03T10:35:37.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.422+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c11d720 con 0x7f514c065eb0 2022-07-03T10:35:37.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.423+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f514c0fbb10 con 0x7f514c065eb0 2022-07-03T10:35:37.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.427+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f514800c040 con 0x7f514c065eb0 2022-07-03T10:35:37.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.428+0000 7f5141ffb700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5138096d60 0x7f5138099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:37.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.428+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f5148014070 con 0x7f514c065eb0 2022-07-03T10:35:37.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.429+0000 7f5150bf0700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5138096d60 0x7f5138099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:37.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.429+0000 7f5141ffb700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] conn(0x7f51380a0910 0x7f51380a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:37.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.429+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51380a3470 con 0x7f51380a0910 2022-07-03T10:35:37.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.429+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51480c06e0 con 0x7f514c065eb0 2022-07-03T10:35:37.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.429+0000 7f5151bf2700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] conn(0x7f51380a0910 0x7f51380a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:37.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.430+0000 7f5150bf0700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5138096d60 0x7f5138099210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f513c0097b0 tx=0x7f513c005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:37.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.430+0000 7f5151bf2700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] conn(0x7f51380a0910 0x7f51380a2da0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:37.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.431+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== osd.4 v2:172.21.15.161:6808/54950390 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f51380a3470 con 0x7f51380a0910 2022-07-03T10:35:37.451 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.449+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 --> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f514c04ccb0 con 0x7f51380a0910 2022-07-03T10:35:37.451 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.449+0000 7f5141ffb700 1 -- 172.21.15.101:0/3642275255 <== osd.4 v2:172.21.15.161:6808/54950390 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f514c04ccb0 con 0x7f51380a0910 2022-07-03T10:35:37.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] conn(0x7f51380a0910 msgr2=0x7f51380a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] conn(0x7f51380a0910 0x7f51380a2da0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5138096d60 msgr2=0x7f5138099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5138096d60 0x7f5138099210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f513c0097b0 tx=0x7f513c005040 comp rx=0 tx=0).stop 2022-07-03T10:35:37.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 msgr2=0x7f514c0fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:37.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 --2- 172.21.15.101:0/3642275255 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f514c065eb0 0x7f514c0fc6c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f514800ea50 tx=0x7f51480091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:37.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5151bf2700 1 -- 172.21.15.101:0/3642275255 reap_dead start 2022-07-03T10:35:37.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 shutdown_connections 2022-07-03T10:35:37.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.450+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 >> 172.21.15.101:0/3642275255 conn(0x7f514c074450 msgr2=0x7f514c0fb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:37.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.451+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 shutdown_connections 2022-07-03T10:35:37.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:37.451+0000 7f5153655700 1 -- 172.21.15.101:0/3642275255 wait complete. 2022-07-03T10:35:37.599 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:37 smithi101 ceph-mon[85126]: pgmap v287: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 318 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:37 smithi161 ceph-mon[91579]: pgmap v287: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 318 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:37 smithi188 ceph-mon[92425]: pgmap v287: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 318 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:38.069 INFO:teuthology.orchestra.run.smithi101.stdout:128849018931 2022-07-03T10:35:38.069 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-07-03T10:35:38.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 -- 172.21.15.101:0/905387190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4181036b0 msgr2=0x7fc418103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 --2- 172.21.15.101:0/905387190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4181036b0 0x7fc418103a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc414009900 tx=0x7fc4140091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 -- 172.21.15.101:0/905387190 shutdown_connections 2022-07-03T10:35:38.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 --2- 172.21.15.101:0/905387190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4180fddc0 0x7fc41810a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 --2- 172.21.15.101:0/905387190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4181036b0 0x7fc418103a80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 --2- 172.21.15.101:0/905387190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc4180fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 -- 172.21.15.101:0/905387190 >> 172.21.15.101:0/905387190 conn(0x7fc4180f9220 msgr2=0x7fc4180fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:38.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 -- 172.21.15.101:0/905387190 shutdown_connections 2022-07-03T10:35:38.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.094+0000 7fc41f815700 1 -- 172.21.15.101:0/905387190 wait complete. 2022-07-03T10:35:38.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.095+0000 7fc41f815700 1 Processor -- start 2022-07-03T10:35:38.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 -- start start 2022-07-03T10:35:38.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 0x7fc418104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:38.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4181036b0 0x7fc418104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:38.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc418104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:38.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418120fb0 con 0x7fc418104050 2022-07-03T10:35:38.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4181210f0 con 0x7fc4181036b0 2022-07-03T10:35:38.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41f815700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418121280 con 0x7fc4180fddc0 2022-07-03T10:35:38.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41ddb2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc418104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:38.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.096+0000 7fc41d5b1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 0x7fc418104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:38.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41ddb2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc418104fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52856/0 (socket says 172.21.15.101:52856) 2022-07-03T10:35:38.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41ddb2700 1 -- 172.21.15.101:0/1788276767 learned_addr learned my addr 172.21.15.101:0/1788276767 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:38.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 0x7fc418104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48170/0 (socket says 172.21.15.101:48170) 2022-07-03T10:35:38.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41cdb0700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4181036b0 0x7fc418104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:38.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 -- 172.21.15.101:0/1788276767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4181036b0 msgr2=0x7fc418104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4181036b0 0x7fc418104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 -- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 msgr2=0x7fc418104fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc418104fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414009440 con 0x7fc4180fddc0 2022-07-03T10:35:38.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41ddb2700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc418104050 0x7fc418104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:38.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.097+0000 7fc41d5b1700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 0x7fc418104530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc414003540 tx=0x7fc4140091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:38.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.098+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41401d070 con 0x7fc4180fddc0 2022-07-03T10:35:38.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.098+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc41400baa0 con 0x7fc4180fddc0 2022-07-03T10:35:38.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.098+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41400fb20 con 0x7fc4180fddc0 2022-07-03T10:35:38.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.098+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc418121500 con 0x7fc4180fddc0 2022-07-03T10:35:38.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.098+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc418121950 con 0x7fc4180fddc0 2022-07-03T10:35:38.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.099+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc41810a3b0 con 0x7fc4180fddc0 2022-07-03T10:35:38.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.101+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fc414028030 con 0x7fc4180fddc0 2022-07-03T10:35:38.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.102+0000 7fc40dffb700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc404096d60 0x7fc404099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:38.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.102+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fc4140c1030 con 0x7fc4180fddc0 2022-07-03T10:35:38.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.102+0000 7fc41cdb0700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc404096d60 0x7fc404099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:38.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.103+0000 7fc40dffb700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] conn(0x7fc4040a0910 0x7fc4040a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:38.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.103+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4040a3470 con 0x7fc4040a0910 2022-07-03T10:35:38.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.103+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4140c13b0 con 0x7fc4180fddc0 2022-07-03T10:35:38.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.103+0000 7fc41ddb2700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] conn(0x7fc4040a0910 0x7fc4040a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:38.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.103+0000 7fc41cdb0700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc404096d60 0x7fc404099210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc418105d20 tx=0x7fc40800b3a0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:38.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.104+0000 7fc41ddb2700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] conn(0x7fc4040a0910 0x7fc4040a2da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:38.112 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.105+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== osd.2 v2:172.21.15.188:6800/2999852966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fc4040a3470 con 0x7fc4040a0910 2022-07-03T10:35:38.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.122+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 --> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc41804ccb0 con 0x7fc4040a0910 2022-07-03T10:35:38.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.122+0000 7fc40dffb700 1 -- 172.21.15.101:0/1788276767 <== osd.2 v2:172.21.15.188:6800/2999852966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc41804ccb0 con 0x7fc4040a0910 2022-07-03T10:35:38.126 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] conn(0x7fc4040a0910 msgr2=0x7fc4040a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.126 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] conn(0x7fc4040a0910 0x7fc4040a2da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc404096d60 msgr2=0x7fc404099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc404096d60 0x7fc404099210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc418105d20 tx=0x7fc40800b3a0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 msgr2=0x7fc418104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:38.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 --2- 172.21.15.101:0/1788276767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4180fddc0 0x7fc418104530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc414003540 tx=0x7fc4140091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:38.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41ddb2700 1 -- 172.21.15.101:0/1788276767 reap_dead start 2022-07-03T10:35:38.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 shutdown_connections 2022-07-03T10:35:38.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 >> 172.21.15.101:0/1788276767 conn(0x7fc4180f9220 msgr2=0x7fc418108950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:38.129 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 shutdown_connections 2022-07-03T10:35:38.129 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:38.123+0000 7fc41f815700 1 -- 172.21.15.101:0/1788276767 wait complete. 2022-07-03T10:35:38.593 INFO:teuthology.orchestra.run.smithi101.stdout:287762808855 2022-07-03T10:35:38.594 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.10 2022-07-03T10:35:38.902 INFO:teuthology.orchestra.run.smithi101.stdout:68719476798 2022-07-03T10:35:38.903 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-07-03T10:35:39.464 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:39 smithi101 ceph-mon[85126]: pgmap v288: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 288 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:39 smithi161 ceph-mon[91579]: pgmap v288: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 288 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:39 smithi188 ceph-mon[92425]: pgmap v288: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 288 B/s wr, 2 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:35:39.794 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:39.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 -- 172.21.15.101:0/1974407514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef940fdc80 msgr2=0x7fef9410a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 --2- 172.21.15.101:0/1974407514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef940fdc80 0x7fef9410a2d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009ca0 tx=0x7fef8c009520 comp rx=0 tx=0).stop 2022-07-03T10:35:39.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 -- 172.21.15.101:0/1974407514 shutdown_connections 2022-07-03T10:35:39.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 --2- 172.21.15.101:0/1974407514 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef94103530 0x7fef94103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 --2- 172.21.15.101:0/1974407514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef940fdc80 0x7fef9410a2d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 --2- 172.21.15.101:0/1974407514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef94103e40 0x7fef940fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 -- 172.21.15.101:0/1974407514 >> 172.21.15.101:0/1974407514 conn(0x7fef940f90b0 msgr2=0x7fef940fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:39.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 -- 172.21.15.101:0/1974407514 shutdown_connections 2022-07-03T10:35:39.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.863+0000 7fef99168700 1 -- 172.21.15.101:0/1974407514 wait complete. 2022-07-03T10:35:39.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.865+0000 7fef99168700 1 Processor -- start 2022-07-03T10:35:39.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.865+0000 7fef99168700 1 -- start start 2022-07-03T10:35:39.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef940fdc80 0x7fef94114200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.870 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef94103530 0x7fef94114740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef94103e40 0x7fef9410e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9410e820 con 0x7fef94103e40 2022-07-03T10:35:39.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9410e990 con 0x7fef94103530 2022-07-03T10:35:39.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef99168700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9410eb50 con 0x7fef940fdc80 2022-07-03T10:35:39.871 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef93fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef940fdc80 0x7fef94114200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef93fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef940fdc80 0x7fef94114200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48034/0 (socket says 172.21.15.101:48034) 2022-07-03T10:35:39.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef93fff700 1 -- 172.21.15.101:0/2236514885 learned_addr learned my addr 172.21.15.101:0/2236514885 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:39.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.866+0000 7fef937fe700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef94103530 0x7fef94114740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.872 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef98967700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef94103e40 0x7fef9410e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 -- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef940fdc80 msgr2=0x7fef94114200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef940fdc80 0x7fef94114200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 -- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef94103e40 msgr2=0x7fef9410e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef94103e40 0x7fef9410e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.873 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c0097e0 con 0x7fef94103530 2022-07-03T10:35:39.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.867+0000 7fef937fe700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef94103530 0x7fef94114740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fef94104230 tx=0x7fef88005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.874 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.868+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88009440 con 0x7fef94103530 2022-07-03T10:35:39.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.868+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fef88010bc0 con 0x7fef94103530 2022-07-03T10:35:39.875 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.868+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88018d90 con 0x7fef94103530 2022-07-03T10:35:39.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.868+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9410ee30 con 0x7fef94103530 2022-07-03T10:35:39.876 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.868+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9410f110 con 0x7fef94103530 2022-07-03T10:35:39.877 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.869+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef940fae50 con 0x7fef94103530 2022-07-03T10:35:39.878 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.875+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fef88028070 con 0x7fef94103530 2022-07-03T10:35:39.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.876+0000 7fef90ff9700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fef7c096d60 0x7fef7c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.876+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fef88014070 con 0x7fef94103530 2022-07-03T10:35:39.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef93fff700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fef7c096d60 0x7fef7c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef90ff9700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] conn(0x7fef7c0a0910 0x7fef7c0a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef7c0a3470 con 0x7fef7c0a0910 2022-07-03T10:35:39.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef880c0dd0 con 0x7fef94103530 2022-07-03T10:35:39.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef98967700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] conn(0x7fef7c0a0910 0x7fef7c0a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.877+0000 7fef93fff700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fef7c096d60 0x7fef7c099210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fef840097d0 tx=0x7fef84005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.878+0000 7fef98967700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] conn(0x7fef7c0a0910 0x7fef7c0a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.879+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== osd.11 v2:172.21.15.188:6824/1726462750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fef7c0a3470 con 0x7fef7c0a0910 2022-07-03T10:35:39.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.895+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 --> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef9404ccb0 con 0x7fef7c0a0910 2022-07-03T10:35:39.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.895+0000 7fef90ff9700 1 -- 172.21.15.101:0/2236514885 <== osd.11 v2:172.21.15.188:6824/1726462750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fef9404ccb0 con 0x7fef7c0a0910 2022-07-03T10:35:39.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.896+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] conn(0x7fef7c0a0910 msgr2=0x7fef7c0a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.896+0000 7fef99168700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] conn(0x7fef7c0a0910 0x7fef7c0a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fef7c096d60 msgr2=0x7fef7c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fef7c096d60 0x7fef7c099210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fef840097d0 tx=0x7fef84005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef94103530 msgr2=0x7fef94114740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 --2- 172.21.15.101:0/2236514885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fef94103530 0x7fef94114740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fef94104230 tx=0x7fef88005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.900 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef98967700 1 -- 172.21.15.101:0/2236514885 reap_dead start 2022-07-03T10:35:39.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 shutdown_connections 2022-07-03T10:35:39.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.897+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 >> 172.21.15.101:0/2236514885 conn(0x7fef940f90b0 msgr2=0x7fef940fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:39.901 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.898+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 shutdown_connections 2022-07-03T10:35:39.902 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.898+0000 7fef99168700 1 -- 172.21.15.101:0/2236514885 wait complete. 2022-07-03T10:35:39.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.945+0000 7fead188a700 1 -- 172.21.15.101:0/980776294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc104a80 msgr2=0x7feacc110920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.945+0000 7fead188a700 1 --2- 172.21.15.101:0/980776294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc104a80 0x7feacc110920 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feabc009900 tx=0x7feabc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.946+0000 7fead188a700 1 -- 172.21.15.101:0/980776294 shutdown_connections 2022-07-03T10:35:39.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.946+0000 7fead188a700 1 --2- 172.21.15.101:0/980776294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc104a80 0x7feacc110920 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.946+0000 7fead188a700 1 --2- 172.21.15.101:0/980776294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109af0 0x7feacc109ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.946+0000 7fead188a700 1 --2- 172.21.15.101:0/980776294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feacc10a490 0x7feacc104350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.946+0000 7fead188a700 1 -- 172.21.15.101:0/980776294 >> 172.21.15.101:0/980776294 conn(0x7feacc0732c0 msgr2=0x7feacc0756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:39.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.947+0000 7fead188a700 1 -- 172.21.15.101:0/980776294 shutdown_connections 2022-07-03T10:35:39.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.947+0000 7fead188a700 1 -- 172.21.15.101:0/980776294 wait complete. 2022-07-03T10:35:39.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.949+0000 7fead188a700 1 Processor -- start 2022-07-03T10:35:39.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.949+0000 7fead188a700 1 -- start start 2022-07-03T10:35:39.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feacc104a80 0x7feacc11a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109af0 0x7feacc11adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 0x7feacc114960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc114f30 con 0x7feacc10a490 2022-07-03T10:35:39.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc1150a0 con 0x7feacc104a80 2022-07-03T10:35:39.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead188a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc115260 con 0x7feacc109af0 2022-07-03T10:35:39.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead1089700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 0x7feacc114960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead1089700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 0x7feacc114960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:55902/0 (socket says 172.21.15.101:55902) 2022-07-03T10:35:39.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.950+0000 7fead1089700 1 -- 172.21.15.101:0/253209787 learned_addr learned my addr 172.21.15.101:0/253209787 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:39.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.951+0000 7feacbfff700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109af0 0x7feacc11adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.951+0000 7fead1089700 1 -- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109af0 msgr2=0x7feacc11adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.951+0000 7fead1089700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feacc109af0 0x7feacc11adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7fead1089700 1 -- 172.21.15.101:0/253209787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feacc104a80 msgr2=0x7feacc11a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7fead1089700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feacc104a80 0x7feacc11a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7fead1089700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc009440 con 0x7feacc10a490 2022-07-03T10:35:39.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7fead1089700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 0x7feacc114960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feabc0097d0 tx=0x7feabc0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc01d070 con 0x7feacc10a490 2022-07-03T10:35:39.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7feabc005580 con 0x7feacc10a490 2022-07-03T10:35:39.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.952+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc1154e0 con 0x7feacc10a490 2022-07-03T10:35:39.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.953+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc022420 con 0x7feacc10a490 2022-07-03T10:35:39.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.953+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc1157b0 con 0x7feacc10a490 2022-07-03T10:35:39.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.953+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feacc1091d0 con 0x7feacc10a490 2022-07-03T10:35:39.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.957+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7feabc00c040 con 0x7feacc10a490 2022-07-03T10:35:39.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.959+0000 7feac97fa700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feab4096d60 0x7feab4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.960+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7feabc0c93a0 con 0x7feacc10a490 2022-07-03T10:35:39.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.960+0000 7feac97fa700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] conn(0x7feab40a0840 0x7feab40a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:39.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.960+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feab40a33a0 con 0x7feab40a0840 2022-07-03T10:35:39.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.960+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7feabc0c96a0 con 0x7feacc10a490 2022-07-03T10:35:39.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.961+0000 7fead0888700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feab4096d60 0x7feab4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.962+0000 7feacbfff700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] conn(0x7feab40a0840 0x7feab40a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:39.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.962+0000 7fead0888700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feab4096d60 0x7feab4099210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feacc0767e0 tx=0x7feac0005c00 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.962+0000 7feacbfff700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] conn(0x7feab40a0840 0x7feab40a2cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:39.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.963+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== osd.8 v2:172.21.15.188:6816/1483136090 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7feab40a33a0 con 0x7feab40a0840 2022-07-03T10:35:39.970 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:39.990 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.988+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 --> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feacc04ccb0 con 0x7feab40a0840 2022-07-03T10:35:39.995 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.992+0000 7feac97fa700 1 -- 172.21.15.101:0/253209787 <== osd.8 v2:172.21.15.188:6816/1483136090 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feacc04ccb0 con 0x7feab40a0840 2022-07-03T10:35:39.996 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] conn(0x7feab40a0840 msgr2=0x7feab40a2cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.997 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] conn(0x7feab40a0840 0x7feab40a2cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.997 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feab4096d60 msgr2=0x7feab4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7feab4096d60 0x7feab4099210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feacc0767e0 tx=0x7feac0005c00 comp rx=0 tx=0).stop 2022-07-03T10:35:39.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 msgr2=0x7feacc114960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:39.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.995+0000 7fead188a700 1 --2- 172.21.15.101:0/253209787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feacc10a490 0x7feacc114960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feabc0097d0 tx=0x7feabc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:39.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.997+0000 7fead1089700 1 -- 172.21.15.101:0/253209787 reap_dead start 2022-07-03T10:35:39.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.997+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 shutdown_connections 2022-07-03T10:35:40.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.997+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 >> 172.21.15.101:0/253209787 conn(0x7feacc0732c0 msgr2=0x7feacc107770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:40.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.998+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 shutdown_connections 2022-07-03T10:35:40.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:39.999+0000 7fead188a700 1 -- 172.21.15.101:0/253209787 wait complete. 2022-07-03T10:35:40.805 INFO:teuthology.orchestra.run.smithi101.stdout:322122547219 2022-07-03T10:35:40.805 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.11 2022-07-03T10:35:40.912 INFO:teuthology.orchestra.run.smithi101.stdout:231928234018 2022-07-03T10:35:40.912 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.8 2022-07-03T10:35:41.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 -- 172.21.15.101:0/1610277209 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 msgr2=0x7f5aec10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/1610277209 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 0x7f5aec10a2e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009ca0 tx=0x7f5ae4009520 comp rx=0 tx=0).stop 2022-07-03T10:35:41.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 -- 172.21.15.101:0/1610277209 shutdown_connections 2022-07-03T10:35:41.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/1610277209 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 0x7f5aec10a2e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/1610277209 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/1610277209 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 0x7f5aec0fd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 -- 172.21.15.101:0/1610277209 >> 172.21.15.101:0/1610277209 conn(0x7f5aec0f9060 msgr2=0x7f5aec0fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:41.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 -- 172.21.15.101:0/1610277209 shutdown_connections 2022-07-03T10:35:41.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.614+0000 7f5af0ab7700 1 -- 172.21.15.101:0/1610277209 wait complete. 2022-07-03T10:35:41.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.615+0000 7f5af0ab7700 1 Processor -- start 2022-07-03T10:35:41.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 -- start start 2022-07-03T10:35:41.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 0x7f5aec1141d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:41.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec114710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:41.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 0x7f5aec10e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:41.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec10e7f0 con 0x7f5aec0fdc90 2022-07-03T10:35:41.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec10e960 con 0x7f5aec103e50 2022-07-03T10:35:41.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.616+0000 7f5af0ab7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec10eb20 con 0x7f5aec103540 2022-07-03T10:35:41.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec114710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:41.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec114710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48078/0 (socket says 172.21.15.101:48078) 2022-07-03T10:35:41.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 -- 172.21.15.101:0/2394873069 learned_addr learned my addr 172.21.15.101:0/2394873069 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:41.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aebfff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 0x7f5aec10e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:41.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeb7fe700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 0x7f5aec1141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:41.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 -- 172.21.15.101:0/2394873069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 msgr2=0x7f5aec10e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 0x7f5aec10e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 -- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 msgr2=0x7f5aec1141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5aec0fdc90 0x7f5aec1141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aeaffd700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40097e0 con 0x7f5aec103540 2022-07-03T10:35:41.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.617+0000 7f5aebfff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec103e50 0x7f5aec10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:41.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5aeaffd700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec114710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000e830 tx=0x7f5ae0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:41.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0010710 con 0x7f5aec103540 2022-07-03T10:35:41.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec10ee00 con 0x7f5aec103540 2022-07-03T10:35:41.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5ae0010d20 con 0x7f5aec103540 2022-07-03T10:35:41.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0018d10 con 0x7f5aec103540 2022-07-03T10:35:41.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.618+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec0709b0 con 0x7f5aec103540 2022-07-03T10:35:41.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.619+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5aec102ae0 con 0x7f5aec103540 2022-07-03T10:35:41.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.621+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f5ae000c040 con 0x7f5aec103540 2022-07-03T10:35:41.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5ad3fff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5ad4096d60 0x7f5ad4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:41.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f5ae0014070 con 0x7f5aec103540 2022-07-03T10:35:41.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5aeb7fe700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5ad4096d60 0x7f5ad4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:41.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5ad3fff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] conn(0x7f5ad40a0910 0x7f5ad40a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:41.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ad40a3470 con 0x7f5ad40a0910 2022-07-03T10:35:41.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.622+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae00bfda0 con 0x7f5aec103540 2022-07-03T10:35:41.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.623+0000 7f5aebfff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] conn(0x7f5ad40a0910 0x7f5ad40a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:41.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.623+0000 7f5aeb7fe700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5ad4096d60 0x7f5ad4099210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009d00 tx=0x7f5adc005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:41.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.623+0000 7f5aebfff700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] conn(0x7f5ad40a0910 0x7f5ad40a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:41.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.624+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== osd.0 v2:172.21.15.101:6802/2068504891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f5ad40a3470 con 0x7f5ad40a0910 2022-07-03T10:35:41.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.640+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 --> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5aec04ccb0 con 0x7f5ad40a0910 2022-07-03T10:35:41.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.641+0000 7f5ad3fff700 1 -- 172.21.15.101:0/2394873069 <== osd.0 v2:172.21.15.101:6802/2068504891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5aec04ccb0 con 0x7f5ad40a0910 2022-07-03T10:35:41.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.641+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] conn(0x7f5ad40a0910 msgr2=0x7f5ad40a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.641+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] conn(0x7f5ad40a0910 0x7f5ad40a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5ad4096d60 msgr2=0x7f5ad4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5ad4096d60 0x7f5ad4099210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009d00 tx=0x7f5adc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 msgr2=0x7f5aec114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:41.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 --2- 172.21.15.101:0/2394873069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5aec103540 0x7f5aec114710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000e830 tx=0x7f5ae0005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:41.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5aebfff700 1 -- 172.21.15.101:0/2394873069 reap_dead start 2022-07-03T10:35:41.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 shutdown_connections 2022-07-03T10:35:41.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 >> 172.21.15.101:0/2394873069 conn(0x7f5aec0f9060 msgr2=0x7f5aec101080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:41.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 shutdown_connections 2022-07-03T10:35:41.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:40.642+0000 7f5af0ab7700 1 -- 172.21.15.101:0/2394873069 wait complete. 2022-07-03T10:35:41.522 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:41 smithi161 ceph-mon[91579]: pgmap v289: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:41 smithi188 ceph-mon[92425]: pgmap v289: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:41.738 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:41 smithi101 ceph-mon[85126]: pgmap v289: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:42.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.046+0000 7fd303fff700 1 -- 172.21.15.101:0/2055546553 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 msgr2=0x7fd30410c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.046+0000 7fd303fff700 1 --2- 172.21.15.101:0/2055546553 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd30410c510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009ca0 tx=0x7fd2fc009520 comp rx=0 tx=0).stop 2022-07-03T10:35:42.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 -- 172.21.15.101:0/2055546553 shutdown_connections 2022-07-03T10:35:42.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 --2- 172.21.15.101:0/2055546553 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd30410c510 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 --2- 172.21.15.101:0/2055546553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 0x7fd304105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 --2- 172.21.15.101:0/2055546553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041060b0 0x7fd3040ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 -- 172.21.15.101:0/2055546553 >> 172.21.15.101:0/2055546553 conn(0x7fd3040fb260 msgr2=0x7fd3040fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:42.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 -- 172.21.15.101:0/2055546553 shutdown_connections 2022-07-03T10:35:42.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.047+0000 7fd303fff700 1 -- 172.21.15.101:0/2055546553 wait complete. 2022-07-03T10:35:42.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.048+0000 7fd303fff700 1 Processor -- start 2022-07-03T10:35:42.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.048+0000 7fd303fff700 1 -- start start 2022-07-03T10:35:42.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.048+0000 7fd303fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd304116350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:42.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.048+0000 7fd303fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 0x7fd304116890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:42.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd303fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041060b0 0x7fd3041103e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:42.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd303fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304110970 con 0x7fd3040ffe20 2022-07-03T10:35:42.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd303fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304110ae0 con 0x7fd3041060b0 2022-07-03T10:35:42.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd303fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304110ca0 con 0x7fd304105710 2022-07-03T10:35:42.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd304116350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:42.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd304116350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:55936/0 (socket says 172.21.15.101:55936) 2022-07-03T10:35:42.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 -- 172.21.15.101:0/4102556464 learned_addr learned my addr 172.21.15.101:0/4102556464 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:42.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd3027fc700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 0x7fd304116890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:42.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd3037fe700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041060b0 0x7fd3041103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:42.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 -- 172.21.15.101:0/4102556464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 msgr2=0x7fd304116890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 0x7fd304116890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 -- 172.21.15.101:0/4102556464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041060b0 msgr2=0x7fd3041103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd3041060b0 0x7fd3041103e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.049+0000 7fd302ffd700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc0097e0 con 0x7fd3040ffe20 2022-07-03T10:35:42.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd3027fc700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd304105710 0x7fd304116890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:42.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd302ffd700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd304116350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ea80 tx=0x7fd2f4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:42.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400c7c0 con 0x7fd3040ffe20 2022-07-03T10:35:42.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd2f400cdd0 con 0x7fd3040ffe20 2022-07-03T10:35:42.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd304110f80 con 0x7fd3040ffe20 2022-07-03T10:35:42.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4010df0 con 0x7fd3040ffe20 2022-07-03T10:35:42.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.050+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30411d340 con 0x7fd3040ffe20 2022-07-03T10:35:42.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.051+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd304104cb0 con 0x7fd3040ffe20 2022-07-03T10:35:42.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.054+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd2f400c040 con 0x7fd3040ffe20 2022-07-03T10:35:42.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd2eb7fe700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd2ec096d60 0x7fd2ec099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:42.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd2f4014070 con 0x7fd3040ffe20 2022-07-03T10:35:42.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd3027fc700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd2ec096d60 0x7fd2ec099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:42.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd2eb7fe700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] conn(0x7fd2ec0a0910 0x7fd2ec0a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:42.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2ec0a3470 con 0x7fd2ec0a0910 2022-07-03T10:35:42.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.055+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2f40c0360 con 0x7fd3040ffe20 2022-07-03T10:35:42.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.056+0000 7fd3037fe700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] conn(0x7fd2ec0a0910 0x7fd2ec0a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:42.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.056+0000 7fd3027fc700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd2ec096d60 0x7fd2ec099210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd304111bd0 tx=0x7fd2f8005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:42.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.057+0000 7fd3037fe700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] conn(0x7fd2ec0a0910 0x7fd2ec0a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:42.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.058+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== osd.6 v2:172.21.15.101:6818/3209986330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fd2ec0a3470 con 0x7fd2ec0a0910 2022-07-03T10:35:42.077 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.075+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 --> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd30404ccb0 con 0x7fd2ec0a0910 2022-07-03T10:35:42.078 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.075+0000 7fd2eb7fe700 1 -- 172.21.15.101:0/4102556464 <== osd.6 v2:172.21.15.101:6818/3209986330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd30404ccb0 con 0x7fd2ec0a0910 2022-07-03T10:35:42.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] conn(0x7fd2ec0a0910 msgr2=0x7fd2ec0a2da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] conn(0x7fd2ec0a0910 0x7fd2ec0a2da0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd2ec096d60 msgr2=0x7fd2ec099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd2ec096d60 0x7fd2ec099210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd304111bd0 tx=0x7fd2f8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 msgr2=0x7fd304116350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:42.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 --2- 172.21.15.101:0/4102556464 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd3040ffe20 0x7fd304116350 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ea80 tx=0x7fd2f4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:42.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd3037fe700 1 -- 172.21.15.101:0/4102556464 reap_dead start 2022-07-03T10:35:42.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 shutdown_connections 2022-07-03T10:35:42.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.076+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 >> 172.21.15.101:0/4102556464 conn(0x7fd3040fb260 msgr2=0x7fd304103290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:42.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.077+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 shutdown_connections 2022-07-03T10:35:42.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:42.077+0000 7fd303fff700 1 -- 172.21.15.101:0/4102556464 wait complete. 2022-07-03T10:35:42.761 INFO:teuthology.orchestra.run.smithi101.stdout:184683593769 2022-07-03T10:35:42.762 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-07-03T10:35:43.081 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:43.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:43 smithi101 ceph-mon[85126]: pgmap v290: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:43 smithi161 ceph-mon[91579]: pgmap v290: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:43 smithi188 ceph-mon[92425]: pgmap v290: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:35:44.181 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:44.289 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:44.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 -- 172.21.15.101:0/827796063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff0073360 msgr2=0x7f0ff010e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 --2- 172.21.15.101:0/827796063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff0073360 0x7f0ff010e070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009930 tx=0x7f0fe00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 -- 172.21.15.101:0/827796063 shutdown_connections 2022-07-03T10:35:44.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 --2- 172.21.15.101:0/827796063 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 0x7f0ff0072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 --2- 172.21.15.101:0/827796063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0110990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 --2- 172.21.15.101:0/827796063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff0073360 0x7f0ff010e070 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.317+0000 7f0ff8359700 1 -- 172.21.15.101:0/827796063 >> 172.21.15.101:0/827796063 conn(0x7f0ff00fb410 msgr2=0x7f0ff00fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:44.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.318+0000 7f0ff8359700 1 -- 172.21.15.101:0/827796063 shutdown_connections 2022-07-03T10:35:44.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.318+0000 7f0ff8359700 1 -- 172.21.15.101:0/827796063 wait complete. 2022-07-03T10:35:44.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.319+0000 7f0ff8359700 1 Processor -- start 2022-07-03T10:35:44.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.319+0000 7f0ff8359700 1 -- start start 2022-07-03T10:35:44.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 0x7f0ff011aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff01152c0 con 0x7f0ff0072a50 2022-07-03T10:35:44.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0115430 con 0x7f0ff010e5b0 2022-07-03T10:35:44.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff8359700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff01155f0 con 0x7f0ff0073360 2022-07-03T10:35:44.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff58f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff58f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48132/0 (socket says 172.21.15.101:48132) 2022-07-03T10:35:44.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff58f4700 1 -- 172.21.15.101:0/2971439592 learned_addr learned my addr 172.21.15.101:0/2971439592 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:44.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.320+0000 7f0ff60f5700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 0x7f0ff011aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff68f6700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 -- 172.21.15.101:0/2971439592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 msgr2=0x7f0ff0114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0114d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 -- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 msgr2=0x7f0ff011aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 0x7f0ff011aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 -- 172.21.15.101:0/2971439592 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec009410 con 0x7f0ff0073360 2022-07-03T10:35:44.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff68f6700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:35:44.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.321+0000 7f0ff58f4700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0003fd0 tx=0x7f0fe00091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:44.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.322+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe001d070 con 0x7f0ff0073360 2022-07-03T10:35:44.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.322+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe0009440 con 0x7f0ff0073360 2022-07-03T10:35:44.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.322+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff0121ab0 con 0x7f0ff0073360 2022-07-03T10:35:44.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.322+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0fe000bbd0 con 0x7f0ff0073360 2022-07-03T10:35:44.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.322+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0022a60 con 0x7f0ff0073360 2022-07-03T10:35:44.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.323+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd80051a0 con 0x7f0ff0073360 2022-07-03T10:35:44.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.325+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f0fe0029030 con 0x7f0ff0073360 2022-07-03T10:35:44.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.327+0000 7f0fe6ffd700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 0x7f0fdc099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.327+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f0fe00c0d50 con 0x7f0ff0073360 2022-07-03T10:35:44.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.327+0000 7f0ff60f5700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 0x7f0fdc099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.328+0000 7f0ff60f5700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 0x7f0fdc099290 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fec009e90 tx=0x7f0fec005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:44.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.329+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f0fe008df50 con 0x7f0ff0073360 2022-07-03T10:35:44.398 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:44.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.586+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f0fd8005b40 con 0x7f0ff0073360 2022-07-03T10:35:44.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.587+0000 7f0fe6ffd700 1 -- 172.21.15.101:0/2971439592 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0fe0027020 con 0x7f0ff0073360 2022-07-03T10:35:44.590 INFO:teuthology.orchestra.run.smithi101.stdout:266287972382 2022-07-03T10:35:44.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 msgr2=0x7f0fdc099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 0x7f0fdc099290 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fec009e90 tx=0x7f0fec005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 msgr2=0x7f0ff011b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0003fd0 tx=0x7f0fe00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 shutdown_connections 2022-07-03T10:35:44.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0fdc096de0 0x7f0fdc099290 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0ff0072a50 0x7f0ff011aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ff010e5b0 0x7f0ff0114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 --2- 172.21.15.101:0/2971439592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ff0073360 0x7f0ff011b1e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 >> 172.21.15.101:0/2971439592 conn(0x7f0ff00fb410 msgr2=0x7f0ff010f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:44.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.590+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 shutdown_connections 2022-07-03T10:35:44.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.591+0000 7f0ff8359700 1 -- 172.21.15.101:0/2971439592 wait complete. 2022-07-03T10:35:44.634 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:44.635 INFO:teuthology.orchestra.run.smithi101.stdout:34359738437 2022-07-03T10:35:44.635 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-07-03T10:35:44.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 -- 172.21.15.101:0/3775822163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 msgr2=0x7fa8140fd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3775822163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa8140fd550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa804009900 tx=0x7fa8040091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 -- 172.21.15.101:0/3775822163 shutdown_connections 2022-07-03T10:35:44.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3775822163 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa81410a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3775822163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 0x7fa814103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3775822163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa8140fd550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 -- 172.21.15.101:0/3775822163 >> 172.21.15.101:0/3775822163 conn(0x7fa8140f90b0 msgr2=0x7fa8140fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:44.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 -- 172.21.15.101:0/3775822163 shutdown_connections 2022-07-03T10:35:44.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.937+0000 7fa81aef2700 1 -- 172.21.15.101:0/3775822163 wait complete. 2022-07-03T10:35:44.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.938+0000 7fa81aef2700 1 Processor -- start 2022-07-03T10:35:44.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.938+0000 7fa81aef2700 1 -- start start 2022-07-03T10:35:44.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 0x7fa814114a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa81410e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410eb80 con 0x7fa814103e40 2022-07-03T10:35:44.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410ecf0 con 0x7fa8140fdc80 2022-07-03T10:35:44.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81aef2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81410eeb0 con 0x7fa814103530 2022-07-03T10:35:44.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa819ef0700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa81a6f1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa81410e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa819ef0700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:52764/0 (socket says 172.21.15.101:52764) 2022-07-03T10:35:44.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa819ef0700 1 -- 172.21.15.101:0/3678862240 learned_addr learned my addr 172.21.15.101:0/3678862240 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:44.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.939+0000 7fa8196ef700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 0x7fa814114a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 -- 172.21.15.101:0/3678862240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 msgr2=0x7fa814114a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 0x7fa814114a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 -- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 msgr2=0x7fa81410e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:44.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa81410e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:44.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 -- 172.21.15.101:0/3678862240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810009410 con 0x7fa8140fdc80 2022-07-03T10:35:44.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa819ef0700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa81000ea80 tx=0x7fa810005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:44.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.940+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810009690 con 0x7fa8140fdc80 2022-07-03T10:35:44.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.941+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa81000c970 con 0x7fa8140fdc80 2022-07-03T10:35:44.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.941+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa804009440 con 0x7fa8140fdc80 2022-07-03T10:35:44.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.941+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa810003b40 con 0x7fa8140fdc80 2022-07-03T10:35:44.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.941+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81411b2e0 con 0x7fa8140fdc80 2022-07-03T10:35:44.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.942+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81404ccb0 con 0x7fa8140fdc80 2022-07-03T10:35:44.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.944+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fa81000c040 con 0x7fa8140fdc80 2022-07-03T10:35:44.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.946+0000 7fa80a7fc700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 0x7fa800099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:44.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.946+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa810014070 con 0x7fa8140fdc80 2022-07-03T10:35:44.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.946+0000 7fa8196ef700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 0x7fa800099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:44.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.947+0000 7fa8196ef700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 0x7fa800099290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa8040092d0 tx=0x7fa8040091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:44.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:44.947+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa81008cf70 con 0x7fa8140fdc80 2022-07-03T10:35:45.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.166+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa81410d2b0 con 0x7fa8140fdc80 2022-07-03T10:35:45.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.167+0000 7fa80a7fc700 1 -- 172.21.15.101:0/3678862240 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa810090590 con 0x7fa8140fdc80 2022-07-03T10:35:45.169 INFO:teuthology.orchestra.run.smithi101.stdout:51539607617 2022-07-03T10:35:45.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.170+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 msgr2=0x7fa800099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:45.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.170+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 0x7fa800099290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa8040092d0 tx=0x7fa8040091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.170+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 msgr2=0x7fa814114500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:45.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.170+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa81000ea80 tx=0x7fa810005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 shutdown_connections 2022-07-03T10:35:45.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa800096de0 0x7fa800099290 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa814103e40 0x7fa81410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8140fdc80 0x7fa814114500 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 --2- 172.21.15.101:0/3678862240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa814103530 0x7fa814114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 >> 172.21.15.101:0/3678862240 conn(0x7fa8140f90b0 msgr2=0x7fa814101270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 shutdown_connections 2022-07-03T10:35:45.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:45.171+0000 7fa81aef2700 1 -- 172.21.15.101:0/3678862240 wait complete. 2022-07-03T10:35:45.610 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:45 smithi101 ceph-mon[85126]: pgmap v291: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:45.611 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:45 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2971439592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-07-03T10:35:45.611 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:45 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3678862240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-07-03T10:35:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:45 smithi161 ceph-mon[91579]: pgmap v291: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:45 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2971439592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-07-03T10:35:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:45 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3678862240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-07-03T10:35:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:45 smithi188 ceph-mon[92425]: pgmap v291: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:45 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2971439592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-07-03T10:35:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:45 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3678862240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-07-03T10:35:46.563 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972379 got 266287972382 for osd.9 2022-07-03T10:35:46.564 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:46.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 -- 172.21.15.101:0/979033138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80708c0 msgr2=0x7faaf8070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:46.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 --2- 172.21.15.101:0/979033138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80708c0 0x7faaf8070c90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faaec009900 tx=0x7faaec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 -- 172.21.15.101:0/979033138 shutdown_connections 2022-07-03T10:35:46.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 --2- 172.21.15.101:0/979033138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80708c0 0x7faaf8070c90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 --2- 172.21.15.101:0/979033138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 0x7faaf8110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 --2- 172.21.15.101:0/979033138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80711d0 0x7faaf810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.785+0000 7faafed6a700 1 -- 172.21.15.101:0/979033138 >> 172.21.15.101:0/979033138 conn(0x7faaf80f9220 msgr2=0x7faaf80fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:46.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.786+0000 7faafed6a700 1 -- 172.21.15.101:0/979033138 shutdown_connections 2022-07-03T10:35:46.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.786+0000 7faafed6a700 1 -- 172.21.15.101:0/979033138 wait complete. 2022-07-03T10:35:46.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.787+0000 7faafed6a700 1 Processor -- start 2022-07-03T10:35:46.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.787+0000 7faafed6a700 1 -- start start 2022-07-03T10:35:46.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:46.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:46.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 0x7faaf8114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:46.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf81152b0 con 0x7faaf80711d0 2022-07-03T10:35:46.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8115420 con 0x7faaf80708c0 2022-07-03T10:35:46.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafed6a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf81155e0 con 0x7faaf810e6f0 2022-07-03T10:35:46.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafcb06700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:46.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faaf7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:46.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faaf7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:56000/0 (socket says 172.21.15.101:56000) 2022-07-03T10:35:46.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faaf7fff700 1 -- 172.21.15.101:0/1932902400 learned_addr learned my addr 172.21.15.101:0/1932902400 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:46.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.788+0000 7faafcb06700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:52788/0 (socket says 172.21.15.101:52788) 2022-07-03T10:35:46.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faafd307700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 0x7faaf8114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:46.796 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 -- 172.21.15.101:0/1932902400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 msgr2=0x7faaf8114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:46.797 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 0x7faaf8114d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.797 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 -- 172.21.15.101:0/1932902400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 msgr2=0x7faaf811ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:46.797 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.798 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 -- 172.21.15.101:0/1932902400 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec009440 con 0x7faaf80711d0 2022-07-03T10:35:46.798 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faafcb06700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:46.798 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.789+0000 7faaf7fff700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faae400ea80 tx=0x7faae4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:46.798 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.790+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4009e10 con 0x7faaf80711d0 2022-07-03T10:35:46.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.790+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81158c0 con 0x7faaf80711d0 2022-07-03T10:35:46.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.790+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faae400ce80 con 0x7faaf80711d0 2022-07-03T10:35:46.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.790+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4010e70 con 0x7faaf80711d0 2022-07-03T10:35:46.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.790+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8121c80 con 0x7faaf80711d0 2022-07-03T10:35:46.801 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.792+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf804ccb0 con 0x7faaf80711d0 2022-07-03T10:35:46.801 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.793+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7faae400c040 con 0x7faaf80711d0 2022-07-03T10:35:46.801 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.794+0000 7faaf57fa700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 0x7faae8099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:46.801 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.794+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7faae4014070 con 0x7faaf80711d0 2022-07-03T10:35:46.801 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.794+0000 7faafcb06700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 0x7faae8099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:46.802 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.795+0000 7faafcb06700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 0x7faae8099210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faaec009f70 tx=0x7faaec0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:46.802 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.798+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7faae404f100 con 0x7faaf80711d0 2022-07-03T10:35:46.996 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.993+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faaf8116280 con 0x7faaf80711d0 2022-07-03T10:35:46.997 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.994+0000 7faaf57fa700 1 -- 172.21.15.101:0/1932902400 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7faae408cd60 con 0x7faaf80711d0 2022-07-03T10:35:46.997 INFO:teuthology.orchestra.run.smithi101.stdout:150323855408 2022-07-03T10:35:46.998 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 msgr2=0x7faae8099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:46.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 0x7faae8099210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faaec009f70 tx=0x7faaec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 msgr2=0x7faaf811b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:46.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faae400ea80 tx=0x7faae4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:46.999 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 shutdown_connections 2022-07-03T10:35:47.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faae8096d60 0x7faae8099210 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:47.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.996+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faaf810e6f0 0x7faaf8114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:47.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.997+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faaf80708c0 0x7faaf811ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:47.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.997+0000 7faafed6a700 1 --2- 172.21.15.101:0/1932902400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80711d0 0x7faaf811b1d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:47.000 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.997+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 >> 172.21.15.101:0/1932902400 conn(0x7faaf80f9220 msgr2=0x7faaf80f9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:47.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.997+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 shutdown_connections 2022-07-03T10:35:47.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:46.997+0000 7faafed6a700 1 -- 172.21.15.101:0/1932902400 wait complete. 2022-07-03T10:35:47.514 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:47.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607614 got 51539607617 for osd.1 2022-07-03T10:35:47.555 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:47 smithi161 ceph-mon[91579]: pgmap v292: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:47 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1932902400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-07-03T10:35:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:47 smithi188 ceph-mon[92425]: pgmap v292: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:47 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1932902400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-07-03T10:35:47.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:47 smithi101 ceph-mon[85126]: pgmap v292: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:47.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:47 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1932902400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-07-03T10:35:48.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855406 got 150323855408 for osd.5 2022-07-03T10:35:48.607 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:49.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 -- 172.21.15.101:0/386213961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b4105720 msgr2=0x7f57b4105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 --2- 172.21.15.101:0/386213961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b4105720 0x7f57b4105af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009900 tx=0x7f57b00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 -- 172.21.15.101:0/386213961 shutdown_connections 2022-07-03T10:35:49.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 --2- 172.21.15.101:0/386213961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b40ffe30 0x7f57b410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 --2- 172.21.15.101:0/386213961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b4105720 0x7f57b4105af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 --2- 172.21.15.101:0/386213961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 0x7f57b40ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 -- 172.21.15.101:0/386213961 >> 172.21.15.101:0/386213961 conn(0x7f57b40fb290 msgr2=0x7f57b40fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:49.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.197+0000 7f57bb40c700 1 -- 172.21.15.101:0/386213961 shutdown_connections 2022-07-03T10:35:49.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.198+0000 7f57bb40c700 1 -- 172.21.15.101:0/386213961 wait complete. 2022-07-03T10:35:49.204 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.198+0000 7f57bb40c700 1 Processor -- start 2022-07-03T10:35:49.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 -- start start 2022-07-03T10:35:49.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 0x7f57b41165f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.205 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 0x7f57b41106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4110c70 con 0x7f57b4105720 2022-07-03T10:35:49.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4110de0 con 0x7f57b40ffe30 2022-07-03T10:35:49.206 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57bb40c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4110fa0 con 0x7f57b41060c0 2022-07-03T10:35:49.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.199+0000 7f57b9c09700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 0x7f57b41165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57b9c09700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52206/0 (socket says 172.21.15.101:52206) 2022-07-03T10:35:49.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57b9c09700 1 -- 172.21.15.101:0/3563960411 learned_addr learned my addr 172.21.15.101:0/3563960411 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:49.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57bac0b700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 0x7f57b41106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 -- 172.21.15.101:0/3563960411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 msgr2=0x7f57b41106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 0x7f57b41106a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 -- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 msgr2=0x7f57b4116b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 -- 172.21.15.101:0/3563960411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0009440 con 0x7f57b40ffe30 2022-07-03T10:35:49.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57b9c09700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:49.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.200+0000 7f57ba40a700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 0x7f57b41165f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f57b00092d0 tx=0x7f57b00091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.201+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b001d070 con 0x7f57b40ffe30 2022-07-03T10:35:49.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.201+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b4111220 con 0x7f57b40ffe30 2022-07-03T10:35:49.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.201+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f57b0003fd0 con 0x7f57b40ffe30 2022-07-03T10:35:49.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.201+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0022ac0 con 0x7f57b40ffe30 2022-07-03T10:35:49.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.201+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b41114f0 con 0x7f57b40ffe30 2022-07-03T10:35:49.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.202+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b404ccb0 con 0x7f57b40ffe30 2022-07-03T10:35:49.213 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.204+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f57b0029030 con 0x7f57b40ffe30 2022-07-03T10:35:49.213 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.205+0000 7f57aaffd700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 0x7f57a0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.205+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f57b00c0b60 con 0x7f57b40ffe30 2022-07-03T10:35:49.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.205+0000 7f57b9c09700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 0x7f57a0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.206+0000 7f57b9c09700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 0x7f57a0099210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57b4111e70 tx=0x7f57a4005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.209+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f57b008ddc0 con 0x7f57b40ffe30 2022-07-03T10:35:49.226 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 -- 172.21.15.101:0/219272422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 msgr2=0x7fde08103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.227 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 --2- 172.21.15.101:0/219272422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08103900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009900 tx=0x7fddf80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.227 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 -- 172.21.15.101:0/219272422 shutdown_connections 2022-07-03T10:35:49.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 --2- 172.21.15.101:0/219272422 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde080fdc80 0x7fde0810a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 --2- 172.21.15.101:0/219272422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08103900 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 --2- 172.21.15.101:0/219272422 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde08103e40 0x7fde080fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 -- 172.21.15.101:0/219272422 >> 172.21.15.101:0/219272422 conn(0x7fde080f90b0 msgr2=0x7fde080fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:49.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.223+0000 7fde07fff700 1 -- 172.21.15.101:0/219272422 shutdown_connections 2022-07-03T10:35:49.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.224+0000 7fde07fff700 1 -- 172.21.15.101:0/219272422 wait complete. 2022-07-03T10:35:49.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.225+0000 7fde07fff700 1 Processor -- start 2022-07-03T10:35:49.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.225+0000 7fde07fff700 1 -- start start 2022-07-03T10:35:49.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08103e40 0x7fde08104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde08120f40 con 0x7fde080fdc80 2022-07-03T10:35:49.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081106e0 con 0x7fde08103e40 2022-07-03T10:35:49.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde07fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081108a0 con 0x7fde08103530 2022-07-03T10:35:49.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde06ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde067fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde067fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:57394/0 (socket says 172.21.15.101:57394) 2022-07-03T10:35:49.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde06ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52210/0 (socket says 172.21.15.101:52210) 2022-07-03T10:35:49.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde06ffd700 1 -- 172.21.15.101:0/1063693288 learned_addr learned my addr 172.21.15.101:0/1063693288 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:49.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.226+0000 7fde077fe700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08103e40 0x7fde08104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 -- 172.21.15.101:0/1063693288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 msgr2=0x7fde08104890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 -- 172.21.15.101:0/1063693288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08103e40 msgr2=0x7fde08104dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08103e40 0x7fde08104dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 -- 172.21.15.101:0/1063693288 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc009410 con 0x7fde080fdc80 2022-07-03T10:35:49.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde067fc700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:49.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.227+0000 7fde06ffd700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fddf8003c70 tx=0x7fddf80091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.228+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf801d070 con 0x7fde080fdc80 2022-07-03T10:35:49.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.228+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fddf800b9c0 con 0x7fde080fdc80 2022-07-03T10:35:49.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.228+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf8009440 con 0x7fde080fdc80 2022-07-03T10:35:49.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.228+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8022b60 con 0x7fde080fdc80 2022-07-03T10:35:49.237 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.228+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde08110e10 con 0x7fde080fdc80 2022-07-03T10:35:49.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.231+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fddf8029030 con 0x7fde080fdc80 2022-07-03T10:35:49.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.232+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0804ccb0 con 0x7fde080fdc80 2022-07-03T10:35:49.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.234+0000 7fddef7fe700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 0x7fddf0099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.234+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fddf80c0a60 con 0x7fde080fdc80 2022-07-03T10:35:49.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.234+0000 7fde067fc700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 0x7fddf0099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.235+0000 7fde067fc700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 0x7fddf0099290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde08105b40 tx=0x7fddfc005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.243 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.241+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fddf808dc60 con 0x7fde080fdc80 2022-07-03T10:35:49.249 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:49.389 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:49.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.436+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f57b411da30 con 0x7f57b40ffe30 2022-07-03T10:35:49.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.436+0000 7f57aaffd700 1 -- 172.21.15.101:0/3563960411 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f57b0027020 con 0x7f57b40ffe30 2022-07-03T10:35:49.439 INFO:teuthology.orchestra.run.smithi101.stdout:287762808857 2022-07-03T10:35:49.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 msgr2=0x7f57a0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 0x7f57a0099210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57b4111e70 tx=0x7f57a4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 msgr2=0x7f57b41165f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 0x7f57b41165f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f57b00092d0 tx=0x7f57b00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 shutdown_connections 2022-07-03T10:35:49.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57a0096d60 0x7f57a0099210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57b41060c0 0x7f57b41106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57b40ffe30 0x7f57b41165f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 --2- 172.21.15.101:0/3563960411 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57b4105720 0x7f57b4116b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.439+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 >> 172.21.15.101:0/3563960411 conn(0x7f57b40fb290 msgr2=0x7f57b40fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:49.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.440+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 shutdown_connections 2022-07-03T10:35:49.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.440+0000 7f57bb40c700 1 -- 172.21.15.101:0/3563960411 wait complete. 2022-07-03T10:35:49.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.457+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fde08105880 con 0x7fde080fdc80 2022-07-03T10:35:49.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.457+0000 7fddef7fe700 1 -- 172.21.15.101:0/1063693288 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddf8027020 con 0x7fde080fdc80 2022-07-03T10:35:49.460 INFO:teuthology.orchestra.run.smithi101.stdout:98784247865 2022-07-03T10:35:49.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 msgr2=0x7fddf0099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 0x7fddf0099290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde08105b40 tx=0x7fddfc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 msgr2=0x7fde08104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fddf8003c70 tx=0x7fddf80091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 shutdown_connections 2022-07-03T10:35:49.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fddf0096de0 0x7fddf0099290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fde080fdc80 0x7fde08104350 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde08103e40 0x7fde08104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 --2- 172.21.15.101:0/1063693288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde08103530 0x7fde08104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 >> 172.21.15.101:0/1063693288 conn(0x7fde080f90b0 msgr2=0x7fde080fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:49.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 shutdown_connections 2022-07-03T10:35:49.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.461+0000 7fde07fff700 1 -- 172.21.15.101:0/1063693288 wait complete. 2022-07-03T10:35:49.616 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:49 smithi101 ceph-mon[85126]: pgmap v293: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:49 smithi161 ceph-mon[91579]: pgmap v293: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:49 smithi188 ceph-mon[92425]: pgmap v293: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:49.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.909+0000 7f6de9e57700 1 -- 172.21.15.101:0/779977759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de40fddc0 msgr2=0x7f6de410a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.909+0000 7f6de9e57700 1 --2- 172.21.15.101:0/779977759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de40fddc0 0x7f6de410a4b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009ca0 tx=0x7f6ddc009520 comp rx=0 tx=0).stop 2022-07-03T10:35:49.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 -- 172.21.15.101:0/779977759 shutdown_connections 2022-07-03T10:35:49.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 --2- 172.21.15.101:0/779977759 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de41036b0 0x7f6de4103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 --2- 172.21.15.101:0/779977759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de40fddc0 0x7f6de410a4b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 --2- 172.21.15.101:0/779977759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de4104050 0x7f6de40fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 -- 172.21.15.101:0/779977759 >> 172.21.15.101:0/779977759 conn(0x7f6de40f9220 msgr2=0x7f6de40fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:49.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 -- 172.21.15.101:0/779977759 shutdown_connections 2022-07-03T10:35:49.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.910+0000 7f6de9e57700 1 -- 172.21.15.101:0/779977759 wait complete. 2022-07-03T10:35:49.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 Processor -- start 2022-07-03T10:35:49.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 -- start start 2022-07-03T10:35:49.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 0x7f6de4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 0x7f6de40712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4072750 con 0x7f6de40fddc0 2022-07-03T10:35:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4120ef0 con 0x7f6de4104050 2022-07-03T10:35:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.911+0000 7f6de9e57700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4121030 con 0x7f6de41036b0 2022-07-03T10:35:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52236/0 (socket says 172.21.15.101:52236) 2022-07-03T10:35:49.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 -- 172.21.15.101:0/677918645 learned_addr learned my addr 172.21.15.101:0/677918645 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:49.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de2ffd700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 0x7f6de4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de3fff700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 0x7f6de40712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 -- 172.21.15.101:0/677918645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 msgr2=0x7f6de4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 0x7f6de4070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 -- 172.21.15.101:0/677918645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 msgr2=0x7f6de40712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:49.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 0x7f6de40712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:49.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de37fe700 1 -- 172.21.15.101:0/677918645 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc0097e0 con 0x7f6de40fddc0 2022-07-03T10:35:49.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de2ffd700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 0x7f6de4070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:49.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.912+0000 7f6de3fff700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 0x7f6de40712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:49.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6de37fe700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400ea80 tx=0x7f6dd4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd400c7c0 con 0x7f6de40fddc0 2022-07-03T10:35:49.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6dd400cdd0 con 0x7f6de40fddc0 2022-07-03T10:35:49.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de4121310 con 0x7f6de40fddc0 2022-07-03T10:35:49.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4010d40 con 0x7f6de40fddc0 2022-07-03T10:35:49.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.913+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de4121870 con 0x7f6de40fddc0 2022-07-03T10:35:49.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.914+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de404ccb0 con 0x7f6de40fddc0 2022-07-03T10:35:49.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.916+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f6dd400c040 con 0x7f6de40fddc0 2022-07-03T10:35:49.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.918+0000 7f6dcbfff700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 0x7f6dcc099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:49.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.918+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f6dd4014070 con 0x7f6de40fddc0 2022-07-03T10:35:49.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.919+0000 7f6de2ffd700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 0x7f6dcc099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:49.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.920+0000 7f6de2ffd700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 0x7f6dcc099290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6de4072010 tx=0x7f6dd8005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:49.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:49.922+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f6dd408d180 con 0x7f6de40fddc0 2022-07-03T10:35:50.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.135+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6de410d450 con 0x7f6de40fddc0 2022-07-03T10:35:50.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.136+0000 7f6dcbfff700 1 -- 172.21.15.101:0/677918645 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6dd4026090 con 0x7f6de40fddc0 2022-07-03T10:35:50.139 INFO:teuthology.orchestra.run.smithi101.stdout:68719476800 2022-07-03T10:35:50.140 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 msgr2=0x7f6dcc099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:50.140 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 0x7f6dcc099290 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6de4072010 tx=0x7f6dd8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 msgr2=0x7f6de4070820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:50.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400ea80 tx=0x7f6dd4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 shutdown_connections 2022-07-03T10:35:50.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6dcc096de0 0x7f6dcc099290 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f6de40fddc0 0x7f6de4070820 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6de4104050 0x7f6de40712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 --2- 172.21.15.101:0/677918645 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de41036b0 0x7f6de4070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 >> 172.21.15.101:0/677918645 conn(0x7f6de40f9220 msgr2=0x7f6de4101230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:50.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 shutdown_connections 2022-07-03T10:35:50.143 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.138+0000 7f6de9e57700 1 -- 172.21.15.101:0/677918645 wait complete. 2022-07-03T10:35:50.360 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247862 got 98784247865 for osd.3 2022-07-03T10:35:50.360 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:50.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3563960411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-07-03T10:35:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1063693288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-07-03T10:35:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/677918645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-07-03T10:35:50.613 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808855 got 287762808857 for osd.10 2022-07-03T10:35:50.613 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:50 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3563960411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-07-03T10:35:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:50 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1063693288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-07-03T10:35:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:50 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/677918645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-07-03T10:35:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:50 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3563960411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-07-03T10:35:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:50 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1063693288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-07-03T10:35:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:50 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/677918645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-07-03T10:35:50.812 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 -- 172.21.15.101:0/3237882441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97840fdc90 msgr2=0x7f978410a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:50.813 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 --2- 172.21.15.101:0/3237882441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97840fdc90 0x7f978410a2e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f977c009ca0 tx=0x7f977c009520 comp rx=0 tx=0).stop 2022-07-03T10:35:50.813 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 -- 172.21.15.101:0/3237882441 shutdown_connections 2022-07-03T10:35:50.813 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 --2- 172.21.15.101:0/3237882441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 --2- 172.21.15.101:0/3237882441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97840fdc90 0x7f978410a2e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 --2- 172.21.15.101:0/3237882441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9784103e50 0x7f97840fd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 -- 172.21.15.101:0/3237882441 >> 172.21.15.101:0/3237882441 conn(0x7f97840f9060 msgr2=0x7f97840fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:50.814 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 -- 172.21.15.101:0/3237882441 shutdown_connections 2022-07-03T10:35:50.815 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.810+0000 7f978a85a700 1 -- 172.21.15.101:0/3237882441 wait complete. 2022-07-03T10:35:50.817 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.811+0000 7f978a85a700 1 Processor -- start 2022-07-03T10:35:50.817 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.812+0000 7f978a85a700 1 -- start start 2022-07-03T10:35:50.817 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.812+0000 7f978a85a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 0x7f97841163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:50.817 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.812+0000 7f978a85a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:50.818 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f978a85a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9784103e50 0x7f9784110460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:50.818 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f978a85a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97841109f0 con 0x7f9784103540 2022-07-03T10:35:50.818 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f978a85a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9784110b60 con 0x7f9784103e50 2022-07-03T10:35:50.819 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f978a85a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9784110d20 con 0x7f97840fdc90 2022-07-03T10:35:50.819 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:50.819 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789858700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 0x7f97841163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:50.819 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52254/0 (socket says 172.21.15.101:52254) 2022-07-03T10:35:50.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 -- 172.21.15.101:0/3232805662 learned_addr learned my addr 172.21.15.101:0/3232805662 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:50.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 -- 172.21.15.101:0/3232805662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 msgr2=0x7f97841163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:50.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 0x7f97841163d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 -- 172.21.15.101:0/3232805662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9784103e50 msgr2=0x7f9784110460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:50.820 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9784103e50 0x7f9784110460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:50.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789057700 1 -- 172.21.15.101:0/3232805662 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9774009410 con 0x7f9784103540 2022-07-03T10:35:50.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.813+0000 7f9789858700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 0x7f97841163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:50.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f9789057700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f977400ea80 tx=0x7f9774005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:50.821 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774009650 con 0x7f9784103540 2022-07-03T10:35:50.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0097e0 con 0x7f9784103540 2022-07-03T10:35:50.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f977400ce80 con 0x7f9784103540 2022-07-03T10:35:50.822 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774018690 con 0x7f9784103540 2022-07-03T10:35:50.823 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.814+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978411d1e0 con 0x7f9784103540 2022-07-03T10:35:50.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.815+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978404ccb0 con 0x7f9784103540 2022-07-03T10:35:50.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.817+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f977400c040 con 0x7f9784103540 2022-07-03T10:35:50.824 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.819+0000 7f977a7fc700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 0x7f9770099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:50.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.819+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f9774014070 con 0x7f9784103540 2022-07-03T10:35:50.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.819+0000 7f9789858700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 0x7f9770099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:50.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.820+0000 7f9789858700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 0x7f9770099290 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9780011560 tx=0x7f9780005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:50.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:50.822+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f977408ce60 con 0x7f9784103540 2022-07-03T10:35:51.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.051+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f978411d790 con 0x7f9784103540 2022-07-03T10:35:51.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.052+0000 7f977a7fc700 1 -- 172.21.15.101:0/3232805662 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9774090480 con 0x7f9784103540 2022-07-03T10:35:51.054 INFO:teuthology.orchestra.run.smithi101.stdout:201863462951 2022-07-03T10:35:51.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 msgr2=0x7f9770099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:51.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 0x7f9770099290 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9780011560 tx=0x7f9780005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 msgr2=0x7f9784116910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:51.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f977400ea80 tx=0x7f9774005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 shutdown_connections 2022-07-03T10:35:51.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9770096de0 0x7f9770099290 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9784103e50 0x7f9784110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97840fdc90 0x7f97841163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 --2- 172.21.15.101:0/3232805662 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9784103540 0x7f9784116910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:51.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.055+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 >> 172.21.15.101:0/3232805662 conn(0x7f97840f9060 msgr2=0x7f9784070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:51.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.056+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 shutdown_connections 2022-07-03T10:35:51.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:51.056+0000 7f978a85a700 1 -- 172.21.15.101:0/3232805662 wait complete. 2022-07-03T10:35:51.295 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476798 got 68719476800 for osd.2 2022-07-03T10:35:51.295 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:51 smithi161 ceph-mon[91579]: pgmap v294: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2022-07-03T10:35:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:51 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3232805662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-07-03T10:35:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:51 smithi188 ceph-mon[92425]: pgmap v294: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2022-07-03T10:35:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:51 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3232805662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-07-03T10:35:51.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:51 smithi101 ceph-mon[85126]: pgmap v294: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2022-07-03T10:35:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:51 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3232805662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-07-03T10:35:52.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462948 got 201863462951 for osd.7 2022-07-03T10:35:52.282 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:53.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 -- 172.21.15.101:0/1311050723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a40fd9b0 msgr2=0x7fe4a4109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 --2- 172.21.15.101:0/1311050723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a4109c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe490009900 tx=0x7fe4900091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 -- 172.21.15.101:0/1311050723 shutdown_connections 2022-07-03T10:35:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 --2- 172.21.15.101:0/1311050723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a410a1a0 0x7fe4a410c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 --2- 172.21.15.101:0/1311050723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 --2- 172.21.15.101:0/1311050723 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a4109c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 -- 172.21.15.101:0/1311050723 >> 172.21.15.101:0/1311050723 conn(0x7fe4a4074450 msgr2=0x7fe4a4074850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:53.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 -- 172.21.15.101:0/1311050723 shutdown_connections 2022-07-03T10:35:53.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.094+0000 7fe4aada4700 1 -- 172.21.15.101:0/1311050723 wait complete. 2022-07-03T10:35:53.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.095+0000 7fe4aada4700 1 Processor -- start 2022-07-03T10:35:53.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.095+0000 7fe4aada4700 1 -- start start 2022-07-03T10:35:53.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.095+0000 7fe4aada4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.095+0000 7fe4aada4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a40fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.095+0000 7fe4aada4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4aada4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41210f0 con 0x7fe4a410a1a0 2022-07-03T10:35:53.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4aada4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4121230 con 0x7fe4a40fd0a0 2022-07-03T10:35:53.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4aada4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41213c0 con 0x7fe4a40fd9b0 2022-07-03T10:35:53.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a8b40700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52274/0 (socket says 172.21.15.101:52274) 2022-07-03T10:35:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 -- 172.21.15.101:0/881312347 learned_addr learned my addr 172.21.15.101:0/881312347 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a3fff700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a40fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 -- 172.21.15.101:0/881312347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd9b0 msgr2=0x7fe4a40fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a40fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 -- 172.21.15.101:0/881312347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 msgr2=0x7fe4a40fe8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fe8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a9341700 1 -- 172.21.15.101:0/881312347 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe490009440 con 0x7fe4a410a1a0 2022-07-03T10:35:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.096+0000 7fe4a8b40700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4a9341700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00eb50 tx=0x7fe49c009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c004a50 con 0x7fe4a410a1a0 2022-07-03T10:35:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a41216a0 con 0x7fe4a410a1a0 2022-07-03T10:35:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe49c00ce80 con 0x7fe4a410a1a0 2022-07-03T10:35:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c005730 con 0x7fe4a410a1a0 2022-07-03T10:35:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.097+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a4121bb0 con 0x7fe4a410a1a0 2022-07-03T10:35:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.099+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a404ccb0 con 0x7fe4a410a1a0 2022-07-03T10:35:53.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.102+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fe49c00c040 con 0x7fe4a410a1a0 2022-07-03T10:35:53.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.103+0000 7fe4a17fa700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 0x7fe494099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.103+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fe49c014070 con 0x7fe4a410a1a0 2022-07-03T10:35:53.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.103+0000 7fe4a8b40700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 0x7fe494099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.104+0000 7fe4a8b40700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 0x7fe494099210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe498011560 tx=0x7fe498005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:53.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.107+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe49c0c4060 con 0x7fe4a410a1a0 2022-07-03T10:35:53.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.307+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe4a4122250 con 0x7fe4a410a1a0 2022-07-03T10:35:53.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.308+0000 7fe4a17fa700 1 -- 172.21.15.101:0/881312347 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe49c08ca50 con 0x7fe4a410a1a0 2022-07-03T10:35:53.311 INFO:teuthology.orchestra.run.smithi101.stdout:128849018934 2022-07-03T10:35:53.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.310+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 msgr2=0x7fe494099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.310+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 0x7fe494099210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe498011560 tx=0x7fe498005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.310+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 msgr2=0x7fe4a40ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.310+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00eb50 tx=0x7fe49c009460 comp rx=0 tx=0).stop 2022-07-03T10:35:53.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 shutdown_connections 2022-07-03T10:35:53.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe494096d60 0x7fe494099210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe4a410a1a0 0x7fe4a40ff350 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4a40fd0a0 0x7fe4a40fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 --2- 172.21.15.101:0/881312347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4a40fd9b0 0x7fe4a40fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 >> 172.21.15.101:0/881312347 conn(0x7fe4a4074450 msgr2=0x7fe4a410b850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:53.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 shutdown_connections 2022-07-03T10:35:53.318 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.311+0000 7fe4aada4700 1 -- 172.21.15.101:0/881312347 wait complete. 2022-07-03T10:35:53.462 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:53.603 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:53 smithi101 ceph-mon[85126]: pgmap v295: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:53.603 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:53 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/881312347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-07-03T10:35:53.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.607+0000 7f776679a700 1 -- 172.21.15.101:0/2081112949 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 msgr2=0x7f7760103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.607+0000 7f776679a700 1 --2- 172.21.15.101:0/2081112949 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760103900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f775c009900 tx=0x7f775c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 -- 172.21.15.101:0/2081112949 shutdown_connections 2022-07-03T10:35:53.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 --2- 172.21.15.101:0/2081112949 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760103900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.611 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 --2- 172.21.15.101:0/2081112949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f77600fdc80 0x7f776010a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 --2- 172.21.15.101:0/2081112949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7760103e40 0x7f77600fd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 -- 172.21.15.101:0/2081112949 >> 172.21.15.101:0/2081112949 conn(0x7f77600f90b0 msgr2=0x7f77600fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:53.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 -- 172.21.15.101:0/2081112949 shutdown_connections 2022-07-03T10:35:53.612 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.608+0000 7f776679a700 1 -- 172.21.15.101:0/2081112949 wait complete. 2022-07-03T10:35:53.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.609+0000 7f776679a700 1 Processor -- start 2022-07-03T10:35:53.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.609+0000 7f776679a700 1 -- start start 2022-07-03T10:35:53.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.609+0000 7f776679a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f776679a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f776679a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760103e40 0x7f776010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f776679a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776010eab0 con 0x7f7760103530 2022-07-03T10:35:53.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f776679a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776010ec20 con 0x7f77600fdc80 2022-07-03T10:35:53.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f776679a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776010ede0 con 0x7f7760103e40 2022-07-03T10:35:53.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7765798700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.617 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7765f99700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760103e40 0x7f776010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52292/0 (socket says 172.21.15.101:52292) 2022-07-03T10:35:53.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 -- 172.21.15.101:0/2285273711 learned_addr learned my addr 172.21.15.101:0/2285273711 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:53.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7765798700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51134/0 (socket says 172.21.15.101:51134) 2022-07-03T10:35:53.618 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 -- 172.21.15.101:0/2285273711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760103e40 msgr2=0x7f776010e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760103e40 0x7f776010e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 -- 172.21.15.101:0/2285273711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 msgr2=0x7f7760114430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.610+0000 7f7764f97700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.619 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7764f97700 1 -- 172.21.15.101:0/2285273711 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7750009410 con 0x7f7760103530 2022-07-03T10:35:53.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7765798700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:53.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7764f97700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f775000ea80 tx=0x7f7750005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:53.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750009650 con 0x7f7760103530 2022-07-03T10:35:53.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f775000ee60 con 0x7f7760103530 2022-07-03T10:35:53.620 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c009440 con 0x7f7760103530 2022-07-03T10:35:53.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750003af0 con 0x7f7760103530 2022-07-03T10:35:53.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.611+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7760070750 con 0x7f7760103530 2022-07-03T10:35:53.621 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.612+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776004ccb0 con 0x7f7760103530 2022-07-03T10:35:53.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.614+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f775001e020 con 0x7f7760103530 2022-07-03T10:35:53.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.616+0000 7f7755ffb700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 0x7f774c099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:53.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.616+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7750014070 con 0x7f7760103530 2022-07-03T10:35:53.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.616+0000 7f7765798700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 0x7f774c099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:53.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.617+0000 7f7765798700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 0x7f774c099290 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f775c0056f0 tx=0x7f775c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:53.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.620+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f775008cd50 con 0x7f7760103530 2022-07-03T10:35:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:53 smithi161 ceph-mon[91579]: pgmap v295: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:53 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/881312347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-07-03T10:35:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:53 smithi188 ceph-mon[92425]: pgmap v295: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:53 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/881312347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-07-03T10:35:53.827 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.825+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f776010d2d0 con 0x7f7760103530 2022-07-03T10:35:53.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.826+0000 7f7755ffb700 1 -- 172.21.15.101:0/2285273711 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f7750090370 con 0x7f7760103530 2022-07-03T10:35:53.828 INFO:teuthology.orchestra.run.smithi101.stdout:322122547222 2022-07-03T10:35:53.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 msgr2=0x7f774c099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 0x7f774c099290 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f775c0056f0 tx=0x7f775c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 msgr2=0x7f7760114970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:53.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f775000ea80 tx=0x7f7750005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 shutdown_connections 2022-07-03T10:35:53.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f774c096de0 0x7f774c099290 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7760103e40 0x7f776010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77600fdc80 0x7f7760114430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 --2- 172.21.15.101:0/2285273711 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7760103530 0x7f7760114970 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:53.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 >> 172.21.15.101:0/2285273711 conn(0x7f77600f90b0 msgr2=0x7f77600fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:53.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.828+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 shutdown_connections 2022-07-03T10:35:53.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:53.829+0000 7f776679a700 1 -- 172.21.15.101:0/2285273711 wait complete. 2022-07-03T10:35:53.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018931 got 128849018934 for osd.4 2022-07-03T10:35:53.945 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:54.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:54 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2285273711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-07-03T10:35:54.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.523+0000 7f75b63b8700 1 -- 172.21.15.101:0/2070617908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 msgr2=0x7f75b00fd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.523+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2070617908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b00fd560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009900 tx=0x7f75a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 -- 172.21.15.101:0/2070617908 shutdown_connections 2022-07-03T10:35:54.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2070617908 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b00fdc90 0x7f75b010a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2070617908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b0103540 0x7f75b0103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2070617908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b00fd560 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 -- 172.21.15.101:0/2070617908 >> 172.21.15.101:0/2070617908 conn(0x7f75b00f9060 msgr2=0x7f75b00fb470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:54.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 -- 172.21.15.101:0/2070617908 shutdown_connections 2022-07-03T10:35:54.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.524+0000 7f75b63b8700 1 -- 172.21.15.101:0/2070617908 wait complete. 2022-07-03T10:35:54.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 Processor -- start 2022-07-03T10:35:54.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 -- start start 2022-07-03T10:35:54.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 0x7f75b01166b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b0110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b0110dd0 con 0x7f75b0103540 2022-07-03T10:35:54.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b0110f40 con 0x7f75b0103e50 2022-07-03T10:35:54.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.525+0000 7f75b63b8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b0111100 con 0x7f75b00fdc90 2022-07-03T10:35:54.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b4bb5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b4bb5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52308/0 (socket says 172.21.15.101:52308) 2022-07-03T10:35:54.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b4bb5700 1 -- 172.21.15.101:0/2063930328 learned_addr learned my addr 172.21.15.101:0/2063930328 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:54.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 0x7f75b01166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b5bb7700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b0110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 -- 172.21.15.101:0/2063930328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 msgr2=0x7f75b0110760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b0110760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 -- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 msgr2=0x7f75b0116c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b53b6700 1 -- 172.21.15.101:0/2063930328 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac009410 con 0x7f75b00fdc90 2022-07-03T10:35:54.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.526+0000 7f75b4bb5700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:35:54.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75b53b6700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 0x7f75b01166b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00ea80 tx=0x7f75ac005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:54.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac009650 con 0x7f75b00fdc90 2022-07-03T10:35:54.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a0009440 con 0x7f75b00fdc90 2022-07-03T10:35:54.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f75ac00ee60 con 0x7f75b00fdc90 2022-07-03T10:35:54.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac003ab0 con 0x7f75b00fdc90 2022-07-03T10:35:54.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.527+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b01115d0 con 0x7f75b00fdc90 2022-07-03T10:35:54.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.529+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b004ccb0 con 0x7f75b00fdc90 2022-07-03T10:35:54.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.530+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f75ac00c040 con 0x7f75b00fdc90 2022-07-03T10:35:54.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.531+0000 7f75a5ffb700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 0x7f759c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.531+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f75ac014070 con 0x7f75b00fdc90 2022-07-03T10:35:54.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.532+0000 7f75b4bb5700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 0x7f759c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.532+0000 7f75b4bb5700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 0x7f759c099210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f75a00056f0 tx=0x7f75a00091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:54.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.535+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f75ac08ce10 con 0x7f75b00fdc90 2022-07-03T10:35:54.583 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2393096025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d80fef90 msgr2=0x7fc4d8107510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.583 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2393096025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d80fef90 0x7fc4d8107510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009900 tx=0x7fc4cc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2393096025 shutdown_connections 2022-07-03T10:35:54.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2393096025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d8107a50 0x7fc4d8109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2393096025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d80fea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2393096025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d80fef90 0x7fc4d8107510 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2393096025 >> 172.21.15.101:0/2393096025 conn(0x7fc4d80f7f20 msgr2=0x7fc4d80fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:54.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2393096025 shutdown_connections 2022-07-03T10:35:54.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.581+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2393096025 wait complete. 2022-07-03T10:35:54.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.582+0000 7fc4d7fff700 1 Processor -- start 2022-07-03T10:35:54.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.582+0000 7fc4d7fff700 1 -- start start 2022-07-03T10:35:54.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d81166c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 0x7fc4d8116c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d8107a50 0x7fc4d8110770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8110de0 con 0x7fc4d80fef90 2022-07-03T10:35:54.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8110f50 con 0x7fc4d80fe680 2022-07-03T10:35:54.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d7fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8111110 con 0x7fc4d8107a50 2022-07-03T10:35:54.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d81166c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d6ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d81166c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51170/0 (socket says 172.21.15.101:51170) 2022-07-03T10:35:54.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d6ffd700 1 -- 172.21.15.101:0/2032624258 learned_addr learned my addr 172.21.15.101:0/2032624258 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:54.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.583+0000 7fc4d67fc700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 0x7fc4d8116c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d77fe700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d8107a50 0x7fc4d8110770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d67fc700 1 -- 172.21.15.101:0/2032624258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d8107a50 msgr2=0x7fc4d8110770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d67fc700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d8107a50 0x7fc4d8110770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d67fc700 1 -- 172.21.15.101:0/2032624258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 msgr2=0x7fc4d81166c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d67fc700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d81166c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.584+0000 7fc4d67fc700 1 -- 172.21.15.101:0/2032624258 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8009410 con 0x7fc4d80fef90 2022-07-03T10:35:54.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.585+0000 7fc4d67fc700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 0x7fc4d8116c40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0092d0 tx=0x7fc4cc0091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:54.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.585+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc01d070 con 0x7fc4d80fef90 2022-07-03T10:35:54.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.585+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc009440 con 0x7fc4d80fef90 2022-07-03T10:35:54.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.585+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc4cc003f50 con 0x7fc4d80fef90 2022-07-03T10:35:54.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.585+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc022a40 con 0x7fc4d80fef90 2022-07-03T10:35:54.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.586+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d811d4a0 con 0x7fc4d80fef90 2022-07-03T10:35:54.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.587+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d804ccb0 con 0x7fc4d80fef90 2022-07-03T10:35:54.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.590+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fc4cc029030 con 0x7fc4d80fef90 2022-07-03T10:35:54.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.592+0000 7fc4bf7fe700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 0x7fc4c0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:54.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.592+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fc4cc0c0820 con 0x7fc4d80fef90 2022-07-03T10:35:54.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.593+0000 7fc4d6ffd700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 0x7fc4c0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:54.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.593+0000 7fc4d6ffd700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 0x7fc4c0099210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4d81091f0 tx=0x7fc4c8005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:54.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.597+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fc4cc05a100 con 0x7fc4d80fef90 2022-07-03T10:35:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:54 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2285273711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-07-03T10:35:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:54 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2285273711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-07-03T10:35:54.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.782+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f75b011db50 con 0x7f75b00fdc90 2022-07-03T10:35:54.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.783+0000 7f75a5ffb700 1 -- 172.21.15.101:0/2063930328 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f75ac090430 con 0x7f75b00fdc90 2022-07-03T10:35:54.785 INFO:teuthology.orchestra.run.smithi101.stdout:231928234021 2022-07-03T10:35:54.787 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 msgr2=0x7f759c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 0x7f759c099210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f75a00056f0 tx=0x7f75a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 msgr2=0x7f75b01166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 0x7f75b01166b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00ea80 tx=0x7f75ac005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 shutdown_connections 2022-07-03T10:35:54.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f759c096d60 0x7f759c099210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75b0103e50 0x7f75b0110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75b00fdc90 0x7f75b01166b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 --2- 172.21.15.101:0/2063930328 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f75b0103540 0x7f75b0116c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 >> 172.21.15.101:0/2063930328 conn(0x7f75b00f9060 msgr2=0x7f75b00fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:54.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.785+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 shutdown_connections 2022-07-03T10:35:54.790 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.786+0000 7f75b63b8700 1 -- 172.21.15.101:0/2063930328 wait complete. 2022-07-03T10:35:54.825 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.823+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc4d8111cb0 con 0x7fc4d80fef90 2022-07-03T10:35:54.826 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.824+0000 7fc4bf7fe700 1 -- 172.21.15.101:0/2032624258 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc4cc08da80 con 0x7fc4d80fef90 2022-07-03T10:35:54.826 INFO:teuthology.orchestra.run.smithi101.stdout:184683593771 2022-07-03T10:35:54.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 msgr2=0x7fc4c0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.828 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 0x7fc4c0099210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4d81091f0 tx=0x7fc4c8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.829 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 msgr2=0x7fc4d8116c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:54.829 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 0x7fc4d8116c40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0092d0 tx=0x7fc4cc0091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.829 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 shutdown_connections 2022-07-03T10:35:54.829 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4c0096d60 0x7fc4c0099210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4d8107a50 0x7fc4d8110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4d80fe680 0x7fc4d81166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 --2- 172.21.15.101:0/2032624258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4d80fef90 0x7fc4d8116c40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:54.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 >> 172.21.15.101:0/2032624258 conn(0x7fc4d80f7f20 msgr2=0x7fc4d80fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:54.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 shutdown_connections 2022-07-03T10:35:54.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:54.826+0000 7fc4d7fff700 1 -- 172.21.15.101:0/2032624258 wait complete. 2022-07-03T10:35:55.577 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234018 got 231928234021 for osd.8 2022-07-03T10:35:55.578 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:55 smithi188 ceph-mon[92425]: pgmap v296: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:55 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2063930328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-07-03T10:35:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:55 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2032624258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-07-03T10:35:55.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:55 smithi101 ceph-mon[85126]: pgmap v296: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:55 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2063930328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-07-03T10:35:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:55 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2032624258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-07-03T10:35:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:55 smithi161 ceph-mon[91579]: pgmap v296: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:55 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2063930328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-07-03T10:35:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:55 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2032624258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-07-03T10:35:55.882 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547219 got 322122547222 for osd.11 2022-07-03T10:35:55.882 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:56.297 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 -- 172.21.15.101:0/2819638988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550109fa0 msgr2=0x7fd55010c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:56.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 --2- 172.21.15.101:0/2819638988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550109fa0 0x7fd55010c380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd548009ca0 tx=0x7fd548009520 comp rx=0 tx=0).stop 2022-07-03T10:35:56.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 -- 172.21.15.101:0/2819638988 shutdown_connections 2022-07-03T10:35:56.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 --2- 172.21.15.101:0/2819638988 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550070840 0x7fd550070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 --2- 172.21.15.101:0/2819638988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550109fa0 0x7fd55010c380 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 --2- 172.21.15.101:0/2819638988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550071150 0x7fd550109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 -- 172.21.15.101:0/2819638988 >> 172.21.15.101:0/2819638988 conn(0x7fd5500f9220 msgr2=0x7fd5500fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:56.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 -- 172.21.15.101:0/2819638988 shutdown_connections 2022-07-03T10:35:56.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.295+0000 7fd557cdb700 1 -- 172.21.15.101:0/2819638988 wait complete. 2022-07-03T10:35:56.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.296+0000 7fd557cdb700 1 Processor -- start 2022-07-03T10:35:56.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.296+0000 7fd557cdb700 1 -- start start 2022-07-03T10:35:56.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 0x7fd550116550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:56.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550071150 0x7fd550116a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:56.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 0x7fd550110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:56.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550110bd0 con 0x7fd550071150 2022-07-03T10:35:56.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550110d40 con 0x7fd550070840 2022-07-03T10:35:56.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd557cdb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550110f00 con 0x7fd550109fa0 2022-07-03T10:35:56.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd556278700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 0x7fd550110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:56.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd555a77700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 0x7fd550116550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:56.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd556278700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 0x7fd550110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:57514/0 (socket says 172.21.15.101:57514) 2022-07-03T10:35:56.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd556278700 1 -- 172.21.15.101:0/1433096572 learned_addr learned my addr 172.21.15.101:0/1433096572 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:56.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.297+0000 7fd555276700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550071150 0x7fd550116a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:56.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 -- 172.21.15.101:0/1433096572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 msgr2=0x7fd550110600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:56.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 0x7fd550110600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 -- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550071150 msgr2=0x7fd550116a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:56.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550071150 0x7fd550116a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 -- 172.21.15.101:0/1433096572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480097e0 con 0x7fd550070840 2022-07-03T10:35:56.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd555a77700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 0x7fd550116550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00e7d0 tx=0x7fd54c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:56.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c010710 con 0x7fd550070840 2022-07-03T10:35:56.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd54c010d20 con 0x7fd550070840 2022-07-03T10:35:56.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5501111e0 con 0x7fd550070840 2022-07-03T10:35:56.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c018dd0 con 0x7fd550070840 2022-07-03T10:35:56.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.298+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55011d510 con 0x7fd550070840 2022-07-03T10:35:56.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.300+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55004ccb0 con 0x7fd550070840 2022-07-03T10:35:56.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.303+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd54c00c040 con 0x7fd550070840 2022-07-03T10:35:56.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.304+0000 7fd5467fc700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 0x7fd53c099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:56.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.304+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd54c014070 con 0x7fd550070840 2022-07-03T10:35:56.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.304+0000 7fd555276700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 0x7fd53c099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:56.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.305+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd54c08d2d0 con 0x7fd550070840 2022-07-03T10:35:56.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.305+0000 7fd555276700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 0x7fd53c099290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd550111dd0 tx=0x7fd540005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:56.317 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593769 got 184683593771 for osd.6 2022-07-03T10:35:56.317 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:56.495 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.493+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd55010f6c0 con 0x7fd550070840 2022-07-03T10:35:56.495 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.493+0000 7fd5467fc700 1 -- 172.21.15.101:0/1433096572 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd54c026090 con 0x7fd550070840 2022-07-03T10:35:56.496 INFO:teuthology.orchestra.run.smithi101.stdout:34359738440 2022-07-03T10:35:56.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 msgr2=0x7fd53c099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:56.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 0x7fd53c099290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd550111dd0 tx=0x7fd540005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 msgr2=0x7fd550116550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:56.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 0x7fd550116550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00e7d0 tx=0x7fd54c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 shutdown_connections 2022-07-03T10:35:56.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd53c096de0 0x7fd53c099290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd550109fa0 0x7fd550110600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd550070840 0x7fd550116550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 --2- 172.21.15.101:0/1433096572 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd550071150 0x7fd550116a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:56.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 >> 172.21.15.101:0/1433096572 conn(0x7fd5500f9220 msgr2=0x7fd550101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:56.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 shutdown_connections 2022-07-03T10:35:56.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:56.496+0000 7fd557cdb700 1 -- 172.21.15.101:0/1433096572 wait complete. 2022-07-03T10:35:56.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738437 got 34359738440 for osd.0 2022-07-03T10:35:56.999 DEBUG:teuthology.parallel:result is None 2022-07-03T10:35:57.000 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-07-03T10:35:57.000 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph pg dump --format=json 2022-07-03T10:35:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:57 smithi188 ceph-mon[92425]: pgmap v297: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:57 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1433096572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-07-03T10:35:57.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:57 smithi101 ceph-mon[85126]: pgmap v297: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:57.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:57 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1433096572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-07-03T10:35:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:57 smithi161 ceph-mon[91579]: pgmap v297: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:35:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:57 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1433096572' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-07-03T10:35:57.885 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:35:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:35:59 smithi101 ceph-mon[85126]: pgmap v298: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:59.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 -- 172.21.15.101:0/226170992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b4103e40 msgr2=0x7f36b40fd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:59.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 --2- 172.21.15.101:0/226170992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b4103e40 0x7f36b40fd550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009900 tx=0x7f36a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 -- 172.21.15.101:0/226170992 shutdown_connections 2022-07-03T10:35:59.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 --2- 172.21.15.101:0/226170992 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103530 0x7f36b4103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 --2- 172.21.15.101:0/226170992 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b40fdc80 0x7f36b410a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 --2- 172.21.15.101:0/226170992 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b4103e40 0x7f36b40fd550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 -- 172.21.15.101:0/226170992 >> 172.21.15.101:0/226170992 conn(0x7f36b40f90b0 msgr2=0x7f36b40fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:59.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 -- 172.21.15.101:0/226170992 shutdown_connections 2022-07-03T10:35:59.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.523+0000 7f36b9f89700 1 -- 172.21.15.101:0/226170992 wait complete. 2022-07-03T10:35:59.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.524+0000 7f36b9f89700 1 Processor -- start 2022-07-03T10:35:59.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.524+0000 7f36b9f89700 1 -- start start 2022-07-03T10:35:59.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:59.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b4103530 0x7f36b4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:59.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103e40 0x7f36b40712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:59.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b411cb10 con 0x7f36b4103e40 2022-07-03T10:35:59.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b411cc80 con 0x7f36b4103530 2022-07-03T10:35:59.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.525+0000 7f36b9f89700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b411ce40 con 0x7f36b40fdc80 2022-07-03T10:35:59.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:59.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53356/0 (socket says 172.21.15.101:53356) 2022-07-03T10:35:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 -- 172.21.15.101:0/1080836413 learned_addr learned my addr 172.21.15.101:0/1080836413 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:35:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b9788700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103e40 0x7f36b40712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b3fff700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b4103530 0x7f36b4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 -- 172.21.15.101:0/1080836413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b4103530 msgr2=0x7f36b4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:59.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b4103530 0x7f36b4070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 -- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103e40 msgr2=0x7f36b40712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:59.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103e40 0x7f36b40712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 -- 172.21.15.101:0/1080836413 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009440 con 0x7f36b40fdc80 2022-07-03T10:35:59.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.526+0000 7f36b8f87700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f36a800ea80 tx=0x7f36a8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:59.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.527+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a800c7c0 con 0x7f36b40fdc80 2022-07-03T10:35:59.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.527+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f36a800cdd0 con 0x7f36b40fdc80 2022-07-03T10:35:59.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.527+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b411d120 con 0x7f36b40fdc80 2022-07-03T10:35:59.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.527+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8010df0 con 0x7f36b40fdc80 2022-07-03T10:35:59.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.527+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b411d600 con 0x7f36b40fdc80 2022-07-03T10:35:59.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.528+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b404ccb0 con 0x7f36b40fdc80 2022-07-03T10:35:59.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.530+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f36a800c040 con 0x7f36b40fdc80 2022-07-03T10:35:59.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.531+0000 7f36b17fa700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 0x7f36a4099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:35:59.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.531+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f36a8014070 con 0x7f36b40fdc80 2022-07-03T10:35:59.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.531+0000 7f36b3fff700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 0x7f36a4099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:35:59.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.532+0000 7f36b3fff700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 0x7f36a4099290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f36a00092d0 tx=0x7f36a00091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:35:59.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.533+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f36a808cbb0 con 0x7f36b40fdc80 2022-07-03T10:35:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:35:59 smithi188 ceph-mon[92425]: pgmap v298: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:35:59.718 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.716+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36b40faed0 con 0x7f36a4096de0 2022-07-03T10:35:59.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.727+0000 7f36b17fa700 1 -- 172.21.15.101:0/1080836413 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+338952 (secure 0 0 0) 0x7f36b40faed0 con 0x7f36a4096de0 2022-07-03T10:35:59.736 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:35:59.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 msgr2=0x7f36a4099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:59.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 0x7f36a4099290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f36a00092d0 tx=0x7f36a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 msgr2=0x7f36b4070820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:35:59.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f36a800ea80 tx=0x7f36a8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 shutdown_connections 2022-07-03T10:35:59.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f36a4096de0 0x7f36a4099290 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b4103e40 0x7f36b40712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36b40fdc80 0x7f36b4070820 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 --2- 172.21.15.101:0/1080836413 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36b4103530 0x7f36b4070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:35:59.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 >> 172.21.15.101:0/1080836413 conn(0x7f36b40f90b0 msgr2=0x7f36b40fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:35:59.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 shutdown_connections 2022-07-03T10:35:59.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:35:59.733+0000 7f36b9f89700 1 -- 172.21.15.101:0/1080836413 wait complete. 2022-07-03T10:35:59.742 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:35:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:35:59 smithi161 ceph-mon[91579]: pgmap v298: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:00.160 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2022-07-03T10:35:57.951386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644869,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":234,"num_read_kb":217,"num_write":251,"num_write_kb":3013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77444,"kb_used_data":10724,"kb_used_omap":0,"kb_used_meta":66624,"kb_avail":1124864380,"statfs":{"total":1151940427776,"available":1151861125120,"internally_reserved":0,"allocated":10981376,"data_stored":7172872,"data_compressed":176772,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":68222976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003224"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307572+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:35:26.307572+0000","last_peered":"2022-07-03T10:35:26.307572+0000","last_clean":"2022-07-03T10:35:26.307572+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:35:26.307572+0000","last_undegraded":"2022-07-03T10:35:26.307572+0000","last_fullsized":"2022-07-03T10:35:26.307572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.345037+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:35:26.345037+0000","last_peered":"2022-07-03T10:35:26.345037+0000","last_clean":"2022-07-03T10:35:26.345037+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:35:26.345037+0000","last_undegraded":"2022-07-03T10:35:26.345037+0000","last_fullsized":"2022-07-03T10:35:26.345037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327600+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:35:26.327600+0000","last_peered":"2022-07-03T10:35:26.327600+0000","last_clean":"2022-07-03T10:35:26.327600+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:35:26.327600+0000","last_undegraded":"2022-07-03T10:35:26.327600+0000","last_fullsized":"2022-07-03T10:35:26.327600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.315233+0000","last_change":"2022-07-03T10:35:27.315233+0000","last_active":"2022-07-03T10:35:27.315233+0000","last_peered":"2022-07-03T10:35:27.315233+0000","last_clean":"2022-07-03T10:35:27.315233+0000","last_became_active":"2022-07-03T10:35:27.314807+0000","last_became_peered":"2022-07-03T10:35:27.314807+0000","last_unstale":"2022-07-03T10:35:27.315233+0000","last_undegraded":"2022-07-03T10:35:27.315233+0000","last_fullsized":"2022-07-03T10:35:27.315233+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:37:28.362457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309008+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:35:26.309008+0000","last_peered":"2022-07-03T10:35:26.309008+0000","last_clean":"2022-07-03T10:35:26.309008+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:35:26.309008+0000","last_undegraded":"2022-07-03T10:35:26.309008+0000","last_fullsized":"2022-07-03T10:35:26.309008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309163+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:35:26.309163+0000","last_peered":"2022-07-03T10:35:26.309163+0000","last_clean":"2022-07-03T10:35:26.309163+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:35:26.309163+0000","last_undegraded":"2022-07-03T10:35:26.309163+0000","last_fullsized":"2022-07-03T10:35:26.309163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347625+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:35:26.347625+0000","last_peered":"2022-07-03T10:35:26.347625+0000","last_clean":"2022-07-03T10:35:26.347625+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:35:26.347625+0000","last_undegraded":"2022-07-03T10:35:26.347625+0000","last_fullsized":"2022-07-03T10:35:26.347625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341101+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:35:26.341101+0000","last_peered":"2022-07-03T10:35:26.341101+0000","last_clean":"2022-07-03T10:35:26.341101+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:35:26.341101+0000","last_undegraded":"2022-07-03T10:35:26.341101+0000","last_fullsized":"2022-07-03T10:35:26.341101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310755+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:35:26.310755+0000","last_peered":"2022-07-03T10:35:26.310755+0000","last_clean":"2022-07-03T10:35:26.310755+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:35:26.310755+0000","last_undegraded":"2022-07-03T10:35:26.310755+0000","last_fullsized":"2022-07-03T10:35:26.310755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309517+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:35:26.309517+0000","last_peered":"2022-07-03T10:35:26.309517+0000","last_clean":"2022-07-03T10:35:26.309517+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:35:26.309517+0000","last_undegraded":"2022-07-03T10:35:26.309517+0000","last_fullsized":"2022-07-03T10:35:26.309517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347953+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:35:26.347953+0000","last_peered":"2022-07-03T10:35:26.347953+0000","last_clean":"2022-07-03T10:35:26.347953+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:35:26.347953+0000","last_undegraded":"2022-07-03T10:35:26.347953+0000","last_fullsized":"2022-07-03T10:35:26.347953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302340+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:35:26.302340+0000","last_peered":"2022-07-03T10:35:26.302340+0000","last_clean":"2022-07-03T10:35:26.302340+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:26.302340+0000","last_undegraded":"2022-07-03T10:35:26.302340+0000","last_fullsized":"2022-07-03T10:35:26.302340+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332994+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:35:26.332994+0000","last_peered":"2022-07-03T10:35:26.332994+0000","last_clean":"2022-07-03T10:35:26.332994+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:35:26.332994+0000","last_undegraded":"2022-07-03T10:35:26.332994+0000","last_fullsized":"2022-07-03T10:35:26.332994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332723+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:35:26.332723+0000","last_peered":"2022-07-03T10:35:26.332723+0000","last_clean":"2022-07-03T10:35:26.332723+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:35:26.332723+0000","last_undegraded":"2022-07-03T10:35:26.332723+0000","last_fullsized":"2022-07-03T10:35:26.332723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327163+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:35:26.327163+0000","last_peered":"2022-07-03T10:35:26.327163+0000","last_clean":"2022-07-03T10:35:26.327163+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:35:26.327163+0000","last_undegraded":"2022-07-03T10:35:26.327163+0000","last_fullsized":"2022-07-03T10:35:26.327163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309877+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:35:26.309877+0000","last_peered":"2022-07-03T10:35:26.309877+0000","last_clean":"2022-07-03T10:35:26.309877+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:35:26.309877+0000","last_undegraded":"2022-07-03T10:35:26.309877+0000","last_fullsized":"2022-07-03T10:35:26.309877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307666+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:35:26.307666+0000","last_peered":"2022-07-03T10:35:26.307666+0000","last_clean":"2022-07-03T10:35:26.307666+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:35:26.307666+0000","last_undegraded":"2022-07-03T10:35:26.307666+0000","last_fullsized":"2022-07-03T10:35:26.307666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325729+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:35:26.325729+0000","last_peered":"2022-07-03T10:35:26.325729+0000","last_clean":"2022-07-03T10:35:26.325729+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:35:26.325729+0000","last_undegraded":"2022-07-03T10:35:26.325729+0000","last_fullsized":"2022-07-03T10:35:26.325729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303772+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:35:26.303772+0000","last_peered":"2022-07-03T10:35:26.303772+0000","last_clean":"2022-07-03T10:35:26.303772+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:35:26.303772+0000","last_undegraded":"2022-07-03T10:35:26.303772+0000","last_fullsized":"2022-07-03T10:35:26.303772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325415+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:26.325415+0000","last_peered":"2022-07-03T10:35:26.325415+0000","last_clean":"2022-07-03T10:35:26.325415+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:26.325415+0000","last_undegraded":"2022-07-03T10:35:26.325415+0000","last_fullsized":"2022-07-03T10:35:26.325415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343425+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:35:26.343425+0000","last_peered":"2022-07-03T10:35:26.343425+0000","last_clean":"2022-07-03T10:35:26.343425+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:35:26.343425+0000","last_undegraded":"2022-07-03T10:35:26.343425+0000","last_fullsized":"2022-07-03T10:35:26.343425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343531+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:26.343531+0000","last_peered":"2022-07-03T10:35:26.343531+0000","last_clean":"2022-07-03T10:35:26.343531+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:26.343531+0000","last_undegraded":"2022-07-03T10:35:26.343531+0000","last_fullsized":"2022-07-03T10:35:26.343531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.328016+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:35:26.328016+0000","last_peered":"2022-07-03T10:35:26.328016+0000","last_clean":"2022-07-03T10:35:26.328016+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:35:26.328016+0000","last_undegraded":"2022-07-03T10:35:26.328016+0000","last_fullsized":"2022-07-03T10:35:26.328016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309637+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:26.309637+0000","last_peered":"2022-07-03T10:35:26.309637+0000","last_clean":"2022-07-03T10:35:26.309637+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:26.309637+0000","last_undegraded":"2022-07-03T10:35:26.309637+0000","last_fullsized":"2022-07-03T10:35:26.309637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310404+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:35:26.310404+0000","last_peered":"2022-07-03T10:35:26.310404+0000","last_clean":"2022-07-03T10:35:26.310404+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:35:26.310404+0000","last_undegraded":"2022-07-03T10:35:26.310404+0000","last_fullsized":"2022-07-03T10:35:26.310404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309394+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:26.309394+0000","last_peered":"2022-07-03T10:35:26.309394+0000","last_clean":"2022-07-03T10:35:26.309394+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:26.309394+0000","last_undegraded":"2022-07-03T10:35:26.309394+0000","last_fullsized":"2022-07-03T10:35:26.309394+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306137+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:35:26.306137+0000","last_peered":"2022-07-03T10:35:26.306137+0000","last_clean":"2022-07-03T10:35:26.306137+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:35:26.306137+0000","last_undegraded":"2022-07-03T10:35:26.306137+0000","last_fullsized":"2022-07-03T10:35:26.306137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305894+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:26.305894+0000","last_peered":"2022-07-03T10:35:26.305894+0000","last_clean":"2022-07-03T10:35:26.305894+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:26.305894+0000","last_undegraded":"2022-07-03T10:35:26.305894+0000","last_fullsized":"2022-07-03T10:35:26.305894+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308941+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:35:26.308941+0000","last_peered":"2022-07-03T10:35:26.308941+0000","last_clean":"2022-07-03T10:35:26.308941+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:35:26.308941+0000","last_undegraded":"2022-07-03T10:35:26.308941+0000","last_fullsized":"2022-07-03T10:35:26.308941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309034+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:26.309034+0000","last_peered":"2022-07-03T10:35:26.309034+0000","last_clean":"2022-07-03T10:35:26.309034+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:26.309034+0000","last_undegraded":"2022-07-03T10:35:26.309034+0000","last_fullsized":"2022-07-03T10:35:26.309034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.324978+0000","last_change":"2022-07-03T10:35:27.324978+0000","last_active":"2022-07-03T10:35:27.324978+0000","last_peered":"2022-07-03T10:35:27.324978+0000","last_clean":"2022-07-03T10:35:27.324978+0000","last_became_active":"2022-07-03T10:35:27.324570+0000","last_became_peered":"2022-07-03T10:35:27.324570+0000","last_unstale":"2022-07-03T10:35:27.324978+0000","last_undegraded":"2022-07-03T10:35:27.324978+0000","last_fullsized":"2022-07-03T10:35:27.324978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:18:06.187457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332535+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:26.332535+0000","last_peered":"2022-07-03T10:35:26.332535+0000","last_clean":"2022-07-03T10:35:26.332535+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:26.332535+0000","last_undegraded":"2022-07-03T10:35:26.332535+0000","last_fullsized":"2022-07-03T10:35:26.332535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341210+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:35:26.341210+0000","last_peered":"2022-07-03T10:35:26.341210+0000","last_clean":"2022-07-03T10:35:26.341210+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:35:26.341210+0000","last_undegraded":"2022-07-03T10:35:26.341210+0000","last_fullsized":"2022-07-03T10:35:26.341210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305427+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:26.305427+0000","last_peered":"2022-07-03T10:35:26.305427+0000","last_clean":"2022-07-03T10:35:26.305427+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:26.305427+0000","last_undegraded":"2022-07-03T10:35:26.305427+0000","last_fullsized":"2022-07-03T10:35:26.305427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.314929+0000","last_change":"2022-07-03T10:35:27.314929+0000","last_active":"2022-07-03T10:35:27.314929+0000","last_peered":"2022-07-03T10:35:27.314929+0000","last_clean":"2022-07-03T10:35:27.314929+0000","last_became_active":"2022-07-03T10:35:27.314542+0000","last_became_peered":"2022-07-03T10:35:27.314542+0000","last_unstale":"2022-07-03T10:35:27.314929+0000","last_undegraded":"2022-07-03T10:35:27.314929+0000","last_fullsized":"2022-07-03T10:35:27.314929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:50:25.278127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309250+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:26.309250+0000","last_peered":"2022-07-03T10:35:26.309250+0000","last_clean":"2022-07-03T10:35:26.309250+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:26.309250+0000","last_undegraded":"2022-07-03T10:35:26.309250+0000","last_fullsized":"2022-07-03T10:35:26.309250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310646+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:35:26.310646+0000","last_peered":"2022-07-03T10:35:26.310646+0000","last_clean":"2022-07-03T10:35:26.310646+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:35:26.310646+0000","last_undegraded":"2022-07-03T10:35:26.310646+0000","last_fullsized":"2022-07-03T10:35:26.310646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347933+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:26.347933+0000","last_peered":"2022-07-03T10:35:26.347933+0000","last_clean":"2022-07-03T10:35:26.347933+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:26.347933+0000","last_undegraded":"2022-07-03T10:35:26.347933+0000","last_fullsized":"2022-07-03T10:35:26.347933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327938+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:35:26.327938+0000","last_peered":"2022-07-03T10:35:26.327938+0000","last_clean":"2022-07-03T10:35:26.327938+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:35:26.327938+0000","last_undegraded":"2022-07-03T10:35:26.327938+0000","last_fullsized":"2022-07-03T10:35:26.327938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306107+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:26.306107+0000","last_peered":"2022-07-03T10:35:26.306107+0000","last_clean":"2022-07-03T10:35:26.306107+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:26.306107+0000","last_undegraded":"2022-07-03T10:35:26.306107+0000","last_fullsized":"2022-07-03T10:35:26.306107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310386+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:35:26.310386+0000","last_peered":"2022-07-03T10:35:26.310386+0000","last_clean":"2022-07-03T10:35:26.310386+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:26.310386+0000","last_undegraded":"2022-07-03T10:35:26.310386+0000","last_fullsized":"2022-07-03T10:35:26.310386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'25","reported_seq":141,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:53.972990+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:35:53.972990+0000","last_peered":"2022-07-03T10:35:53.972990+0000","last_clean":"2022-07-03T10:35:53.972990+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:35:53.972990+0000","last_undegraded":"2022-07-03T10:35:53.972990+0000","last_fullsized":"2022-07-03T10:35:53.972990+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53473,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":25,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309385+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:35:26.309385+0000","last_peered":"2022-07-03T10:35:26.309385+0000","last_clean":"2022-07-03T10:35:26.309385+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:35:26.309385+0000","last_undegraded":"2022-07-03T10:35:26.309385+0000","last_fullsized":"2022-07-03T10:35:26.309385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.326689+0000","last_change":"2022-07-03T10:35:27.326689+0000","last_active":"2022-07-03T10:35:27.326689+0000","last_peered":"2022-07-03T10:35:27.326689+0000","last_clean":"2022-07-03T10:35:27.326689+0000","last_became_active":"2022-07-03T10:35:27.320346+0000","last_became_peered":"2022-07-03T10:35:27.320346+0000","last_unstale":"2022-07-03T10:35:27.326689+0000","last_undegraded":"2022-07-03T10:35:27.326689+0000","last_fullsized":"2022-07-03T10:35:27.326689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:57:55.061972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330365+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:35:26.330365+0000","last_peered":"2022-07-03T10:35:26.330365+0000","last_clean":"2022-07-03T10:35:26.330365+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:35:26.330365+0000","last_undegraded":"2022-07-03T10:35:26.330365+0000","last_fullsized":"2022-07-03T10:35:26.330365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305459+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:26.305459+0000","last_peered":"2022-07-03T10:35:26.305459+0000","last_clean":"2022-07-03T10:35:26.305459+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:26.305459+0000","last_undegraded":"2022-07-03T10:35:26.305459+0000","last_fullsized":"2022-07-03T10:35:26.305459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347839+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:35:26.347839+0000","last_peered":"2022-07-03T10:35:26.347839+0000","last_clean":"2022-07-03T10:35:26.347839+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:35:26.347839+0000","last_undegraded":"2022-07-03T10:35:26.347839+0000","last_fullsized":"2022-07-03T10:35:26.347839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302039+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:26.302039+0000","last_peered":"2022-07-03T10:35:26.302039+0000","last_clean":"2022-07-03T10:35:26.302039+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:26.302039+0000","last_undegraded":"2022-07-03T10:35:26.302039+0000","last_fullsized":"2022-07-03T10:35:26.302039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347546+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:35:26.347546+0000","last_peered":"2022-07-03T10:35:26.347546+0000","last_clean":"2022-07-03T10:35:26.347546+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:35:26.347546+0000","last_undegraded":"2022-07-03T10:35:26.347546+0000","last_fullsized":"2022-07-03T10:35:26.347546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308728+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:26.308728+0000","last_peered":"2022-07-03T10:35:26.308728+0000","last_clean":"2022-07-03T10:35:26.308728+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:26.308728+0000","last_undegraded":"2022-07-03T10:35:26.308728+0000","last_fullsized":"2022-07-03T10:35:26.308728+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.623637+0000","last_change":"2022-07-03T10:35:27.623637+0000","last_active":"2022-07-03T10:35:27.623637+0000","last_peered":"2022-07-03T10:35:27.623637+0000","last_clean":"2022-07-03T10:35:27.623637+0000","last_became_active":"2022-07-03T10:35:27.623202+0000","last_became_peered":"2022-07-03T10:35:27.623202+0000","last_unstale":"2022-07-03T10:35:27.623637+0000","last_undegraded":"2022-07-03T10:35:27.623637+0000","last_fullsized":"2022-07-03T10:35:27.623637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:39:41.407533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310134+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:26.310134+0000","last_peered":"2022-07-03T10:35:26.310134+0000","last_clean":"2022-07-03T10:35:26.310134+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:26.310134+0000","last_undegraded":"2022-07-03T10:35:26.310134+0000","last_fullsized":"2022-07-03T10:35:26.310134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307218+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:35:26.307218+0000","last_peered":"2022-07-03T10:35:26.307218+0000","last_clean":"2022-07-03T10:35:26.307218+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:35:26.307218+0000","last_undegraded":"2022-07-03T10:35:26.307218+0000","last_fullsized":"2022-07-03T10:35:26.307218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.324898+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:26.324898+0000","last_peered":"2022-07-03T10:35:26.324898+0000","last_clean":"2022-07-03T10:35:26.324898+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:26.324898+0000","last_undegraded":"2022-07-03T10:35:26.324898+0000","last_fullsized":"2022-07-03T10:35:26.324898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309083+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:35:26.309083+0000","last_peered":"2022-07-03T10:35:26.309083+0000","last_clean":"2022-07-03T10:35:26.309083+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:35:26.309083+0000","last_undegraded":"2022-07-03T10:35:26.309083+0000","last_fullsized":"2022-07-03T10:35:26.309083+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347744+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:26.347744+0000","last_peered":"2022-07-03T10:35:26.347744+0000","last_clean":"2022-07-03T10:35:26.347744+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:26.347744+0000","last_undegraded":"2022-07-03T10:35:26.347744+0000","last_fullsized":"2022-07-03T10:35:26.347744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309643+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:35:26.309643+0000","last_peered":"2022-07-03T10:35:26.309643+0000","last_clean":"2022-07-03T10:35:26.309643+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:35:26.309643+0000","last_undegraded":"2022-07-03T10:35:26.309643+0000","last_fullsized":"2022-07-03T10:35:26.309643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305537+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:35:26.305537+0000","last_peered":"2022-07-03T10:35:26.305537+0000","last_clean":"2022-07-03T10:35:26.305537+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:35:26.305537+0000","last_undegraded":"2022-07-03T10:35:26.305537+0000","last_fullsized":"2022-07-03T10:35:26.305537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332452+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:26.332452+0000","last_peered":"2022-07-03T10:35:26.332452+0000","last_clean":"2022-07-03T10:35:26.332452+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:26.332452+0000","last_undegraded":"2022-07-03T10:35:26.332452+0000","last_fullsized":"2022-07-03T10:35:26.332452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341576+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:35:26.341576+0000","last_peered":"2022-07-03T10:35:26.341576+0000","last_clean":"2022-07-03T10:35:26.341576+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:35:26.341576+0000","last_undegraded":"2022-07-03T10:35:26.341576+0000","last_fullsized":"2022-07-03T10:35:26.341576+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302670+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:26.302670+0000","last_peered":"2022-07-03T10:35:26.302670+0000","last_clean":"2022-07-03T10:35:26.302670+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:26.302670+0000","last_undegraded":"2022-07-03T10:35:26.302670+0000","last_fullsized":"2022-07-03T10:35:26.302670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347662+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:26.347662+0000","last_peered":"2022-07-03T10:35:26.347662+0000","last_clean":"2022-07-03T10:35:26.347662+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:26.347662+0000","last_undegraded":"2022-07-03T10:35:26.347662+0000","last_fullsized":"2022-07-03T10:35:26.347662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332353+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:35:26.332353+0000","last_peered":"2022-07-03T10:35:26.332353+0000","last_clean":"2022-07-03T10:35:26.332353+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:35:26.332353+0000","last_undegraded":"2022-07-03T10:35:26.332353+0000","last_fullsized":"2022-07-03T10:35:26.332353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347752+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:26.347752+0000","last_peered":"2022-07-03T10:35:26.347752+0000","last_clean":"2022-07-03T10:35:26.347752+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:26.347752+0000","last_undegraded":"2022-07-03T10:35:26.347752+0000","last_fullsized":"2022-07-03T10:35:26.347752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309497+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:26.309497+0000","last_peered":"2022-07-03T10:35:26.309497+0000","last_clean":"2022-07-03T10:35:26.309497+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:26.309497+0000","last_undegraded":"2022-07-03T10:35:26.309497+0000","last_fullsized":"2022-07-03T10:35:26.309497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.333194+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:35:26.333194+0000","last_peered":"2022-07-03T10:35:26.333194+0000","last_clean":"2022-07-03T10:35:26.333194+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:35:26.333194+0000","last_undegraded":"2022-07-03T10:35:26.333194+0000","last_fullsized":"2022-07-03T10:35:26.333194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.414221+0000","last_change":"2022-07-03T10:35:27.414221+0000","last_active":"2022-07-03T10:35:27.414221+0000","last_peered":"2022-07-03T10:35:27.414221+0000","last_clean":"2022-07-03T10:35:27.414221+0000","last_became_active":"2022-07-03T10:35:27.323296+0000","last_became_peered":"2022-07-03T10:35:27.323296+0000","last_unstale":"2022-07-03T10:35:27.414221+0000","last_undegraded":"2022-07-03T10:35:27.414221+0000","last_fullsized":"2022-07-03T10:35:27.414221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:01:20.773865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341933+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:35:26.341933+0000","last_peered":"2022-07-03T10:35:26.341933+0000","last_clean":"2022-07-03T10:35:26.341933+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:35:26.341933+0000","last_undegraded":"2022-07-03T10:35:26.341933+0000","last_fullsized":"2022-07-03T10:35:26.341933+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302521+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:26.302521+0000","last_peered":"2022-07-03T10:35:26.302521+0000","last_clean":"2022-07-03T10:35:26.302521+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:26.302521+0000","last_undegraded":"2022-07-03T10:35:26.302521+0000","last_fullsized":"2022-07-03T10:35:26.302521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.325484+0000","last_change":"2022-07-03T10:35:27.325484+0000","last_active":"2022-07-03T10:35:27.325484+0000","last_peered":"2022-07-03T10:35:27.325484+0000","last_clean":"2022-07-03T10:35:27.325484+0000","last_became_active":"2022-07-03T10:35:27.324947+0000","last_became_peered":"2022-07-03T10:35:27.324947+0000","last_unstale":"2022-07-03T10:35:27.325484+0000","last_undegraded":"2022-07-03T10:35:27.325484+0000","last_fullsized":"2022-07-03T10:35:27.325484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:36.475275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309995+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:26.309995+0000","last_peered":"2022-07-03T10:35:26.309995+0000","last_clean":"2022-07-03T10:35:26.309995+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:26.309995+0000","last_undegraded":"2022-07-03T10:35:26.309995+0000","last_fullsized":"2022-07-03T10:35:26.309995+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308702+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:35:26.308702+0000","last_peered":"2022-07-03T10:35:26.308702+0000","last_clean":"2022-07-03T10:35:26.308702+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:35:26.308702+0000","last_undegraded":"2022-07-03T10:35:26.308702+0000","last_fullsized":"2022-07-03T10:35:26.308702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341921+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:26.341921+0000","last_peered":"2022-07-03T10:35:26.341921+0000","last_clean":"2022-07-03T10:35:26.341921+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:26.341921+0000","last_undegraded":"2022-07-03T10:35:26.341921+0000","last_fullsized":"2022-07-03T10:35:26.341921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306168+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:35:26.306168+0000","last_peered":"2022-07-03T10:35:26.306168+0000","last_clean":"2022-07-03T10:35:26.306168+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:35:26.306168+0000","last_undegraded":"2022-07-03T10:35:26.306168+0000","last_fullsized":"2022-07-03T10:35:26.306168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303560+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:26.303560+0000","last_peered":"2022-07-03T10:35:26.303560+0000","last_clean":"2022-07-03T10:35:26.303560+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:26.303560+0000","last_undegraded":"2022-07-03T10:35:26.303560+0000","last_fullsized":"2022-07-03T10:35:26.303560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.342577+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:35:26.342577+0000","last_peered":"2022-07-03T10:35:26.342577+0000","last_clean":"2022-07-03T10:35:26.342577+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:35:26.342577+0000","last_undegraded":"2022-07-03T10:35:26.342577+0000","last_fullsized":"2022-07-03T10:35:26.342577+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309116+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:26.309116+0000","last_peered":"2022-07-03T10:35:26.309116+0000","last_clean":"2022-07-03T10:35:26.309116+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:26.309116+0000","last_undegraded":"2022-07-03T10:35:26.309116+0000","last_fullsized":"2022-07-03T10:35:26.309116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347763+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:35:26.347763+0000","last_peered":"2022-07-03T10:35:26.347763+0000","last_clean":"2022-07-03T10:35:26.347763+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:35:26.347763+0000","last_undegraded":"2022-07-03T10:35:26.347763+0000","last_fullsized":"2022-07-03T10:35:26.347763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.312207+0000","last_change":"2022-07-03T10:35:27.312207+0000","last_active":"2022-07-03T10:35:27.312207+0000","last_peered":"2022-07-03T10:35:27.312207+0000","last_clean":"2022-07-03T10:35:27.312207+0000","last_became_active":"2022-07-03T10:35:27.311839+0000","last_became_peered":"2022-07-03T10:35:27.311839+0000","last_unstale":"2022-07-03T10:35:27.312207+0000","last_undegraded":"2022-07-03T10:35:27.312207+0000","last_fullsized":"2022-07-03T10:35:27.312207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:21:12.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347987+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:35:26.347987+0000","last_peered":"2022-07-03T10:35:26.347987+0000","last_clean":"2022-07-03T10:35:26.347987+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:35:26.347987+0000","last_undegraded":"2022-07-03T10:35:26.347987+0000","last_fullsized":"2022-07-03T10:35:26.347987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305852+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:26.305852+0000","last_peered":"2022-07-03T10:35:26.305852+0000","last_clean":"2022-07-03T10:35:26.305852+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:26.305852+0000","last_undegraded":"2022-07-03T10:35:26.305852+0000","last_fullsized":"2022-07-03T10:35:26.305852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347831+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:35:26.347831+0000","last_peered":"2022-07-03T10:35:26.347831+0000","last_clean":"2022-07-03T10:35:26.347831+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:35:26.347831+0000","last_undegraded":"2022-07-03T10:35:26.347831+0000","last_fullsized":"2022-07-03T10:35:26.347831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330620+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:26.330620+0000","last_peered":"2022-07-03T10:35:26.330620+0000","last_clean":"2022-07-03T10:35:26.330620+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:26.330620+0000","last_undegraded":"2022-07-03T10:35:26.330620+0000","last_fullsized":"2022-07-03T10:35:26.330620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347988+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:35:26.347988+0000","last_peered":"2022-07-03T10:35:26.347988+0000","last_clean":"2022-07-03T10:35:26.347988+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:35:26.347988+0000","last_undegraded":"2022-07-03T10:35:26.347988+0000","last_fullsized":"2022-07-03T10:35:26.347988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306271+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:35:26.306271+0000","last_peered":"2022-07-03T10:35:26.306271+0000","last_clean":"2022-07-03T10:35:26.306271+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:35:26.306271+0000","last_undegraded":"2022-07-03T10:35:26.306271+0000","last_fullsized":"2022-07-03T10:35:26.306271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.313043+0000","last_change":"2022-07-03T10:35:27.313043+0000","last_active":"2022-07-03T10:35:27.313043+0000","last_peered":"2022-07-03T10:35:27.313043+0000","last_clean":"2022-07-03T10:35:27.313043+0000","last_became_active":"2022-07-03T10:35:27.312553+0000","last_became_peered":"2022-07-03T10:35:27.312553+0000","last_unstale":"2022-07-03T10:35:27.313043+0000","last_undegraded":"2022-07-03T10:35:27.313043+0000","last_fullsized":"2022-07-03T10:35:27.313043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:05:28.297497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303480+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:35:26.303480+0000","last_peered":"2022-07-03T10:35:26.303480+0000","last_clean":"2022-07-03T10:35:26.303480+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:35:26.303480+0000","last_undegraded":"2022-07-03T10:35:26.303480+0000","last_fullsized":"2022-07-03T10:35:26.303480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327435+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:35:26.327435+0000","last_peered":"2022-07-03T10:35:26.327435+0000","last_clean":"2022-07-03T10:35:26.327435+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:35:26.327435+0000","last_undegraded":"2022-07-03T10:35:26.327435+0000","last_fullsized":"2022-07-03T10:35:26.327435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.333275+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:35:26.333275+0000","last_peered":"2022-07-03T10:35:26.333275+0000","last_clean":"2022-07-03T10:35:26.333275+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:35:26.333275+0000","last_undegraded":"2022-07-03T10:35:26.333275+0000","last_fullsized":"2022-07-03T10:35:26.333275+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307182+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:35:26.307182+0000","last_peered":"2022-07-03T10:35:26.307182+0000","last_clean":"2022-07-03T10:35:26.307182+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:35:26.307182+0000","last_undegraded":"2022-07-03T10:35:26.307182+0000","last_fullsized":"2022-07-03T10:35:26.307182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.304295+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:35:26.304295+0000","last_peered":"2022-07-03T10:35:26.304295+0000","last_clean":"2022-07-03T10:35:26.304295+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:35:26.304295+0000","last_undegraded":"2022-07-03T10:35:26.304295+0000","last_fullsized":"2022-07-03T10:35:26.304295+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341842+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:35:26.341842+0000","last_peered":"2022-07-03T10:35:26.341842+0000","last_clean":"2022-07-03T10:35:26.341842+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:35:26.341842+0000","last_undegraded":"2022-07-03T10:35:26.341842+0000","last_fullsized":"2022-07-03T10:35:26.341842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305468+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:35:26.305468+0000","last_peered":"2022-07-03T10:35:26.305468+0000","last_clean":"2022-07-03T10:35:26.305468+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:35:26.305468+0000","last_undegraded":"2022-07-03T10:35:26.305468+0000","last_fullsized":"2022-07-03T10:35:26.305468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341814+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:35:26.341814+0000","last_peered":"2022-07-03T10:35:26.341814+0000","last_clean":"2022-07-03T10:35:26.341814+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:35:26.341814+0000","last_undegraded":"2022-07-03T10:35:26.341814+0000","last_fullsized":"2022-07-03T10:35:26.341814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310230+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:35:26.310230+0000","last_peered":"2022-07-03T10:35:26.310230+0000","last_clean":"2022-07-03T10:35:26.310230+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:35:26.310230+0000","last_undegraded":"2022-07-03T10:35:26.310230+0000","last_fullsized":"2022-07-03T10:35:26.310230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332910+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:35:26.332910+0000","last_peered":"2022-07-03T10:35:26.332910+0000","last_clean":"2022-07-03T10:35:26.332910+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:35:26.332910+0000","last_undegraded":"2022-07-03T10:35:26.332910+0000","last_fullsized":"2022-07-03T10:35:26.332910+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307775+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:35:26.307775+0000","last_peered":"2022-07-03T10:35:26.307775+0000","last_clean":"2022-07-03T10:35:26.307775+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:35:26.307775+0000","last_undegraded":"2022-07-03T10:35:26.307775+0000","last_fullsized":"2022-07-03T10:35:26.307775+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54255,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":68,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":162765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547223,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5928,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739224,"statfs":{"total":95995035648,"available":95988965376,"internally_reserved":0,"allocated":884736,"data_stored":436583,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65500000000000003}]},{"osd":1,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.77300000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.68600000000000005}]},{"osd":5,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.59999999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.83399999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.877},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.754}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.89800000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.85499999999999998}]}]},{"osd":10,"up_from":67,"seq":287762808859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":1080384,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.52600000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.55500000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.501}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.66500000000000004,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71899999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59799999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.75700000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.63100000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85899999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.626}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.77600000000000002,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.752}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.46400000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74199999999999999,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.84699999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.73099999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.57899999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.879,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.81499999999999995},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.79400000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:35:18 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.76400000000000001}]}]},{"osd":9,"up_from":62,"seq":266287972385,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":437115,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":0.81899999999999995,"15min":0.71699999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":11.505000000000001,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.7350000000000001,"5min":0.92100000000000004,"15min":0.78500000000000003},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.582000000000001,"5min":11.582000000000001,"15min":11.582000000000001},"last":0.40400000000000003}]},{"osd":1,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":0.95099999999999996,"15min":0.85699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":11.933,"5min":11.933,"15min":11.933},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.393,"5min":1.0009999999999999,"15min":0.93500000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":11.411,"5min":11.411,"15min":11.411},"last":0.748}]},{"osd":2,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":0.95499999999999996,"15min":0.85899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.97699999999999998,"15min":0.90500000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":11.331,"5min":11.331,"15min":11.331},"last":0.626}]},{"osd":3,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":0.94899999999999995,"15min":0.86599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":9.8859999999999992,"5min":9.8859999999999992,"15min":9.8859999999999992},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.6830000000000001,"5min":0.96699999999999997,"15min":0.84799999999999998},"min":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":12.041,"5min":12.041,"15min":12.041},"last":0.57299999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.054,"15min":0.98799999999999999},"min":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":11.369,"5min":11.369,"15min":11.369},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.478,"5min":1.038,"15min":0.96499999999999997},"min":{"1min":0.54200000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":9.6769999999999996,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.77500000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":9.734},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":11.968}]},{"osd":6,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.129,"15min":1.042},"min":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":12.311,"5min":12.311,"15min":12.311},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.155,"15min":1.087},"min":{"1min":0.69999999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":11.534000000000001,"5min":11.534000000000001,"15min":11.534000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":9.8079999999999998,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.181,"15min":1.099},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":12.000999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.83599999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.206,"15min":1.1439999999999999},"min":{"1min":0.71899999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":11.441000000000001,"5min":11.441000000000001,"15min":11.441000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.1739999999999999,"15min":1.109},"min":{"1min":0.71299999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":9.8539999999999992,"5min":9.8539999999999992,"15min":9.8539999999999992},"last":0.73299999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.68799999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":8,"up_from":54,"seq":231928234021,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":888832,"data_stored":437137,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.67100000000000004,"15min":0.63},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.6449999999999996,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55900000000000005}]},{"osd":2,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.436,"15min":0.433},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.40999999999999998}]},{"osd":3,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.879,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.624},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59699999999999998}]},{"osd":4,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.66000000000000003,"15min":0.61399999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.6600000000000001,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.311,"15min":0.311},"max":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.53400000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.73799999999999999,"15min":0.69599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.6959999999999997,"5min":4.6959999999999997,"15min":4.6959999999999997},"last":1.357},{"interface":"front","average":{"1min":1.018,"5min":0.77300000000000002,"15min":0.73199999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.60499999999999998}]},{"osd":7,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.72099999999999997}]},{"osd":9,"last update":"Sun Jul 3 10:35:36 2022","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93000000000000005,"5min":1.117,"15min":1.117},"last":0.70599999999999996}]},{"osd":10,"last update":"Sun Jul 3 10:34:56 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.77400000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":7,"up_from":47,"seq":201863462953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":937984,"data_stored":1027606,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.37},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.309,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.45900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64000000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":1.222,"15min":1.222},"last":0.58799999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48999999999999999,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94799999999999995,"5min":5.077,"15min":5.077},"last":0.623},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70299999999999996,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.82999999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.52600000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.79900000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:35:42 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.92000000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.89600000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.74099999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":6,"up_from":43,"seq":184683593772,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":884736,"data_stored":436672,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.63400000000000001}]},{"osd":1,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63900000000000001},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83599999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.70599999999999996,"15min":0.68400000000000005},"min":{"1min":0.62,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95499999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.89200000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.87},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72799999999999998,"15min":0.71199999999999997},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.016,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":1.1140000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68500000000000005,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97299999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":1.05},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72099999999999997,"15min":0.70599999999999996},"min":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.226}]},{"osd":4,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.76600000000000001,"15min":0.755},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1559999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80200000000000005,"15min":0.79000000000000004},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.0800000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.01}]},{"osd":5,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.83199999999999996,"15min":0.81200000000000006},"min":{"1min":0.66500000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":1.0669999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:35:52 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86299999999999999,"15min":0.84099999999999997},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2869999999999999,"5min":1.333,"15min":1.333},"last":0.998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.878,"15min":0.86099999999999999},"min":{"1min":0.70399999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.151}]},{"osd":8,"last update":"Sun Jul 3 10:35:04 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.02}]},{"osd":9,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.96099999999999997,"15min":0.97399999999999998},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.1719999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":1.073}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":1.046},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":1.2370000000000001}]}]},{"osd":1,"up_from":12,"seq":51539607620,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7136,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738016,"statfs":{"total":95995035648,"available":95987728384,"internally_reserved":0,"allocated":942080,"data_stored":490022,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:43 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.38200000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.753,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.57499999999999996},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.44900000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:47 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.58899999999999997,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.879,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.54800000000000004},"min":{"1min":0.48599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Jul 3 10:35:20 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.54800000000000004},"min":{"1min":0.438,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82699999999999996,"5min":0.88,"15min":0.88},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77900000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48499999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:43 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.44,"15min":0.40500000000000003},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64700000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.441,"15min":0.40699999999999997},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.745,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.63800000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.55200000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.51600000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.83199999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:35:32 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.629},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67300000000000004,"15min":0.66500000000000004},"min":{"1min":0.46200000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.83199999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:51 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65600000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:35:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.93100000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.83499999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.751}]},{"osd":9,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79000000000000004,"15min":0.79200000000000004},"min":{"1min":0.622,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95299999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.006,"5min":1.103,"15min":1.103},"last":0.81799999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7016,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738136,"statfs":{"total":95995035648,"available":95987851264,"internally_reserved":0,"allocated":884736,"data_stored":436639,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.59599999999999997,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.59799999999999998,"15min":0.56699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.78300000000000003}]},{"osd":2,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.68100000000000005,"15min":0.63100000000000001},"min":{"1min":0.52900000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.67200000000000004}]},{"osd":3,"last update":"Sun Jul 3 10:33:16 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.46600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.81100000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70099999999999996,"15min":0.67000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.99299999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.63100000000000001},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.62}]},{"osd":5,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.069,"5min":1.141,"15min":1.141},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.03,"5min":1.256,"15min":1.256},"last":0.84699999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:35:29 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77300000000000002,"15min":0.749},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.339,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.749},"min":{"1min":0.52700000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1279999999999999,"5min":1.478,"15min":1.478},"last":0.71199999999999997}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.5,"5min":0.497,"15min":0.497},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85199999999999998},"min":{"1min":0.63500000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.60899999999999999}]},{"osd":8,"last update":"Sun Jul 3 10:35:12 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.137,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.087,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.89100000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83299999999999996,"15min":0.83199999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.96199999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.85699999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.65200000000000002}]},{"osd":11,"last update":"Sun Jul 3 10:35:14 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.754}]}]},{"osd":2,"up_from":16,"seq":68719476802,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7264,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737888,"statfs":{"total":95995035648,"available":95987597312,"internally_reserved":0,"allocated":942080,"data_stored":490022,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":1.595}]},{"osd":1,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69599999999999995,"5min":0.874,"15min":0.874},"last":0.71799999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84199999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56599999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.67500000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76400000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:35:00 2022","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.443,"15min":0.436},"min":{"1min":0.33700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Sun Jul 3 10:35:31 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.83499999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79100000000000004,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.68500000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.72899999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:35:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.83399999999999996,"5min":0.877,"15min":0.877},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.91100000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.58399999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.80400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":3,"up_from":23,"seq":98784247867,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":436899,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46999999999999997,"15min":0.432},"min":{"1min":0.379,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.502,"15min":0.45200000000000001},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.52500000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.623,"15min":0.58499999999999996},"min":{"1min":0.46200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.65400000000000003,"15min":0.59699999999999998},"min":{"1min":0.46500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.68300000000000005}]},{"osd":2,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69599999999999995,"15min":0.65100000000000002},"min":{"1min":0.54900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":2.581,"15min":2.581},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70499999999999996,"15min":0.66100000000000003},"min":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96499999999999997,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.66700000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.755,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.443,"15min":0.443},"max":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.751,"15min":0.66600000000000004},"min":{"1min":0.52000000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.57299999999999995}]},{"osd":5,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.77800000000000002,"15min":0.73099999999999998},"min":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.125,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.78900000000000003,"15min":0.754},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1180000000000001,"5min":2.46,"15min":2.46},"last":0.77000000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:34:26 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.64100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0589999999999999,"5min":1.214,"15min":1.214},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.79400000000000004,"15min":0.77400000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.90300000000000002,"15min":0.91100000000000003},"min":{"1min":0.65600000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.159,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.88900000000000001,"15min":0.88400000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.79300000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.95399999999999996,"15min":0.97599999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.145,"5min":1.256,"15min":1.256},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.98699999999999999,"15min":1.0109999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.119,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.89900000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.84199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":436571,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52500000000000002,"15min":0.48999999999999999},"min":{"1min":0.45300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79800000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.52800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.51300000000000001}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:40 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.55100000000000005},"min":{"1min":0.39700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78600000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.623},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57299999999999995,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.75800000000000001,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.57899999999999996}]},{"osd":3,"last update":"Sun Jul 3 10:35:40 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60099999999999998,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83499999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.55600000000000005},"min":{"1min":0.434,"5min":0.433,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.58099999999999996,"15min":0.56100000000000005},"min":{"1min":0.51400000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53400000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67600000000000005,"15min":0.65500000000000003},"min":{"1min":0.51600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.79200000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:07 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.46300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.77900000000000003}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95699999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.76900000000000002,"15min":0.76600000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.86599999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.84199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]}]},{"osd":5,"up_from":35,"seq":150323855410,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6436,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738716,"statfs":{"total":95995035648,"available":95988445184,"internally_reserved":0,"allocated":946176,"data_stored":1027222,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70899999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.39800000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55100000000000005}]},{"osd":1,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54200000000000004},"min":{"1min":0.42099999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78200000000000003,"5min":0.874,"15min":0.874},"last":0.626},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53000000000000003,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.315,"15min":0.315},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66400000000000003}]},{"osd":2,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40699999999999997,"15min":0.40600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.48299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.496},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.55600000000000005,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52900000000000003}]},{"osd":3,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.60499999999999998,"15min":0.59599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4129999999999998,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.68100000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59399999999999997,"15min":0.58299999999999996},"min":{"1min":0.47799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.79200000000000004,"5min":0.874,"15min":0.874},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.84799999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:35:31 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.50700000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":5.476,"5min":5.476,"15min":5.476},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.749,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:52 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.45600000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.89400000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.747},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.753,"15min":0.753},"last":0.79300000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":0.81299999999999994,"15min":0.76700000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.82499999999999996,"15min":0.78700000000000003},"min":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":5.4240000000000004,"5min":5.4240000000000004,"15min":5.4240000000000004},"last":0.875}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.93000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":53667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:36:00.174 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph pg dump --format=json 2022-07-03T10:36:00.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:00 smithi101 ceph-mon[85126]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:00 smithi188 ceph-mon[92425]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:00 smithi161 ceph-mon[91579]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:01.059 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:01 smithi188 ceph-mon[92425]: pgmap v299: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:01.693 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:01 smithi101 ceph-mon[85126]: pgmap v299: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:01 smithi161 ceph-mon[91579]: pgmap v299: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:02.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 -- 172.21.15.101:0/3563083584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 msgr2=0x7f0e540fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:02.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 --2- 172.21.15.101:0/3563083584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e540fd6e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0e44009860 tx=0x7f0e44005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 -- 172.21.15.101:0/3563083584 shutdown_connections 2022-07-03T10:36:02.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 --2- 172.21.15.101:0/3563083584 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e5410a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 --2- 172.21.15.101:0/3563083584 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 0x7f0e54103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 --2- 172.21.15.101:0/3563083584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e540fd6e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.630+0000 7f0e5c784700 1 -- 172.21.15.101:0/3563083584 >> 172.21.15.101:0/3563083584 conn(0x7f0e540f9220 msgr2=0x7f0e540fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:02.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.631+0000 7f0e5c784700 1 -- 172.21.15.101:0/3563083584 shutdown_connections 2022-07-03T10:36:02.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.631+0000 7f0e5c784700 1 -- 172.21.15.101:0/3563083584 wait complete. 2022-07-03T10:36:02.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.631+0000 7f0e5c784700 1 Processor -- start 2022-07-03T10:36:02.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 -- start start 2022-07-03T10:36:02.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:02.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 0x7f0e54116d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:02.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e54110900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:02.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54110ed0 con 0x7f0e540fddc0 2022-07-03T10:36:02.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54111040 con 0x7f0e541036b0 2022-07-03T10:36:02.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5c784700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54111200 con 0x7f0e54104050 2022-07-03T10:36:02.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5a520700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:02.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5a520700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42762/0 (socket says 172.21.15.101:42762) 2022-07-03T10:36:02.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.632+0000 7f0e5a520700 1 -- 172.21.15.101:0/2543684062 learned_addr learned my addr 172.21.15.101:0/2543684062 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:02.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5ad21700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e54110900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:02.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e59d1f700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 0x7f0e54116d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:02.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 -- 172.21.15.101:0/2543684062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 msgr2=0x7f0e54110900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:02.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e54110900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 -- 172.21.15.101:0/2543684062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 msgr2=0x7f0e54116d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:02.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 0x7f0e54116d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 -- 172.21.15.101:0/2543684062 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44009410 con 0x7f0e540fddc0 2022-07-03T10:36:02.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.633+0000 7f0e5a520700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000ea80 tx=0x7f0e50005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:02.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.634+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5000c7c0 con 0x7f0e540fddc0 2022-07-03T10:36:02.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.634+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e541114e0 con 0x7f0e540fddc0 2022-07-03T10:36:02.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.634+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0e5000cdd0 con 0x7f0e540fddc0 2022-07-03T10:36:02.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.634+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50010df0 con 0x7f0e540fddc0 2022-07-03T10:36:02.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.634+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5411d810 con 0x7f0e540fddc0 2022-07-03T10:36:02.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.635+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5404ccb0 con 0x7f0e540fddc0 2022-07-03T10:36:02.644 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.637+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f0e5000c040 con 0x7f0e540fddc0 2022-07-03T10:36:02.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.638+0000 7f0e4affd700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 0x7f0e40099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:02.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.638+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f0e50014070 con 0x7f0e540fddc0 2022-07-03T10:36:02.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.639+0000 7f0e59d1f700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 0x7f0e40099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:02.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.639+0000 7f0e59d1f700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 0x7f0e40099210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e54111c30 tx=0x7f0e44005040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:02.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.642+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f0e5008cd60 con 0x7f0e540fddc0 2022-07-03T10:36:02.826 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.824+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e54111f20 con 0x7f0e40096d60 2022-07-03T10:36:02.837 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.835+0000 7f0e4affd700 1 -- 172.21.15.101:0/2543684062 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+339001 (secure 0 0 0) 0x7f0e54111f20 con 0x7f0e40096d60 2022-07-03T10:36:02.845 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:02.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 msgr2=0x7f0e40099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:02.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 0x7f0e40099210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e54111c30 tx=0x7f0e44005040 comp rx=0 tx=0).stop 2022-07-03T10:36:02.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 msgr2=0x7f0e54116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:02.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0e5000ea80 tx=0x7f0e50005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 shutdown_connections 2022-07-03T10:36:02.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0e40096d60 0x7f0e40099210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e540fddc0 0x7f0e54116850 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e54104050 0x7f0e54110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 --2- 172.21.15.101:0/2543684062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e541036b0 0x7f0e54116d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:02.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 >> 172.21.15.101:0/2543684062 conn(0x7f0e540f9220 msgr2=0x7f0e54108a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:02.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 shutdown_connections 2022-07-03T10:36:02.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:02.841+0000 7f0e5c784700 1 -- 172.21.15.101:0/2543684062 wait complete. 2022-07-03T10:36:02.851 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:36:03.343 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2022-07-03T10:36:01.952374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644869,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":244,"num_read_kb":222,"num_write":251,"num_write_kb":3013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77444,"kb_used_data":10724,"kb_used_omap":0,"kb_used_meta":66624,"kb_avail":1124864380,"statfs":{"total":1151940427776,"available":1151861125120,"internally_reserved":0,"allocated":10981376,"data_stored":7172872,"data_compressed":176772,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":68222976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307572+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:35:26.307572+0000","last_peered":"2022-07-03T10:35:26.307572+0000","last_clean":"2022-07-03T10:35:26.307572+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:35:26.307572+0000","last_undegraded":"2022-07-03T10:35:26.307572+0000","last_fullsized":"2022-07-03T10:35:26.307572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.345037+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:35:26.345037+0000","last_peered":"2022-07-03T10:35:26.345037+0000","last_clean":"2022-07-03T10:35:26.345037+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:35:26.345037+0000","last_undegraded":"2022-07-03T10:35:26.345037+0000","last_fullsized":"2022-07-03T10:35:26.345037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327600+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:35:26.327600+0000","last_peered":"2022-07-03T10:35:26.327600+0000","last_clean":"2022-07-03T10:35:26.327600+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:35:26.327600+0000","last_undegraded":"2022-07-03T10:35:26.327600+0000","last_fullsized":"2022-07-03T10:35:26.327600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.315233+0000","last_change":"2022-07-03T10:35:27.315233+0000","last_active":"2022-07-03T10:35:27.315233+0000","last_peered":"2022-07-03T10:35:27.315233+0000","last_clean":"2022-07-03T10:35:27.315233+0000","last_became_active":"2022-07-03T10:35:27.314807+0000","last_became_peered":"2022-07-03T10:35:27.314807+0000","last_unstale":"2022-07-03T10:35:27.315233+0000","last_undegraded":"2022-07-03T10:35:27.315233+0000","last_fullsized":"2022-07-03T10:35:27.315233+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:37:28.362457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309008+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:35:26.309008+0000","last_peered":"2022-07-03T10:35:26.309008+0000","last_clean":"2022-07-03T10:35:26.309008+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:35:26.309008+0000","last_undegraded":"2022-07-03T10:35:26.309008+0000","last_fullsized":"2022-07-03T10:35:26.309008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309163+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:35:26.309163+0000","last_peered":"2022-07-03T10:35:26.309163+0000","last_clean":"2022-07-03T10:35:26.309163+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:35:26.309163+0000","last_undegraded":"2022-07-03T10:35:26.309163+0000","last_fullsized":"2022-07-03T10:35:26.309163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347625+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:35:26.347625+0000","last_peered":"2022-07-03T10:35:26.347625+0000","last_clean":"2022-07-03T10:35:26.347625+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:35:26.347625+0000","last_undegraded":"2022-07-03T10:35:26.347625+0000","last_fullsized":"2022-07-03T10:35:26.347625+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341101+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:35:26.341101+0000","last_peered":"2022-07-03T10:35:26.341101+0000","last_clean":"2022-07-03T10:35:26.341101+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:35:26.341101+0000","last_undegraded":"2022-07-03T10:35:26.341101+0000","last_fullsized":"2022-07-03T10:35:26.341101+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310755+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:35:26.310755+0000","last_peered":"2022-07-03T10:35:26.310755+0000","last_clean":"2022-07-03T10:35:26.310755+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:35:26.310755+0000","last_undegraded":"2022-07-03T10:35:26.310755+0000","last_fullsized":"2022-07-03T10:35:26.310755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309517+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:35:26.309517+0000","last_peered":"2022-07-03T10:35:26.309517+0000","last_clean":"2022-07-03T10:35:26.309517+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:35:26.309517+0000","last_undegraded":"2022-07-03T10:35:26.309517+0000","last_fullsized":"2022-07-03T10:35:26.309517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347953+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:35:26.347953+0000","last_peered":"2022-07-03T10:35:26.347953+0000","last_clean":"2022-07-03T10:35:26.347953+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:35:26.347953+0000","last_undegraded":"2022-07-03T10:35:26.347953+0000","last_fullsized":"2022-07-03T10:35:26.347953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302340+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:35:26.302340+0000","last_peered":"2022-07-03T10:35:26.302340+0000","last_clean":"2022-07-03T10:35:26.302340+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:26.302340+0000","last_undegraded":"2022-07-03T10:35:26.302340+0000","last_fullsized":"2022-07-03T10:35:26.302340+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332994+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:35:26.332994+0000","last_peered":"2022-07-03T10:35:26.332994+0000","last_clean":"2022-07-03T10:35:26.332994+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:35:26.332994+0000","last_undegraded":"2022-07-03T10:35:26.332994+0000","last_fullsized":"2022-07-03T10:35:26.332994+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332723+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:35:26.332723+0000","last_peered":"2022-07-03T10:35:26.332723+0000","last_clean":"2022-07-03T10:35:26.332723+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:35:26.332723+0000","last_undegraded":"2022-07-03T10:35:26.332723+0000","last_fullsized":"2022-07-03T10:35:26.332723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327163+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:35:26.327163+0000","last_peered":"2022-07-03T10:35:26.327163+0000","last_clean":"2022-07-03T10:35:26.327163+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:35:26.327163+0000","last_undegraded":"2022-07-03T10:35:26.327163+0000","last_fullsized":"2022-07-03T10:35:26.327163+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309877+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:35:26.309877+0000","last_peered":"2022-07-03T10:35:26.309877+0000","last_clean":"2022-07-03T10:35:26.309877+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:35:26.309877+0000","last_undegraded":"2022-07-03T10:35:26.309877+0000","last_fullsized":"2022-07-03T10:35:26.309877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307666+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:35:26.307666+0000","last_peered":"2022-07-03T10:35:26.307666+0000","last_clean":"2022-07-03T10:35:26.307666+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:35:26.307666+0000","last_undegraded":"2022-07-03T10:35:26.307666+0000","last_fullsized":"2022-07-03T10:35:26.307666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325729+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:35:26.325729+0000","last_peered":"2022-07-03T10:35:26.325729+0000","last_clean":"2022-07-03T10:35:26.325729+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:35:26.325729+0000","last_undegraded":"2022-07-03T10:35:26.325729+0000","last_fullsized":"2022-07-03T10:35:26.325729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303772+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:35:26.303772+0000","last_peered":"2022-07-03T10:35:26.303772+0000","last_clean":"2022-07-03T10:35:26.303772+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:35:26.303772+0000","last_undegraded":"2022-07-03T10:35:26.303772+0000","last_fullsized":"2022-07-03T10:35:26.303772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325415+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:26.325415+0000","last_peered":"2022-07-03T10:35:26.325415+0000","last_clean":"2022-07-03T10:35:26.325415+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:26.325415+0000","last_undegraded":"2022-07-03T10:35:26.325415+0000","last_fullsized":"2022-07-03T10:35:26.325415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343425+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:35:26.343425+0000","last_peered":"2022-07-03T10:35:26.343425+0000","last_clean":"2022-07-03T10:35:26.343425+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:35:26.343425+0000","last_undegraded":"2022-07-03T10:35:26.343425+0000","last_fullsized":"2022-07-03T10:35:26.343425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343531+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:26.343531+0000","last_peered":"2022-07-03T10:35:26.343531+0000","last_clean":"2022-07-03T10:35:26.343531+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:26.343531+0000","last_undegraded":"2022-07-03T10:35:26.343531+0000","last_fullsized":"2022-07-03T10:35:26.343531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.328016+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:35:26.328016+0000","last_peered":"2022-07-03T10:35:26.328016+0000","last_clean":"2022-07-03T10:35:26.328016+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:35:26.328016+0000","last_undegraded":"2022-07-03T10:35:26.328016+0000","last_fullsized":"2022-07-03T10:35:26.328016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309637+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:26.309637+0000","last_peered":"2022-07-03T10:35:26.309637+0000","last_clean":"2022-07-03T10:35:26.309637+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:26.309637+0000","last_undegraded":"2022-07-03T10:35:26.309637+0000","last_fullsized":"2022-07-03T10:35:26.309637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310404+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:35:26.310404+0000","last_peered":"2022-07-03T10:35:26.310404+0000","last_clean":"2022-07-03T10:35:26.310404+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:35:26.310404+0000","last_undegraded":"2022-07-03T10:35:26.310404+0000","last_fullsized":"2022-07-03T10:35:26.310404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309394+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:26.309394+0000","last_peered":"2022-07-03T10:35:26.309394+0000","last_clean":"2022-07-03T10:35:26.309394+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:26.309394+0000","last_undegraded":"2022-07-03T10:35:26.309394+0000","last_fullsized":"2022-07-03T10:35:26.309394+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306137+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:35:26.306137+0000","last_peered":"2022-07-03T10:35:26.306137+0000","last_clean":"2022-07-03T10:35:26.306137+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:35:26.306137+0000","last_undegraded":"2022-07-03T10:35:26.306137+0000","last_fullsized":"2022-07-03T10:35:26.306137+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305894+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:26.305894+0000","last_peered":"2022-07-03T10:35:26.305894+0000","last_clean":"2022-07-03T10:35:26.305894+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:26.305894+0000","last_undegraded":"2022-07-03T10:35:26.305894+0000","last_fullsized":"2022-07-03T10:35:26.305894+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308941+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:35:26.308941+0000","last_peered":"2022-07-03T10:35:26.308941+0000","last_clean":"2022-07-03T10:35:26.308941+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:35:26.308941+0000","last_undegraded":"2022-07-03T10:35:26.308941+0000","last_fullsized":"2022-07-03T10:35:26.308941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309034+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:26.309034+0000","last_peered":"2022-07-03T10:35:26.309034+0000","last_clean":"2022-07-03T10:35:26.309034+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:26.309034+0000","last_undegraded":"2022-07-03T10:35:26.309034+0000","last_fullsized":"2022-07-03T10:35:26.309034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.324978+0000","last_change":"2022-07-03T10:35:27.324978+0000","last_active":"2022-07-03T10:35:27.324978+0000","last_peered":"2022-07-03T10:35:27.324978+0000","last_clean":"2022-07-03T10:35:27.324978+0000","last_became_active":"2022-07-03T10:35:27.324570+0000","last_became_peered":"2022-07-03T10:35:27.324570+0000","last_unstale":"2022-07-03T10:35:27.324978+0000","last_undegraded":"2022-07-03T10:35:27.324978+0000","last_fullsized":"2022-07-03T10:35:27.324978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:18:06.187457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332535+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:26.332535+0000","last_peered":"2022-07-03T10:35:26.332535+0000","last_clean":"2022-07-03T10:35:26.332535+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:26.332535+0000","last_undegraded":"2022-07-03T10:35:26.332535+0000","last_fullsized":"2022-07-03T10:35:26.332535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341210+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:35:26.341210+0000","last_peered":"2022-07-03T10:35:26.341210+0000","last_clean":"2022-07-03T10:35:26.341210+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:35:26.341210+0000","last_undegraded":"2022-07-03T10:35:26.341210+0000","last_fullsized":"2022-07-03T10:35:26.341210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305427+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:26.305427+0000","last_peered":"2022-07-03T10:35:26.305427+0000","last_clean":"2022-07-03T10:35:26.305427+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:26.305427+0000","last_undegraded":"2022-07-03T10:35:26.305427+0000","last_fullsized":"2022-07-03T10:35:26.305427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.314929+0000","last_change":"2022-07-03T10:35:27.314929+0000","last_active":"2022-07-03T10:35:27.314929+0000","last_peered":"2022-07-03T10:35:27.314929+0000","last_clean":"2022-07-03T10:35:27.314929+0000","last_became_active":"2022-07-03T10:35:27.314542+0000","last_became_peered":"2022-07-03T10:35:27.314542+0000","last_unstale":"2022-07-03T10:35:27.314929+0000","last_undegraded":"2022-07-03T10:35:27.314929+0000","last_fullsized":"2022-07-03T10:35:27.314929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:50:25.278127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309250+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:26.309250+0000","last_peered":"2022-07-03T10:35:26.309250+0000","last_clean":"2022-07-03T10:35:26.309250+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:26.309250+0000","last_undegraded":"2022-07-03T10:35:26.309250+0000","last_fullsized":"2022-07-03T10:35:26.309250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310646+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:35:26.310646+0000","last_peered":"2022-07-03T10:35:26.310646+0000","last_clean":"2022-07-03T10:35:26.310646+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:35:26.310646+0000","last_undegraded":"2022-07-03T10:35:26.310646+0000","last_fullsized":"2022-07-03T10:35:26.310646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347933+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:26.347933+0000","last_peered":"2022-07-03T10:35:26.347933+0000","last_clean":"2022-07-03T10:35:26.347933+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:26.347933+0000","last_undegraded":"2022-07-03T10:35:26.347933+0000","last_fullsized":"2022-07-03T10:35:26.347933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327938+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:35:26.327938+0000","last_peered":"2022-07-03T10:35:26.327938+0000","last_clean":"2022-07-03T10:35:26.327938+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:35:26.327938+0000","last_undegraded":"2022-07-03T10:35:26.327938+0000","last_fullsized":"2022-07-03T10:35:26.327938+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306107+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:26.306107+0000","last_peered":"2022-07-03T10:35:26.306107+0000","last_clean":"2022-07-03T10:35:26.306107+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:26.306107+0000","last_undegraded":"2022-07-03T10:35:26.306107+0000","last_fullsized":"2022-07-03T10:35:26.306107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310386+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:35:26.310386+0000","last_peered":"2022-07-03T10:35:26.310386+0000","last_clean":"2022-07-03T10:35:26.310386+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:35:26.310386+0000","last_undegraded":"2022-07-03T10:35:26.310386+0000","last_fullsized":"2022-07-03T10:35:26.310386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'25","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:58.981174+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:35:58.981174+0000","last_peered":"2022-07-03T10:35:58.981174+0000","last_clean":"2022-07-03T10:35:58.981174+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:35:58.981174+0000","last_undegraded":"2022-07-03T10:35:58.981174+0000","last_fullsized":"2022-07-03T10:35:58.981174+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53473,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":113,"num_read_kb":108,"num_write":25,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309385+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:35:26.309385+0000","last_peered":"2022-07-03T10:35:26.309385+0000","last_clean":"2022-07-03T10:35:26.309385+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:35:26.309385+0000","last_undegraded":"2022-07-03T10:35:26.309385+0000","last_fullsized":"2022-07-03T10:35:26.309385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.326689+0000","last_change":"2022-07-03T10:35:27.326689+0000","last_active":"2022-07-03T10:35:27.326689+0000","last_peered":"2022-07-03T10:35:27.326689+0000","last_clean":"2022-07-03T10:35:27.326689+0000","last_became_active":"2022-07-03T10:35:27.320346+0000","last_became_peered":"2022-07-03T10:35:27.320346+0000","last_unstale":"2022-07-03T10:35:27.326689+0000","last_undegraded":"2022-07-03T10:35:27.326689+0000","last_fullsized":"2022-07-03T10:35:27.326689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:57:55.061972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330365+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:35:26.330365+0000","last_peered":"2022-07-03T10:35:26.330365+0000","last_clean":"2022-07-03T10:35:26.330365+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:35:26.330365+0000","last_undegraded":"2022-07-03T10:35:26.330365+0000","last_fullsized":"2022-07-03T10:35:26.330365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305459+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:26.305459+0000","last_peered":"2022-07-03T10:35:26.305459+0000","last_clean":"2022-07-03T10:35:26.305459+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:26.305459+0000","last_undegraded":"2022-07-03T10:35:26.305459+0000","last_fullsized":"2022-07-03T10:35:26.305459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347839+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:35:26.347839+0000","last_peered":"2022-07-03T10:35:26.347839+0000","last_clean":"2022-07-03T10:35:26.347839+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:35:26.347839+0000","last_undegraded":"2022-07-03T10:35:26.347839+0000","last_fullsized":"2022-07-03T10:35:26.347839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302039+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:26.302039+0000","last_peered":"2022-07-03T10:35:26.302039+0000","last_clean":"2022-07-03T10:35:26.302039+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:26.302039+0000","last_undegraded":"2022-07-03T10:35:26.302039+0000","last_fullsized":"2022-07-03T10:35:26.302039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347546+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:35:26.347546+0000","last_peered":"2022-07-03T10:35:26.347546+0000","last_clean":"2022-07-03T10:35:26.347546+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:35:26.347546+0000","last_undegraded":"2022-07-03T10:35:26.347546+0000","last_fullsized":"2022-07-03T10:35:26.347546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308728+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:26.308728+0000","last_peered":"2022-07-03T10:35:26.308728+0000","last_clean":"2022-07-03T10:35:26.308728+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:26.308728+0000","last_undegraded":"2022-07-03T10:35:26.308728+0000","last_fullsized":"2022-07-03T10:35:26.308728+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.623637+0000","last_change":"2022-07-03T10:35:27.623637+0000","last_active":"2022-07-03T10:35:27.623637+0000","last_peered":"2022-07-03T10:35:27.623637+0000","last_clean":"2022-07-03T10:35:27.623637+0000","last_became_active":"2022-07-03T10:35:27.623202+0000","last_became_peered":"2022-07-03T10:35:27.623202+0000","last_unstale":"2022-07-03T10:35:27.623637+0000","last_undegraded":"2022-07-03T10:35:27.623637+0000","last_fullsized":"2022-07-03T10:35:27.623637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:39:41.407533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310134+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:26.310134+0000","last_peered":"2022-07-03T10:35:26.310134+0000","last_clean":"2022-07-03T10:35:26.310134+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:26.310134+0000","last_undegraded":"2022-07-03T10:35:26.310134+0000","last_fullsized":"2022-07-03T10:35:26.310134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307218+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:35:26.307218+0000","last_peered":"2022-07-03T10:35:26.307218+0000","last_clean":"2022-07-03T10:35:26.307218+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:35:26.307218+0000","last_undegraded":"2022-07-03T10:35:26.307218+0000","last_fullsized":"2022-07-03T10:35:26.307218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.324898+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:26.324898+0000","last_peered":"2022-07-03T10:35:26.324898+0000","last_clean":"2022-07-03T10:35:26.324898+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:26.324898+0000","last_undegraded":"2022-07-03T10:35:26.324898+0000","last_fullsized":"2022-07-03T10:35:26.324898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309083+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:35:26.309083+0000","last_peered":"2022-07-03T10:35:26.309083+0000","last_clean":"2022-07-03T10:35:26.309083+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:35:26.309083+0000","last_undegraded":"2022-07-03T10:35:26.309083+0000","last_fullsized":"2022-07-03T10:35:26.309083+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347744+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:26.347744+0000","last_peered":"2022-07-03T10:35:26.347744+0000","last_clean":"2022-07-03T10:35:26.347744+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:26.347744+0000","last_undegraded":"2022-07-03T10:35:26.347744+0000","last_fullsized":"2022-07-03T10:35:26.347744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309643+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:35:26.309643+0000","last_peered":"2022-07-03T10:35:26.309643+0000","last_clean":"2022-07-03T10:35:26.309643+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:35:26.309643+0000","last_undegraded":"2022-07-03T10:35:26.309643+0000","last_fullsized":"2022-07-03T10:35:26.309643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305537+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:35:26.305537+0000","last_peered":"2022-07-03T10:35:26.305537+0000","last_clean":"2022-07-03T10:35:26.305537+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:35:26.305537+0000","last_undegraded":"2022-07-03T10:35:26.305537+0000","last_fullsized":"2022-07-03T10:35:26.305537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332452+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:26.332452+0000","last_peered":"2022-07-03T10:35:26.332452+0000","last_clean":"2022-07-03T10:35:26.332452+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:26.332452+0000","last_undegraded":"2022-07-03T10:35:26.332452+0000","last_fullsized":"2022-07-03T10:35:26.332452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341576+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:35:26.341576+0000","last_peered":"2022-07-03T10:35:26.341576+0000","last_clean":"2022-07-03T10:35:26.341576+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:35:26.341576+0000","last_undegraded":"2022-07-03T10:35:26.341576+0000","last_fullsized":"2022-07-03T10:35:26.341576+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302670+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:26.302670+0000","last_peered":"2022-07-03T10:35:26.302670+0000","last_clean":"2022-07-03T10:35:26.302670+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:26.302670+0000","last_undegraded":"2022-07-03T10:35:26.302670+0000","last_fullsized":"2022-07-03T10:35:26.302670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347662+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:26.347662+0000","last_peered":"2022-07-03T10:35:26.347662+0000","last_clean":"2022-07-03T10:35:26.347662+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:26.347662+0000","last_undegraded":"2022-07-03T10:35:26.347662+0000","last_fullsized":"2022-07-03T10:35:26.347662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332353+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:35:26.332353+0000","last_peered":"2022-07-03T10:35:26.332353+0000","last_clean":"2022-07-03T10:35:26.332353+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:35:26.332353+0000","last_undegraded":"2022-07-03T10:35:26.332353+0000","last_fullsized":"2022-07-03T10:35:26.332353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347752+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:26.347752+0000","last_peered":"2022-07-03T10:35:26.347752+0000","last_clean":"2022-07-03T10:35:26.347752+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:26.347752+0000","last_undegraded":"2022-07-03T10:35:26.347752+0000","last_fullsized":"2022-07-03T10:35:26.347752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309497+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:26.309497+0000","last_peered":"2022-07-03T10:35:26.309497+0000","last_clean":"2022-07-03T10:35:26.309497+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:26.309497+0000","last_undegraded":"2022-07-03T10:35:26.309497+0000","last_fullsized":"2022-07-03T10:35:26.309497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.333194+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:35:26.333194+0000","last_peered":"2022-07-03T10:35:26.333194+0000","last_clean":"2022-07-03T10:35:26.333194+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:35:26.333194+0000","last_undegraded":"2022-07-03T10:35:26.333194+0000","last_fullsized":"2022-07-03T10:35:26.333194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'3","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.414221+0000","last_change":"2022-07-03T10:35:27.414221+0000","last_active":"2022-07-03T10:35:27.414221+0000","last_peered":"2022-07-03T10:35:27.414221+0000","last_clean":"2022-07-03T10:35:27.414221+0000","last_became_active":"2022-07-03T10:35:27.323296+0000","last_became_peered":"2022-07-03T10:35:27.323296+0000","last_unstale":"2022-07-03T10:35:27.414221+0000","last_undegraded":"2022-07-03T10:35:27.414221+0000","last_fullsized":"2022-07-03T10:35:27.414221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:01:20.773865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341933+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:35:26.341933+0000","last_peered":"2022-07-03T10:35:26.341933+0000","last_clean":"2022-07-03T10:35:26.341933+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:35:26.341933+0000","last_undegraded":"2022-07-03T10:35:26.341933+0000","last_fullsized":"2022-07-03T10:35:26.341933+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302521+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:26.302521+0000","last_peered":"2022-07-03T10:35:26.302521+0000","last_clean":"2022-07-03T10:35:26.302521+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:26.302521+0000","last_undegraded":"2022-07-03T10:35:26.302521+0000","last_fullsized":"2022-07-03T10:35:26.302521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.325484+0000","last_change":"2022-07-03T10:35:27.325484+0000","last_active":"2022-07-03T10:35:27.325484+0000","last_peered":"2022-07-03T10:35:27.325484+0000","last_clean":"2022-07-03T10:35:27.325484+0000","last_became_active":"2022-07-03T10:35:27.324947+0000","last_became_peered":"2022-07-03T10:35:27.324947+0000","last_unstale":"2022-07-03T10:35:27.325484+0000","last_undegraded":"2022-07-03T10:35:27.325484+0000","last_fullsized":"2022-07-03T10:35:27.325484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:36.475275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309995+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:26.309995+0000","last_peered":"2022-07-03T10:35:26.309995+0000","last_clean":"2022-07-03T10:35:26.309995+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:26.309995+0000","last_undegraded":"2022-07-03T10:35:26.309995+0000","last_fullsized":"2022-07-03T10:35:26.309995+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308702+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:35:26.308702+0000","last_peered":"2022-07-03T10:35:26.308702+0000","last_clean":"2022-07-03T10:35:26.308702+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:35:26.308702+0000","last_undegraded":"2022-07-03T10:35:26.308702+0000","last_fullsized":"2022-07-03T10:35:26.308702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341921+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:26.341921+0000","last_peered":"2022-07-03T10:35:26.341921+0000","last_clean":"2022-07-03T10:35:26.341921+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:26.341921+0000","last_undegraded":"2022-07-03T10:35:26.341921+0000","last_fullsized":"2022-07-03T10:35:26.341921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306168+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:35:26.306168+0000","last_peered":"2022-07-03T10:35:26.306168+0000","last_clean":"2022-07-03T10:35:26.306168+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:35:26.306168+0000","last_undegraded":"2022-07-03T10:35:26.306168+0000","last_fullsized":"2022-07-03T10:35:26.306168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303560+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:26.303560+0000","last_peered":"2022-07-03T10:35:26.303560+0000","last_clean":"2022-07-03T10:35:26.303560+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:26.303560+0000","last_undegraded":"2022-07-03T10:35:26.303560+0000","last_fullsized":"2022-07-03T10:35:26.303560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.342577+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:35:26.342577+0000","last_peered":"2022-07-03T10:35:26.342577+0000","last_clean":"2022-07-03T10:35:26.342577+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:35:26.342577+0000","last_undegraded":"2022-07-03T10:35:26.342577+0000","last_fullsized":"2022-07-03T10:35:26.342577+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309116+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:26.309116+0000","last_peered":"2022-07-03T10:35:26.309116+0000","last_clean":"2022-07-03T10:35:26.309116+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:26.309116+0000","last_undegraded":"2022-07-03T10:35:26.309116+0000","last_fullsized":"2022-07-03T10:35:26.309116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347763+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:35:26.347763+0000","last_peered":"2022-07-03T10:35:26.347763+0000","last_clean":"2022-07-03T10:35:26.347763+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:35:26.347763+0000","last_undegraded":"2022-07-03T10:35:26.347763+0000","last_fullsized":"2022-07-03T10:35:26.347763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.312207+0000","last_change":"2022-07-03T10:35:27.312207+0000","last_active":"2022-07-03T10:35:27.312207+0000","last_peered":"2022-07-03T10:35:27.312207+0000","last_clean":"2022-07-03T10:35:27.312207+0000","last_became_active":"2022-07-03T10:35:27.311839+0000","last_became_peered":"2022-07-03T10:35:27.311839+0000","last_unstale":"2022-07-03T10:35:27.312207+0000","last_undegraded":"2022-07-03T10:35:27.312207+0000","last_fullsized":"2022-07-03T10:35:27.312207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:21:12.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347987+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:35:26.347987+0000","last_peered":"2022-07-03T10:35:26.347987+0000","last_clean":"2022-07-03T10:35:26.347987+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:35:26.347987+0000","last_undegraded":"2022-07-03T10:35:26.347987+0000","last_fullsized":"2022-07-03T10:35:26.347987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305852+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:26.305852+0000","last_peered":"2022-07-03T10:35:26.305852+0000","last_clean":"2022-07-03T10:35:26.305852+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:26.305852+0000","last_undegraded":"2022-07-03T10:35:26.305852+0000","last_fullsized":"2022-07-03T10:35:26.305852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347831+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:35:26.347831+0000","last_peered":"2022-07-03T10:35:26.347831+0000","last_clean":"2022-07-03T10:35:26.347831+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:35:26.347831+0000","last_undegraded":"2022-07-03T10:35:26.347831+0000","last_fullsized":"2022-07-03T10:35:26.347831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330620+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:26.330620+0000","last_peered":"2022-07-03T10:35:26.330620+0000","last_clean":"2022-07-03T10:35:26.330620+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:26.330620+0000","last_undegraded":"2022-07-03T10:35:26.330620+0000","last_fullsized":"2022-07-03T10:35:26.330620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347988+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:35:26.347988+0000","last_peered":"2022-07-03T10:35:26.347988+0000","last_clean":"2022-07-03T10:35:26.347988+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:35:26.347988+0000","last_undegraded":"2022-07-03T10:35:26.347988+0000","last_fullsized":"2022-07-03T10:35:26.347988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306271+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:35:26.306271+0000","last_peered":"2022-07-03T10:35:26.306271+0000","last_clean":"2022-07-03T10:35:26.306271+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:35:26.306271+0000","last_undegraded":"2022-07-03T10:35:26.306271+0000","last_fullsized":"2022-07-03T10:35:26.306271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.313043+0000","last_change":"2022-07-03T10:35:27.313043+0000","last_active":"2022-07-03T10:35:27.313043+0000","last_peered":"2022-07-03T10:35:27.313043+0000","last_clean":"2022-07-03T10:35:27.313043+0000","last_became_active":"2022-07-03T10:35:27.312553+0000","last_became_peered":"2022-07-03T10:35:27.312553+0000","last_unstale":"2022-07-03T10:35:27.313043+0000","last_undegraded":"2022-07-03T10:35:27.313043+0000","last_fullsized":"2022-07-03T10:35:27.313043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:05:28.297497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303480+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:35:26.303480+0000","last_peered":"2022-07-03T10:35:26.303480+0000","last_clean":"2022-07-03T10:35:26.303480+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:35:26.303480+0000","last_undegraded":"2022-07-03T10:35:26.303480+0000","last_fullsized":"2022-07-03T10:35:26.303480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.327435+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:35:26.327435+0000","last_peered":"2022-07-03T10:35:26.327435+0000","last_clean":"2022-07-03T10:35:26.327435+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:35:26.327435+0000","last_undegraded":"2022-07-03T10:35:26.327435+0000","last_fullsized":"2022-07-03T10:35:26.327435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.333275+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:35:26.333275+0000","last_peered":"2022-07-03T10:35:26.333275+0000","last_clean":"2022-07-03T10:35:26.333275+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:35:26.333275+0000","last_undegraded":"2022-07-03T10:35:26.333275+0000","last_fullsized":"2022-07-03T10:35:26.333275+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307182+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:35:26.307182+0000","last_peered":"2022-07-03T10:35:26.307182+0000","last_clean":"2022-07-03T10:35:26.307182+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:35:26.307182+0000","last_undegraded":"2022-07-03T10:35:26.307182+0000","last_fullsized":"2022-07-03T10:35:26.307182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.304295+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:35:26.304295+0000","last_peered":"2022-07-03T10:35:26.304295+0000","last_clean":"2022-07-03T10:35:26.304295+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:35:26.304295+0000","last_undegraded":"2022-07-03T10:35:26.304295+0000","last_fullsized":"2022-07-03T10:35:26.304295+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341842+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:35:26.341842+0000","last_peered":"2022-07-03T10:35:26.341842+0000","last_clean":"2022-07-03T10:35:26.341842+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:35:26.341842+0000","last_undegraded":"2022-07-03T10:35:26.341842+0000","last_fullsized":"2022-07-03T10:35:26.341842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305468+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:35:26.305468+0000","last_peered":"2022-07-03T10:35:26.305468+0000","last_clean":"2022-07-03T10:35:26.305468+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:35:26.305468+0000","last_undegraded":"2022-07-03T10:35:26.305468+0000","last_fullsized":"2022-07-03T10:35:26.305468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341814+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:35:26.341814+0000","last_peered":"2022-07-03T10:35:26.341814+0000","last_clean":"2022-07-03T10:35:26.341814+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:35:26.341814+0000","last_undegraded":"2022-07-03T10:35:26.341814+0000","last_fullsized":"2022-07-03T10:35:26.341814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310230+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:35:26.310230+0000","last_peered":"2022-07-03T10:35:26.310230+0000","last_clean":"2022-07-03T10:35:26.310230+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:35:26.310230+0000","last_undegraded":"2022-07-03T10:35:26.310230+0000","last_fullsized":"2022-07-03T10:35:26.310230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332910+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:35:26.332910+0000","last_peered":"2022-07-03T10:35:26.332910+0000","last_clean":"2022-07-03T10:35:26.332910+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:35:26.332910+0000","last_undegraded":"2022-07-03T10:35:26.332910+0000","last_fullsized":"2022-07-03T10:35:26.332910+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.307775+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:35:26.307775+0000","last_peered":"2022-07-03T10:35:26.307775+0000","last_clean":"2022-07-03T10:35:26.307775+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:35:26.307775+0000","last_undegraded":"2022-07-03T10:35:26.307775+0000","last_fullsized":"2022-07-03T10:35:26.307775+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54255,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":118,"num_read_kb":113,"num_write":68,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":162765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547224,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5928,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739224,"statfs":{"total":95995035648,"available":95988965376,"internally_reserved":0,"allocated":884736,"data_stored":436583,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.41399999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.628}]},{"osd":2,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.60499999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.55800000000000005}]},{"osd":6,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.83399999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.68400000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":9,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.84299999999999997}]},{"osd":10,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.73099999999999998}]}]},{"osd":10,"up_from":67,"seq":287762808860,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":1003520,"data_stored":1080384,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.52600000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.55500000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.498}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.66500000000000004,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71899999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.63700000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.75700000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.623}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.67600000000000005}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85899999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66600000000000004}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.77600000000000002,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.46400000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74199999999999999,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.84699999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.77500000000000002}]},{"osd":9,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.57899999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.879,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.81499999999999995},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.80400000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:35:18 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.73899999999999999}]}]},{"osd":9,"up_from":62,"seq":266287972385,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":437115,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":0.81899999999999995,"15min":0.71699999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":11.505000000000001,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.7350000000000001,"5min":0.92100000000000004,"15min":0.78500000000000003},"min":{"1min":0.39700000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":11.582000000000001,"5min":11.582000000000001,"15min":11.582000000000001},"last":0.40400000000000003}]},{"osd":1,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":0.95099999999999996,"15min":0.85699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":11.933,"5min":11.933,"15min":11.933},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.393,"5min":1.0009999999999999,"15min":0.93500000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":11.411,"5min":11.411,"15min":11.411},"last":0.748}]},{"osd":2,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":0.95499999999999996,"15min":0.85899999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.97699999999999998,"15min":0.90500000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":11.331,"5min":11.331,"15min":11.331},"last":0.626}]},{"osd":3,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":0.94899999999999995,"15min":0.86599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":9.8859999999999992,"5min":9.8859999999999992,"15min":9.8859999999999992},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.6830000000000001,"5min":0.96699999999999997,"15min":0.84799999999999998},"min":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":12.041,"5min":12.041,"15min":12.041},"last":0.57299999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.054,"15min":0.98799999999999999},"min":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":11.369,"5min":11.369,"15min":11.369},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.478,"5min":1.038,"15min":0.96499999999999997},"min":{"1min":0.54200000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":9.6769999999999996,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.77500000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":9.734},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":11.968}]},{"osd":6,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.129,"15min":1.042},"min":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":12.311,"5min":12.311,"15min":12.311},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.155,"15min":1.087},"min":{"1min":0.69999999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":11.534000000000001,"5min":11.534000000000001,"15min":11.534000000000001},"last":0.55300000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":9.8079999999999998,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.181,"15min":1.099},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":12.000999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.83599999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.206,"15min":1.1439999999999999},"min":{"1min":0.71899999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":11.441000000000001,"5min":11.441000000000001,"15min":11.441000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.1739999999999999,"15min":1.109},"min":{"1min":0.71299999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":9.8539999999999992,"5min":9.8539999999999992,"15min":9.8539999999999992},"last":0.73299999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.68799999999999994}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":8,"up_from":54,"seq":231928234022,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":888832,"data_stored":437137,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.67100000000000004,"15min":0.63},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":4.6449999999999996,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.54900000000000004}]},{"osd":1,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.749,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61099999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.436,"15min":0.433},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.44900000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.60699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.879,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.60699999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.61899999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69899999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.66000000000000003,"15min":0.61399999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.6600000000000001,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.312,"5min":0.311,"15min":0.311},"max":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.48499999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.73799999999999999,"15min":0.69599999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":4.6959999999999997,"5min":4.6959999999999997,"15min":4.6959999999999997},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.018,"5min":0.77300000000000002,"15min":0.73199999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.66900000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:35:09 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64200000000000002,"15min":0.63800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73399999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:35:36 2022","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.79300000000000004,"15min":0.753},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93000000000000005,"5min":1.117,"15min":1.117},"last":0.72499999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.51300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.75}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":7,"up_from":47,"seq":201863462953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6300,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":937984,"data_stored":1027606,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.37},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.309,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.45900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64000000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":1.222,"15min":1.222},"last":0.58799999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48999999999999999,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56299999999999994}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.70899999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94799999999999995,"5min":5.077,"15min":5.077},"last":0.623},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70299999999999996,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.82999999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66400000000000003,"15min":0.65800000000000003},"min":{"1min":0.52600000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.79900000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:35:42 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.92000000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.89600000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.74099999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:34:55 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":6,"up_from":43,"seq":184683593773,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6184,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738968,"statfs":{"total":95995035648,"available":95988703232,"internally_reserved":0,"allocated":884736,"data_stored":436672,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48599999999999999}]},{"osd":1,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.63900000000000001},"min":{"1min":0.48099999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83599999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.70599999999999996,"15min":0.68400000000000005},"min":{"1min":0.62,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.95499999999999996,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.66500000000000004}]},{"osd":2,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92000000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72799999999999998,"15min":0.71199999999999997},"min":{"1min":0.56100000000000005,"5min":0.495,"15min":0.495},"max":{"1min":1.016,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76200000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68500000000000005,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.97299999999999998,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72099999999999997,"15min":0.70599999999999996},"min":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.72299999999999998}]},{"osd":4,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.76600000000000001,"15min":0.755},"min":{"1min":0.52200000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1559999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80200000000000005,"15min":0.79000000000000004},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.0800000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.81100000000000005,"15min":0.79600000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.83199999999999996,"15min":0.81200000000000006},"min":{"1min":0.66500000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.79700000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:35:52 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86299999999999999,"15min":0.84099999999999997},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2869999999999999,"5min":1.333,"15min":1.333},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.878,"15min":0.86099999999999999},"min":{"1min":0.70399999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.78800000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:35:04 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.85999999999999999,"15min":0.84999999999999998},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.624,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2509999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.84299999999999997}]},{"osd":9,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.96099999999999997,"15min":0.97399999999999998},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.1719999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.89500000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.877}]}]},{"osd":1,"up_from":12,"seq":51539607621,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7136,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738016,"statfs":{"total":95995035648,"available":95987728384,"internally_reserved":0,"allocated":942080,"data_stored":490022,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:43 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.38200000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.753,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.59899999999999998,"15min":0.57499999999999996},"min":{"1min":0.45000000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88100000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.61599999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:35:47 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.58899999999999997,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.879,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.54800000000000004},"min":{"1min":0.48599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.66800000000000004}]},{"osd":3,"last update":"Sun Jul 3 10:35:20 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.54800000000000004},"min":{"1min":0.438,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82699999999999996,"5min":0.88,"15min":0.88},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77900000000000003,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.68600000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:35:43 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.44,"15min":0.40500000000000003},"min":{"1min":0.34599999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64700000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.441,"15min":0.40699999999999997},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.745,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.79000000000000004}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":1.044,"15min":1.044},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.60999999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88800000000000001,"5min":1.028,"15min":1.028},"last":0.88500000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:35:32 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86399999999999999,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.67300000000000004,"15min":0.66500000000000004},"min":{"1min":0.46200000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.83199999999999996,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.77600000000000002}]},{"osd":7,"last update":"Sun Jul 3 10:35:51 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.85899999999999999}]},{"osd":8,"last update":"Sun Jul 3 10:35:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.93100000000000005,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.66000000000000003,"15min":0.66100000000000003},"min":{"1min":0.52400000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.83499999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.83399999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:35:37 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79000000000000004,"15min":0.79200000000000004},"min":{"1min":0.622,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95299999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60799999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.006,"5min":1.103,"15min":1.103},"last":0.96999999999999997}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90800000000000003,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.752,"15min":0.748},"min":{"1min":0.53800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.89900000000000002,"5min":1.069,"15min":1.069},"last":0.91300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7016,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738136,"statfs":{"total":95995035648,"available":95987851264,"internally_reserved":0,"allocated":884736,"data_stored":436639,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.59599999999999997,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.59799999999999998,"15min":0.56699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.42799999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.68100000000000005,"15min":0.63100000000000001},"min":{"1min":0.52900000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Jul 3 10:33:16 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.46600000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.629},{"interface":"front","average":{"1min":0.622,"5min":0.56200000000000006,"15min":0.55200000000000005},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67800000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70099999999999996,"15min":0.67000000000000004},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.99299999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66100000000000003,"15min":0.63100000000000001},"min":{"1min":0.47199999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:34:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.72899999999999998,"15min":0.71299999999999997},"min":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.069,"5min":1.141,"15min":1.141},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.03,"5min":1.256,"15min":1.256},"last":0.81000000000000005}]},{"osd":6,"last update":"Sun Jul 3 10:35:29 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.77300000000000002,"15min":0.749},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.339,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.749},"min":{"1min":0.52700000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1279999999999999,"5min":1.478,"15min":1.478},"last":0.54000000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72299999999999998,"15min":0.71899999999999997},"min":{"1min":0.5,"5min":0.497,"15min":0.497},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85099999999999998,"15min":0.85199999999999998},"min":{"1min":0.63500000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.59299999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:35:12 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.876,"15min":0.88200000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.137,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.087,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.73899999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83299999999999996,"15min":0.83199999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.96199999999999997,"15min":0.96699999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.66500000000000004}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.90200000000000002,"15min":0.90700000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72199999999999998}]},{"osd":11,"last update":"Sun Jul 3 10:35:14 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.83099999999999996}]}]},{"osd":2,"up_from":16,"seq":68719476802,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7264,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737888,"statfs":{"total":95995035648,"available":95987597312,"internally_reserved":0,"allocated":942080,"data_stored":490022,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":1.595}]},{"osd":1,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69599999999999995,"5min":0.874,"15min":0.874},"last":0.71799999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.38300000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84199999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.41599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56599999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:34:36 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54400000000000004},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70399999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.67500000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76400000000000001}]},{"osd":5,"last update":"Sun Jul 3 10:35:00 2022","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.443,"15min":0.436},"min":{"1min":0.33700000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.625,"5min":0.629,"15min":0.629},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67600000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Sun Jul 3 10:35:31 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66100000000000003,"15min":0.64700000000000002},"min":{"1min":0.44700000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.83499999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79100000000000004,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.68500000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.86099999999999999,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.72899999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:35:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.83399999999999996,"5min":0.877,"15min":0.877},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.74299999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.91100000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.58399999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.80400000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":3,"up_from":23,"seq":98784247868,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":436899,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.46999999999999997,"15min":0.432},"min":{"1min":0.379,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.502,"15min":0.45200000000000001},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.39200000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.623,"15min":0.58499999999999996},"min":{"1min":0.46200000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.626},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.65400000000000003,"15min":0.59699999999999998},"min":{"1min":0.46500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Sun Jul 3 10:35:17 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.69599999999999995,"15min":0.65100000000000002},"min":{"1min":0.54900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0289999999999999,"5min":2.581,"15min":2.581},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70499999999999996,"15min":0.66100000000000003},"min":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96499999999999997,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.69599999999999995}]},{"osd":4,"last update":"Sun Jul 3 10:35:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":0.755,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.443,"15min":0.443},"max":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.751,"15min":0.66600000000000004},"min":{"1min":0.52000000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":5.1749999999999998,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.55200000000000005}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.657,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81100000000000005,"15min":0.76100000000000001},"min":{"1min":0.52100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.2519999999999998,"5min":2.46,"15min":2.46},"last":0.747}]},{"osd":6,"last update":"Sun Jul 3 10:34:26 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.80600000000000005,"15min":0.79500000000000004},"min":{"1min":0.64100000000000001,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0589999999999999,"5min":1.214,"15min":1.214},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.79400000000000004,"15min":0.77400000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.71799999999999997}]},{"osd":7,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90500000000000003,"15min":0.91200000000000003},"min":{"1min":0.65400000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.65900000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:35:33 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.95399999999999996,"15min":0.97599999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.145,"5min":1.256,"15min":1.256},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.98699999999999999,"15min":1.0109999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.119,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.72799999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]}]},{"osd":4,"up_from":30,"seq":128849018936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":436571,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52500000000000002,"15min":0.48999999999999999},"min":{"1min":0.45300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79800000000000004,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.52800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.40699999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:40 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.55100000000000005},"min":{"1min":0.39700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78600000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57299999999999995,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.75800000000000001,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58599999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:35:40 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60099999999999998,"15min":0.56299999999999994},"min":{"1min":0.47299999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83499999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.503},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59599999999999997,"15min":0.55600000000000005},"min":{"1min":0.434,"5min":0.433,"15min":0.433},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.498}]},{"osd":5,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.40200000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.627},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.58099999999999996,"15min":0.56100000000000005},"min":{"1min":0.51400000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48799999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.82499999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67600000000000005,"15min":0.65500000000000003},"min":{"1min":0.51600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.45000000000000001}]},{"osd":7,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53400000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:35:07 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.65900000000000003,"15min":0.64500000000000002},"min":{"1min":0.46300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.65500000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.60799999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.79600000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95699999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.76900000000000002,"15min":0.76600000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.68500000000000005}]},{"osd":10,"last update":"Sun Jul 3 10:34:59 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]}]},{"osd":5,"up_from":35,"seq":150323855411,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6436,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738716,"statfs":{"total":95995035648,"available":95988445184,"internally_reserved":0,"allocated":946176,"data_stored":1027222,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.64000000000000001,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"last":0.55500000000000005}]},{"osd":1,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.32900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.433,"15min":0.432},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56499999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53400000000000003}]},{"osd":3,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.69799999999999995,"15min":0.627},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.21,"5min":7.21,"15min":7.21},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.02},"last":0.745}]},{"osd":4,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63600000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:35:31 2022","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.50700000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":5.476,"5min":5.476,"15min":5.476},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.749,"15min":0.72099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":3.27,"5min":3.27,"15min":3.27},"last":0.65200000000000002}]},{"osd":7,"last update":"Sun Jul 3 10:35:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":0.875,"15min":0.875},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66200000000000003,"15min":0.64600000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.76600000000000001}]},{"osd":8,"last update":"Sun Jul 3 10:35:06 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.39500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.753,"15min":0.753},"last":0.60299999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:35:34 2022","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":0.81299999999999994,"15min":0.76700000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.82499999999999996,"15min":0.78700000000000003},"min":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":5.4240000000000004,"5min":5.4240000000000004,"15min":5.4240000000000004},"last":0.75900000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":53473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":53667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:36:03.351 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-07-03T10:36:03.351 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-07-03T10:36:03.352 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-07-03T10:36:03.352 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph health --format=json 2022-07-03T10:36:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:03 smithi188 ceph-mon[92425]: pgmap v300: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:03.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:03 smithi101 ceph-mon[85126]: pgmap v300: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:03 smithi161 ceph-mon[91579]: pgmap v300: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:04.240 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:04 smithi188 ceph-mon[92425]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:04.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:04 smithi101 ceph-mon[85126]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:04 smithi161 ceph-mon[91579]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:05 smithi188 ceph-mon[92425]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:05.681 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:05 smithi101 ceph-mon[85126]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:05 smithi161 ceph-mon[91579]: pgmap v301: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:05.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 -- 172.21.15.101:0/28634448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 msgr2=0x7f1af810a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:05.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 --2- 172.21.15.101:0/28634448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af810a4b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009bf0 tx=0x7f1af0009460 comp rx=0 tx=0).stop 2022-07-03T10:36:05.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 -- 172.21.15.101:0/28634448 shutdown_connections 2022-07-03T10:36:05.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 --2- 172.21.15.101:0/28634448 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af810a4b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:05.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 --2- 172.21.15.101:0/28634448 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 0x7f1af8103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:05.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 --2- 172.21.15.101:0/28634448 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 0x7f1af80fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:05.983 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 -- 172.21.15.101:0/28634448 >> 172.21.15.101:0/28634448 conn(0x7f1af80f9200 msgr2=0x7f1af80fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:05.983 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 -- 172.21.15.101:0/28634448 shutdown_connections 2022-07-03T10:36:05.983 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.979+0000 7f1afe853700 1 -- 172.21.15.101:0/28634448 wait complete. 2022-07-03T10:36:05.985 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.980+0000 7f1afe853700 1 Processor -- start 2022-07-03T10:36:05.985 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.980+0000 7f1afe853700 1 -- start start 2022-07-03T10:36:05.986 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:05.986 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 0x7f1af8116b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:05.986 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 0x7f1af8110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:05.986 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af8110c40 con 0x7f1af8104050 2022-07-03T10:36:05.987 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af8110db0 con 0x7f1af80fddc0 2022-07-03T10:36:05.987 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afe853700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af8110f70 con 0x7f1af81036b0 2022-07-03T10:36:05.987 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:05.987 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:35982/0 (socket says 172.21.15.101:35982) 2022-07-03T10:36:05.988 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 -- 172.21.15.101:0/3235117611 learned_addr learned my addr 172.21.15.101:0/3235117611 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:05.988 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1afcdf0700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 0x7f1af8110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:05.988 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af77fe700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 0x7f1af8116b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:05.988 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 -- 172.21.15.101:0/3235117611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 msgr2=0x7f1af8116b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:05.989 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 0x7f1af8116b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:05.989 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 -- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 msgr2=0x7f1af8110670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:05.989 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 0x7f1af8110670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:05.989 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.981+0000 7f1af7fff700 1 -- 172.21.15.101:0/3235117611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0009700 con 0x7f1af80fddc0 2022-07-03T10:36:05.990 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1af7fff700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800ea80 tx=0x7f1ae80091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:05.990 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8005020 con 0x7f1af80fddc0 2022-07-03T10:36:05.990 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1ae800c7c0 con 0x7f1af80fddc0 2022-07-03T10:36:05.990 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8004920 con 0x7f1af80fddc0 2022-07-03T10:36:05.991 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af8111250 con 0x7f1af80fddc0 2022-07-03T10:36:05.991 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.982+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af811d580 con 0x7f1af80fddc0 2022-07-03T10:36:05.991 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.983+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af804ccb0 con 0x7f1af80fddc0 2022-07-03T10:36:05.992 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.985+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f1ae800c040 con 0x7f1af80fddc0 2022-07-03T10:36:05.992 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.987+0000 7f1af4ff9700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 0x7f1ae0099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:05.993 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.987+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f1ae8014070 con 0x7f1af80fddc0 2022-07-03T10:36:05.993 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.987+0000 7f1af77fe700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 0x7f1ae0099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:05.993 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.987+0000 7f1af77fe700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 0x7f1ae0099290 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1af8111e40 tx=0x7f1aec005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:05.993 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:05.989+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f1ae808d460 con 0x7f1af80fddc0 2022-07-03T10:36:06.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.212+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1af811db90 con 0x7f1af80fddc0 2022-07-03T10:36:06.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.213+0000 7f1af4ff9700 1 -- 172.21.15.101:0/3235117611 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1ae8095080 con 0x7f1af80fddc0 2022-07-03T10:36:06.215 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:06.215 INFO:teuthology.orchestra.run.smithi101.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-03T10:36:06.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 msgr2=0x7f1ae0099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:06.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 0x7f1ae0099290 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1af8111e40 tx=0x7f1aec005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 msgr2=0x7f1af81165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:06.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800ea80 tx=0x7f1ae80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 shutdown_connections 2022-07-03T10:36:06.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1ae0096de0 0x7f1ae0099290 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1af8104050 0x7f1af8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1af80fddc0 0x7f1af81165c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 --2- 172.21.15.101:0/3235117611 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1af81036b0 0x7f1af8116b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 >> 172.21.15.101:0/3235117611 conn(0x7f1af80f9200 msgr2=0x7f1af8070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:06.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.215+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 shutdown_connections 2022-07-03T10:36:06.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.216+0000 7f1afe853700 1 -- 172.21.15.101:0/3235117611 wait complete. 2022-07-03T10:36:06.656 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:06 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3235117611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-07-03T10:36:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:06 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3235117611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-07-03T10:36:06.716 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-07-03T10:36:06.717 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:06 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3235117611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-07-03T10:36:06.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 -- 172.21.15.101:0/267356569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 msgr2=0x7fae380fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:06.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 --2- 172.21.15.101:0/267356569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae380fd7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fae28009900 tx=0x7fae280091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 -- 172.21.15.101:0/267356569 shutdown_connections 2022-07-03T10:36:06.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 --2- 172.21.15.101:0/267356569 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fefe0 0x7fae381058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 --2- 172.21.15.101:0/267356569 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae380fd7e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 --2- 172.21.15.101:0/267356569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fe650 0x7fae380feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 -- 172.21.15.101:0/267356569 >> 172.21.15.101:0/267356569 conn(0x7fae380f8b20 msgr2=0x7fae380faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:06.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 -- 172.21.15.101:0/267356569 shutdown_connections 2022-07-03T10:36:06.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.947+0000 7fae37fff700 1 -- 172.21.15.101:0/267356569 wait complete. 2022-07-03T10:36:06.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.949+0000 7fae37fff700 1 Processor -- start 2022-07-03T10:36:06.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.949+0000 7fae37fff700 1 -- start start 2022-07-03T10:36:06.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae3810e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:06.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 0x7fae3810e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:06.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 0x7fae38115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:06.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38115f00 con 0x7fae380fe650 2022-07-03T10:36:06.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38116040 con 0x7fae380fd3d0 2022-07-03T10:36:06.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae37fff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae38116200 con 0x7fae380fefe0 2022-07-03T10:36:06.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae367fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 0x7fae3810e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:06.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae36ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae3810e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:06.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae367fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 0x7fae3810e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42804/0 (socket says 172.21.15.101:42804) 2022-07-03T10:36:06.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae367fc700 1 -- 172.21.15.101:0/1941648231 learned_addr learned my addr 172.21.15.101:0/1941648231 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:06.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.950+0000 7fae377fe700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 0x7fae38115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:06.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 -- 172.21.15.101:0/1941648231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 msgr2=0x7fae38115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:06.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 0x7fae38115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 -- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 msgr2=0x7fae3810e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:06.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 0x7fae3810e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:06.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 -- 172.21.15.101:0/1941648231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28009440 con 0x7fae380fd3d0 2022-07-03T10:36:06.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae377fe700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 0x7fae38115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:06.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae36ffd700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae3810e3d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fae28006010 tx=0x7fae280091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:06.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.951+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2801d070 con 0x7fae380fd3d0 2022-07-03T10:36:06.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.952+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae38116480 con 0x7fae380fd3d0 2022-07-03T10:36:06.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.952+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fae2800b9c0 con 0x7fae380fd3d0 2022-07-03T10:36:06.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.952+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28022ac0 con 0x7fae380fd3d0 2022-07-03T10:36:06.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.952+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae38116980 con 0x7fae380fd3d0 2022-07-03T10:36:06.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.953+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae38108ba0 con 0x7fae380fd3d0 2022-07-03T10:36:06.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.955+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fae28029030 con 0x7fae380fd3d0 2022-07-03T10:36:06.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.956+0000 7fae1f7fe700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 0x7fae20099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:06.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.956+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fae280c0db0 con 0x7fae380fd3d0 2022-07-03T10:36:06.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.956+0000 7fae367fc700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 0x7fae20099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:06.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.957+0000 7fae367fc700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 0x7fae20099210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009d70 tx=0x7fae2c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:06.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:06.958+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fae2808e010 con 0x7fae380fd3d0 2022-07-03T10:36:07.149 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.147+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae381045c0 con 0x7fae380fd3d0 2022-07-03T10:36:07.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.150+0000 7fae1f7fe700 1 -- 172.21.15.101:0/1941648231 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7fae28027090 con 0x7fae380fd3d0 2022-07-03T10:36:07.152 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:07.152 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.151+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 msgr2=0x7fae20099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.151+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 0x7fae20099210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009d70 tx=0x7fae2c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.151+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 msgr2=0x7fae3810e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.151+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae3810e3d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fae28006010 tx=0x7fae280091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 shutdown_connections 2022-07-03T10:36:07.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fae20096d60 0x7fae20099210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.157 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae380fefe0 0x7fae38115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae380fd3d0 0x7fae3810e3d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 --2- 172.21.15.101:0/1941648231 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fae380fe650 0x7fae3810e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 >> 172.21.15.101:0/1941648231 conn(0x7fae380f8b20 msgr2=0x7fae38101880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 shutdown_connections 2022-07-03T10:36:07.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.152+0000 7fae37fff700 1 -- 172.21.15.101:0/1941648231 wait complete. 2022-07-03T10:36:07.183 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:07.183 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:07.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 -- 172.21.15.101:0/536667681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 msgr2=0x7efd640fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 --2- 172.21.15.101:0/536667681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd640fd9f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efd54009900 tx=0x7efd540091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 -- 172.21.15.101:0/536667681 shutdown_connections 2022-07-03T10:36:07.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 --2- 172.21.15.101:0/536667681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640ff1f0 0x7efd64105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 --2- 172.21.15.101:0/536667681 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd640fd9f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 --2- 172.21.15.101:0/536667681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640fe860 0x7efd640fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 -- 172.21.15.101:0/536667681 >> 172.21.15.101:0/536667681 conn(0x7efd640f8d50 msgr2=0x7efd640fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:07.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 -- 172.21.15.101:0/536667681 shutdown_connections 2022-07-03T10:36:07.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.422+0000 7efd691ff700 1 -- 172.21.15.101:0/536667681 wait complete. 2022-07-03T10:36:07.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 Processor -- start 2022-07-03T10:36:07.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 -- start start 2022-07-03T10:36:07.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 0x7efd6410eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 0x7efd64115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd64116160 con 0x7efd640fe860 2022-07-03T10:36:07.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd641162a0 con 0x7efd640ff1f0 2022-07-03T10:36:07.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd691ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd64116460 con 0x7efd640fd5e0 2022-07-03T10:36:07.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd62d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd62d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53428/0 (socket says 172.21.15.101:53428) 2022-07-03T10:36:07.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd62d9d700 1 -- 172.21.15.101:0/42649371 learned_addr learned my addr 172.21.15.101:0/42649371 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:07.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.423+0000 7efd6359e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 0x7efd64115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd6259c700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 0x7efd6410eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 -- 172.21.15.101:0/42649371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 msgr2=0x7efd64115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 0x7efd64115be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 -- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 msgr2=0x7efd6410eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 0x7efd6410eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 -- 172.21.15.101:0/42649371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54009440 con 0x7efd640fd5e0 2022-07-03T10:36:07.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd6359e700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 0x7efd64115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:07.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd6259c700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 0x7efd6410eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:07.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd62d9d700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd54005e10 tx=0x7efd540091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:07.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.424+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5401d070 con 0x7efd640fd5e0 2022-07-03T10:36:07.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.425+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efd5400baa0 con 0x7efd640fd5e0 2022-07-03T10:36:07.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.425+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54022d00 con 0x7efd640fd5e0 2022-07-03T10:36:07.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.425+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd641166e0 con 0x7efd640fd5e0 2022-07-03T10:36:07.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.425+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd64116b30 con 0x7efd640fd5e0 2022-07-03T10:36:07.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.426+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd64108df0 con 0x7efd640fd5e0 2022-07-03T10:36:07.435 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:07 smithi101 ceph-mon[85126]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:07.435 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:07 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1941648231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.435 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.435 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:36:07.436 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:07 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:36:07.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.428+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7efd54029030 con 0x7efd640fd5e0 2022-07-03T10:36:07.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.429+0000 7efd4b7fe700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 0x7efd4c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.429+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7efd540c0eb0 con 0x7efd640fd5e0 2022-07-03T10:36:07.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.430+0000 7efd6259c700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 0x7efd4c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.430+0000 7efd6259c700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 0x7efd4c099210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd58009cf0 tx=0x7efd58005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:07.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.433+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7efd5408e110 con 0x7efd640fd5e0 2022-07-03T10:36:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:07 smithi161 ceph-mon[91579]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:07 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1941648231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:36:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:07 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:36:07.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.631+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd64108f80 con 0x7efd640fd5e0 2022-07-03T10:36:07.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.631+0000 7efd4b7fe700 1 -- 172.21.15.101:0/42649371 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd64108f80 con 0x7efd640fd5e0 2022-07-03T10:36:07.633 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:07.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 msgr2=0x7efd4c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 0x7efd4c099210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efd58009cf0 tx=0x7efd58005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 msgr2=0x7efd6410e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd54005e10 tx=0x7efd540091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.638 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 shutdown_connections 2022-07-03T10:36:07.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7efd4c096d60 0x7efd4c099210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd640ff1f0 0x7efd64115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.639 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efd640fd5e0 0x7efd6410e630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 --2- 172.21.15.101:0/42649371 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efd640fe860 0x7efd6410eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 >> 172.21.15.101:0/42649371 conn(0x7efd640f8d50 msgr2=0x7efd64101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:07.640 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 shutdown_connections 2022-07-03T10:36:07.641 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.634+0000 7efd691ff700 1 -- 172.21.15.101:0/42649371 wait complete. 2022-07-03T10:36:07.664 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:07 smithi188 ceph-mon[92425]: pgmap v302: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:07 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1941648231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:36:07.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:36:07.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:07 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:36:07.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 -- 172.21.15.101:0/1700980961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fe670 msgr2=0x7f58a80feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 --2- 172.21.15.101:0/1700980961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fe670 0x7f58a80feac0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f589c009900 tx=0x7f589c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 -- 172.21.15.101:0/1700980961 shutdown_connections 2022-07-03T10:36:07.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 --2- 172.21.15.101:0/1700980961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80ff000 0x7f58a81058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 --2- 172.21.15.101:0/1700980961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80fd3f0 0x7f58a80fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 --2- 172.21.15.101:0/1700980961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fe670 0x7f58a80feac0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 -- 172.21.15.101:0/1700980961 >> 172.21.15.101:0/1700980961 conn(0x7f58a80f8b40 msgr2=0x7f58a80fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:07.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.907+0000 7f58a759e700 1 -- 172.21.15.101:0/1700980961 shutdown_connections 2022-07-03T10:36:07.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.908+0000 7f58a759e700 1 -- 172.21.15.101:0/1700980961 wait complete. 2022-07-03T10:36:07.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 Processor -- start 2022-07-03T10:36:07.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 -- start start 2022-07-03T10:36:07.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80fe670 0x7f58a8110ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80ff000 0x7f58a8117b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81180d0 con 0x7f58a80fe670 2022-07-03T10:36:07.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a8118210 con 0x7f58a80ff000 2022-07-03T10:36:07.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a759e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81183d0 con 0x7f58a80fd3f0 2022-07-03T10:36:07.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.909+0000 7f58a659c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53444/0 (socket says 172.21.15.101:53444) 2022-07-03T10:36:07.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 -- 172.21.15.101:0/1181954214 learned_addr learned my addr 172.21.15.101:0/1181954214 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:07.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a5d9b700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80fe670 0x7f58a8110ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a6d9d700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80ff000 0x7f58a8117b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 -- 172.21.15.101:0/1181954214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80ff000 msgr2=0x7f58a8117b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80ff000 0x7f58a8117b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 -- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80fe670 msgr2=0x7f58a8110ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:07.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80fe670 0x7f58a8110ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:07.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 -- 172.21.15.101:0/1181954214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c009440 con 0x7f58a80fd3f0 2022-07-03T10:36:07.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.910+0000 7f58a659c700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f589800e840 tx=0x7f5898005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:07.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.911+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898004030 con 0x7f58a80fd3f0 2022-07-03T10:36:07.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.911+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5898004190 con 0x7f58a80fd3f0 2022-07-03T10:36:07.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.911+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898003a20 con 0x7f58a80fd3f0 2022-07-03T10:36:07.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.911+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a81186b0 con 0x7f58a80fd3f0 2022-07-03T10:36:07.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.911+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a8118bf0 con 0x7f58a80fd3f0 2022-07-03T10:36:07.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.912+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a8108c00 con 0x7f58a80fd3f0 2022-07-03T10:36:07.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.914+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f589800c040 con 0x7f58a80fd3f0 2022-07-03T10:36:07.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.915+0000 7f5896ffd700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 0x7f5890099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:07.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.915+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f5898014070 con 0x7f58a80fd3f0 2022-07-03T10:36:07.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.915+0000 7f58a5d9b700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 0x7f5890099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:07.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.916+0000 7f58a5d9b700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 0x7f5890099290 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f589c005d00 tx=0x7f589c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:07.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:07.918+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f589808d230 con 0x7f58a80fd3f0 2022-07-03T10:36:08.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.126+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f58a8119420 con 0x7f58a80fd3f0 2022-07-03T10:36:08.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.126+0000 7f5896ffd700 1 -- 172.21.15.101:0/1181954214 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5898090850 con 0x7f58a80fd3f0 2022-07-03T10:36:08.130 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:08.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 msgr2=0x7f5890099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 0x7f5890099290 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f589c005d00 tx=0x7f589c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 msgr2=0x7f58a81105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f589800e840 tx=0x7f5898005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 shutdown_connections 2022-07-03T10:36:08.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5890096de0 0x7f5890099290 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a80ff000 0x7f58a8117b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f58a80fd3f0 0x7f58a81105a0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f589800e840 tx=0x7f5898005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.134 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 --2- 172.21.15.101:0/1181954214 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f58a80fe670 0x7f58a8110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.134 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.129+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 >> 172.21.15.101:0/1181954214 conn(0x7f58a80f8b40 msgr2=0x7f58a80fae60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:08.134 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.130+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 shutdown_connections 2022-07-03T10:36:08.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.130+0000 7f58a759e700 1 -- 172.21.15.101:0/1181954214 wait complete. 2022-07-03T10:36:08.160 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:08.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.401+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2822950929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fd3f0 msgr2=0x7f7e940fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.401+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2822950929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fd3f0 0x7f7e940fd800 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009900 tx=0x7f7e900091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2822950929 shutdown_connections 2022-07-03T10:36:08.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2822950929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e941058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2822950929 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fd3f0 0x7f7e940fd800 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2822950929 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fe670 0x7f7e940feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2822950929 >> 172.21.15.101:0/2822950929 conn(0x7f7e940f8b40 msgr2=0x7f7e940fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:08.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2822950929 shutdown_connections 2022-07-03T10:36:08.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.402+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2822950929 wait complete. 2022-07-03T10:36:08.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 Processor -- start 2022-07-03T10:36:08.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 -- start start 2022-07-03T10:36:08.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fd3f0 0x7f7e94072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fe670 0x7f7e94070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94071480 con 0x7f7e940fd3f0 2022-07-03T10:36:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940715f0 con 0x7f7e940fe670 2022-07-03T10:36:08.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.403+0000 7f7e9a19d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94118510 con 0x7f7e940ff000 2022-07-03T10:36:08.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51286/0 (socket says 172.21.15.101:51286) 2022-07-03T10:36:08.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9899a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fe670 0x7f7e94070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 -- 172.21.15.101:0/2535210760 learned_addr learned my addr 172.21.15.101:0/2535210760 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:08.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9919b700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fd3f0 0x7f7e94072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 -- 172.21.15.101:0/2535210760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fe670 msgr2=0x7f7e94070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fe670 0x7f7e94070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 -- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fd3f0 msgr2=0x7f7e94072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fd3f0 0x7f7e94072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 -- 172.21.15.101:0/2535210760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e90009440 con 0x7f7e940ff000 2022-07-03T10:36:08.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.404+0000 7f7e9999c700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00eb80 tx=0x7f7e8c009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:08.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.405+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c004a50 con 0x7f7e940ff000 2022-07-03T10:36:08.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.405+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7e8c00ce80 con 0x7f7e940ff000 2022-07-03T10:36:08.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.405+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e94118770 con 0x7f7e940ff000 2022-07-03T10:36:08.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.405+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c005730 con 0x7f7e940ff000 2022-07-03T10:36:08.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.405+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94118cd0 con 0x7f7e940ff000 2022-07-03T10:36:08.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.406+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e94108bd0 con 0x7f7e940ff000 2022-07-03T10:36:08.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.408+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f7e8c00c040 con 0x7f7e940ff000 2022-07-03T10:36:08.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.409+0000 7f7e89ffb700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 0x7f7e80099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.409+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7e8c014070 con 0x7f7e940ff000 2022-07-03T10:36:08.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.409+0000 7f7e9919b700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 0x7f7e80099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.410+0000 7f7e9919b700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 0x7f7e80099210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e90006010 tx=0x7f7e900091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:08.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.415+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f7e8c08d470 con 0x7f7e940ff000 2022-07-03T10:36:08.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.627+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e94104c10 con 0x7f7e940ff000 2022-07-03T10:36:08.629 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.627+0000 7f7e89ffb700 1 -- 172.21.15.101:0/2535210760 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e94104c10 con 0x7f7e940ff000 2022-07-03T10:36:08.629 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:08.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 msgr2=0x7f7e80099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 0x7f7e80099210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e90006010 tx=0x7f7e900091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 msgr2=0x7f7e94070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00eb80 tx=0x7f7e8c009460 comp rx=0 tx=0).stop 2022-07-03T10:36:08.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 shutdown_connections 2022-07-03T10:36:08.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7e80096d60 0x7f7e80099210 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e940fd3f0 0x7f7e94072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e940ff000 0x7f7e94070db0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 --2- 172.21.15.101:0/2535210760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e940fe670 0x7f7e94070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 >> 172.21.15.101:0/2535210760 conn(0x7f7e940f8b40 msgr2=0x7f7e94101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:08.636 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.630+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 shutdown_connections 2022-07-03T10:36:08.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.631+0000 7f7e9a19d700 1 -- 172.21.15.101:0/2535210760 wait complete. 2022-07-03T10:36:08.669 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:36:08.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:08 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/42649371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:08 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1181954214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:08 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/42649371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:08 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1181954214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:08 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/42649371' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:08 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1181954214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:08.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 -- 172.21.15.101:0/2806783196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00ff1e0 msgr2=0x7fd4c0105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/2806783196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0105aa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009bf0 tx=0x7fd4b8009460 comp rx=0 tx=0).stop 2022-07-03T10:36:08.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 -- 172.21.15.101:0/2806783196 shutdown_connections 2022-07-03T10:36:08.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/2806783196 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 0x7fd4c00fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/2806783196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0105aa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/2806783196 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00fe850 0x7fd4c00feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 -- 172.21.15.101:0/2806783196 >> 172.21.15.101:0/2806783196 conn(0x7fd4c00f8d60 msgr2=0x7fd4c00fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:08.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 -- 172.21.15.101:0/2806783196 shutdown_connections 2022-07-03T10:36:08.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.931+0000 7fd4c73ff700 1 -- 172.21.15.101:0/2806783196 wait complete. 2022-07-03T10:36:08.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.932+0000 7fd4c73ff700 1 Processor -- start 2022-07-03T10:36:08.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.932+0000 7fd4c73ff700 1 -- start start 2022-07-03T10:36:08.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 0x7fd4c0110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0117b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01180c0 con 0x7fd4c00fd5d0 2022-07-03T10:36:08.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0118200 con 0x7fd4c00ff1e0 2022-07-03T10:36:08.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c73ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01183c0 con 0x7fd4c00fe850 2022-07-03T10:36:08.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51316/0 (socket says 172.21.15.101:51316) 2022-07-03T10:36:08.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 -- 172.21.15.101:0/4033767002 learned_addr learned my addr 172.21.15.101:0/4033767002 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:08.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c599c700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0117b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c519b700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 0x7fd4c0110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 -- 172.21.15.101:0/4033767002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00ff1e0 msgr2=0x7fd4c0117b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0117b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 -- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 msgr2=0x7fd4c0110590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:08.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 0x7fd4c0110590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:08.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.933+0000 7fd4c499a700 1 -- 172.21.15.101:0/4033767002 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b8009700 con 0x7fd4c00fe850 2022-07-03T10:36:08.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4c499a700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000ea50 tx=0x7fd4b00091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:08.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0005020 con 0x7fd4c00fe850 2022-07-03T10:36:08.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd4b000c740 con 0x7fd4c00fe850 2022-07-03T10:36:08.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c01186a0 con 0x7fd4c00fe850 2022-07-03T10:36:08.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0004940 con 0x7fd4c00fe850 2022-07-03T10:36:08.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.934+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c0118c30 con 0x7fd4c00fe850 2022-07-03T10:36:08.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.935+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c0108db0 con 0x7fd4c00fe850 2022-07-03T10:36:08.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.937+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd4b000c040 con 0x7fd4c00fe850 2022-07-03T10:36:08.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.938+0000 7fd4b5ffb700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 0x7fd4ac099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:08.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.938+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd4b0014070 con 0x7fd4c00fe850 2022-07-03T10:36:08.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.939+0000 7fd4c519b700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 0x7fd4ac099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:08.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.939+0000 7fd4c519b700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 0x7fd4ac099210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00fe6b0 tx=0x7fd4bc005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:08.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:08.942+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd4b008d080 con 0x7fd4c00fe850 2022-07-03T10:36:09.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.190+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c0119490 con 0x7fd4c00fe850 2022-07-03T10:36:09.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.191+0000 7fd4b5ffb700 1 -- 172.21.15.101:0/4033767002 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7fd4b00906a0 con 0x7fd4c00fe850 2022-07-03T10:36:09.192 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:09.193 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:36:09.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 msgr2=0x7fd4ac099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 0x7fd4ac099210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00fe6b0 tx=0x7fd4bc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 msgr2=0x7fd4c0110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000ea50 tx=0x7fd4b00091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 shutdown_connections 2022-07-03T10:36:09.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd4ac096d60 0x7fd4ac099210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd4c00fd5d0 0x7fd4c0110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4c00ff1e0 0x7fd4c0117b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 --2- 172.21.15.101:0/4033767002 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd4c00fe850 0x7fd4c0110ad0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.194+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 >> 172.21.15.101:0/4033767002 conn(0x7fd4c00f8d60 msgr2=0x7fd4c01040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.195+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 shutdown_connections 2022-07-03T10:36:09.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.195+0000 7fd4c73ff700 1 -- 172.21.15.101:0/4033767002 wait complete. 2022-07-03T10:36:09.199 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:09.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-07-03T10:35:03.427264+0000', 'modified': '2022-07-03T10:35:08.208608+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24569}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24557': {'gid': 24557, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6837/639939509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 639939509}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 639939509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.188:6839/204174115', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 204174115}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 204174115}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-03T10:36:09.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-03T10:36:09.225 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-07-03T10:36:09.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 -- 172.21.15.101:0/2106454008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c100a70 msgr2=0x7f635c100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 --2- 172.21.15.101:0/2106454008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c100a70 0x7f635c100ec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6350009900 tx=0x7f63500091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 -- 172.21.15.101:0/2106454008 shutdown_connections 2022-07-03T10:36:09.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 --2- 172.21.15.101:0/2106454008 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c101400 0x7f635c107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 --2- 172.21.15.101:0/2106454008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c0ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 --2- 172.21.15.101:0/2106454008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c100a70 0x7f635c100ec0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 -- 172.21.15.101:0/2106454008 >> 172.21.15.101:0/2106454008 conn(0x7f635c0faf80 msgr2=0x7f635c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 -- 172.21.15.101:0/2106454008 shutdown_connections 2022-07-03T10:36:09.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.473+0000 7f63625ff700 1 -- 172.21.15.101:0/2106454008 wait complete. 2022-07-03T10:36:09.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.474+0000 7f63625ff700 1 Processor -- start 2022-07-03T10:36:09.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.474+0000 7f63625ff700 1 -- start start 2022-07-03T10:36:09.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.480 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c100a70 0x7f635c110cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 0x7f635c117d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1182e0 con 0x7f635c100a70 2022-07-03T10:36:09.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c118420 con 0x7f635c0ff7f0 2022-07-03T10:36:09.481 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f63625ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c1185e0 con 0x7f635c101400 2022-07-03T10:36:09.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:57630/0 (socket says 172.21.15.101:57630) 2022-07-03T10:36:09.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 -- 172.21.15.101:0/2684953770 learned_addr learned my addr 172.21.15.101:0/2684953770 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:09.482 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f6360b9c700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 0x7f635c117d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635b7fe700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c100a70 0x7f635c110cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 -- 172.21.15.101:0/2684953770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 msgr2=0x7f635c117d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 0x7f635c117d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 -- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c100a70 msgr2=0x7f635c110cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.475+0000 7f635bfff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c100a70 0x7f635c110cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f635bfff700 1 -- 172.21.15.101:0/2684953770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350009440 con 0x7f635c0ff7f0 2022-07-03T10:36:09.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f6360b9c700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 0x7f635c117d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:09.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f635bfff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ea50 tx=0x7f634c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f6358ff9700 1 -- 172.21.15.101:0/2684953770 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c004020 con 0x7f635c0ff7f0 2022-07-03T10:36:09.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f6358ff9700 1 -- 172.21.15.101:0/2684953770 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f634c00cc70 con 0x7f635c0ff7f0 2022-07-03T10:36:09.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c1188c0 con 0x7f635c0ff7f0 2022-07-03T10:36:09.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f6358ff9700 1 -- 172.21.15.101:0/2684953770 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0039d0 con 0x7f635c0ff7f0 2022-07-03T10:36:09.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.476+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c118ea0 con 0x7f635c0ff7f0 2022-07-03T10:36:09.498 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.481+0000 7f6358ff9700 1 -- 172.21.15.101:0/2684953770 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f634c00c040 con 0x7f635c0ff7f0 2022-07-03T10:36:09.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.482+0000 7f6376605b80 1 Processor -- start 2022-07-03T10:36:09.499 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.482+0000 7f6358ff9700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 0x7f6344099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.482+0000 7f6376605b80 1 -- start start 2022-07-03T10:36:09.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.482+0000 7f6358ff9700 1 -- 172.21.15.101:0/2684953770 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f634c014070 con 0x7f635c0ff7f0 2022-07-03T10:36:09.500 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.482+0000 7f6376605b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6360b9c700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 0x7f6344099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6376605b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fdf110 0x55a059fdfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6376605b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fe0050 0x55a059fe04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6376605b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059ecb1c0 con 0x55a059fdf110 2022-07-03T10:36:09.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6376605b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059f34cb0 con 0x55a059fe0050 2022-07-03T10:36:09.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f6376605b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059f2e4b0 con 0x55a059fde7c0 2022-07-03T10:36:09.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51344/0 (socket says 172.21.15.101:51344) 2022-07-03T10:36:09.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 -- 172.21.15.101:0/714562961 learned_addr learned my addr 172.21.15.101:0/714562961 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:09.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 -- 172.21.15.101:0/714562961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fe0050 msgr2=0x55a059fe04c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-03T10:36:09.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fe0050 0x55a059fe04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.483+0000 7f635bfff700 1 -- 172.21.15.101:0/714562961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fdf110 msgr2=0x55a059fdfa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-03T10:36:09.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f635bfff700 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fdf110 0x55a059fdfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f635bfff700 1 -- 172.21.15.101:0/714562961 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a059db7360 con 0x55a059fde7c0 2022-07-03T10:36:09.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f635bfff700 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f634c009680 tx=0x7f634c004370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa52039933a1049 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f6360b9c700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 0x7f6344099290 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6354009fd0 tx=0x7f6354009460 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f63419fe700 1 -- 172.21.15.101:0/714562961 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c004180 con 0x55a059fde7c0 2022-07-03T10:36:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f63419fe700 1 -- 172.21.15.101:0/714562961 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f634c091010 con 0x55a059fde7c0 2022-07-03T10:36:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.484+0000 7f63419fe700 1 -- 172.21.15.101:0/714562961 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c08d020 con 0x55a059fde7c0 2022-07-03T10:36:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 -- 172.21.15.101:0/714562961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 msgr2=0x55a059fdebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f634c009680 tx=0x7f634c004370 comp rx=0 tx=0).stop 2022-07-03T10:36:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 -- 172.21.15.101:0/714562961 shutdown_connections 2022-07-03T10:36:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fe0050 0x55a059fe04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fdebd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 --2- 172.21.15.101:0/714562961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fdf110 0x55a059fdfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 -- 172.21.15.101:0/714562961 >> 172.21.15.101:0/714562961 conn(0x55a059fd47b0 msgr2=0x55a059fdb140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 -- 172.21.15.101:0/714562961 shutdown_connections 2022-07-03T10:36:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.485+0000 7f6376605b80 1 -- 172.21.15.101:0/714562961 wait complete. 2022-07-03T10:36:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 -- start start 2022-07-03T10:36:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 Processor -- start 2022-07-03T10:36:09.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059fe88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fef900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059e1ce40 con 0x55a059fe0050 2022-07-03T10:36:09.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059fefeb0 con 0x55a059fdf110 2022-07-03T10:36:09.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.486+0000 7f6376605b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059ff0070 con 0x55a059fde7c0 2022-07-03T10:36:09.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f6376605b80 10 client.0 _ll_register_callbacks cb 0x55a059eda1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:09.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51360/0 (socket says 172.21.15.101:51360) 2022-07-03T10:36:09.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 -- 172.21.15.101:0/3954397941 learned_addr learned my addr 172.21.15.101:0/3954397941 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f6360b9c700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635b7fe700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059fe88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 -- 172.21.15.101:0/3954397941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 msgr2=0x55a059fe88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059fe88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 -- 172.21.15.101:0/3954397941 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 msgr2=0x55a059fef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 -- 172.21.15.101:0/3954397941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c118ea0 con 0x55a059fde7c0 2022-07-03T10:36:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635b7fe700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059fe88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:09.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f635bfff700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f634c0908a0 tx=0x7f634c004370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4230f7da30fd7d82 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.487+0000 7f6336ffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:36:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c097000 con 0x55a059fde7c0 2022-07-03T10:36:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f634c08fe00 con 0x55a059fde7c0 2022-07-03T10:36:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a059fd1f70 con 0x55a059fde7c0 2022-07-03T10:36:09.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f634c096b70 con 0x55a059fde7c0 2022-07-03T10:36:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 10 client.24920 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a059d63b10 con 0x55a059fde7c0 2022-07-03T10:36:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55a059fd1f70 con 0x55a059fde7c0 2022-07-03T10:36:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f634c08c700 con 0x55a059fde7c0 2022-07-03T10:36:09.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 10 client.24920 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 10 client.24920 resolve_mds: resolved name 'l' to daemon mds.l 2022-07-03T10:36:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 20 client.24920 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a059ff34c0 0x55a059ff8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 4 client.24920 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.488+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a059f27a20 con 0x55a059ff34c0 2022-07-03T10:36:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.489+0000 7f635b7fe700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a059ff34c0 0x55a059ff8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.489+0000 7f635b7fe700 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a059ff34c0 0x55a059ff8760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8ad306161f6b63 server_cookie=90c4551cd050d16c in_seq=0 out_seq=0 2022-07-03T10:36:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.489+0000 7f6336ffd700 10 client.24920 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:09.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6336ffd700 1 -- 172.21.15.101:0/3954397941 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55a059f27a20 con 0x55a059ff34c0 2022-07-03T10:36:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6336ffd700 10 client.24920 handle_command_reply: tid=0 2022-07-03T10:36:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 client.24920 shutdown 2022-07-03T10:36:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 msgr2=0x55a059fe8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f634c0908a0 tx=0x7f634c004370 comp rx=0 tx=0).stop 2022-07-03T10:36:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 shutdown_connections 2022-07-03T10:36:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a059ff34c0 0x55a059ff8760 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fe8380 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 --2- 172.21.15.101:0/3954397941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059fe88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6336ffd700 0 client.24920 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.490+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 >> 172.21.15.101:0/3954397941 conn(0x55a059fd47b0 msgr2=0x55a059fdad80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.518 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.491+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 shutdown_connections 2022-07-03T10:36:09.519 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.491+0000 7f6376605b80 1 -- 172.21.15.101:0/3954397941 wait complete. 2022-07-03T10:36:09.519 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.491+0000 7f6376605b80 20 client.24920 trim_cache size 0 max 16384 2022-07-03T10:36:09.519 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:09 smithi101 ceph-mon[85126]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:09.520 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:09 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2535210760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:09.520 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:09 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/4033767002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:09.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.508+0000 7f6376605b80 1 Processor -- start 2022-07-03T10:36:09.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.508+0000 7f6376605b80 1 -- start start 2022-07-03T10:36:09.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 0x55a059fe31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fe6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059f27a20 con 0x55a059fe0050 2022-07-03T10:36:09.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059fde3d0 con 0x55a059fde7c0 2022-07-03T10:36:09.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6376605b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059ef2460 con 0x55a059fdf110 2022-07-03T10:36:09.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f635bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f635bfff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:57650/0 (socket says 172.21.15.101:57650) 2022-07-03T10:36:09.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f635bfff700 1 -- 172.21.15.101:0/280599533 learned_addr learned my addr 172.21.15.101:0/280599533 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:09.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f635b7fe700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 0x55a059fe31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.509+0000 7f6360b9c700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fe6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 -- 172.21.15.101:0/280599533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 msgr2=0x55a059fe31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 0x55a059fe31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 -- 172.21.15.101:0/280599533 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 msgr2=0x55a059fe6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fe6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 -- 172.21.15.101:0/280599533 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a059fe78d0 con 0x55a059fde7c0 2022-07-03T10:36:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635b7fe700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 0x55a059fe31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f635bfff700 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f634c08d330 tx=0x7f634c004370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96f93d81ae87b3fc server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f63421ff700 1 -- 172.21.15.101:0/280599533 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c0953d0 con 0x55a059fde7c0 2022-07-03T10:36:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f63421ff700 1 -- 172.21.15.101:0/280599533 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f634c08fe00 con 0x55a059fde7c0 2022-07-03T10:36:09.532 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.510+0000 7f63421ff700 1 -- 172.21.15.101:0/280599533 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c08c700 con 0x55a059fde7c0 2022-07-03T10:36:09.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- 172.21.15.101:0/280599533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 msgr2=0x55a059fe2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f634c08d330 tx=0x7f634c004370 comp rx=0 tx=0).stop 2022-07-03T10:36:09.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- 172.21.15.101:0/280599533 shutdown_connections 2022-07-03T10:36:09.533 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fe6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fde7c0 0x55a059fe2c60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 --2- 172.21.15.101:0/280599533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fdf110 0x55a059fe31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- 172.21.15.101:0/280599533 >> 172.21.15.101:0/280599533 conn(0x55a059fd47b0 msgr2=0x55a059fdad60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.534 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- 172.21.15.101:0/280599533 shutdown_connections 2022-07-03T10:36:09.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- 172.21.15.101:0/280599533 wait complete. 2022-07-03T10:36:09.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.511+0000 7f6376605b80 1 -- start start 2022-07-03T10:36:09.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 Processor -- start 2022-07-03T10:36:09.535 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fed730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.536 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fec480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a059fecb90 con 0x55a059fe0050 2022-07-03T10:36:09.537 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05a045700 con 0x55a059fdf110 2022-07-03T10:36:09.538 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05a045840 con 0x55a059fde7c0 2022-07-03T10:36:09.538 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f6376605b80 10 client.0 _ll_register_callbacks cb 0x55a059eda1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:09.538 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.538 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:57660/0 (socket says 172.21.15.101:57660) 2022-07-03T10:36:09.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f635b7fe700 1 -- 172.21.15.101:0/1588693292 learned_addr learned my addr 172.21.15.101:0/1588693292 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:09.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.512+0000 7f635bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f6360b9c700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.539 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 -- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 msgr2=0x55a059fed730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fed730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 -- 172.21.15.101:0/1588693292 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 msgr2=0x55a059fec480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:09.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fec480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 -- 172.21.15.101:0/1588693292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a059d63b10 con 0x55a059fdf110 2022-07-03T10:36:09.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f635b7fe700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f635000bea0 tx=0x7f63500055a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2db565e509945561 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:09.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f6336ffd700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:36:09.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.513+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635001d040 con 0x55a059fdf110 2022-07-03T10:36:09.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f634c091010 con 0x55a059fdf110 2022-07-03T10:36:09.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f6350003bf0 con 0x55a059fdf110 2022-07-03T10:36:09.542 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6350003fc0 con 0x55a059fdf110 2022-07-03T10:36:09.542 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 10 client.24934 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:09.542 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a059d61fe0 con 0x55a059fdf110 2022-07-03T10:36:09.542 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f634c091010 con 0x55a059fdf110 2022-07-03T10:36:09.543 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f6350009930 con 0x55a059fdf110 2022-07-03T10:36:09.543 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 10 client.24934 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:09.543 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 10 client.24934 resolve_mds: resolved name 'l' to daemon mds.l 2022-07-03T10:36:09.544 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 20 client.24934 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:09.544 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a05a04a560 0x55a05a046ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:09.544 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 4 client.24934 mds_command: new command op to 24569 tid=0[{"prefix": "status"}] 2022-07-03T10:36:09.544 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.514+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a059f27800 con 0x55a05a04a560 2022-07-03T10:36:09.544 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.515+0000 7f635bfff700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a05a04a560 0x55a05a046ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:09.545 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.515+0000 7f635bfff700 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a05a04a560 0x55a05a046ef0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d4b5a780715fb93 server_cookie=173c7a93aa358445 in_seq=0 out_seq=0 2022-07-03T10:36:09.545 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.515+0000 7f6336ffd700 10 client.24934 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:09.545 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.515+0000 7f6336ffd700 1 -- 172.21.15.101:0/1588693292 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a059f27800 con 0x55a05a04a560 2022-07-03T10:36:09.546 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.515+0000 7f6336ffd700 10 client.24934 handle_command_reply: tid=0 2022-07-03T10:36:09.546 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 client.24934 shutdown 2022-07-03T10:36:09.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 msgr2=0x55a059febde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f635000bea0 tx=0x7f63500055a0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 shutdown_connections 2022-07-03T10:36:09.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a05a04a560 0x55a05a046ef0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a059fe0050 0x55a059fec480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.549 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a059fde7c0 0x55a059fed730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.549 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 --2- 172.21.15.101:0/1588693292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a059fdf110 0x55a059febde0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.550 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6336ffd700 0 client.24934 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:09.550 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 >> 172.21.15.101:0/1588693292 conn(0x55a059fd47b0 msgr2=0x55a059fe2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 shutdown_connections 2022-07-03T10:36:09.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 1 -- 172.21.15.101:0/1588693292 wait complete. 2022-07-03T10:36:09.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.516+0000 7f6376605b80 20 client.24934 trim_cache size 0 max 16384 2022-07-03T10:36:09.552 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: "cluster_fsid": "23a851ae-fab9-11ec-842c-001a4aab830c", 2022-07-03T10:36:09.553 INFO:teuthology.orchestra.run.smithi101.stdout: "whoami": 0, 2022-07-03T10:36:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "id": 24569, 2022-07-03T10:36:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "want_state": "up:active", 2022-07-03T10:36:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "up:active", 2022-07-03T10:36:09.554 INFO:teuthology.orchestra.run.smithi101.stdout: "fs_name": "cephfs", 2022-07-03T10:36:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "rank_uptime": 66.085410287000002, 2022-07-03T10:36:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "mdsmap_epoch": 20, 2022-07-03T10:36:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "osdmap_epoch": 82, 2022-07-03T10:36:09.555 INFO:teuthology.orchestra.run.smithi101.stdout: "osdmap_epoch_barrier": 80, 2022-07-03T10:36:09.556 INFO:teuthology.orchestra.run.smithi101.stdout: "uptime": 71.744559136999996 2022-07-03T10:36:09.556 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:09.557 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 msgr2=0x7f6344099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 0x7f6344099290 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6354009fd0 tx=0x7f6354009460 comp rx=0 tx=0).stop 2022-07-03T10:36:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 msgr2=0x7f635c1107b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f634c00ea50 tx=0x7f634c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.558 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 shutdown_connections 2022-07-03T10:36:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6344096de0 0x7f6344099290 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f635c101400 0x7f635c117d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0ff7f0 0x7f635c1107b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 --2- 172.21.15.101:0/2684953770 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f635c100a70 0x7f635c110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:09.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 >> 172.21.15.101:0/2684953770 conn(0x7f635c0faf80 msgr2=0x7f635c103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:09.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 shutdown_connections 2022-07-03T10:36:09.560 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:09.517+0000 7f63625ff700 1 -- 172.21.15.101:0/2684953770 wait complete. 2022-07-03T10:36:09.560 INFO:tasks.cephadm:Setup complete, yielding 2022-07-03T10:36:09.561 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-07-03T10:36:09.572 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi101.front.sepia.ceph.com 2022-07-03T10:36:09.572 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph orch status' 2022-07-03T10:36:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:09 smithi161 ceph-mon[91579]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:09 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2535210760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:09 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/4033767002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:09 smithi188 ceph-mon[92425]: pgmap v303: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:09 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2535210760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:09 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/4033767002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:10.508 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:11.125 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:10 smithi101 ceph-mon[85126]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:11.125 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.125 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.126 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.126 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:10 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:10 smithi161 ceph-mon[91579]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:10 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:10 smithi188 ceph-mon[92425]: pgmap v304: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:10 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:12.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 -- 172.21.15.101:0/3402097317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fe740 msgr2=0x7fd7840feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:12.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 --2- 172.21.15.101:0/3402097317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fe740 0x7fd7840feb90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd774009900 tx=0x7fd7740091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 -- 172.21.15.101:0/3402097317 shutdown_connections 2022-07-03T10:36:12.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 --2- 172.21.15.101:0/3402097317 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fd4c0 0x7fd7840fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 --2- 172.21.15.101:0/3402097317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 --2- 172.21.15.101:0/3402097317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fe740 0x7fd7840feb90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 -- 172.21.15.101:0/3402097317 >> 172.21.15.101:0/3402097317 conn(0x7fd7840f8c30 msgr2=0x7fd7840fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:12.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 -- 172.21.15.101:0/3402097317 shutdown_connections 2022-07-03T10:36:12.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.105+0000 7fd78a089700 1 -- 172.21.15.101:0/3402097317 wait complete. 2022-07-03T10:36:12.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 Processor -- start 2022-07-03T10:36:12.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 -- start start 2022-07-03T10:36:12.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:12.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 0x7fd784110be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:12.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784117c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:12.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7841181d0 con 0x7fd7840fe740 2022-07-03T10:36:12.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784118310 con 0x7fd7840ff0d0 2022-07-03T10:36:12.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.107+0000 7fd78a089700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7841184d0 con 0x7fd7840fd4c0 2022-07-03T10:36:12.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd788886700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 0x7fd784110be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:12.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:12.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd788886700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 0x7fd784110be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:37062/0 (socket says 172.21.15.101:37062) 2022-07-03T10:36:12.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51380/0 (socket says 172.21.15.101:51380) 2022-07-03T10:36:12.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd788886700 1 -- 172.21.15.101:0/4122972012 learned_addr learned my addr 172.21.15.101:0/4122972012 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:12.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789888700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784117c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:12.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 -- 172.21.15.101:0/4122972012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 msgr2=0x7fd784117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:12.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784117c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 -- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 msgr2=0x7fd784110be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:12.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 0x7fd784110be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 -- 172.21.15.101:0/4122972012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd774009440 con 0x7fd7840fd4c0 2022-07-03T10:36:12.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789888700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784117c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:12.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.108+0000 7fd789087700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd78000e7a0 tx=0x7fd780005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:12.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.109+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780009460 con 0x7fd7840fd4c0 2022-07-03T10:36:12.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.109+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7841187b0 con 0x7fd7840fd4c0 2022-07-03T10:36:12.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.109+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd780004230 con 0x7fd7840fd4c0 2022-07-03T10:36:12.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.109+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780003890 con 0x7fd7840fd4c0 2022-07-03T10:36:12.126 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.109+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd784118d90 con 0x7fd7840fd4c0 2022-07-03T10:36:12.126 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.110+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd784108cb0 con 0x7fd7840fd4c0 2022-07-03T10:36:12.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.112+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd78000c040 con 0x7fd7840fd4c0 2022-07-03T10:36:12.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.113+0000 7fd779ffb700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 0x7fd770099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:12.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.113+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd780014070 con 0x7fd7840fd4c0 2022-07-03T10:36:12.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.113+0000 7fd788886700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 0x7fd770099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:12.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.114+0000 7fd788886700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 0x7fd770099210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd774000e50 tx=0x7fd7740091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:12.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.117+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd78008cd20 con 0x7fd7840fd4c0 2022-07-03T10:36:12.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.339+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7841051d0 con 0x7fd770096d60 2022-07-03T10:36:12.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.340+0000 7fd779ffb700 1 -- 172.21.15.101:0/4122972012 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd7841051d0 con 0x7fd770096d60 2022-07-03T10:36:12.341 INFO:teuthology.orchestra.run.smithi101.stdout:Backend: cephadm 2022-07-03T10:36:12.341 INFO:teuthology.orchestra.run.smithi101.stdout:Available: Yes 2022-07-03T10:36:12.342 INFO:teuthology.orchestra.run.smithi101.stdout:Paused: No 2022-07-03T10:36:12.344 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.342+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 msgr2=0x7fd770099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:12.344 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.342+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 0x7fd770099210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd774000e50 tx=0x7fd7740091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.342+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 msgr2=0x7fd7841106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:12.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.342+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd78000e7a0 tx=0x7fd780005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 shutdown_connections 2022-07-03T10:36:12.346 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd770096d60 0x7fd770099210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.346 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7840ff0d0 0x7fd784117c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.346 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7840fd4c0 0x7fd7841106a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 --2- 172.21.15.101:0/4122972012 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7840fe740 0x7fd784110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:12.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 >> 172.21.15.101:0/4122972012 conn(0x7fd7840f8c30 msgr2=0x7fd7840faf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:12.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 shutdown_connections 2022-07-03T10:36:12.348 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:12.343+0000 7fd78a089700 1 -- 172.21.15.101:0/4122972012 wait complete. 2022-07-03T10:36:12.824 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph orch ps' 2022-07-03T10:36:13.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:12 smithi101 ceph-mon[85126]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:13.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:12 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:12 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:12 smithi101 ceph-mon[85126]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:12 smithi161 ceph-mon[91579]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:12 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:12 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:12 smithi161 ceph-mon[91579]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:12 smithi188 ceph-mon[92425]: pgmap v305: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:12 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:12 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:13.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:12 smithi188 ceph-mon[92425]: from='client.24932 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:13.721 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:14.367 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:14 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:14 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:14 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:36:15.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:15 smithi101 ceph-mon[85126]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:15.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 -- 172.21.15.101:0/796670240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 msgr2=0x7f0960105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:15.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 --2- 172.21.15.101:0/796670240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f0960105b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0958009ca0 tx=0x7f0958009520 comp rx=0 tx=0).stop 2022-07-03T10:36:15.250 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 -- 172.21.15.101:0/796670240 shutdown_connections 2022-07-03T10:36:15.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 --2- 172.21.15.101:0/796670240 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f0960105b30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 --2- 172.21.15.101:0/796670240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fd660 0x7f09600fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 --2- 172.21.15.101:0/796670240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fe8e0 0x7f09600fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.251 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.247+0000 7f09667d4700 1 -- 172.21.15.101:0/796670240 >> 172.21.15.101:0/796670240 conn(0x7f09600f8dd0 msgr2=0x7f09600fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:15.252 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.248+0000 7f09667d4700 1 -- 172.21.15.101:0/796670240 shutdown_connections 2022-07-03T10:36:15.252 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.248+0000 7f09667d4700 1 -- 172.21.15.101:0/796670240 wait complete. 2022-07-03T10:36:15.254 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 Processor -- start 2022-07-03T10:36:15.254 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 -- start start 2022-07-03T10:36:15.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:15.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:15.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f09601158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:15.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0960115e60 con 0x7f09600ff270 2022-07-03T10:36:15.255 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0960115fa0 con 0x7f09600fe8e0 2022-07-03T10:36:15.256 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.249+0000 7f09667d4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0960116160 con 0x7f09600fd660 2022-07-03T10:36:15.256 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:15.256 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:15.256 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:57672/0 (socket says 172.21.15.101:57672) 2022-07-03T10:36:15.257 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:51400/0 (socket says 172.21.15.101:51400) 2022-07-03T10:36:15.257 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 -- 172.21.15.101:0/1337429478 learned_addr learned my addr 172.21.15.101:0/1337429478 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:15.257 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f0964d71700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f09601158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:15.257 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 -- 172.21.15.101:0/1337429478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 msgr2=0x7f096010e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:15.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 -- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 msgr2=0x7f09601158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:15.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f09601158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.265 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095f7fe700 1 -- 172.21.15.101:0/1337429478 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580097e0 con 0x7f09600fe8e0 2022-07-03T10:36:15.265 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f095ffff700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:15.265 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.250+0000 7f0964d71700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f09601158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:15.265 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f095f7fe700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f095400ea80 tx=0x7f0954005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:15.266 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954009e10 con 0x7f09600fe8e0 2022-07-03T10:36:15.266 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f095400ce80 con 0x7f09600fe8e0 2022-07-03T10:36:15.266 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09600707b0 con 0x7f09600fe8e0 2022-07-03T10:36:15.266 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954010e00 con 0x7f09600fe8e0 2022-07-03T10:36:15.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.251+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0960070d40 con 0x7f09600fe8e0 2022-07-03T10:36:15.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.252+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0960108ec0 con 0x7f09600fe8e0 2022-07-03T10:36:15.268 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.254+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f095400c040 con 0x7f09600fe8e0 2022-07-03T10:36:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.255+0000 7f095cff9700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 0x7f0948099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.255+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f0954014070 con 0x7f09600fe8e0 2022-07-03T10:36:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.255+0000 7f095ffff700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 0x7f0948099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:15.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.256+0000 7f095ffff700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 0x7f0948099210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09600fe740 tx=0x7f0950005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:15.270 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.257+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f095408cd90 con 0x7f09600fe8e0 2022-07-03T10:36:15.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.458+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f09600721f0 con 0x7f0948096d60 2022-07-03T10:36:15.475 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.473+0000 7f095cff9700 1 -- 172.21.15.101:0/1337429478 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4061 (secure 0 0 0) 0x7f09600721f0 con 0x7f0948096d60 2022-07-03T10:36:15.476 INFO:teuthology.orchestra.run.smithi101.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-07-03T10:36:15.476 INFO:teuthology.orchestra.run.smithi101.stdout:mds.a smithi101 running (105s) 3s ago 105s 13.5M - 17.0.0-13322-g61829bae ac818e5c233e 96a23032e64b 2022-07-03T10:36:15.477 INFO:teuthology.orchestra.run.smithi101.stdout:mds.b smithi161 running (92s) 5s ago 92s 17.3M - 17.0.0-13322-g61829bae ac818e5c233e e9e9f7d1cb36 2022-07-03T10:36:15.477 INFO:teuthology.orchestra.run.smithi101.stdout:mds.c smithi188 running (84s) 5s ago 84s 14.9M - 17.0.0-13322-g61829bae ac818e5c233e 6bb8865a5c8f 2022-07-03T10:36:15.477 INFO:teuthology.orchestra.run.smithi101.stdout:mds.d smithi101 running (102s) 3s ago 101s 13.6M - 17.0.0-13322-g61829bae ac818e5c233e a4baaa09326f 2022-07-03T10:36:15.477 INFO:teuthology.orchestra.run.smithi101.stdout:mds.e smithi161 running (90s) 5s ago 90s 16.2M - 17.0.0-13322-g61829bae ac818e5c233e 1cf8271c0070 2022-07-03T10:36:15.478 INFO:teuthology.orchestra.run.smithi101.stdout:mds.f smithi188 running (82s) 5s ago 81s 14.5M - 17.0.0-13322-g61829bae ac818e5c233e 5f0b76866bbd 2022-07-03T10:36:15.478 INFO:teuthology.orchestra.run.smithi101.stdout:mds.g smithi101 running (98s) 3s ago 98s 13.8M - 17.0.0-13322-g61829bae ac818e5c233e 22942aba4c00 2022-07-03T10:36:15.478 INFO:teuthology.orchestra.run.smithi101.stdout:mds.h smithi161 running (88s) 5s ago 88s 14.8M - 17.0.0-13322-g61829bae ac818e5c233e 7dc2d86a0285 2022-07-03T10:36:15.478 INFO:teuthology.orchestra.run.smithi101.stdout:mds.i smithi188 running (80s) 5s ago 79s 18.3M - 17.0.0-13322-g61829bae ac818e5c233e 92ed3cdb4f48 2022-07-03T10:36:15.479 INFO:teuthology.orchestra.run.smithi101.stdout:mds.j smithi101 running (95s) 3s ago 94s 13.5M - 17.0.0-13322-g61829bae ac818e5c233e 757dd143ed7b 2022-07-03T10:36:15.479 INFO:teuthology.orchestra.run.smithi101.stdout:mds.k smithi161 running (86s) 5s ago 86s 14.9M - 17.0.0-13322-g61829bae ac818e5c233e f43dafd10881 2022-07-03T10:36:15.479 INFO:teuthology.orchestra.run.smithi101.stdout:mds.l smithi188 running (78s) 5s ago 77s 30.5M - 17.0.0-13322-g61829bae ac818e5c233e 43e917df20a6 2022-07-03T10:36:15.479 INFO:teuthology.orchestra.run.smithi101.stdout:mgr.x smithi101 *:9283 running (8m) 3s ago 8m 499M - 17.0.0-13322-g61829bae ac818e5c233e a4a271435dab 2022-07-03T10:36:15.480 INFO:teuthology.orchestra.run.smithi101.stdout:mgr.y smithi161 *:8443 running (6m) 5s ago 6m 429M - 17.0.0-13322-g61829bae ac818e5c233e 66d37cba56fd 2022-07-03T10:36:15.480 INFO:teuthology.orchestra.run.smithi101.stdout:mgr.z smithi188 *:8443 running (6m) 5s ago 6m 428M - 17.0.0-13322-g61829bae ac818e5c233e 5f6cd708a843 2022-07-03T10:36:15.480 INFO:teuthology.orchestra.run.smithi101.stdout:mon.a smithi101 running (8m) 3s ago 8m 89.6M 2048M 17.0.0-13322-g61829bae ac818e5c233e 89cf619fbd64 2022-07-03T10:36:15.480 INFO:teuthology.orchestra.run.smithi101.stdout:mon.b smithi161 running (7m) 5s ago 7m 67.9M 2048M 17.0.0-13322-g61829bae ac818e5c233e bf56df17d9f5 2022-07-03T10:36:15.481 INFO:teuthology.orchestra.run.smithi101.stdout:mon.c smithi188 running (7m) 5s ago 7m 63.5M 2048M 17.0.0-13322-g61829bae ac818e5c233e 0212006b729d 2022-07-03T10:36:15.481 INFO:teuthology.orchestra.run.smithi101.stdout:osd.0 smithi101 running (6m) 3s ago 6m 64.0M 4317M 17.0.0-13322-g61829bae ac818e5c233e 18f15118f10f 2022-07-03T10:36:15.481 INFO:teuthology.orchestra.run.smithi101.stdout:osd.1 smithi161 running (6m) 5s ago 6m 70.6M 4292M 17.0.0-13322-g61829bae ac818e5c233e 6ed63a4a7f84 2022-07-03T10:36:15.481 INFO:teuthology.orchestra.run.smithi101.stdout:osd.2 smithi188 running (5m) 5s ago 5m 67.8M 4292M 17.0.0-13322-g61829bae ac818e5c233e 5d40c3667712 2022-07-03T10:36:15.481 INFO:teuthology.orchestra.run.smithi101.stdout:osd.3 smithi101 running (5m) 3s ago 5m 59.9M 4317M 17.0.0-13322-g61829bae ac818e5c233e 2ffc2b7a92d0 2022-07-03T10:36:15.482 INFO:teuthology.orchestra.run.smithi101.stdout:osd.4 smithi161 running (4m) 5s ago 4m 59.6M 4292M 17.0.0-13322-g61829bae ac818e5c233e e6e2b077b6fe 2022-07-03T10:36:15.482 INFO:teuthology.orchestra.run.smithi101.stdout:osd.5 smithi188 running (4m) 5s ago 4m 62.4M 4292M 17.0.0-13322-g61829bae ac818e5c233e 9808f8a9742c 2022-07-03T10:36:15.482 INFO:teuthology.orchestra.run.smithi101.stdout:osd.6 smithi101 running (4m) 3s ago 4m 55.2M 4317M 17.0.0-13322-g61829bae ac818e5c233e d8ebcca26f98 2022-07-03T10:36:15.482 INFO:teuthology.orchestra.run.smithi101.stdout:osd.7 smithi161 running (3m) 5s ago 3m 58.2M 4292M 17.0.0-13322-g61829bae ac818e5c233e 8566325fc21f 2022-07-03T10:36:15.483 INFO:teuthology.orchestra.run.smithi101.stdout:osd.8 smithi188 running (3m) 5s ago 3m 58.9M 4292M 17.0.0-13322-g61829bae ac818e5c233e 0f2aae5401b0 2022-07-03T10:36:15.483 INFO:teuthology.orchestra.run.smithi101.stdout:osd.9 smithi101 running (2m) 3s ago 2m 55.8M 4317M 17.0.0-13322-g61829bae ac818e5c233e 4be7eeccf804 2022-07-03T10:36:15.484 INFO:teuthology.orchestra.run.smithi101.stdout:osd.10 smithi161 running (2m) 5s ago 2m 58.4M 4292M 17.0.0-13322-g61829bae ac818e5c233e fc3fe6f47459 2022-07-03T10:36:15.484 INFO:teuthology.orchestra.run.smithi101.stdout:osd.11 smithi188 running (2m) 5s ago 2m 51.1M 4292M 17.0.0-13322-g61829bae ac818e5c233e e8ef4349a7ca 2022-07-03T10:36:15.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 msgr2=0x7f0948099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:15.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 0x7f0948099210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f09600fe740 tx=0x7f0950005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 msgr2=0x7f096010e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:15.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f095400ea80 tx=0x7f0954005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 shutdown_connections 2022-07-03T10:36:15.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0948096d60 0x7f0948099210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ff270 0x7f09601158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09600fd660 0x7f096010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 --2- 172.21.15.101:0/1337429478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09600fe8e0 0x7f096010e870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:15.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 >> 172.21.15.101:0/1337429478 conn(0x7f09600f8dd0 msgr2=0x7f0960101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:15.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.475+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 shutdown_connections 2022-07-03T10:36:15.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:15.476+0000 7f09667d4700 1 -- 172.21.15.101:0/1337429478 wait complete. 2022-07-03T10:36:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:15 smithi161 ceph-mon[91579]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:15 smithi188 ceph-mon[92425]: pgmap v306: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:15.995 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph orch ls' 2022-07-03T10:36:16.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:16 smithi101 ceph-mon[85126]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:16 smithi161 ceph-mon[91579]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:16 smithi188 ceph-mon[92425]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:16.882 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:17.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:17 smithi101 ceph-mon[85126]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:17 smithi161 ceph-mon[91579]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:17 smithi188 ceph-mon[92425]: pgmap v307: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:18.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.429+0000 7fe03354a700 1 -- 172.21.15.101:0/714395873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0fd4c0 msgr2=0x7fe02c0fd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:18.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.429+0000 7fe03354a700 1 --2- 172.21.15.101:0/714395873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c0fd8d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe028009900 tx=0x7fe0280091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 -- 172.21.15.101:0/714395873 shutdown_connections 2022-07-03T10:36:18.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 --2- 172.21.15.101:0/714395873 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 --2- 172.21.15.101:0/714395873 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c0fd8d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 --2- 172.21.15.101:0/714395873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 0x7fe02c0feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 -- 172.21.15.101:0/714395873 >> 172.21.15.101:0/714395873 conn(0x7fe02c0f8c30 msgr2=0x7fe02c0fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:18.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 -- 172.21.15.101:0/714395873 shutdown_connections 2022-07-03T10:36:18.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.430+0000 7fe03354a700 1 -- 172.21.15.101:0/714395873 wait complete. 2022-07-03T10:36:18.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 Processor -- start 2022-07-03T10:36:18.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 -- start start 2022-07-03T10:36:18.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:18.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 0x7fe02c070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:18.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:18.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c071480 con 0x7fe02c0fd4c0 2022-07-03T10:36:18.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0715f0 con 0x7fe02c0fe740 2022-07-03T10:36:18.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe03354a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c118600 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.431+0000 7fe032548700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:18.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032548700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:37410/0 (socket says 172.21.15.101:37410) 2022-07-03T10:36:18.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032548700 1 -- 172.21.15.101:0/3460305726 learned_addr learned my addr 172.21.15.101:0/3460305726 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:18.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe031d47700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 0x7fe02c070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:18.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:18.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 -- 172.21.15.101:0/3460305726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 msgr2=0x7fe02c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:18.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 0x7fe02c070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 -- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 msgr2=0x7fe02c072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:18.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 -- 172.21.15.101:0/3460305726 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009440 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032548700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:18.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.432+0000 7fe032d49700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c070db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe02400eab0 tx=0x7fe024009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:18.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.433+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024009850 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.433+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c118860 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.433+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe02400c9d0 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.433+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe024004020 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.433+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c118df0 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.434+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c108cd0 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.436+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fe02401e020 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.437+0000 7fe022ffd700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 0x7fe018099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:18.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.437+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fe024014070 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.437+0000 7fe032548700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 0x7fe018099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:18.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.438+0000 7fe032548700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 0x7fe018099290 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0280056f0 tx=0x7fe0280091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:18.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.441+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe0240c4070 con 0x7fe02c0ff0d0 2022-07-03T10:36:18.637 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.635+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe02c05e960 con 0x7fe018096de0 2022-07-03T10:36:18.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.641+0000 7fe022ffd700 1 -- 172.21.15.101:0/3460305726 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7fe02c05e960 con 0x7fe018096de0 2022-07-03T10:36:18.643 INFO:teuthology.orchestra.run.smithi101.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-07-03T10:36:18.643 INFO:teuthology.orchestra.run.smithi101.stdout:mds.all 12/12 8s ago 112s smithi101=a;smithi101=d;smithi101=g;smithi101=j;smithi161=b;smithi161=e;smithi161=h;smithi161=k;smithi188=c;smithi188=f;smithi188=i;smithi188=l;count:12 2022-07-03T10:36:18.644 INFO:teuthology.orchestra.run.smithi101.stdout:mgr 3/3 8s ago 6m smithi101=x;smithi161=y;smithi188=z;count:3 2022-07-03T10:36:18.644 INFO:teuthology.orchestra.run.smithi101.stdout:mon 3/3 8s ago 7m smithi101:172.21.15.101=a;smithi161:172.21.15.161=b;smithi188:172.21.15.188=c;count:3 2022-07-03T10:36:18.644 INFO:teuthology.orchestra.run.smithi101.stdout:osd 12 8s ago - 2022-07-03T10:36:18.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 msgr2=0x7fe018099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:18.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 0x7fe018099290 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0280056f0 tx=0x7fe0280091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 msgr2=0x7fe02c070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:18.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c070db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe02400eab0 tx=0x7fe024009460 comp rx=0 tx=0).stop 2022-07-03T10:36:18.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 shutdown_connections 2022-07-03T10:36:18.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe018096de0 0x7fe018099290 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe02c0fd4c0 0x7fe02c072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe02c0ff0d0 0x7fe02c070db0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 --2- 172.21.15.101:0/3460305726 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe02c0fe740 0x7fe02c070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:18.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.643+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 >> 172.21.15.101:0/3460305726 conn(0x7fe02c0f8c30 msgr2=0x7fe02c0fb040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:18.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.644+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 shutdown_connections 2022-07-03T10:36:18.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:18.644+0000 7fe03354a700 1 -- 172.21.15.101:0/3460305726 wait complete. 2022-07-03T10:36:19.161 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph orch host ls' 2022-07-03T10:36:19.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:19 smithi101 ceph-mon[85126]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:19.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:19 smithi101 ceph-mon[85126]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:19 smithi161 ceph-mon[91579]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:19 smithi161 ceph-mon[91579]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:19 smithi188 ceph-mon[92425]: pgmap v308: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:19 smithi188 ceph-mon[92425]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:20.045 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:21.276 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:21 smithi101 ceph-mon[85126]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:21.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1903938064 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0fe740 msgr2=0x7fd5bc0feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:21.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1903938064 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc0feb90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009900 tx=0x7fd5ac0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1903938064 shutdown_connections 2022-07-03T10:36:21.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1903938064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1903938064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc0fd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1903938064 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc0feb90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1903938064 >> 172.21.15.101:0/1903938064 conn(0x7fd5bc0f8c30 msgr2=0x7fd5bc0fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:21.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1903938064 shutdown_connections 2022-07-03T10:36:21.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.581+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1903938064 wait complete. 2022-07-03T10:36:21.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.582+0000 7fd5c2ff1700 1 Processor -- start 2022-07-03T10:36:21.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.582+0000 7fd5c2ff1700 1 -- start start 2022-07-03T10:36:21.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.582+0000 7fd5c2ff1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:21.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.582+0000 7fd5c2ff1700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc110be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:21.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.582+0000 7fd5c2ff1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc117c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:21.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c2ff1700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1181d0 con 0x7fd5bc0ff0d0 2022-07-03T10:36:21.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c2ff1700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc118310 con 0x7fd5bc0fe740 2022-07-03T10:36:21.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c2ff1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc1184d0 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:21.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c27f0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc117c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:21.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59560/0 (socket says 172.21.15.101:59560) 2022-07-03T10:36:21.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 -- 172.21.15.101:0/1620224585 learned_addr learned my addr 172.21.15.101:0/1620224585 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:21.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c17ee700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc110be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:21.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 -- 172.21.15.101:0/1620224585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0fe740 msgr2=0x7fd5bc110be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:21.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc110be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 -- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 msgr2=0x7fd5bc117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:21.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc117c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c1fef700 1 -- 172.21.15.101:0/1620224585 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac009440 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.583+0000 7fd5c27f0700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc117c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:21.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5c1fef700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800ea50 tx=0x7fd5b8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:21.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8004020 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd5b800cc70 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc1187b0 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80039d0 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.584+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc118d40 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.585+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc108cb0 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.587+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7fd5b800c040 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.588+0000 7fd5b27fc700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 0x7fd5a8099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:21.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.588+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd5b8014070 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.589+0000 7fd5c17ee700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 0x7fd5a8099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:21.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.589+0000 7fd5c17ee700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 0x7fd5a8099210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0092d0 tx=0x7fd5ac0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:21.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.592+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fd5b808cce0 con 0x7fd5bc0fd4c0 2022-07-03T10:36:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:21 smithi161 ceph-mon[91579]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:21 smithi188 ceph-mon[92425]: pgmap v309: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:21.788 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.785+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5bc1051d0 con 0x7fd5a8096d60 2022-07-03T10:36:21.789 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.787+0000 7fd5b27fc700 1 -- 172.21.15.101:0/1620224585 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fd5bc1051d0 con 0x7fd5a8096d60 2022-07-03T10:36:21.790 INFO:teuthology.orchestra.run.smithi101.stdout:HOST ADDR LABELS STATUS 2022-07-03T10:36:21.790 INFO:teuthology.orchestra.run.smithi101.stdout:smithi101 172.21.15.101 2022-07-03T10:36:21.790 INFO:teuthology.orchestra.run.smithi101.stdout:smithi161 172.21.15.161 2022-07-03T10:36:21.791 INFO:teuthology.orchestra.run.smithi101.stdout:smithi188 172.21.15.188 2022-07-03T10:36:21.791 INFO:teuthology.orchestra.run.smithi101.stdout:3 hosts in cluster 2022-07-03T10:36:21.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 msgr2=0x7fd5a8099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:21.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 0x7fd5a8099210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac0092d0 tx=0x7fd5ac0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 msgr2=0x7fd5bc1106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:21.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800ea50 tx=0x7fd5b8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 shutdown_connections 2022-07-03T10:36:21.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd5a8096d60 0x7fd5a8099210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd5bc0ff0d0 0x7fd5bc117c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5bc0fd4c0 0x7fd5bc1106a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 --2- 172.21.15.101:0/1620224585 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5bc0fe740 0x7fd5bc110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:21.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 >> 172.21.15.101:0/1620224585 conn(0x7fd5bc0f8c30 msgr2=0x7fd5bc0fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:21.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 shutdown_connections 2022-07-03T10:36:21.795 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:21.790+0000 7fd5c2ff1700 1 -- 172.21.15.101:0/1620224585 wait complete. 2022-07-03T10:36:22.321 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph orch device ls' 2022-07-03T10:36:23.204 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:23.490 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:23 smithi101 ceph-mon[85126]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:23.491 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:23 smithi101 ceph-mon[85126]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:23 smithi161 ceph-mon[91579]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:23 smithi161 ceph-mon[91579]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:23 smithi188 ceph-mon[92425]: from='client.24950 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:23 smithi188 ceph-mon[92425]: pgmap v310: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:24.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 -- 172.21.15.101:0/1609776769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00ff270 msgr2=0x7f25e0105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 --2- 172.21.15.101:0/1609776769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00ff270 0x7f25e0105b30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009ca0 tx=0x7f25d8009520 comp rx=0 tx=0).stop 2022-07-03T10:36:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 -- 172.21.15.101:0/1609776769 shutdown_connections 2022-07-03T10:36:24.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 --2- 172.21.15.101:0/1609776769 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fd660 0x7f25e00fda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 --2- 172.21.15.101:0/1609776769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00ff270 0x7f25e0105b30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 --2- 172.21.15.101:0/1609776769 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00fe8e0 0x7f25e00fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 -- 172.21.15.101:0/1609776769 >> 172.21.15.101:0/1609776769 conn(0x7f25e00f8dd0 msgr2=0x7f25e00fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:24.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 -- 172.21.15.101:0/1609776769 shutdown_connections 2022-07-03T10:36:24.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.730+0000 7f25e6e16700 1 -- 172.21.15.101:0/1609776769 wait complete. 2022-07-03T10:36:24.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.731+0000 7f25e6e16700 1 Processor -- start 2022-07-03T10:36:24.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.731+0000 7f25e6e16700 1 -- start start 2022-07-03T10:36:24.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:24.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 0x7f25e0070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:24.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 0x7f25e0070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:24.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e0071480 con 0x7f25e00fe8e0 2022-07-03T10:36:24.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e00715f0 con 0x7f25e00ff270 2022-07-03T10:36:24.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e6e16700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e0118440 con 0x7f25e00fd660 2022-07-03T10:36:24.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e4bb2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:24.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25dffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 0x7f25e0070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:24.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e4bb2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:59580/0 (socket says 172.21.15.101:59580) 2022-07-03T10:36:24.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e53b3700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 0x7f25e0070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:24.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25e4bb2700 1 -- 172.21.15.101:0/2157970285 learned_addr learned my addr 172.21.15.101:0/2157970285 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:24.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.732+0000 7f25dffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 0x7f25e0070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:37440/0 (socket says 172.21.15.101:37440) 2022-07-03T10:36:24.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 -- 172.21.15.101:0/2157970285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 msgr2=0x7f25e0072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:24.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 -- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 msgr2=0x7f25e0070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:24.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 0x7f25e0070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 -- 172.21.15.101:0/2157970285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4009410 con 0x7f25e00ff270 2022-07-03T10:36:24.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e4bb2700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:24.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e53b3700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 0x7f25e0070db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009b70 tx=0x7f25d8009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:24.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d801d070 con 0x7f25e00ff270 2022-07-03T10:36:24.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d80097e0 con 0x7f25e00ff270 2022-07-03T10:36:24.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f25d8005000 con 0x7f25e00ff270 2022-07-03T10:36:24.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d800f460 con 0x7f25e00ff270 2022-07-03T10:36:24.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.733+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e0118950 con 0x7f25e00ff270 2022-07-03T10:36:24.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.734+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e0108ec0 con 0x7f25e00ff270 2022-07-03T10:36:24.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.736+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7f25d8029030 con 0x7f25e00ff270 2022-07-03T10:36:24.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.738+0000 7f25dd7fa700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 0x7f25d0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:24.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.738+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f25d80c1120 con 0x7f25e00ff270 2022-07-03T10:36:24.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.738+0000 7f25e4bb2700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 0x7f25d0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:24.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.739+0000 7f25e4bb2700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 0x7f25d0099210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009fd0 tx=0x7f25d4005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:24.746 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.740+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f25d808e380 con 0x7f25e00ff270 2022-07-03T10:36:24.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.935+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f25e005e960 con 0x7f25d0096d60 2022-07-03T10:36:24.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.937+0000 7f25dd7fa700 1 -- 172.21.15.101:0/2157970285 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f25e005e960 con 0x7f25d0096d60 2022-07-03T10:36:24.939 INFO:teuthology.orchestra.run.smithi101.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-07-03T10:36:24.939 INFO:teuthology.orchestra.run.smithi101.stdout:smithi101 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204017B400BGN 400G 2m ago LVM detected, locked 2022-07-03T10:36:24.940 INFO:teuthology.orchestra.run.smithi101.stdout:smithi161 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001X400BGN 400G 2m ago LVM detected, locked 2022-07-03T10:36:24.940 INFO:teuthology.orchestra.run.smithi101.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 400G 2m ago LVM detected, locked 2022-07-03T10:36:24.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.939+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 msgr2=0x7f25d0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:24.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.939+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 0x7f25d0099210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009fd0 tx=0x7f25d4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.939+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 msgr2=0x7f25e0070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:24.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.939+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 0x7f25e0070db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009b70 tx=0x7f25d8009520 comp rx=0 tx=0).stop 2022-07-03T10:36:24.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 shutdown_connections 2022-07-03T10:36:24.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f25d0096d60 0x7f25d0099210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25e00ff270 0x7f25e0070db0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25e00fd660 0x7f25e0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 --2- 172.21.15.101:0/2157970285 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f25e00fe8e0 0x7f25e0070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:24.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 >> 172.21.15.101:0/2157970285 conn(0x7f25e00f8dd0 msgr2=0x7f25e00faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:24.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 shutdown_connections 2022-07-03T10:36:24.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:24.940+0000 7f25e6e16700 1 -- 172.21.15.101:0/2157970285 wait complete. 2022-07-03T10:36:25.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:25 smithi101 ceph-mon[85126]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:25.478 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-07-03T10:36:25.493 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi101.front.sepia.ceph.com 2022-07-03T10:36:25.494 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph fs dump' 2022-07-03T10:36:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:25 smithi161 ceph-mon[91579]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:25 smithi188 ceph-mon[92425]: pgmap v311: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:26.420 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:26.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:26 smithi101 ceph-mon[85126]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:26.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:36:26.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:36:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:26 smithi161 ceph-mon[91579]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:36:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:36:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:26 smithi188 ceph-mon[92425]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-07-03T10:36:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:36:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:36:27.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:27 smithi101 ceph-mon[85126]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:27 smithi161 ceph-mon[91579]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:27 smithi188 ceph-mon[92425]: pgmap v312: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:36:27.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.936+0000 7ff9238de700 1 -- 172.21.15.101:0/233370399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 msgr2=0x7ff91c0fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:27.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.936+0000 7ff9238de700 1 --2- 172.21.15.101:0/233370399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c0fda70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff918009900 tx=0x7ff9180091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 -- 172.21.15.101:0/233370399 shutdown_connections 2022-07-03T10:36:27.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 --2- 172.21.15.101:0/233370399 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c0fda70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 --2- 172.21.15.101:0/233370399 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 0x7ff91c105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 --2- 172.21.15.101:0/233370399 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 0x7ff91c0fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 -- 172.21.15.101:0/233370399 >> 172.21.15.101:0/233370399 conn(0x7ff91c0f8dd0 msgr2=0x7ff91c0fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:27.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 -- 172.21.15.101:0/233370399 shutdown_connections 2022-07-03T10:36:27.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.937+0000 7ff9238de700 1 -- 172.21.15.101:0/233370399 wait complete. 2022-07-03T10:36:27.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.938+0000 7ff9238de700 1 Processor -- start 2022-07-03T10:36:27.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.938+0000 7ff9238de700 1 -- start start 2022-07-03T10:36:27.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:27.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 0x7ff91c10ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:27.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 0x7ff91c115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:27.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c1161a0 con 0x7ff91c0fd660 2022-07-03T10:36:27.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c1162e0 con 0x7ff91c0ff270 2022-07-03T10:36:27.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff9238de700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c1164a0 con 0x7ff91c0fe8e0 2022-07-03T10:36:27.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff92167a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:27.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff92167a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:37450/0 (socket says 172.21.15.101:37450) 2022-07-03T10:36:27.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff92167a700 1 -- 172.21.15.101:0/1560642001 learned_addr learned my addr 172.21.15.101:0/1560642001 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:27.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff920e79700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 0x7ff91c10ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:27.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.939+0000 7ff921e7b700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 0x7ff91c115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:27.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 -- 172.21.15.101:0/1560642001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 msgr2=0x7ff91c10ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:27.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 0x7ff91c10ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 -- 172.21.15.101:0/1560642001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 msgr2=0x7ff91c115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:27.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 0x7ff91c115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:27.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 -- 172.21.15.101:0/1560642001 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918009440 con 0x7ff91c0fd660 2022-07-03T10:36:27.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff92167a700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff918009f70 tx=0x7ff9180091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:27.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91801d070 con 0x7ff91c0fd660 2022-07-03T10:36:27.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff91800b9c0 con 0x7ff91c0fd660 2022-07-03T10:36:27.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.940+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c116720 con 0x7ff91c0fd660 2022-07-03T10:36:27.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.941+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c116c20 con 0x7ff91c0fd660 2022-07-03T10:36:27.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.941+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918022c40 con 0x7ff91c0fd660 2022-07-03T10:36:27.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.942+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c108ec0 con 0x7ff91c0fd660 2022-07-03T10:36:27.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.943+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 125172+0+0 (secure 0 0 0) 0x7ff918029030 con 0x7ff91c0fd660 2022-07-03T10:36:27.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.945+0000 7ff911ffb700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 0x7ff908099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:27.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.945+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff9180c0b30 con 0x7ff91c0fd660 2022-07-03T10:36:27.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.945+0000 7ff920e79700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 0x7ff908099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:27.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.946+0000 7ff920e79700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 0x7ff908099290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009d00 tx=0x7ff90c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:27.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:27.949+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7ff91808dd30 con 0x7ff91c0fd660 2022-07-03T10:36:28.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.177+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff91c05f130 con 0x7ff91c0fd660 2022-07-03T10:36:28.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.177+0000 7ff911ffb700 1 -- 172.21.15.101:0/1560642001 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+2839 (secure 0 0 0) 0x7ff918027b30 con 0x7ff91c0fd660 2022-07-03T10:36:28.180 INFO:teuthology.orchestra.run.smithi101.stdout:e20 2022-07-03T10:36:28.180 INFO:teuthology.orchestra.run.smithi101.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-07-03T10:36:28.180 INFO:teuthology.orchestra.run.smithi101.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-07-03T10:36:28.181 INFO:teuthology.orchestra.run.smithi101.stdout:legacy client fscid: 1 2022-07-03T10:36:28.181 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:28.182 INFO:teuthology.orchestra.run.smithi101.stdout:Filesystem 'cephfs' (1) 2022-07-03T10:36:28.182 INFO:teuthology.orchestra.run.smithi101.stdout:fs_name cephfs 2022-07-03T10:36:28.183 INFO:teuthology.orchestra.run.smithi101.stdout:epoch 20 2022-07-03T10:36:28.183 INFO:teuthology.orchestra.run.smithi101.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-07-03T10:36:28.183 INFO:teuthology.orchestra.run.smithi101.stdout:created 2022-07-03T10:35:03.427264+0000 2022-07-03T10:36:28.183 INFO:teuthology.orchestra.run.smithi101.stdout:modified 2022-07-03T10:35:08.208608+0000 2022-07-03T10:36:28.184 INFO:teuthology.orchestra.run.smithi101.stdout:tableserver 0 2022-07-03T10:36:28.184 INFO:teuthology.orchestra.run.smithi101.stdout:root 0 2022-07-03T10:36:28.184 INFO:teuthology.orchestra.run.smithi101.stdout:session_timeout 300 2022-07-03T10:36:28.184 INFO:teuthology.orchestra.run.smithi101.stdout:session_autoclose 300 2022-07-03T10:36:28.184 INFO:teuthology.orchestra.run.smithi101.stdout:max_file_size 1099511627776 2022-07-03T10:36:28.185 INFO:teuthology.orchestra.run.smithi101.stdout:required_client_features {} 2022-07-03T10:36:28.185 INFO:teuthology.orchestra.run.smithi101.stdout:last_failure 0 2022-07-03T10:36:28.185 INFO:teuthology.orchestra.run.smithi101.stdout:last_failure_osd_epoch 0 2022-07-03T10:36:28.185 INFO:teuthology.orchestra.run.smithi101.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-07-03T10:36:28.186 INFO:teuthology.orchestra.run.smithi101.stdout:max_mds 1 2022-07-03T10:36:28.186 INFO:teuthology.orchestra.run.smithi101.stdout:in 0 2022-07-03T10:36:28.187 INFO:teuthology.orchestra.run.smithi101.stdout:up {0=24569} 2022-07-03T10:36:28.187 INFO:teuthology.orchestra.run.smithi101.stdout:failed 2022-07-03T10:36:28.187 INFO:teuthology.orchestra.run.smithi101.stdout:damaged 2022-07-03T10:36:28.187 INFO:teuthology.orchestra.run.smithi101.stdout:stopped 2022-07-03T10:36:28.187 INFO:teuthology.orchestra.run.smithi101.stdout:data_pools [3] 2022-07-03T10:36:28.188 INFO:teuthology.orchestra.run.smithi101.stdout:metadata_pool 2 2022-07-03T10:36:28.188 INFO:teuthology.orchestra.run.smithi101.stdout:inline_data disabled 2022-07-03T10:36:28.188 INFO:teuthology.orchestra.run.smithi101.stdout:balancer 2022-07-03T10:36:28.188 INFO:teuthology.orchestra.run.smithi101.stdout:standby_count_wanted 0 2022-07-03T10:36:28.189 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.l{0:24569} state up:active seq 3 addr [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.189 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.i{0:24557} state up:standby-replay seq 1 addr [v2:172.21.15.188:6836/639939509,v1:172.21.15.188:6837/639939509] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.189 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:28.189 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:28.189 INFO:teuthology.orchestra.run.smithi101.stdout:Standby daemons: 2022-07-03T10:36:28.190 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:28.190 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.a{-1:24466} state up:standby seq 1 addr [v2:172.21.15.101:6834/369400191,v1:172.21.15.101:6835/369400191] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.190 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.g{-1:24482} state up:standby seq 1 addr [v2:172.21.15.101:6838/3965232315,v1:172.21.15.101:6839/3965232315] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.190 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.d{-1:24484} state up:standby seq 1 addr [v2:172.21.15.101:6836/3291034276,v1:172.21.15.101:6837/3291034276] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.191 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.j{-1:24500} state up:standby seq 1 addr [v2:172.21.15.101:6840/2161782899,v1:172.21.15.101:6841/2161782899] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.191 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.b{-1:24517} state up:standby seq 1 addr [v2:172.21.15.161:6832/2910285108,v1:172.21.15.161:6833/2910285108] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.191 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.e{-1:24529} state up:standby seq 1 addr [v2:172.21.15.161:6834/429921526,v1:172.21.15.161:6835/429921526] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.192 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.c{-1:24539} state up:standby seq 1 addr [v2:172.21.15.188:6832/1718258929,v1:172.21.15.188:6833/1718258929] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.192 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.h{-1:24541} state up:standby seq 1 addr [v2:172.21.15.161:6836/1658187203,v1:172.21.15.161:6837/1658187203] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.192 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.f{-1:24545} state up:standby seq 1 addr [v2:172.21.15.188:6834/1026299276,v1:172.21.15.188:6835/1026299276] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.193 INFO:teuthology.orchestra.run.smithi101.stdout:[mds.k{-1:24553} state up:standby seq 1 addr [v2:172.21.15.161:6838/2119471810,v1:172.21.15.161:6839/2119471810] compat {c=[1],r=[1],i=[7ff]}] 2022-07-03T10:36:28.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 msgr2=0x7ff908099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:28.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 0x7ff908099290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009d00 tx=0x7ff90c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 msgr2=0x7ff91c10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:28.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff918009f70 tx=0x7ff9180091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 shutdown_connections 2022-07-03T10:36:28.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff908096de0 0x7ff908099290 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff91c0fd660 0x7ff91c10e670 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff91c0ff270 0x7ff91c115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 --2- 172.21.15.101:0/1560642001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff91c0fe8e0 0x7ff91c10ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:28.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 >> 172.21.15.101:0/1560642001 conn(0x7ff91c0f8dd0 msgr2=0x7ff91c101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:28.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 shutdown_connections 2022-07-03T10:36:28.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:28.180+0000 7ff9238de700 1 -- 172.21.15.101:0/1560642001 wait complete. 2022-07-03T10:36:28.197 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:28.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:28 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1560642001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-07-03T10:36:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:28 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1560642001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-07-03T10:36:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:28 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1560642001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-07-03T10:36:28.696 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- bash -c 'ceph osd dump' 2022-07-03T10:36:29.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:29 smithi101 ceph-mon[85126]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:29.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:29 smithi101 ceph-mon[85126]: mgrmap e18: x(active, since 8m), standbys: y, z 2022-07-03T10:36:29.584 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:29 smithi161 ceph-mon[91579]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:29 smithi161 ceph-mon[91579]: mgrmap e18: x(active, since 8m), standbys: y, z 2022-07-03T10:36:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:29 smithi188 ceph-mon[92425]: pgmap v313: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:36:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:29 smithi188 ceph-mon[92425]: mgrmap e18: x(active, since 8m), standbys: y, z 2022-07-03T10:36:31.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 -- 172.21.15.101:0/1676963848 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 msgr2=0x7fe9480fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:31.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 --2- 172.21.15.101:0/1676963848 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe9480fda70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe938009900 tx=0x7fe9380091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 -- 172.21.15.101:0/1676963848 shutdown_connections 2022-07-03T10:36:31.116 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 --2- 172.21.15.101:0/1676963848 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe9480fda70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 --2- 172.21.15.101:0/1676963848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 --2- 172.21.15.101:0/1676963848 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 0x7fe9480fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 -- 172.21.15.101:0/1676963848 >> 172.21.15.101:0/1676963848 conn(0x7fe9480f8dd0 msgr2=0x7fe9480fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:31.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.113+0000 7fe94c933700 1 -- 172.21.15.101:0/1676963848 shutdown_connections 2022-07-03T10:36:31.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.114+0000 7fe94c933700 1 -- 172.21.15.101:0/1676963848 wait complete. 2022-07-03T10:36:31.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.114+0000 7fe94c933700 1 Processor -- start 2022-07-03T10:36:31.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 -- start start 2022-07-03T10:36:31.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:31.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 0x7fe948110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:31.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948117dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:31.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe948118340 con 0x7fe9480fd660 2022-07-03T10:36:31.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe948118480 con 0x7fe9480ff270 2022-07-03T10:36:31.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94c933700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe948118640 con 0x7fe9480fe8e0 2022-07-03T10:36:31.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:31.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52660/0 (socket says 172.21.15.101:52660) 2022-07-03T10:36:31.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe94659c700 1 -- 172.21.15.101:0/2408208589 learned_addr learned my addr 172.21.15.101:0/2408208589 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:31.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe946d9d700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948117dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:31.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.115+0000 7fe945d9b700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 0x7fe948110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:31.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 -- 172.21.15.101:0/2408208589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 msgr2=0x7fe948110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:31.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 0x7fe948110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 -- 172.21.15.101:0/2408208589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 msgr2=0x7fe948117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:31.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948117dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 -- 172.21.15.101:0/2408208589 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe938009440 con 0x7fe9480fd660 2022-07-03T10:36:31.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe946d9d700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:31.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe94659c700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe938006010 tx=0x7fe9380091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:31.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.116+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93801d070 con 0x7fe9480fd660 2022-07-03T10:36:31.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.117+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe93800b9c0 con 0x7fe9480fd660 2022-07-03T10:36:31.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.117+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9481188c0 con 0x7fe9480fd660 2022-07-03T10:36:31.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.117+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe938022c10 con 0x7fe9480fd660 2022-07-03T10:36:31.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.117+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe948118dc0 con 0x7fe9480fd660 2022-07-03T10:36:31.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.118+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe948108e70 con 0x7fe9480fd660 2022-07-03T10:36:31.126 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.120+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fe938022d70 con 0x7fe9480fd660 2022-07-03T10:36:31.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.121+0000 7fe936ffd700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 0x7fe930099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:31.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.121+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fe9380c0d60 con 0x7fe9480fd660 2022-07-03T10:36:31.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.121+0000 7fe945d9b700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 0x7fe930099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:31.127 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.122+0000 7fe945d9b700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 0x7fe930099250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009d00 tx=0x7fe93c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:31.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.125+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe93808dfa0 con 0x7fe9480fd660 2022-07-03T10:36:31.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.313+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe948119400 con 0x7fe9480fd660 2022-07-03T10:36:31.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.314+0000 7fe936ffd700 1 -- 172.21.15.101:0/2408208589 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v82) v1 ==== 56+0+5261 (secure 0 0 0) 0x7fe938027070 con 0x7fe9480fd660 2022-07-03T10:36:31.317 INFO:teuthology.orchestra.run.smithi101.stdout:epoch 82 2022-07-03T10:36:31.317 INFO:teuthology.orchestra.run.smithi101.stdout:fsid 23a851ae-fab9-11ec-842c-001a4aab830c 2022-07-03T10:36:31.317 INFO:teuthology.orchestra.run.smithi101.stdout:created 2022-07-03T10:27:26.718252+0000 2022-07-03T10:36:31.317 INFO:teuthology.orchestra.run.smithi101.stdout:modified 2022-07-03T10:35:27.299095+0000 2022-07-03T10:36:31.318 INFO:teuthology.orchestra.run.smithi101.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2022-07-03T10:36:31.318 INFO:teuthology.orchestra.run.smithi101.stdout:crush_version 26 2022-07-03T10:36:31.318 INFO:teuthology.orchestra.run.smithi101.stdout:full_ratio 0.95 2022-07-03T10:36:31.319 INFO:teuthology.orchestra.run.smithi101.stdout:backfillfull_ratio 0.9 2022-07-03T10:36:31.319 INFO:teuthology.orchestra.run.smithi101.stdout:nearfull_ratio 0.85 2022-07-03T10:36:31.319 INFO:teuthology.orchestra.run.smithi101.stdout:require_min_compat_client luminous 2022-07-03T10:36:31.319 INFO:teuthology.orchestra.run.smithi101.stdout:min_compat_client luminous 2022-07-03T10:36:31.320 INFO:teuthology.orchestra.run.smithi101.stdout:require_osd_release quincy 2022-07-03T10:36:31.320 INFO:teuthology.orchestra.run.smithi101.stdout:stretch_mode_enabled false 2022-07-03T10:36:31.320 INFO:teuthology.orchestra.run.smithi101.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 19 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2022-07-03T10:36:31.321 INFO:teuthology.orchestra.run.smithi101.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs 2022-07-03T10:36:31.321 INFO:teuthology.orchestra.run.smithi101.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs 2022-07-03T10:36:31.321 INFO:teuthology.orchestra.run.smithi101.stdout:max_osd 12 2022-07-03T10:36:31.321 INFO:teuthology.orchestra.run.smithi101.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6802/2068504891,v1:172.21.15.101:6803/2068504891] [v2:172.21.15.101:6804/2068504891,v1:172.21.15.101:6805/2068504891] exists,up b6b433d6-6618-4afa-9a41-a4dbde9dfd3d 2022-07-03T10:36:31.321 INFO:teuthology.orchestra.run.smithi101.stdout:osd.1 up in weight 1 up_from 12 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6800/967237995,v1:172.21.15.161:6801/967237995] [v2:172.21.15.161:6802/967237995,v1:172.21.15.161:6803/967237995] exists,up 59dc2862-3622-479a-a620-87a0f8de138b 2022-07-03T10:36:31.322 INFO:teuthology.orchestra.run.smithi101.stdout:osd.2 up in weight 1 up_from 16 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6800/2999852966,v1:172.21.15.188:6801/2999852966] [v2:172.21.15.188:6802/2999852966,v1:172.21.15.188:6803/2999852966] exists,up acec9c8a-8e42-45f1-83a0-f598007cc2e1 2022-07-03T10:36:31.322 INFO:teuthology.orchestra.run.smithi101.stdout:osd.3 up in weight 1 up_from 23 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6810/3606989058,v1:172.21.15.101:6811/3606989058] [v2:172.21.15.101:6812/3606989058,v1:172.21.15.101:6813/3606989058] exists,up 14eb692b-ed90-438b-9b81-5b71756db9c5 2022-07-03T10:36:31.322 INFO:teuthology.orchestra.run.smithi101.stdout:osd.4 up in weight 1 up_from 30 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6808/54950390,v1:172.21.15.161:6809/54950390] [v2:172.21.15.161:6810/54950390,v1:172.21.15.161:6811/54950390] exists,up 4de30afa-ff39-4be7-9258-990f21fe1fc9 2022-07-03T10:36:31.323 INFO:teuthology.orchestra.run.smithi101.stdout:osd.5 up in weight 1 up_from 35 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6808/1424487492,v1:172.21.15.188:6809/1424487492] [v2:172.21.15.188:6810/1424487492,v1:172.21.15.188:6811/1424487492] exists,up 93593083-57f3-4a9c-a08d-c32067460386 2022-07-03T10:36:31.323 INFO:teuthology.orchestra.run.smithi101.stdout:osd.6 up in weight 1 up_from 43 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6818/3209986330,v1:172.21.15.101:6819/3209986330] [v2:172.21.15.101:6820/3209986330,v1:172.21.15.101:6821/3209986330] exists,up a3e65880-acae-4f11-9494-023c542b5e8c 2022-07-03T10:36:31.323 INFO:teuthology.orchestra.run.smithi101.stdout:osd.7 up in weight 1 up_from 47 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6816/328327819,v1:172.21.15.161:6817/328327819] [v2:172.21.15.161:6818/328327819,v1:172.21.15.161:6819/328327819] exists,up ed7c95a0-e815-4121-8542-947140c403f7 2022-07-03T10:36:31.323 INFO:teuthology.orchestra.run.smithi101.stdout:osd.8 up in weight 1 up_from 54 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6816/1483136090,v1:172.21.15.188:6817/1483136090] [v2:172.21.15.188:6818/1483136090,v1:172.21.15.188:6819/1483136090] exists,up 2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f 2022-07-03T10:36:31.324 INFO:teuthology.orchestra.run.smithi101.stdout:osd.9 up in weight 1 up_from 62 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6826/1335772894,v1:172.21.15.101:6827/1335772894] [v2:172.21.15.101:6828/1335772894,v1:172.21.15.101:6829/1335772894] exists,up 241d0da9-abc5-4259-8e73-454c3a885ebf 2022-07-03T10:36:31.324 INFO:teuthology.orchestra.run.smithi101.stdout:osd.10 up in weight 1 up_from 67 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.161:6824/1352813899,v1:172.21.15.161:6825/1352813899] [v2:172.21.15.161:6826/1352813899,v1:172.21.15.161:6827/1352813899] exists,up bee2129f-d332-4a19-b6c8-3e308c9e262a 2022-07-03T10:36:31.325 INFO:teuthology.orchestra.run.smithi101.stdout:osd.11 up in weight 1 up_from 75 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.188:6824/1726462750,v1:172.21.15.188:6825/1726462750] [v2:172.21.15.188:6826/1726462750,v1:172.21.15.188:6827/1726462750] exists,up 818815ae-5ae2-4fa9-a4bd-237e7f420337 2022-07-03T10:36:31.325 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 2.6 [8,11] 2022-07-03T10:36:31.325 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 2.c [2,4] 2022-07-03T10:36:31.325 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 2.13 [2,11] 2022-07-03T10:36:31.326 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.6 [4,7] 2022-07-03T10:36:31.326 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.11 [4,8] 2022-07-03T10:36:31.326 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.19 [3,6] 2022-07-03T10:36:31.326 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.1b [5,9] 2022-07-03T10:36:31.327 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.21 [5,9] 2022-07-03T10:36:31.327 INFO:teuthology.orchestra.run.smithi101.stdout:pg_upmap_items 3.3c [4,9] 2022-07-03T10:36:31.328 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/3515793351 expires 2022-07-04T10:28:25.795153+0000 2022-07-03T10:36:31.328 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:6801/2446540439 expires 2022-07-04T10:28:25.795153+0000 2022-07-03T10:36:31.329 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:6800/2446540439 expires 2022-07-04T10:28:25.795153+0000 2022-07-03T10:36:31.329 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/2333710927 expires 2022-07-04T10:28:25.795153+0000 2022-07-03T10:36:31.330 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/2063980612 expires 2022-07-04T10:28:25.795153+0000 2022-07-03T10:36:31.330 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/2182043394 expires 2022-07-04T10:27:58.929183+0000 2022-07-03T10:36:31.330 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/2680816285 expires 2022-07-04T10:27:58.929183+0000 2022-07-03T10:36:31.331 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:0/4072829405 expires 2022-07-04T10:27:58.929183+0000 2022-07-03T10:36:31.331 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:6800/1657735167 expires 2022-07-04T10:27:58.929183+0000 2022-07-03T10:36:31.332 INFO:teuthology.orchestra.run.smithi101.stdout:blocklist 172.21.15.101:6801/1657735167 expires 2022-07-04T10:27:58.929183+0000 2022-07-03T10:36:31.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.316+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 msgr2=0x7fe930099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:31.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.316+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 0x7fe930099250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009d00 tx=0x7fe93c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.316+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 msgr2=0x7fe948110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:31.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.316+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe938006010 tx=0x7fe9380091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 shutdown_connections 2022-07-03T10:36:31.335 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe930096da0 0x7fe930099250 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.336 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9480fd660 0x7fe948110810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.336 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9480ff270 0x7fe948117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.336 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 --2- 172.21.15.101:0/2408208589 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9480fe8e0 0x7fe948110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:31.336 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 >> 172.21.15.101:0/2408208589 conn(0x7fe9480f8dd0 msgr2=0x7fe9480fb210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:31.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 shutdown_connections 2022-07-03T10:36:31.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:31.317+0000 7fe94c933700 1 -- 172.21.15.101:0/2408208589 wait complete. 2022-07-03T10:36:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:31 smithi161 ceph-mon[91579]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:31 smithi188 ceph-mon[92425]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:31.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:31 smithi101 ceph-mon[85126]: pgmap v314: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:31.860 INFO:teuthology.run_tasks:Running task fs.ready... 2022-07-03T10:36:31.881 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:32.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 -- 172.21.15.101:0/776980235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fd3d0 msgr2=0x7f38600fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 --2- 172.21.15.101:0/776980235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fd3d0 0x7f38600fd7e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3850009900 tx=0x7f38500091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 -- 172.21.15.101:0/776980235 shutdown_connections 2022-07-03T10:36:32.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 --2- 172.21.15.101:0/776980235 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fd3d0 0x7f38600fd7e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 --2- 172.21.15.101:0/776980235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fefe0 0x7f38601058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 --2- 172.21.15.101:0/776980235 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fe650 0x7f38600feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 -- 172.21.15.101:0/776980235 >> 172.21.15.101:0/776980235 conn(0x7f38600f8b20 msgr2=0x7f38600faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:32.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 -- 172.21.15.101:0/776980235 shutdown_connections 2022-07-03T10:36:32.115 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.111+0000 7f38649fe700 1 -- 172.21.15.101:0/776980235 wait complete. 2022-07-03T10:36:32.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.112+0000 7f38649fe700 1 Processor -- start 2022-07-03T10:36:32.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.112+0000 7f38649fe700 1 -- start start 2022-07-03T10:36:32.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fd3d0 0x7f3860072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 0x7f3860070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3860071480 con 0x7f38600fefe0 2022-07-03T10:36:32.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600715f0 con 0x7f38600fd3d0 2022-07-03T10:36:32.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f38649fe700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3860118500 con 0x7f38600fe650 2022-07-03T10:36:32.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385f59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 0x7f3860070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:47856/0 (socket says 172.21.15.101:47856) 2022-07-03T10:36:32.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 -- 172.21.15.101:0/1158635441 learned_addr learned my addr 172.21.15.101:0/1158635441 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:32.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385ed9d700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fd3d0 0x7f3860072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 -- 172.21.15.101:0/1158635441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fd3d0 msgr2=0x7f3860072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fd3d0 0x7f3860072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 -- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 msgr2=0x7f3860070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 0x7f3860070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.121 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385e59c700 1 -- 172.21.15.101:0/1158635441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3850009440 con 0x7f38600fe650 2022-07-03T10:36:32.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.113+0000 7f385f59e700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 0x7f3860070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:32.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f385e59c700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f385400ea80 tx=0x7f3854005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:32.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400c7c0 con 0x7f38600fe650 2022-07-03T10:36:32.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3860118760 con 0x7f38600fe650 2022-07-03T10:36:32.122 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3854010430 con 0x7f38600fe650 2022-07-03T10:36:32.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3860118cf0 con 0x7f38600fe650 2022-07-03T10:36:32.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.114+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854018420 con 0x7f38600fe650 2022-07-03T10:36:32.123 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.115+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860108be0 con 0x7f38600fe650 2022-07-03T10:36:32.124 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.117+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f385400c040 con 0x7f38600fe650 2022-07-03T10:36:32.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.118+0000 7f38477fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 0x7f38480991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.119+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f3854014070 con 0x7f38600fe650 2022-07-03T10:36:32.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.119+0000 7f385ed9d700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 0x7f38480991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.119+0000 7f385ed9d700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 0x7f38480991d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f38500097d0 tx=0x7f38500091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:32.125 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.121+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f385408cb90 con 0x7f38600fe650 2022-07-03T10:36:32.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.312+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38601045c0 con 0x7f38600fe650 2022-07-03T10:36:32.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.314+0000 7f38477fe700 1 -- 172.21.15.101:0/1158635441 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7f38540901b0 con 0x7f38600fe650 2022-07-03T10:36:32.317 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:32.317 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:32.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 msgr2=0x7f38480991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 0x7f38480991d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f38500097d0 tx=0x7f38500091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 msgr2=0x7f3860070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f385400ea80 tx=0x7f3854005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 shutdown_connections 2022-07-03T10:36:32.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3848096d20 0x7f38480991d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f38600fefe0 0x7f3860070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f38600fd3d0 0x7f3860072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 --2- 172.21.15.101:0/1158635441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38600fe650 0x7f3860070870 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 >> 172.21.15.101:0/1158635441 conn(0x7f38600f8b20 msgr2=0x7f3860101880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:32.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 shutdown_connections 2022-07-03T10:36:32.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.317+0000 7f38649fe700 1 -- 172.21.15.101:0/1158635441 wait complete. 2022-07-03T10:36:32.348 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:32.349 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:32.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 -- 172.21.15.101:0/2514344215 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fd5e0 msgr2=0x7f62f80fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/2514344215 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fd5e0 0x7f62f80fd9f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f62e8009900 tx=0x7f62e80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 -- 172.21.15.101:0/2514344215 shutdown_connections 2022-07-03T10:36:32.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/2514344215 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fd5e0 0x7f62f80fd9f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/2514344215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80ff1f0 0x7f62f8105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/2514344215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80fe860 0x7f62f80fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 -- 172.21.15.101:0/2514344215 >> 172.21.15.101:0/2514344215 conn(0x7f62f80f8d50 msgr2=0x7f62f80fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:32.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 -- 172.21.15.101:0/2514344215 shutdown_connections 2022-07-03T10:36:32.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.577+0000 7f62fe9ff700 1 -- 172.21.15.101:0/2514344215 wait complete. 2022-07-03T10:36:32.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.578+0000 7f62fe9ff700 1 Processor -- start 2022-07-03T10:36:32.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.578+0000 7f62fe9ff700 1 -- start start 2022-07-03T10:36:32.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.578+0000 7f62fe9ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80fd5e0 0x7f62f810e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.578+0000 7f62fe9ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62fe9ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80ff1f0 0x7f62f8115ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62fe9ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8116120 con 0x7f62f80fe860 2022-07-03T10:36:32.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62fe9ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8116260 con 0x7f62f80ff1f0 2022-07-03T10:36:32.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62fe9ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8116420 con 0x7f62f80fd5e0 2022-07-03T10:36:32.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52684/0 (socket says 172.21.15.101:52684) 2022-07-03T10:36:32.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 -- 172.21.15.101:0/1141815260 learned_addr learned my addr 172.21.15.101:0/1141815260 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:32.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f7fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80fd5e0 0x7f62f810e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62fcf9c700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80ff1f0 0x7f62f8115ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 -- 172.21.15.101:0/1141815260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80fd5e0 msgr2=0x7f62f810e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80fd5e0 0x7f62f810e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 -- 172.21.15.101:0/1141815260 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80ff1f0 msgr2=0x7f62f8115ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80ff1f0 0x7f62f8115ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.579+0000 7f62f77fe700 1 -- 172.21.15.101:0/1141815260 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f0009700 con 0x7f62f80fe860 2022-07-03T10:36:32.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62f77fe700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00ea80 tx=0x7f62ec005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:32.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec00c7c0 con 0x7f62f80fe860 2022-07-03T10:36:32.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f62ec00cdd0 con 0x7f62f80fe860 2022-07-03T10:36:32.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec010df0 con 0x7f62f80fe860 2022-07-03T10:36:32.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e8009440 con 0x7f62f80fe860 2022-07-03T10:36:32.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.580+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f8116ab0 con 0x7f62f80fe860 2022-07-03T10:36:32.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.581+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f8108d80 con 0x7f62f80fe860 2022-07-03T10:36:32.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.583+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f62ec00c040 con 0x7f62f80fe860 2022-07-03T10:36:32.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.584+0000 7f62f4ff9700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 0x7f62e0099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:32.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.585+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f62ec014070 con 0x7f62f80fe860 2022-07-03T10:36:32.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.585+0000 7f62f7fff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 0x7f62e0099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:32.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.586+0000 7f62f7fff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 0x7f62e0099250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097d0 tx=0x7f62e80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:32.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.588+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f62ec04f100 con 0x7f62f80fe860 2022-07-03T10:36:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:32 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2408208589' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-07-03T10:36:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:32 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2408208589' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-07-03T10:36:32.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:32 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2408208589' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-07-03T10:36:32.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.775+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62f8108f10 con 0x7f62f80fe860 2022-07-03T10:36:32.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.776+0000 7f62f4ff9700 1 -- 172.21.15.101:0/1141815260 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62f8108f10 con 0x7f62f80fe860 2022-07-03T10:36:32.779 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:32.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 msgr2=0x7f62e0099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.782 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 0x7f62e0099250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f62e80097d0 tx=0x7f62e80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.782 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 msgr2=0x7f62f810eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:32.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00ea80 tx=0x7f62ec005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 shutdown_connections 2022-07-03T10:36:32.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f62e0096da0 0x7f62e0099250 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62f80ff1f0 0x7f62f8115ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62f80fd5e0 0x7f62f810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 --2- 172.21.15.101:0/1141815260 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f62f80fe860 0x7f62f810eb30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:32.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 >> 172.21.15.101:0/1141815260 conn(0x7f62f80f8d50 msgr2=0x7f62f8101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:32.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 shutdown_connections 2022-07-03T10:36:32.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:32.778+0000 7f62fe9ff700 1 -- 172.21.15.101:0/1141815260 wait complete. 2022-07-03T10:36:32.807 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:33.041 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 -- 172.21.15.101:0/2168147478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 msgr2=0x7fdc0c0fd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2168147478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c0fd810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009900 tx=0x7fdc080091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 -- 172.21.15.101:0/2168147478 shutdown_connections 2022-07-03T10:36:33.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2168147478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c0fd810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2168147478 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 0x7fdc0c1058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2168147478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c0fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.045 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.035+0000 7fdc1299d700 1 -- 172.21.15.101:0/2168147478 >> 172.21.15.101:0/2168147478 conn(0x7fdc0c0f8b70 msgr2=0x7fdc0c0fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:33.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.036+0000 7fdc1299d700 1 -- 172.21.15.101:0/2168147478 shutdown_connections 2022-07-03T10:36:33.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.036+0000 7fdc1299d700 1 -- 172.21.15.101:0/2168147478 wait complete. 2022-07-03T10:36:33.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 Processor -- start 2022-07-03T10:36:33.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 -- start start 2022-07-03T10:36:33.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c110640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 0x7fdc0c117bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c118170 con 0x7fdc0c0fd400 2022-07-03T10:36:33.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c1182b0 con 0x7fdc0c0ff010 2022-07-03T10:36:33.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1299d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c118470 con 0x7fdc0c0fe680 2022-07-03T10:36:33.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1119a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.037+0000 7fdc1119a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:47908/0 (socket says 172.21.15.101:47908) 2022-07-03T10:36:33.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1219c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 0x7fdc0c117bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 -- 172.21.15.101:0/2511112917 learned_addr learned my addr 172.21.15.101:0/2511112917 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:33.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1199b700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 -- 172.21.15.101:0/2511112917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 msgr2=0x7fdc0c117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 0x7fdc0c117bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 -- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 msgr2=0x7fdc0c110640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c110640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 -- 172.21.15.101:0/2511112917 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc08009440 con 0x7fdc0c0fe680 2022-07-03T10:36:33.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc1119a700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00ea80 tx=0x7fdbfc005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.038+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc005020 con 0x7fdc0c0fe680 2022-07-03T10:36:33.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.039+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c118750 con 0x7fdc0c0fe680 2022-07-03T10:36:33.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.039+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdbfc00c7c0 con 0x7fdc0c0fe680 2022-07-03T10:36:33.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.039+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc004940 con 0x7fdc0c0fe680 2022-07-03T10:36:33.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.039+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c118ce0 con 0x7fdc0c0fe680 2022-07-03T10:36:33.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.040+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c108c00 con 0x7fdc0c0fe680 2022-07-03T10:36:33.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.042+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fdbfc00c040 con 0x7fdc0c0fe680 2022-07-03T10:36:33.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.043+0000 7fdc027fc700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 0x7fdbf80991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.043+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fdbfc014070 con 0x7fdc0c0fe680 2022-07-03T10:36:33.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.043+0000 7fdc1199b700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 0x7fdbf80991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.044+0000 7fdc1199b700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 0x7fdbf80991d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc08006010 tx=0x7fdc080091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.047+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fdbfc08d0d0 con 0x7fdc0c0fe680 2022-07-03T10:36:33.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.234+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdc0c108d90 con 0x7fdc0c0fe680 2022-07-03T10:36:33.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.234+0000 7fdc027fc700 1 -- 172.21.15.101:0/2511112917 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdc0c108d90 con 0x7fdc0c0fe680 2022-07-03T10:36:33.237 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:33.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 msgr2=0x7fdbf80991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 0x7fdbf80991d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc08006010 tx=0x7fdc080091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 msgr2=0x7fdc0c110b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00ea80 tx=0x7fdbfc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 shutdown_connections 2022-07-03T10:36:33.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fdbf8096d20 0x7fdbf80991d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fdc0c0fd400 0x7fdc0c110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdc0c0ff010 0x7fdc0c117bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 --2- 172.21.15.101:0/2511112917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc0c0fe680 0x7fdc0c110b80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 >> 172.21.15.101:0/2511112917 conn(0x7fdc0c0f8b70 msgr2=0x7fdc0c0faf10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:33.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 shutdown_connections 2022-07-03T10:36:33.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.237+0000 7fdc1299d700 1 -- 172.21.15.101:0/2511112917 wait complete. 2022-07-03T10:36:33.266 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:33.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.493+0000 7f427ef9d700 1 -- 172.21.15.101:0/1245807905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 msgr2=0x7f42781058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1245807905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781058d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4270009bf0 tx=0x7f4270009460 comp rx=0 tx=0).stop 2022-07-03T10:36:33.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 -- 172.21.15.101:0/1245807905 shutdown_connections 2022-07-03T10:36:33.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1245807905 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f42780fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1245807905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781058d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1245807905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f42780fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 -- 172.21.15.101:0/1245807905 >> 172.21.15.101:0/1245807905 conn(0x7f42780f8b70 msgr2=0x7f42780fafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:33.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 -- 172.21.15.101:0/1245807905 shutdown_connections 2022-07-03T10:36:33.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.494+0000 7f427ef9d700 1 -- 172.21.15.101:0/1245807905 wait complete. 2022-07-03T10:36:33.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 Processor -- start 2022-07-03T10:36:33.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 -- start start 2022-07-03T10:36:33.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f427810e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.506 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f427810e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.507 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781156f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4278115c70 con 0x7f42780fd400 2022-07-03T10:36:33.508 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4278115db0 con 0x7f42780fe680 2022-07-03T10:36:33.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.495+0000 7f427ef9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4278115f70 con 0x7f42780ff010 2022-07-03T10:36:33.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427df9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f427810e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f427810e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.509 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427df9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f427810e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52716/0 (socket says 172.21.15.101:52716) 2022-07-03T10:36:33.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427df9b700 1 -- 172.21.15.101:0/1396286250 learned_addr learned my addr 172.21.15.101:0/1396286250 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:33.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427e79c700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 -- 172.21.15.101:0/1396286250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 msgr2=0x7f42781156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.510 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781156f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 -- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 msgr2=0x7f427810e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f427810e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 -- 172.21.15.101:0/1396286250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4270009700 con 0x7f42780fe680 2022-07-03T10:36:33.511 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.496+0000 7f427d79a700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f427810e680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f426800e7d0 tx=0x7f4268005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.497+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268010710 con 0x7f42780fe680 2022-07-03T10:36:33.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.497+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4268010d20 con 0x7f42780fe680 2022-07-03T10:36:33.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.497+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4278116250 con 0x7f42780fe680 2022-07-03T10:36:33.512 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.497+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268018df0 con 0x7f42780fe680 2022-07-03T10:36:33.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.497+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4278116830 con 0x7f42780fe680 2022-07-03T10:36:33.513 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.498+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4278108b90 con 0x7f42780fe680 2022-07-03T10:36:33.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.500+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f426800c040 con 0x7f42780fe680 2022-07-03T10:36:33.514 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.501+0000 7f426e7fc700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 0x7f4264099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.501+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f4268014070 con 0x7f42780fe680 2022-07-03T10:36:33.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.501+0000 7f427df9b700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 0x7f4264099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.502+0000 7f427df9b700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 0x7f4264099250 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4274009e30 tx=0x7f42740091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.503+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f426808cc40 con 0x7f42780fe680 2022-07-03T10:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:33 smithi161 ceph-mon[91579]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2022-07-03T10:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:33 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1158635441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:33 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1141815260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:33 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2511112917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:33 smithi188 ceph-mon[92425]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2022-07-03T10:36:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:33 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1158635441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:33 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1141815260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:33 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2511112917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.689+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4278108d20 con 0x7f42780fe680 2022-07-03T10:36:33.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.689+0000 7f426e7fc700 1 -- 172.21.15.101:0/1396286250 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4278108d20 con 0x7f42780fe680 2022-07-03T10:36:33.692 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:33.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 msgr2=0x7f4264099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 0x7f4264099250 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4274009e30 tx=0x7f42740091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 msgr2=0x7f427810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f427810e680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f426800e7d0 tx=0x7f4268005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 shutdown_connections 2022-07-03T10:36:33.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4264096da0 0x7f4264099250 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42780fd400 0x7f427810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42780ff010 0x7f42781156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 --2- 172.21.15.101:0/1396286250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42780fe680 0x7f427810e680 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 >> 172.21.15.101:0/1396286250 conn(0x7f42780f8b70 msgr2=0x7f42780fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:33.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 shutdown_connections 2022-07-03T10:36:33.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.692+0000 7f427ef9d700 1 -- 172.21.15.101:0/1396286250 wait complete. 2022-07-03T10:36:33.718 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:33 smithi101 ceph-mon[85126]: pgmap v315: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2022-07-03T10:36:33.719 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:33 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1158635441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:33.719 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:33 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1141815260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.719 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:33 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2511112917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:33.721 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:36:33.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 -- 172.21.15.101:0/4231173731 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fe830 msgr2=0x7ff5100fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 --2- 172.21.15.101:0/4231173731 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fe830 0x7ff5100fec80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff504009900 tx=0x7ff5040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 -- 172.21.15.101:0/4231173731 shutdown_connections 2022-07-03T10:36:33.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 --2- 172.21.15.101:0/4231173731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 0x7ff510105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 --2- 172.21.15.101:0/4231173731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fd5b0 0x7ff5100fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 --2- 172.21.15.101:0/4231173731 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fe830 0x7ff5100fec80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 -- 172.21.15.101:0/4231173731 >> 172.21.15.101:0/4231173731 conn(0x7ff5100f8d00 msgr2=0x7ff5100fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:33.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 -- 172.21.15.101:0/4231173731 shutdown_connections 2022-07-03T10:36:33.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.946+0000 7ff5161ff700 1 -- 172.21.15.101:0/4231173731 wait complete. 2022-07-03T10:36:33.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 Processor -- start 2022-07-03T10:36:33.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 -- start start 2022-07-03T10:36:33.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fe830 0x7ff510110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 0x7ff510117dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510118340 con 0x7ff5100fd5b0 2022-07-03T10:36:33.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510118480 con 0x7ff5100ff1c0 2022-07-03T10:36:33.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.947+0000 7ff5161ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510118640 con 0x7ff5100fe830 2022-07-03T10:36:33.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52732/0 (socket says 172.21.15.101:52732) 2022-07-03T10:36:33.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 -- 172.21.15.101:0/2472902651 learned_addr learned my addr 172.21.15.101:0/2472902651 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:33.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50ffff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 0x7ff510117dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50effd700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fe830 0x7ff510110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 -- 172.21.15.101:0/2472902651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fe830 msgr2=0x7ff510110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fe830 0x7ff510110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 -- 172.21.15.101:0/2472902651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 msgr2=0x7ff510117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:33.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 0x7ff510117dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:33.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 -- 172.21.15.101:0/2472902651 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508009700 con 0x7ff5100fd5b0 2022-07-03T10:36:33.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.948+0000 7ff50f7fe700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff50000e840 tx=0x7ff500005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.949+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500004030 con 0x7ff5100fd5b0 2022-07-03T10:36:33.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.949+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff504009440 con 0x7ff5100fd5b0 2022-07-03T10:36:33.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.949+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff500004190 con 0x7ff5100fd5b0 2022-07-03T10:36:33.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.949+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500003a20 con 0x7ff5100fd5b0 2022-07-03T10:36:33.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.949+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff510118cd0 con 0x7ff5100fd5b0 2022-07-03T10:36:33.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.950+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff510108dc0 con 0x7ff5100fd5b0 2022-07-03T10:36:33.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.952+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7ff50000c040 con 0x7ff5100fd5b0 2022-07-03T10:36:33.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.953+0000 7ff4f7fff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 0x7ff4f80991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:33.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.953+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff500014070 con 0x7ff5100fd5b0 2022-07-03T10:36:33.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.953+0000 7ff50effd700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 0x7ff4f80991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:33.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.954+0000 7ff50effd700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 0x7ff4f80991d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff5040092d0 tx=0x7ff5040091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:33.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:33.957+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7ff50008d140 con 0x7ff5100fd5b0 2022-07-03T10:36:34.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.191+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff510108f50 con 0x7ff5100fd5b0 2022-07-03T10:36:34.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.192+0000 7ff4f7fff700 1 -- 172.21.15.101:0/2472902651 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7ff510108f50 con 0x7ff5100fd5b0 2022-07-03T10:36:34.194 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:34.195 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:36:34.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.194+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 msgr2=0x7ff4f80991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.194+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 0x7ff4f80991d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff5040092d0 tx=0x7ff5040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 msgr2=0x7ff510110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff50000e840 tx=0x7ff500005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 shutdown_connections 2022-07-03T10:36:34.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff4f8096d20 0x7ff4f80991d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff5100fd5b0 0x7ff510110810 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff5100ff1c0 0x7ff510117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 --2- 172.21.15.101:0/2472902651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff5100fe830 0x7ff510110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 >> 172.21.15.101:0/2472902651 conn(0x7ff5100f8d00 msgr2=0x7ff5100fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:34.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 shutdown_connections 2022-07-03T10:36:34.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.195+0000 7ff5161ff700 1 -- 172.21.15.101:0/2472902651 wait complete. 2022-07-03T10:36:34.200 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:34.224 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:34.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.453+0000 7fc11ab9d700 1 -- 172.21.15.101:0/649063035 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 msgr2=0x7fc1140feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.453+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/649063035 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc1140feaa0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc104009900 tx=0x7fc1040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 -- 172.21.15.101:0/649063035 shutdown_connections 2022-07-03T10:36:34.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/649063035 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fefe0 0x7fc1141058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/649063035 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fd3d0 0x7fc1140fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/649063035 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc1140feaa0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 -- 172.21.15.101:0/649063035 >> 172.21.15.101:0/649063035 conn(0x7fc1140f8b20 msgr2=0x7fc1140faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:34.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 -- 172.21.15.101:0/649063035 shutdown_connections 2022-07-03T10:36:34.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.454+0000 7fc11ab9d700 1 -- 172.21.15.101:0/649063035 wait complete. 2022-07-03T10:36:34.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 Processor -- start 2022-07-03T10:36:34.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 -- start start 2022-07-03T10:36:34.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fd3d0 0x7fc114072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fefe0 0x7fc114070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc114071480 con 0x7fc1140fe650 2022-07-03T10:36:34.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1140715f0 con 0x7fc1140fefe0 2022-07-03T10:36:34.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11ab9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1141184e0 con 0x7fc1140fd3d0 2022-07-03T10:36:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11939a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11939a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52750/0 (socket says 172.21.15.101:52750) 2022-07-03T10:36:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.455+0000 7fc11939a700 1 -- 172.21.15.101:0/375488560 learned_addr learned my addr 172.21.15.101:0/375488560 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc119b9b700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fd3d0 0x7fc114072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11a39c700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fefe0 0x7fc114070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 -- 172.21.15.101:0/375488560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fd3d0 msgr2=0x7fc114072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fd3d0 0x7fc114072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 -- 172.21.15.101:0/375488560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fefe0 msgr2=0x7fc114070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fefe0 0x7fc114070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 -- 172.21.15.101:0/375488560 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc104009440 con 0x7fc1140fe650 2022-07-03T10:36:34.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.456+0000 7fc11939a700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc104005d00 tx=0x7fc1040091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:34.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.457+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10401d070 con 0x7fc1140fe650 2022-07-03T10:36:34.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.457+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc10400b9c0 con 0x7fc1140fe650 2022-07-03T10:36:34.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.457+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1141186e0 con 0x7fc1140fe650 2022-07-03T10:36:34.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.457+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc104022b80 con 0x7fc1140fe650 2022-07-03T10:36:34.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.457+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc114118b80 con 0x7fc1140fe650 2022-07-03T10:36:34.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.458+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114108bf0 con 0x7fc1140fe650 2022-07-03T10:36:34.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.459+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fc104029030 con 0x7fc1140fe650 2022-07-03T10:36:34.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.461+0000 7fc10a7fc700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 0x7fc1000991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.461+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fc1040c0ad0 con 0x7fc1140fe650 2022-07-03T10:36:34.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.461+0000 7fc119b9b700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 0x7fc1000991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.462+0000 7fc119b9b700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 0x7fc1000991d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1140fe4b0 tx=0x7fc110005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:34.473 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.465+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fc10405a100 con 0x7fc1140fe650 2022-07-03T10:36:34.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:34 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1396286250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:34.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:34 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2472902651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:34 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1396286250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:34 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2472902651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:34.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.656+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc11405f130 con 0x7fc1140fe650 2022-07-03T10:36:34.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.657+0000 7fc10a7fc700 1 -- 172.21.15.101:0/375488560 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7fc104027090 con 0x7fc1140fe650 2022-07-03T10:36:34.660 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:34.660 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:34.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.659+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 msgr2=0x7fc1000991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.659+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 0x7fc1000991d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1140fe4b0 tx=0x7fc110005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.659+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 msgr2=0x7fc114070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.659+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc104005d00 tx=0x7fc1040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 shutdown_connections 2022-07-03T10:36:34.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc100096d20 0x7fc1000991d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1140fefe0 0x7fc114070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1140fd3d0 0x7fc114072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 --2- 172.21.15.101:0/375488560 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc1140fe650 0x7fc114070870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 >> 172.21.15.101:0/375488560 conn(0x7fc1140f8b20 msgr2=0x7fc114101880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:34.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 shutdown_connections 2022-07-03T10:36:34.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.660+0000 7fc11ab9d700 1 -- 172.21.15.101:0/375488560 wait complete. 2022-07-03T10:36:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:34 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1396286250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:34 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2472902651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:34.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:34.690 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:34.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 -- 172.21.15.101:0/2629412538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fd5c0 msgr2=0x7f12d80fd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2629412538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fd5c0 0x7f12d80fd9d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009900 tx=0x7f12c80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 -- 172.21.15.101:0/2629412538 shutdown_connections 2022-07-03T10:36:34.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2629412538 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80ff1d0 0x7f12d8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2629412538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fd5c0 0x7f12d80fd9d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2629412538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80fe840 0x7f12d80fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 -- 172.21.15.101:0/2629412538 >> 172.21.15.101:0/2629412538 conn(0x7f12d80f8d50 msgr2=0x7f12d80fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:34.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 -- 172.21.15.101:0/2629412538 shutdown_connections 2022-07-03T10:36:34.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.915+0000 7f12dcdff700 1 -- 172.21.15.101:0/2629412538 wait complete. 2022-07-03T10:36:34.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 Processor -- start 2022-07-03T10:36:34.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 -- start start 2022-07-03T10:36:34.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 0x7f12d8072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80ff1d0 0x7f12d8070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8071480 con 0x7f12d80fd5c0 2022-07-03T10:36:34.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d80715f0 con 0x7f12d80fe840 2022-07-03T10:36:34.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.916+0000 7f12dcdff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8118780 con 0x7f12d80ff1d0 2022-07-03T10:36:34.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12d659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 0x7f12d8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12d659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 0x7f12d8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52764/0 (socket says 172.21.15.101:52764) 2022-07-03T10:36:34.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12d659c700 1 -- 172.21.15.101:0/2466069026 learned_addr learned my addr 172.21.15.101:0/2466069026 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:34.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:59014/0 (socket says 172.21.15.101:59014) 2022-07-03T10:36:34.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12d6d9d700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80ff1d0 0x7f12d8070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 -- 172.21.15.101:0/2466069026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80ff1d0 msgr2=0x7f12d8070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80ff1d0 0x7f12d8070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 -- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 msgr2=0x7f12d8072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:34.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 0x7f12d8072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:34.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 -- 172.21.15.101:0/2466069026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8009440 con 0x7f12d80fe840 2022-07-03T10:36:34.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.917+0000 7f12cffff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12c000e850 tx=0x7f12c0005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:34.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.918+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0009420 con 0x7f12d80fe840 2022-07-03T10:36:34.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.918+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d81189e0 con 0x7f12d80fe840 2022-07-03T10:36:34.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.918+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f12c0005020 con 0x7f12d80fe840 2022-07-03T10:36:34.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.918+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c0010d60 con 0x7f12d80fe840 2022-07-03T10:36:34.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.918+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d8118f40 con 0x7f12d80fe840 2022-07-03T10:36:34.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.919+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d8108dd0 con 0x7f12d80fe840 2022-07-03T10:36:34.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.921+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f12c001e020 con 0x7f12d80fe840 2022-07-03T10:36:34.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.922+0000 7f12ceffd700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 0x7f12c40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:34.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.922+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f12c0014070 con 0x7f12d80fe840 2022-07-03T10:36:34.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.922+0000 7f12d659c700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 0x7f12c40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:34.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.923+0000 7f12d659c700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 0x7f12c40991d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f12c80092d0 tx=0x7f12c80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:35.006 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:34.924+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f12c008d1b0 con 0x7f12d80fe840 2022-07-03T10:36:35.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.112+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12d8108f60 con 0x7f12d80fe840 2022-07-03T10:36:35.114 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.113+0000 7f12ceffd700 1 -- 172.21.15.101:0/2466069026 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12d8108f60 con 0x7f12d80fe840 2022-07-03T10:36:35.115 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:35.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 msgr2=0x7f12c40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.117 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 0x7f12c40991d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f12c80092d0 tx=0x7f12c80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 msgr2=0x7f12d8070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12c000e850 tx=0x7f12c0005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 shutdown_connections 2022-07-03T10:36:35.118 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f12c4096d20 0x7f12c40991d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12d80fd5c0 0x7f12d8072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f12d80ff1d0 0x7f12d8070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 --2- 172.21.15.101:0/2466069026 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f12d80fe840 0x7f12d8070870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.119 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 >> 172.21.15.101:0/2466069026 conn(0x7f12d80f8d50 msgr2=0x7f12d8101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:35.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 shutdown_connections 2022-07-03T10:36:35.120 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.115+0000 7f12dcdff700 1 -- 172.21.15.101:0/2466069026 wait complete. 2022-07-03T10:36:35.145 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:35.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 -- 172.21.15.101:0/3857199421 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3530103250 msgr2=0x7f35301056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 --2- 172.21.15.101:0/3857199421 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3530103250 0x7f35301056b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3528009ca0 tx=0x7f3528009520 comp rx=0 tx=0).stop 2022-07-03T10:36:35.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 -- 172.21.15.101:0/3857199421 shutdown_connections 2022-07-03T10:36:35.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 --2- 172.21.15.101:0/3857199421 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3530103250 0x7f35301056b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 --2- 172.21.15.101:0/3857199421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f35300662c0 0x7f35301002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 --2- 172.21.15.101:0/3857199421 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.374+0000 7f35365ff700 1 -- 172.21.15.101:0/3857199421 >> 172.21.15.101:0/3857199421 conn(0x7f35300f7c50 msgr2=0x7f35300fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:35.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.375+0000 7f35365ff700 1 -- 172.21.15.101:0/3857199421 shutdown_connections 2022-07-03T10:36:35.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.375+0000 7f35365ff700 1 -- 172.21.15.101:0/3857199421 wait complete. 2022-07-03T10:36:35.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.375+0000 7f35365ff700 1 Processor -- start 2022-07-03T10:36:35.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 -- start start 2022-07-03T10:36:35.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530110a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530103250 0x7f3530117ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530118040 con 0x7f35300662c0 2022-07-03T10:36:35.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530118180 con 0x7f3530100800 2022-07-03T10:36:35.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f35365ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3530118340 con 0x7f3530103250 2022-07-03T10:36:35.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f352ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f352ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52784/0 (socket says 172.21.15.101:52784) 2022-07-03T10:36:35.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f352ffff700 1 -- 172.21.15.101:0/2594024412 learned_addr learned my addr 172.21.15.101:0/2594024412 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:35.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.376+0000 7f352f7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530110a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f3534b9c700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530103250 0x7f3530117ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 -- 172.21.15.101:0/2594024412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530103250 msgr2=0x7f3530117ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530103250 0x7f3530117ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 -- 172.21.15.101:0/2594024412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 msgr2=0x7f3530110a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530110a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 -- 172.21.15.101:0/2594024412 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524009410 con 0x7f35300662c0 2022-07-03T10:36:35.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352f7fe700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:35.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352ffff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f352000ea50 tx=0x7f3520005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:35.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.377+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520004020 con 0x7f35300662c0 2022-07-03T10:36:35.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.378+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35280097e0 con 0x7f35300662c0 2022-07-03T10:36:35.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.378+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3520004180 con 0x7f35300662c0 2022-07-03T10:36:35.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.378+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200039b0 con 0x7f35300662c0 2022-07-03T10:36:35.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.378+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3530118a20 con 0x7f35300662c0 2022-07-03T10:36:35.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.379+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3530108da0 con 0x7f35300662c0 2022-07-03T10:36:35.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.380+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f352000c040 con 0x7f35300662c0 2022-07-03T10:36:35.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.382+0000 7f352cff9700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 0x7f35180991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.382+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f3520014070 con 0x7f35300662c0 2022-07-03T10:36:35.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.382+0000 7f352f7fe700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 0x7f35180991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.383+0000 7f352f7fe700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 0x7f35180991d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35240116e0 tx=0x7f3524005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:35.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.385+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f352008cc40 con 0x7f35300662c0 2022-07-03T10:36:35.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:35 smithi101 ceph-mon[85126]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:35.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:35 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/375488560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:35.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:35 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2466069026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:35.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.572+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3530108f30 con 0x7f35300662c0 2022-07-03T10:36:35.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.573+0000 7f352cff9700 1 -- 172.21.15.101:0/2594024412 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3530108f30 con 0x7f35300662c0 2022-07-03T10:36:35.576 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:35.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 msgr2=0x7f35180991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 0x7f35180991d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35240116e0 tx=0x7f3524005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 msgr2=0x7f3530110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f352000ea50 tx=0x7f3520005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 shutdown_connections 2022-07-03T10:36:35.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f3518096d20 0x7f35180991d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35300662c0 0x7f3530110510 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3530103250 0x7f3530117ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 --2- 172.21.15.101:0/2594024412 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3530100800 0x7f3530110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 >> 172.21.15.101:0/2594024412 conn(0x7f35300f7c50 msgr2=0x7f35300fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:35.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 shutdown_connections 2022-07-03T10:36:35.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.575+0000 7f35365ff700 1 -- 172.21.15.101:0/2594024412 wait complete. 2022-07-03T10:36:35.604 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:35 smithi161 ceph-mon[91579]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:35 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/375488560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:35 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2466069026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:35 smithi188 ceph-mon[92425]: pgmap v316: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:35 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/375488560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:35 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2466069026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:35.844 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.838+0000 7f5ba1bff700 1 -- 172.21.15.101:0/401362263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 msgr2=0x7f5b9c0fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.838+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/401362263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c0fecb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009900 tx=0x7f5b900091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 -- 172.21.15.101:0/401362263 shutdown_connections 2022-07-03T10:36:35.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/401362263 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c0fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/401362263 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/401362263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c0fecb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 -- 172.21.15.101:0/401362263 >> 172.21.15.101:0/401362263 conn(0x7f5b9c0f8d50 msgr2=0x7f5b9c0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:35.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 -- 172.21.15.101:0/401362263 shutdown_connections 2022-07-03T10:36:35.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.839+0000 7f5ba1bff700 1 -- 172.21.15.101:0/401362263 wait complete. 2022-07-03T10:36:35.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 Processor -- start 2022-07-03T10:36:35.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 -- start start 2022-07-03T10:36:35.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c110d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c118350 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c118490 con 0x7f5b9c0fd5e0 2022-07-03T10:36:35.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.840+0000 7f5ba1bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c118650 con 0x7f5b9c0fe860 2022-07-03T10:36:35.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9affd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c110d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52808/0 (socket says 172.21.15.101:52808) 2022-07-03T10:36:35.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:59072/0 (socket says 172.21.15.101:59072) 2022-07-03T10:36:35.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 -- 172.21.15.101:0/3258614737 learned_addr learned my addr 172.21.15.101:0/3258614737 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:35.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 -- 172.21.15.101:0/3258614737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 msgr2=0x7f5b9c110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c110d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 -- 172.21.15.101:0/3258614737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 msgr2=0x7f5b9c110820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:35.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:35.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 -- 172.21.15.101:0/3258614737 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c009410 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9b7fe700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:35.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.841+0000 7f5b9bfff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400eb80 tx=0x7f5b94009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:35.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.842+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94004a50 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.842+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b90009440 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.842+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5b9400ce80 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.842+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b940056f0 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.842+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c118d30 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.843+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c108df0 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.844+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f5b9400c040 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.846+0000 7f5b83fff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 0x7f5b840991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:35.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.846+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f5b94014070 con 0x7f5b9c0ff1f0 2022-07-03T10:36:35.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.846+0000 7f5b9b7fe700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 0x7f5b840991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:35.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.847+0000 7f5b9b7fe700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 0x7f5b840991d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0fe6c0 tx=0x7f5b8c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:35.861 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:35.849+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f5b9408d480 con 0x7f5b9c0ff1f0 2022-07-03T10:36:36.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.041+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b9c119560 con 0x7f5b9c0ff1f0 2022-07-03T10:36:36.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.041+0000 7f5b83fff700 1 -- 172.21.15.101:0/3258614737 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b94090aa0 con 0x7f5b9c0ff1f0 2022-07-03T10:36:36.044 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:36.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.043+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 msgr2=0x7f5b840991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.043+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 0x7f5b840991d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0fe6c0 tx=0x7f5b8c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.043+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 msgr2=0x7f5b9c117dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.043+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400eb80 tx=0x7f5b94009460 comp rx=0 tx=0).stop 2022-07-03T10:36:36.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 shutdown_connections 2022-07-03T10:36:36.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f5b84096d20 0x7f5b840991d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5b9c0ff1f0 0x7f5b9c117dd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b9c0fd5e0 0x7f5b9c110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 --2- 172.21.15.101:0/3258614737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5b9c0fe860 0x7f5b9c110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 >> 172.21.15.101:0/3258614737 conn(0x7f5b9c0f8d50 msgr2=0x7f5b9c0fb100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:36.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 shutdown_connections 2022-07-03T10:36:36.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.044+0000 7f5ba1bff700 1 -- 172.21.15.101:0/3258614737 wait complete. 2022-07-03T10:36:36.073 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:36:36.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/2847191740 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180ff1c0 msgr2=0x7f7f18105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/2847191740 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180ff1c0 0x7f7f18105a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009ca0 tx=0x7f7f10009520 comp rx=0 tx=0).stop 2022-07-03T10:36:36.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/2847191740 shutdown_connections 2022-07-03T10:36:36.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/2847191740 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180ff1c0 0x7f7f18105a80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/2847191740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f180fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/2847191740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180fe830 0x7f7f180fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/2847191740 >> 172.21.15.101:0/2847191740 conn(0x7f7f180f8d00 msgr2=0x7f7f180fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:36.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/2847191740 shutdown_connections 2022-07-03T10:36:36.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.298+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/2847191740 wait complete. 2022-07-03T10:36:36.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.299+0000 7f7f1d7ff700 1 Processor -- start 2022-07-03T10:36:36.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.299+0000 7f7f1d7ff700 1 -- start start 2022-07-03T10:36:36.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.299+0000 7f7f1d7ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f18072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f1d7ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f1d7ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 0x7f7f18070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f1d7ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18071480 con 0x7f7f180fe830 2022-07-03T10:36:36.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f1d7ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180715f0 con 0x7f7f180fd5b0 2022-07-03T10:36:36.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f1d7ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18118360 con 0x7f7f180ff1c0 2022-07-03T10:36:36.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f167fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f167fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52822/0 (socket says 172.21.15.101:52822) 2022-07-03T10:36:36.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f177fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 0x7f7f18070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f18072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f167fc700 1 -- 172.21.15.101:0/3216252023 learned_addr learned my addr 172.21.15.101:0/3216252023 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:36.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 -- 172.21.15.101:0/3216252023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 msgr2=0x7f7f18070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 0x7f7f18070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 -- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 msgr2=0x7f7f18070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f16ffd700 1 -- 172.21.15.101:0/3216252023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08009410 con 0x7f7f180fd5b0 2022-07-03T10:36:36.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f177fe700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 0x7f7f18070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:36.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.300+0000 7f7f167fc700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:36.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7f16ffd700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f18072230 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800ea50 tx=0x7f7f08005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:36.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08009650 con 0x7f7f180fd5b0 2022-07-03T10:36:36.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f100097e0 con 0x7f7f180fd5b0 2022-07-03T10:36:36.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7f0800cc70 con 0x7f7f180fd5b0 2022-07-03T10:36:36.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08010ea0 con 0x7f7f180fd5b0 2022-07-03T10:36:36.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.301+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f181189c0 con 0x7f7f180fd5b0 2022-07-03T10:36:36.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.302+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18108d90 con 0x7f7f180fd5b0 2022-07-03T10:36:36.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.304+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f7f0800c040 con 0x7f7f180fd5b0 2022-07-03T10:36:36.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.305+0000 7f7eff7fe700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 0x7f7f000991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.305+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7f08014070 con 0x7f7f180fd5b0 2022-07-03T10:36:36.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.305+0000 7f7f167fc700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 0x7f7f000991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.306+0000 7f7f167fc700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 0x7f7f000991d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c011640 tx=0x7f7f0c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:36.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.309+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f7f0808cd90 con 0x7f7f180fd5b0 2022-07-03T10:36:36.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2594024412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:36 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3258614737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.540 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.538+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f18108f20 con 0x7f7f180fd5b0 2022-07-03T10:36:36.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.539+0000 7f7eff7fe700 1 -- 172.21.15.101:0/3216252023 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7f7f18108f20 con 0x7f7f180fd5b0 2022-07-03T10:36:36.542 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:36.543 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:36:36.546 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 msgr2=0x7f7f000991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 0x7f7f000991d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c011640 tx=0x7f7f0c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 msgr2=0x7f7f18072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f18072230 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800ea50 tx=0x7f7f08005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 shutdown_connections 2022-07-03T10:36:36.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7f00096d20 0x7f7f000991d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7f180ff1c0 0x7f7f18070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7f180fd5b0 0x7f7f18072230 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 --2- 172.21.15.101:0/3216252023 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7f180fe830 0x7f7f18070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.548 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 >> 172.21.15.101:0/3216252023 conn(0x7f7f180f8d00 msgr2=0x7f7f180fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:36.549 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 shutdown_connections 2022-07-03T10:36:36.549 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.542+0000 7f7f1d7ff700 1 -- 172.21.15.101:0/3216252023 wait complete. 2022-07-03T10:36:36.549 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:36.572 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2594024412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:36 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3258614737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2594024412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:36 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3258614737' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:36.798 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 -- 172.21.15.101:0/1309804775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0ff1c0 msgr2=0x7f696c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 --2- 172.21.15.101:0/1309804775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0ff1c0 0x7f696c105a80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6964009bf0 tx=0x7f6964009460 comp rx=0 tx=0).stop 2022-07-03T10:36:36.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 -- 172.21.15.101:0/1309804775 shutdown_connections 2022-07-03T10:36:36.799 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 --2- 172.21.15.101:0/1309804775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0ff1c0 0x7f696c105a80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.800 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 --2- 172.21.15.101:0/1309804775 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fd5b0 0x7f696c0fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.800 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 --2- 172.21.15.101:0/1309804775 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0fe830 0x7f696c0fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.800 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 -- 172.21.15.101:0/1309804775 >> 172.21.15.101:0/1309804775 conn(0x7f696c0f8d00 msgr2=0x7f696c0fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:36.800 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 -- 172.21.15.101:0/1309804775 shutdown_connections 2022-07-03T10:36:36.800 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.796+0000 7f69735ff700 1 -- 172.21.15.101:0/1309804775 wait complete. 2022-07-03T10:36:36.804 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.797+0000 7f69735ff700 1 Processor -- start 2022-07-03T10:36:36.804 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 -- start start 2022-07-03T10:36:36.804 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0fd5b0 0x7f696c110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.805 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.805 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0ff1c0 0x7f696c117b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.805 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1180c0 con 0x7f696c0ff1c0 2022-07-03T10:36:36.805 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c118200 con 0x7f696c0fd5b0 2022-07-03T10:36:36.805 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f69735ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1183c0 con 0x7f696c0fe830 2022-07-03T10:36:36.806 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f6970b9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.806 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f697139b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0fd5b0 0x7f696c110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.806 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f6970b9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48064/0 (socket says 172.21.15.101:48064) 2022-07-03T10:36:36.806 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f6970b9a700 1 -- 172.21.15.101:0/3109542359 learned_addr learned my addr 172.21.15.101:0/3109542359 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:36.807 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.798+0000 7f6971b9c700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0ff1c0 0x7f696c117b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.807 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 -- 172.21.15.101:0/3109542359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0fd5b0 msgr2=0x7f696c110590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.807 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0fd5b0 0x7f696c110590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.807 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 -- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0ff1c0 msgr2=0x7f696c117b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:36.808 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0ff1c0 0x7f696c117b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:36.808 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 -- 172.21.15.101:0/3109542359 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964009700 con 0x7f696c0fe830 2022-07-03T10:36:36.808 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6970b9a700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f695c00ea50 tx=0x7f695c0091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:36.808 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c005020 con 0x7f696c0fe830 2022-07-03T10:36:36.809 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1186a0 con 0x7f696c0fe830 2022-07-03T10:36:36.809 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f695c00c740 con 0x7f696c0fe830 2022-07-03T10:36:36.809 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c004920 con 0x7f696c0fe830 2022-07-03T10:36:36.809 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.799+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c118c80 con 0x7f696c0fe830 2022-07-03T10:36:36.809 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.800+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c108db0 con 0x7f696c0fe830 2022-07-03T10:36:36.810 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.802+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f695c00c040 con 0x7f696c0fe830 2022-07-03T10:36:36.811 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.803+0000 7f6961ffb700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 0x7f69580991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:36.811 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.804+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f695c014070 con 0x7f696c0fe830 2022-07-03T10:36:36.811 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.804+0000 7f697139b700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 0x7f69580991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:36.811 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.804+0000 7f697139b700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 0x7f69580991d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f696c0fe690 tx=0x7f6968005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:36.812 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:36.808+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f695c08d070 con 0x7f696c0fe830 2022-07-03T10:36:37.001 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.000+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f696c119320 con 0x7f696c0fe830 2022-07-03T10:36:37.003 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.001+0000 7f6961ffb700 1 -- 172.21.15.101:0/3109542359 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7f695c090690 con 0x7f696c0fe830 2022-07-03T10:36:37.004 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:37.004 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:37.007 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 msgr2=0x7f69580991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.008 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 0x7f69580991d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f696c0fe690 tx=0x7f6968005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.008 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 msgr2=0x7f696c110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.008 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f695c00ea50 tx=0x7f695c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.008 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 shutdown_connections 2022-07-03T10:36:37.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6958096d20 0x7f69580991d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f696c0ff1c0 0x7f696c117b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f696c0fd5b0 0x7f696c110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.009 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 --2- 172.21.15.101:0/3109542359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f696c0fe830 0x7f696c110ad0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 >> 172.21.15.101:0/3109542359 conn(0x7f696c0f8d00 msgr2=0x7f696c104070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 shutdown_connections 2022-07-03T10:36:37.010 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.004+0000 7f69735ff700 1 -- 172.21.15.101:0/3109542359 wait complete. 2022-07-03T10:36:37.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-07-03T10:35:03.427264+0000', 'modified': '2022-07-03T10:35:08.208608+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24569}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24557': {'gid': 24557, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.188:6837/639939509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6836', 'nonce': 639939509}, {'type': 'v1', 'addr': '172.21.15.188:6837', 'nonce': 639939509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.188:6839/204174115', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.188:6838', 'nonce': 204174115}, {'type': 'v1', 'addr': '172.21.15.188:6839', 'nonce': 204174115}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-03T10:36:37.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-03T10:36:37.034 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-07-03T10:36:37.263 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 -- 172.21.15.101:0/886666076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fe860 msgr2=0x7f8cbc0fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.263 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/886666076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc0fecb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009860 tx=0x7f8cac005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.263 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 -- 172.21.15.101:0/886666076 shutdown_connections 2022-07-03T10:36:37.263 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/886666076 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc0fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/886666076 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/886666076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc0fecb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 -- 172.21.15.101:0/886666076 >> 172.21.15.101:0/886666076 conn(0x7f8cbc0f8d90 msgr2=0x7f8cbc0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.264 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 -- 172.21.15.101:0/886666076 shutdown_connections 2022-07-03T10:36:37.265 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.260+0000 7f8cc3bff700 1 -- 172.21.15.101:0/886666076 wait complete. 2022-07-03T10:36:37.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.261+0000 7f8cc3bff700 1 Processor -- start 2022-07-03T10:36:37.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 -- start start 2022-07-03T10:36:37.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.267 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.268 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1160e0 con 0x7f8cbc0ff1f0 2022-07-03T10:36:37.268 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc116220 con 0x7f8cbc0fd5e0 2022-07-03T10:36:37.268 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc3bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1163e0 con 0x7f8cbc0fe860 2022-07-03T10:36:37.268 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc219c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:59132/0 (socket says 172.21.15.101:59132) 2022-07-03T10:36:37.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.262+0000 7f8cc199b700 1 -- 172.21.15.101:0/2618441349 learned_addr learned my addr 172.21.15.101:0/2618441349 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:37.269 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 -- 172.21.15.101:0/2618441349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 msgr2=0x7f8cbc10e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.270 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.270 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 -- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 msgr2=0x7f8cbc115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.270 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.270 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 -- 172.21.15.101:0/2618441349 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb8009410 con 0x7f8cbc0fe860 2022-07-03T10:36:37.271 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc219c700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:37.271 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:37.271 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.263+0000 7f8cc119a700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc10eaf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0093e0 tx=0x7f8cac005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.272 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.267+0000 7f8cb27fc700 1 -- 172.21.15.101:0/2618441349 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac01d070 con 0x7f8cbc0fe860 2022-07-03T10:36:37.273 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.268+0000 7f8cb27fc700 1 -- 172.21.15.101:0/2618441349 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8cac005590 con 0x7f8cbc0fe860 2022-07-03T10:36:37.273 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.268+0000 7f8cb27fc700 1 -- 172.21.15.101:0/2618441349 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac017bf0 con 0x7f8cbc0fe860 2022-07-03T10:36:37.273 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.268+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb40097e0 con 0x7f8cbc0fe860 2022-07-03T10:36:37.274 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.268+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac009410 con 0x7f8cbc0fe860 2022-07-03T10:36:37.274 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.271+0000 7f8cb27fc700 1 -- 172.21.15.101:0/2618441349 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f8cac028020 con 0x7f8cbc0fe860 2022-07-03T10:36:37.276 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.272+0000 7f8cb27fc700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 0x7f8ca8099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.277 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.272+0000 7f8cd7bb5b80 1 Processor -- start 2022-07-03T10:36:37.277 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.272+0000 7f8cb27fc700 1 -- 172.21.15.101:0/2618441349 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f8cac0c0d90 con 0x7f8cbc0fe860 2022-07-03T10:36:37.277 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.272+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 0x7f8ca8099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.277 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 -- start start 2022-07-03T10:36:37.277 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 0x563e9538afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.278 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.278 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e9538c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.278 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e952731c0 con 0x563e9538b520 2022-07-03T10:36:37.278 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e952dccb0 con 0x563e9538abd0 2022-07-03T10:36:37.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e952d64b0 con 0x563e9538c460 2022-07-03T10:36:37.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 0x7f8ca8099250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009fd0 tx=0x7f8cb8005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.279 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52874/0 (socket says 172.21.15.101:52874) 2022-07-03T10:36:37.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc219c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e9538c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc119a700 1 -- 172.21.15.101:0/460116108 learned_addr learned my addr 172.21.15.101:0/460116108 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:37.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.273+0000 7f8cc199b700 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 0x563e9538afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.280 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 -- 172.21.15.101:0/460116108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 msgr2=0x563e9538c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e9538c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 -- 172.21.15.101:0/460116108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 msgr2=0x563e9538be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.281 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 -- 172.21.15.101:0/460116108 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e9515f360 con 0x563e9538abd0 2022-07-03T10:36:37.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc119a700 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:37.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8cc199b700 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 0x563e9538afe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8011ea0 tx=0x7f8cb8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b20d458e428b345 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8c9f1fe700 1 -- 172.21.15.101:0/460116108 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb801c050 con 0x563e9538abd0 2022-07-03T10:36:37.282 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8c9f1fe700 1 -- 172.21.15.101:0/460116108 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8cb801c1b0 con 0x563e9538abd0 2022-07-03T10:36:37.283 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.274+0000 7f8c9f1fe700 1 -- 172.21.15.101:0/460116108 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8003d10 con 0x563e9538abd0 2022-07-03T10:36:37.288 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/460116108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 msgr2=0x563e9538afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.288 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 0x563e9538afe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8011ea0 tx=0x7f8cb8019000 comp rx=0 tx=0).stop 2022-07-03T10:36:37.288 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/460116108 shutdown_connections 2022-07-03T10:36:37.288 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e9538c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.289 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538abd0 0x563e9538afe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.289 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/460116108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538b520 0x563e9538be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.289 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/460116108 >> 172.21.15.101:0/460116108 conn(0x563e9537c7b0 msgr2=0x563e95383140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.289 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/460116108 shutdown_connections 2022-07-03T10:36:37.290 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.275+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/460116108 wait complete. 2022-07-03T10:36:37.290 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.276+0000 7f8cd7bb5b80 1 -- start start 2022-07-03T10:36:37.290 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.276+0000 7f8cd7bb5b80 1 Processor -- start 2022-07-03T10:36:37.290 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.291 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e95394ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.291 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538c460 0x563e9539bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.291 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e951c4e40 con 0x563e9538c460 2022-07-03T10:36:37.291 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e9539c0c0 con 0x563e9538b520 2022-07-03T10:36:37.291 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e9539c280 con 0x563e9538abd0 2022-07-03T10:36:37.292 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cd7bb5b80 10 client.0 _ll_register_callbacks cb 0x563e952821a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:37.292 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.292 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48104/0 (socket says 172.21.15.101:48104) 2022-07-03T10:36:37.292 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 -- 172.21.15.101:0/546401661 learned_addr learned my addr 172.21.15.101:0/546401661 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:37.293 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc219c700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538c460 0x563e9539bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.293 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc119a700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e95394ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.293 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 -- 172.21.15.101:0/546401661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 msgr2=0x563e95394ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.293 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e95394ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.293 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 -- 172.21.15.101:0/546401661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538c460 msgr2=0x563e9539bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.294 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538c460 0x563e9539bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.294 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc199b700 1 -- 172.21.15.101:0/546401661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac009410 con 0x563e9538abd0 2022-07-03T10:36:37.294 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.277+0000 7f8cc119a700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e95394ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:37.294 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8cc199b700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80039d0 tx=0x7f8cb8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d879cb48e63e8ca4 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.295 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8c9c9f9700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:36:37.295 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8005330 con 0x563e9538abd0 2022-07-03T10:36:37.295 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e95379f70 con 0x563e9538abd0 2022-07-03T10:36:37.295 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f8cb8005490 con 0x563e9538abd0 2022-07-03T10:36:37.295 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb8003fc0 con 0x563e9538abd0 2022-07-03T10:36:37.296 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8cd7bb5b80 10 client.25046 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:37.296 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.278+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7f8cb8017050 con 0x563e9538abd0 2022-07-03T10:36:37.296 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x563e95379f70 con 0x563e9538abd0 2022-07-03T10:36:37.297 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f8cb801b610 con 0x563e9538abd0 2022-07-03T10:36:37.297 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 10 client.25046 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:37.297 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 10 client.25046 resolve_mds: resolved name 'l' to daemon mds.l 2022-07-03T10:36:37.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 20 client.25046 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:37.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e9539f740 0x563e953a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.298 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 4 client.25046 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:37.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e952cfa20 con 0x563e9539f740 2022-07-03T10:36:37.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.279+0000 7f8cc119a700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e9539f740 0x563e953a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.280+0000 7f8cc119a700 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e9539f740 0x563e953a4a20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3c165048c9bda16 server_cookie=4c51ad825b67441b in_seq=0 out_seq=0 2022-07-03T10:36:37.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.280+0000 7f8c9c9f9700 10 client.25046 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:37.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/546401661 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x563e952cfa20 con 0x563e9539f740 2022-07-03T10:36:37.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8c9c9f9700 10 client.25046 handle_command_reply: tid=0 2022-07-03T10:36:37.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 client.25046 shutdown 2022-07-03T10:36:37.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 msgr2=0x563e95394590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80039d0 tx=0x7f8cb8019000 comp rx=0 tx=0).stop 2022-07-03T10:36:37.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 shutdown_connections 2022-07-03T10:36:37.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e9539f740 0x563e953a4a20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538c460 0x563e9539bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538abd0 0x563e95394590 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/546401661 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e95394ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8c9c9f9700 0 client.25046 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:37.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 >> 172.21.15.101:0/546401661 conn(0x563e9537c7b0 msgr2=0x563e95382d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 shutdown_connections 2022-07-03T10:36:37.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.281+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/546401661 wait complete. 2022-07-03T10:36:37.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.282+0000 7f8cd7bb5b80 20 client.25046 trim_cache size 0 max 16384 2022-07-03T10:36:37.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.299+0000 7f8cd7bb5b80 1 Processor -- start 2022-07-03T10:36:37.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.299+0000 7f8cd7bb5b80 1 -- start start 2022-07-03T10:36:37.312 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e9539c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e95392430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e952cfa20 con 0x563e9538abd0 2022-07-03T10:36:37.313 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e9529a460 con 0x563e9538b520 2022-07-03T10:36:37.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e95278fd0 con 0x563e9538c460 2022-07-03T10:36:37.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e9539c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.314 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc219c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e95392430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc219c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e95392430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:48108/0 (socket says 172.21.15.101:48108) 2022-07-03T10:36:37.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc219c700 1 -- 172.21.15.101:0/3284215061 learned_addr learned my addr 172.21.15.101:0/3284215061 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:37.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:59146/0 (socket says 172.21.15.101:59146) 2022-07-03T10:36:37.315 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.300+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e9539c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52892/0 (socket says 172.21.15.101:52892) 2022-07-03T10:36:37.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 -- 172.21.15.101:0/3284215061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 msgr2=0x563e95392430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e95392430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 -- 172.21.15.101:0/3284215061 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 msgr2=0x563e9539c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.316 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e9539c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 -- 172.21.15.101:0/3284215061 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e9510bb10 con 0x563e9538b520 2022-07-03T10:36:37.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8cc119a700 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8cac08e190 tx=0x7f8cac024040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f236d027105feadc server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8c9f9ff700 1 -- 172.21.15.101:0/3284215061 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac016e80 con 0x563e9538b520 2022-07-03T10:36:37.317 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8c9f9ff700 1 -- 172.21.15.101:0/3284215061 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8cac0917d0 con 0x563e9538b520 2022-07-03T10:36:37.318 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.301+0000 7f8c9f9ff700 1 -- 172.21.15.101:0/3284215061 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac090030 con 0x563e9538b520 2022-07-03T10:36:37.318 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/3284215061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 msgr2=0x563e9538f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.318 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8cac08e190 tx=0x7f8cac024040 comp rx=0 tx=0).stop 2022-07-03T10:36:37.318 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/3284215061 shutdown_connections 2022-07-03T10:36:37.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e9539c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538c460 0x563e95392430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/3284215061 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538b520 0x563e9538f070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.319 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/3284215061 >> 172.21.15.101:0/3284215061 conn(0x563e9537c7b0 msgr2=0x563e953a8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/3284215061 shutdown_connections 2022-07-03T10:36:37.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/3284215061 wait complete. 2022-07-03T10:36:37.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.302+0000 7f8cd7bb5b80 1 -- start start 2022-07-03T10:36:37.320 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 Processor -- start 2022-07-03T10:36:37.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538b520 0x563e95398da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538c460 0x563e953f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.321 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e95399490 con 0x563e9538abd0 2022-07-03T10:36:37.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e953f1f00 con 0x563e9538c460 2022-07-03T10:36:37.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.303+0000 7f8cd7bb5b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e953f20c0 con 0x563e9538b520 2022-07-03T10:36:37.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cd7bb5b80 10 client.0 _ll_register_callbacks cb 0x563e952821a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:37.322 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.323 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc119a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538b520 0x563e95398da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.323 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:52906/0 (socket says 172.21.15.101:52906) 2022-07-03T10:36:37.323 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 -- 172.21.15.101:0/2223608613 learned_addr learned my addr 172.21.15.101:0/2223608613 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:37.323 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc219c700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538c460 0x563e953f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.324 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 -- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538b520 msgr2=0x563e95398da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.324 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538b520 0x563e95398da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.324 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 -- 172.21.15.101:0/2223608613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538c460 msgr2=0x563e953f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:37.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538c460 0x563e953f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 -- 172.21.15.101:0/2223608613 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb40097e0 con 0x563e9538abd0 2022-07-03T10:36:37.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8cc199b700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8005cd0 tx=0x7f8cb801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bfdd22999feef3 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:37.325 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.304+0000 7f8c9c9f9700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:36:37.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8005330 con 0x563e9538abd0 2022-07-03T10:36:37.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cb801c1b0 con 0x563e9538abd0 2022-07-03T10:36:37.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f8cb8005490 con 0x563e9538abd0 2022-07-03T10:36:37.326 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb80041d0 con 0x563e9538abd0 2022-07-03T10:36:37.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8cd7bb5b80 10 client.14988 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:37.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563e95109fe0 con 0x563e9538abd0 2022-07-03T10:36:37.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.305+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8cb801c1b0 con 0x563e9538abd0 2022-07-03T10:36:37.327 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f8cb802f440 con 0x563e9538abd0 2022-07-03T10:36:37.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 10 client.14988 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:37.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 10 client.14988 resolve_mds: resolved name 'l' to daemon mds.l 2022-07-03T10:36:37.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 20 client.14988 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:37.328 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e953f6a90 0x563e953f3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:37.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 4 client.14988 mds_command: new command op to 24569 tid=0[{"prefix": "status"}] 2022-07-03T10:36:37.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563e952cf800 con 0x563e953f6a90 2022-07-03T10:36:37.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cc119a700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e953f6a90 0x563e953f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:37.329 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8cc119a700 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e953f6a90 0x563e953f3400 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c2c7ddae945de0b server_cookie=f5895d9ec4a13af5 in_seq=0 out_seq=0 2022-07-03T10:36:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.306+0000 7f8c9c9f9700 10 client.14988 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8c9c9f9700 1 -- 172.21.15.101:0/2223608613 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x563e952cf800 con 0x563e953f6a90 2022-07-03T10:36:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8c9c9f9700 10 client.14988 handle_command_reply: tid=0 2022-07-03T10:36:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 client.14988 shutdown 2022-07-03T10:36:37.330 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 msgr2=0x563e95398860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8005cd0 tx=0x7f8cb801e040 comp rx=0 tx=0).stop 2022-07-03T10:36:37.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 shutdown_connections 2022-07-03T10:36:37.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x563e953f6a90 0x563e953f3400 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.331 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x563e9538abd0 0x563e95398860 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563e9538c460 0x563e953f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8c9c9f9700 0 client.14988 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 --2- 172.21.15.101:0/2223608613 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x563e9538b520 0x563e95398da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 >> 172.21.15.101:0/2223608613 conn(0x563e9537c7b0 msgr2=0x563e9538f070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 shutdown_connections 2022-07-03T10:36:37.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 1 -- 172.21.15.101:0/2223608613 wait complete. 2022-07-03T10:36:37.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.307+0000 7f8cd7bb5b80 20 client.14988 trim_cache size 0 max 16384 2022-07-03T10:36:37.334 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:37.334 INFO:teuthology.orchestra.run.smithi101.stdout: "cluster_fsid": "23a851ae-fab9-11ec-842c-001a4aab830c", 2022-07-03T10:36:37.334 INFO:teuthology.orchestra.run.smithi101.stdout: "whoami": 0, 2022-07-03T10:36:37.334 INFO:teuthology.orchestra.run.smithi101.stdout: "id": 24569, 2022-07-03T10:36:37.335 INFO:teuthology.orchestra.run.smithi101.stdout: "want_state": "up:active", 2022-07-03T10:36:37.335 INFO:teuthology.orchestra.run.smithi101.stdout: "state": "up:active", 2022-07-03T10:36:37.335 INFO:teuthology.orchestra.run.smithi101.stdout: "fs_name": "cephfs", 2022-07-03T10:36:37.335 INFO:teuthology.orchestra.run.smithi101.stdout: "rank_uptime": 93.877610270000005, 2022-07-03T10:36:37.336 INFO:teuthology.orchestra.run.smithi101.stdout: "mdsmap_epoch": 20, 2022-07-03T10:36:37.336 INFO:teuthology.orchestra.run.smithi101.stdout: "osdmap_epoch": 82, 2022-07-03T10:36:37.336 INFO:teuthology.orchestra.run.smithi101.stdout: "osdmap_epoch_barrier": 80, 2022-07-03T10:36:37.336 INFO:teuthology.orchestra.run.smithi101.stdout: "uptime": 99.53676179 2022-07-03T10:36:37.337 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:37.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 msgr2=0x7f8ca8099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 0x7f8ca8099250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009fd0 tx=0x7f8cb8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 msgr2=0x7f8cbc10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc10eaf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0093e0 tx=0x7f8cac005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 shutdown_connections 2022-07-03T10:36:37.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f8ca8096da0 0x7f8ca8099250 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8cbc0ff1f0 0x7f8cbc115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8cbc0fd5e0 0x7f8cbc10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 --2- 172.21.15.101:0/2618441349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8cbc0fe860 0x7f8cbc10eaf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:37.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 >> 172.21.15.101:0/2618441349 conn(0x7f8cbc0f8d90 msgr2=0x7f8cbc101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:37.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 shutdown_connections 2022-07-03T10:36:37.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:37.308+0000 7f8cc3bff700 1 -- 172.21.15.101:0/2618441349 wait complete. 2022-07-03T10:36:37.341 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:37 smithi101 ceph-mon[85126]: pgmap v317: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:37.342 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:37 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3216252023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:37.342 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:37 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3109542359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:37.343 INFO:teuthology.run_tasks:Running task kclient... 2022-07-03T10:36:37.368 INFO:tasks.kclient:Mounting kernel clients... 2022-07-03T10:36:37.369 INFO:tasks.kclient:config is {'client.0': {}} 2022-07-03T10:36:37.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:37.371 DEBUG:teuthology.orchestra.run.smithi101:> ip netns list 2022-07-03T10:36:37.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:37.432 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link delete ceph-brx 2022-07-03T10:36:37.523 INFO:teuthology.orchestra.run.smithi101.stderr:Cannot find device "ceph-brx" 2022-07-03T10:36:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-03T10:36:37.529 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2022-07-03T10:36:37.529 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph fs ls 2022-07-03T10:36:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:37 smithi161 ceph-mon[91579]: pgmap v317: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:37 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3216252023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:37 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3109542359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:37 smithi188 ceph-mon[92425]: pgmap v317: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:37 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3216252023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:37 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3109542359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:38.411 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:36:39.331 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:39 smithi101 ceph-mon[85126]: pgmap v318: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:39 smithi161 ceph-mon[91579]: pgmap v318: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:39 smithi188 ceph-mon[92425]: pgmap v318: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:39.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 -- 172.21.15.101:0/1175989737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c0fddc0 msgr2=0x7ff56c10a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:39.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1175989737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c0fddc0 0x7ff56c10a4b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff564009ca0 tx=0x7ff564009520 comp rx=0 tx=0).stop 2022-07-03T10:36:39.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 -- 172.21.15.101:0/1175989737 shutdown_connections 2022-07-03T10:36:39.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1175989737 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c0fddc0 0x7ff56c10a4b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:39.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1175989737 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:39.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1175989737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c104050 0x7ff56c0fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:39.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 -- 172.21.15.101:0/1175989737 >> 172.21.15.101:0/1175989737 conn(0x7ff56c0f9220 msgr2=0x7ff56c0fb630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:39.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 -- 172.21.15.101:0/1175989737 shutdown_connections 2022-07-03T10:36:39.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.968+0000 7ff571d9e700 1 -- 172.21.15.101:0/1175989737 wait complete. 2022-07-03T10:36:39.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.969+0000 7ff571d9e700 1 Processor -- start 2022-07-03T10:36:39.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.969+0000 7ff571d9e700 1 -- start start 2022-07-03T10:36:39.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c0fddc0 0x7ff56c116520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:39.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:39.975 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c104050 0x7ff56c1105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:39.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c110ba0 con 0x7ff56c104050 2022-07-03T10:36:39.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c110d10 con 0x7ff56c1036b0 2022-07-03T10:36:39.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff571d9e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56c110ed0 con 0x7ff56c0fddc0 2022-07-03T10:36:39.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff56affd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:39.976 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff56affd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51684/0 (socket says 172.21.15.101:51684) 2022-07-03T10:36:39.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff56affd700 1 -- 172.21.15.101:0/1315820664 learned_addr learned my addr 172.21.15.101:0/1315820664 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:39.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff56b7fe700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c0fddc0 0x7ff56c116520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:39.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.970+0000 7ff56bfff700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c104050 0x7ff56c1105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:39.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 -- 172.21.15.101:0/1315820664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c0fddc0 msgr2=0x7ff56c116520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:39.977 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c0fddc0 0x7ff56c116520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:39.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 -- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c104050 msgr2=0x7ff56c1105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:39.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c104050 0x7ff56c1105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:39.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 -- 172.21.15.101:0/1315820664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5640097e0 con 0x7ff56c1036b0 2022-07-03T10:36:39.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff56affd700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff56000ea80 tx=0x7ff560005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560009e10 con 0x7ff56c1036b0 2022-07-03T10:36:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff56000ce80 con 0x7ff56c1036b0 2022-07-03T10:36:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c1111b0 con 0x7ff56c1036b0 2022-07-03T10:36:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.971+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff560010e70 con 0x7ff56c1036b0 2022-07-03T10:36:39.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.972+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c11d4e0 con 0x7ff56c1036b0 2022-07-03T10:36:39.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.972+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff56c04ccb0 con 0x7ff56c1036b0 2022-07-03T10:36:39.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.974+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7ff56000c040 con 0x7ff56c1036b0 2022-07-03T10:36:39.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.976+0000 7ff553fff700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 0x7ff5540991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:39.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.976+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff560014070 con 0x7ff56c1036b0 2022-07-03T10:36:39.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.976+0000 7ff56b7fe700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 0x7ff5540991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:39.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.977+0000 7ff56b7fe700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 0x7ff5540991d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009d00 tx=0x7ff55c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:39.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:39.978+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7ff56008d4f0 con 0x7ff56c1036b0 2022-07-03T10:36:40.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.209+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff56c10d450 con 0x7ff56c1036b0 2022-07-03T10:36:40.212 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.209+0000 7ff553fff700 1 -- 172.21.15.101:0/1315820664 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff560026020 con 0x7ff56c1036b0 2022-07-03T10:36:40.212 INFO:teuthology.orchestra.run.smithi101.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2022-07-03T10:36:40.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 msgr2=0x7ff5540991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:40.214 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 0x7ff5540991d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009d00 tx=0x7ff55c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 msgr2=0x7ff56c116a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:40.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff56000ea80 tx=0x7ff560005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 shutdown_connections 2022-07-03T10:36:40.215 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff554096d20 0x7ff5540991d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff56c104050 0x7ff56c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff56c0fddc0 0x7ff56c116520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 --2- 172.21.15.101:0/1315820664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff56c1036b0 0x7ff56c116a60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:40.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 >> 172.21.15.101:0/1315820664 conn(0x7ff56c0f9220 msgr2=0x7ff56c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:40.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 shutdown_connections 2022-07-03T10:36:40.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:40.212+0000 7ff571d9e700 1 -- 172.21.15.101:0/1315820664 wait complete. 2022-07-03T10:36:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:40 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1315820664' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-07-03T10:36:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:40 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1315820664' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-07-03T10:36:40.671 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:40 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1315820664' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-07-03T10:36:40.695 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-07-03T10:36:40.695 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-07-03T10:36:40.696 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi101.front.sepia.ceph.com 2022-07-03T10:36:40.696 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-07-03T10:36:40.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:40.697 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-07-03T10:36:40.697 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-07-03T10:36:40.697 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-07-03T10:36:40.698 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-07-03T10:36:40.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:40.700 DEBUG:teuthology.orchestra.run.smithi101:> ip addr 2022-07-03T10:36:40.760 INFO:teuthology.orchestra.run.smithi101.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-07-03T10:36:40.761 INFO:teuthology.orchestra.run.smithi101.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-07-03T10:36:40.761 INFO:teuthology.orchestra.run.smithi101.stdout: inet 127.0.0.1/8 scope host lo 2022-07-03T10:36:40.762 INFO:teuthology.orchestra.run.smithi101.stdout: valid_lft forever preferred_lft forever 2022-07-03T10:36:40.762 INFO:teuthology.orchestra.run.smithi101.stdout: inet6 ::1/128 scope host 2022-07-03T10:36:40.762 INFO:teuthology.orchestra.run.smithi101.stdout: valid_lft forever preferred_lft forever 2022-07-03T10:36:40.762 INFO:teuthology.orchestra.run.smithi101.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-07-03T10:36:40.763 INFO:teuthology.orchestra.run.smithi101.stdout: link/ether 0c:c4:7a:d8:a1:f2 brd ff:ff:ff:ff:ff:ff 2022-07-03T10:36:40.763 INFO:teuthology.orchestra.run.smithi101.stdout: altname enp7s0f0 2022-07-03T10:36:40.763 INFO:teuthology.orchestra.run.smithi101.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-07-03T10:36:40.763 INFO:teuthology.orchestra.run.smithi101.stdout: link/ether 0c:c4:7a:d8:a1:f3 brd ff:ff:ff:ff:ff:ff 2022-07-03T10:36:40.764 INFO:teuthology.orchestra.run.smithi101.stdout: altname enp7s0f1 2022-07-03T10:36:40.764 INFO:teuthology.orchestra.run.smithi101.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-07-03T10:36:40.764 INFO:teuthology.orchestra.run.smithi101.stdout: link/ether 0c:c4:7a:8f:d7:22 brd ff:ff:ff:ff:ff:ff 2022-07-03T10:36:40.764 INFO:teuthology.orchestra.run.smithi101.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-07-03T10:36:40.765 INFO:teuthology.orchestra.run.smithi101.stdout: link/ether 0c:c4:7a:8f:d7:23 brd ff:ff:ff:ff:ff:ff 2022-07-03T10:36:40.765 INFO:teuthology.orchestra.run.smithi101.stdout: inet 172.21.15.101/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-07-03T10:36:40.765 INFO:teuthology.orchestra.run.smithi101.stdout: valid_lft 41116sec preferred_lft 41116sec 2022-07-03T10:36:40.765 INFO:teuthology.orchestra.run.smithi101.stdout: inet6 fe80::ec4:7aff:fe8f:d723/64 scope link 2022-07-03T10:36:40.765 INFO:teuthology.orchestra.run.smithi101.stdout: valid_lft forever preferred_lft forever 2022-07-03T10:36:40.767 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-07-03T10:36:40.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:40.768 DEBUG:teuthology.orchestra.run.smithi101:> (cd / && exec bash -c ' 2022-07-03T10:36:40.768 DEBUG:teuthology.orchestra.run.smithi101:> set -e 2022-07-03T10:36:40.769 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link add name ceph-brx type bridge 2022-07-03T10:36:40.769 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip addr flush dev ceph-brx 2022-07-03T10:36:40.769 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link set ceph-brx up 2022-07-03T10:36:40.770 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-07-03T10:36:40.770 DEBUG:teuthology.orchestra.run.smithi101:> ') 2022-07-03T10:36:41.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:41.261 DEBUG:teuthology.orchestra.run.smithi101:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-07-03T10:36:41.351 INFO:teuthology.orchestra.run.smithi101.stdout:1 2022-07-03T10:36:41.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:41.356 DEBUG:teuthology.orchestra.run.smithi101:> route 2022-07-03T10:36:41.443 INFO:teuthology.orchestra.run.smithi101.stdout:Kernel IP routing table 2022-07-03T10:36:41.443 INFO:teuthology.orchestra.run.smithi101.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-07-03T10:36:41.444 INFO:teuthology.orchestra.run.smithi101.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-07-03T10:36:41.444 INFO:teuthology.orchestra.run.smithi101.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-07-03T10:36:41.445 INFO:teuthology.orchestra.run.smithi101.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-07-03T10:36:41.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:41.448 DEBUG:teuthology.orchestra.run.smithi101:> (cd / && exec bash -c ' 2022-07-03T10:36:41.448 DEBUG:teuthology.orchestra.run.smithi101:> set -e 2022-07-03T10:36:41.449 DEBUG:teuthology.orchestra.run.smithi101:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-07-03T10:36:41.449 DEBUG:teuthology.orchestra.run.smithi101:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-07-03T10:36:41.449 DEBUG:teuthology.orchestra.run.smithi101:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-07-03T10:36:41.450 DEBUG:teuthology.orchestra.run.smithi101:> ') 2022-07-03T10:36:41.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:41 smithi101 ceph-mon[85126]: pgmap v319: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:41 smithi161 ceph-mon[91579]: pgmap v319: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:41 smithi188 ceph-mon[92425]: pgmap v319: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T10:36:42.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:42.163 DEBUG:teuthology.orchestra.run.smithi101:> ip netns list 2022-07-03T10:36:42.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:42.263 DEBUG:teuthology.orchestra.run.smithi101:> ip netns list-id 2022-07-03T10:36:42.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:42.324 DEBUG:teuthology.orchestra.run.smithi101:> (cd / && exec bash -c ' 2022-07-03T10:36:42.325 DEBUG:teuthology.orchestra.run.smithi101:> set -e 2022-07-03T10:36:42.325 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-07-03T10:36:42.325 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-07-03T10:36:42.326 DEBUG:teuthology.orchestra.run.smithi101:> ') 2022-07-03T10:36:42.513 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-07-03T10:36:42.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:42.514 DEBUG:teuthology.orchestra.run.smithi101:> (cd / && exec bash -c ' 2022-07-03T10:36:42.515 DEBUG:teuthology.orchestra.run.smithi101:> set -e 2022-07-03T10:36:42.516 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-07-03T10:36:42.516 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-07-03T10:36:42.517 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-07-03T10:36:42.518 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-07-03T10:36:42.518 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-07-03T10:36:42.519 DEBUG:teuthology.orchestra.run.smithi101:> ') 2022-07-03T10:36:42.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:42.864 DEBUG:teuthology.orchestra.run.smithi101:> (cd / && exec bash -c ' 2022-07-03T10:36:42.864 DEBUG:teuthology.orchestra.run.smithi101:> set -e 2022-07-03T10:36:42.864 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link set brx.0 up 2022-07-03T10:36:42.865 DEBUG:teuthology.orchestra.run.smithi101:> sudo ip link set dev brx.0 master ceph-brx 2022-07-03T10:36:42.865 DEBUG:teuthology.orchestra.run.smithi101:> ') 2022-07-03T10:36:43.081 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-03T10:36:43.082 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:43.143 INFO:teuthology.orchestra.run.smithi101.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-07-03T10:36:43.144 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-03T10:36:43.145 DEBUG:teuthology.orchestra.run.smithi101:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:43.205 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/f64dc3a0-3e2a-4139-9ee3-035a6e925e48 2022-07-03T10:36:43.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:43.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/f64dc3a0-3e2a-4139-9ee3-035a6e925e48 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2022-07-03T10:36:43.345 INFO:teuthology.orchestra.run.smithi101.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2022-07-03T10:36:43.346 INFO:teuthology.orchestra.run.smithi101.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2022-07-03T10:36:43.346 INFO:teuthology.orchestra.run.smithi101.stdout:invalid new device string format 2022-07-03T10:36:43.623 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:43 smithi101 ceph-mon[85126]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:43 smithi161 ceph-mon[91579]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:43 smithi188 ceph-mon[92425]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:43.826 INFO:teuthology.orchestra.run.smithi101.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2022-07-03T10:36:43.827 INFO:teuthology.orchestra.run.smithi101.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2022-07-03T10:36:43.827 INFO:teuthology.orchestra.run.smithi101.stdout:invalid new device string format 2022-07-03T10:36:43.827 INFO:teuthology.orchestra.run.smithi101.stdout:mount.ceph: resolved to: "172.21.15.101:3300,172.21.15.161:3300,172.21.15.188:3300" 2022-07-03T10:36:43.827 INFO:teuthology.orchestra.run.smithi101.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.101:3300,172.21.15.161:3300,172.21.15.188:3300:/volumes/_nogroup/sv_1/f64dc3a0-3e2a-4139-9ee3-035a6e925e48 2022-07-03T10:36:43.828 INFO:teuthology.orchestra.run.smithi101.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=23a851ae-fab9-11ec-842c-001a4aab830c" will pass to kernel 2022-07-03T10:36:43.840 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-07-03T10:36:43.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-03T10:36:43.842 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:43.933 INFO:teuthology.run_tasks:Running task fwd_scrub... 2022-07-03T10:36:43.950 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:44.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.175+0000 7f93683ff700 1 -- 172.21.15.101:0/2145938953 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600ff1c0 msgr2=0x7f9360105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.175+0000 7f93683ff700 1 --2- 172.21.15.101:0/2145938953 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600ff1c0 0x7f9360105a80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9358009ca0 tx=0x7f9358009520 comp rx=0 tx=0).stop 2022-07-03T10:36:44.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 -- 172.21.15.101:0/2145938953 shutdown_connections 2022-07-03T10:36:44.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 --2- 172.21.15.101:0/2145938953 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600ff1c0 0x7f9360105a80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 --2- 172.21.15.101:0/2145938953 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600fd5b0 0x7f93600fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 --2- 172.21.15.101:0/2145938953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fe830 0x7f93600fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 -- 172.21.15.101:0/2145938953 >> 172.21.15.101:0/2145938953 conn(0x7f93600f8d00 msgr2=0x7f93600fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:44.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 -- 172.21.15.101:0/2145938953 shutdown_connections 2022-07-03T10:36:44.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.176+0000 7f93683ff700 1 -- 172.21.15.101:0/2145938953 wait complete. 2022-07-03T10:36:44.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 Processor -- start 2022-07-03T10:36:44.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 -- start start 2022-07-03T10:36:44.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fd5b0 0x7f9360110470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 0x7f9360117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9360117fa0 con 0x7f93600fe830 2022-07-03T10:36:44.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601180e0 con 0x7f93600ff1c0 2022-07-03T10:36:44.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f93683ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601182a0 con 0x7f93600fd5b0 2022-07-03T10:36:44.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.177+0000 7f936599a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 0x7f9360117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936599a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:45578/0 (socket says 172.21.15.101:45578) 2022-07-03T10:36:44.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936599a700 1 -- 172.21.15.101:0/3036461908 learned_addr learned my addr 172.21.15.101:0/3036461908 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:44.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936619b700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fd5b0 0x7f9360110470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 -- 172.21.15.101:0/3036461908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fd5b0 msgr2=0x7f9360110470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fd5b0 0x7f9360110470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 -- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 msgr2=0x7f93601109b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 -- 172.21.15.101:0/3036461908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93580097e0 con 0x7f93600ff1c0 2022-07-03T10:36:44.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936599a700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:44.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.178+0000 7f936699c700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 0x7f9360117a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9358009b70 tx=0x7f9358009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:44.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.179+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935801d070 con 0x7f93600ff1c0 2022-07-03T10:36:44.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.179+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360118520 con 0x7f93600ff1c0 2022-07-03T10:36:44.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.179+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9358005000 con 0x7f93600ff1c0 2022-07-03T10:36:44.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.179+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935800f460 con 0x7f93600ff1c0 2022-07-03T10:36:44.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.179+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93601189c0 con 0x7f93600ff1c0 2022-07-03T10:36:44.190 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.180+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9360108d90 con 0x7f93600ff1c0 2022-07-03T10:36:44.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.182+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f9358029030 con 0x7f93600ff1c0 2022-07-03T10:36:44.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.183+0000 7f9356ffd700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 0x7f934c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.183+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f93580c10e0 con 0x7f93600ff1c0 2022-07-03T10:36:44.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.183+0000 7f936619b700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 0x7f934c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.184+0000 7f936619b700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 0x7f934c0991d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f93600fe690 tx=0x7f935c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:44.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.185+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f935808e380 con 0x7f93600ff1c0 2022-07-03T10:36:44.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.378+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9360119000 con 0x7f93600ff1c0 2022-07-03T10:36:44.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.380+0000 7f9356ffd700 1 -- 172.21.15.101:0/3036461908 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7f9358027020 con 0x7f93600ff1c0 2022-07-03T10:36:44.382 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:44.383 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:44.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 msgr2=0x7f934c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 0x7f934c0991d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f93600fe690 tx=0x7f935c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 msgr2=0x7f9360117a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 0x7f9360117a20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9358009b70 tx=0x7f9358009520 comp rx=0 tx=0).stop 2022-07-03T10:36:44.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 shutdown_connections 2022-07-03T10:36:44.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f934c096d20 0x7f934c0991d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93600ff1c0 0x7f9360117a20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93600fd5b0 0x7f9360110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 --2- 172.21.15.101:0/3036461908 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93600fe830 0x7f93601109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 >> 172.21.15.101:0/3036461908 conn(0x7f93600f8d00 msgr2=0x7f93600fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:44.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 shutdown_connections 2022-07-03T10:36:44.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.383+0000 7f93683ff700 1 -- 172.21.15.101:0/3036461908 wait complete. 2022-07-03T10:36:44.411 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:44.412 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:44.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 -- 172.21.15.101:0/2814410773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 msgr2=0x7fc0600fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 --2- 172.21.15.101:0/2814410773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc0600fecb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc054009900 tx=0x7fc0540091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 -- 172.21.15.101:0/2814410773 shutdown_connections 2022-07-03T10:36:44.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 --2- 172.21.15.101:0/2814410773 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600ff1f0 0x7fc060105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 --2- 172.21.15.101:0/2814410773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600fd5e0 0x7fc0600fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 --2- 172.21.15.101:0/2814410773 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc0600fecb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 -- 172.21.15.101:0/2814410773 >> 172.21.15.101:0/2814410773 conn(0x7fc0600f8d50 msgr2=0x7fc0600fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:44.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 -- 172.21.15.101:0/2814410773 shutdown_connections 2022-07-03T10:36:44.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.639+0000 7fc0657ff700 1 -- 172.21.15.101:0/2814410773 wait complete. 2022-07-03T10:36:44.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.640+0000 7fc0657ff700 1 Processor -- start 2022-07-03T10:36:44.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.640+0000 7fc0657ff700 1 -- start start 2022-07-03T10:36:44.651 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.651 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc060110cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.652 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600ff1f0 0x7fc060117d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601182e0 con 0x7fc0600fd5e0 2022-07-03T10:36:44.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060118420 con 0x7fc0600fe860 2022-07-03T10:36:44.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc0657ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0601185e0 con 0x7fc0600ff1f0 2022-07-03T10:36:44.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc05effd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc05e7fc700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc060110cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc05effd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:45592/0 (socket says 172.21.15.101:45592) 2022-07-03T10:36:44.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc05effd700 1 -- 172.21.15.101:0/596258010 learned_addr learned my addr 172.21.15.101:0/596258010 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:44.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.641+0000 7fc05f7fe700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600ff1f0 0x7fc060117d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 -- 172.21.15.101:0/596258010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600ff1f0 msgr2=0x7fc060117d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600ff1f0 0x7fc060117d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 -- 172.21.15.101:0/596258010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 msgr2=0x7fc060110cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc060110cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 -- 172.21.15.101:0/596258010 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054009440 con 0x7fc0600fd5e0 2022-07-03T10:36:44.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05e7fc700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc060110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:44.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc05effd700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e840 tx=0x7fc050005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:44.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050004030 con 0x7fc0600fd5e0 2022-07-03T10:36:44.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc050004190 con 0x7fc0600fd5e0 2022-07-03T10:36:44.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0601188c0 con 0x7fc0600fd5e0 2022-07-03T10:36:44.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050003a20 con 0x7fc0600fd5e0 2022-07-03T10:36:44.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.642+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc060118e50 con 0x7fc0600fd5e0 2022-07-03T10:36:44.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.643+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060108da0 con 0x7fc0600fd5e0 2022-07-03T10:36:44.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.645+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fc05000c040 con 0x7fc0600fd5e0 2022-07-03T10:36:44.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.646+0000 7fc0477fe700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 0x7fc048099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:44.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.647+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fc050014070 con 0x7fc0600fd5e0 2022-07-03T10:36:44.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.647+0000 7fc05e7fc700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 0x7fc048099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:44.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.647+0000 7fc05e7fc700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 0x7fc048099250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc054009f70 tx=0x7fc0540091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:44.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.650+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fc0500c6080 con 0x7fc0600fd5e0 2022-07-03T10:36:44.840 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.838+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc060108f30 con 0x7fc0600fd5e0 2022-07-03T10:36:44.841 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.839+0000 7fc0477fe700 1 -- 172.21.15.101:0/596258010 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc060108f30 con 0x7fc0600fd5e0 2022-07-03T10:36:44.841 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:44.843 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 msgr2=0x7fc048099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.843 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 0x7fc048099250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc054009f70 tx=0x7fc0540091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.844 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 msgr2=0x7fc0601107b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:44.844 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc05000e840 tx=0x7fc050005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.844 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 shutdown_connections 2022-07-03T10:36:44.844 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc048096da0 0x7fc048099250 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc0600fd5e0 0x7fc0601107b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0600ff1f0 0x7fc060117d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 --2- 172.21.15.101:0/596258010 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0600fe860 0x7fc060110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:44.845 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 >> 172.21.15.101:0/596258010 conn(0x7fc0600f8d50 msgr2=0x7fc060070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:44.846 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 shutdown_connections 2022-07-03T10:36:44.846 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:44.841+0000 7fc0657ff700 1 -- 172.21.15.101:0/596258010 wait complete. 2022-07-03T10:36:44.870 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:45.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 -- 172.21.15.101:0/2538077963 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0ff210 msgr2=0x7f256c105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 --2- 172.21.15.101:0/2538077963 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0ff210 0x7f256c105ad0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2564009ca0 tx=0x7f2564009520 comp rx=0 tx=0).stop 2022-07-03T10:36:45.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 -- 172.21.15.101:0/2538077963 shutdown_connections 2022-07-03T10:36:45.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 --2- 172.21.15.101:0/2538077963 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0ff210 0x7f256c105ad0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 --2- 172.21.15.101:0/2538077963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0fd600 0x7f256c0fda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 --2- 172.21.15.101:0/2538077963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fe880 0x7f256c0fecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 -- 172.21.15.101:0/2538077963 >> 172.21.15.101:0/2538077963 conn(0x7f256c0f8db0 msgr2=0x7f256c0fb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:45.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.098+0000 7f25733ff700 1 -- 172.21.15.101:0/2538077963 shutdown_connections 2022-07-03T10:36:45.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.099+0000 7f25733ff700 1 -- 172.21.15.101:0/2538077963 wait complete. 2022-07-03T10:36:45.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.099+0000 7f25733ff700 1 Processor -- start 2022-07-03T10:36:45.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 -- start start 2022-07-03T10:36:45.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 0x7f256c110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0ff210 0x7f256c117b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c118090 con 0x7f256c0fe880 2022-07-03T10:36:45.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c1181d0 con 0x7f256c0ff210 2022-07-03T10:36:45.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f25733ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c118390 con 0x7f256c0fd600 2022-07-03T10:36:45.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f257119b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f257119b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:33060/0 (socket says 172.21.15.101:33060) 2022-07-03T10:36:45.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f257119b700 1 -- 172.21.15.101:0/3649629926 learned_addr learned my addr 172.21.15.101:0/3649629926 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:45.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f257099a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 0x7f256c110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.100+0000 7f257199c700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0ff210 0x7f256c117b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 -- 172.21.15.101:0/3649629926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0ff210 msgr2=0x7f256c117b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0ff210 0x7f256c117b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 -- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 msgr2=0x7f256c110aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 0x7f256c110aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 -- 172.21.15.101:0/3649629926 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2568009410 con 0x7f256c0fd600 2022-07-03T10:36:45.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257099a700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 0x7f256c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:45.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f257119b700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f256800ea00 tx=0x7f2568005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:45.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568009690 con 0x7f256c0fd600 2022-07-03T10:36:45.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25640097e0 con 0x7f256c0fd600 2022-07-03T10:36:45.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f256800ce00 con 0x7f256c0fd600 2022-07-03T10:36:45.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568018420 con 0x7f256c0fd600 2022-07-03T10:36:45.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.101+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c118ac0 con 0x7f256c0fd600 2022-07-03T10:36:45.111 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.102+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c108e80 con 0x7f256c0fd600 2022-07-03T10:36:45.112 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.104+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f256800c040 con 0x7f256c0fd600 2022-07-03T10:36:45.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.106+0000 7f2561ffb700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 0x7f25580991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.106+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f2568014070 con 0x7f256c0fd600 2022-07-03T10:36:45.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.106+0000 7f257099a700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 0x7f25580991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.107+0000 7f257099a700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 0x7f25580991d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f255c009cf0 tx=0x7f255c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:45.113 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.108+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f256808d010 con 0x7f256c0fd600 2022-07-03T10:36:45.302 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.301+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f256c109010 con 0x7f256c0fd600 2022-07-03T10:36:45.303 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.301+0000 7f2561ffb700 1 -- 172.21.15.101:0/3649629926 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f256c109010 con 0x7f256c0fd600 2022-07-03T10:36:45.304 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:45.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 msgr2=0x7f25580991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 0x7f25580991d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f255c009cf0 tx=0x7f255c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 msgr2=0x7f256c110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f256800ea00 tx=0x7f2568005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 shutdown_connections 2022-07-03T10:36:45.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f2558096d20 0x7f25580991d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f256c0ff210 0x7f256c117b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f256c0fd600 0x7f256c110560 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 --2- 172.21.15.101:0/3649629926 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f256c0fe880 0x7f256c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 >> 172.21.15.101:0/3649629926 conn(0x7f256c0f8db0 msgr2=0x7f256c104080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:45.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 shutdown_connections 2022-07-03T10:36:45.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.304+0000 7f25733ff700 1 -- 172.21.15.101:0/3649629926 wait complete. 2022-07-03T10:36:45.333 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:45.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 -- 172.21.15.101:0/2008074319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fd470 msgr2=0x7f426c0fd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 --2- 172.21.15.101:0/2008074319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fd470 0x7f426c0fd880 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4260009900 tx=0x7f42600091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 -- 172.21.15.101:0/2008074319 shutdown_connections 2022-07-03T10:36:45.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 --2- 172.21.15.101:0/2008074319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0feec0 0x7f426c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 --2- 172.21.15.101:0/2008074319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fd470 0x7f426c0fd880 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 --2- 172.21.15.101:0/2008074319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0fddc0 0x7f426c0fe790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 -- 172.21.15.101:0/2008074319 >> 172.21.15.101:0/2008074319 conn(0x7f426c0f8c00 msgr2=0x7f426c0fb030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:45.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.561+0000 7f4271b9d700 1 -- 172.21.15.101:0/2008074319 shutdown_connections 2022-07-03T10:36:45.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.562+0000 7f4271b9d700 1 -- 172.21.15.101:0/2008074319 wait complete. 2022-07-03T10:36:45.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.562+0000 7f4271b9d700 1 Processor -- start 2022-07-03T10:36:45.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 -- start start 2022-07-03T10:36:45.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fddc0 0x7f426c10e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0feec0 0x7f426c1159e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c115f60 con 0x7f426c0feec0 2022-07-03T10:36:45.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1160a0 con 0x7f426c0fd470 2022-07-03T10:36:45.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4271b9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c116260 con 0x7f426c0fddc0 2022-07-03T10:36:45.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4270b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4270b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51750/0 (socket says 172.21.15.101:51750) 2022-07-03T10:36:45.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f4270b9b700 1 -- 172.21.15.101:0/3243408300 learned_addr learned my addr 172.21.15.101:0/3243408300 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:45.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f426bfff700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fddc0 0x7f426c10e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.563+0000 7f427139c700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0feec0 0x7f426c1159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 -- 172.21.15.101:0/3243408300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fddc0 msgr2=0x7f426c10e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fddc0 0x7f426c10e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 -- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0feec0 msgr2=0x7f426c1159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0feec0 0x7f426c1159e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 -- 172.21.15.101:0/3243408300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258009410 con 0x7f426c0fd470 2022-07-03T10:36:45.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4270b9b700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4260005d00 tx=0x7f42600091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:45.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426001d070 con 0x7f426c0fd470 2022-07-03T10:36:45.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260009440 con 0x7f426c0fd470 2022-07-03T10:36:45.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f426000b9c0 con 0x7f426c0fd470 2022-07-03T10:36:45.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.564+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260022b80 con 0x7f426c0fd470 2022-07-03T10:36:45.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.565+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c116850 con 0x7f426c0fd470 2022-07-03T10:36:45.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.566+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c108c60 con 0x7f426c0fd470 2022-07-03T10:36:45.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.567+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f4260029030 con 0x7f426c0fd470 2022-07-03T10:36:45.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.568+0000 7f42697fa700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 0x7f425c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:45.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.569+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f42600c0a70 con 0x7f426c0fd470 2022-07-03T10:36:45.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.569+0000 7f426bfff700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 0x7f425c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:45.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.569+0000 7f426bfff700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 0x7f425c0991d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f426c0fe190 tx=0x7f4258005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:45.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.571+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f426008dd10 con 0x7f426c0fd470 2022-07-03T10:36:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:45 smithi161 ceph-mon[91579]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:45 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3036461908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:45 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/596258010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:45 smithi188 ceph-mon[92425]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:45 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3036461908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:45 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/596258010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:45.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:45 smithi101 ceph-mon[85126]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T10:36:45.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:45 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3036461908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:45.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:45 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/596258010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:45.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.758+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f426c116e90 con 0x7f426c0fd470 2022-07-03T10:36:45.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.759+0000 7f42697fa700 1 -- 172.21.15.101:0/3243408300 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4260027020 con 0x7f426c0fd470 2022-07-03T10:36:45.761 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:45.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 msgr2=0x7f425c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 0x7f425c0991d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f426c0fe190 tx=0x7f4258005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 msgr2=0x7f426c10e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:45.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4260005d00 tx=0x7f42600091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 shutdown_connections 2022-07-03T10:36:45.764 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f425c096d20 0x7f425c0991d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f426c0feec0 0x7f426c1159e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426c0fd470 0x7f426c10e430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.766 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 --2- 172.21.15.101:0/3243408300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f426c0fddc0 0x7f426c10e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:45.766 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 >> 172.21.15.101:0/3243408300 conn(0x7f426c0f8c00 msgr2=0x7f426c0fafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:45.767 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 shutdown_connections 2022-07-03T10:36:45.767 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:45.761+0000 7f4271b9d700 1 -- 172.21.15.101:0/3243408300 wait complete. 2022-07-03T10:36:45.791 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:46.018 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 -- 172.21.15.101:0/3008168044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40fd5b0 msgr2=0x7f57c40fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/3008168044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40fd5b0 0x7f57c40fd9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009900 tx=0x7f57b80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 -- 172.21.15.101:0/3008168044 shutdown_connections 2022-07-03T10:36:46.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/3008168044 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40ff1c0 0x7f57c4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/3008168044 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40fd5b0 0x7f57c40fd9c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/3008168044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fe830 0x7f57c40fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.020 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 -- 172.21.15.101:0/3008168044 >> 172.21.15.101:0/3008168044 conn(0x7f57c40f8d00 msgr2=0x7f57c40fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:46.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 -- 172.21.15.101:0/3008168044 shutdown_connections 2022-07-03T10:36:46.021 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.016+0000 7f57cb1ff700 1 -- 172.21.15.101:0/3008168044 wait complete. 2022-07-03T10:36:46.024 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.017+0000 7f57cb1ff700 1 Processor -- start 2022-07-03T10:36:46.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 -- start start 2022-07-03T10:36:46.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 0x7f57c4072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40fe830 0x7f57c4070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.025 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 0x7f57c4070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4071480 con 0x7f57c40fe830 2022-07-03T10:36:46.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c40715f0 con 0x7f57c40ff1c0 2022-07-03T10:36:46.026 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57cb1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4118720 con 0x7f57c40fd5b0 2022-07-03T10:36:46.027 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.018+0000 7f57c8f9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 0x7f57c4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.027 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c8f9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 0x7f57c4072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:33088/0 (socket says 172.21.15.101:33088) 2022-07-03T10:36:46.027 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c8f9b700 1 -- 172.21.15.101:0/173949615 learned_addr learned my addr 172.21.15.101:0/173949615 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:46.027 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c3fff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40fe830 0x7f57c4070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 0x7f57c4070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 -- 172.21.15.101:0/173949615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 msgr2=0x7f57c4072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 0x7f57c4072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.028 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 -- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40fe830 msgr2=0x7f57c4070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40fe830 0x7f57c4070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 -- 172.21.15.101:0/173949615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc009700 con 0x7f57c40ff1c0 2022-07-03T10:36:46.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.019+0000 7f57c979c700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 0x7f57c4070db0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fd0 tx=0x7f57bc009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.020+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00caa0 con 0x7f57c40ff1c0 2022-07-03T10:36:46.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.020+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b8009440 con 0x7f57c40ff1c0 2022-07-03T10:36:46.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.020+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f57bc003d30 con 0x7f57c40ff1c0 2022-07-03T10:36:46.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.020+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc004630 con 0x7f57c40ff1c0 2022-07-03T10:36:46.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.020+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c4118d80 con 0x7f57c40ff1c0 2022-07-03T10:36:46.031 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.021+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c4108db0 con 0x7f57c40ff1c0 2022-07-03T10:36:46.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.023+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f57bc00c040 con 0x7f57c40ff1c0 2022-07-03T10:36:46.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.024+0000 7f57c17fa700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.024+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f57bc014070 con 0x7f57c40ff1c0 2022-07-03T10:36:46.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.025+0000 7f57c8f9b700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.025+0000 7f57c8f9b700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f57b80092d0 tx=0x7f57b80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.027+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f57bc08d320 con 0x7f57c40ff1c0 2022-07-03T10:36:46.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.214+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57c41047a0 con 0x7f57c40ff1c0 2022-07-03T10:36:46.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.216+0000 7f57c17fa700 1 -- 172.21.15.101:0/173949615 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7f57bc090940 con 0x7f57c40ff1c0 2022-07-03T10:36:46.218 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:46.219 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:46.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 msgr2=0x7f57b40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f57b80092d0 tx=0x7f57b80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 msgr2=0x7f57c4070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 0x7f57c4070db0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009fd0 tx=0x7f57bc009460 comp rx=0 tx=0).stop 2022-07-03T10:36:46.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 shutdown_connections 2022-07-03T10:36:46.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f57b4096d20 0x7f57b40991d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57c40ff1c0 0x7f57c4070db0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57c40fd5b0 0x7f57c4072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 --2- 172.21.15.101:0/173949615 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57c40fe830 0x7f57c4070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 >> 172.21.15.101:0/173949615 conn(0x7f57c40f8d00 msgr2=0x7f57c4101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:46.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 shutdown_connections 2022-07-03T10:36:46.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.218+0000 7f57cb1ff700 1 -- 172.21.15.101:0/173949615 wait complete. 2022-07-03T10:36:46.247 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:46.248 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:46.483 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 -- 172.21.15.101:0/2100757977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04072430 msgr2=0x7f0e04070910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/2100757977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04072430 0x7f0e04070910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e00009900 tx=0x7f0e000091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 -- 172.21.15.101:0/2100757977 shutdown_connections 2022-07-03T10:36:46.484 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/2100757977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04105300 0x7f0e041076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/2100757977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04072430 0x7f0e04070910 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/2100757977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e040712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 -- 172.21.15.101:0/2100757977 >> 172.21.15.101:0/2100757977 conn(0x7f0e040f8b40 msgr2=0x7f0e040fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:46.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 -- 172.21.15.101:0/2100757977 shutdown_connections 2022-07-03T10:36:46.485 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.476+0000 7f0e0b39d700 1 -- 172.21.15.101:0/2100757977 wait complete. 2022-07-03T10:36:46.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 Processor -- start 2022-07-03T10:36:46.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 -- start start 2022-07-03T10:36:46.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e04110680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.486 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 0x7f0e04110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041181b0 con 0x7f0e04072430 2022-07-03T10:36:46.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041182f0 con 0x7f0e04070e50 2022-07-03T10:36:46.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.477+0000 7f0e0b39d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e041184b0 con 0x7f0e04105300 2022-07-03T10:36:46.487 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e09b9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 0x7f0e04110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:33110/0 (socket says 172.21.15.101:33110) 2022-07-03T10:36:46.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 -- 172.21.15.101:0/3861294230 learned_addr learned my addr 172.21.15.101:0/3861294230 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:46.488 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0a39b700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e04110680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 -- 172.21.15.101:0/3861294230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 msgr2=0x7f0e04110680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e04110680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 -- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 msgr2=0x7f0e04110bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 0x7f0e04110bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.489 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0ab9c700 1 -- 172.21.15.101:0/3861294230 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00009440 con 0x7f0e04105300 2022-07-03T10:36:46.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e09b9a700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 0x7f0e04110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:46.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.478+0000 7f0e0a39b700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e04110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:46.490 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0e0ab9c700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00eb50 tx=0x7f0dfc009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc004a50 con 0x7f0e04105300 2022-07-03T10:36:46.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e04118790 con 0x7f0e04105300 2022-07-03T10:36:46.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0dfc00ce80 con 0x7f0e04105300 2022-07-03T10:36:46.491 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc005690 con 0x7f0e04105300 2022-07-03T10:36:46.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.479+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e04118d70 con 0x7f0e04105300 2022-07-03T10:36:46.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.480+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0410ae20 con 0x7f0e04105300 2022-07-03T10:36:46.492 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.482+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f0dfc00c040 con 0x7f0e04105300 2022-07-03T10:36:46.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.483+0000 7f0dfaffd700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 0x7f0df00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.493 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.483+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f0dfc014070 con 0x7f0e04105300 2022-07-03T10:36:46.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.483+0000 7f0e0a39b700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 0x7f0df00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.484+0000 7f0e0a39b700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 0x7f0df00991d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e000092d0 tx=0x7f0e000091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.494 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.485+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f0dfc08d380 con 0x7f0e04105300 2022-07-03T10:36:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:46 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3649629926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:46 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3243408300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:46 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/173949615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:46 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3649629926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:46 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3243408300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:46 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/173949615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:46.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.670+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e0410afb0 con 0x7f0e04105300 2022-07-03T10:36:46.672 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.671+0000 7f0dfaffd700 1 -- 172.21.15.101:0/3861294230 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e0410afb0 con 0x7f0e04105300 2022-07-03T10:36:46.673 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:46.676 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 msgr2=0x7f0df00991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 0x7f0df00991d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e000092d0 tx=0x7f0e000091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 msgr2=0x7f0e04117c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00eb50 tx=0x7f0dfc009460 comp rx=0 tx=0).stop 2022-07-03T10:36:46.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 shutdown_connections 2022-07-03T10:36:46.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f0df0096d20 0x7f0df00991d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e04105300 0x7f0e04117c30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e04070e50 0x7f0e04110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 --2- 172.21.15.101:0/3861294230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0e04072430 0x7f0e04110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 >> 172.21.15.101:0/3861294230 conn(0x7f0e040f8b40 msgr2=0x7f0e040faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:46.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 shutdown_connections 2022-07-03T10:36:46.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.673+0000 7f0e0b39d700 1 -- 172.21.15.101:0/3861294230 wait complete. 2022-07-03T10:36:46.700 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:46 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3649629926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.700 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:46 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3243408300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:46.701 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:46 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/173949615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:46.703 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:46.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 -- 172.21.15.101:0/3412300382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fddc0 msgr2=0x7faa400fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 --2- 172.21.15.101:0/3412300382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fddc0 0x7faa400fe790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa30009900 tx=0x7faa300091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 -- 172.21.15.101:0/3412300382 shutdown_connections 2022-07-03T10:36:46.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 --2- 172.21.15.101:0/3412300382 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 0x7faa401059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 --2- 172.21.15.101:0/3412300382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fd470 0x7faa400fd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 --2- 172.21.15.101:0/3412300382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fddc0 0x7faa400fe790 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 -- 172.21.15.101:0/3412300382 >> 172.21.15.101:0/3412300382 conn(0x7faa400f8c00 msgr2=0x7faa400fb030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:46.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 -- 172.21.15.101:0/3412300382 shutdown_connections 2022-07-03T10:36:46.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.929+0000 7faa46d9d700 1 -- 172.21.15.101:0/3412300382 wait complete. 2022-07-03T10:36:46.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 Processor -- start 2022-07-03T10:36:46.937 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 -- start start 2022-07-03T10:36:46.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fddc0 0x7faa4010e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.938 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 0x7faa40115990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40115f10 con 0x7faa400feec0 2022-07-03T10:36:46.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40116050 con 0x7faa400fd470 2022-07-03T10:36:46.939 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.930+0000 7faa46d9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa40116210 con 0x7faa400fddc0 2022-07-03T10:36:46.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51786/0 (socket says 172.21.15.101:51786) 2022-07-03T10:36:46.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa4659c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 0x7faa40115990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.940 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 -- 172.21.15.101:0/72398179 learned_addr learned my addr 172.21.15.101:0/72398179 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:46.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa4559a700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fddc0 0x7faa4010e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 -- 172.21.15.101:0/72398179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fddc0 msgr2=0x7faa4010e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.941 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fddc0 0x7faa4010e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 -- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 msgr2=0x7faa40115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:46.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 0x7faa40115990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:46.942 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.931+0000 7faa45d9b700 1 -- 172.21.15.101:0/72398179 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30009440 con 0x7faa400fd470 2022-07-03T10:36:46.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa45d9b700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00ea80 tx=0x7faa3c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c005020 con 0x7faa400fd470 2022-07-03T10:36:46.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faa3c00c7c0 con 0x7faa400fd470 2022-07-03T10:36:46.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa401164f0 con 0x7faa400fd470 2022-07-03T10:36:46.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c004940 con 0x7faa400fd470 2022-07-03T10:36:46.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.932+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa40116a30 con 0x7faa400fd470 2022-07-03T10:36:46.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.933+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa40108c10 con 0x7faa400fd470 2022-07-03T10:36:46.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.935+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7faa3c00c040 con 0x7faa400fd470 2022-07-03T10:36:46.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.936+0000 7faa367fc700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 0x7faa2c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:46.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.936+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7faa3c014070 con 0x7faa400fd470 2022-07-03T10:36:46.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.936+0000 7faa4559a700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 0x7faa2c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:46.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.937+0000 7faa4559a700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 0x7faa2c0991d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa300097d0 tx=0x7faa300091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:46.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:46.940+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7faa3c08d0d0 con 0x7faa400fd470 2022-07-03T10:36:47.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.126+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa40108da0 con 0x7faa400fd470 2022-07-03T10:36:47.128 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.126+0000 7faa367fc700 1 -- 172.21.15.101:0/72398179 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa40108da0 con 0x7faa400fd470 2022-07-03T10:36:47.129 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:47.131 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 msgr2=0x7faa2c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.131 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 0x7faa2c0991d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa300097d0 tx=0x7faa300091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.131 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 msgr2=0x7faa4010e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00ea80 tx=0x7faa3c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 shutdown_connections 2022-07-03T10:36:47.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7faa2c096d20 0x7faa2c0991d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faa400feec0 0x7faa40115990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa400fd470 0x7faa4010e3e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 --2- 172.21.15.101:0/72398179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa400fddc0 0x7faa4010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 >> 172.21.15.101:0/72398179 conn(0x7faa400f8c00 msgr2=0x7faa400faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:47.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 shutdown_connections 2022-07-03T10:36:47.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.129+0000 7faa46d9d700 1 -- 172.21.15.101:0/72398179 wait complete. 2022-07-03T10:36:47.157 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:47.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 -- 172.21.15.101:0/3100560220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 msgr2=0x7f182c1056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 --2- 172.21.15.101:0/3100560220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c1056b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1824009ca0 tx=0x7f1824009520 comp rx=0 tx=0).stop 2022-07-03T10:36:47.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 -- 172.21.15.101:0/3100560220 shutdown_connections 2022-07-03T10:36:47.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 --2- 172.21.15.101:0/3100560220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c1056b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 --2- 172.21.15.101:0/3100560220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 --2- 172.21.15.101:0/3100560220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 0x7f182c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 -- 172.21.15.101:0/3100560220 >> 172.21.15.101:0/3100560220 conn(0x7f182c0f7c50 msgr2=0x7f182c0fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:47.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 -- 172.21.15.101:0/3100560220 shutdown_connections 2022-07-03T10:36:47.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.389+0000 7f18327ff700 1 -- 172.21.15.101:0/3100560220 wait complete. 2022-07-03T10:36:47.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 Processor -- start 2022-07-03T10:36:47.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 -- start start 2022-07-03T10:36:47.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c1104e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 0x7f182c110a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c118010 con 0x7f182c103230 2022-07-03T10:36:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c118150 con 0x7f182c100740 2022-07-03T10:36:47.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.390+0000 7f18327ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182c118310 con 0x7f182c066380 2022-07-03T10:36:47.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:45714/0 (socket says 172.21.15.101:45714) 2022-07-03T10:36:47.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 -- 172.21.15.101:0/2042126472 learned_addr learned my addr 172.21.15.101:0/2042126472 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:47.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f182b7fe700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 0x7f182c110a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f182bfff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c1104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 -- 172.21.15.101:0/2042126472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 msgr2=0x7f182c1104e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c1104e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 -- 172.21.15.101:0/2042126472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 msgr2=0x7f182c110a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 0x7f182c110a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f1830d9c700 1 -- 172.21.15.101:0/2042126472 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240097e0 con 0x7f182c103230 2022-07-03T10:36:47.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.391+0000 7f182bfff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:47.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f1830d9c700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1824004bf0 tx=0x7f1824009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:47.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182401d070 con 0x7f182c103230 2022-07-03T10:36:47.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c118590 con 0x7f182c103230 2022-07-03T10:36:47.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1824004e10 con 0x7f182c103230 2022-07-03T10:36:47.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182400f6d0 con 0x7f182c103230 2022-07-03T10:36:47.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.392+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c118a30 con 0x7f182c103230 2022-07-03T10:36:47.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.393+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f182c108da0 con 0x7f182c103230 2022-07-03T10:36:47.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.395+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f1824029030 con 0x7f182c103230 2022-07-03T10:36:47.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.396+0000 7f1828ff9700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 0x7f18140991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.396+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f1824022e40 con 0x7f182c103230 2022-07-03T10:36:47.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.396+0000 7f182bfff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 0x7f18140991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.397+0000 7f182bfff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 0x7f18140991d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f182c066130 tx=0x7f181c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:47.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.398+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f182408e930 con 0x7f182c103230 2022-07-03T10:36:47.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:47 smithi101 ceph-mon[85126]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:36:47.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:47 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3861294230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:47 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/72398179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.587+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f182c108f30 con 0x7f182c103230 2022-07-03T10:36:47.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.587+0000 7f1828ff9700 1 -- 172.21.15.101:0/2042126472 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f182c108f30 con 0x7f182c103230 2022-07-03T10:36:47.589 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:47.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.589+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 msgr2=0x7f18140991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.589+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 0x7f18140991d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f182c066130 tx=0x7f181c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.589+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 msgr2=0x7f182c117a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.589+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1824004bf0 tx=0x7f1824009520 comp rx=0 tx=0).stop 2022-07-03T10:36:47.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 shutdown_connections 2022-07-03T10:36:47.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f1814096d20 0x7f18140991d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f182c103230 0x7f182c117a90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f182c066380 0x7f182c1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 --2- 172.21.15.101:0/2042126472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f182c100740 0x7f182c110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 >> 172.21.15.101:0/2042126472 conn(0x7f182c0f7c50 msgr2=0x7f182c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:47.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 shutdown_connections 2022-07-03T10:36:47.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.590+0000 7f18327ff700 1 -- 172.21.15.101:0/2042126472 wait complete. 2022-07-03T10:36:47.619 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2022-07-03T10:36:47.619 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:36:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:47 smithi161 ceph-mon[91579]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:36:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:47 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3861294230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:47 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/72398179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:47 smithi188 ceph-mon[92425]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:36:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:47 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3861294230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:47 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/72398179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:47.846 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 -- 172.21.15.101:0/3340862678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0fd5b0 msgr2=0x7f408c0fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 --2- 172.21.15.101:0/3340862678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0fd5b0 0x7f408c0fd9c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f407c009900 tx=0x7f407c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 -- 172.21.15.101:0/3340862678 shutdown_connections 2022-07-03T10:36:47.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 --2- 172.21.15.101:0/3340862678 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0ff1c0 0x7f408c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.847 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 --2- 172.21.15.101:0/3340862678 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0fd5b0 0x7f408c0fd9c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 --2- 172.21.15.101:0/3340862678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fe830 0x7f408c0fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 -- 172.21.15.101:0/3340862678 >> 172.21.15.101:0/3340862678 conn(0x7f408c0f8d00 msgr2=0x7f408c0fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:47.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 -- 172.21.15.101:0/3340862678 shutdown_connections 2022-07-03T10:36:47.848 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.844+0000 7f40927ff700 1 -- 172.21.15.101:0/3340862678 wait complete. 2022-07-03T10:36:47.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 Processor -- start 2022-07-03T10:36:47.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 -- start start 2022-07-03T10:36:47.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 0x7f408c1107e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 0x7f408c110d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 0x7f408c117d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c118310 con 0x7f408c0fe830 2022-07-03T10:36:47.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c118450 con 0x7f408c0ff1c0 2022-07-03T10:36:47.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.845+0000 7f40927ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c118610 con 0x7f408c0fd5b0 2022-07-03T10:36:47.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f4090d9c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 0x7f408c117d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 0x7f408c1107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408b7fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 0x7f408c110d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f4090d9c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 0x7f408c117d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51818/0 (socket says 172.21.15.101:51818) 2022-07-03T10:36:47.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f4090d9c700 1 -- 172.21.15.101:0/375184658 learned_addr learned my addr 172.21.15.101:0/375184658 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:47.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 -- 172.21.15.101:0/375184658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 msgr2=0x7f408c117d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 0x7f408c117d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.861 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 -- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 msgr2=0x7f408c110d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:47.861 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 0x7f408c110d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:47.861 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408bfff700 1 -- 172.21.15.101:0/375184658 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c009440 con 0x7f408c0fd5b0 2022-07-03T10:36:47.861 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.846+0000 7f408b7fe700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 0x7f408c110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:47.862 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f408bfff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 0x7f408c1107e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f407c006010 tx=0x7f407c0091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:47.862 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c01d070 con 0x7f408c0fd5b0 2022-07-03T10:36:47.862 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c118890 con 0x7f408c0fd5b0 2022-07-03T10:36:47.862 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f407c00b9c0 con 0x7f408c0fd5b0 2022-07-03T10:36:47.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c022ac0 con 0x7f408c0fd5b0 2022-07-03T10:36:47.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.847+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c118d90 con 0x7f408c0fd5b0 2022-07-03T10:36:47.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.848+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c108d60 con 0x7f408c0fd5b0 2022-07-03T10:36:47.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.850+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f407c029030 con 0x7f408c0fd5b0 2022-07-03T10:36:47.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.851+0000 7f4088ff9700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 0x7f40740991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:47.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.851+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f407c0c1250 con 0x7f408c0fd5b0 2022-07-03T10:36:47.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.851+0000 7f408b7fe700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 0x7f40740991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:47.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.852+0000 7f408b7fe700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 0x7f40740991d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4080011560 tx=0x7f4080005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:47.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:47.853+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f407c08e4f0 con 0x7f408c0fd5b0 2022-07-03T10:36:48.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.089+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f408c108ef0 con 0x7f408c0fd5b0 2022-07-03T10:36:48.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.090+0000 7f4088ff9700 1 -- 172.21.15.101:0/375184658 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7f408c108ef0 con 0x7f408c0fd5b0 2022-07-03T10:36:48.092 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:48.092 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:36:48.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 msgr2=0x7f40740991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 0x7f40740991d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4080011560 tx=0x7f4080005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 msgr2=0x7f408c1107e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 0x7f408c1107e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f407c006010 tx=0x7f407c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 shutdown_connections 2022-07-03T10:36:48.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4074096d20 0x7f40740991d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f408c0ff1c0 0x7f408c117d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f408c0fd5b0 0x7f408c1107e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 --2- 172.21.15.101:0/375184658 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f408c0fe830 0x7f408c110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 >> 172.21.15.101:0/375184658 conn(0x7f408c0f8d00 msgr2=0x7f408c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:48.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.092+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 shutdown_connections 2022-07-03T10:36:48.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.093+0000 7f40927ff700 1 -- 172.21.15.101:0/375184658 wait complete. 2022-07-03T10:36:48.097 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:48.121 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2022-07-03T10:36:48.121 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2022-07-03T10:36:48.122 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-03T10:36:48.352 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 -- 172.21.15.101:0/2085366025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00fd5d0 msgr2=0x7f22a00fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.353 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 --2- 172.21.15.101:0/2085366025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00fd5d0 0x7f22a00fd9e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f229c009900 tx=0x7f229c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.353 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 -- 172.21.15.101:0/2085366025 shutdown_connections 2022-07-03T10:36:48.353 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 --2- 172.21.15.101:0/2085366025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00ff1e0 0x7f22a0105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.353 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 --2- 172.21.15.101:0/2085366025 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00fd5d0 0x7f22a00fd9e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.354 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 --2- 172.21.15.101:0/2085366025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fe850 0x7f22a00feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.354 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 -- 172.21.15.101:0/2085366025 >> 172.21.15.101:0/2085366025 conn(0x7f22a00f8d20 msgr2=0x7f22a00fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:48.354 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 -- 172.21.15.101:0/2085366025 shutdown_connections 2022-07-03T10:36:48.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.350+0000 7f22a87ff700 1 -- 172.21.15.101:0/2085366025 wait complete. 2022-07-03T10:36:48.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 Processor -- start 2022-07-03T10:36:48.357 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 -- start start 2022-07-03T10:36:48.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fd5d0 0x7f22a0072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 0x7f22a0070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0071480 con 0x7f22a00fe850 2022-07-03T10:36:48.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00715f0 con 0x7f22a00ff1e0 2022-07-03T10:36:48.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.351+0000 7f22a87ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a01186f0 con 0x7f22a00fd5d0 2022-07-03T10:36:48.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:46910/0 (socket says 172.21.15.101:46910) 2022-07-03T10:36:48.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 -- 172.21.15.101:0/1446771096 learned_addr learned my addr 172.21.15.101:0/1446771096 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:48.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a6d9c700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 0x7f22a0070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a659b700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fd5d0 0x7f22a0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 -- 172.21.15.101:0/1446771096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fd5d0 msgr2=0x7f22a0072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fd5d0 0x7f22a0072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 -- 172.21.15.101:0/1446771096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 msgr2=0x7f22a0070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 0x7f22a0070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 -- 172.21.15.101:0/1446771096 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c009440 con 0x7f22a00fe850 2022-07-03T10:36:48.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a6d9c700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 0x7f22a0070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:48.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.352+0000 7f22a5d9a700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f229000ea80 tx=0x7f2290005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:48.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.353+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290005020 con 0x7f22a00fe850 2022-07-03T10:36:48.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.353+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a0118950 con 0x7f22a00fe850 2022-07-03T10:36:48.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.353+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f229000c7c0 con 0x7f22a00fe850 2022-07-03T10:36:48.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.353+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290004940 con 0x7f22a00fe850 2022-07-03T10:36:48.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.353+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a0118eb0 con 0x7f22a00fe850 2022-07-03T10:36:48.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.354+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a0108dd0 con 0x7f22a00fe850 2022-07-03T10:36:48.364 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.356+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f229000c040 con 0x7f22a00fe850 2022-07-03T10:36:48.364 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.357+0000 7f2296ffd700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 0x7f228c0991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.365 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.357+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f2290014070 con 0x7f22a00fe850 2022-07-03T10:36:48.365 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.357+0000 7f22a659b700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 0x7f228c0991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.365 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.358+0000 7f22a659b700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 0x7f228c0991d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f229c0097d0 tx=0x7f229c0091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:48.366 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.361+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f229008d0d0 con 0x7f22a00fe850 2022-07-03T10:36:48.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:48 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2042126472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:48.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:48 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/375184658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:48.547 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.545+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f22a005e960 con 0x7f228c096d20 2022-07-03T10:36:48.559 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.557+0000 7f2296ffd700 1 -- 172.21.15.101:0/1446771096 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343667 (secure 0 0 0) 0x7f22a005e960 con 0x7f228c096d20 2022-07-03T10:36:48.564 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:48.567 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 msgr2=0x7f228c0991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 0x7f228c0991d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f229c0097d0 tx=0x7f229c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 msgr2=0x7f22a0070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f229000ea80 tx=0x7f2290005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.568 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 shutdown_connections 2022-07-03T10:36:48.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f228c096d20 0x7f228c0991d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22a00ff1e0 0x7f22a0070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22a00fd5d0 0x7f22a0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 --2- 172.21.15.101:0/1446771096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f22a00fe850 0x7f22a0070870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 >> 172.21.15.101:0/1446771096 conn(0x7f22a00f8d20 msgr2=0x7f22a0101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:48.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 shutdown_connections 2022-07-03T10:36:48.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.562+0000 7f22a87ff700 1 -- 172.21.15.101:0/1446771096 wait complete. 2022-07-03T10:36:48.570 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:36:48.591 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2022-07-03T10:36:47.964118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645440,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":336,"num_read_kb":269,"num_write":258,"num_write_kb":3018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":705,"ondisk_log_size":705,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78288,"kb_used_data":10736,"kb_used_omap":0,"kb_used_meta":67456,"kb_avail":1124863536,"statfs":{"total":1151940427776,"available":1151860260864,"internally_reserved":0,"allocated":10993664,"data_stored":7174585,"data_compressed":176772,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":69074944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":322,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003008"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244527+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:36:26.244527+0000","last_peered":"2022-07-03T10:36:26.244527+0000","last_clean":"2022-07-03T10:36:26.244527+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:36:26.244527+0000","last_undegraded":"2022-07-03T10:36:26.244527+0000","last_fullsized":"2022-07-03T10:36:26.244527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.261640+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:36:26.261640+0000","last_peered":"2022-07-03T10:36:26.261640+0000","last_clean":"2022-07-03T10:36:26.261640+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:36:26.261640+0000","last_undegraded":"2022-07-03T10:36:26.261640+0000","last_fullsized":"2022-07-03T10:36:26.261640+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260480+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:36:26.260480+0000","last_peered":"2022-07-03T10:36:26.260480+0000","last_clean":"2022-07-03T10:36:26.260480+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:36:26.260480+0000","last_undegraded":"2022-07-03T10:36:26.260480+0000","last_fullsized":"2022-07-03T10:36:26.260480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248817+0000","last_change":"2022-07-03T10:35:27.315233+0000","last_active":"2022-07-03T10:36:26.248817+0000","last_peered":"2022-07-03T10:36:26.248817+0000","last_clean":"2022-07-03T10:36:26.248817+0000","last_became_active":"2022-07-03T10:35:27.314807+0000","last_became_peered":"2022-07-03T10:35:27.314807+0000","last_unstale":"2022-07-03T10:36:26.248817+0000","last_undegraded":"2022-07-03T10:36:26.248817+0000","last_fullsized":"2022-07-03T10:36:26.248817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:37:28.362457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240860+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:36:26.240860+0000","last_peered":"2022-07-03T10:36:26.240860+0000","last_clean":"2022-07-03T10:36:26.240860+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:36:26.240860+0000","last_undegraded":"2022-07-03T10:36:26.240860+0000","last_fullsized":"2022-07-03T10:36:26.240860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234621+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:36:26.234621+0000","last_peered":"2022-07-03T10:36:26.234621+0000","last_clean":"2022-07-03T10:36:26.234621+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:36:26.234621+0000","last_undegraded":"2022-07-03T10:36:26.234621+0000","last_fullsized":"2022-07-03T10:36:26.234621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247958+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:36:26.247958+0000","last_peered":"2022-07-03T10:36:26.247958+0000","last_clean":"2022-07-03T10:36:26.247958+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:36:26.247958+0000","last_undegraded":"2022-07-03T10:36:26.247958+0000","last_fullsized":"2022-07-03T10:36:26.247958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.252954+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:36:26.252954+0000","last_peered":"2022-07-03T10:36:26.252954+0000","last_clean":"2022-07-03T10:36:26.252954+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:36:26.252954+0000","last_undegraded":"2022-07-03T10:36:26.252954+0000","last_fullsized":"2022-07-03T10:36:26.252954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251354+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:36:26.251354+0000","last_peered":"2022-07-03T10:36:26.251354+0000","last_clean":"2022-07-03T10:36:26.251354+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:36:26.251354+0000","last_undegraded":"2022-07-03T10:36:26.251354+0000","last_fullsized":"2022-07-03T10:36:26.251354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241581+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:36:26.241581+0000","last_peered":"2022-07-03T10:36:26.241581+0000","last_clean":"2022-07-03T10:36:26.241581+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:36:26.241581+0000","last_undegraded":"2022-07-03T10:36:26.241581+0000","last_fullsized":"2022-07-03T10:36:26.241581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246804+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:36:26.246804+0000","last_peered":"2022-07-03T10:36:26.246804+0000","last_clean":"2022-07-03T10:36:26.246804+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:36:26.246804+0000","last_undegraded":"2022-07-03T10:36:26.246804+0000","last_fullsized":"2022-07-03T10:36:26.246804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237345+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:36:26.237345+0000","last_peered":"2022-07-03T10:36:26.237345+0000","last_clean":"2022-07-03T10:36:26.237345+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.237345+0000","last_undegraded":"2022-07-03T10:36:26.237345+0000","last_fullsized":"2022-07-03T10:36:26.237345+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245178+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:36:26.245178+0000","last_peered":"2022-07-03T10:36:26.245178+0000","last_clean":"2022-07-03T10:36:26.245178+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:36:26.245178+0000","last_undegraded":"2022-07-03T10:36:26.245178+0000","last_fullsized":"2022-07-03T10:36:26.245178+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.262152+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:36:26.262152+0000","last_peered":"2022-07-03T10:36:26.262152+0000","last_clean":"2022-07-03T10:36:26.262152+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:36:26.262152+0000","last_undegraded":"2022-07-03T10:36:26.262152+0000","last_fullsized":"2022-07-03T10:36:26.262152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259927+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:36:26.259927+0000","last_peered":"2022-07-03T10:36:26.259927+0000","last_clean":"2022-07-03T10:36:26.259927+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:36:26.259927+0000","last_undegraded":"2022-07-03T10:36:26.259927+0000","last_fullsized":"2022-07-03T10:36:26.259927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256218+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:36:26.256218+0000","last_peered":"2022-07-03T10:36:26.256218+0000","last_clean":"2022-07-03T10:36:26.256218+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:36:26.256218+0000","last_undegraded":"2022-07-03T10:36:26.256218+0000","last_fullsized":"2022-07-03T10:36:26.256218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248902+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:36:26.248902+0000","last_peered":"2022-07-03T10:36:26.248902+0000","last_clean":"2022-07-03T10:36:26.248902+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:36:26.248902+0000","last_undegraded":"2022-07-03T10:36:26.248902+0000","last_fullsized":"2022-07-03T10:36:26.248902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248745+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:36:26.248745+0000","last_peered":"2022-07-03T10:36:26.248745+0000","last_clean":"2022-07-03T10:36:26.248745+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:36:26.248745+0000","last_undegraded":"2022-07-03T10:36:26.248745+0000","last_fullsized":"2022-07-03T10:36:26.248745+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246317+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:36:26.246317+0000","last_peered":"2022-07-03T10:36:26.246317+0000","last_clean":"2022-07-03T10:36:26.246317+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:36:26.246317+0000","last_undegraded":"2022-07-03T10:36:26.246317+0000","last_fullsized":"2022-07-03T10:36:26.246317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325415+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:26.325415+0000","last_peered":"2022-07-03T10:35:26.325415+0000","last_clean":"2022-07-03T10:35:26.325415+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:26.325415+0000","last_undegraded":"2022-07-03T10:35:26.325415+0000","last_fullsized":"2022-07-03T10:35:26.325415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242008+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:36:26.242008+0000","last_peered":"2022-07-03T10:36:26.242008+0000","last_clean":"2022-07-03T10:36:26.242008+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:36:26.242008+0000","last_undegraded":"2022-07-03T10:36:26.242008+0000","last_fullsized":"2022-07-03T10:36:26.242008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343531+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:26.343531+0000","last_peered":"2022-07-03T10:35:26.343531+0000","last_clean":"2022-07-03T10:35:26.343531+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:26.343531+0000","last_undegraded":"2022-07-03T10:35:26.343531+0000","last_fullsized":"2022-07-03T10:35:26.343531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260621+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:36:26.260621+0000","last_peered":"2022-07-03T10:36:26.260621+0000","last_clean":"2022-07-03T10:36:26.260621+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:36:26.260621+0000","last_undegraded":"2022-07-03T10:36:26.260621+0000","last_fullsized":"2022-07-03T10:36:26.260621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309637+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:26.309637+0000","last_peered":"2022-07-03T10:35:26.309637+0000","last_clean":"2022-07-03T10:35:26.309637+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:26.309637+0000","last_undegraded":"2022-07-03T10:35:26.309637+0000","last_fullsized":"2022-07-03T10:35:26.309637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250215+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:36:26.250215+0000","last_peered":"2022-07-03T10:36:26.250215+0000","last_clean":"2022-07-03T10:36:26.250215+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:36:26.250215+0000","last_undegraded":"2022-07-03T10:36:26.250215+0000","last_fullsized":"2022-07-03T10:36:26.250215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309394+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:26.309394+0000","last_peered":"2022-07-03T10:35:26.309394+0000","last_clean":"2022-07-03T10:35:26.309394+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:26.309394+0000","last_undegraded":"2022-07-03T10:35:26.309394+0000","last_fullsized":"2022-07-03T10:35:26.309394+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248977+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:36:26.248977+0000","last_peered":"2022-07-03T10:36:26.248977+0000","last_clean":"2022-07-03T10:36:26.248977+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:36:26.248977+0000","last_undegraded":"2022-07-03T10:36:26.248977+0000","last_fullsized":"2022-07-03T10:36:26.248977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305894+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:26.305894+0000","last_peered":"2022-07-03T10:35:26.305894+0000","last_clean":"2022-07-03T10:35:26.305894+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:26.305894+0000","last_undegraded":"2022-07-03T10:35:26.305894+0000","last_fullsized":"2022-07-03T10:35:26.305894+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234357+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:36:26.234357+0000","last_peered":"2022-07-03T10:36:26.234357+0000","last_clean":"2022-07-03T10:36:26.234357+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:36:26.234357+0000","last_undegraded":"2022-07-03T10:36:26.234357+0000","last_fullsized":"2022-07-03T10:36:26.234357+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309034+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:26.309034+0000","last_peered":"2022-07-03T10:35:26.309034+0000","last_clean":"2022-07-03T10:35:26.309034+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:26.309034+0000","last_undegraded":"2022-07-03T10:35:26.309034+0000","last_fullsized":"2022-07-03T10:35:26.309034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256408+0000","last_change":"2022-07-03T10:35:27.324978+0000","last_active":"2022-07-03T10:36:26.256408+0000","last_peered":"2022-07-03T10:36:26.256408+0000","last_clean":"2022-07-03T10:36:26.256408+0000","last_became_active":"2022-07-03T10:35:27.324570+0000","last_became_peered":"2022-07-03T10:35:27.324570+0000","last_unstale":"2022-07-03T10:36:26.256408+0000","last_undegraded":"2022-07-03T10:36:26.256408+0000","last_fullsized":"2022-07-03T10:36:26.256408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:18:06.187457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332535+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:26.332535+0000","last_peered":"2022-07-03T10:35:26.332535+0000","last_clean":"2022-07-03T10:35:26.332535+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:26.332535+0000","last_undegraded":"2022-07-03T10:35:26.332535+0000","last_fullsized":"2022-07-03T10:35:26.332535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.254411+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:36:26.254411+0000","last_peered":"2022-07-03T10:36:26.254411+0000","last_clean":"2022-07-03T10:36:26.254411+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:36:26.254411+0000","last_undegraded":"2022-07-03T10:36:26.254411+0000","last_fullsized":"2022-07-03T10:36:26.254411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305427+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:26.305427+0000","last_peered":"2022-07-03T10:35:26.305427+0000","last_clean":"2022-07-03T10:35:26.305427+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:26.305427+0000","last_undegraded":"2022-07-03T10:35:26.305427+0000","last_fullsized":"2022-07-03T10:35:26.305427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255985+0000","last_change":"2022-07-03T10:35:27.314929+0000","last_active":"2022-07-03T10:36:26.255985+0000","last_peered":"2022-07-03T10:36:26.255985+0000","last_clean":"2022-07-03T10:36:26.255985+0000","last_became_active":"2022-07-03T10:35:27.314542+0000","last_became_peered":"2022-07-03T10:35:27.314542+0000","last_unstale":"2022-07-03T10:36:26.255985+0000","last_undegraded":"2022-07-03T10:36:26.255985+0000","last_fullsized":"2022-07-03T10:36:26.255985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:50:25.278127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309250+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:26.309250+0000","last_peered":"2022-07-03T10:35:26.309250+0000","last_clean":"2022-07-03T10:35:26.309250+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:26.309250+0000","last_undegraded":"2022-07-03T10:35:26.309250+0000","last_fullsized":"2022-07-03T10:35:26.309250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251048+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:36:26.251048+0000","last_peered":"2022-07-03T10:36:26.251048+0000","last_clean":"2022-07-03T10:36:26.251048+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:36:26.251048+0000","last_undegraded":"2022-07-03T10:36:26.251048+0000","last_fullsized":"2022-07-03T10:36:26.251048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347933+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:26.347933+0000","last_peered":"2022-07-03T10:35:26.347933+0000","last_clean":"2022-07-03T10:35:26.347933+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:26.347933+0000","last_undegraded":"2022-07-03T10:35:26.347933+0000","last_fullsized":"2022-07-03T10:35:26.347933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258203+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:36:26.258203+0000","last_peered":"2022-07-03T10:36:26.258203+0000","last_clean":"2022-07-03T10:36:26.258203+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:36:26.258203+0000","last_undegraded":"2022-07-03T10:36:26.258203+0000","last_fullsized":"2022-07-03T10:36:26.258203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306107+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:26.306107+0000","last_peered":"2022-07-03T10:35:26.306107+0000","last_clean":"2022-07-03T10:35:26.306107+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:26.306107+0000","last_undegraded":"2022-07-03T10:35:26.306107+0000","last_fullsized":"2022-07-03T10:35:26.306107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250505+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:36:26.250505+0000","last_peered":"2022-07-03T10:36:26.250505+0000","last_clean":"2022-07-03T10:36:26.250505+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.250505+0000","last_undegraded":"2022-07-03T10:36:26.250505+0000","last_fullsized":"2022-07-03T10:36:26.250505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'30","reported_seq":248,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:44.057423+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:36:44.057423+0000","last_peered":"2022-07-03T10:36:44.057423+0000","last_clean":"2022-07-03T10:36:44.057423+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:36:44.057423+0000","last_undegraded":"2022-07-03T10:36:44.057423+0000","last_fullsized":"2022-07-03T10:36:44.057423+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54044,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":205,"num_read_kb":155,"num_write":30,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.235656+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:36:26.235656+0000","last_peered":"2022-07-03T10:36:26.235656+0000","last_clean":"2022-07-03T10:36:26.235656+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:36:26.235656+0000","last_undegraded":"2022-07-03T10:36:26.235656+0000","last_fullsized":"2022-07-03T10:36:26.235656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.326689+0000","last_change":"2022-07-03T10:35:27.326689+0000","last_active":"2022-07-03T10:35:27.326689+0000","last_peered":"2022-07-03T10:35:27.326689+0000","last_clean":"2022-07-03T10:35:27.326689+0000","last_became_active":"2022-07-03T10:35:27.320346+0000","last_became_peered":"2022-07-03T10:35:27.320346+0000","last_unstale":"2022-07-03T10:35:27.326689+0000","last_undegraded":"2022-07-03T10:35:27.326689+0000","last_fullsized":"2022-07-03T10:35:27.326689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:57:55.061972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250431+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:36:26.250431+0000","last_peered":"2022-07-03T10:36:26.250431+0000","last_clean":"2022-07-03T10:36:26.250431+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:36:26.250431+0000","last_undegraded":"2022-07-03T10:36:26.250431+0000","last_fullsized":"2022-07-03T10:36:26.250431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305459+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:26.305459+0000","last_peered":"2022-07-03T10:35:26.305459+0000","last_clean":"2022-07-03T10:35:26.305459+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:26.305459+0000","last_undegraded":"2022-07-03T10:35:26.305459+0000","last_fullsized":"2022-07-03T10:35:26.305459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245612+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:36:26.245612+0000","last_peered":"2022-07-03T10:36:26.245612+0000","last_clean":"2022-07-03T10:36:26.245612+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:36:26.245612+0000","last_undegraded":"2022-07-03T10:36:26.245612+0000","last_fullsized":"2022-07-03T10:36:26.245612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302039+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:26.302039+0000","last_peered":"2022-07-03T10:35:26.302039+0000","last_clean":"2022-07-03T10:35:26.302039+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:26.302039+0000","last_undegraded":"2022-07-03T10:35:26.302039+0000","last_fullsized":"2022-07-03T10:35:26.302039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241776+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:36:26.241776+0000","last_peered":"2022-07-03T10:36:26.241776+0000","last_clean":"2022-07-03T10:36:26.241776+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:36:26.241776+0000","last_undegraded":"2022-07-03T10:36:26.241776+0000","last_fullsized":"2022-07-03T10:36:26.241776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308728+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:26.308728+0000","last_peered":"2022-07-03T10:35:26.308728+0000","last_clean":"2022-07-03T10:35:26.308728+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:26.308728+0000","last_undegraded":"2022-07-03T10:35:26.308728+0000","last_fullsized":"2022-07-03T10:35:26.308728+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248392+0000","last_change":"2022-07-03T10:35:27.623637+0000","last_active":"2022-07-03T10:36:26.248392+0000","last_peered":"2022-07-03T10:36:26.248392+0000","last_clean":"2022-07-03T10:36:26.248392+0000","last_became_active":"2022-07-03T10:35:27.623202+0000","last_became_peered":"2022-07-03T10:35:27.623202+0000","last_unstale":"2022-07-03T10:36:26.248392+0000","last_undegraded":"2022-07-03T10:36:26.248392+0000","last_fullsized":"2022-07-03T10:36:26.248392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:39:41.407533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310134+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:26.310134+0000","last_peered":"2022-07-03T10:35:26.310134+0000","last_clean":"2022-07-03T10:35:26.310134+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:26.310134+0000","last_undegraded":"2022-07-03T10:35:26.310134+0000","last_fullsized":"2022-07-03T10:35:26.310134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240643+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:36:26.240643+0000","last_peered":"2022-07-03T10:36:26.240643+0000","last_clean":"2022-07-03T10:36:26.240643+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:36:26.240643+0000","last_undegraded":"2022-07-03T10:36:26.240643+0000","last_fullsized":"2022-07-03T10:36:26.240643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.324898+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:26.324898+0000","last_peered":"2022-07-03T10:35:26.324898+0000","last_clean":"2022-07-03T10:35:26.324898+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:26.324898+0000","last_undegraded":"2022-07-03T10:35:26.324898+0000","last_fullsized":"2022-07-03T10:35:26.324898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244794+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:36:26.244794+0000","last_peered":"2022-07-03T10:36:26.244794+0000","last_clean":"2022-07-03T10:36:26.244794+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:36:26.244794+0000","last_undegraded":"2022-07-03T10:36:26.244794+0000","last_fullsized":"2022-07-03T10:36:26.244794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347744+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:26.347744+0000","last_peered":"2022-07-03T10:35:26.347744+0000","last_clean":"2022-07-03T10:35:26.347744+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:26.347744+0000","last_undegraded":"2022-07-03T10:35:26.347744+0000","last_fullsized":"2022-07-03T10:35:26.347744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255206+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:36:26.255206+0000","last_peered":"2022-07-03T10:36:26.255206+0000","last_clean":"2022-07-03T10:36:26.255206+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:36:26.255206+0000","last_undegraded":"2022-07-03T10:36:26.255206+0000","last_fullsized":"2022-07-03T10:36:26.255206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245693+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:36:26.245693+0000","last_peered":"2022-07-03T10:36:26.245693+0000","last_clean":"2022-07-03T10:36:26.245693+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:36:26.245693+0000","last_undegraded":"2022-07-03T10:36:26.245693+0000","last_fullsized":"2022-07-03T10:36:26.245693+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332452+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:26.332452+0000","last_peered":"2022-07-03T10:35:26.332452+0000","last_clean":"2022-07-03T10:35:26.332452+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:26.332452+0000","last_undegraded":"2022-07-03T10:35:26.332452+0000","last_fullsized":"2022-07-03T10:35:26.332452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247527+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:36:26.247527+0000","last_peered":"2022-07-03T10:36:26.247527+0000","last_clean":"2022-07-03T10:36:26.247527+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:36:26.247527+0000","last_undegraded":"2022-07-03T10:36:26.247527+0000","last_fullsized":"2022-07-03T10:36:26.247527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302670+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:26.302670+0000","last_peered":"2022-07-03T10:35:26.302670+0000","last_clean":"2022-07-03T10:35:26.302670+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:26.302670+0000","last_undegraded":"2022-07-03T10:35:26.302670+0000","last_fullsized":"2022-07-03T10:35:26.302670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347662+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:26.347662+0000","last_peered":"2022-07-03T10:35:26.347662+0000","last_clean":"2022-07-03T10:35:26.347662+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:26.347662+0000","last_undegraded":"2022-07-03T10:35:26.347662+0000","last_fullsized":"2022-07-03T10:35:26.347662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245672+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:36:26.245672+0000","last_peered":"2022-07-03T10:36:26.245672+0000","last_clean":"2022-07-03T10:36:26.245672+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:36:26.245672+0000","last_undegraded":"2022-07-03T10:36:26.245672+0000","last_fullsized":"2022-07-03T10:36:26.245672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347752+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:26.347752+0000","last_peered":"2022-07-03T10:35:26.347752+0000","last_clean":"2022-07-03T10:35:26.347752+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:26.347752+0000","last_undegraded":"2022-07-03T10:35:26.347752+0000","last_fullsized":"2022-07-03T10:35:26.347752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309497+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:26.309497+0000","last_peered":"2022-07-03T10:35:26.309497+0000","last_clean":"2022-07-03T10:35:26.309497+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:26.309497+0000","last_undegraded":"2022-07-03T10:35:26.309497+0000","last_fullsized":"2022-07-03T10:35:26.309497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.253049+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:36:26.253049+0000","last_peered":"2022-07-03T10:36:26.253049+0000","last_clean":"2022-07-03T10:36:26.253049+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:36:26.253049+0000","last_undegraded":"2022-07-03T10:36:26.253049+0000","last_fullsized":"2022-07-03T10:36:26.253049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'4","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.415059+0000","last_change":"2022-07-03T10:35:27.414221+0000","last_active":"2022-07-03T10:35:27.415059+0000","last_peered":"2022-07-03T10:35:27.415059+0000","last_clean":"2022-07-03T10:35:27.415059+0000","last_became_active":"2022-07-03T10:35:27.323296+0000","last_became_peered":"2022-07-03T10:35:27.323296+0000","last_unstale":"2022-07-03T10:35:27.415059+0000","last_undegraded":"2022-07-03T10:35:27.415059+0000","last_fullsized":"2022-07-03T10:35:27.415059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:01:20.773865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302521+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:26.302521+0000","last_peered":"2022-07-03T10:35:26.302521+0000","last_clean":"2022-07-03T10:35:26.302521+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:26.302521+0000","last_undegraded":"2022-07-03T10:35:26.302521+0000","last_fullsized":"2022-07-03T10:35:26.302521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260780+0000","last_change":"2022-07-03T10:35:27.325484+0000","last_active":"2022-07-03T10:36:26.260780+0000","last_peered":"2022-07-03T10:36:26.260780+0000","last_clean":"2022-07-03T10:36:26.260780+0000","last_became_active":"2022-07-03T10:35:27.324947+0000","last_became_peered":"2022-07-03T10:35:27.324947+0000","last_unstale":"2022-07-03T10:36:26.260780+0000","last_undegraded":"2022-07-03T10:36:26.260780+0000","last_fullsized":"2022-07-03T10:36:26.260780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:36.475275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309995+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:26.309995+0000","last_peered":"2022-07-03T10:35:26.309995+0000","last_clean":"2022-07-03T10:35:26.309995+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:26.309995+0000","last_undegraded":"2022-07-03T10:35:26.309995+0000","last_fullsized":"2022-07-03T10:35:26.309995+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.214344+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:36:26.214344+0000","last_peered":"2022-07-03T10:36:26.214344+0000","last_clean":"2022-07-03T10:36:26.214344+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:36:26.214344+0000","last_undegraded":"2022-07-03T10:36:26.214344+0000","last_fullsized":"2022-07-03T10:36:26.214344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341921+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:26.341921+0000","last_peered":"2022-07-03T10:35:26.341921+0000","last_clean":"2022-07-03T10:35:26.341921+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:26.341921+0000","last_undegraded":"2022-07-03T10:35:26.341921+0000","last_fullsized":"2022-07-03T10:35:26.341921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242220+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:36:26.242220+0000","last_peered":"2022-07-03T10:36:26.242220+0000","last_clean":"2022-07-03T10:36:26.242220+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:36:26.242220+0000","last_undegraded":"2022-07-03T10:36:26.242220+0000","last_fullsized":"2022-07-03T10:36:26.242220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303560+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:26.303560+0000","last_peered":"2022-07-03T10:35:26.303560+0000","last_clean":"2022-07-03T10:35:26.303560+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:26.303560+0000","last_undegraded":"2022-07-03T10:35:26.303560+0000","last_fullsized":"2022-07-03T10:35:26.303560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256486+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:36:26.256486+0000","last_peered":"2022-07-03T10:36:26.256486+0000","last_clean":"2022-07-03T10:36:26.256486+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:36:26.256486+0000","last_undegraded":"2022-07-03T10:36:26.256486+0000","last_fullsized":"2022-07-03T10:36:26.256486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309116+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:26.309116+0000","last_peered":"2022-07-03T10:35:26.309116+0000","last_clean":"2022-07-03T10:35:26.309116+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:26.309116+0000","last_undegraded":"2022-07-03T10:35:26.309116+0000","last_fullsized":"2022-07-03T10:35:26.309116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246522+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:36:26.246522+0000","last_peered":"2022-07-03T10:36:26.246522+0000","last_clean":"2022-07-03T10:36:26.246522+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:36:26.246522+0000","last_undegraded":"2022-07-03T10:36:26.246522+0000","last_fullsized":"2022-07-03T10:36:26.246522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.312207+0000","last_change":"2022-07-03T10:35:27.312207+0000","last_active":"2022-07-03T10:35:27.312207+0000","last_peered":"2022-07-03T10:35:27.312207+0000","last_clean":"2022-07-03T10:35:27.312207+0000","last_became_active":"2022-07-03T10:35:27.311839+0000","last_became_peered":"2022-07-03T10:35:27.311839+0000","last_unstale":"2022-07-03T10:35:27.312207+0000","last_undegraded":"2022-07-03T10:35:27.312207+0000","last_fullsized":"2022-07-03T10:35:27.312207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:21:12.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246912+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:36:26.246912+0000","last_peered":"2022-07-03T10:36:26.246912+0000","last_clean":"2022-07-03T10:36:26.246912+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:36:26.246912+0000","last_undegraded":"2022-07-03T10:36:26.246912+0000","last_fullsized":"2022-07-03T10:36:26.246912+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305852+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:26.305852+0000","last_peered":"2022-07-03T10:35:26.305852+0000","last_clean":"2022-07-03T10:35:26.305852+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:26.305852+0000","last_undegraded":"2022-07-03T10:35:26.305852+0000","last_fullsized":"2022-07-03T10:35:26.305852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242322+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:36:26.242322+0000","last_peered":"2022-07-03T10:36:26.242322+0000","last_clean":"2022-07-03T10:36:26.242322+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:36:26.242322+0000","last_undegraded":"2022-07-03T10:36:26.242322+0000","last_fullsized":"2022-07-03T10:36:26.242322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330620+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:26.330620+0000","last_peered":"2022-07-03T10:35:26.330620+0000","last_clean":"2022-07-03T10:35:26.330620+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:26.330620+0000","last_undegraded":"2022-07-03T10:35:26.330620+0000","last_fullsized":"2022-07-03T10:35:26.330620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237445+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:36:26.237445+0000","last_peered":"2022-07-03T10:36:26.237445+0000","last_clean":"2022-07-03T10:36:26.237445+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:36:26.237445+0000","last_undegraded":"2022-07-03T10:36:26.237445+0000","last_fullsized":"2022-07-03T10:36:26.237445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242972+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:36:26.242972+0000","last_peered":"2022-07-03T10:36:26.242972+0000","last_clean":"2022-07-03T10:36:26.242972+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:36:26.242972+0000","last_undegraded":"2022-07-03T10:36:26.242972+0000","last_fullsized":"2022-07-03T10:36:26.242972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250077+0000","last_change":"2022-07-03T10:35:27.313043+0000","last_active":"2022-07-03T10:36:26.250077+0000","last_peered":"2022-07-03T10:36:26.250077+0000","last_clean":"2022-07-03T10:36:26.250077+0000","last_became_active":"2022-07-03T10:35:27.312553+0000","last_became_peered":"2022-07-03T10:35:27.312553+0000","last_unstale":"2022-07-03T10:36:26.250077+0000","last_undegraded":"2022-07-03T10:36:26.250077+0000","last_fullsized":"2022-07-03T10:36:26.250077+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:05:28.297497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256573+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:36:26.256573+0000","last_peered":"2022-07-03T10:36:26.256573+0000","last_clean":"2022-07-03T10:36:26.256573+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:36:26.256573+0000","last_undegraded":"2022-07-03T10:36:26.256573+0000","last_fullsized":"2022-07-03T10:36:26.256573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259834+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:36:26.259834+0000","last_peered":"2022-07-03T10:36:26.259834+0000","last_clean":"2022-07-03T10:36:26.259834+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:36:26.259834+0000","last_undegraded":"2022-07-03T10:36:26.259834+0000","last_fullsized":"2022-07-03T10:36:26.259834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.249554+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:36:26.249554+0000","last_peered":"2022-07-03T10:36:26.249554+0000","last_clean":"2022-07-03T10:36:26.249554+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:36:26.249554+0000","last_undegraded":"2022-07-03T10:36:26.249554+0000","last_fullsized":"2022-07-03T10:36:26.249554+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241377+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:36:26.241377+0000","last_peered":"2022-07-03T10:36:26.241377+0000","last_clean":"2022-07-03T10:36:26.241377+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:36:26.241377+0000","last_undegraded":"2022-07-03T10:36:26.241377+0000","last_fullsized":"2022-07-03T10:36:26.241377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242254+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:36:26.242254+0000","last_peered":"2022-07-03T10:36:26.242254+0000","last_clean":"2022-07-03T10:36:26.242254+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:36:26.242254+0000","last_undegraded":"2022-07-03T10:36:26.242254+0000","last_fullsized":"2022-07-03T10:36:26.242254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.243944+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:36:26.243944+0000","last_peered":"2022-07-03T10:36:26.243944+0000","last_clean":"2022-07-03T10:36:26.243944+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:36:26.243944+0000","last_undegraded":"2022-07-03T10:36:26.243944+0000","last_fullsized":"2022-07-03T10:36:26.243944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255263+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:36:26.255263+0000","last_peered":"2022-07-03T10:36:26.255263+0000","last_clean":"2022-07-03T10:36:26.255263+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:36:26.255263+0000","last_undegraded":"2022-07-03T10:36:26.255263+0000","last_fullsized":"2022-07-03T10:36:26.255263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255288+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:36:26.255288+0000","last_peered":"2022-07-03T10:36:26.255288+0000","last_clean":"2022-07-03T10:36:26.255288+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:36:26.255288+0000","last_undegraded":"2022-07-03T10:36:26.255288+0000","last_fullsized":"2022-07-03T10:36:26.255288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258181+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:36:26.258181+0000","last_peered":"2022-07-03T10:36:26.258181+0000","last_clean":"2022-07-03T10:36:26.258181+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:36:26.258181+0000","last_undegraded":"2022-07-03T10:36:26.258181+0000","last_fullsized":"2022-07-03T10:36:26.258181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241416+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:36:26.241416+0000","last_peered":"2022-07-03T10:36:26.241416+0000","last_clean":"2022-07-03T10:36:26.241416+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:36:26.241416+0000","last_undegraded":"2022-07-03T10:36:26.241416+0000","last_fullsized":"2022-07-03T10:36:26.241416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54826,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":210,"num_read_kb":160,"num_write":75,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":164478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547233,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":436583,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.501,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65800000000000003}]},{"osd":4,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.58799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.875,"5min":1.026,"15min":1.026},"last":0.625}]},{"osd":5,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.623},"min":{"1min":0.41899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92300000000000004,"5min":1.151,"15min":1.151},"last":0.68300000000000005}]},{"osd":6,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.83399999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.76000000000000001,"15min":0.76100000000000001},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.96799999999999997,"5min":1.075,"15min":1.075},"last":0.69399999999999995}]},{"osd":8,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72699999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.80600000000000005,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.044,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.77600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.78100000000000003,"15min":0.77700000000000002},"min":{"1min":0.63900000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.115,"15min":1.115},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.61399999999999999,"5min":0.496,"15min":0.496},"max":{"1min":0.94999999999999996,"5min":1.179,"15min":1.179},"last":0.63700000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6368,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738784,"statfs":{"total":95995035648,"available":95988514816,"internally_reserved":0,"allocated":1007616,"data_stored":1080955,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.437},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.52600000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.55500000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.66500000000000004,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71899999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65000000000000002}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.75700000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85899999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.77600000000000002,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.46400000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74199999999999999,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.84699999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.57899999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.879,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.81499999999999995},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.67500000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.72799999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972395,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":884736,"data_stored":437115,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.80500000000000005,"15min":0.71199999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6359999999999999,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.442},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.89800000000000002,"15min":0.77800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0720000000000001,"5min":11.582000000000001,"15min":11.582000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.93000000000000005,"15min":0.84999999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.905,"5min":11.933,"15min":11.933},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.55600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.5669999999999999,"5min":11.411,"15min":11.411},"last":0.89000000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.94099999999999995,"15min":0.85499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.944,"5min":11.887,"15min":11.887},"last":0.748},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.94699999999999995,"15min":0.89500000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6080000000000001,"5min":11.331,"15min":11.331},"last":0.95099999999999996}]},{"osd":3,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.85699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2490000000000001,"5min":9.8859999999999992,"15min":9.8859999999999992},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.95399999999999996,"15min":0.84399999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8009999999999999,"5min":12.041,"15min":12.041},"last":0.60899999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.016,"15min":0.97599999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":11.369,"15min":11.369},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":1,"15min":0.95199999999999996},"min":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95299999999999996,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":9.734},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":11.968}]},{"osd":6,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.0880000000000001,"15min":1.028},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.869,"5min":12.311,"15min":12.311},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1040000000000001,"15min":1.0700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.0419999999999998,"5min":11.534000000000001,"15min":11.534000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.0860000000000001,"15min":1.0529999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.272,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.1419999999999999,"15min":1.0860000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0019999999999998,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.78400000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.147,"15min":1.125},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6639999999999999,"5min":11.441000000000001,"15min":11.441000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.1160000000000001,"15min":1.0900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1839999999999999,"5min":9.8539999999999992,"15min":9.8539999999999992},"last":0.90500000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.1459999999999999,"15min":1.0920000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":11.474,"5min":11.474,"15min":11.474},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.107,"15min":1.0549999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":9.7729999999999997,"5min":9.7729999999999997,"15min":9.7729999999999997},"last":0.93500000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.008}]}]},{"osd":8,"up_from":54,"seq":231928234031,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738900,"statfs":{"total":95995035648,"available":95988633600,"internally_reserved":0,"allocated":888832,"data_stored":437137,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67200000000000004,"15min":0.63},"min":{"1min":0.47199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.871,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81799999999999995,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.47599999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.57199999999999995,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80700000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72999999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61299999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.437},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63700000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.52700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.62}]},{"osd":4,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57499999999999996,"15min":0.54800000000000004},"min":{"1min":0.45200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67900000000000005,"15min":0.621},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83699999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.77200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.69999999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.75900000000000001,"15min":0.70299999999999996},"min":{"1min":0.47699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.357,"5min":4.6959999999999997,"15min":4.6959999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.73399999999999999},"min":{"1min":0.56499999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67000000000000004,"15min":0.65400000000000003},"min":{"1min":0.51200000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.57499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.82799999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.75900000000000001},"min":{"1min":0.64000000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.77200000000000002,"15min":0.76800000000000002},"min":{"1min":0.63700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.997,"5min":1.117,"15min":1.117},"last":0.82399999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.51300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80600000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83999999999999997}]}]},{"osd":7,"up_from":47,"seq":201863462963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":1027606,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.37},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.373},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.309,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.45900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.626}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":1.222,"15min":1.222},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48999999999999999,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.40200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94799999999999995,"5min":5.077,"15min":5.077},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70299999999999996,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.67600000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:36:09 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66400000000000003},"min":{"1min":0.48199999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.96699999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.54400000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:36:42 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79100000000000004,"15min":0.79700000000000004},"min":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.002,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.94699999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.79900000000000004,"15min":0.81100000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.034,"15min":1.034},"last":0.70599999999999996}]},{"osd":11,"last update":"Sun Jul 3 10:36:04 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75}]}]},{"osd":6,"up_from":43,"seq":184683593782,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":436672,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.56799999999999995,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.93899999999999995}]},{"osd":1,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.70199999999999996,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.55100000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71899999999999997,"15min":0.69499999999999995},"min":{"1min":0.56499999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.748,"15min":0.71899999999999997},"min":{"1min":0.53600000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70099999999999996,"15min":0.68100000000000005},"min":{"1min":0.437,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.78200000000000003,"15min":0.72599999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.83899999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76700000000000002,"15min":0.755},"min":{"1min":0.51600000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.137,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81100000000000005,"15min":0.79300000000000004},"min":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3129999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.95399999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.82699999999999996,"15min":0.81100000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0669999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81799999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:35:52 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86299999999999999,"15min":0.84099999999999997},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2869999999999999,"5min":1.333,"15min":1.333},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.878,"15min":0.86099999999999999},"min":{"1min":0.70399999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.008}]},{"osd":8,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86499999999999999,"15min":0.85199999999999998},"min":{"1min":0.65400000000000003,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.121,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.879}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96599999999999997},"min":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.03},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.96499999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.89600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.98299999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1279999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.004,"15min":1.01},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.069}]}]},{"osd":1,"up_from":12,"seq":51539607630,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7268,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737884,"statfs":{"total":95995035648,"available":95987593216,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.38700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83899999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.57599999999999996},"min":{"1min":0.40899999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69799999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:47 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.58899999999999997,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.879,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.54800000000000004},"min":{"1min":0.48599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.86699999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:24 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.55500000000000005},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.55900000000000005},"min":{"1min":0.48499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83899999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.41399999999999998},"min":{"1min":0.39900000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77300000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.47399999999999998,"15min":0.41799999999999998},"min":{"1min":0.40200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79900000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.73799999999999999}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":1.044,"15min":1.044},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.60999999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88800000000000001,"5min":1.028,"15min":1.028},"last":0.91300000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.872,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.89300000000000002,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:51 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.70099999999999996,"15min":0.68500000000000005},"min":{"1min":0.53200000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98099999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.88},{"interface":"front","average":{"1min":0.755,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.57399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.88900000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79700000000000004,"15min":0.79400000000000004},"min":{"1min":0.626,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.68500000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0609999999999999,"5min":1.103,"15min":1.103},"last":0.996}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90800000000000003,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.752,"15min":0.748},"min":{"1min":0.53800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.89900000000000002,"5min":1.069,"15min":1.069},"last":0.96099999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":1.0349999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7080,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738072,"statfs":{"total":95995035648,"available":95987785728,"internally_reserved":0,"allocated":884736,"data_stored":436639,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.63700000000000001,"15min":0.56399999999999995},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":0.94499999999999995},"last":0.57399999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.68100000000000005,"15min":0.63100000000000001},"min":{"1min":0.52900000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.496}]},{"osd":3,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.39500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.73199999999999998,"15min":0.68000000000000005},"min":{"1min":0.51600000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.69099999999999995,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:35:59 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.73699999999999999,"15min":0.71399999999999997},"min":{"1min":0.59899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.45800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80800000000000005,"15min":0.76100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":0.752},{"interface":"front","average":{"1min":0.879,"5min":0.79200000000000004,"15min":0.75800000000000001},"min":{"1min":0.54000000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.89600000000000002,"15min":0.88800000000000001},"min":{"1min":0.64500000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.85399999999999998,"15min":0.84999999999999998},"min":{"1min":0.59299999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.48999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.84199999999999997,"15min":0.83499999999999996},"min":{"1min":0.64200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97399999999999998,"15min":0.97099999999999997},"min":{"1min":0.66500000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.76800000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.90200000000000002,"15min":0.90700000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72399999999999998}]},{"osd":11,"last update":"Sun Jul 3 10:36:15 2022","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.95499999999999996,"15min":0.93700000000000006},"min":{"1min":0.71299999999999997,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.88500000000000001,"15min":0.878},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.63300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476812,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7332,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737820,"statfs":{"total":95995035648,"available":95987527680,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.496}]},{"osd":1,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69599999999999995,"5min":0.874,"15min":0.874},"last":0.62}]},{"osd":3,"last update":"Sun Jul 3 10:36:16 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83299999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58199999999999996}]},{"osd":4,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.71199999999999997}]},{"osd":5,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.501,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63500000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:36:32 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67100000000000004,"15min":0.65100000000000002},"min":{"1min":0.52200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.88900000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66200000000000003,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83599999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78400000000000003,"5min":0.871,"15min":0.871},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.42699999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.69799999999999995}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.746,"15min":0.74199999999999999},"min":{"1min":0.56799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.88100000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.58399999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.79900000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72899999999999998}]}]},{"osd":3,"up_from":23,"seq":98784247877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":897024,"data_stored":436899,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.51600000000000001,"15min":0.44700000000000001},"min":{"1min":0.34100000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.9609999999999999,"5min":3.9609999999999999,"15min":3.9609999999999999},"last":0.433},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67200000000000004,"15min":0.60299999999999998},"min":{"1min":0.46500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95199999999999996,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.21,"5min":2.581,"15min":2.581},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.903,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.80500000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76900000000000002,"15min":0.66400000000000003},"min":{"1min":0.58599999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.82099999999999995,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90700000000000003,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.657,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81100000000000005,"15min":0.76100000000000001},"min":{"1min":0.52100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.2519999999999998,"5min":2.46,"15min":2.46},"last":0.86399999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.745},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90500000000000003,"15min":0.91200000000000003},"min":{"1min":0.65400000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.78900000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.91400000000000003,"15min":0.96299999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.131,"5min":1.256,"15min":1.256},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.95499999999999996,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.61199999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.71499999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.878}]}]},{"osd":4,"up_from":30,"seq":128849018945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6312,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738840,"statfs":{"total":95995035648,"available":95988572160,"internally_reserved":0,"allocated":884736,"data_stored":436571,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.59899999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58899999999999997,"15min":0.55900000000000005},"min":{"1min":0.44800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.46899999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.70899999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.56499999999999995},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73199999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.75},{"interface":"front","average":{"1min":0.628,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.48199999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.76900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.57499999999999996},"min":{"1min":0.46400000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79000000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56399999999999995},"min":{"1min":0.42699999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.84399999999999997}]},{"osd":6,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.53300000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.67000000000000004,"15min":0.65300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.76700000000000002,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.64700000000000002},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66800000000000004,"15min":0.64500000000000002},"min":{"1min":0.53900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":1,"15min":1},"last":0.86899999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.79100000000000004},"min":{"1min":0.56200000000000006,"5min":0.497,"15min":0.497},"max":{"1min":0.89100000000000001,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.96499999999999997,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.97299999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.72999999999999998,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92100000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.91300000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90400000000000003}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.98299999999999998}]}]},{"osd":5,"up_from":35,"seq":150323855420,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":946176,"data_stored":1027222,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.64000000000000001,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"last":0.77700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.32900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.433,"15min":0.432},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56499999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.69799999999999995,"15min":0.627},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.21,"5min":7.21,"15min":7.21},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.02},"last":0.64500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59799999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73399999999999999,"15min":0.69599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86699999999999999,"5min":5.476,"15min":5.476},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.72299999999999998},"min":{"1min":0.58599999999999997,"5min":0.505,"15min":0.505},"max":{"1min":0.91000000000000003,"5min":3.27,"15min":3.27},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":0.875,"15min":0.875},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66200000000000003,"15min":0.64600000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56200000000000006}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.50700000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.77200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95999999999999996,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82199999999999995,"15min":0.78600000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.90800000000000003,"5min":5.4240000000000004,"15min":5.4240000000000004},"last":0.75800000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.66900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":54238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:36:48.599 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-03T10:36:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:48 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2042126472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:48 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/375184658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:48 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2042126472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:48 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/375184658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:48.829 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 -- 172.21.15.101:0/2759353208 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 msgr2=0x7fe9500feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 --2- 172.21.15.101:0/2759353208 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe9500feaa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe944009900 tx=0x7fe9440091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 -- 172.21.15.101:0/2759353208 shutdown_connections 2022-07-03T10:36:48.830 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 --2- 172.21.15.101:0/2759353208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 0x7fe9501058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 --2- 172.21.15.101:0/2759353208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe9500fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 --2- 172.21.15.101:0/2759353208 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe9500feaa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 -- 172.21.15.101:0/2759353208 >> 172.21.15.101:0/2759353208 conn(0x7fe9500f8b20 msgr2=0x7fe9500faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:48.831 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 -- 172.21.15.101:0/2759353208 shutdown_connections 2022-07-03T10:36:48.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.823+0000 7fe94ffff700 1 -- 172.21.15.101:0/2759353208 wait complete. 2022-07-03T10:36:48.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.824+0000 7fe94ffff700 1 Processor -- start 2022-07-03T10:36:48.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.824+0000 7fe94ffff700 1 -- start start 2022-07-03T10:36:48.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.824+0000 7fe94ffff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.832 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.824+0000 7fe94ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe95010e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.824+0000 7fe94ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 0x7fe950115990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94ffff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe950115f10 con 0x7fe9500fe650 2022-07-03T10:36:48.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94ffff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe950116050 con 0x7fe9500fd3d0 2022-07-03T10:36:48.833 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94ffff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe950116210 con 0x7fe9500fefe0 2022-07-03T10:36:48.834 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.834 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51638/0 (socket says 172.21.15.101:51638) 2022-07-03T10:36:48.834 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 -- 172.21.15.101:0/1532175190 learned_addr learned my addr 172.21.15.101:0/1532175190 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:48.834 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94f7fe700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 0x7fe950115990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.835 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94e7fc700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe95010e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.835 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 -- 172.21.15.101:0/1532175190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 msgr2=0x7fe950115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.835 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 0x7fe950115990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.835 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 -- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 msgr2=0x7fe95010e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:48.835 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe95010e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:48.836 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.825+0000 7fe94effd700 1 -- 172.21.15.101:0/1532175190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944009440 con 0x7fe9500fd3d0 2022-07-03T10:36:48.836 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe94effd700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe94000ea50 tx=0x7fe940005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:48.836 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940004020 con 0x7fe9500fd3d0 2022-07-03T10:36:48.836 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe94000cc70 con 0x7fe9500fd3d0 2022-07-03T10:36:48.836 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9501164f0 con 0x7fe9500fd3d0 2022-07-03T10:36:48.837 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9400039d0 con 0x7fe9500fd3d0 2022-07-03T10:36:48.837 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.826+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe950116a80 con 0x7fe9500fd3d0 2022-07-03T10:36:48.837 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.827+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe950108bf0 con 0x7fe9500fd3d0 2022-07-03T10:36:48.838 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.829+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fe94000c040 con 0x7fe9500fd3d0 2022-07-03T10:36:48.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.830+0000 7fe9377fe700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 0x7fe9380991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:48.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.830+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fe940014070 con 0x7fe9500fd3d0 2022-07-03T10:36:48.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.831+0000 7fe94e7fc700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 0x7fe9380991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:48.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.831+0000 7fe94e7fc700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 0x7fe9380991d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe9440092d0 tx=0x7fe9440091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:48.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:48.832+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fe94008ccb0 con 0x7fe9500fd3d0 2022-07-03T10:36:49.019 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.017+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe95005e960 con 0x7fe938096d20 2022-07-03T10:36:49.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.028+0000 7fe9377fe700 1 -- 172.21.15.101:0/1532175190 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343667 (secure 0 0 0) 0x7fe95005e960 con 0x7fe938096d20 2022-07-03T10:36:49.037 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:49.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 msgr2=0x7fe9380991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 0x7fe9380991d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe9440092d0 tx=0x7fe9440091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 msgr2=0x7fe95010e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe94000ea50 tx=0x7fe940005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 shutdown_connections 2022-07-03T10:36:49.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fe938096d20 0x7fe9380991d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9500fefe0 0x7fe950115990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9500fd3d0 0x7fe95010e3e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 --2- 172.21.15.101:0/1532175190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9500fe650 0x7fe95010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 >> 172.21.15.101:0/1532175190 conn(0x7fe9500f8b20 msgr2=0x7fe950101880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:49.045 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 shutdown_connections 2022-07-03T10:36:49.045 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.033+0000 7fe94ffff700 1 -- 172.21.15.101:0/1532175190 wait complete. 2022-07-03T10:36:49.045 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:36:49.062 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2022-07-03T10:36:47.964118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645440,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":336,"num_read_kb":269,"num_write":258,"num_write_kb":3018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":705,"ondisk_log_size":705,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78288,"kb_used_data":10736,"kb_used_omap":0,"kb_used_meta":67456,"kb_avail":1124863536,"statfs":{"total":1151940427776,"available":1151860260864,"internally_reserved":0,"allocated":10993664,"data_stored":7174585,"data_compressed":176772,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":69074944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":322,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003008"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244527+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:36:26.244527+0000","last_peered":"2022-07-03T10:36:26.244527+0000","last_clean":"2022-07-03T10:36:26.244527+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:36:26.244527+0000","last_undegraded":"2022-07-03T10:36:26.244527+0000","last_fullsized":"2022-07-03T10:36:26.244527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.261640+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:36:26.261640+0000","last_peered":"2022-07-03T10:36:26.261640+0000","last_clean":"2022-07-03T10:36:26.261640+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:36:26.261640+0000","last_undegraded":"2022-07-03T10:36:26.261640+0000","last_fullsized":"2022-07-03T10:36:26.261640+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260480+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:36:26.260480+0000","last_peered":"2022-07-03T10:36:26.260480+0000","last_clean":"2022-07-03T10:36:26.260480+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:36:26.260480+0000","last_undegraded":"2022-07-03T10:36:26.260480+0000","last_fullsized":"2022-07-03T10:36:26.260480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248817+0000","last_change":"2022-07-03T10:35:27.315233+0000","last_active":"2022-07-03T10:36:26.248817+0000","last_peered":"2022-07-03T10:36:26.248817+0000","last_clean":"2022-07-03T10:36:26.248817+0000","last_became_active":"2022-07-03T10:35:27.314807+0000","last_became_peered":"2022-07-03T10:35:27.314807+0000","last_unstale":"2022-07-03T10:36:26.248817+0000","last_undegraded":"2022-07-03T10:36:26.248817+0000","last_fullsized":"2022-07-03T10:36:26.248817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:37:28.362457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240860+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:36:26.240860+0000","last_peered":"2022-07-03T10:36:26.240860+0000","last_clean":"2022-07-03T10:36:26.240860+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:36:26.240860+0000","last_undegraded":"2022-07-03T10:36:26.240860+0000","last_fullsized":"2022-07-03T10:36:26.240860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234621+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:36:26.234621+0000","last_peered":"2022-07-03T10:36:26.234621+0000","last_clean":"2022-07-03T10:36:26.234621+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:36:26.234621+0000","last_undegraded":"2022-07-03T10:36:26.234621+0000","last_fullsized":"2022-07-03T10:36:26.234621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247958+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:36:26.247958+0000","last_peered":"2022-07-03T10:36:26.247958+0000","last_clean":"2022-07-03T10:36:26.247958+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:36:26.247958+0000","last_undegraded":"2022-07-03T10:36:26.247958+0000","last_fullsized":"2022-07-03T10:36:26.247958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.252954+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:36:26.252954+0000","last_peered":"2022-07-03T10:36:26.252954+0000","last_clean":"2022-07-03T10:36:26.252954+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:36:26.252954+0000","last_undegraded":"2022-07-03T10:36:26.252954+0000","last_fullsized":"2022-07-03T10:36:26.252954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251354+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:36:26.251354+0000","last_peered":"2022-07-03T10:36:26.251354+0000","last_clean":"2022-07-03T10:36:26.251354+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:36:26.251354+0000","last_undegraded":"2022-07-03T10:36:26.251354+0000","last_fullsized":"2022-07-03T10:36:26.251354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241581+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:36:26.241581+0000","last_peered":"2022-07-03T10:36:26.241581+0000","last_clean":"2022-07-03T10:36:26.241581+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:36:26.241581+0000","last_undegraded":"2022-07-03T10:36:26.241581+0000","last_fullsized":"2022-07-03T10:36:26.241581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246804+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:36:26.246804+0000","last_peered":"2022-07-03T10:36:26.246804+0000","last_clean":"2022-07-03T10:36:26.246804+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:36:26.246804+0000","last_undegraded":"2022-07-03T10:36:26.246804+0000","last_fullsized":"2022-07-03T10:36:26.246804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237345+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:36:26.237345+0000","last_peered":"2022-07-03T10:36:26.237345+0000","last_clean":"2022-07-03T10:36:26.237345+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.237345+0000","last_undegraded":"2022-07-03T10:36:26.237345+0000","last_fullsized":"2022-07-03T10:36:26.237345+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245178+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:36:26.245178+0000","last_peered":"2022-07-03T10:36:26.245178+0000","last_clean":"2022-07-03T10:36:26.245178+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:36:26.245178+0000","last_undegraded":"2022-07-03T10:36:26.245178+0000","last_fullsized":"2022-07-03T10:36:26.245178+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.262152+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:36:26.262152+0000","last_peered":"2022-07-03T10:36:26.262152+0000","last_clean":"2022-07-03T10:36:26.262152+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:36:26.262152+0000","last_undegraded":"2022-07-03T10:36:26.262152+0000","last_fullsized":"2022-07-03T10:36:26.262152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259927+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:36:26.259927+0000","last_peered":"2022-07-03T10:36:26.259927+0000","last_clean":"2022-07-03T10:36:26.259927+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:36:26.259927+0000","last_undegraded":"2022-07-03T10:36:26.259927+0000","last_fullsized":"2022-07-03T10:36:26.259927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256218+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:36:26.256218+0000","last_peered":"2022-07-03T10:36:26.256218+0000","last_clean":"2022-07-03T10:36:26.256218+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:36:26.256218+0000","last_undegraded":"2022-07-03T10:36:26.256218+0000","last_fullsized":"2022-07-03T10:36:26.256218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248902+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:36:26.248902+0000","last_peered":"2022-07-03T10:36:26.248902+0000","last_clean":"2022-07-03T10:36:26.248902+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:36:26.248902+0000","last_undegraded":"2022-07-03T10:36:26.248902+0000","last_fullsized":"2022-07-03T10:36:26.248902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248745+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:36:26.248745+0000","last_peered":"2022-07-03T10:36:26.248745+0000","last_clean":"2022-07-03T10:36:26.248745+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:36:26.248745+0000","last_undegraded":"2022-07-03T10:36:26.248745+0000","last_fullsized":"2022-07-03T10:36:26.248745+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246317+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:36:26.246317+0000","last_peered":"2022-07-03T10:36:26.246317+0000","last_clean":"2022-07-03T10:36:26.246317+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:36:26.246317+0000","last_undegraded":"2022-07-03T10:36:26.246317+0000","last_fullsized":"2022-07-03T10:36:26.246317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325415+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:26.325415+0000","last_peered":"2022-07-03T10:35:26.325415+0000","last_clean":"2022-07-03T10:35:26.325415+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:26.325415+0000","last_undegraded":"2022-07-03T10:35:26.325415+0000","last_fullsized":"2022-07-03T10:35:26.325415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242008+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:36:26.242008+0000","last_peered":"2022-07-03T10:36:26.242008+0000","last_clean":"2022-07-03T10:36:26.242008+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:36:26.242008+0000","last_undegraded":"2022-07-03T10:36:26.242008+0000","last_fullsized":"2022-07-03T10:36:26.242008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343531+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:26.343531+0000","last_peered":"2022-07-03T10:35:26.343531+0000","last_clean":"2022-07-03T10:35:26.343531+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:26.343531+0000","last_undegraded":"2022-07-03T10:35:26.343531+0000","last_fullsized":"2022-07-03T10:35:26.343531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260621+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:36:26.260621+0000","last_peered":"2022-07-03T10:36:26.260621+0000","last_clean":"2022-07-03T10:36:26.260621+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:36:26.260621+0000","last_undegraded":"2022-07-03T10:36:26.260621+0000","last_fullsized":"2022-07-03T10:36:26.260621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309637+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:26.309637+0000","last_peered":"2022-07-03T10:35:26.309637+0000","last_clean":"2022-07-03T10:35:26.309637+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:26.309637+0000","last_undegraded":"2022-07-03T10:35:26.309637+0000","last_fullsized":"2022-07-03T10:35:26.309637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250215+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:36:26.250215+0000","last_peered":"2022-07-03T10:36:26.250215+0000","last_clean":"2022-07-03T10:36:26.250215+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:36:26.250215+0000","last_undegraded":"2022-07-03T10:36:26.250215+0000","last_fullsized":"2022-07-03T10:36:26.250215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309394+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:26.309394+0000","last_peered":"2022-07-03T10:35:26.309394+0000","last_clean":"2022-07-03T10:35:26.309394+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:26.309394+0000","last_undegraded":"2022-07-03T10:35:26.309394+0000","last_fullsized":"2022-07-03T10:35:26.309394+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248977+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:36:26.248977+0000","last_peered":"2022-07-03T10:36:26.248977+0000","last_clean":"2022-07-03T10:36:26.248977+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:36:26.248977+0000","last_undegraded":"2022-07-03T10:36:26.248977+0000","last_fullsized":"2022-07-03T10:36:26.248977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305894+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:26.305894+0000","last_peered":"2022-07-03T10:35:26.305894+0000","last_clean":"2022-07-03T10:35:26.305894+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:26.305894+0000","last_undegraded":"2022-07-03T10:35:26.305894+0000","last_fullsized":"2022-07-03T10:35:26.305894+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234357+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:36:26.234357+0000","last_peered":"2022-07-03T10:36:26.234357+0000","last_clean":"2022-07-03T10:36:26.234357+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:36:26.234357+0000","last_undegraded":"2022-07-03T10:36:26.234357+0000","last_fullsized":"2022-07-03T10:36:26.234357+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309034+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:26.309034+0000","last_peered":"2022-07-03T10:35:26.309034+0000","last_clean":"2022-07-03T10:35:26.309034+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:26.309034+0000","last_undegraded":"2022-07-03T10:35:26.309034+0000","last_fullsized":"2022-07-03T10:35:26.309034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256408+0000","last_change":"2022-07-03T10:35:27.324978+0000","last_active":"2022-07-03T10:36:26.256408+0000","last_peered":"2022-07-03T10:36:26.256408+0000","last_clean":"2022-07-03T10:36:26.256408+0000","last_became_active":"2022-07-03T10:35:27.324570+0000","last_became_peered":"2022-07-03T10:35:27.324570+0000","last_unstale":"2022-07-03T10:36:26.256408+0000","last_undegraded":"2022-07-03T10:36:26.256408+0000","last_fullsized":"2022-07-03T10:36:26.256408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:18:06.187457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332535+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:26.332535+0000","last_peered":"2022-07-03T10:35:26.332535+0000","last_clean":"2022-07-03T10:35:26.332535+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:26.332535+0000","last_undegraded":"2022-07-03T10:35:26.332535+0000","last_fullsized":"2022-07-03T10:35:26.332535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.254411+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:36:26.254411+0000","last_peered":"2022-07-03T10:36:26.254411+0000","last_clean":"2022-07-03T10:36:26.254411+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:36:26.254411+0000","last_undegraded":"2022-07-03T10:36:26.254411+0000","last_fullsized":"2022-07-03T10:36:26.254411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305427+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:26.305427+0000","last_peered":"2022-07-03T10:35:26.305427+0000","last_clean":"2022-07-03T10:35:26.305427+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:26.305427+0000","last_undegraded":"2022-07-03T10:35:26.305427+0000","last_fullsized":"2022-07-03T10:35:26.305427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255985+0000","last_change":"2022-07-03T10:35:27.314929+0000","last_active":"2022-07-03T10:36:26.255985+0000","last_peered":"2022-07-03T10:36:26.255985+0000","last_clean":"2022-07-03T10:36:26.255985+0000","last_became_active":"2022-07-03T10:35:27.314542+0000","last_became_peered":"2022-07-03T10:35:27.314542+0000","last_unstale":"2022-07-03T10:36:26.255985+0000","last_undegraded":"2022-07-03T10:36:26.255985+0000","last_fullsized":"2022-07-03T10:36:26.255985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:50:25.278127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309250+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:26.309250+0000","last_peered":"2022-07-03T10:35:26.309250+0000","last_clean":"2022-07-03T10:35:26.309250+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:26.309250+0000","last_undegraded":"2022-07-03T10:35:26.309250+0000","last_fullsized":"2022-07-03T10:35:26.309250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251048+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:36:26.251048+0000","last_peered":"2022-07-03T10:36:26.251048+0000","last_clean":"2022-07-03T10:36:26.251048+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:36:26.251048+0000","last_undegraded":"2022-07-03T10:36:26.251048+0000","last_fullsized":"2022-07-03T10:36:26.251048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347933+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:26.347933+0000","last_peered":"2022-07-03T10:35:26.347933+0000","last_clean":"2022-07-03T10:35:26.347933+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:26.347933+0000","last_undegraded":"2022-07-03T10:35:26.347933+0000","last_fullsized":"2022-07-03T10:35:26.347933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258203+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:36:26.258203+0000","last_peered":"2022-07-03T10:36:26.258203+0000","last_clean":"2022-07-03T10:36:26.258203+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:36:26.258203+0000","last_undegraded":"2022-07-03T10:36:26.258203+0000","last_fullsized":"2022-07-03T10:36:26.258203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306107+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:26.306107+0000","last_peered":"2022-07-03T10:35:26.306107+0000","last_clean":"2022-07-03T10:35:26.306107+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:26.306107+0000","last_undegraded":"2022-07-03T10:35:26.306107+0000","last_fullsized":"2022-07-03T10:35:26.306107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250505+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:36:26.250505+0000","last_peered":"2022-07-03T10:36:26.250505+0000","last_clean":"2022-07-03T10:36:26.250505+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.250505+0000","last_undegraded":"2022-07-03T10:36:26.250505+0000","last_fullsized":"2022-07-03T10:36:26.250505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'30","reported_seq":248,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:44.057423+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:36:44.057423+0000","last_peered":"2022-07-03T10:36:44.057423+0000","last_clean":"2022-07-03T10:36:44.057423+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:36:44.057423+0000","last_undegraded":"2022-07-03T10:36:44.057423+0000","last_fullsized":"2022-07-03T10:36:44.057423+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54044,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":205,"num_read_kb":155,"num_write":30,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.235656+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:36:26.235656+0000","last_peered":"2022-07-03T10:36:26.235656+0000","last_clean":"2022-07-03T10:36:26.235656+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:36:26.235656+0000","last_undegraded":"2022-07-03T10:36:26.235656+0000","last_fullsized":"2022-07-03T10:36:26.235656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.326689+0000","last_change":"2022-07-03T10:35:27.326689+0000","last_active":"2022-07-03T10:35:27.326689+0000","last_peered":"2022-07-03T10:35:27.326689+0000","last_clean":"2022-07-03T10:35:27.326689+0000","last_became_active":"2022-07-03T10:35:27.320346+0000","last_became_peered":"2022-07-03T10:35:27.320346+0000","last_unstale":"2022-07-03T10:35:27.326689+0000","last_undegraded":"2022-07-03T10:35:27.326689+0000","last_fullsized":"2022-07-03T10:35:27.326689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:57:55.061972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250431+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:36:26.250431+0000","last_peered":"2022-07-03T10:36:26.250431+0000","last_clean":"2022-07-03T10:36:26.250431+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:36:26.250431+0000","last_undegraded":"2022-07-03T10:36:26.250431+0000","last_fullsized":"2022-07-03T10:36:26.250431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305459+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:26.305459+0000","last_peered":"2022-07-03T10:35:26.305459+0000","last_clean":"2022-07-03T10:35:26.305459+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:26.305459+0000","last_undegraded":"2022-07-03T10:35:26.305459+0000","last_fullsized":"2022-07-03T10:35:26.305459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245612+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:36:26.245612+0000","last_peered":"2022-07-03T10:36:26.245612+0000","last_clean":"2022-07-03T10:36:26.245612+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:36:26.245612+0000","last_undegraded":"2022-07-03T10:36:26.245612+0000","last_fullsized":"2022-07-03T10:36:26.245612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302039+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:26.302039+0000","last_peered":"2022-07-03T10:35:26.302039+0000","last_clean":"2022-07-03T10:35:26.302039+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:26.302039+0000","last_undegraded":"2022-07-03T10:35:26.302039+0000","last_fullsized":"2022-07-03T10:35:26.302039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241776+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:36:26.241776+0000","last_peered":"2022-07-03T10:36:26.241776+0000","last_clean":"2022-07-03T10:36:26.241776+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:36:26.241776+0000","last_undegraded":"2022-07-03T10:36:26.241776+0000","last_fullsized":"2022-07-03T10:36:26.241776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308728+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:26.308728+0000","last_peered":"2022-07-03T10:35:26.308728+0000","last_clean":"2022-07-03T10:35:26.308728+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:26.308728+0000","last_undegraded":"2022-07-03T10:35:26.308728+0000","last_fullsized":"2022-07-03T10:35:26.308728+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248392+0000","last_change":"2022-07-03T10:35:27.623637+0000","last_active":"2022-07-03T10:36:26.248392+0000","last_peered":"2022-07-03T10:36:26.248392+0000","last_clean":"2022-07-03T10:36:26.248392+0000","last_became_active":"2022-07-03T10:35:27.623202+0000","last_became_peered":"2022-07-03T10:35:27.623202+0000","last_unstale":"2022-07-03T10:36:26.248392+0000","last_undegraded":"2022-07-03T10:36:26.248392+0000","last_fullsized":"2022-07-03T10:36:26.248392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:39:41.407533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310134+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:26.310134+0000","last_peered":"2022-07-03T10:35:26.310134+0000","last_clean":"2022-07-03T10:35:26.310134+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:26.310134+0000","last_undegraded":"2022-07-03T10:35:26.310134+0000","last_fullsized":"2022-07-03T10:35:26.310134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240643+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:36:26.240643+0000","last_peered":"2022-07-03T10:36:26.240643+0000","last_clean":"2022-07-03T10:36:26.240643+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:36:26.240643+0000","last_undegraded":"2022-07-03T10:36:26.240643+0000","last_fullsized":"2022-07-03T10:36:26.240643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.324898+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:26.324898+0000","last_peered":"2022-07-03T10:35:26.324898+0000","last_clean":"2022-07-03T10:35:26.324898+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:26.324898+0000","last_undegraded":"2022-07-03T10:35:26.324898+0000","last_fullsized":"2022-07-03T10:35:26.324898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244794+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:36:26.244794+0000","last_peered":"2022-07-03T10:36:26.244794+0000","last_clean":"2022-07-03T10:36:26.244794+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:36:26.244794+0000","last_undegraded":"2022-07-03T10:36:26.244794+0000","last_fullsized":"2022-07-03T10:36:26.244794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347744+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:26.347744+0000","last_peered":"2022-07-03T10:35:26.347744+0000","last_clean":"2022-07-03T10:35:26.347744+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:26.347744+0000","last_undegraded":"2022-07-03T10:35:26.347744+0000","last_fullsized":"2022-07-03T10:35:26.347744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255206+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:36:26.255206+0000","last_peered":"2022-07-03T10:36:26.255206+0000","last_clean":"2022-07-03T10:36:26.255206+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:36:26.255206+0000","last_undegraded":"2022-07-03T10:36:26.255206+0000","last_fullsized":"2022-07-03T10:36:26.255206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245693+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:36:26.245693+0000","last_peered":"2022-07-03T10:36:26.245693+0000","last_clean":"2022-07-03T10:36:26.245693+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:36:26.245693+0000","last_undegraded":"2022-07-03T10:36:26.245693+0000","last_fullsized":"2022-07-03T10:36:26.245693+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332452+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:26.332452+0000","last_peered":"2022-07-03T10:35:26.332452+0000","last_clean":"2022-07-03T10:35:26.332452+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:26.332452+0000","last_undegraded":"2022-07-03T10:35:26.332452+0000","last_fullsized":"2022-07-03T10:35:26.332452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247527+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:36:26.247527+0000","last_peered":"2022-07-03T10:36:26.247527+0000","last_clean":"2022-07-03T10:36:26.247527+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:36:26.247527+0000","last_undegraded":"2022-07-03T10:36:26.247527+0000","last_fullsized":"2022-07-03T10:36:26.247527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302670+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:26.302670+0000","last_peered":"2022-07-03T10:35:26.302670+0000","last_clean":"2022-07-03T10:35:26.302670+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:26.302670+0000","last_undegraded":"2022-07-03T10:35:26.302670+0000","last_fullsized":"2022-07-03T10:35:26.302670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347662+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:26.347662+0000","last_peered":"2022-07-03T10:35:26.347662+0000","last_clean":"2022-07-03T10:35:26.347662+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:26.347662+0000","last_undegraded":"2022-07-03T10:35:26.347662+0000","last_fullsized":"2022-07-03T10:35:26.347662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245672+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:36:26.245672+0000","last_peered":"2022-07-03T10:36:26.245672+0000","last_clean":"2022-07-03T10:36:26.245672+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:36:26.245672+0000","last_undegraded":"2022-07-03T10:36:26.245672+0000","last_fullsized":"2022-07-03T10:36:26.245672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347752+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:26.347752+0000","last_peered":"2022-07-03T10:35:26.347752+0000","last_clean":"2022-07-03T10:35:26.347752+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:26.347752+0000","last_undegraded":"2022-07-03T10:35:26.347752+0000","last_fullsized":"2022-07-03T10:35:26.347752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309497+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:26.309497+0000","last_peered":"2022-07-03T10:35:26.309497+0000","last_clean":"2022-07-03T10:35:26.309497+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:26.309497+0000","last_undegraded":"2022-07-03T10:35:26.309497+0000","last_fullsized":"2022-07-03T10:35:26.309497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.253049+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:36:26.253049+0000","last_peered":"2022-07-03T10:36:26.253049+0000","last_clean":"2022-07-03T10:36:26.253049+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:36:26.253049+0000","last_undegraded":"2022-07-03T10:36:26.253049+0000","last_fullsized":"2022-07-03T10:36:26.253049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'4","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.415059+0000","last_change":"2022-07-03T10:35:27.414221+0000","last_active":"2022-07-03T10:35:27.415059+0000","last_peered":"2022-07-03T10:35:27.415059+0000","last_clean":"2022-07-03T10:35:27.415059+0000","last_became_active":"2022-07-03T10:35:27.323296+0000","last_became_peered":"2022-07-03T10:35:27.323296+0000","last_unstale":"2022-07-03T10:35:27.415059+0000","last_undegraded":"2022-07-03T10:35:27.415059+0000","last_fullsized":"2022-07-03T10:35:27.415059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:01:20.773865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302521+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:26.302521+0000","last_peered":"2022-07-03T10:35:26.302521+0000","last_clean":"2022-07-03T10:35:26.302521+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:26.302521+0000","last_undegraded":"2022-07-03T10:35:26.302521+0000","last_fullsized":"2022-07-03T10:35:26.302521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260780+0000","last_change":"2022-07-03T10:35:27.325484+0000","last_active":"2022-07-03T10:36:26.260780+0000","last_peered":"2022-07-03T10:36:26.260780+0000","last_clean":"2022-07-03T10:36:26.260780+0000","last_became_active":"2022-07-03T10:35:27.324947+0000","last_became_peered":"2022-07-03T10:35:27.324947+0000","last_unstale":"2022-07-03T10:36:26.260780+0000","last_undegraded":"2022-07-03T10:36:26.260780+0000","last_fullsized":"2022-07-03T10:36:26.260780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:36.475275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309995+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:26.309995+0000","last_peered":"2022-07-03T10:35:26.309995+0000","last_clean":"2022-07-03T10:35:26.309995+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:26.309995+0000","last_undegraded":"2022-07-03T10:35:26.309995+0000","last_fullsized":"2022-07-03T10:35:26.309995+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.214344+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:36:26.214344+0000","last_peered":"2022-07-03T10:36:26.214344+0000","last_clean":"2022-07-03T10:36:26.214344+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:36:26.214344+0000","last_undegraded":"2022-07-03T10:36:26.214344+0000","last_fullsized":"2022-07-03T10:36:26.214344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341921+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:26.341921+0000","last_peered":"2022-07-03T10:35:26.341921+0000","last_clean":"2022-07-03T10:35:26.341921+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:26.341921+0000","last_undegraded":"2022-07-03T10:35:26.341921+0000","last_fullsized":"2022-07-03T10:35:26.341921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242220+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:36:26.242220+0000","last_peered":"2022-07-03T10:36:26.242220+0000","last_clean":"2022-07-03T10:36:26.242220+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:36:26.242220+0000","last_undegraded":"2022-07-03T10:36:26.242220+0000","last_fullsized":"2022-07-03T10:36:26.242220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303560+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:26.303560+0000","last_peered":"2022-07-03T10:35:26.303560+0000","last_clean":"2022-07-03T10:35:26.303560+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:26.303560+0000","last_undegraded":"2022-07-03T10:35:26.303560+0000","last_fullsized":"2022-07-03T10:35:26.303560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256486+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:36:26.256486+0000","last_peered":"2022-07-03T10:36:26.256486+0000","last_clean":"2022-07-03T10:36:26.256486+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:36:26.256486+0000","last_undegraded":"2022-07-03T10:36:26.256486+0000","last_fullsized":"2022-07-03T10:36:26.256486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309116+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:26.309116+0000","last_peered":"2022-07-03T10:35:26.309116+0000","last_clean":"2022-07-03T10:35:26.309116+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:26.309116+0000","last_undegraded":"2022-07-03T10:35:26.309116+0000","last_fullsized":"2022-07-03T10:35:26.309116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246522+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:36:26.246522+0000","last_peered":"2022-07-03T10:36:26.246522+0000","last_clean":"2022-07-03T10:36:26.246522+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:36:26.246522+0000","last_undegraded":"2022-07-03T10:36:26.246522+0000","last_fullsized":"2022-07-03T10:36:26.246522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.312207+0000","last_change":"2022-07-03T10:35:27.312207+0000","last_active":"2022-07-03T10:35:27.312207+0000","last_peered":"2022-07-03T10:35:27.312207+0000","last_clean":"2022-07-03T10:35:27.312207+0000","last_became_active":"2022-07-03T10:35:27.311839+0000","last_became_peered":"2022-07-03T10:35:27.311839+0000","last_unstale":"2022-07-03T10:35:27.312207+0000","last_undegraded":"2022-07-03T10:35:27.312207+0000","last_fullsized":"2022-07-03T10:35:27.312207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:21:12.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246912+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:36:26.246912+0000","last_peered":"2022-07-03T10:36:26.246912+0000","last_clean":"2022-07-03T10:36:26.246912+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:36:26.246912+0000","last_undegraded":"2022-07-03T10:36:26.246912+0000","last_fullsized":"2022-07-03T10:36:26.246912+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305852+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:26.305852+0000","last_peered":"2022-07-03T10:35:26.305852+0000","last_clean":"2022-07-03T10:35:26.305852+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:26.305852+0000","last_undegraded":"2022-07-03T10:35:26.305852+0000","last_fullsized":"2022-07-03T10:35:26.305852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242322+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:36:26.242322+0000","last_peered":"2022-07-03T10:36:26.242322+0000","last_clean":"2022-07-03T10:36:26.242322+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:36:26.242322+0000","last_undegraded":"2022-07-03T10:36:26.242322+0000","last_fullsized":"2022-07-03T10:36:26.242322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330620+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:26.330620+0000","last_peered":"2022-07-03T10:35:26.330620+0000","last_clean":"2022-07-03T10:35:26.330620+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:26.330620+0000","last_undegraded":"2022-07-03T10:35:26.330620+0000","last_fullsized":"2022-07-03T10:35:26.330620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237445+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:36:26.237445+0000","last_peered":"2022-07-03T10:36:26.237445+0000","last_clean":"2022-07-03T10:36:26.237445+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:36:26.237445+0000","last_undegraded":"2022-07-03T10:36:26.237445+0000","last_fullsized":"2022-07-03T10:36:26.237445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242972+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:36:26.242972+0000","last_peered":"2022-07-03T10:36:26.242972+0000","last_clean":"2022-07-03T10:36:26.242972+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:36:26.242972+0000","last_undegraded":"2022-07-03T10:36:26.242972+0000","last_fullsized":"2022-07-03T10:36:26.242972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250077+0000","last_change":"2022-07-03T10:35:27.313043+0000","last_active":"2022-07-03T10:36:26.250077+0000","last_peered":"2022-07-03T10:36:26.250077+0000","last_clean":"2022-07-03T10:36:26.250077+0000","last_became_active":"2022-07-03T10:35:27.312553+0000","last_became_peered":"2022-07-03T10:35:27.312553+0000","last_unstale":"2022-07-03T10:36:26.250077+0000","last_undegraded":"2022-07-03T10:36:26.250077+0000","last_fullsized":"2022-07-03T10:36:26.250077+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:05:28.297497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256573+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:36:26.256573+0000","last_peered":"2022-07-03T10:36:26.256573+0000","last_clean":"2022-07-03T10:36:26.256573+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:36:26.256573+0000","last_undegraded":"2022-07-03T10:36:26.256573+0000","last_fullsized":"2022-07-03T10:36:26.256573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259834+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:36:26.259834+0000","last_peered":"2022-07-03T10:36:26.259834+0000","last_clean":"2022-07-03T10:36:26.259834+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:36:26.259834+0000","last_undegraded":"2022-07-03T10:36:26.259834+0000","last_fullsized":"2022-07-03T10:36:26.259834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.249554+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:36:26.249554+0000","last_peered":"2022-07-03T10:36:26.249554+0000","last_clean":"2022-07-03T10:36:26.249554+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:36:26.249554+0000","last_undegraded":"2022-07-03T10:36:26.249554+0000","last_fullsized":"2022-07-03T10:36:26.249554+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241377+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:36:26.241377+0000","last_peered":"2022-07-03T10:36:26.241377+0000","last_clean":"2022-07-03T10:36:26.241377+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:36:26.241377+0000","last_undegraded":"2022-07-03T10:36:26.241377+0000","last_fullsized":"2022-07-03T10:36:26.241377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242254+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:36:26.242254+0000","last_peered":"2022-07-03T10:36:26.242254+0000","last_clean":"2022-07-03T10:36:26.242254+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:36:26.242254+0000","last_undegraded":"2022-07-03T10:36:26.242254+0000","last_fullsized":"2022-07-03T10:36:26.242254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.243944+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:36:26.243944+0000","last_peered":"2022-07-03T10:36:26.243944+0000","last_clean":"2022-07-03T10:36:26.243944+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:36:26.243944+0000","last_undegraded":"2022-07-03T10:36:26.243944+0000","last_fullsized":"2022-07-03T10:36:26.243944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255263+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:36:26.255263+0000","last_peered":"2022-07-03T10:36:26.255263+0000","last_clean":"2022-07-03T10:36:26.255263+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:36:26.255263+0000","last_undegraded":"2022-07-03T10:36:26.255263+0000","last_fullsized":"2022-07-03T10:36:26.255263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255288+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:36:26.255288+0000","last_peered":"2022-07-03T10:36:26.255288+0000","last_clean":"2022-07-03T10:36:26.255288+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:36:26.255288+0000","last_undegraded":"2022-07-03T10:36:26.255288+0000","last_fullsized":"2022-07-03T10:36:26.255288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258181+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:36:26.258181+0000","last_peered":"2022-07-03T10:36:26.258181+0000","last_clean":"2022-07-03T10:36:26.258181+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:36:26.258181+0000","last_undegraded":"2022-07-03T10:36:26.258181+0000","last_fullsized":"2022-07-03T10:36:26.258181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241416+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:36:26.241416+0000","last_peered":"2022-07-03T10:36:26.241416+0000","last_clean":"2022-07-03T10:36:26.241416+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:36:26.241416+0000","last_undegraded":"2022-07-03T10:36:26.241416+0000","last_fullsized":"2022-07-03T10:36:26.241416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54826,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":210,"num_read_kb":160,"num_write":75,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":164478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547233,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":436583,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.501,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65800000000000003}]},{"osd":4,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.58799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.875,"5min":1.026,"15min":1.026},"last":0.625}]},{"osd":5,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.623},"min":{"1min":0.41899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92300000000000004,"5min":1.151,"15min":1.151},"last":0.68300000000000005}]},{"osd":6,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.83399999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.76000000000000001,"15min":0.76100000000000001},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.96799999999999997,"5min":1.075,"15min":1.075},"last":0.69399999999999995}]},{"osd":8,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72699999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.80600000000000005,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.044,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.77600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.78100000000000003,"15min":0.77700000000000002},"min":{"1min":0.63900000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.115,"15min":1.115},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.61399999999999999,"5min":0.496,"15min":0.496},"max":{"1min":0.94999999999999996,"5min":1.179,"15min":1.179},"last":0.63700000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6368,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738784,"statfs":{"total":95995035648,"available":95988514816,"internally_reserved":0,"allocated":1007616,"data_stored":1080955,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.437},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.52600000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.55500000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.66500000000000004,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71899999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65000000000000002}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.75700000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85899999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.77600000000000002,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.46400000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74199999999999999,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.84699999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.57899999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.879,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.81499999999999995},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.67500000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.72799999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972395,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":884736,"data_stored":437115,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.80500000000000005,"15min":0.71199999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6359999999999999,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.442},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.89800000000000002,"15min":0.77800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0720000000000001,"5min":11.582000000000001,"15min":11.582000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.93000000000000005,"15min":0.84999999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.905,"5min":11.933,"15min":11.933},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.55600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.5669999999999999,"5min":11.411,"15min":11.411},"last":0.89000000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.94099999999999995,"15min":0.85499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.944,"5min":11.887,"15min":11.887},"last":0.748},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.94699999999999995,"15min":0.89500000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6080000000000001,"5min":11.331,"15min":11.331},"last":0.95099999999999996}]},{"osd":3,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.85699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2490000000000001,"5min":9.8859999999999992,"15min":9.8859999999999992},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.95399999999999996,"15min":0.84399999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8009999999999999,"5min":12.041,"15min":12.041},"last":0.60899999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.016,"15min":0.97599999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":11.369,"15min":11.369},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":1,"15min":0.95199999999999996},"min":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95299999999999996,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":9.734},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":11.968}]},{"osd":6,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.0880000000000001,"15min":1.028},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.869,"5min":12.311,"15min":12.311},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1040000000000001,"15min":1.0700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.0419999999999998,"5min":11.534000000000001,"15min":11.534000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.0860000000000001,"15min":1.0529999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.272,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.1419999999999999,"15min":1.0860000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0019999999999998,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.78400000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.147,"15min":1.125},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6639999999999999,"5min":11.441000000000001,"15min":11.441000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.1160000000000001,"15min":1.0900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1839999999999999,"5min":9.8539999999999992,"15min":9.8539999999999992},"last":0.90500000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.1459999999999999,"15min":1.0920000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":11.474,"5min":11.474,"15min":11.474},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.107,"15min":1.0549999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":9.7729999999999997,"5min":9.7729999999999997,"15min":9.7729999999999997},"last":0.93500000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.008}]}]},{"osd":8,"up_from":54,"seq":231928234031,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738900,"statfs":{"total":95995035648,"available":95988633600,"internally_reserved":0,"allocated":888832,"data_stored":437137,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67200000000000004,"15min":0.63},"min":{"1min":0.47199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.871,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81799999999999995,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.47599999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.57199999999999995,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80700000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72999999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61299999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.437},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63700000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.52700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.62}]},{"osd":4,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57499999999999996,"15min":0.54800000000000004},"min":{"1min":0.45200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67900000000000005,"15min":0.621},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83699999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.77200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.69999999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.75900000000000001,"15min":0.70299999999999996},"min":{"1min":0.47699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.357,"5min":4.6959999999999997,"15min":4.6959999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.73399999999999999},"min":{"1min":0.56499999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67000000000000004,"15min":0.65400000000000003},"min":{"1min":0.51200000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.57499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.82799999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.75900000000000001},"min":{"1min":0.64000000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.77200000000000002,"15min":0.76800000000000002},"min":{"1min":0.63700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.997,"5min":1.117,"15min":1.117},"last":0.82399999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.51300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80600000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83999999999999997}]}]},{"osd":7,"up_from":47,"seq":201863462963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":1027606,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.37},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.373},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.309,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.45900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.626}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":1.222,"15min":1.222},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48999999999999999,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.40200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94799999999999995,"5min":5.077,"15min":5.077},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70299999999999996,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.67600000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:36:09 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66400000000000003},"min":{"1min":0.48199999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.96699999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.54400000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:36:42 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79100000000000004,"15min":0.79700000000000004},"min":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.002,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.94699999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.79900000000000004,"15min":0.81100000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.034,"15min":1.034},"last":0.70599999999999996}]},{"osd":11,"last update":"Sun Jul 3 10:36:04 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75}]}]},{"osd":6,"up_from":43,"seq":184683593782,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":436672,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.56799999999999995,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.93899999999999995}]},{"osd":1,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.70199999999999996,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.55100000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71899999999999997,"15min":0.69499999999999995},"min":{"1min":0.56499999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.748,"15min":0.71899999999999997},"min":{"1min":0.53600000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70099999999999996,"15min":0.68100000000000005},"min":{"1min":0.437,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.78200000000000003,"15min":0.72599999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.83899999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76700000000000002,"15min":0.755},"min":{"1min":0.51600000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.137,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81100000000000005,"15min":0.79300000000000004},"min":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3129999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.95399999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.82699999999999996,"15min":0.81100000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0669999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81799999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:35:52 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86299999999999999,"15min":0.84099999999999997},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2869999999999999,"5min":1.333,"15min":1.333},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.878,"15min":0.86099999999999999},"min":{"1min":0.70399999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.008}]},{"osd":8,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86499999999999999,"15min":0.85199999999999998},"min":{"1min":0.65400000000000003,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.121,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.879}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96599999999999997},"min":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.03},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.96499999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.89600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.98299999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1279999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.004,"15min":1.01},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.069}]}]},{"osd":1,"up_from":12,"seq":51539607630,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7268,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737884,"statfs":{"total":95995035648,"available":95987593216,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.38700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83899999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.57599999999999996},"min":{"1min":0.40899999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69799999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:47 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.58899999999999997,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.879,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.54800000000000004},"min":{"1min":0.48599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.86699999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:24 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.55500000000000005},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.55900000000000005},"min":{"1min":0.48499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83899999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.41399999999999998},"min":{"1min":0.39900000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77300000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.47399999999999998,"15min":0.41799999999999998},"min":{"1min":0.40200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79900000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.73799999999999999}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":1.044,"15min":1.044},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.60999999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88800000000000001,"5min":1.028,"15min":1.028},"last":0.91300000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.872,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.89300000000000002,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:51 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.70099999999999996,"15min":0.68500000000000005},"min":{"1min":0.53200000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98099999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.88},{"interface":"front","average":{"1min":0.755,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.57399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.88900000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79700000000000004,"15min":0.79400000000000004},"min":{"1min":0.626,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.68500000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0609999999999999,"5min":1.103,"15min":1.103},"last":0.996}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90800000000000003,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.752,"15min":0.748},"min":{"1min":0.53800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.89900000000000002,"5min":1.069,"15min":1.069},"last":0.96099999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":1.0349999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7080,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738072,"statfs":{"total":95995035648,"available":95987785728,"internally_reserved":0,"allocated":884736,"data_stored":436639,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.63700000000000001,"15min":0.56399999999999995},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":0.94499999999999995},"last":0.57399999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.68100000000000005,"15min":0.63100000000000001},"min":{"1min":0.52900000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.496}]},{"osd":3,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.39500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.73199999999999998,"15min":0.68000000000000005},"min":{"1min":0.51600000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.69099999999999995,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:35:59 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.73699999999999999,"15min":0.71399999999999997},"min":{"1min":0.59899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.45800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80800000000000005,"15min":0.76100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":0.752},{"interface":"front","average":{"1min":0.879,"5min":0.79200000000000004,"15min":0.75800000000000001},"min":{"1min":0.54000000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.89600000000000002,"15min":0.88800000000000001},"min":{"1min":0.64500000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.85399999999999998,"15min":0.84999999999999998},"min":{"1min":0.59299999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.48999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.84199999999999997,"15min":0.83499999999999996},"min":{"1min":0.64200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97399999999999998,"15min":0.97099999999999997},"min":{"1min":0.66500000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.76800000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.90200000000000002,"15min":0.90700000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72399999999999998}]},{"osd":11,"last update":"Sun Jul 3 10:36:15 2022","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.95499999999999996,"15min":0.93700000000000006},"min":{"1min":0.71299999999999997,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.88500000000000001,"15min":0.878},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.63300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476812,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7332,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737820,"statfs":{"total":95995035648,"available":95987527680,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.496}]},{"osd":1,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69599999999999995,"5min":0.874,"15min":0.874},"last":0.62}]},{"osd":3,"last update":"Sun Jul 3 10:36:16 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83299999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58199999999999996}]},{"osd":4,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.71199999999999997}]},{"osd":5,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.501,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63500000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:36:32 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67100000000000004,"15min":0.65100000000000002},"min":{"1min":0.52200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.88900000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66200000000000003,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83599999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78400000000000003,"5min":0.871,"15min":0.871},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.42699999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.69799999999999995}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.746,"15min":0.74199999999999999},"min":{"1min":0.56799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.88100000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.58399999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.79900000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72899999999999998}]}]},{"osd":3,"up_from":23,"seq":98784247877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":897024,"data_stored":436899,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.51600000000000001,"15min":0.44700000000000001},"min":{"1min":0.34100000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.9609999999999999,"5min":3.9609999999999999,"15min":3.9609999999999999},"last":0.433},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67200000000000004,"15min":0.60299999999999998},"min":{"1min":0.46500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95199999999999996,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.21,"5min":2.581,"15min":2.581},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.903,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.80500000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76900000000000002,"15min":0.66400000000000003},"min":{"1min":0.58599999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.82099999999999995,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90700000000000003,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.657,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81100000000000005,"15min":0.76100000000000001},"min":{"1min":0.52100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.2519999999999998,"5min":2.46,"15min":2.46},"last":0.86399999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.745},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90500000000000003,"15min":0.91200000000000003},"min":{"1min":0.65400000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.78900000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.91400000000000003,"15min":0.96299999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.131,"5min":1.256,"15min":1.256},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.95499999999999996,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.61199999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.71499999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.878}]}]},{"osd":4,"up_from":30,"seq":128849018945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6312,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738840,"statfs":{"total":95995035648,"available":95988572160,"internally_reserved":0,"allocated":884736,"data_stored":436571,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.59899999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58899999999999997,"15min":0.55900000000000005},"min":{"1min":0.44800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.46899999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.70899999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.56499999999999995},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73199999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.75},{"interface":"front","average":{"1min":0.628,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.48199999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.76900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.57499999999999996},"min":{"1min":0.46400000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79000000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56399999999999995},"min":{"1min":0.42699999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.84399999999999997}]},{"osd":6,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.53300000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.67000000000000004,"15min":0.65300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.76700000000000002,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.64700000000000002},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66800000000000004,"15min":0.64500000000000002},"min":{"1min":0.53900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":1,"15min":1},"last":0.86899999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.79100000000000004},"min":{"1min":0.56200000000000006,"5min":0.497,"15min":0.497},"max":{"1min":0.89100000000000001,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.96499999999999997,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.97299999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.72999999999999998,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92100000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.91300000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90400000000000003}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.98299999999999998}]}]},{"osd":5,"up_from":35,"seq":150323855420,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":946176,"data_stored":1027222,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.64000000000000001,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"last":0.77700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.32900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.433,"15min":0.432},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56499999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.69799999999999995,"15min":0.627},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.21,"5min":7.21,"15min":7.21},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.02},"last":0.64500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59799999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73399999999999999,"15min":0.69599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86699999999999999,"5min":5.476,"15min":5.476},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.72299999999999998},"min":{"1min":0.58599999999999997,"5min":0.505,"15min":0.505},"max":{"1min":0.91000000000000003,"5min":3.27,"15min":3.27},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":0.875,"15min":0.875},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66200000000000003,"15min":0.64600000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56200000000000006}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.50700000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.77200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95999999999999996,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82199999999999995,"15min":0.78600000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.90800000000000003,"5min":5.4240000000000004,"15min":5.4240000000000004},"last":0.75800000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.66900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":54238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:36:49.074 INFO:tasks.fwd_scrub.ceph_manager:clean! 2022-07-03T10:36:49.075 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-03T10:36:49.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.329+0000 7f931df9d700 1 -- 172.21.15.101:0/2601008687 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9318070e50 msgr2=0x7f93180712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.332 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.329+0000 7f931df9d700 1 --2- 172.21.15.101:0/2601008687 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9318070e50 0x7f93180712c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9304009900 tx=0x7f93040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 -- 172.21.15.101:0/2601008687 shutdown_connections 2022-07-03T10:36:49.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 --2- 172.21.15.101:0/2601008687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93181052e0 0x7f93181076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 --2- 172.21.15.101:0/2601008687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 0x7f9318070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.333 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 --2- 172.21.15.101:0/2601008687 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9318070e50 0x7f93180712c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 -- 172.21.15.101:0/2601008687 >> 172.21.15.101:0/2601008687 conn(0x7f93180f8b20 msgr2=0x7f93180faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:49.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 -- 172.21.15.101:0/2601008687 shutdown_connections 2022-07-03T10:36:49.334 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.330+0000 7f931df9d700 1 -- 172.21.15.101:0/2601008687 wait complete. 2022-07-03T10:36:49.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 Processor -- start 2022-07-03T10:36:49.337 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 -- start start 2022-07-03T10:36:49.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 0x7f9318110b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 0x7f9318117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.338 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318118160 con 0x7f93181052e0 2022-07-03T10:36:49.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93181182a0 con 0x7f9318072430 2022-07-03T10:36:49.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.331+0000 7f931df9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9318118460 con 0x7f9318070e50 2022-07-03T10:36:49.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.339 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52728/0 (socket says 172.21.15.101:52728) 2022-07-03T10:36:49.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 -- 172.21.15.101:0/3042258725 learned_addr learned my addr 172.21.15.101:0/3042258725 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:49.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931d79c700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 0x7f9318117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.340 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f9317fff700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 0x7f9318110b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 -- 172.21.15.101:0/3042258725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 msgr2=0x7f9318110b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 0x7f9318110b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.341 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 -- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 msgr2=0x7f9318117be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.342 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 0x7f9318117be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.342 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 -- 172.21.15.101:0/3042258725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9304009440 con 0x7f9318070e50 2022-07-03T10:36:49.343 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931d79c700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 0x7f9318117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:49.343 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.332+0000 7f931cf9b700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f930c00ea80 tx=0x7f930c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:49.343 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.333+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c00c7c0 con 0x7f9318070e50 2022-07-03T10:36:49.344 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.333+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9318118740 con 0x7f9318070e50 2022-07-03T10:36:49.344 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.333+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f930c00cdd0 con 0x7f9318070e50 2022-07-03T10:36:49.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.333+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c010df0 con 0x7f9318070e50 2022-07-03T10:36:49.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.333+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9318118cd0 con 0x7f9318070e50 2022-07-03T10:36:49.345 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.334+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931810adf0 con 0x7f9318070e50 2022-07-03T10:36:49.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.336+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f930c00c040 con 0x7f9318070e50 2022-07-03T10:36:49.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.337+0000 7f93157fa700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 0x7f93080991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.347 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.337+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f930c014070 con 0x7f9318070e50 2022-07-03T10:36:49.348 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.337+0000 7f9317fff700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 0x7f93080991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.348 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.338+0000 7f9317fff700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 0x7f93080991d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9304005d00 tx=0x7f93040091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:49.348 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.345+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f930c04f100 con 0x7f9318070e50 2022-07-03T10:36:49.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:49 smithi101 ceph-mon[85126]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-07-03T10:36:49.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:49 smithi101 ceph-mon[85126]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:49.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.529+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 --> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f931805e960 con 0x7f9308096d20 2022-07-03T10:36:49.541 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.539+0000 7f93157fa700 1 -- 172.21.15.101:0/3042258725 <== mgr.14154 v2:172.21.15.101:6800/1865562803 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343667 (secure 0 0 0) 0x7f931805e960 con 0x7f9308096d20 2022-07-03T10:36:49.547 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:49.550 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 msgr2=0x7f93080991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.550 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 0x7f93080991d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9304005d00 tx=0x7f93040091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 msgr2=0x7f9318110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f930c00ea80 tx=0x7f930c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 shutdown_connections 2022-07-03T10:36:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f9308096d20 0x7f93080991d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.551 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f93181052e0 0x7f9318117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.552 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9318070e50 0x7f9318110630 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.552 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 --2- 172.21.15.101:0/3042258725 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9318072430 0x7f9318110b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.552 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 >> 172.21.15.101:0/3042258725 conn(0x7f93180f8b20 msgr2=0x7f93180faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:49.553 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 shutdown_connections 2022-07-03T10:36:49.553 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.544+0000 7f931df9d700 1 -- 172.21.15.101:0/3042258725 wait complete. 2022-07-03T10:36:49.553 INFO:teuthology.orchestra.run.smithi101.stderr:dumped all 2022-07-03T10:36:49.574 INFO:teuthology.orchestra.run.smithi101.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2022-07-03T10:36:47.964118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645440,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":336,"num_read_kb":269,"num_write":258,"num_write_kb":3018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":705,"ondisk_log_size":705,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78288,"kb_used_data":10736,"kb_used_omap":0,"kb_used_meta":67456,"kb_avail":1124863536,"statfs":{"total":1151940427776,"available":1151860260864,"internally_reserved":0,"allocated":10993664,"data_stored":7174585,"data_compressed":176772,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":69074944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":322,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003008"},"pg_stats":[{"pgid":"3.3f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244527+0000","last_change":"2022-07-03T10:35:03.461005+0000","last_active":"2022-07-03T10:36:26.244527+0000","last_peered":"2022-07-03T10:36:26.244527+0000","last_clean":"2022-07-03T10:36:26.244527+0000","last_became_active":"2022-07-03T10:35:03.460522+0000","last_became_peered":"2022-07-03T10:35:03.460522+0000","last_unstale":"2022-07-03T10:36:26.244527+0000","last_undegraded":"2022-07-03T10:36:26.244527+0000","last_fullsized":"2022-07-03T10:36:26.244527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:22.446824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.261640+0000","last_change":"2022-07-03T10:35:03.470408+0000","last_active":"2022-07-03T10:36:26.261640+0000","last_peered":"2022-07-03T10:36:26.261640+0000","last_clean":"2022-07-03T10:36:26.261640+0000","last_became_active":"2022-07-03T10:35:03.461694+0000","last_became_peered":"2022-07-03T10:35:03.461694+0000","last_unstale":"2022-07-03T10:36:26.261640+0000","last_undegraded":"2022-07-03T10:36:26.261640+0000","last_fullsized":"2022-07-03T10:36:26.261640+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:09:49.299429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260480+0000","last_change":"2022-07-03T10:35:03.460065+0000","last_active":"2022-07-03T10:36:26.260480+0000","last_peered":"2022-07-03T10:36:26.260480+0000","last_clean":"2022-07-03T10:36:26.260480+0000","last_became_active":"2022-07-03T10:35:03.459700+0000","last_became_peered":"2022-07-03T10:35:03.459700+0000","last_unstale":"2022-07-03T10:36:26.260480+0000","last_undegraded":"2022-07-03T10:36:26.260480+0000","last_fullsized":"2022-07-03T10:36:26.260480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:19:28.047449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248817+0000","last_change":"2022-07-03T10:35:27.315233+0000","last_active":"2022-07-03T10:36:26.248817+0000","last_peered":"2022-07-03T10:36:26.248817+0000","last_clean":"2022-07-03T10:36:26.248817+0000","last_became_active":"2022-07-03T10:35:27.314807+0000","last_became_peered":"2022-07-03T10:35:27.314807+0000","last_unstale":"2022-07-03T10:36:26.248817+0000","last_undegraded":"2022-07-03T10:36:26.248817+0000","last_fullsized":"2022-07-03T10:36:26.248817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:37:28.362457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240860+0000","last_change":"2022-07-03T10:35:03.465746+0000","last_active":"2022-07-03T10:36:26.240860+0000","last_peered":"2022-07-03T10:36:26.240860+0000","last_clean":"2022-07-03T10:36:26.240860+0000","last_became_active":"2022-07-03T10:35:03.465438+0000","last_became_peered":"2022-07-03T10:35:03.465438+0000","last_unstale":"2022-07-03T10:36:26.240860+0000","last_undegraded":"2022-07-03T10:36:26.240860+0000","last_fullsized":"2022-07-03T10:36:26.240860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:26:47.365260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234621+0000","last_change":"2022-07-03T10:35:03.450988+0000","last_active":"2022-07-03T10:36:26.234621+0000","last_peered":"2022-07-03T10:36:26.234621+0000","last_clean":"2022-07-03T10:36:26.234621+0000","last_became_active":"2022-07-03T10:35:03.450639+0000","last_became_peered":"2022-07-03T10:35:03.450639+0000","last_unstale":"2022-07-03T10:36:26.234621+0000","last_undegraded":"2022-07-03T10:36:26.234621+0000","last_fullsized":"2022-07-03T10:36:26.234621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:42:59.191981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247958+0000","last_change":"2022-07-03T10:35:03.453447+0000","last_active":"2022-07-03T10:36:26.247958+0000","last_peered":"2022-07-03T10:36:26.247958+0000","last_clean":"2022-07-03T10:36:26.247958+0000","last_became_active":"2022-07-03T10:35:03.453119+0000","last_became_peered":"2022-07-03T10:35:03.453119+0000","last_unstale":"2022-07-03T10:36:26.247958+0000","last_undegraded":"2022-07-03T10:36:26.247958+0000","last_fullsized":"2022-07-03T10:36:26.247958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:17:43.541499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.252954+0000","last_change":"2022-07-03T10:35:03.439396+0000","last_active":"2022-07-03T10:36:26.252954+0000","last_peered":"2022-07-03T10:36:26.252954+0000","last_clean":"2022-07-03T10:36:26.252954+0000","last_became_active":"2022-07-03T10:35:03.439055+0000","last_became_peered":"2022-07-03T10:35:03.439055+0000","last_unstale":"2022-07-03T10:36:26.252954+0000","last_undegraded":"2022-07-03T10:36:26.252954+0000","last_fullsized":"2022-07-03T10:36:26.252954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:55:11.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251354+0000","last_change":"2022-07-03T10:35:03.452023+0000","last_active":"2022-07-03T10:36:26.251354+0000","last_peered":"2022-07-03T10:36:26.251354+0000","last_clean":"2022-07-03T10:36:26.251354+0000","last_became_active":"2022-07-03T10:35:03.451683+0000","last_became_peered":"2022-07-03T10:35:03.451683+0000","last_unstale":"2022-07-03T10:36:26.251354+0000","last_undegraded":"2022-07-03T10:36:26.251354+0000","last_fullsized":"2022-07-03T10:36:26.251354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:51:50.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"82'16","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241581+0000","last_change":"2022-07-03T10:35:03.454452+0000","last_active":"2022-07-03T10:36:26.241581+0000","last_peered":"2022-07-03T10:36:26.241581+0000","last_clean":"2022-07-03T10:36:26.241581+0000","last_became_active":"2022-07-03T10:35:03.454106+0000","last_became_peered":"2022-07-03T10:35:03.454106+0000","last_unstale":"2022-07-03T10:36:26.241581+0000","last_undegraded":"2022-07-03T10:36:26.241581+0000","last_fullsized":"2022-07-03T10:36:26.241581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:43:01.649257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246804+0000","last_change":"2022-07-03T10:35:03.455428+0000","last_active":"2022-07-03T10:36:26.246804+0000","last_peered":"2022-07-03T10:36:26.246804+0000","last_clean":"2022-07-03T10:36:26.246804+0000","last_became_active":"2022-07-03T10:35:03.455116+0000","last_became_peered":"2022-07-03T10:35:03.455116+0000","last_unstale":"2022-07-03T10:36:26.246804+0000","last_undegraded":"2022-07-03T10:36:26.246804+0000","last_fullsized":"2022-07-03T10:36:26.246804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:53:14.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237345+0000","last_change":"2022-07-03T10:35:03.458920+0000","last_active":"2022-07-03T10:36:26.237345+0000","last_peered":"2022-07-03T10:36:26.237345+0000","last_clean":"2022-07-03T10:36:26.237345+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.237345+0000","last_undegraded":"2022-07-03T10:36:26.237345+0000","last_fullsized":"2022-07-03T10:36:26.237345+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:07:38.605807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245178+0000","last_change":"2022-07-03T10:35:03.442331+0000","last_active":"2022-07-03T10:36:26.245178+0000","last_peered":"2022-07-03T10:36:26.245178+0000","last_clean":"2022-07-03T10:36:26.245178+0000","last_became_active":"2022-07-03T10:35:03.441968+0000","last_became_peered":"2022-07-03T10:35:03.441968+0000","last_unstale":"2022-07-03T10:36:26.245178+0000","last_undegraded":"2022-07-03T10:36:26.245178+0000","last_fullsized":"2022-07-03T10:36:26.245178+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:24:36.925484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.262152+0000","last_change":"2022-07-03T10:35:03.461276+0000","last_active":"2022-07-03T10:36:26.262152+0000","last_peered":"2022-07-03T10:36:26.262152+0000","last_clean":"2022-07-03T10:36:26.262152+0000","last_became_active":"2022-07-03T10:35:03.460923+0000","last_became_peered":"2022-07-03T10:35:03.460923+0000","last_unstale":"2022-07-03T10:36:26.262152+0000","last_undegraded":"2022-07-03T10:36:26.262152+0000","last_fullsized":"2022-07-03T10:36:26.262152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:27:17.979893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259927+0000","last_change":"2022-07-03T10:35:03.469733+0000","last_active":"2022-07-03T10:36:26.259927+0000","last_peered":"2022-07-03T10:36:26.259927+0000","last_clean":"2022-07-03T10:36:26.259927+0000","last_became_active":"2022-07-03T10:35:03.469311+0000","last_became_peered":"2022-07-03T10:35:03.469311+0000","last_unstale":"2022-07-03T10:36:26.259927+0000","last_undegraded":"2022-07-03T10:36:26.259927+0000","last_fullsized":"2022-07-03T10:36:26.259927+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:54:39.873279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256218+0000","last_change":"2022-07-03T10:35:03.460994+0000","last_active":"2022-07-03T10:36:26.256218+0000","last_peered":"2022-07-03T10:36:26.256218+0000","last_clean":"2022-07-03T10:36:26.256218+0000","last_became_active":"2022-07-03T10:35:03.460667+0000","last_became_peered":"2022-07-03T10:35:03.460667+0000","last_unstale":"2022-07-03T10:36:26.256218+0000","last_undegraded":"2022-07-03T10:36:26.256218+0000","last_fullsized":"2022-07-03T10:36:26.256218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:40:26.198901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248902+0000","last_change":"2022-07-03T10:35:03.461695+0000","last_active":"2022-07-03T10:36:26.248902+0000","last_peered":"2022-07-03T10:36:26.248902+0000","last_clean":"2022-07-03T10:36:26.248902+0000","last_became_active":"2022-07-03T10:35:03.460971+0000","last_became_peered":"2022-07-03T10:35:03.460971+0000","last_unstale":"2022-07-03T10:36:26.248902+0000","last_undegraded":"2022-07-03T10:36:26.248902+0000","last_fullsized":"2022-07-03T10:36:26.248902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:52:24.141698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248745+0000","last_change":"2022-07-03T10:35:03.451732+0000","last_active":"2022-07-03T10:36:26.248745+0000","last_peered":"2022-07-03T10:36:26.248745+0000","last_clean":"2022-07-03T10:36:26.248745+0000","last_became_active":"2022-07-03T10:35:03.451367+0000","last_became_peered":"2022-07-03T10:35:03.451367+0000","last_unstale":"2022-07-03T10:36:26.248745+0000","last_undegraded":"2022-07-03T10:36:26.248745+0000","last_fullsized":"2022-07-03T10:36:26.248745+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:42:59.598503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246317+0000","last_change":"2022-07-03T10:35:03.472392+0000","last_active":"2022-07-03T10:36:26.246317+0000","last_peered":"2022-07-03T10:36:26.246317+0000","last_clean":"2022-07-03T10:36:26.246317+0000","last_became_active":"2022-07-03T10:35:03.472124+0000","last_became_peered":"2022-07-03T10:35:03.472124+0000","last_unstale":"2022-07-03T10:36:26.246317+0000","last_undegraded":"2022-07-03T10:36:26.246317+0000","last_fullsized":"2022-07-03T10:36:26.246317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:53:25.394204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.325415+0000","last_change":"2022-07-03T10:35:02.873839+0000","last_active":"2022-07-03T10:35:26.325415+0000","last_peered":"2022-07-03T10:35:26.325415+0000","last_clean":"2022-07-03T10:35:26.325415+0000","last_became_active":"2022-07-03T10:35:02.873468+0000","last_became_peered":"2022-07-03T10:35:02.873468+0000","last_unstale":"2022-07-03T10:35:26.325415+0000","last_undegraded":"2022-07-03T10:35:26.325415+0000","last_fullsized":"2022-07-03T10:35:26.325415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:51:58.738965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242008+0000","last_change":"2022-07-03T10:35:03.462657+0000","last_active":"2022-07-03T10:36:26.242008+0000","last_peered":"2022-07-03T10:36:26.242008+0000","last_clean":"2022-07-03T10:36:26.242008+0000","last_became_active":"2022-07-03T10:35:03.462325+0000","last_became_peered":"2022-07-03T10:35:03.462325+0000","last_unstale":"2022-07-03T10:36:26.242008+0000","last_undegraded":"2022-07-03T10:36:26.242008+0000","last_fullsized":"2022-07-03T10:36:26.242008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:49:01.654363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.343531+0000","last_change":"2022-07-03T10:35:02.879961+0000","last_active":"2022-07-03T10:35:26.343531+0000","last_peered":"2022-07-03T10:35:26.343531+0000","last_clean":"2022-07-03T10:35:26.343531+0000","last_became_active":"2022-07-03T10:35:02.879645+0000","last_became_peered":"2022-07-03T10:35:02.879645+0000","last_unstale":"2022-07-03T10:35:26.343531+0000","last_undegraded":"2022-07-03T10:35:26.343531+0000","last_fullsized":"2022-07-03T10:35:26.343531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:44:30.649895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260621+0000","last_change":"2022-07-03T10:35:03.470902+0000","last_active":"2022-07-03T10:36:26.260621+0000","last_peered":"2022-07-03T10:36:26.260621+0000","last_clean":"2022-07-03T10:36:26.260621+0000","last_became_active":"2022-07-03T10:35:03.462514+0000","last_became_peered":"2022-07-03T10:35:03.462514+0000","last_unstale":"2022-07-03T10:36:26.260621+0000","last_undegraded":"2022-07-03T10:36:26.260621+0000","last_fullsized":"2022-07-03T10:36:26.260621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:08.204954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309637+0000","last_change":"2022-07-03T10:35:02.871783+0000","last_active":"2022-07-03T10:35:26.309637+0000","last_peered":"2022-07-03T10:35:26.309637+0000","last_clean":"2022-07-03T10:35:26.309637+0000","last_became_active":"2022-07-03T10:35:02.871428+0000","last_became_peered":"2022-07-03T10:35:02.871428+0000","last_unstale":"2022-07-03T10:35:26.309637+0000","last_undegraded":"2022-07-03T10:35:26.309637+0000","last_fullsized":"2022-07-03T10:35:26.309637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:07:45.748922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250215+0000","last_change":"2022-07-03T10:35:03.454372+0000","last_active":"2022-07-03T10:36:26.250215+0000","last_peered":"2022-07-03T10:36:26.250215+0000","last_clean":"2022-07-03T10:36:26.250215+0000","last_became_active":"2022-07-03T10:35:03.453992+0000","last_became_peered":"2022-07-03T10:35:03.453992+0000","last_unstale":"2022-07-03T10:36:26.250215+0000","last_undegraded":"2022-07-03T10:36:26.250215+0000","last_fullsized":"2022-07-03T10:36:26.250215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:42.387884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309394+0000","last_change":"2022-07-03T10:35:02.867816+0000","last_active":"2022-07-03T10:35:26.309394+0000","last_peered":"2022-07-03T10:35:26.309394+0000","last_clean":"2022-07-03T10:35:26.309394+0000","last_became_active":"2022-07-03T10:35:02.866636+0000","last_became_peered":"2022-07-03T10:35:02.866636+0000","last_unstale":"2022-07-03T10:35:26.309394+0000","last_undegraded":"2022-07-03T10:35:26.309394+0000","last_fullsized":"2022-07-03T10:35:26.309394+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:04:18.758651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248977+0000","last_change":"2022-07-03T10:35:03.450728+0000","last_active":"2022-07-03T10:36:26.248977+0000","last_peered":"2022-07-03T10:36:26.248977+0000","last_clean":"2022-07-03T10:36:26.248977+0000","last_became_active":"2022-07-03T10:35:03.450482+0000","last_became_peered":"2022-07-03T10:35:03.450482+0000","last_unstale":"2022-07-03T10:36:26.248977+0000","last_undegraded":"2022-07-03T10:36:26.248977+0000","last_fullsized":"2022-07-03T10:36:26.248977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:31.059898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305894+0000","last_change":"2022-07-03T10:35:02.870981+0000","last_active":"2022-07-03T10:35:26.305894+0000","last_peered":"2022-07-03T10:35:26.305894+0000","last_clean":"2022-07-03T10:35:26.305894+0000","last_became_active":"2022-07-03T10:35:02.870638+0000","last_became_peered":"2022-07-03T10:35:02.870638+0000","last_unstale":"2022-07-03T10:35:26.305894+0000","last_undegraded":"2022-07-03T10:35:26.305894+0000","last_fullsized":"2022-07-03T10:35:26.305894+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:12:11.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.234357+0000","last_change":"2022-07-03T10:35:03.456372+0000","last_active":"2022-07-03T10:36:26.234357+0000","last_peered":"2022-07-03T10:36:26.234357+0000","last_clean":"2022-07-03T10:36:26.234357+0000","last_became_active":"2022-07-03T10:35:03.454690+0000","last_became_peered":"2022-07-03T10:35:03.454690+0000","last_unstale":"2022-07-03T10:36:26.234357+0000","last_undegraded":"2022-07-03T10:36:26.234357+0000","last_fullsized":"2022-07-03T10:36:26.234357+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:59:24.720031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'1","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309034+0000","last_change":"2022-07-03T10:35:02.881227+0000","last_active":"2022-07-03T10:35:26.309034+0000","last_peered":"2022-07-03T10:35:26.309034+0000","last_clean":"2022-07-03T10:35:26.309034+0000","last_became_active":"2022-07-03T10:35:02.880920+0000","last_became_peered":"2022-07-03T10:35:02.880920+0000","last_unstale":"2022-07-03T10:35:26.309034+0000","last_undegraded":"2022-07-03T10:35:26.309034+0000","last_fullsized":"2022-07-03T10:35:26.309034+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:42:23.762477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"82'8","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256408+0000","last_change":"2022-07-03T10:35:27.324978+0000","last_active":"2022-07-03T10:36:26.256408+0000","last_peered":"2022-07-03T10:36:26.256408+0000","last_clean":"2022-07-03T10:36:26.256408+0000","last_became_active":"2022-07-03T10:35:27.324570+0000","last_became_peered":"2022-07-03T10:35:27.324570+0000","last_unstale":"2022-07-03T10:36:26.256408+0000","last_undegraded":"2022-07-03T10:36:26.256408+0000","last_fullsized":"2022-07-03T10:36:26.256408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:18:06.187457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332535+0000","last_change":"2022-07-03T10:35:02.870197+0000","last_active":"2022-07-03T10:35:26.332535+0000","last_peered":"2022-07-03T10:35:26.332535+0000","last_clean":"2022-07-03T10:35:26.332535+0000","last_became_active":"2022-07-03T10:35:02.869693+0000","last_became_peered":"2022-07-03T10:35:02.869693+0000","last_unstale":"2022-07-03T10:35:26.332535+0000","last_undegraded":"2022-07-03T10:35:26.332535+0000","last_fullsized":"2022-07-03T10:35:26.332535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:33:48.229036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.254411+0000","last_change":"2022-07-03T10:35:03.451651+0000","last_active":"2022-07-03T10:36:26.254411+0000","last_peered":"2022-07-03T10:36:26.254411+0000","last_clean":"2022-07-03T10:36:26.254411+0000","last_became_active":"2022-07-03T10:35:03.451331+0000","last_became_peered":"2022-07-03T10:35:03.451331+0000","last_unstale":"2022-07-03T10:36:26.254411+0000","last_undegraded":"2022-07-03T10:36:26.254411+0000","last_fullsized":"2022-07-03T10:36:26.254411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:34:25.104668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305427+0000","last_change":"2022-07-03T10:35:02.872676+0000","last_active":"2022-07-03T10:35:26.305427+0000","last_peered":"2022-07-03T10:35:26.305427+0000","last_clean":"2022-07-03T10:35:26.305427+0000","last_became_active":"2022-07-03T10:35:02.872319+0000","last_became_peered":"2022-07-03T10:35:02.872319+0000","last_unstale":"2022-07-03T10:35:26.305427+0000","last_undegraded":"2022-07-03T10:35:26.305427+0000","last_fullsized":"2022-07-03T10:35:26.305427+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:37:18.724144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'11","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255985+0000","last_change":"2022-07-03T10:35:27.314929+0000","last_active":"2022-07-03T10:36:26.255985+0000","last_peered":"2022-07-03T10:36:26.255985+0000","last_clean":"2022-07-03T10:36:26.255985+0000","last_became_active":"2022-07-03T10:35:27.314542+0000","last_became_peered":"2022-07-03T10:35:27.314542+0000","last_unstale":"2022-07-03T10:36:26.255985+0000","last_undegraded":"2022-07-03T10:36:26.255985+0000","last_fullsized":"2022-07-03T10:36:26.255985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:50:25.278127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309250+0000","last_change":"2022-07-03T10:35:02.877290+0000","last_active":"2022-07-03T10:35:26.309250+0000","last_peered":"2022-07-03T10:35:26.309250+0000","last_clean":"2022-07-03T10:35:26.309250+0000","last_became_active":"2022-07-03T10:35:02.877032+0000","last_became_peered":"2022-07-03T10:35:02.877032+0000","last_unstale":"2022-07-03T10:35:26.309250+0000","last_undegraded":"2022-07-03T10:35:26.309250+0000","last_fullsized":"2022-07-03T10:35:26.309250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:18:49.486658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.251048+0000","last_change":"2022-07-03T10:35:03.450233+0000","last_active":"2022-07-03T10:36:26.251048+0000","last_peered":"2022-07-03T10:36:26.251048+0000","last_clean":"2022-07-03T10:36:26.251048+0000","last_became_active":"2022-07-03T10:35:03.449918+0000","last_became_peered":"2022-07-03T10:35:03.449918+0000","last_unstale":"2022-07-03T10:36:26.251048+0000","last_undegraded":"2022-07-03T10:36:26.251048+0000","last_fullsized":"2022-07-03T10:36:26.251048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:37:20.728122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347933+0000","last_change":"2022-07-03T10:35:02.870733+0000","last_active":"2022-07-03T10:35:26.347933+0000","last_peered":"2022-07-03T10:35:26.347933+0000","last_clean":"2022-07-03T10:35:26.347933+0000","last_became_active":"2022-07-03T10:35:02.870380+0000","last_became_peered":"2022-07-03T10:35:02.870380+0000","last_unstale":"2022-07-03T10:35:26.347933+0000","last_undegraded":"2022-07-03T10:35:26.347933+0000","last_fullsized":"2022-07-03T10:35:26.347933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:07:59.928243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258203+0000","last_change":"2022-07-03T10:35:03.460507+0000","last_active":"2022-07-03T10:36:26.258203+0000","last_peered":"2022-07-03T10:36:26.258203+0000","last_clean":"2022-07-03T10:36:26.258203+0000","last_became_active":"2022-07-03T10:35:03.460185+0000","last_became_peered":"2022-07-03T10:35:03.460185+0000","last_unstale":"2022-07-03T10:36:26.258203+0000","last_undegraded":"2022-07-03T10:36:26.258203+0000","last_fullsized":"2022-07-03T10:36:26.258203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:19:14.022173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'3","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.306107+0000","last_change":"2022-07-03T10:35:02.874642+0000","last_active":"2022-07-03T10:35:26.306107+0000","last_peered":"2022-07-03T10:35:26.306107+0000","last_clean":"2022-07-03T10:35:26.306107+0000","last_became_active":"2022-07-03T10:35:02.874304+0000","last_became_peered":"2022-07-03T10:35:02.874304+0000","last_unstale":"2022-07-03T10:35:26.306107+0000","last_undegraded":"2022-07-03T10:35:26.306107+0000","last_fullsized":"2022-07-03T10:35:26.306107+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:28:32.834780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250505+0000","last_change":"2022-07-03T10:35:03.458433+0000","last_active":"2022-07-03T10:36:26.250505+0000","last_peered":"2022-07-03T10:36:26.250505+0000","last_clean":"2022-07-03T10:36:26.250505+0000","last_became_active":"2022-07-03T10:35:03.458124+0000","last_became_peered":"2022-07-03T10:35:03.458124+0000","last_unstale":"2022-07-03T10:36:26.250505+0000","last_undegraded":"2022-07-03T10:36:26.250505+0000","last_fullsized":"2022-07-03T10:36:26.250505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:36:35.804481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'30","reported_seq":248,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:44.057423+0000","last_change":"2022-07-03T10:35:02.868521+0000","last_active":"2022-07-03T10:36:44.057423+0000","last_peered":"2022-07-03T10:36:44.057423+0000","last_clean":"2022-07-03T10:36:44.057423+0000","last_became_active":"2022-07-03T10:35:02.868187+0000","last_became_peered":"2022-07-03T10:35:02.868187+0000","last_unstale":"2022-07-03T10:36:44.057423+0000","last_undegraded":"2022-07-03T10:36:44.057423+0000","last_fullsized":"2022-07-03T10:36:44.057423+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:26:52.705187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54044,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":205,"num_read_kb":155,"num_write":30,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.235656+0000","last_change":"2022-07-03T10:35:03.455469+0000","last_active":"2022-07-03T10:36:26.235656+0000","last_peered":"2022-07-03T10:36:26.235656+0000","last_clean":"2022-07-03T10:36:26.235656+0000","last_became_active":"2022-07-03T10:35:03.455090+0000","last_became_peered":"2022-07-03T10:35:03.455090+0000","last_unstale":"2022-07-03T10:36:26.235656+0000","last_undegraded":"2022-07-03T10:36:26.235656+0000","last_fullsized":"2022-07-03T10:36:26.235656+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:18:49.272050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.326689+0000","last_change":"2022-07-03T10:35:27.326689+0000","last_active":"2022-07-03T10:35:27.326689+0000","last_peered":"2022-07-03T10:35:27.326689+0000","last_clean":"2022-07-03T10:35:27.326689+0000","last_became_active":"2022-07-03T10:35:27.320346+0000","last_became_peered":"2022-07-03T10:35:27.320346+0000","last_unstale":"2022-07-03T10:35:27.326689+0000","last_undegraded":"2022-07-03T10:35:27.326689+0000","last_fullsized":"2022-07-03T10:35:27.326689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:57:55.061972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250431+0000","last_change":"2022-07-03T10:35:03.450803+0000","last_active":"2022-07-03T10:36:26.250431+0000","last_peered":"2022-07-03T10:36:26.250431+0000","last_clean":"2022-07-03T10:36:26.250431+0000","last_became_active":"2022-07-03T10:35:03.450111+0000","last_became_peered":"2022-07-03T10:35:03.450111+0000","last_unstale":"2022-07-03T10:36:26.250431+0000","last_undegraded":"2022-07-03T10:36:26.250431+0000","last_fullsized":"2022-07-03T10:36:26.250431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:45:36.850685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305459+0000","last_change":"2022-07-03T10:35:02.870917+0000","last_active":"2022-07-03T10:35:26.305459+0000","last_peered":"2022-07-03T10:35:26.305459+0000","last_clean":"2022-07-03T10:35:26.305459+0000","last_became_active":"2022-07-03T10:35:02.870567+0000","last_became_peered":"2022-07-03T10:35:02.870567+0000","last_unstale":"2022-07-03T10:35:26.305459+0000","last_undegraded":"2022-07-03T10:35:26.305459+0000","last_fullsized":"2022-07-03T10:35:26.305459+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:28:26.922970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245612+0000","last_change":"2022-07-03T10:35:03.449051+0000","last_active":"2022-07-03T10:36:26.245612+0000","last_peered":"2022-07-03T10:36:26.245612+0000","last_clean":"2022-07-03T10:36:26.245612+0000","last_became_active":"2022-07-03T10:35:03.448746+0000","last_became_peered":"2022-07-03T10:35:03.448746+0000","last_unstale":"2022-07-03T10:36:26.245612+0000","last_undegraded":"2022-07-03T10:36:26.245612+0000","last_fullsized":"2022-07-03T10:36:26.245612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:19:29.794764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302039+0000","last_change":"2022-07-03T10:35:02.872424+0000","last_active":"2022-07-03T10:35:26.302039+0000","last_peered":"2022-07-03T10:35:26.302039+0000","last_clean":"2022-07-03T10:35:26.302039+0000","last_became_active":"2022-07-03T10:35:02.872068+0000","last_became_peered":"2022-07-03T10:35:02.872068+0000","last_unstale":"2022-07-03T10:35:26.302039+0000","last_undegraded":"2022-07-03T10:35:26.302039+0000","last_fullsized":"2022-07-03T10:35:26.302039+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:27:37.438080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241776+0000","last_change":"2022-07-03T10:35:03.450235+0000","last_active":"2022-07-03T10:36:26.241776+0000","last_peered":"2022-07-03T10:36:26.241776+0000","last_clean":"2022-07-03T10:36:26.241776+0000","last_became_active":"2022-07-03T10:35:03.449878+0000","last_became_peered":"2022-07-03T10:35:03.449878+0000","last_unstale":"2022-07-03T10:36:26.241776+0000","last_undegraded":"2022-07-03T10:36:26.241776+0000","last_fullsized":"2022-07-03T10:36:26.241776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:03:45.510465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'8","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.308728+0000","last_change":"2022-07-03T10:35:02.879536+0000","last_active":"2022-07-03T10:35:26.308728+0000","last_peered":"2022-07-03T10:35:26.308728+0000","last_clean":"2022-07-03T10:35:26.308728+0000","last_became_active":"2022-07-03T10:35:02.879116+0000","last_became_peered":"2022-07-03T10:35:02.879116+0000","last_unstale":"2022-07-03T10:35:26.308728+0000","last_undegraded":"2022-07-03T10:35:26.308728+0000","last_fullsized":"2022-07-03T10:35:26.308728+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:59:03.908995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"82'6","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.248392+0000","last_change":"2022-07-03T10:35:27.623637+0000","last_active":"2022-07-03T10:36:26.248392+0000","last_peered":"2022-07-03T10:36:26.248392+0000","last_clean":"2022-07-03T10:36:26.248392+0000","last_became_active":"2022-07-03T10:35:27.623202+0000","last_became_peered":"2022-07-03T10:35:27.623202+0000","last_unstale":"2022-07-03T10:36:26.248392+0000","last_undegraded":"2022-07-03T10:36:26.248392+0000","last_fullsized":"2022-07-03T10:36:26.248392+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:39:41.407533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.310134+0000","last_change":"2022-07-03T10:35:02.868113+0000","last_active":"2022-07-03T10:35:26.310134+0000","last_peered":"2022-07-03T10:35:26.310134+0000","last_clean":"2022-07-03T10:35:26.310134+0000","last_became_active":"2022-07-03T10:35:02.867797+0000","last_became_peered":"2022-07-03T10:35:02.867797+0000","last_unstale":"2022-07-03T10:35:26.310134+0000","last_undegraded":"2022-07-03T10:35:26.310134+0000","last_fullsized":"2022-07-03T10:35:26.310134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:09:14.642081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.240643+0000","last_change":"2022-07-03T10:35:03.450159+0000","last_active":"2022-07-03T10:36:26.240643+0000","last_peered":"2022-07-03T10:36:26.240643+0000","last_clean":"2022-07-03T10:36:26.240643+0000","last_became_active":"2022-07-03T10:35:03.449875+0000","last_became_peered":"2022-07-03T10:35:03.449875+0000","last_unstale":"2022-07-03T10:36:26.240643+0000","last_undegraded":"2022-07-03T10:36:26.240643+0000","last_fullsized":"2022-07-03T10:36:26.240643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T16:38:39.399387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.324898+0000","last_change":"2022-07-03T10:35:02.883582+0000","last_active":"2022-07-03T10:35:26.324898+0000","last_peered":"2022-07-03T10:35:26.324898+0000","last_clean":"2022-07-03T10:35:26.324898+0000","last_became_active":"2022-07-03T10:35:02.883205+0000","last_became_peered":"2022-07-03T10:35:02.883205+0000","last_unstale":"2022-07-03T10:35:26.324898+0000","last_undegraded":"2022-07-03T10:35:26.324898+0000","last_fullsized":"2022-07-03T10:35:26.324898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:00:54.232978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.244794+0000","last_change":"2022-07-03T10:35:03.442114+0000","last_active":"2022-07-03T10:36:26.244794+0000","last_peered":"2022-07-03T10:36:26.244794+0000","last_clean":"2022-07-03T10:36:26.244794+0000","last_became_active":"2022-07-03T10:35:03.441436+0000","last_became_peered":"2022-07-03T10:35:03.441436+0000","last_unstale":"2022-07-03T10:36:26.244794+0000","last_undegraded":"2022-07-03T10:36:26.244794+0000","last_fullsized":"2022-07-03T10:36:26.244794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T10:35:10.145754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347744+0000","last_change":"2022-07-03T10:35:02.868513+0000","last_active":"2022-07-03T10:35:26.347744+0000","last_peered":"2022-07-03T10:35:26.347744+0000","last_clean":"2022-07-03T10:35:26.347744+0000","last_became_active":"2022-07-03T10:35:02.867812+0000","last_became_peered":"2022-07-03T10:35:02.867812+0000","last_unstale":"2022-07-03T10:35:26.347744+0000","last_undegraded":"2022-07-03T10:35:26.347744+0000","last_fullsized":"2022-07-03T10:35:26.347744+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:52:47.609836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255206+0000","last_change":"2022-07-03T10:35:03.458922+0000","last_active":"2022-07-03T10:36:26.255206+0000","last_peered":"2022-07-03T10:36:26.255206+0000","last_clean":"2022-07-03T10:36:26.255206+0000","last_became_active":"2022-07-03T10:35:03.458609+0000","last_became_peered":"2022-07-03T10:35:03.458609+0000","last_unstale":"2022-07-03T10:36:26.255206+0000","last_undegraded":"2022-07-03T10:36:26.255206+0000","last_fullsized":"2022-07-03T10:36:26.255206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:52:40.473605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245693+0000","last_change":"2022-07-03T10:35:03.446088+0000","last_active":"2022-07-03T10:36:26.245693+0000","last_peered":"2022-07-03T10:36:26.245693+0000","last_clean":"2022-07-03T10:36:26.245693+0000","last_became_active":"2022-07-03T10:35:03.445814+0000","last_became_peered":"2022-07-03T10:35:03.445814+0000","last_unstale":"2022-07-03T10:36:26.245693+0000","last_undegraded":"2022-07-03T10:36:26.245693+0000","last_fullsized":"2022-07-03T10:36:26.245693+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:26:44.454798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.332452+0000","last_change":"2022-07-03T10:35:02.874441+0000","last_active":"2022-07-03T10:35:26.332452+0000","last_peered":"2022-07-03T10:35:26.332452+0000","last_clean":"2022-07-03T10:35:26.332452+0000","last_became_active":"2022-07-03T10:35:02.874061+0000","last_became_peered":"2022-07-03T10:35:02.874061+0000","last_unstale":"2022-07-03T10:35:26.332452+0000","last_undegraded":"2022-07-03T10:35:26.332452+0000","last_fullsized":"2022-07-03T10:35:26.332452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:31:54.094619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.247527+0000","last_change":"2022-07-03T10:35:03.450766+0000","last_active":"2022-07-03T10:36:26.247527+0000","last_peered":"2022-07-03T10:36:26.247527+0000","last_clean":"2022-07-03T10:36:26.247527+0000","last_became_active":"2022-07-03T10:35:03.450418+0000","last_became_peered":"2022-07-03T10:35:03.450418+0000","last_unstale":"2022-07-03T10:36:26.247527+0000","last_undegraded":"2022-07-03T10:36:26.247527+0000","last_fullsized":"2022-07-03T10:36:26.247527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:07:12.143643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302670+0000","last_change":"2022-07-03T10:35:02.869154+0000","last_active":"2022-07-03T10:35:26.302670+0000","last_peered":"2022-07-03T10:35:26.302670+0000","last_clean":"2022-07-03T10:35:26.302670+0000","last_became_active":"2022-07-03T10:35:02.868806+0000","last_became_peered":"2022-07-03T10:35:02.868806+0000","last_unstale":"2022-07-03T10:35:26.302670+0000","last_undegraded":"2022-07-03T10:35:26.302670+0000","last_fullsized":"2022-07-03T10:35:26.302670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:36:19.440190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":359,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347662+0000","last_change":"2022-07-03T10:33:57.905608+0000","last_active":"2022-07-03T10:35:26.347662+0000","last_peered":"2022-07-03T10:35:26.347662+0000","last_clean":"2022-07-03T10:35:26.347662+0000","last_became_active":"2022-07-03T10:33:57.904940+0000","last_became_peered":"2022-07-03T10:33:57.904940+0000","last_unstale":"2022-07-03T10:35:26.347662+0000","last_undegraded":"2022-07-03T10:35:26.347662+0000","last_fullsized":"2022-07-03T10:35:26.347662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:30:42.693870+0000","last_clean_scrub_stamp":"2022-07-03T10:30:42.693870+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:07:31.406549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.245672+0000","last_change":"2022-07-03T10:35:03.459586+0000","last_active":"2022-07-03T10:36:26.245672+0000","last_peered":"2022-07-03T10:36:26.245672+0000","last_clean":"2022-07-03T10:36:26.245672+0000","last_became_active":"2022-07-03T10:35:03.459273+0000","last_became_peered":"2022-07-03T10:35:03.459273+0000","last_unstale":"2022-07-03T10:36:26.245672+0000","last_undegraded":"2022-07-03T10:36:26.245672+0000","last_fullsized":"2022-07-03T10:36:26.245672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:33:01.520965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.347752+0000","last_change":"2022-07-03T10:35:02.871273+0000","last_active":"2022-07-03T10:35:26.347752+0000","last_peered":"2022-07-03T10:35:26.347752+0000","last_clean":"2022-07-03T10:35:26.347752+0000","last_became_active":"2022-07-03T10:35:02.870932+0000","last_became_peered":"2022-07-03T10:35:02.870932+0000","last_unstale":"2022-07-03T10:35:26.347752+0000","last_undegraded":"2022-07-03T10:35:26.347752+0000","last_fullsized":"2022-07-03T10:35:26.347752+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:13:56.000997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309497+0000","last_change":"2022-07-03T10:35:02.879548+0000","last_active":"2022-07-03T10:35:26.309497+0000","last_peered":"2022-07-03T10:35:26.309497+0000","last_clean":"2022-07-03T10:35:26.309497+0000","last_became_active":"2022-07-03T10:35:02.879128+0000","last_became_peered":"2022-07-03T10:35:02.879128+0000","last_unstale":"2022-07-03T10:35:26.309497+0000","last_undegraded":"2022-07-03T10:35:26.309497+0000","last_fullsized":"2022-07-03T10:35:26.309497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:19.770269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.253049+0000","last_change":"2022-07-03T10:35:03.467431+0000","last_active":"2022-07-03T10:36:26.253049+0000","last_peered":"2022-07-03T10:36:26.253049+0000","last_clean":"2022-07-03T10:36:26.253049+0000","last_became_active":"2022-07-03T10:35:03.467035+0000","last_became_peered":"2022-07-03T10:35:03.467035+0000","last_unstale":"2022-07-03T10:36:26.253049+0000","last_undegraded":"2022-07-03T10:36:26.253049+0000","last_fullsized":"2022-07-03T10:36:26.253049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:36:51.883730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"82'4","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.415059+0000","last_change":"2022-07-03T10:35:27.414221+0000","last_active":"2022-07-03T10:35:27.415059+0000","last_peered":"2022-07-03T10:35:27.415059+0000","last_clean":"2022-07-03T10:35:27.415059+0000","last_became_active":"2022-07-03T10:35:27.323296+0000","last_became_peered":"2022-07-03T10:35:27.323296+0000","last_unstale":"2022-07-03T10:35:27.415059+0000","last_undegraded":"2022-07-03T10:35:27.415059+0000","last_fullsized":"2022-07-03T10:35:27.415059+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:01:20.773865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451855+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451540+0000","last_became_peered":"2022-07-03T10:35:03.451540+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:13:43.991819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.302521+0000","last_change":"2022-07-03T10:35:02.871425+0000","last_active":"2022-07-03T10:35:26.302521+0000","last_peered":"2022-07-03T10:35:26.302521+0000","last_clean":"2022-07-03T10:35:26.302521+0000","last_became_active":"2022-07-03T10:35:02.871093+0000","last_became_peered":"2022-07-03T10:35:02.871093+0000","last_unstale":"2022-07-03T10:35:26.302521+0000","last_undegraded":"2022-07-03T10:35:26.302521+0000","last_fullsized":"2022-07-03T10:35:26.302521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:21:00.958228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"82'7","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.260780+0000","last_change":"2022-07-03T10:35:27.325484+0000","last_active":"2022-07-03T10:36:26.260780+0000","last_peered":"2022-07-03T10:36:26.260780+0000","last_clean":"2022-07-03T10:36:26.260780+0000","last_became_active":"2022-07-03T10:35:27.324947+0000","last_became_peered":"2022-07-03T10:35:27.324947+0000","last_unstale":"2022-07-03T10:36:26.260780+0000","last_undegraded":"2022-07-03T10:36:26.260780+0000","last_fullsized":"2022-07-03T10:36:26.260780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:36.475275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309995+0000","last_change":"2022-07-03T10:35:02.879377+0000","last_active":"2022-07-03T10:35:26.309995+0000","last_peered":"2022-07-03T10:35:26.309995+0000","last_clean":"2022-07-03T10:35:26.309995+0000","last_became_active":"2022-07-03T10:35:02.879020+0000","last_became_peered":"2022-07-03T10:35:02.879020+0000","last_unstale":"2022-07-03T10:35:26.309995+0000","last_undegraded":"2022-07-03T10:35:26.309995+0000","last_fullsized":"2022-07-03T10:35:26.309995+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:45:37.856189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.214344+0000","last_change":"2022-07-03T10:35:03.455039+0000","last_active":"2022-07-03T10:36:26.214344+0000","last_peered":"2022-07-03T10:36:26.214344+0000","last_clean":"2022-07-03T10:36:26.214344+0000","last_became_active":"2022-07-03T10:35:03.454610+0000","last_became_peered":"2022-07-03T10:35:03.454610+0000","last_unstale":"2022-07-03T10:36:26.214344+0000","last_undegraded":"2022-07-03T10:36:26.214344+0000","last_fullsized":"2022-07-03T10:36:26.214344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:32:09.647222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.341921+0000","last_change":"2022-07-03T10:35:02.862953+0000","last_active":"2022-07-03T10:35:26.341921+0000","last_peered":"2022-07-03T10:35:26.341921+0000","last_clean":"2022-07-03T10:35:26.341921+0000","last_became_active":"2022-07-03T10:35:02.862616+0000","last_became_peered":"2022-07-03T10:35:02.862616+0000","last_unstale":"2022-07-03T10:35:26.341921+0000","last_undegraded":"2022-07-03T10:35:26.341921+0000","last_fullsized":"2022-07-03T10:35:26.341921+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:09:53.538033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242220+0000","last_change":"2022-07-03T10:35:03.460792+0000","last_active":"2022-07-03T10:36:26.242220+0000","last_peered":"2022-07-03T10:36:26.242220+0000","last_clean":"2022-07-03T10:36:26.242220+0000","last_became_active":"2022-07-03T10:35:03.460459+0000","last_became_peered":"2022-07-03T10:35:03.460459+0000","last_unstale":"2022-07-03T10:36:26.242220+0000","last_undegraded":"2022-07-03T10:36:26.242220+0000","last_fullsized":"2022-07-03T10:36:26.242220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T20:16:23.857016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.303560+0000","last_change":"2022-07-03T10:35:02.870161+0000","last_active":"2022-07-03T10:35:26.303560+0000","last_peered":"2022-07-03T10:35:26.303560+0000","last_clean":"2022-07-03T10:35:26.303560+0000","last_became_active":"2022-07-03T10:35:02.869841+0000","last_became_peered":"2022-07-03T10:35:02.869841+0000","last_unstale":"2022-07-03T10:35:26.303560+0000","last_undegraded":"2022-07-03T10:35:26.303560+0000","last_fullsized":"2022-07-03T10:35:26.303560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T18:24:08.278792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256486+0000","last_change":"2022-07-03T10:35:03.464601+0000","last_active":"2022-07-03T10:36:26.256486+0000","last_peered":"2022-07-03T10:36:26.256486+0000","last_clean":"2022-07-03T10:36:26.256486+0000","last_became_active":"2022-07-03T10:35:03.464150+0000","last_became_peered":"2022-07-03T10:35:03.464150+0000","last_unstale":"2022-07-03T10:36:26.256486+0000","last_undegraded":"2022-07-03T10:36:26.256486+0000","last_fullsized":"2022-07-03T10:36:26.256486+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T14:00:09.740180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.309116+0000","last_change":"2022-07-03T10:35:02.871000+0000","last_active":"2022-07-03T10:35:26.309116+0000","last_peered":"2022-07-03T10:35:26.309116+0000","last_clean":"2022-07-03T10:35:26.309116+0000","last_became_active":"2022-07-03T10:35:02.870633+0000","last_became_peered":"2022-07-03T10:35:02.870633+0000","last_unstale":"2022-07-03T10:35:26.309116+0000","last_undegraded":"2022-07-03T10:35:26.309116+0000","last_fullsized":"2022-07-03T10:35:26.309116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:57:35.430142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"82'5","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246522+0000","last_change":"2022-07-03T10:35:03.449080+0000","last_active":"2022-07-03T10:36:26.246522+0000","last_peered":"2022-07-03T10:36:26.246522+0000","last_clean":"2022-07-03T10:36:26.246522+0000","last_became_active":"2022-07-03T10:35:03.448788+0000","last_became_peered":"2022-07-03T10:35:03.448788+0000","last_unstale":"2022-07-03T10:36:26.246522+0000","last_undegraded":"2022-07-03T10:36:26.246522+0000","last_fullsized":"2022-07-03T10:36:26.246522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:13:00.181729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:35:27.312207+0000","last_change":"2022-07-03T10:35:27.312207+0000","last_active":"2022-07-03T10:35:27.312207+0000","last_peered":"2022-07-03T10:35:27.312207+0000","last_clean":"2022-07-03T10:35:27.312207+0000","last_became_active":"2022-07-03T10:35:27.311839+0000","last_became_peered":"2022-07-03T10:35:27.311839+0000","last_unstale":"2022-07-03T10:35:27.312207+0000","last_undegraded":"2022-07-03T10:35:27.312207+0000","last_fullsized":"2022-07-03T10:35:27.312207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:21:12.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.246912+0000","last_change":"2022-07-03T10:35:03.453553+0000","last_active":"2022-07-03T10:36:26.246912+0000","last_peered":"2022-07-03T10:36:26.246912+0000","last_clean":"2022-07-03T10:36:26.246912+0000","last_became_active":"2022-07-03T10:35:03.453223+0000","last_became_peered":"2022-07-03T10:35:03.453223+0000","last_unstale":"2022-07-03T10:36:26.246912+0000","last_undegraded":"2022-07-03T10:36:26.246912+0000","last_fullsized":"2022-07-03T10:36:26.246912+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:27:27.615348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.305852+0000","last_change":"2022-07-03T10:35:02.866496+0000","last_active":"2022-07-03T10:35:26.305852+0000","last_peered":"2022-07-03T10:35:26.305852+0000","last_clean":"2022-07-03T10:35:26.305852+0000","last_became_active":"2022-07-03T10:35:02.866099+0000","last_became_peered":"2022-07-03T10:35:02.866099+0000","last_unstale":"2022-07-03T10:35:26.305852+0000","last_undegraded":"2022-07-03T10:35:26.305852+0000","last_fullsized":"2022-07-03T10:35:26.305852+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T11:28:18.628009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242322+0000","last_change":"2022-07-03T10:35:03.457460+0000","last_active":"2022-07-03T10:36:26.242322+0000","last_peered":"2022-07-03T10:36:26.242322+0000","last_clean":"2022-07-03T10:36:26.242322+0000","last_became_active":"2022-07-03T10:35:03.457124+0000","last_became_peered":"2022-07-03T10:35:03.457124+0000","last_unstale":"2022-07-03T10:36:26.242322+0000","last_undegraded":"2022-07-03T10:36:26.242322+0000","last_fullsized":"2022-07-03T10:36:26.242322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:47:51.082268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-07-03T10:35:26.330620+0000","last_change":"2022-07-03T10:35:02.870154+0000","last_active":"2022-07-03T10:35:26.330620+0000","last_peered":"2022-07-03T10:35:26.330620+0000","last_clean":"2022-07-03T10:35:26.330620+0000","last_became_active":"2022-07-03T10:35:02.869697+0000","last_became_peered":"2022-07-03T10:35:02.869697+0000","last_unstale":"2022-07-03T10:35:26.330620+0000","last_undegraded":"2022-07-03T10:35:26.330620+0000","last_fullsized":"2022-07-03T10:35:26.330620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:01.613866+0000","last_clean_scrub_stamp":"2022-07-03T10:35:01.613866+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T17:33:52.456927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.237445+0000","last_change":"2022-07-03T10:35:03.455967+0000","last_active":"2022-07-03T10:36:26.237445+0000","last_peered":"2022-07-03T10:36:26.237445+0000","last_clean":"2022-07-03T10:36:26.237445+0000","last_became_active":"2022-07-03T10:35:03.455596+0000","last_became_peered":"2022-07-03T10:35:03.455596+0000","last_unstale":"2022-07-03T10:36:26.237445+0000","last_undegraded":"2022-07-03T10:36:26.237445+0000","last_fullsized":"2022-07-03T10:36:26.237445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:20:29.927509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'9","reported_seq":21,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242972+0000","last_change":"2022-07-03T10:35:03.450226+0000","last_active":"2022-07-03T10:36:26.242972+0000","last_peered":"2022-07-03T10:36:26.242972+0000","last_clean":"2022-07-03T10:36:26.242972+0000","last_became_active":"2022-07-03T10:35:03.449906+0000","last_became_peered":"2022-07-03T10:35:03.449906+0000","last_unstale":"2022-07-03T10:36:26.242972+0000","last_undegraded":"2022-07-03T10:36:26.242972+0000","last_fullsized":"2022-07-03T10:36:26.242972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:24:30.066903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"82'12","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.250077+0000","last_change":"2022-07-03T10:35:27.313043+0000","last_active":"2022-07-03T10:36:26.250077+0000","last_peered":"2022-07-03T10:36:26.250077+0000","last_clean":"2022-07-03T10:36:26.250077+0000","last_became_active":"2022-07-03T10:35:27.312553+0000","last_became_peered":"2022-07-03T10:35:27.312553+0000","last_unstale":"2022-07-03T10:36:26.250077+0000","last_undegraded":"2022-07-03T10:36:26.250077+0000","last_fullsized":"2022-07-03T10:36:26.250077+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:05:28.297497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.256573+0000","last_change":"2022-07-03T10:35:03.457798+0000","last_active":"2022-07-03T10:36:26.256573+0000","last_peered":"2022-07-03T10:36:26.256573+0000","last_clean":"2022-07-03T10:36:26.256573+0000","last_became_active":"2022-07-03T10:35:03.457501+0000","last_became_peered":"2022-07-03T10:35:03.457501+0000","last_unstale":"2022-07-03T10:36:26.256573+0000","last_undegraded":"2022-07-03T10:36:26.256573+0000","last_fullsized":"2022-07-03T10:36:26.256573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T19:57:21.370253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"82'7","reported_seq":19,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.259834+0000","last_change":"2022-07-03T10:35:03.458713+0000","last_active":"2022-07-03T10:36:26.259834+0000","last_peered":"2022-07-03T10:36:26.259834+0000","last_clean":"2022-07-03T10:36:26.259834+0000","last_became_active":"2022-07-03T10:35:03.458265+0000","last_became_peered":"2022-07-03T10:35:03.458265+0000","last_unstale":"2022-07-03T10:36:26.259834+0000","last_undegraded":"2022-07-03T10:36:26.259834+0000","last_fullsized":"2022-07-03T10:36:26.259834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:13:37.429306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.249554+0000","last_change":"2022-07-03T10:35:03.467756+0000","last_active":"2022-07-03T10:36:26.249554+0000","last_peered":"2022-07-03T10:36:26.249554+0000","last_clean":"2022-07-03T10:36:26.249554+0000","last_became_active":"2022-07-03T10:35:03.467167+0000","last_became_peered":"2022-07-03T10:35:03.467167+0000","last_unstale":"2022-07-03T10:36:26.249554+0000","last_undegraded":"2022-07-03T10:36:26.249554+0000","last_fullsized":"2022-07-03T10:36:26.249554+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:49:03.226109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241377+0000","last_change":"2022-07-03T10:35:03.454702+0000","last_active":"2022-07-03T10:36:26.241377+0000","last_peered":"2022-07-03T10:36:26.241377+0000","last_clean":"2022-07-03T10:36:26.241377+0000","last_became_active":"2022-07-03T10:35:03.454369+0000","last_became_peered":"2022-07-03T10:35:03.454369+0000","last_unstale":"2022-07-03T10:36:26.241377+0000","last_undegraded":"2022-07-03T10:36:26.241377+0000","last_fullsized":"2022-07-03T10:36:26.241377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:39:28.500238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"82'15","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.242254+0000","last_change":"2022-07-03T10:35:03.458953+0000","last_active":"2022-07-03T10:36:26.242254+0000","last_peered":"2022-07-03T10:36:26.242254+0000","last_clean":"2022-07-03T10:36:26.242254+0000","last_became_active":"2022-07-03T10:35:03.458721+0000","last_became_peered":"2022-07-03T10:35:03.458721+0000","last_unstale":"2022-07-03T10:36:26.242254+0000","last_undegraded":"2022-07-03T10:36:26.242254+0000","last_fullsized":"2022-07-03T10:36:26.242254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:08:32.053556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.243944+0000","last_change":"2022-07-03T10:35:03.456103+0000","last_active":"2022-07-03T10:36:26.243944+0000","last_peered":"2022-07-03T10:36:26.243944+0000","last_clean":"2022-07-03T10:36:26.243944+0000","last_became_active":"2022-07-03T10:35:03.455806+0000","last_became_peered":"2022-07-03T10:35:03.455806+0000","last_unstale":"2022-07-03T10:36:26.243944+0000","last_undegraded":"2022-07-03T10:36:26.243944+0000","last_fullsized":"2022-07-03T10:36:26.243944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:52.332569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255263+0000","last_change":"2022-07-03T10:35:03.450023+0000","last_active":"2022-07-03T10:36:26.255263+0000","last_peered":"2022-07-03T10:36:26.255263+0000","last_clean":"2022-07-03T10:36:26.255263+0000","last_became_active":"2022-07-03T10:35:03.449605+0000","last_became_peered":"2022-07-03T10:35:03.449605+0000","last_unstale":"2022-07-03T10:36:26.255263+0000","last_undegraded":"2022-07-03T10:36:26.255263+0000","last_fullsized":"2022-07-03T10:36:26.255263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T13:46:36.926162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241678+0000","last_change":"2022-07-03T10:35:03.451895+0000","last_active":"2022-07-03T10:36:26.241678+0000","last_peered":"2022-07-03T10:36:26.241678+0000","last_clean":"2022-07-03T10:36:26.241678+0000","last_became_active":"2022-07-03T10:35:03.451567+0000","last_became_peered":"2022-07-03T10:35:03.451567+0000","last_unstale":"2022-07-03T10:36:26.241678+0000","last_undegraded":"2022-07-03T10:36:26.241678+0000","last_fullsized":"2022-07-03T10:36:26.241678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T15:30:58.922354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"82'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.255288+0000","last_change":"2022-07-03T10:35:03.462507+0000","last_active":"2022-07-03T10:36:26.255288+0000","last_peered":"2022-07-03T10:36:26.255288+0000","last_clean":"2022-07-03T10:36:26.255288+0000","last_became_active":"2022-07-03T10:35:03.462177+0000","last_became_peered":"2022-07-03T10:35:03.462177+0000","last_unstale":"2022-07-03T10:36:26.255288+0000","last_undegraded":"2022-07-03T10:36:26.255288+0000","last_fullsized":"2022-07-03T10:36:26.255288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T21:18:04.125762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"82'11","reported_seq":23,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.258181+0000","last_change":"2022-07-03T10:35:03.466207+0000","last_active":"2022-07-03T10:36:26.258181+0000","last_peered":"2022-07-03T10:36:26.258181+0000","last_clean":"2022-07-03T10:36:26.258181+0000","last_became_active":"2022-07-03T10:35:03.465597+0000","last_became_peered":"2022-07-03T10:35:03.465597+0000","last_unstale":"2022-07-03T10:36:26.258181+0000","last_undegraded":"2022-07-03T10:36:26.258181+0000","last_fullsized":"2022-07-03T10:36:26.258181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T22:01:56.776581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-07-03T10:36:26.241416+0000","last_change":"2022-07-03T10:35:03.455982+0000","last_active":"2022-07-03T10:36:26.241416+0000","last_peered":"2022-07-03T10:36:26.241416+0000","last_clean":"2022-07-03T10:36:26.241416+0000","last_became_active":"2022-07-03T10:35:03.455675+0000","last_became_peered":"2022-07-03T10:35:03.455675+0000","last_unstale":"2022-07-03T10:36:26.241416+0000","last_undegraded":"2022-07-03T10:36:26.241416+0000","last_fullsized":"2022-07-03T10:36:26.241416+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-03T10:35:02.839839+0000","last_clean_scrub_stamp":"2022-07-03T10:35:02.839839+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-07-04T12:14:36.064272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245568,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54826,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":210,"num_read_kb":160,"num_write":75,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":164478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547233,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5992,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739160,"statfs":{"total":95995035648,"available":95988899840,"internally_reserved":0,"allocated":884736,"data_stored":436583,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81899999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.501,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67300000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65800000000000003}]},{"osd":4,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.58799999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.52500000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.875,"5min":1.026,"15min":1.026},"last":0.625}]},{"osd":5,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85799999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.623},"min":{"1min":0.41899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92300000000000004,"5min":1.151,"15min":1.151},"last":0.68300000000000005}]},{"osd":6,"last update":"Sun Jul 3 10:35:15 2022","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.83399999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.59299999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.76000000000000001,"15min":0.76100000000000001},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.96799999999999997,"5min":1.075,"15min":1.075},"last":0.69399999999999995}]},{"osd":8,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72699999999999998}]},{"osd":9,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.80600000000000005,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.7229999999999999,"5min":2.7229999999999999,"15min":2.7229999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81799999999999995,"15min":0.81999999999999995},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.044,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.77600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.78100000000000003,"15min":0.77700000000000002},"min":{"1min":0.63900000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.999,"5min":1.115,"15min":1.115},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.61399999999999999,"5min":0.496,"15min":0.496},"max":{"1min":0.94999999999999996,"5min":1.179,"15min":1.179},"last":0.63700000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6368,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738784,"statfs":{"total":95995035648,"available":95988514816,"internally_reserved":0,"allocated":1007616,"data_stored":1080955,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.437},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.52600000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.55500000000000005,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.41099999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.66500000000000004,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.71899999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65000000000000002}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.64700000000000002,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.75700000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.46200000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.316,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71399999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.53300000000000003}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.64600000000000002,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70099999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.85899999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.77600000000000002,"15min":0.79300000000000004},"min":{"1min":0.52900000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92800000000000005,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.72699999999999998,"15min":0.73299999999999998},"min":{"1min":0.48499999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.61299999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:34:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54100000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69499999999999995,"15min":0.69999999999999996},"min":{"1min":0.46400000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.74199999999999999,"15min":0.753},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.84699999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.75800000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.81399999999999995,"15min":0.82699999999999996},"min":{"1min":0.57899999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.879,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.80600000000000005,"15min":0.81499999999999995},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.67500000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.73699999999999999,"15min":0.74399999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.79000000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.73699999999999999,"15min":0.73899999999999999},"min":{"1min":0.54600000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91100000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.72799999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972395,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":884736,"data_stored":437115,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.80500000000000005,"15min":0.71199999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6359999999999999,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.442},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.89800000000000002,"15min":0.77800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0720000000000001,"5min":11.582000000000001,"15min":11.582000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.93000000000000005,"15min":0.84999999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.905,"5min":11.933,"15min":11.933},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.96299999999999997,"15min":0.92300000000000004},"min":{"1min":0.55600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.5669999999999999,"5min":11.411,"15min":11.411},"last":0.89000000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.94099999999999995,"15min":0.85499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.944,"5min":11.887,"15min":11.887},"last":0.748},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.94699999999999995,"15min":0.89500000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6080000000000001,"5min":11.331,"15min":11.331},"last":0.95099999999999996}]},{"osd":3,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.85699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2490000000000001,"5min":9.8859999999999992,"15min":9.8859999999999992},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.95399999999999996,"15min":0.84399999999999997},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8009999999999999,"5min":12.041,"15min":12.041},"last":0.60899999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.016,"15min":0.97599999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":11.369,"15min":11.369},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":1,"15min":0.95199999999999996},"min":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95299999999999996,"5min":9.6769999999999996,"15min":9.6769999999999996},"last":0.82799999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:34:33 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":9.734},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":11.968}]},{"osd":6,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.0880000000000001,"15min":1.028},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.869,"5min":12.311,"15min":12.311},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1040000000000001,"15min":1.0700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.0419999999999998,"5min":11.534000000000001,"15min":11.534000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.0860000000000001,"15min":1.0529999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.272,"5min":9.8079999999999998,"15min":9.8079999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.1419999999999999,"15min":1.0860000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.0019999999999998,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.78400000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.147,"15min":1.125},"min":{"1min":0.60999999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.6639999999999999,"5min":11.441000000000001,"15min":11.441000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.1160000000000001,"15min":1.0900000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1839999999999999,"5min":9.8539999999999992,"15min":9.8539999999999992},"last":0.90500000000000003}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.1459999999999999,"15min":1.0920000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":11.474,"5min":11.474,"15min":11.474},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.107,"15min":1.0549999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":9.7729999999999997,"5min":9.7729999999999997,"15min":9.7729999999999997},"last":0.93500000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":4.6399999999999997,"5min":4.6399999999999997,"15min":4.6399999999999997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.008}]}]},{"osd":8,"up_from":54,"seq":231928234031,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6252,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738900,"statfs":{"total":95995035648,"available":95988633600,"internally_reserved":0,"allocated":888832,"data_stored":437137,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67200000000000004,"15min":0.63},"min":{"1min":0.47199999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.871,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81799999999999995,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.47599999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.57199999999999995,"15min":0.55200000000000005},"min":{"1min":0.371,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80700000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.38800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72999999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61299999999999999}]},{"osd":2,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42899999999999999,"15min":0.41499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.437},"min":{"1min":0.34899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.63700000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48399999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.52700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.62}]},{"osd":4,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.57499999999999996,"15min":0.54800000000000004},"min":{"1min":0.45200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.67900000000000005,"15min":0.621},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83699999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.77200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.311,"15min":0.311},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.69999999999999996}]},{"osd":6,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.75900000000000001,"15min":0.70299999999999996},"min":{"1min":0.47699999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.357,"5min":4.6959999999999997,"15min":4.6959999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77800000000000002,"15min":0.73399999999999999},"min":{"1min":0.56499999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92000000000000004,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:36:14 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67000000000000004,"15min":0.65400000000000003},"min":{"1min":0.51200000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.66700000000000004,"15min":0.64700000000000002},"min":{"1min":0.57499999999999996,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.82799999999999996}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.75900000000000001},"min":{"1min":0.64000000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.77200000000000002,"15min":0.76800000000000002},"min":{"1min":0.63700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.997,"5min":1.117,"15min":1.117},"last":0.82399999999999995}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70499999999999996,"15min":0.69799999999999995},"min":{"1min":0.51300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80600000000000005}]},{"osd":11,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.83999999999999997}]}]},{"osd":7,"up_from":47,"seq":201863462963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":937984,"data_stored":1027606,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:34:49 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.56499999999999995},"min":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73399999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81399999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.39800000000000002,"15min":0.37},"min":{"1min":0.33800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.373},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.41899999999999998,"15min":0.40000000000000002},"min":{"1min":0.309,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.45900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56499999999999995,"15min":0.54200000000000004},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.626}]},{"osd":3,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.58699999999999997,"15min":0.56100000000000005},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80100000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":1.222,"15min":1.222},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.48999999999999999,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66500000000000004,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.40200000000000002}]},{"osd":5,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63500000000000001,"15min":0.61899999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61399999999999999},"min":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.84099999999999997,"15min":0.88500000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.94799999999999995,"5min":5.077,"15min":5.077},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.70299999999999996,"15min":0.67100000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.67600000000000005}]},{"osd":8,"last update":"Sun Jul 3 10:36:09 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66400000000000003},"min":{"1min":0.48199999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.69699999999999995,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.96699999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.54400000000000004}]},{"osd":9,"last update":"Sun Jul 3 10:36:42 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.79100000000000004,"15min":0.79700000000000004},"min":{"1min":0.54900000000000004,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.002,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76700000000000002,"15min":0.76400000000000001},"min":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.94699999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.73999999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:58 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.79900000000000004,"15min":0.81100000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.96399999999999997,"5min":1.107,"15min":1.107},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72899999999999998,"15min":0.73199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.034,"15min":1.034},"last":0.70599999999999996}]},{"osd":11,"last update":"Sun Jul 3 10:36:04 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75}]}]},{"osd":6,"up_from":43,"seq":184683593782,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6248,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738904,"statfs":{"total":95995035648,"available":95988637696,"internally_reserved":0,"allocated":884736,"data_stored":436672,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52200000000000002,"15min":0.502},"min":{"1min":0.39100000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.56799999999999995,"15min":0.53300000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.93899999999999995}]},{"osd":1,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.70199999999999996,"15min":0.65400000000000003},"min":{"1min":0.45800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.958,"5min":1.958,"15min":1.958},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71499999999999997,"15min":0.68700000000000006},"min":{"1min":0.55100000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71899999999999997,"15min":0.69499999999999995},"min":{"1min":0.56499999999999995,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.98999999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.748,"15min":0.71899999999999997},"min":{"1min":0.53600000000000003,"5min":0.495,"15min":0.495},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70099999999999996,"15min":0.68100000000000005},"min":{"1min":0.437,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.78200000000000003,"15min":0.72599999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.83899999999999997}]},{"osd":4,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.76700000000000002,"15min":0.755},"min":{"1min":0.51600000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.137,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81100000000000005,"15min":0.79300000000000004},"min":{"1min":0.60999999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3129999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.95399999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.82499999999999996,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.82699999999999996,"15min":0.81100000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0669999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81799999999999995}]},{"osd":7,"last update":"Sun Jul 3 10:35:52 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.86299999999999999,"15min":0.84099999999999997},"min":{"1min":0.56799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2869999999999999,"5min":1.333,"15min":1.333},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.878,"15min":0.86099999999999999},"min":{"1min":0.70399999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.008}]},{"osd":8,"last update":"Sun Jul 3 10:36:06 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.86499999999999999,"15min":0.85199999999999998},"min":{"1min":0.65400000000000003,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90800000000000003,"15min":0.91000000000000003},"min":{"1min":0.67700000000000005,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.121,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.879}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94799999999999995,"15min":0.96599999999999997},"min":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.03},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.93300000000000005,"15min":0.96499999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.89600000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97499999999999998,"15min":0.98299999999999998},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.1279999999999999,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.004,"15min":1.01},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.069}]}]},{"osd":1,"up_from":12,"seq":51539607630,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7268,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737884,"statfs":{"total":95995035648,"available":95987593216,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.38700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.76000000000000001,"5min":0.83299999999999996,"15min":0.83899999999999997},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.57599999999999996},"min":{"1min":0.40899999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69799999999999995,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60699999999999998}]},{"osd":2,"last update":"Sun Jul 3 10:35:47 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.58899999999999997,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.879,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59999999999999998,"15min":0.54800000000000004},"min":{"1min":0.48599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.81399999999999995,"5min":0.877,"15min":0.877},"last":0.86699999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:24 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.55500000000000005},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59099999999999997,"15min":0.55900000000000005},"min":{"1min":0.48499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83899999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Jul 3 10:36:45 2022","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.41399999999999998},"min":{"1min":0.39900000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77300000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.47399999999999998,"15min":0.41799999999999998},"min":{"1min":0.40200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79900000000000004,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.73799999999999999}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.40500000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":1.044,"15min":1.044},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.63200000000000001,"15min":0.60999999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88800000000000001,"5min":1.028,"15min":1.028},"last":0.91300000000000003}]},{"osd":6,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.872,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68000000000000005,"15min":0.66800000000000004},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.89300000000000002,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.70099999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:51 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57199999999999995,"15min":0.54500000000000004},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.84599999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.61599999999999999,"15min":0.58199999999999996},"min":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93600000000000005,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.70099999999999996,"15min":0.68500000000000005},"min":{"1min":0.53200000000000003,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98099999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.88},{"interface":"front","average":{"1min":0.755,"5min":0.67900000000000005,"15min":0.66700000000000004},"min":{"1min":0.57399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.88900000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.79700000000000004,"15min":0.79400000000000004},"min":{"1min":0.626,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0069999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.81699999999999995,"15min":0.81200000000000006},"min":{"1min":0.68500000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0609999999999999,"5min":1.103,"15min":1.103},"last":0.996}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73099999999999998,"15min":0.72899999999999998},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90800000000000003,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.752,"15min":0.748},"min":{"1min":0.53800000000000003,"5min":0.503,"15min":0.503},"max":{"1min":0.89900000000000002,"5min":1.069,"15min":1.069},"last":0.96099999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":1.0349999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7080,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738072,"statfs":{"total":95995035648,"available":95987785728,"internally_reserved":0,"allocated":884736,"data_stored":436639,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.63700000000000001,"15min":0.56399999999999995},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.88800000000000001,"5min":0.91500000000000004,"15min":0.94499999999999995},"last":0.57399999999999995}]},{"osd":2,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.63700000000000001,"15min":0.627},"min":{"1min":0.47199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.68100000000000005,"15min":0.63100000000000001},"min":{"1min":0.52900000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":0.496}]},{"osd":3,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.39500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:36:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.73199999999999998,"15min":0.68000000000000005},"min":{"1min":0.51600000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.69099999999999995,"15min":0.64100000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.70199999999999996}]},{"osd":5,"last update":"Sun Jul 3 10:35:59 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.753,"15min":0.72099999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.73699999999999999,"15min":0.71399999999999997},"min":{"1min":0.59899999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.718,"5min":1.718,"15min":1.718},"last":0.45800000000000002}]},{"osd":6,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80800000000000005,"15min":0.76100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":0.752},{"interface":"front","average":{"1min":0.879,"5min":0.79200000000000004,"15min":0.75800000000000001},"min":{"1min":0.54000000000000004,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Sun Jul 3 10:36:30 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.79000000000000004}]},{"osd":8,"last update":"Sun Jul 3 10:36:12 2022","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.89600000000000002,"15min":0.88800000000000001},"min":{"1min":0.64500000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.3580000000000001,"5min":2.3580000000000001,"15min":2.3580000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.85399999999999998,"15min":0.84999999999999998},"min":{"1min":0.59299999999999997,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.48999999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.84199999999999997,"15min":0.83499999999999996},"min":{"1min":0.64200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97399999999999998,"15min":0.97099999999999997},"min":{"1min":0.66500000000000004,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.76800000000000002}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.90200000000000002,"15min":0.90700000000000003},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72399999999999998}]},{"osd":11,"last update":"Sun Jul 3 10:36:15 2022","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.95499999999999996,"15min":0.93700000000000006},"min":{"1min":0.71299999999999997,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.88500000000000001,"15min":0.878},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.63300000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476812,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7332,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737820,"statfs":{"total":95995035648,"available":95987527680,"internally_reserved":0,"allocated":946176,"data_stored":490593,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.35699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56100000000000005,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.73199999999999998,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.496}]},{"osd":1,"last update":"Sun Jul 3 10:35:50 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.51600000000000001},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.69599999999999995,"5min":0.874,"15min":0.874},"last":0.62}]},{"osd":3,"last update":"Sun Jul 3 10:36:16 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83299999999999996,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.371,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58199999999999996}]},{"osd":4,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.71199999999999997}]},{"osd":5,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.39300000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.501,"15min":0.48499999999999999},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.63500000000000001}]},{"osd":6,"last update":"Sun Jul 3 10:36:32 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.67100000000000004,"15min":0.65100000000000002},"min":{"1min":0.52200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.88900000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66200000000000003,"15min":0.65900000000000003},"min":{"1min":0.44900000000000001,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83599999999999997,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.67600000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:34:50 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.88,"5min":0.71599999999999997,"15min":0.68799999999999994},"min":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":4.9130000000000003,"5min":4.9130000000000003,"15min":4.9130000000000003},"last":0.57899999999999996}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.47699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78400000000000003,"5min":0.871,"15min":0.871},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.55500000000000005,"15min":0.55000000000000004},"min":{"1min":0.42699999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.747,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.69799999999999995}]},{"osd":9,"last update":"Sun Jul 3 10:36:38 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.69499999999999995,"15min":0.69199999999999995},"min":{"1min":0.54600000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.746,"15min":0.74199999999999999},"min":{"1min":0.56799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.88100000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:55 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65000000000000002,"15min":0.64400000000000002},"min":{"1min":0.58399999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65800000000000003,"15min":0.65200000000000002},"min":{"1min":0.57099999999999995,"5min":0.436,"15min":0.436},"max":{"1min":0.80400000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.79900000000000004}]},{"osd":11,"last update":"Sun Jul 3 10:36:02 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72899999999999998}]}]},{"osd":3,"up_from":23,"seq":98784247877,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":897024,"data_stored":436899,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.51600000000000001,"15min":0.44700000000000001},"min":{"1min":0.34100000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.9609999999999999,"5min":3.9609999999999999,"15min":3.9609999999999999},"last":0.433},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53200000000000003,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.65000000000000002,"15min":0.59399999999999997},"min":{"1min":0.44,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.67200000000000004,"15min":0.60299999999999998},"min":{"1min":0.46500000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95199999999999996,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:18 2022","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.73799999999999999,"15min":0.66500000000000004},"min":{"1min":0.443,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.21,"5min":2.581,"15min":2.581},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72499999999999998,"15min":0.66700000000000004},"min":{"1min":0.501,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.903,"5min":4.2649999999999997,"15min":4.2649999999999997},"last":0.80500000000000005}]},{"osd":4,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.76900000000000002,"15min":0.66400000000000003},"min":{"1min":0.58599999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.82099999999999995,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.76000000000000001,"15min":0.66900000000000004},"min":{"1min":0.49099999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.90700000000000003,"5min":5.1749999999999998,"15min":5.1749999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79800000000000004,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.657,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.81100000000000005,"15min":0.76100000000000001},"min":{"1min":0.52100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.2519999999999998,"5min":2.46,"15min":2.46},"last":0.86399999999999999}]},{"osd":6,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":0.745},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.68100000000000005}]},{"osd":7,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.90500000000000003,"15min":0.91200000000000003},"min":{"1min":0.65400000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.879,"15min":0.88100000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":2.27,"5min":2.27,"15min":2.27},"last":0.78900000000000003}]},{"osd":8,"last update":"Sun Jul 3 10:34:06 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.873}]},{"osd":9,"last update":"Sun Jul 3 10:36:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.91400000000000003,"15min":0.96299999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.131,"5min":1.256,"15min":1.256},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.95499999999999996,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.61199999999999999}]},{"osd":10,"last update":"Sun Jul 3 10:35:56 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.93799999999999994,"15min":0.94299999999999995},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.71499999999999997}]},{"osd":11,"last update":"Sun Jul 3 10:36:05 2022","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.878}]}]},{"osd":4,"up_from":30,"seq":128849018945,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6312,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738840,"statfs":{"total":95995035648,"available":95988572160,"internally_reserved":0,"allocated":884736,"data_stored":436571,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:36:27 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.59899999999999998}]},{"osd":1,"last update":"Sun Jul 3 10:34:35 2022","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.35799999999999998},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.59099999999999997,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.39000000000000001,"15min":0.377},"min":{"1min":0.29599999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58899999999999997,"15min":0.55900000000000005},"min":{"1min":0.44800000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58299999999999996,"15min":0.56299999999999994},"min":{"1min":0.46899999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.70899999999999996,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Sun Jul 3 10:36:41 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60699999999999998,"15min":0.56499999999999995},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.73199999999999998,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.75},{"interface":"front","average":{"1min":0.628,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.48199999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.76900000000000002,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.72899999999999998}]},{"osd":5,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59399999999999997,"15min":0.57499999999999996},"min":{"1min":0.46400000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79000000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56399999999999995},"min":{"1min":0.42699999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.76000000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.84399999999999997}]},{"osd":6,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.53300000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.67000000000000004,"15min":0.65300000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.76700000000000002,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Jul 3 10:35:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77600000000000002,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.77600000000000002}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.64700000000000002},"min":{"1min":0.52300000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75900000000000001,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.66800000000000004,"15min":0.64500000000000002},"min":{"1min":0.53900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86099999999999999,"5min":1,"15min":1},"last":0.86899999999999999}]},{"osd":9,"last update":"Sun Jul 3 10:36:37 2022","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.77500000000000002,"15min":0.79100000000000004},"min":{"1min":0.56200000000000006,"5min":0.497,"15min":0.497},"max":{"1min":0.89100000000000001,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.96499999999999997,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.97299999999999998}]},{"osd":10,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.72999999999999998,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.92100000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.755,"15min":0.76200000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.91300000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90400000000000003}]},{"osd":11,"last update":"Sun Jul 3 10:36:03 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.98299999999999998}]}]},{"osd":5,"up_from":35,"seq":150323855420,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6500,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738652,"statfs":{"total":95995035648,"available":95988379648,"internally_reserved":0,"allocated":946176,"data_stored":1027222,"data_compressed":41569,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.64000000000000001,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.1820000000000004,"5min":4.1820000000000004,"15min":4.1820000000000004},"last":0.77700000000000002}]},{"osd":1,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.377,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.40699999999999997,"5min":0.315,"15min":0.315},"max":{"1min":0.78600000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63100000000000001}]},{"osd":2,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.32900000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.433,"15min":0.432},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56499999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.69799999999999995,"15min":0.627},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":7.21,"5min":7.21,"15min":7.21},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.79800000000000004,"5min":1.02,"15min":1.02},"last":0.64500000000000002}]},{"osd":4,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.59999999999999998,"15min":0.59399999999999997},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59799999999999998}]},{"osd":6,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73399999999999999,"15min":0.69599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86699999999999999,"5min":5.476,"15min":5.476},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.755,"15min":0.72299999999999998},"min":{"1min":0.58599999999999997,"5min":0.505,"15min":0.505},"max":{"1min":0.91000000000000003,"5min":3.27,"15min":3.27},"last":0.57899999999999996}]},{"osd":7,"last update":"Sun Jul 3 10:35:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.621,"15min":0.61499999999999999},"min":{"1min":0.46600000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.86399999999999999,"5min":0.875,"15min":0.875},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66200000000000003,"15min":0.64600000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56200000000000006}]},{"osd":8,"last update":"Sun Jul 3 10:36:07 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.50700000000000001}]},{"osd":9,"last update":"Sun Jul 3 10:36:35 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.77200000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95999999999999996,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.82199999999999995,"15min":0.78600000000000003},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.90800000000000003,"5min":5.4240000000000004,"15min":5.4240000000000004},"last":0.75800000000000001}]},{"osd":10,"last update":"Sun Jul 3 10:35:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.47799999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.73699999999999999,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.66900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34144,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35784,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":54238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-07-03T10:36:49.582 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:49 smithi161 ceph-mon[91579]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-07-03T10:36:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:49 smithi161 ceph-mon[91579]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:49 smithi188 ceph-mon[92425]: pgmap v323: 97 pgs: 97 active+clean; 630 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2022-07-03T10:36:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:49 smithi188 ceph-mon[92425]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:49.839 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1266336930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fd5d0 msgr2=0x7f61a80fd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.840 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1266336930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fd5d0 0x7f61a80fd9e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6198009900 tx=0x7f61980091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.840 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1266336930 shutdown_connections 2022-07-03T10:36:49.840 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1266336930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fd5d0 0x7f61a80fd9e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.841 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1266336930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80ff1e0 0x7f61a8105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.841 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1266336930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80fe850 0x7f61a80feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.841 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1266336930 >> 172.21.15.101:0/1266336930 conn(0x7f61a80f8d20 msgr2=0x7f61a80fb190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:49.842 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1266336930 shutdown_connections 2022-07-03T10:36:49.842 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.837+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1266336930 wait complete. 2022-07-03T10:36:49.843 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.838+0000 7f61ad5ff700 1 Processor -- start 2022-07-03T10:36:49.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.838+0000 7f61ad5ff700 1 -- start start 2022-07-03T10:36:49.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.849 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 0x7f61a8070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 0x7f61a8070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.850 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8071480 con 0x7f61a80fe850 2022-07-03T10:36:49.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80715f0 con 0x7f61a80ff1e0 2022-07-03T10:36:49.851 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61ad5ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8118690 con 0x7f61a80fd5d0 2022-07-03T10:36:49.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.852 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f619ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 0x7f61a8070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52758/0 (socket says 172.21.15.101:52758) 2022-07-03T10:36:49.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 -- 172.21.15.101:0/1478458097 learned_addr learned my addr 172.21.15.101:0/1478458097 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:49.853 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a77fe700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 0x7f61a8070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 -- 172.21.15.101:0/1478458097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 msgr2=0x7f61a8070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.854 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 0x7f61a8070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 -- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 msgr2=0x7f61a8070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:49.855 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.839+0000 7f61a6ffd700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 0x7f61a8070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:49.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f61a6ffd700 1 -- 172.21.15.101:0/1478458097 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198009440 con 0x7f61a80fd5d0 2022-07-03T10:36:49.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f61a77fe700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 0x7f61a8070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:49.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f619ffff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 0x7f61a8070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:49.856 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f61a6ffd700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6198006010 tx=0x7f61980091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:49.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619801d070 con 0x7f61a80fd5d0 2022-07-03T10:36:49.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a8118890 con 0x7f61a80fd5d0 2022-07-03T10:36:49.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f619800b9c0 con 0x7f61a80fd5d0 2022-07-03T10:36:49.857 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198022be0 con 0x7f61a80fd5d0 2022-07-03T10:36:49.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.840+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a8118d30 con 0x7f61a80fd5d0 2022-07-03T10:36:49.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.841+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a8108d80 con 0x7f61a80fd5d0 2022-07-03T10:36:49.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.843+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f6198029030 con 0x7f61a80fd5d0 2022-07-03T10:36:49.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.844+0000 7f619f7fe700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 0x7f61940991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:49.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.845+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f61980c1670 con 0x7f61a80fd5d0 2022-07-03T10:36:49.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.845+0000 7f619ffff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 0x7f61940991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:49.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.845+0000 7f619ffff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 0x7f61940991d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6190011560 tx=0x7f6190005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:49.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:49.847+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f619808e8b0 con 0x7f61a80fd5d0 2022-07-03T10:36:50.038 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.036+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61a81047c0 con 0x7f61a80fd5d0 2022-07-03T10:36:50.039 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.039+0000 7f619f7fe700 1 -- 172.21.15.101:0/1478458097 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7f6198027090 con 0x7f61a80fd5d0 2022-07-03T10:36:50.040 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:50.040 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:50.042 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 msgr2=0x7f61940991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 0x7f61940991d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6190011560 tx=0x7f6190005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 msgr2=0x7f61a8072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6198006010 tx=0x7f61980091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 shutdown_connections 2022-07-03T10:36:50.043 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6194096d20 0x7f61940991d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61a80ff1e0 0x7f61a8070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f61a80fd5d0 0x7f61a8072230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 --2- 172.21.15.101:0/1478458097 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f61a80fe850 0x7f61a8070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.044 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 >> 172.21.15.101:0/1478458097 conn(0x7f61a80f8d20 msgr2=0x7f61a8101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:50.045 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 shutdown_connections 2022-07-03T10:36:50.045 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.040+0000 7f61ad5ff700 1 -- 172.21.15.101:0/1478458097 wait complete. 2022-07-03T10:36:50.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-07-03T10:30:42.660749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-03T10:35:01.420999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-07-03T10:35:02.839114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-03T10:36:50.070 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-07-03T10:36:50.299 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 -- 172.21.15.101:0/325721042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 msgr2=0x7f659c0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 --2- 172.21.15.101:0/325721042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c0fd9f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f658c009900 tx=0x7f658c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 -- 172.21.15.101:0/325721042 shutdown_connections 2022-07-03T10:36:50.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 --2- 172.21.15.101:0/325721042 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0ff1f0 0x7f659c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 --2- 172.21.15.101:0/325721042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c0fd9f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.300 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 --2- 172.21.15.101:0/325721042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0fe860 0x7f659c0fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 -- 172.21.15.101:0/325721042 >> 172.21.15.101:0/325721042 conn(0x7f659c0f8d50 msgr2=0x7f659c0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:50.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 -- 172.21.15.101:0/325721042 shutdown_connections 2022-07-03T10:36:50.301 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.297+0000 7f65a1bff700 1 -- 172.21.15.101:0/325721042 wait complete. 2022-07-03T10:36:50.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.298+0000 7f65a1bff700 1 Processor -- start 2022-07-03T10:36:50.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.298+0000 7f65a1bff700 1 -- start start 2022-07-03T10:36:50.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0fe860 0x7f659c110d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.304 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0ff1f0 0x7f659c117de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c118360 con 0x7f659c0fe860 2022-07-03T10:36:50.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1184a0 con 0x7f659c0ff1f0 2022-07-03T10:36:50.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f65a1bff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c118660 con 0x7f659c0fd5e0 2022-07-03T10:36:50.305 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f659affd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0fe860 0x7f659c110d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52784/0 (socket says 172.21.15.101:52784) 2022-07-03T10:36:50.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f659b7fe700 1 -- 172.21.15.101:0/2379595147 learned_addr learned my addr 172.21.15.101:0/2379595147 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:50.306 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.299+0000 7f659bfff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0ff1f0 0x7f659c117de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 -- 172.21.15.101:0/2379595147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0ff1f0 msgr2=0x7f659c117de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0ff1f0 0x7f659c117de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 -- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0fe860 msgr2=0x7f659c110d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0fe860 0x7f659c110d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.307 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 -- 172.21.15.101:0/2379595147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c009440 con 0x7f659c0fd5e0 2022-07-03T10:36:50.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.300+0000 7f659b7fe700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f658c009f70 tx=0x7f658c0091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:50.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.301+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c01d070 con 0x7f659c0fd5e0 2022-07-03T10:36:50.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.301+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c1188e0 con 0x7f659c0fd5e0 2022-07-03T10:36:50.308 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.301+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f658c00b9c0 con 0x7f659c0fd5e0 2022-07-03T10:36:50.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.301+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c022b90 con 0x7f659c0fd5e0 2022-07-03T10:36:50.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.301+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c118de0 con 0x7f659c0fd5e0 2022-07-03T10:36:50.309 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.302+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c108df0 con 0x7f659c0fd5e0 2022-07-03T10:36:50.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.304+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f658c029030 con 0x7f659c0fd5e0 2022-07-03T10:36:50.310 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.306+0000 7f6583fff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 0x7f6584099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.306+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f658c0c0960 con 0x7f659c0fd5e0 2022-07-03T10:36:50.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.306+0000 7f659affd700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 0x7f6584099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.307+0000 7f659affd700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 0x7f6584099250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6590009c70 tx=0x7f6590005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:50.311 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.310+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f658c05a100 con 0x7f659c0fd5e0 2022-07-03T10:36:50.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:50 smithi101 ceph-mon[85126]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:50 smithi101 ceph-mon[85126]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:50 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1478458097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:50.496 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.494+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f659c119480 con 0x7f659c0fd5e0 2022-07-03T10:36:50.497 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.495+0000 7f6583fff700 1 -- 172.21.15.101:0/2379595147 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f658c08dba0 con 0x7f659c0fd5e0 2022-07-03T10:36:50.497 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 1 2022-07-03T10:36:50.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.497+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 msgr2=0x7f6584099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.497+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 0x7f6584099250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6590009c70 tx=0x7f6590005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.501 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.497+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 msgr2=0x7f659c110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.497+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f658c009f70 tx=0x7f658c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.502 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 shutdown_connections 2022-07-03T10:36:50.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6584096da0 0x7f6584099250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f659c0ff1f0 0x7f659c117de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.503 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f659c0fd5e0 0x7f659c110830 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 --2- 172.21.15.101:0/2379595147 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f659c0fe860 0x7f659c110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.504 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 >> 172.21.15.101:0/2379595147 conn(0x7f659c0f8d50 msgr2=0x7f659c070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:50.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 shutdown_connections 2022-07-03T10:36:50.505 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.498+0000 7f65a1bff700 1 -- 172.21.15.101:0/2379595147 wait complete. 2022-07-03T10:36:50.527 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-03T10:36:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:50 smithi161 ceph-mon[91579]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:50 smithi161 ceph-mon[91579]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:50 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1478458097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:50 smithi188 ceph-mon[92425]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:50 smithi188 ceph-mon[92425]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-07-03T10:36:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:50 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1478458097' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:50.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 -- 172.21.15.101:0/2631913175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fd5e0 msgr2=0x7f07ec0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.760 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 --2- 172.21.15.101:0/2631913175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec0fd9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009900 tx=0x7f07e80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.761 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 -- 172.21.15.101:0/2631913175 shutdown_connections 2022-07-03T10:36:50.761 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 --2- 172.21.15.101:0/2631913175 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0ff1f0 0x7f07ec105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.762 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 --2- 172.21.15.101:0/2631913175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec0fd9f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.762 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 --2- 172.21.15.101:0/2631913175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0fe860 0x7f07ec0fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.757+0000 7f07f33ff700 1 -- 172.21.15.101:0/2631913175 >> 172.21.15.101:0/2631913175 conn(0x7f07ec0f8d50 msgr2=0x7f07ec0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:50.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.758+0000 7f07f33ff700 1 -- 172.21.15.101:0/2631913175 shutdown_connections 2022-07-03T10:36:50.763 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.758+0000 7f07f33ff700 1 -- 172.21.15.101:0/2631913175 wait complete. 2022-07-03T10:36:50.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.758+0000 7f07f33ff700 1 Processor -- start 2022-07-03T10:36:50.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 -- start start 2022-07-03T10:36:50.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fe860 0x7f07ec110d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0ff1f0 0x7f07ec117de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec118360 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec1184a0 con 0x7f07ec0ff1f0 2022-07-03T10:36:50.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f33ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec118660 con 0x7f07ec0fe860 2022-07-03T10:36:50.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f119b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f119b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47026/0 (socket says 172.21.15.101:47026) 2022-07-03T10:36:50.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f119b700 1 -- 172.21.15.101:0/683784370 learned_addr learned my addr 172.21.15.101:0/683784370 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:50.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f099a700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fe860 0x7f07ec110d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.759+0000 7f07f199c700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0ff1f0 0x7f07ec117de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 -- 172.21.15.101:0/683784370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fe860 msgr2=0x7f07ec110d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fe860 0x7f07ec110d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 -- 172.21.15.101:0/683784370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0ff1f0 msgr2=0x7f07ec117de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0ff1f0 0x7f07ec117de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 -- 172.21.15.101:0/683784370 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8009440 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f119b700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009f70 tx=0x7f07e80091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:50.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e801d070 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec1188e0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f07e800b9c0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.760+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8022b90 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.761+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec118d80 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.762+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec108df0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.764+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f07e8029030 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.765+0000 7f07e1ffb700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 0x7f07d80991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:50.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.765+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f07e80c09c0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.765+0000 7f07f099a700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 0x7f07d80991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:50.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.766+0000 7f07f099a700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 0x7f07d80991d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009d00 tx=0x7f07dc005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:50.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.768+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7f07e805a100 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.954+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07ec1193c0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.955+0000 7f07e1ffb700 1 -- 172.21.15.101:0/683784370 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07e80270a0 con 0x7f07ec0fd5e0 2022-07-03T10:36:50.957 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 32 2022-07-03T10:36:50.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 msgr2=0x7f07d80991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 0x7f07d80991d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009d00 tx=0x7f07dc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 msgr2=0x7f07ec110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:50.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009f70 tx=0x7f07e80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 shutdown_connections 2022-07-03T10:36:50.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f07d8096d20 0x7f07d80991d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f07ec0fd5e0 0x7f07ec110830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07ec0ff1f0 0x7f07ec117de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 --2- 172.21.15.101:0/683784370 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07ec0fe860 0x7f07ec110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:50.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.957+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 >> 172.21.15.101:0/683784370 conn(0x7f07ec0f8d50 msgr2=0x7f07ec070750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:50.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.958+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 shutdown_connections 2022-07-03T10:36:50.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:50.958+0000 7f07f33ff700 1 -- 172.21.15.101:0/683784370 wait complete. 2022-07-03T10:36:50.987 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-03T10:36:51.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/3410967084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40fd5e0 msgr2=0x7fc4e40fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/3410967084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e40fd9f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009900 tx=0x7fc4d40091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/3410967084 shutdown_connections 2022-07-03T10:36:51.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/3410967084 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40ff1f0 0x7fc4e4105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/3410967084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e40fd9f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/3410967084 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e40fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.215+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/3410967084 >> 172.21.15.101:0/3410967084 conn(0x7fc4e40f8d50 msgr2=0x7fc4e40fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:51.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.216+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/3410967084 shutdown_connections 2022-07-03T10:36:51.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.216+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/3410967084 wait complete. 2022-07-03T10:36:51.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.216+0000 7fc4ea5ff700 1 Processor -- start 2022-07-03T10:36:51.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 -- start start 2022-07-03T10:36:51.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e4072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40ff1f0 0x7fc4e4070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4071480 con 0x7fc4e40fd5e0 2022-07-03T10:36:51.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e40715f0 con 0x7fc4e40fe860 2022-07-03T10:36:51.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4ea5ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4118740 con 0x7fc4e40ff1f0 2022-07-03T10:36:51.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4e3fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4e37fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.217+0000 7fc4e37fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51718/0 (socket says 172.21.15.101:51718) 2022-07-03T10:36:51.225 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 -- 172.21.15.101:0/1392043664 learned_addr learned my addr 172.21.15.101:0/1392043664 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:51.225 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e8b9c700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40ff1f0 0x7fc4e4070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.225 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 -- 172.21.15.101:0/1392043664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40ff1f0 msgr2=0x7fc4e4070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.226 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40ff1f0 0x7fc4e4070db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.226 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 -- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40fd5e0 msgr2=0x7fc4e4072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.227 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e4072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.227 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 -- 172.21.15.101:0/1392043664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4009440 con 0x7fc4e40fe860 2022-07-03T10:36:51.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.218+0000 7fc4e37fe700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800ea80 tx=0x7fc4d8005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:51.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.219+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800c7c0 con 0x7fc4e40fe860 2022-07-03T10:36:51.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.219+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc4d800cdd0 con 0x7fc4e40fe860 2022-07-03T10:36:51.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.219+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e41189a0 con 0x7fc4e40fe860 2022-07-03T10:36:51.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.219+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8010df0 con 0x7fc4e40fe860 2022-07-03T10:36:51.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.219+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e4118f30 con 0x7fc4e40fe860 2022-07-03T10:36:51.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.220+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e4108dd0 con 0x7fc4e40fe860 2022-07-03T10:36:51.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.222+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fc4d800c040 con 0x7fc4e40fe860 2022-07-03T10:36:51.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.223+0000 7fc4e0ff9700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 0x7fc4cc099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.223+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fc4d8014070 con 0x7fc4e40fe860 2022-07-03T10:36:51.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.223+0000 7fc4e3fff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 0x7fc4cc099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.224+0000 7fc4e3fff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 0x7fc4cc099250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40097d0 tx=0x7fc4d40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:51.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.225+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fc4d808cff0 con 0x7fc4e40fe860 2022-07-03T10:36:51.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.417+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4e4108f60 con 0x7fc4e40fe860 2022-07-03T10:36:51.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.417+0000 7fc4e0ff9700 1 -- 172.21.15.101:0/1392043664 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4e4108f60 con 0x7fc4e40fe860 2022-07-03T10:36:51.420 INFO:teuthology.orchestra.run.smithi101.stdout:pg_num: 64 2022-07-03T10:36:51.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 msgr2=0x7fc4cc099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 0x7fc4cc099250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40097d0 tx=0x7fc4d40091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 msgr2=0x7fc4e4070870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800ea80 tx=0x7fc4d8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 shutdown_connections 2022-07-03T10:36:51.423 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fc4cc096da0 0x7fc4cc099250 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc4e40fd5e0 0x7fc4e4072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc4e40ff1f0 0x7fc4e4070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 --2- 172.21.15.101:0/1392043664 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc4e40fe860 0x7fc4e4070870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.424 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 >> 172.21.15.101:0/1392043664 conn(0x7fc4e40f8d50 msgr2=0x7fc4e4101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:51.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 shutdown_connections 2022-07-03T10:36:51.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.420+0000 7fc4ea5ff700 1 -- 172.21.15.101:0/1392043664 wait complete. 2022-07-03T10:36:51.449 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-03T10:36:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:51 smithi161 ceph-mon[91579]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:51 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2379595147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:51 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/683784370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:51 smithi188 ceph-mon[92425]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:51 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2379595147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:51 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/683784370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.676 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.673+0000 7fb57c9fe700 1 -- 172.21.15.101:0/3298952959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 msgr2=0x7fb5781058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.673+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/3298952959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb5781058b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb570009bf0 tx=0x7fb570009460 comp rx=0 tx=0).stop 2022-07-03T10:36:51.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 -- 172.21.15.101:0/3298952959 shutdown_connections 2022-07-03T10:36:51.677 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/3298952959 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb5780fd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/3298952959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb5781058b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/3298952959 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb5780feab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 -- 172.21.15.101:0/3298952959 >> 172.21.15.101:0/3298952959 conn(0x7fb5780f8b30 msgr2=0x7fb5780fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:51.678 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 -- 172.21.15.101:0/3298952959 shutdown_connections 2022-07-03T10:36:51.679 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.674+0000 7fb57c9fe700 1 -- 172.21.15.101:0/3298952959 wait complete. 2022-07-03T10:36:51.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.675+0000 7fb57c9fe700 1 Processor -- start 2022-07-03T10:36:51.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.675+0000 7fb57c9fe700 1 -- start start 2022-07-03T10:36:51.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb578070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb578070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578071480 con 0x7fb5780fd3e0 2022-07-03T10:36:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780715f0 con 0x7fb5780fe660 2022-07-03T10:36:51.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57c9fe700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5781182a0 con 0x7fb5780feff0 2022-07-03T10:36:51.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb576d9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57759e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb578070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb576d9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47038/0 (socket says 172.21.15.101:47038) 2022-07-03T10:36:51.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb576d9d700 1 -- 172.21.15.101:0/2485440471 learned_addr learned my addr 172.21.15.101:0/2485440471 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:51.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.676+0000 7fb57659c700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb578070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 -- 172.21.15.101:0/2485440471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 msgr2=0x7fb578070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb578070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 -- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 msgr2=0x7fb578072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 -- 172.21.15.101:0/2485440471 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb570009700 con 0x7fb5780feff0 2022-07-03T10:36:51.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57659c700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb578070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:51.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb576d9d700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:51.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57759e700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb578070db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5700096d0 tx=0x7fb570009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:51.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57001d070 con 0x7fb5780feff0 2022-07-03T10:36:51.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb570005b60 con 0x7fb5780feff0 2022-07-03T10:36:51.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57000fc10 con 0x7fb5780feff0 2022-07-03T10:36:51.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.677+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5781184a0 con 0x7fb5780feff0 2022-07-03T10:36:51.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.678+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5781188f0 con 0x7fb5780feff0 2022-07-03T10:36:51.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.679+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb578108b90 con 0x7fb5780feff0 2022-07-03T10:36:51.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.681+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fb570029030 con 0x7fb5780feff0 2022-07-03T10:36:51.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.682+0000 7fb55f7fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 0x7fb5600991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:51.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.682+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fb5700c15a0 con 0x7fb5780feff0 2022-07-03T10:36:51.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.682+0000 7fb576d9d700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 0x7fb5600991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:51.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.683+0000 7fb576d9d700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 0x7fb5600991d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb568011680 tx=0x7fb5680091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:51.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.684+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fb57008e840 con 0x7fb5780feff0 2022-07-03T10:36:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:51 smithi101 ceph-mon[85126]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:51.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:51 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2379595147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:51 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/683784370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-07-03T10:36:51.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.914+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb578108d20 con 0x7fb5780feff0 2022-07-03T10:36:51.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.915+0000 7fb55f7fe700 1 -- 172.21.15.101:0/2485440471 <== mon.2 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+10692 (secure 0 0 0) 0x7fb578108d20 con 0x7fb5780feff0 2022-07-03T10:36:51.917 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:51.918 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/369400191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":369400191},{"type":"v1","addr":"172.21.15.101:6835","nonce":369400191}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24482,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/3965232315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":3965232315},{"type":"v1","addr":"172.21.15.101:6839","nonce":3965232315}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/3291034276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":3291034276},{"type":"v1","addr":"172.21.15.101:6837","nonce":3291034276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24500,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6841/2161782899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6840","nonce":2161782899},{"type":"v1","addr":"172.21.15.101:6841","nonce":2161782899}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24517,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/2910285108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2910285108},{"type":"v1","addr":"172.21.15.161:6833","nonce":2910285108}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/429921526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":429921526},{"type":"v1","addr":"172.21.15.161:6835","nonce":429921526}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6833/1718258929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6832","nonce":1718258929},{"type":"v1","addr":"172.21.15.188:6833","nonce":1718258929}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24541,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6837/1658187203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6836","nonce":1658187203},{"type":"v1","addr":"172.21.15.161:6837","nonce":1658187203}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24545,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6835/1026299276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6834","nonce":1026299276},{"type":"v1","addr":"172.21.15.188:6835","nonce":1026299276}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24553,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6839/2119471810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6838","nonce":2119471810},{"type":"v1","addr":"172.21.15.161:6839","nonce":2119471810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-07-03T10:35:03.427264+0000","modified":"2022-07-03T10:35:08.208608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24557":{"gid":24557,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6837/639939509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6836","nonce":639939509},{"type":"v1","addr":"172.21.15.188:6837","nonce":639939509}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.188:6839/204174115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6838","nonce":204174115},{"type":"v1","addr":"172.21.15.188:6839","nonce":204174115}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-03T10:36:51.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 msgr2=0x7fb5600991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 0x7fb5600991d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb568011680 tx=0x7fb5680091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 msgr2=0x7fb578070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:51.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb578070db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5700096d0 tx=0x7fb570009460 comp rx=0 tx=0).stop 2022-07-03T10:36:51.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 shutdown_connections 2022-07-03T10:36:51.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fb560096d20 0x7fb5600991d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb5780fd3e0 0x7fb578072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb5780feff0 0x7fb578070db0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 --2- 172.21.15.101:0/2485440471 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5780fe660 0x7fb578070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:51.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 >> 172.21.15.101:0/2485440471 conn(0x7fb5780f8b30 msgr2=0x7fb5780fad50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:51.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 shutdown_connections 2022-07-03T10:36:51.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:51.918+0000 7fb57c9fe700 1 -- 172.21.15.101:0/2485440471 wait complete. 2022-07-03T10:36:51.923 INFO:teuthology.orchestra.run.smithi101.stderr:dumped fsmap epoch 20 2022-07-03T10:36:51.947 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-03T10:36:52.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.171+0000 7fa88b19d700 1 -- 172.21.15.101:0/863391767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 msgr2=0x7fa8840fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.171+0000 7fa88b19d700 1 --2- 172.21.15.101:0/863391767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa8840fd7e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa880009900 tx=0x7fa8800091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 -- 172.21.15.101:0/863391767 shutdown_connections 2022-07-03T10:36:52.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 --2- 172.21.15.101:0/863391767 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fefe0 0x7fa8841058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 --2- 172.21.15.101:0/863391767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa8840fd7e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 --2- 172.21.15.101:0/863391767 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fe650 0x7fa8840feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 -- 172.21.15.101:0/863391767 >> 172.21.15.101:0/863391767 conn(0x7fa8840f8b20 msgr2=0x7fa8840faf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 -- 172.21.15.101:0/863391767 shutdown_connections 2022-07-03T10:36:52.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.172+0000 7fa88b19d700 1 -- 172.21.15.101:0/863391767 wait complete. 2022-07-03T10:36:52.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.173+0000 7fa88b19d700 1 Processor -- start 2022-07-03T10:36:52.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.173+0000 7fa88b19d700 1 -- start start 2022-07-03T10:36:52.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa88410e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 0x7fa884115940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa884115ec0 con 0x7fa8840fefe0 2022-07-03T10:36:52.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa884116000 con 0x7fa8840fd3d0 2022-07-03T10:36:52.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88b19d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8841161c0 con 0x7fa8840fe650 2022-07-03T10:36:52.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88a99c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 0x7fa884115940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88999a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88a19b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa88410e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88999a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52846/0 (socket says 172.21.15.101:52846) 2022-07-03T10:36:52.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.174+0000 7fa88999a700 1 -- 172.21.15.101:0/3711970233 learned_addr learned my addr 172.21.15.101:0/3711970233 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 -- 172.21.15.101:0/3711970233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 msgr2=0x7fa88410e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 -- 172.21.15.101:0/3711970233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 msgr2=0x7fa88410e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa88410e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 -- 172.21.15.101:0/3711970233 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874009410 con 0x7fa8840fefe0 2022-07-03T10:36:52.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88999a700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:52.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88a99c700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 0x7fa884115940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00eab0 tx=0x7fa87c009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c009850 con 0x7fa8840fefe0 2022-07-03T10:36:52.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa87c00c9d0 con 0x7fa8840fefe0 2022-07-03T10:36:52.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa880009440 con 0x7fa8840fefe0 2022-07-03T10:36:52.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c004020 con 0x7fa8840fefe0 2022-07-03T10:36:52.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.175+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa884116850 con 0x7fa8840fefe0 2022-07-03T10:36:52.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.176+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa884108b90 con 0x7fa8840fefe0 2022-07-03T10:36:52.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.178+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa87c01e020 con 0x7fa8840fefe0 2022-07-03T10:36:52.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.179+0000 7fa87affd700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 0x7fa8700991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.180+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa87c014070 con 0x7fa8840fefe0 2022-07-03T10:36:52.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.180+0000 7fa88a19b700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 0x7fa8700991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.180+0000 7fa88a19b700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 0x7fa8700991d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097d0 tx=0x7fa8800091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.184+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168162 (secure 0 0 0) 0x7fa87c08d1d0 con 0x7fa8840fefe0 2022-07-03T10:36:52.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.377+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa884104350 con 0x7fa8840fefe0 2022-07-03T10:36:52.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.378+0000 7fa87affd700 1 -- 172.21.15.101:0/3711970233 <== mon.0 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21844 (secure 0 0 0) 0x7fa87c00cb40 con 0x7fa8840fefe0 2022-07-03T10:36:52.380 INFO:teuthology.orchestra.run.smithi101.stdout: 2022-07-03T10:36:52.381 INFO:teuthology.orchestra.run.smithi101.stdout:{"epoch":82,"fsid":"23a851ae-fab9-11ec-842c-001a4aab830c","created":"2022-07-03T10:27:26.718252+0000","modified":"2022-07-03T10:35:27.299095+0000","last_up_change":"2022-07-03T10:34:14.235440+0000","last_in_change":"2022-07-03T10:33:56.896833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-07-03T10:30:42.660749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-07-03T10:35:01.420999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-07-03T10:35:02.839114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b6b433d6-6618-4afa-9a41-a4dbde9dfd3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6803","nonce":2068504891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6805","nonce":2068504891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6809","nonce":2068504891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2068504891},{"type":"v1","addr":"172.21.15.101:6807","nonce":2068504891}]},"public_addr":"172.21.15.101:6803/2068504891","cluster_addr":"172.21.15.101:6805/2068504891","heartbeat_back_addr":"172.21.15.101:6809/2068504891","heartbeat_front_addr":"172.21.15.101:6807/2068504891","state":["exists","up"]},{"osd":1,"uuid":"59dc2862-3622-479a-a620-87a0f8de138b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6801","nonce":967237995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6803","nonce":967237995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6807","nonce":967237995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":967237995},{"type":"v1","addr":"172.21.15.161:6805","nonce":967237995}]},"public_addr":"172.21.15.161:6801/967237995","cluster_addr":"172.21.15.161:6803/967237995","heartbeat_back_addr":"172.21.15.161:6807/967237995","heartbeat_front_addr":"172.21.15.161:6805/967237995","state":["exists","up"]},{"osd":2,"uuid":"acec9c8a-8e42-45f1-83a0-f598007cc2e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6801","nonce":2999852966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6803","nonce":2999852966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6807","nonce":2999852966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2999852966},{"type":"v1","addr":"172.21.15.188:6805","nonce":2999852966}]},"public_addr":"172.21.15.188:6801/2999852966","cluster_addr":"172.21.15.188:6803/2999852966","heartbeat_back_addr":"172.21.15.188:6807/2999852966","heartbeat_front_addr":"172.21.15.188:6805/2999852966","state":["exists","up"]},{"osd":3,"uuid":"14eb692b-ed90-438b-9b81-5b71756db9c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6811","nonce":3606989058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6813","nonce":3606989058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6817","nonce":3606989058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":3606989058},{"type":"v1","addr":"172.21.15.101:6815","nonce":3606989058}]},"public_addr":"172.21.15.101:6811/3606989058","cluster_addr":"172.21.15.101:6813/3606989058","heartbeat_back_addr":"172.21.15.101:6817/3606989058","heartbeat_front_addr":"172.21.15.101:6815/3606989058","state":["exists","up"]},{"osd":4,"uuid":"4de30afa-ff39-4be7-9258-990f21fe1fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6809","nonce":54950390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6810","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6811","nonce":54950390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6815","nonce":54950390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":54950390},{"type":"v1","addr":"172.21.15.161:6813","nonce":54950390}]},"public_addr":"172.21.15.161:6809/54950390","cluster_addr":"172.21.15.161:6811/54950390","heartbeat_back_addr":"172.21.15.161:6815/54950390","heartbeat_front_addr":"172.21.15.161:6813/54950390","state":["exists","up"]},{"osd":5,"uuid":"93593083-57f3-4a9c-a08d-c32067460386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6809","nonce":1424487492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6811","nonce":1424487492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6815","nonce":1424487492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1424487492},{"type":"v1","addr":"172.21.15.188:6813","nonce":1424487492}]},"public_addr":"172.21.15.188:6809/1424487492","cluster_addr":"172.21.15.188:6811/1424487492","heartbeat_back_addr":"172.21.15.188:6815/1424487492","heartbeat_front_addr":"172.21.15.188:6813/1424487492","state":["exists","up"]},{"osd":6,"uuid":"a3e65880-acae-4f11-9494-023c542b5e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6819","nonce":3209986330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6821","nonce":3209986330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6825","nonce":3209986330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":3209986330},{"type":"v1","addr":"172.21.15.101:6823","nonce":3209986330}]},"public_addr":"172.21.15.101:6819/3209986330","cluster_addr":"172.21.15.101:6821/3209986330","heartbeat_back_addr":"172.21.15.101:6825/3209986330","heartbeat_front_addr":"172.21.15.101:6823/3209986330","state":["exists","up"]},{"osd":7,"uuid":"ed7c95a0-e815-4121-8542-947140c403f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6816","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6817","nonce":328327819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6819","nonce":328327819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6822","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6823","nonce":328327819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":328327819},{"type":"v1","addr":"172.21.15.161:6821","nonce":328327819}]},"public_addr":"172.21.15.161:6817/328327819","cluster_addr":"172.21.15.161:6819/328327819","heartbeat_back_addr":"172.21.15.161:6823/328327819","heartbeat_front_addr":"172.21.15.161:6821/328327819","state":["exists","up"]},{"osd":8,"uuid":"2d274e0b-0fd9-4e1c-8044-6295c0eb3b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6817","nonce":1483136090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6819","nonce":1483136090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6823","nonce":1483136090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1483136090},{"type":"v1","addr":"172.21.15.188:6821","nonce":1483136090}]},"public_addr":"172.21.15.188:6817/1483136090","cluster_addr":"172.21.15.188:6819/1483136090","heartbeat_back_addr":"172.21.15.188:6823/1483136090","heartbeat_front_addr":"172.21.15.188:6821/1483136090","state":["exists","up"]},{"osd":9,"uuid":"241d0da9-abc5-4259-8e73-454c3a885ebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6827","nonce":1335772894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6829","nonce":1335772894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6833","nonce":1335772894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":1335772894},{"type":"v1","addr":"172.21.15.101:6831","nonce":1335772894}]},"public_addr":"172.21.15.101:6827/1335772894","cluster_addr":"172.21.15.101:6829/1335772894","heartbeat_back_addr":"172.21.15.101:6833/1335772894","heartbeat_front_addr":"172.21.15.101:6831/1335772894","state":["exists","up"]},{"osd":10,"uuid":"bee2129f-d332-4a19-b6c8-3e308c9e262a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6825","nonce":1352813899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6826","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6827","nonce":1352813899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6831","nonce":1352813899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":1352813899},{"type":"v1","addr":"172.21.15.161:6829","nonce":1352813899}]},"public_addr":"172.21.15.161:6825/1352813899","cluster_addr":"172.21.15.161:6827/1352813899","heartbeat_back_addr":"172.21.15.161:6831/1352813899","heartbeat_front_addr":"172.21.15.161:6829/1352813899","state":["exists","up"]},{"osd":11,"uuid":"818815ae-5ae2-4fa9-a4bd-237e7f420337","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6825","nonce":1726462750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6827","nonce":1726462750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6831","nonce":1726462750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":1726462750},{"type":"v1","addr":"172.21.15.188:6829","nonce":1726462750}]},"public_addr":"172.21.15.188:6825/1726462750","cluster_addr":"172.21.15.188:6827/1726462750","heartbeat_back_addr":"172.21.15.188:6831/1726462750","heartbeat_front_addr":"172.21.15.188:6829/1726462750","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:02.581990+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:24.414539+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:30:39.989813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:08.977755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:30.864173+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:31:51.959401+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:24.307995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:32:46.373626+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:01.981150+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:29.958367+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:33:51.935965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-03T10:34:12.645947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":6}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.101:0/3515793351":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6801/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:6800/2446540439":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2333710927":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2063980612":"2022-07-04T10:28:25.795153+0000","172.21.15.101:0/2182043394":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/2680816285":"2022-07-04T10:27:58.929183+0000","172.21.15.101:0/4072829405":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6800/1657735167":"2022-07-04T10:27:58.929183+0000","172.21.15.101:6801/1657735167":"2022-07-04T10:27:58.929183+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-07-03T10:36:52.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 msgr2=0x7fa8700991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 0x7fa8700991d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097d0 tx=0x7fa8800091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 msgr2=0x7fa884115940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 0x7fa884115940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00eab0 tx=0x7fa87c009460 comp rx=0 tx=0).stop 2022-07-03T10:36:52.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 shutdown_connections 2022-07-03T10:36:52.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa870096d20 0x7fa8700991d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa8840fefe0 0x7fa884115940 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa8840fd3d0 0x7fa88410e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 --2- 172.21.15.101:0/3711970233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8840fe650 0x7fa88410e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 >> 172.21.15.101:0/3711970233 conn(0x7fa8840f8b20 msgr2=0x7fa8840faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 shutdown_connections 2022-07-03T10:36:52.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.381+0000 7fa88b19d700 1 -- 172.21.15.101:0/3711970233 wait complete. 2022-07-03T10:36:52.410 DEBUG:tasks.fwd_scrub:Yielding 2022-07-03T10:36:52.411 INFO:teuthology.run_tasks:Running task check-counter... 2022-07-03T10:36:52.425 INFO:teuthology.run_tasks:Running task workunit... 2022-07-03T10:36:52.449 INFO:tasks.workunit:Pulling workunits from ref 61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:36:52.451 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-07-03T10:36:52.451 INFO:tasks.workunit:timeout=3h 2022-07-03T10:36:52.451 INFO:tasks.workunit:cleanup=True 2022-07-03T10:36:52.452 DEBUG:teuthology.orchestra.run.smithi101:> stat -- /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:52.453 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2022-07-03T10:36:52.453 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2022-07-03T10:36:52.454 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2022-07-03T10:36:52.517 INFO:teuthology.orchestra.run.smithi101.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:52.517 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-07-03T10:36:52.518 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2022-07-03T10:36:52.518 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-03T10:36:52.518 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:cephfs_t:s0 2022-07-03T10:36:52.518 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2022-07-03 10:35:09.735299692 +0000 2022-07-03T10:36:52.519 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2022-07-03 10:35:09.735299692 +0000 2022-07-03T10:36:52.519 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2022-07-03 10:36:43.925322395 +0000 2022-07-03T10:36:52.519 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: 2022-07-03 10:35:09.735299692 +0000 2022-07-03T10:36:52.519 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-07-03T10:36:52.520 DEBUG:teuthology.orchestra.run.smithi101:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-07-03T10:36:52.598 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:52 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/1392043664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:52.598 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:52 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/2485440471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:52.619 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61829bae536fb4126cfaa87ca353b9576e018e7d 2022-07-03T10:36:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:52 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/1392043664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:52 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/2485440471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:52 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/1392043664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-07-03T10:36:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:52 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/2485440471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-07-03T10:36:52.683 INFO:tasks.workunit.client.0.smithi101.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-07-03T10:36:52.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.698+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/423233139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 msgr2=0x7ff8dc0feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.698+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/423233139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc0feac0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009860 tx=0x7ff8cc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/423233139 shutdown_connections 2022-07-03T10:36:52.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/423233139 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc0fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/423233139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 0x7ff8dc1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/423233139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc0feac0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/423233139 >> 172.21.15.101:0/423233139 conn(0x7ff8dc0f8b40 msgr2=0x7ff8dc0fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/423233139 shutdown_connections 2022-07-03T10:36:52.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.699+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/423233139 wait complete. 2022-07-03T10:36:52.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 Processor -- start 2022-07-03T10:36:52.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 -- start start 2022-07-03T10:36:52.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 0x7ff8dc117b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1180c0 con 0x7ff8dc0fd3f0 2022-07-03T10:36:52.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc118200 con 0x7ff8dc0ff000 2022-07-03T10:36:52.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.700+0000 7ff8e2f9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1183c0 con 0x7ff8dc0fe670 2022-07-03T10:36:52.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc110590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47066/0 (socket says 172.21.15.101:47066) 2022-07-03T10:36:52.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/2246379013 learned_addr learned my addr 172.21.15.101:0/2246379013 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e279c700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 0x7ff8dc117b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 -- 172.21.15.101:0/2246379013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 msgr2=0x7ff8dc117b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 0x7ff8dc117b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 -- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 msgr2=0x7ff8dc110590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc110590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.701+0000 7ff8e179a700 1 -- 172.21.15.101:0/2246379013 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8009440 con 0x7ff8dc0fe670 2022-07-03T10:36:52.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc110ad0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0093e0 tx=0x7ff8cc005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8d27fc700 1 -- 172.21.15.101:0/2246379013 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc01d070 con 0x7ff8dc0fe670 2022-07-03T10:36:52.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8d27fc700 1 -- 172.21.15.101:0/2246379013 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff8cc005590 con 0x7ff8dc0fe670 2022-07-03T10:36:52.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc009410 con 0x7ff8dc0fe670 2022-07-03T10:36:52.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8d27fc700 1 -- 172.21.15.101:0/2246379013 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc017bf0 con 0x7ff8dc0fe670 2022-07-03T10:36:52.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.702+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc118950 con 0x7ff8dc0fe670 2022-07-03T10:36:52.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.706+0000 7ff8d27fc700 1 -- 172.21.15.101:0/2246379013 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7ff8cc028020 con 0x7ff8dc0fe670 2022-07-03T10:36:52.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.707+0000 7ff8d27fc700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 0x7ff8c80991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.707+0000 7ff8d27fc700 1 -- 172.21.15.101:0/2246379013 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff8cc0c0e30 con 0x7ff8dc0fe670 2022-07-03T10:36:52.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.708+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 0x7ff8c80991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.708+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 0x7ff8c80991d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80057f0 tx=0x7ff8d80091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.709+0000 7ff8f80e2b80 1 Processor -- start 2022-07-03T10:36:52.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.709+0000 7ff8f80e2b80 1 -- start start 2022-07-03T10:36:52.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5affcedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affcf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5afeba290 con 0x55d5affcf2f0 2022-07-03T10:36:52.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5aff23d80 con 0x55d5affce940 2022-07-03T10:36:52.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5aff1d580 con 0x55d5affcdff0 2022-07-03T10:36:52.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52870/0 (socket says 172.21.15.101:52870) 2022-07-03T10:36:52.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1140772192 learned_addr learned my addr 172.21.15.101:0/1140772192 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8e179a700 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5affcedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.710+0000 7ff8e279c700 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affcf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.724 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1140772192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 msgr2=0x55d5affcedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.724 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5affcedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.724 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1140772192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 msgr2=0x55d5affcf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affcf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1140772192 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5afda6430 con 0x55d5affcdff0 2022-07-03T10:36:52.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009780 tx=0x7ff8d8018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2615e4aa368c952 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8bf7fe700 1 -- 172.21.15.101:0/1140772192 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8026430 con 0x55d5affcdff0 2022-07-03T10:36:52.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8bf7fe700 1 -- 172.21.15.101:0/1140772192 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff8d8021070 con 0x55d5affcdff0 2022-07-03T10:36:52.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.711+0000 7ff8bf7fe700 1 -- 172.21.15.101:0/1140772192 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d801d550 con 0x55d5affcdff0 2022-07-03T10:36:52.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1140772192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 msgr2=0x55d5affce400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009780 tx=0x7ff8d8018000 comp rx=0 tx=0).stop 2022-07-03T10:36:52.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1140772192 shutdown_connections 2022-07-03T10:36:52.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affcf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affce400 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1140772192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5affcedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1140772192 >> 172.21.15.101:0/1140772192 conn(0x55d5affc3880 msgr2=0x55d5affc8830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1140772192 shutdown_connections 2022-07-03T10:36:52.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.712+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1140772192 wait complete. 2022-07-03T10:36:52.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.713+0000 7ff8f80e2b80 1 -- start start 2022-07-03T10:36:52.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 Processor -- start 2022-07-03T10:36:52.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affd7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 0x55d5affde9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5afe0bf10 con 0x55d5affce940 2022-07-03T10:36:52.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5affdef80 con 0x55d5affcf2f0 2022-07-03T10:36:52.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.714+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5affdf140 con 0x55d5affcdff0 2022-07-03T10:36:52.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8f80e2b80 10 client.0 _ll_register_callbacks cb 0x55d5afec9270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:52.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e179a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affd7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e179a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47082/0 (socket says 172.21.15.101:47082) 2022-07-03T10:36:52.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e179a700 1 -- 172.21.15.101:0/1695918015 learned_addr learned my addr 172.21.15.101:0/1695918015 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e279c700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 0x55d5affde9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1695918015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 msgr2=0x55d5affde9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 0x55d5affde9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1695918015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 msgr2=0x55d5affd7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/1695918015 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d8017070 con 0x55d5affcdff0 2022-07-03T10:36:52.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e179a700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:52.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.715+0000 7ff8e279c700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 0x55d5affde9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:52.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affd7450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8014040 tx=0x7ff8d801f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4ef1e39499f699 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:36:52.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8005040 con 0x55d5affcdff0 2022-07-03T10:36:52.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d5affc1040 con 0x55d5affcdff0 2022-07-03T10:36:52.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff8d80051a0 con 0x55d5affcdff0 2022-07-03T10:36:52.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8d801c410 con 0x55d5affcdff0 2022-07-03T10:36:52.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8f80e2b80 10 client.25181 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:52.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.716+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d5afd52be0 con 0x55d5affcdff0 2022-07-03T10:36:52.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff8d803d020 con 0x55d5affcdff0 2022-07-03T10:36:52.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7ff8d8004ac0 con 0x55d5affcdff0 2022-07-03T10:36:52.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 10 client.25181 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:52.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 10 client.25181 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:52.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 20 client.25181 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:52.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5affe3190 0x55d5affe55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 4 client.25181 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:52.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d5aff16af0 con 0x55d5affe3190 2022-07-03T10:36:52.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.717+0000 7ff8e179a700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5affe3190 0x55d5affe55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.718+0000 7ff8bcff9700 10 client.25181 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:52.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.718+0000 7ff8e179a700 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5affe3190 0x55d5affe55a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=294a4d88f5283241 server_cookie=3630904583ddff79 in_seq=0 out_seq=0 2022-07-03T10:36:52.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8bcff9700 1 -- 172.21.15.101:0/1695918015 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55d5aff16af0 con 0x55d5affe3190 2022-07-03T10:36:52.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8bcff9700 10 client.25181 handle_command_reply: tid=0 2022-07-03T10:36:52.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 client.25181 shutdown 2022-07-03T10:36:52.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 msgr2=0x55d5affd7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affd7450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8014040 tx=0x7ff8d801f040 comp rx=0 tx=0).stop 2022-07-03T10:36:52.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 shutdown_connections 2022-07-03T10:36:52.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5affe3190 0x55d5affe55a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcf2f0 0x55d5affde9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8bcff9700 0 client.25181 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:52.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcdff0 0x55d5affd7450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/1695918015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affce940 0x55d5affd7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.744 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.719+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 >> 172.21.15.101:0/1695918015 conn(0x55d5affc3880 msgr2=0x55d5affc8800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.720+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 shutdown_connections 2022-07-03T10:36:52.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.720+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/1695918015 wait complete. 2022-07-03T10:36:52.745 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.720+0000 7ff8f80e2b80 20 client.25181 trim_cache size 0 max 16384 2022-07-03T10:36:52.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 Processor -- start 2022-07-03T10:36:52.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 -- start start 2022-07-03T10:36:52.754 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affce940 0x55d5affd7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affd75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5affc9710 con 0x55d5affcf2f0 2022-07-03T10:36:52.755 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5affd67b0 con 0x55d5affcdff0 2022-07-03T10:36:52.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.752+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5affdedb0 con 0x55d5affce940 2022-07-03T10:36:52.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e279c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affd75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.756 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51818/0 (socket says 172.21.15.101:51818) 2022-07-03T10:36:52.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/3498735230 learned_addr learned my addr 172.21.15.101:0/3498735230 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.757 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e179a700 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affce940 0x55d5affd7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/3498735230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affce940 msgr2=0x55d5affd7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affce940 0x55d5affd7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.758 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/3498735230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 msgr2=0x55d5affd75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affd75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.759 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.753+0000 7ff8e1f9b700 1 -- 172.21.15.101:0/3498735230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5affc4a00 con 0x55d5affcdff0 2022-07-03T10:36:52.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.754+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff8d803e050 tx=0x7ff8d801f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8737587f684b2999 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.765 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.754+0000 7ff8bffff700 1 -- 172.21.15.101:0/3498735230 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8039480 con 0x55d5affcdff0 2022-07-03T10:36:52.766 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.754+0000 7ff8bffff700 1 -- 172.21.15.101:0/3498735230 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff8d803d030 con 0x55d5affcdff0 2022-07-03T10:36:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.754+0000 7ff8bffff700 1 -- 172.21.15.101:0/3498735230 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8004c30 con 0x55d5affcdff0 2022-07-03T10:36:52.767 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/3498735230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 msgr2=0x55d5affdf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff8d803e050 tx=0x7ff8d801f040 comp rx=0 tx=0).stop 2022-07-03T10:36:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/3498735230 shutdown_connections 2022-07-03T10:36:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcf2f0 0x55d5affd75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.768 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affcdff0 0x55d5affdf5e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/3498735230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affce940 0x55d5affd7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/3498735230 >> 172.21.15.101:0/3498735230 conn(0x55d5affc3880 msgr2=0x55d5affc8800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/3498735230 shutdown_connections 2022-07-03T10:36:52.769 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/3498735230 wait complete. 2022-07-03T10:36:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.755+0000 7ff8f80e2b80 1 -- start start 2022-07-03T10:36:52.770 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.756+0000 7ff8f80e2b80 1 Processor -- start 2022-07-03T10:36:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.756+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcf2f0 0x55d5b0034500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5b0034a80 con 0x55d5affcdff0 2022-07-03T10:36:52.771 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5b0034bf0 con 0x55d5affce940 2022-07-03T10:36:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5b0034db0 con 0x55d5affcf2f0 2022-07-03T10:36:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8f80e2b80 10 client.0 _ll_register_callbacks cb 0x55d5afec9270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.772 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e1f9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47102/0 (socket says 172.21.15.101:47102) 2022-07-03T10:36:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51830/0 (socket says 172.21.15.101:51830) 2022-07-03T10:36:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 -- 172.21.15.101:0/2589074481 learned_addr learned my addr 172.21.15.101:0/2589074481 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:52.773 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e279c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcf2f0 0x55d5b0034500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 -- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcf2f0 msgr2=0x55d5b0034500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcf2f0 0x55d5b0034500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 -- 172.21.15.101:0/2589074481 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 msgr2=0x55d5b002d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:52.774 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.757+0000 7ff8e179a700 1 -- 172.21.15.101:0/2589074481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5afd52be0 con 0x55d5affce940 2022-07-03T10:36:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:52.775 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8e179a700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc005c40 tx=0x7ff8cc024040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b8ff7e883af058 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:36:52.776 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc0a8040 con 0x55d5affce940 2022-07-03T10:36:52.777 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8d8021070 con 0x55d5affce940 2022-07-03T10:36:52.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff8cc0a81a0 con 0x55d5affce940 2022-07-03T10:36:52.778 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8cc092430 con 0x55d5affce940 2022-07-03T10:36:52.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8f80e2b80 10 client.25192 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:52.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.758+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d5afd510b0 con 0x55d5affce940 2022-07-03T10:36:52.779 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff8d8021070 con 0x55d5affce940 2022-07-03T10:36:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7ff8cc00b680 con 0x55d5affce940 2022-07-03T10:36:52.780 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 10 client.25192 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 10 client.25192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 20 client.25192 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:52.781 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5b00364e0 0x55d5b00388f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 4 client.25192 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2022-07-03T10:36:52.782 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d5aff168d0 con 0x55d5b00364e0 2022-07-03T10:36:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.759+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5b00364e0 0x55d5b00388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.760+0000 7ff8bcff9700 10 client.25192 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.760+0000 7ff8e1f9b700 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5b00364e0 0x55d5b00388f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a600900045dc5777 server_cookie=4b715890cf16df40 in_seq=0 out_seq=0 2022-07-03T10:36:52.783 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.761+0000 7ff8bcff9700 1 -- 172.21.15.101:0/2589074481 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d5aff168d0 con 0x55d5b00364e0 2022-07-03T10:36:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.761+0000 7ff8bcff9700 10 client.25192 handle_command_reply: tid=0 2022-07-03T10:36:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.761+0000 7ff8f80e2b80 1 client.25192 shutdown 2022-07-03T10:36:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 msgr2=0x55d5b002d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.784 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc005c40 tx=0x7ff8cc024040 comp rx=0 tx=0).stop 2022-07-03T10:36:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 shutdown_connections 2022-07-03T10:36:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d5b00364e0 0x55d5b00388f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d5affcdff0 0x55d5b002d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d5affcf2f0 0x55d5b0034500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.785 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 --2- 172.21.15.101:0/2589074481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d5affce940 0x55d5b002d560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 >> 172.21.15.101:0/2589074481 conn(0x55d5affc3880 msgr2=0x55d5affc7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8bcff9700 0 client.25192 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 shutdown_connections 2022-07-03T10:36:52.786 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 1 -- 172.21.15.101:0/2589074481 wait complete. 2022-07-03T10:36:52.787 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.762+0000 7ff8f80e2b80 20 client.25192 trim_cache size 0 max 16384 2022-07-03T10:36:52.787 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:52.788 INFO:teuthology.orchestra.run.smithi101.stdout: "return_code": 0, 2022-07-03T10:36:52.788 INFO:teuthology.orchestra.run.smithi101.stdout: "scrub_tag": "4edd48d7-1324-4f79-8245-9d5e198f3e5a", 2022-07-03T10:36:52.788 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "asynchronous" 2022-07-03T10:36:52.789 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 msgr2=0x7ff8c80991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.791 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 0x7ff8c80991d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80057f0 tx=0x7ff8d80091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 msgr2=0x7ff8dc110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc110ad0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0093e0 tx=0x7ff8cc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 shutdown_connections 2022-07-03T10:36:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff8c8096d20 0x7ff8c80991d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.792 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff8dc0fd3f0 0x7ff8dc110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8dc0ff000 0x7ff8dc117b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 --2- 172.21.15.101:0/2246379013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8dc0fe670 0x7ff8dc110ad0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 >> 172.21.15.101:0/2246379013 conn(0x7ff8dc0f8b40 msgr2=0x7ff8dc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:52.793 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 shutdown_connections 2022-07-03T10:36:52.794 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:52.763+0000 7ff8e2f9d700 1 -- 172.21.15.101:0/2246379013 wait complete. 2022-07-03T10:36:52.794 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:36:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 -- 172.21.15.101:0/2790270344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 msgr2=0x7f67180fecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.029 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 --2- 172.21.15.101:0/2790270344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f67180fecb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6708009900 tx=0x7f67080091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 -- 172.21.15.101:0/2790270344 shutdown_connections 2022-07-03T10:36:53.030 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 --2- 172.21.15.101:0/2790270344 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180ff1f0 0x7f6718105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 --2- 172.21.15.101:0/2790270344 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180fd5e0 0x7f67180fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.031 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 --2- 172.21.15.101:0/2790270344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f67180fecb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 -- 172.21.15.101:0/2790270344 >> 172.21.15.101:0/2790270344 conn(0x7f67180f8d90 msgr2=0x7f67180fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 -- 172.21.15.101:0/2790270344 shutdown_connections 2022-07-03T10:36:53.032 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.024+0000 7f671f9ff700 1 -- 172.21.15.101:0/2790270344 wait complete. 2022-07-03T10:36:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.025+0000 7f671f9ff700 1 Processor -- start 2022-07-03T10:36:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.025+0000 7f671f9ff700 1 -- start start 2022-07-03T10:36:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180fd5e0 0x7f6718110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.033 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 0x7f6718117e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67181183b0 con 0x7f67180fd5e0 2022-07-03T10:36:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67181184f0 con 0x7f67180ff1f0 2022-07-03T10:36:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671f9ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67181186b0 con 0x7f67180fe860 2022-07-03T10:36:53.034 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671cf9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671df9c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 0x7f6718117e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671d79b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180fd5e0 0x7f6718110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671cf9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52908/0 (socket says 172.21.15.101:52908) 2022-07-03T10:36:53.035 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671df9c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 0x7f6718117e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51856/0 (socket says 172.21.15.101:51856) 2022-07-03T10:36:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.026+0000 7f671cf9a700 1 -- 172.21.15.101:0/3722261282 learned_addr learned my addr 172.21.15.101:0/3722261282 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 -- 172.21.15.101:0/3722261282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 msgr2=0x7f6718117e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 0x7f6718117e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 -- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180fd5e0 msgr2=0x7f6718110880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.036 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180fd5e0 0x7f6718110880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 -- 172.21.15.101:0/3722261282 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009440 con 0x7f67180fe860 2022-07-03T10:36:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671cf9a700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6708005e10 tx=0x7f67080091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f670dffb700 1 -- 172.21.15.101:0/3722261282 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670801d070 con 0x7f67180fe860 2022-07-03T10:36:53.037 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f670dffb700 1 -- 172.21.15.101:0/3722261282 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f670800b9c0 con 0x7f67180fe860 2022-07-03T10:36:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f670dffb700 1 -- 172.21.15.101:0/3722261282 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708022bd0 con 0x7f67180fe860 2022-07-03T10:36:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.027+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6718118930 con 0x7f67180fe860 2022-07-03T10:36:53.038 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.028+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6718118dd0 con 0x7f67180fe860 2022-07-03T10:36:53.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.031+0000 7f670dffb700 1 -- 172.21.15.101:0/3722261282 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f6708029030 con 0x7f67180fe860 2022-07-03T10:36:53.046 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 Processor -- start 2022-07-03T10:36:53.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 -- start start 2022-07-03T10:36:53.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f670dffb700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 0x7f6704099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 0x55bbfa353c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.047 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa355510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa240210 con 0x55bbfa354160 2022-07-03T10:36:53.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa2a9d00 con 0x55bbfa3550a0 2022-07-03T10:36:53.048 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f6733910b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa2a3500 con 0x55bbfa353810 2022-07-03T10:36:53.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.032+0000 7f670dffb700 1 -- 172.21.15.101:0/3722261282 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f67080c09b0 con 0x7f67180fe860 2022-07-03T10:36:53.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671cf9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671cf9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47132/0 (socket says 172.21.15.101:47132) 2022-07-03T10:36:53.049 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 0x55bbfa353c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671cf9a700 1 -- 172.21.15.101:0/4182356850 learned_addr learned my addr 172.21.15.101:0/4182356850 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:53.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671df9c700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 0x7f6704099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671df9c700 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa355510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.050 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 -- 172.21.15.101:0/4182356850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 msgr2=0x55bbfa355510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa355510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 -- 172.21.15.101:0/4182356850 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 msgr2=0x55bbfa354a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671d79b700 1 -- 172.21.15.101:0/4182356850 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbfa12c3b0 con 0x55bbfa353810 2022-07-03T10:36:53.051 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.033+0000 7f671cf9a700 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:53.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.034+0000 7f671d79b700 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 0x55bbfa353c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6714009c70 tx=0x7f6714005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238c178b768bd896 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.034+0000 7f671df9c700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 0x7f6704099250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6710009950 tx=0x7f6710009460 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.052 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.034+0000 7f6702ffd700 1 -- 172.21.15.101:0/4182356850 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714009e80 con 0x55bbfa353810 2022-07-03T10:36:53.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.034+0000 7f6702ffd700 1 -- 172.21.15.101:0/4182356850 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6714003500 con 0x55bbfa353810 2022-07-03T10:36:53.053 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.034+0000 7f6702ffd700 1 -- 172.21.15.101:0/4182356850 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714019680 con 0x55bbfa353810 2022-07-03T10:36:53.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 -- 172.21.15.101:0/4182356850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 msgr2=0x55bbfa353c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 0x55bbfa353c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6714009c70 tx=0x7f6714005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.054 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 -- 172.21.15.101:0/4182356850 shutdown_connections 2022-07-03T10:36:53.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa355510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa353810 0x55bbfa353c20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 --2- 172.21.15.101:0/4182356850 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa354160 0x55bbfa354a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.055 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 -- 172.21.15.101:0/4182356850 >> 172.21.15.101:0/4182356850 conn(0x55bbfa349800 msgr2=0x55bbfa350190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 -- 172.21.15.101:0/4182356850 shutdown_connections 2022-07-03T10:36:53.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.035+0000 7f6733910b80 1 -- 172.21.15.101:0/4182356850 wait complete. 2022-07-03T10:36:53.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 -- start start 2022-07-03T10:36:53.056 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 Processor -- start 2022-07-03T10:36:53.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa35d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa3550a0 0x55bbfa364830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa191e90 con 0x55bbfa3550a0 2022-07-03T10:36:53.057 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa364de0 con 0x55bbfa353810 2022-07-03T10:36:53.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa364fa0 con 0x55bbfa354160 2022-07-03T10:36:53.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.036+0000 7f6733910b80 10 client.0 _ll_register_callbacks cb 0x55bbfa24f1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:53.058 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51870/0 (socket says 172.21.15.101:51870) 2022-07-03T10:36:53.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 -- 172.21.15.101:0/251237453 learned_addr learned my addr 172.21.15.101:0/251237453 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:53.059 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671df9c700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa3550a0 0x55bbfa364830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671cf9a700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa35d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 -- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 msgr2=0x55bbfa35d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa35d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.060 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 -- 172.21.15.101:0/251237453 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa3550a0 msgr2=0x55bbfa364830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa3550a0 0x55bbfa364830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 -- 172.21.15.101:0/251237453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6718118dd0 con 0x55bbfa353810 2022-07-03T10:36:53.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f671d79b700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f671401ae70 tx=0x7f6714005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4493973bd90e76a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.061 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.037+0000 7f66f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:36:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67140042e0 con 0x55bbfa353810 2022-07-03T10:36:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f6714003670 con 0x55bbfa353810 2022-07-03T10:36:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bbfa346fc0 con 0x55bbfa353810 2022-07-03T10:36:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6714003a40 con 0x55bbfa353810 2022-07-03T10:36:53.062 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f6733910b80 10 client.25204 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:53.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7f6714009950 con 0x55bbfa353810 2022-07-03T10:36:53.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.038+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55bbfa346fc0 con 0x55bbfa353810 2022-07-03T10:36:53.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f6714020070 con 0x55bbfa353810 2022-07-03T10:36:53.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 10 client.25204 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:53.063 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 10 client.25204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:53.064 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 20 client.25204 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:53.064 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3690a0 0x55bbfa36b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.064 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 4 client.25204 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:53.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbfa29ca70 con 0x55bbfa3690a0 2022-07-03T10:36:53.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f671cf9a700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3690a0 0x55bbfa36b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f671cf9a700 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3690a0 0x55bbfa36b4b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=788fa0bcd8c6f18b server_cookie=d4514682f696d524 in_seq=0 out_seq=0 2022-07-03T10:36:53.065 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.039+0000 7f66f3fff700 10 client.25204 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.040+0000 7f66f3fff700 1 -- 172.21.15.101:0/251237453 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55bbfa29ca70 con 0x55bbfa3690a0 2022-07-03T10:36:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.040+0000 7f66f3fff700 10 client.25204 handle_command_reply: tid=0 2022-07-03T10:36:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.040+0000 7f6733910b80 1 client.25204 shutdown 2022-07-03T10:36:53.066 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 msgr2=0x55bbfa35d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f671401ae70 tx=0x7f6714005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 shutdown_connections 2022-07-03T10:36:53.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3690a0 0x55bbfa36b4b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa3550a0 0x55bbfa364830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.067 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa353810 0x55bbfa35d2b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.068 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 --2- 172.21.15.101:0/251237453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa35d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.068 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 >> 172.21.15.101:0/251237453 conn(0x55bbfa349800 msgr2=0x55bbfa34fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.068 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f66f3fff700 0 client.25204 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:53.068 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 shutdown_connections 2022-07-03T10:36:53.068 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 1 -- 172.21.15.101:0/251237453 wait complete. 2022-07-03T10:36:53.069 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.041+0000 7f6733910b80 20 client.25204 trim_cache size 0 max 16384 2022-07-03T10:36:53.075 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 Processor -- start 2022-07-03T10:36:53.075 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 -- start start 2022-07-03T10:36:53.076 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.076 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa354160 0x55bbfa34dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.076 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa3550a0 0x55bbfa34e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.076 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa35be70 con 0x55bbfa353810 2022-07-03T10:36:53.076 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa370060 con 0x55bbfa354160 2022-07-03T10:36:53.077 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.058+0000 7f6733910b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa368c10 con 0x55bbfa3550a0 2022-07-03T10:36:53.077 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.077 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47152/0 (socket says 172.21.15.101:47152) 2022-07-03T10:36:53.077 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 -- 172.21.15.101:0/2288082191 learned_addr learned my addr 172.21.15.101:0/2288082191 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:53.078 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671cf9a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa354160 0x55bbfa34dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.078 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671df9c700 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa3550a0 0x55bbfa34e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.078 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 -- 172.21.15.101:0/2288082191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa3550a0 msgr2=0x55bbfa34e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.078 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa3550a0 0x55bbfa34e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 -- 172.21.15.101:0/2288082191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa354160 msgr2=0x55bbfa34dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa354160 0x55bbfa34dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 -- 172.21.15.101:0/2288082191 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbfa0d8b60 con 0x55bbfa353810 2022-07-03T10:36:53.079 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.059+0000 7f671d79b700 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6714009bc0 tx=0x7f6714005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168d7c426b7c4b82 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f67037fe700 1 -- 172.21.15.101:0/2288082191 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714003670 con 0x55bbfa353810 2022-07-03T10:36:53.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f67037fe700 1 -- 172.21.15.101:0/2288082191 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6714003a40 con 0x55bbfa353810 2022-07-03T10:36:53.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f67037fe700 1 -- 172.21.15.101:0/2288082191 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714019c90 con 0x55bbfa353810 2022-07-03T10:36:53.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 -- 172.21.15.101:0/2288082191 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 msgr2=0x55bbfa35b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.080 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6714009bc0 tx=0x7f6714005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 -- 172.21.15.101:0/2288082191 shutdown_connections 2022-07-03T10:36:53.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa35b7e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa3550a0 0x55bbfa34e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.081 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 --2- 172.21.15.101:0/2288082191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa354160 0x55bbfa34dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.082 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.060+0000 7f6733910b80 1 -- 172.21.15.101:0/2288082191 >> 172.21.15.101:0/2288082191 conn(0x55bbfa349800 msgr2=0x55bbfa35b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.082 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.061+0000 7f6733910b80 1 -- 172.21.15.101:0/2288082191 shutdown_connections 2022-07-03T10:36:53.082 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.061+0000 7f6733910b80 1 -- 172.21.15.101:0/2288082191 wait complete. 2022-07-03T10:36:53.082 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.061+0000 7f6733910b80 1 -- start start 2022-07-03T10:36:53.082 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.061+0000 7f6733910b80 1 Processor -- start 2022-07-03T10:36:53.083 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.083 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa3b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.083 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa3ba050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.083 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa3ba5d0 con 0x55bbfa353810 2022-07-03T10:36:53.084 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa3ba740 con 0x55bbfa3550a0 2022-07-03T10:36:53.084 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfa3ba900 con 0x55bbfa354160 2022-07-03T10:36:53.084 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f6733910b80 10 client.0 _ll_register_callbacks cb 0x55bbfa24f1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:53.085 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.062+0000 7f671d79b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.085 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671d79b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47154/0 (socket says 172.21.15.101:47154) 2022-07-03T10:36:53.085 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671d79b700 1 -- 172.21.15.101:0/202786843 learned_addr learned my addr 172.21.15.101:0/202786843 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:53.085 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa3b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.086 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671df9c700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa3ba050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.086 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 -- 172.21.15.101:0/202786843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 msgr2=0x55bbfa3ba050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.086 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa3ba050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.086 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 -- 172.21.15.101:0/202786843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 msgr2=0x55bbfa3b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:53.086 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.087 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 -- 172.21.15.101:0/202786843 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6714009950 con 0x55bbfa354160 2022-07-03T10:36:53.087 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671d79b700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:53.087 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f671cf9a700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa3b30b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f670808e380 tx=0x7f67080c0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c647c9c00f970e server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:53.087 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f66f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:36:53.088 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.063+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708091910 con 0x55bbfa354160 2022-07-03T10:36:53.088 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f6708098000 con 0x55bbfa354160 2022-07-03T10:36:53.088 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6714003500 con 0x55bbfa354160 2022-07-03T10:36:53.088 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f670808fcb0 con 0x55bbfa354160 2022-07-03T10:36:53.088 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 10 client.25205 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:53.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55bbfa0d7030 con 0x55bbfa354160 2022-07-03T10:36:53.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f67080b7080 con 0x55bbfa354160 2022-07-03T10:36:53.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f6708090e00 con 0x55bbfa354160 2022-07-03T10:36:53.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 10 client.25205 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:53.089 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 10 client.25205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:53.090 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 20 client.25205 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:53.090 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3bc040 0x55bbfa3be420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:53.090 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 4 client.25205 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:36:53.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.064+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bbfa29c850 con 0x55bbfa3bc040 2022-07-03T10:36:53.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f671d79b700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3bc040 0x55bbfa3be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:53.091 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f66f3fff700 10 client.25205 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:53.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f671d79b700 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3bc040 0x55bbfa3be420 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12fe0fd97f8d3681 server_cookie=9fedc91268a30670 in_seq=0 out_seq=0 2022-07-03T10:36:53.092 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f66f3fff700 1 -- 172.21.15.101:0/202786843 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bbfa29c850 con 0x55bbfa3bc040 2022-07-03T10:36:53.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f66f3fff700 10 client.25205 handle_command_reply: tid=0 2022-07-03T10:36:53.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.065+0000 7f6733910b80 1 client.25205 shutdown 2022-07-03T10:36:53.093 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 msgr2=0x55bbfa3b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa3b30b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f670808e380 tx=0x7f67080c0040 comp rx=0 tx=0).stop 2022-07-03T10:36:53.094 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 shutdown_connections 2022-07-03T10:36:53.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55bbfa3bc040 0x55bbfa3be420 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55bbfa353810 0x55bbfa3b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f66f3fff700 0 client.25205 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:53.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55bbfa3550a0 0x55bbfa3ba050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 --2- 172.21.15.101:0/202786843 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55bbfa354160 0x55bbfa3b30b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 >> 172.21.15.101:0/202786843 conn(0x55bbfa349800 msgr2=0x55bbfa35c800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 shutdown_connections 2022-07-03T10:36:53.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 1 -- 172.21.15.101:0/202786843 wait complete. 2022-07-03T10:36:53.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.066+0000 7f6733910b80 20 client.25205 trim_cache size 0 max 16384 2022-07-03T10:36:53.099 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:53.099 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "no active scrubs running", 2022-07-03T10:36:53.100 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": {} 2022-07-03T10:36:53.100 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 msgr2=0x7f6704099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 0x7f6704099250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6710009950 tx=0x7f6710009460 comp rx=0 tx=0).stop 2022-07-03T10:36:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 msgr2=0x7f6718110dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6708005e10 tx=0x7f67080091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 shutdown_connections 2022-07-03T10:36:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f6704096da0 0x7f6704099250 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f67180fd5e0 0x7f6718110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67180ff1f0 0x7f6718117e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 --2- 172.21.15.101:0/3722261282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67180fe860 0x7f6718110dc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:53.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 >> 172.21.15.101:0/3722261282 conn(0x7f67180f8d90 msgr2=0x7f67180fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:53.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 shutdown_connections 2022-07-03T10:36:53.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:53.067+0000 7f671f9ff700 1 -- 172.21.15.101:0/3722261282 wait complete. 2022-07-03T10:36:53.109 INFO:tasks.cephfs.filesystem:all active scrubs completed 2022-07-03T10:36:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: from='client.? 172.21.15.101:0/3711970233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: scrub queued for path: / 2022-07-03T10:36:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: scrub summary: active paths [/] 2022-07-03T10:36:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:53 smithi161 ceph-mon[91579]: scrub summary: idle 2022-07-03T10:36:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: from='client.? 172.21.15.101:0/3711970233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: scrub queued for path: / 2022-07-03T10:36:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: scrub summary: active paths [/] 2022-07-03T10:36:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:53 smithi188 ceph-mon[92425]: scrub summary: idle 2022-07-03T10:36:53.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:36:53.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: from='client.? 172.21.15.101:0/3711970233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-07-03T10:36:53.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: scrub queued for path: / 2022-07-03T10:36:53.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: scrub summary: active paths [/] 2022-07-03T10:36:53.744 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:53.745 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:53 smithi101 ceph-mon[85126]: scrub summary: idle 2022-07-03T10:36:54.110 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2022-07-03T10:36:54.111 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2022-07-03T10:36:54.350 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 -- 172.21.15.101:0/1689263096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0fe670 msgr2=0x7fd6ac0feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.350 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/1689263096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac0feac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009900 tx=0x7fd6a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.350 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 -- 172.21.15.101:0/1689263096 shutdown_connections 2022-07-03T10:36:54.351 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/1689263096 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.351 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/1689263096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fd3f0 0x7fd6ac0fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.351 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/1689263096 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac0feac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.351 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.348+0000 7fd6ab59e700 1 -- 172.21.15.101:0/1689263096 >> 172.21.15.101:0/1689263096 conn(0x7fd6ac0f8b40 msgr2=0x7fd6ac0fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.351 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.349+0000 7fd6ab59e700 1 -- 172.21.15.101:0/1689263096 shutdown_connections 2022-07-03T10:36:54.352 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.349+0000 7fd6ab59e700 1 -- 172.21.15.101:0/1689263096 wait complete. 2022-07-03T10:36:54.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.349+0000 7fd6ab59e700 1 Processor -- start 2022-07-03T10:36:54.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 -- start start 2022-07-03T10:36:54.355 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0fd3f0 0x7fd6ac10e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.356 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac10e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.356 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.356 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac115f40 con 0x7fd6ac0ff000 2022-07-03T10:36:54.357 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac116080 con 0x7fd6ac0fd3f0 2022-07-03T10:36:54.357 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6ab59e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac116240 con 0x7fd6ac0fe670 2022-07-03T10:36:54.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6aad9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0fd3f0 0x7fd6ac10e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.358 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6a9d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac10e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6aad9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47178/0 (socket says 172.21.15.101:47178) 2022-07-03T10:36:54.359 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.350+0000 7fd6aad9d700 1 -- 172.21.15.101:0/3561480037 learned_addr learned my addr 172.21.15.101:0/3561480037 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 -- 172.21.15.101:0/3561480037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fe670 msgr2=0x7fd6ac10e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.360 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac10e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 -- 172.21.15.101:0/3561480037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0fd3f0 msgr2=0x7fd6ac10e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.361 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0fd3f0 0x7fd6ac10e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 -- 172.21.15.101:0/3561480037 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a0009440 con 0x7fd6ac0ff000 2022-07-03T10:36:54.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400eb80 tx=0x7fd6a4009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.362 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd69affd700 1 -- 172.21.15.101:0/3561480037 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4004a50 con 0x7fd6ac0ff000 2022-07-03T10:36:54.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.351+0000 7fd69affd700 1 -- 172.21.15.101:0/3561480037 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd6a400ce80 con 0x7fd6ac0ff000 2022-07-03T10:36:54.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.352+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac0707b0 con 0x7fd6ac0ff000 2022-07-03T10:36:54.363 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.352+0000 7fd69affd700 1 -- 172.21.15.101:0/3561480037 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4005730 con 0x7fd6ac0ff000 2022-07-03T10:36:54.364 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.352+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac070d40 con 0x7fd6ac0ff000 2022-07-03T10:36:54.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.355+0000 7fd69affd700 1 -- 172.21.15.101:0/3561480037 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fd6a400c040 con 0x7fd6ac0ff000 2022-07-03T10:36:54.373 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.356+0000 7fd69affd700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 0x7fd6940991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.374 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.356+0000 7fd69affd700 1 -- 172.21.15.101:0/3561480037 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd6a4014070 con 0x7fd6ac0ff000 2022-07-03T10:36:54.374 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.356+0000 7fd6c492bb80 1 Processor -- start 2022-07-03T10:36:54.374 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 0x7fd6940991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.374 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 -- start start 2022-07-03T10:36:54.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 0x7fd6940991d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0fe4d0 tx=0x7fd69c005ef0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af46ff7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af46ff8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46ee3290 con 0x55af46ff6ff0 2022-07-03T10:36:54.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46f4cd80 con 0x55af46ff82f0 2022-07-03T10:36:54.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.357+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46f46580 con 0x55af46ff7940 2022-07-03T10:36:54.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af46ff7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:52954/0 (socket says 172.21.15.101:52954) 2022-07-03T10:36:54.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1425422347 learned_addr learned my addr 172.21.15.101:0/1425422347 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af46ff8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1425422347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 msgr2=0x55af46ff8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af46ff8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1425422347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 msgr2=0x55af46ff7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af46ff7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1425422347 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af46dcf430 con 0x55af46ff7940 2022-07-03T10:36:54.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.358+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af46ff7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:54.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009900 tx=0x7fd6a00091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9951d1942fe7fb3d server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd693fff700 1 -- 172.21.15.101:0/1425422347 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0021070 con 0x55af46ff7940 2022-07-03T10:36:54.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd693fff700 1 -- 172.21.15.101:0/1425422347 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd6a0022040 con 0x55af46ff7940 2022-07-03T10:36:54.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd693fff700 1 -- 172.21.15.101:0/1425422347 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a001ba70 con 0x55af46ff7940 2022-07-03T10:36:54.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1425422347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 msgr2=0x55af46ff7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.359+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009900 tx=0x7fd6a00091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1425422347 shutdown_connections 2022-07-03T10:36:54.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af46ff7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af46ff8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1425422347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af46ff7db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1425422347 >> 172.21.15.101:0/1425422347 conn(0x55af46fec880 msgr2=0x55af46ff1830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1425422347 shutdown_connections 2022-07-03T10:36:54.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1425422347 wait complete. 2022-07-03T10:36:54.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.360+0000 7fd6c492bb80 1 -- start start 2022-07-03T10:36:54.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.361+0000 7fd6c492bb80 1 Processor -- start 2022-07-03T10:36:54.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af47000ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af47007af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46e34f10 con 0x55af46ff6ff0 2022-07-03T10:36:54.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af470080a0 con 0x55af46ff82f0 2022-07-03T10:36:54.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af47008260 con 0x55af46ff7940 2022-07-03T10:36:54.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6c492bb80 10 client.0 _ll_register_callbacks cb 0x55af46ef2270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:54.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47198/0 (socket says 172.21.15.101:47198) 2022-07-03T10:36:54.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 -- 172.21.15.101:0/3177627697 learned_addr learned my addr 172.21.15.101:0/3177627697 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af47000ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af47007af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 -- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 msgr2=0x55af47000ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af47000ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.386 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 -- 172.21.15.101:0/3177627697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 msgr2=0x55af47007af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af47007af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.362+0000 7fd6aa59c700 1 -- 172.21.15.101:0/3177627697 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c017070 con 0x55af46ff6ff0 2022-07-03T10:36:54.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009fd0 tx=0x7fd69c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa25184c2a61033 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6917fa700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:36:54.387 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c022070 con 0x55af46ff6ff0 2022-07-03T10:36:54.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af46fea040 con 0x55af46ff6ff0 2022-07-03T10:36:54.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd69c011e00 con 0x55af46ff6ff0 2022-07-03T10:36:54.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd69c01d830 con 0x55af46ff6ff0 2022-07-03T10:36:54.388 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6c492bb80 10 client.15150 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:54.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.363+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7fd6ac1061b0 con 0x55af46ff6ff0 2022-07-03T10:36:54.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55af46fea040 con 0x55af46ff6ff0 2022-07-03T10:36:54.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fd69c0279e0 con 0x55af46ff6ff0 2022-07-03T10:36:54.389 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 10 client.15150 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:54.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 10 client.15150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:54.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 20 client.15150 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:54.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af4700c580 0x55af4700e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 4 client.15150 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:54.390 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.364+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af46f3faf0 con 0x55af4700c580 2022-07-03T10:36:54.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.365+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af4700c580 0x55af4700e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.365+0000 7fd6917fa700 10 client.15150 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.365+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af4700c580 0x55af4700e990 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df894a775ece22d3 server_cookie=5be9484cf16fb222 in_seq=0 out_seq=0 2022-07-03T10:36:54.391 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6917fa700 1 -- 172.21.15.101:0/3177627697 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55af46f3faf0 con 0x55af4700c580 2022-07-03T10:36:54.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6917fa700 10 client.15150 handle_command_reply: tid=0 2022-07-03T10:36:54.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 client.15150 shutdown 2022-07-03T10:36:54.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 msgr2=0x55af47000570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009fd0 tx=0x7fd69c018000 comp rx=0 tx=0).stop 2022-07-03T10:36:54.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 shutdown_connections 2022-07-03T10:36:54.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af4700c580 0x55af4700e990 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47000570 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6917fa700 0 client.15150 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff82f0 0x55af47007af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/3177627697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff7940 0x55af47000ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.366+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 >> 172.21.15.101:0/3177627697 conn(0x55af46fec880 msgr2=0x55af46ff0180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.367+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 shutdown_connections 2022-07-03T10:36:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.367+0000 7fd6c492bb80 1 -- 172.21.15.101:0/3177627697 wait complete. 2022-07-03T10:36:54.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.367+0000 7fd6c492bb80 20 client.15150 trim_cache size 0 max 16384 2022-07-03T10:36:54.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.383+0000 7fd6c492bb80 1 Processor -- start 2022-07-03T10:36:54.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 -- start start 2022-07-03T10:36:54.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff7940 0x55af46fffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af47000200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af47013540 con 0x55af46ff7940 2022-07-03T10:36:54.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46ff3230 con 0x55af46ff6ff0 2022-07-03T10:36:54.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46fefef0 con 0x55af46ff82f0 2022-07-03T10:36:54.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51938/0 (socket says 172.21.15.101:51938) 2022-07-03T10:36:54.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6aa59c700 1 -- 172.21.15.101:0/2709859530 learned_addr learned my addr 172.21.15.101:0/2709859530 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af47000200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.384+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff7940 0x55af46fffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 -- 172.21.15.101:0/2709859530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 msgr2=0x55af47000200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af47000200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 -- 172.21.15.101:0/2709859530 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff7940 msgr2=0x55af46fffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff7940 0x55af46fffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 -- 172.21.15.101:0/2709859530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af46d7bbe0 con 0x55af46ff6ff0 2022-07-03T10:36:54.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd69c027dc0 tx=0x7fd69c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c90728acc81ce7 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6989ff700 1 -- 172.21.15.101:0/2709859530 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c022040 con 0x55af46ff6ff0 2022-07-03T10:36:54.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6989ff700 1 -- 172.21.15.101:0/2709859530 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd69c000c30 con 0x55af46ff6ff0 2022-07-03T10:36:54.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.385+0000 7fd6989ff700 1 -- 172.21.15.101:0/2709859530 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c027830 con 0x55af46ff6ff0 2022-07-03T10:36:54.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- 172.21.15.101:0/2709859530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 msgr2=0x55af46ff6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd69c027dc0 tx=0x7fd69c018000 comp rx=0 tx=0).stop 2022-07-03T10:36:54.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- 172.21.15.101:0/2709859530 shutdown_connections 2022-07-03T10:36:54.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af47000200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff6ff0 0x55af46ff6500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/2709859530 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff7940 0x55af46fffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- 172.21.15.101:0/2709859530 >> 172.21.15.101:0/2709859530 conn(0x55af46fec880 msgr2=0x55af46ff5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- 172.21.15.101:0/2709859530 shutdown_connections 2022-07-03T10:36:54.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- 172.21.15.101:0/2709859530 wait complete. 2022-07-03T10:36:54.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.386+0000 7fd6c492bb80 1 -- start start 2022-07-03T10:36:54.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 Processor -- start 2022-07-03T10:36:54.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47056c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af4705e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4705e670 con 0x55af46ff6ff0 2022-07-03T10:36:54.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4705e7e0 con 0x55af46ff7940 2022-07-03T10:36:54.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4705e9a0 con 0x55af46ff82f0 2022-07-03T10:36:54.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6c492bb80 10 client.0 _ll_register_callbacks cb 0x55af46ef2270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:54.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6a9d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6aad9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af4705e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6a9d9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51954/0 (socket says 172.21.15.101:51954) 2022-07-03T10:36:54.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1482744685 learned_addr learned my addr 172.21.15.101:0/1482744685 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.387+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47056c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 msgr2=0x55af4705e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af4705e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1482744685 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 msgr2=0x55af47056c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47056c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 -- 172.21.15.101:0/1482744685 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac1061b0 con 0x55af46ff7940 2022-07-03T10:36:54.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6aad9d700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af4705e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:54.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47056c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:54.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6a9d9b700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009900 tx=0x7fd6a0032040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9f5031f016f8614 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6917fa700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:36:54.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0035050 con 0x55af46ff7940 2022-07-03T10:36:54.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6a0022040 con 0x55af46ff7940 2022-07-03T10:36:54.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.388+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fd6a0003570 con 0x55af46ff7940 2022-07-03T10:36:54.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6a0021040 con 0x55af46ff7940 2022-07-03T10:36:54.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 10 client.25228 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:54.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55af46d7a0b0 con 0x55af46ff7940 2022-07-03T10:36:54.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd6a0022040 con 0x55af46ff7940 2022-07-03T10:36:54.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fd6a000b440 con 0x55af46ff7940 2022-07-03T10:36:54.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 10 client.25228 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:54.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 10 client.25228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:54.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 20 client.25228 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:54.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af47060060 0x55af47062440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 4 client.25228 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2022-07-03T10:36:54.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.389+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55af46f3f8d0 con 0x55af47060060 2022-07-03T10:36:54.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.390+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af47060060 0x55af47062440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.390+0000 7fd6aa59c700 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af47060060 0x55af47062440 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f34fb0959c69576d server_cookie=7b3f38d1fa5ba95e in_seq=0 out_seq=0 2022-07-03T10:36:54.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.390+0000 7fd6917fa700 10 client.25228 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6917fa700 1 -- 172.21.15.101:0/1482744685 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55af46f3f8d0 con 0x55af47060060 2022-07-03T10:36:54.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6917fa700 10 client.25228 handle_command_reply: tid=0 2022-07-03T10:36:54.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 client.25228 shutdown 2022-07-03T10:36:54.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 msgr2=0x55af47057150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009900 tx=0x7fd6a0032040 comp rx=0 tx=0).stop 2022-07-03T10:36:54.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 shutdown_connections 2022-07-03T10:36:54.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55af47060060 0x55af47062440 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55af46ff6ff0 0x55af47056c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55af46ff82f0 0x55af4705e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 --2- 172.21.15.101:0/1482744685 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55af46ff7940 0x55af47057150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 >> 172.21.15.101:0/1482744685 conn(0x55af46fec880 msgr2=0x55af46ff6660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6917fa700 0 client.25228 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 shutdown_connections 2022-07-03T10:36:54.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 1 -- 172.21.15.101:0/1482744685 wait complete. 2022-07-03T10:36:54.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.392+0000 7fd6c492bb80 20 client.25228 trim_cache size 0 max 16384 2022-07-03T10:36:54.422 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:54.423 INFO:teuthology.orchestra.run.smithi101.stdout: "return_code": 0, 2022-07-03T10:36:54.423 INFO:teuthology.orchestra.run.smithi101.stdout: "scrub_tag": "9a356bd9-7506-45e5-ae75-3354d44f5562", 2022-07-03T10:36:54.423 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "asynchronous" 2022-07-03T10:36:54.423 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:54.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 msgr2=0x7fd6940991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.425 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 0x7fd6940991d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0fe4d0 tx=0x7fd69c005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 msgr2=0x7fd6ac1159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400eb80 tx=0x7fd6a4009460 comp rx=0 tx=0).stop 2022-07-03T10:36:54.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 shutdown_connections 2022-07-03T10:36:54.426 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fd694096d20 0x7fd6940991d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd6ac0ff000 0x7fd6ac1159c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ac0fd3f0 0x7fd6ac10e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.427 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 --2- 172.21.15.101:0/3561480037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6ac0fe670 0x7fd6ac10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 >> 172.21.15.101:0/3561480037 conn(0x7fd6ac0f8b40 msgr2=0x7fd6ac101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.428 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 shutdown_connections 2022-07-03T10:36:54.429 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.393+0000 7fd6ab59e700 1 -- 172.21.15.101:0/3561480037 wait complete. 2022-07-03T10:36:54.429 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:36:54.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/3568812851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 msgr2=0x7f7d58100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/3568812851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58100ec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009900 tx=0x7f7d4c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/3568812851 shutdown_connections 2022-07-03T10:36:54.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/3568812851 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d580ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/3568812851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 0x7f7d58107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/3568812851 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58100ec0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/3568812851 >> 172.21.15.101:0/3568812851 conn(0x7f7d580faf80 msgr2=0x7f7d580fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/3568812851 shutdown_connections 2022-07-03T10:36:54.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.656+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/3568812851 wait complete. 2022-07-03T10:36:54.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 Processor -- start 2022-07-03T10:36:54.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 -- start start 2022-07-03T10:36:54.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d581107a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 0x7f7d58117d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d581182d0 con 0x7f7d580ff7f0 2022-07-03T10:36:54.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d58118410 con 0x7f7d58100a70 2022-07-03T10:36:54.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.657+0000 7f7d5e7ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d581185d0 con 0x7f7d58101400 2022-07-03T10:36:54.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d581107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51972/0 (socket says 172.21.15.101:51972) 2022-07-03T10:36:54.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 -- 172.21.15.101:0/1192187576 learned_addr learned my addr 172.21.15.101:0/1192187576 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 0x7f7d58117d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 -- 172.21.15.101:0/1192187576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 msgr2=0x7f7d58117d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 0x7f7d58117d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 -- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 msgr2=0x7f7d581107a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d581107a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d577fe700 1 -- 172.21.15.101:0/1192187576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c009440 con 0x7f7d58100a70 2022-07-03T10:36:54.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.658+0000 7f7d57fff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d581107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:54.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d577fe700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009f70 tx=0x7f7d4c0091e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.670 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d54ff9700 1 -- 172.21.15.101:0/1192187576 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c01d070 con 0x7f7d58100a70 2022-07-03T10:36:54.670 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d58118850 con 0x7f7d58100a70 2022-07-03T10:36:54.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d54ff9700 1 -- 172.21.15.101:0/1192187576 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7d4c00b9c0 con 0x7f7d58100a70 2022-07-03T10:36:54.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d54ff9700 1 -- 172.21.15.101:0/1192187576 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c022c70 con 0x7f7d58100a70 2022-07-03T10:36:54.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.659+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d58118d50 con 0x7f7d58100a70 2022-07-03T10:36:54.680 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.662+0000 7f7d54ff9700 1 -- 172.21.15.101:0/1192187576 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f7d4c029030 con 0x7f7d58100a70 2022-07-03T10:36:54.681 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.663+0000 7f7d72677b80 1 Processor -- start 2022-07-03T10:36:54.681 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.663+0000 7f7d72677b80 1 -- start start 2022-07-03T10:36:54.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a6810 0x5558c14a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d54ff9700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 0x7f7d400991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a80a0 0x5558c14a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c1393210 con 0x5558c14a6810 2022-07-03T10:36:54.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c13fcd00 con 0x5558c14a7160 2022-07-03T10:36:54.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d72677b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c13f6500 con 0x5558c14a80a0 2022-07-03T10:36:54.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d54ff9700 1 -- 172.21.15.101:0/1192187576 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7d4c0c0dc0 con 0x7f7d58100a70 2022-07-03T10:36:54.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 0x7f7d400991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:51978/0 (socket says 172.21.15.101:51978) 2022-07-03T10:36:54.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d577fe700 1 -- 172.21.15.101:0/2767914835 learned_addr learned my addr 172.21.15.101:0/2767914835 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d57fff700 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a6810 0x5558c14a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.664+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a80a0 0x5558c14a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 -- 172.21.15.101:0/2767914835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a80a0 msgr2=0x5558c14a8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a80a0 0x5558c14a8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 -- 172.21.15.101:0/2767914835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a6810 msgr2=0x5558c14a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a6810 0x5558c14a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 -- 172.21.15.101:0/2767914835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558c127f3b0 con 0x5558c14a7160 2022-07-03T10:36:54.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 0x7f7d400991d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009950 tx=0x7f7d50009460 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c091620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22f63e2f1d00adbf server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d3dffb700 1 -- 172.21.15.101:0/2767914835 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0a8350 con 0x5558c14a7160 2022-07-03T10:36:54.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d3dffb700 1 -- 172.21.15.101:0/2767914835 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7d4c091a90 con 0x5558c14a7160 2022-07-03T10:36:54.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.665+0000 7f7d3dffb700 1 -- 172.21.15.101:0/2767914835 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c022470 con 0x5558c14a7160 2022-07-03T10:36:54.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 -- 172.21.15.101:0/2767914835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 msgr2=0x5558c14a7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c091620 comp rx=0 tx=0).stop 2022-07-03T10:36:54.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 -- 172.21.15.101:0/2767914835 shutdown_connections 2022-07-03T10:36:54.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a6810 0x5558c14a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a80a0 0x5558c14a8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2767914835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14a7a80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 -- 172.21.15.101:0/2767914835 >> 172.21.15.101:0/2767914835 conn(0x5558c149c800 msgr2=0x5558c14a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 -- 172.21.15.101:0/2767914835 shutdown_connections 2022-07-03T10:36:54.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.666+0000 7f7d72677b80 1 -- 172.21.15.101:0/2767914835 wait complete. 2022-07-03T10:36:54.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.667+0000 7f7d72677b80 1 -- start start 2022-07-03T10:36:54.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.667+0000 7f7d72677b80 1 Processor -- start 2022-07-03T10:36:54.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14b7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c12e4e90 con 0x5558c14a80a0 2022-07-03T10:36:54.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c14b7bc0 con 0x5558c14a7160 2022-07-03T10:36:54.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c14b7d80 con 0x5558c14a6810 2022-07-03T10:36:54.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d72677b80 10 client.0 _ll_register_callbacks cb 0x5558c13a21f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:54.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53022/0 (socket says 172.21.15.101:53022) 2022-07-03T10:36:54.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 -- 172.21.15.101:0/2426898605 learned_addr learned my addr 172.21.15.101:0/2426898605 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 -- 172.21.15.101:0/2426898605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 msgr2=0x5558c14b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 -- 172.21.15.101:0/2426898605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 msgr2=0x5558c14b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d57fff700 1 -- 172.21.15.101:0/2426898605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d58118d50 con 0x5558c14a6810 2022-07-03T10:36:54.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.668+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:54.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d57fff700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7d48012cf0 tx=0x7f7d48005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31ac965aacd47be9 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:36:54.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48017070 con 0x5558c14a6810 2022-07-03T10:36:54.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7d4800f8b0 con 0x5558c14a6810 2022-07-03T10:36:54.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558c1499fc0 con 0x5558c14a6810 2022-07-03T10:36:54.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d48009e90 con 0x5558c14a6810 2022-07-03T10:36:54.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d72677b80 10 client.25235 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:54.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.669+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5558c122bb60 con 0x5558c14a6810 2022-07-03T10:36:54.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5558c1499fc0 con 0x5558c14a6810 2022-07-03T10:36:54.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f7d48010430 con 0x5558c14a6810 2022-07-03T10:36:54.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 10 client.25235 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:54.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 10 client.25235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:54.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 20 client.25235 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:54.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c14bc0a0 0x5558c14be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 4 client.25235 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:36:54.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558c13efa70 con 0x5558c14bc0a0 2022-07-03T10:36:54.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.670+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c14bc0a0 0x5558c14be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.671+0000 7f7d32ffd700 10 client.25235 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.671+0000 7f7d577fe700 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c14bc0a0 0x5558c14be4b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f12d64a6f1a456cc server_cookie=6bd0a2f4edb430a8 in_seq=0 out_seq=0 2022-07-03T10:36:54.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d32ffd700 1 -- 172.21.15.101:0/2426898605 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5558c13efa70 con 0x5558c14bc0a0 2022-07-03T10:36:54.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d32ffd700 10 client.25235 handle_command_reply: tid=0 2022-07-03T10:36:54.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 client.25235 shutdown 2022-07-03T10:36:54.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 msgr2=0x5558c14b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7d48012cf0 tx=0x7f7d48005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 shutdown_connections 2022-07-03T10:36:54.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c14bc0a0 0x5558c14be4b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14b0130 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.712 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 --2- 172.21.15.101:0/2426898605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.712 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d32ffd700 0 client.25235 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.713 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 >> 172.21.15.101:0/2426898605 conn(0x5558c149c800 msgr2=0x5558c14a2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.713 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 shutdown_connections 2022-07-03T10:36:54.713 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 1 -- 172.21.15.101:0/2426898605 wait complete. 2022-07-03T10:36:54.713 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.672+0000 7f7d72677b80 20 client.25235 trim_cache size 0 max 16384 2022-07-03T10:36:54.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 Processor -- start 2022-07-03T10:36:54.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 -- start start 2022-07-03T10:36:54.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.725 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c14a6390 con 0x5558c14a80a0 2022-07-03T10:36:54.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c14af6b0 con 0x5558c14a7160 2022-07-03T10:36:54.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.689+0000 7f7d72677b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c149fe70 con 0x5558c14a6810 2022-07-03T10:36:54.726 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d5cd9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53034/0 (socket says 172.21.15.101:53034) 2022-07-03T10:36:54.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 -- 172.21.15.101:0/67907395 learned_addr learned my addr 172.21.15.101:0/67907395 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.727 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d577fe700 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 -- 172.21.15.101:0/67907395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 msgr2=0x5558c14ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 -- 172.21.15.101:0/67907395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 msgr2=0x5558c14a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.728 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 -- 172.21.15.101:0/67907395 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558c122a030 con 0x5558c14a6810 2022-07-03T10:36:54.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d5cd9c700 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:36:54.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.690+0000 7f7d57fff700 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009380 tx=0x7f7d48005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19e4cc5e10b78962 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.691+0000 7f7d3e7fc700 1 -- 172.21.15.101:0/67907395 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48017040 con 0x5558c14a6810 2022-07-03T10:36:54.729 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.691+0000 7f7d3e7fc700 1 -- 172.21.15.101:0/67907395 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7d48009e90 con 0x5558c14a6810 2022-07-03T10:36:54.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.691+0000 7f7d3e7fc700 1 -- 172.21.15.101:0/67907395 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48010ab0 con 0x5558c14a6810 2022-07-03T10:36:54.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- 172.21.15.101:0/67907395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 msgr2=0x5558c14ae0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009380 tx=0x7f7d48005ef0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- 172.21.15.101:0/67907395 shutdown_connections 2022-07-03T10:36:54.730 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c14a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c14ae0e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 --2- 172.21.15.101:0/67907395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c14ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- 172.21.15.101:0/67907395 >> 172.21.15.101:0/67907395 conn(0x5558c149c800 msgr2=0x5558c14aacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- 172.21.15.101:0/67907395 shutdown_connections 2022-07-03T10:36:54.731 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- 172.21.15.101:0/67907395 wait complete. 2022-07-03T10:36:54.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.692+0000 7f7d72677b80 1 -- start start 2022-07-03T10:36:54.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 Processor -- start 2022-07-03T10:36:54.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c150e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.732 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c15150b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c15155f0 con 0x5558c14a80a0 2022-07-03T10:36:54.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c1515760 con 0x5558c14a7160 2022-07-03T10:36:54.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558c1515920 con 0x5558c14a6810 2022-07-03T10:36:54.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d72677b80 10 client.0 _ll_register_callbacks cb 0x5558c13a21f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:36:54.733 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d5cd9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c15150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d577fe700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c150e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d5cd9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c15150b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:47300/0 (socket says 172.21.15.101:47300) 2022-07-03T10:36:54.734 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d57fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53048/0 (socket says 172.21.15.101:53048) 2022-07-03T10:36:54.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.693+0000 7f7d5cd9c700 1 -- 172.21.15.101:0/3365514357 learned_addr learned my addr 172.21.15.101:0/3365514357 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:36:54.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 -- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 msgr2=0x5558c150e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 -- 172.21.15.101:0/3365514357 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 msgr2=0x5558c15150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:36:54.735 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c15150b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 -- 172.21.15.101:0/3365514357 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558c122bb60 con 0x5558c14a7160 2022-07-03T10:36:54.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d57fff700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:36:54.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d577fe700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c150e7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0097b0 tx=0x7f7d4c0a5040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20b6424eb89e3cc server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:36:54.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.694+0000 7f7d32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:36:54.736 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0923c0 con 0x5558c14a7160 2022-07-03T10:36:54.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f7d4c090060 con 0x5558c14a7160 2022-07-03T10:36:54.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d4c091a90 con 0x5558c14a7160 2022-07-03T10:36:54.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d4c022470 con 0x5558c14a7160 2022-07-03T10:36:54.737 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 10 client.25246 fetch_fsmap learned FSMap version 20 2022-07-03T10:36:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5558c12f8800 con 0x5558c14a7160 2022-07-03T10:36:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7d4c091a90 con 0x5558c14a7160 2022-07-03T10:36:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f7d4c0bed20 con 0x5558c14a7160 2022-07-03T10:36:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 10 client.25246 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:36:54.738 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 10 client.25246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:36:54.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 20 client.25246 populate_metadata read hostname 'smithi101' 2022-07-03T10:36:54.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c1516e60 0x5558c1519270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:36:54.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 4 client.25246 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:36:54.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.695+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5558c13ef850 con 0x5558c1516e60 2022-07-03T10:36:54.739 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.696+0000 7f7d57fff700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c1516e60 0x5558c1519270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:36:54.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.696+0000 7f7d57fff700 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c1516e60 0x5558c1519270 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f40d66e9679df55 server_cookie=31bc1de4a3ad47dd in_seq=0 out_seq=0 2022-07-03T10:36:54.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.696+0000 7f7d32ffd700 10 client.25246 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.696+0000 7f7d32ffd700 1 -- 172.21.15.101:0/3365514357 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5558c13ef850 con 0x5558c1516e60 2022-07-03T10:36:54.740 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.696+0000 7f7d32ffd700 10 client.25246 handle_command_reply: tid=0 2022-07-03T10:36:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 client.25246 shutdown 2022-07-03T10:36:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 msgr2=0x5558c150e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c150e7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0097b0 tx=0x7f7d4c0a5040 comp rx=0 tx=0).stop 2022-07-03T10:36:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 shutdown_connections 2022-07-03T10:36:54.741 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5558c1516e60 0x5558c1519270 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5558c14a80a0 0x5558c15150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d32ffd700 0 client.25246 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:36:54.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5558c14a6810 0x5558c150e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 --2- 172.21.15.101:0/3365514357 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5558c14a7160 0x5558c150e7c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.742 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 >> 172.21.15.101:0/3365514357 conn(0x5558c149c800 msgr2=0x5558c14ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 shutdown_connections 2022-07-03T10:36:54.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 1 -- 172.21.15.101:0/3365514357 wait complete. 2022-07-03T10:36:54.743 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.697+0000 7f7d72677b80 20 client.25246 trim_cache size 0 max 16384 2022-07-03T10:36:54.744 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:36:54.744 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "IDLE (waiting for more scrubs)", 2022-07-03T10:36:54.744 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": { 2022-07-03T10:36:54.744 INFO:teuthology.orchestra.run.smithi101.stdout: "9a356bd9-7506-45e5-ae75-3354d44f5562": { 2022-07-03T10:36:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: "path": "/", 2022-07-03T10:36:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "9a356bd9-7506-45e5-ae75-3354d44f5562", 2022-07-03T10:36:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: "options": "recursive,force" 2022-07-03T10:36:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:36:54.745 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:36:54.746 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:36:54.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 msgr2=0x7f7d400991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.747 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 0x7f7d400991d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009950 tx=0x7f7d50009460 comp rx=0 tx=0).stop 2022-07-03T10:36:54.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 msgr2=0x7f7d58110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:36:54.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009f70 tx=0x7f7d4c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 shutdown_connections 2022-07-03T10:36:54.748 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f7d40096d20 0x7f7d400991d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7d580ff7f0 0x7f7d581107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d58101400 0x7f7d58117d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 --2- 172.21.15.101:0/1192187576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d58100a70 0x7f7d58110ce0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:36:54.749 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 >> 172.21.15.101:0/1192187576 conn(0x7f7d580faf80 msgr2=0x7f7d58103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:36:54.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 shutdown_connections 2022-07-03T10:36:54.750 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:36:54.698+0000 7f7d5e7ff700 1 -- 172.21.15.101:0/1192187576 wait complete. 2022-07-03T10:36:54.750 INFO:tasks.cephfs.filesystem:scrub status for tag:9a356bd9-7506-45e5-ae75-3354d44f5562 - {'path': '/', 'tag': '9a356bd9-7506-45e5-ae75-3354d44f5562', 'options': 'recursive,force'} 2022-07-03T10:36:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:55 smithi161 ceph-mon[91579]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2022-07-03T10:36:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:55 smithi161 ceph-mon[91579]: scrub queued for path: / 2022-07-03T10:36:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:55 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:55 smithi161 ceph-mon[91579]: scrub summary: active paths [/] 2022-07-03T10:36:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:55 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:55 smithi188 ceph-mon[92425]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2022-07-03T10:36:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:55 smithi188 ceph-mon[92425]: scrub queued for path: / 2022-07-03T10:36:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:55 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:55 smithi188 ceph-mon[92425]: scrub summary: active paths [/] 2022-07-03T10:36:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:55 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:55.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:55 smithi101 ceph-mon[85126]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2022-07-03T10:36:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:55 smithi101 ceph-mon[85126]: scrub queued for path: / 2022-07-03T10:36:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:55 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:55 smithi101 ceph-mon[85126]: scrub summary: active paths [/] 2022-07-03T10:36:55.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:55 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:36:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:57 smithi161 ceph-mon[91579]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:36:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:57 smithi188 ceph-mon[92425]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:36:57.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:57 smithi101 ceph-mon[85126]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:36:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:58 smithi161 ceph-mon[91579]: scrub summary: idle 2022-07-03T10:36:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:58 smithi188 ceph-mon[92425]: scrub summary: idle 2022-07-03T10:36:58.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:58 smithi101 ceph-mon[85126]: scrub summary: idle 2022-07-03T10:36:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:36:59 smithi161 ceph-mon[91579]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-07-03T10:36:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:36:59 smithi188 ceph-mon[92425]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-07-03T10:36:59.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:36:59 smithi101 ceph-mon[85126]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2022-07-03T10:37:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:01 smithi161 ceph-mon[91579]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:37:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:01 smithi188 ceph-mon[92425]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:37:01.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:01 smithi101 ceph-mon[85126]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2022-07-03T10:37:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:03 smithi161 ceph-mon[91579]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:03 smithi188 ceph-mon[92425]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:03.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:03 smithi101 ceph-mon[85126]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:05 smithi161 ceph-mon[91579]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:05 smithi188 ceph-mon[92425]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:05.749 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:05 smithi101 ceph-mon[85126]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 682 B/s wr, 2 op/s 2022-07-03T10:37:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:07 smithi161 ceph-mon[91579]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2022-07-03T10:37:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:07 smithi188 ceph-mon[92425]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2022-07-03T10:37:07.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:07 smithi101 ceph-mon[85126]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2022-07-03T10:37:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:09 smithi161 ceph-mon[91579]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:37:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:09 smithi188 ceph-mon[92425]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:37:09.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:09 smithi101 ceph-mon[85126]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:37:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:11 smithi161 ceph-mon[91579]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:11 smithi188 ceph-mon[92425]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:11.748 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:11 smithi101 ceph-mon[85126]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:13 smithi161 ceph-mon[91579]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:13 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:37:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:13 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:37:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:13 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:37:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:13 smithi188 ceph-mon[92425]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:13 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:37:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:13 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:37:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:13 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:37:13.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:13 smithi101 ceph-mon[85126]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:13 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:37:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:13 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:37:13.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:13 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:37:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:15 smithi161 ceph-mon[91579]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:15 smithi188 ceph-mon[92425]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:15.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:15 smithi101 ceph-mon[85126]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:16 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:16 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:16 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:16 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:16 smithi188 ceph-mon[92425]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:17.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:16 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.243 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:16 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:16 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.244 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:16 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.245 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:16 smithi101 ceph-mon[85126]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:16 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:16 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:16 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:16 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:16 smithi161 ceph-mon[91579]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:19.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:19 smithi101 ceph-mon[85126]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:19.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:19.494 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:19 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:19 smithi161 ceph-mon[91579]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:19 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:19 smithi188 ceph-mon[92425]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:19 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:21.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:21 smithi101 ceph-mon[85126]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:21.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:21 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:21 smithi161 ceph-mon[91579]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:21 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:21 smithi188 ceph-mon[92425]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:37:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:21 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' 2022-07-03T10:37:23.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:23 smithi101 ceph-mon[85126]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:23 smithi161 ceph-mon[91579]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:23 smithi188 ceph-mon[92425]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:24.751 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:37:25.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.150+0000 7ff385b9d700 1 -- 172.21.15.101:0/3961957545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fe670 msgr2=0x7ff3800feac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.150+0000 7ff385b9d700 1 --2- 172.21.15.101:0/3961957545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fe670 0x7ff3800feac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009900 tx=0x7ff36c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 -- 172.21.15.101:0/3961957545 shutdown_connections 2022-07-03T10:37:25.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 --2- 172.21.15.101:0/3961957545 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800fd3f0 0x7ff3800fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 --2- 172.21.15.101:0/3961957545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800ff000 0x7ff3801058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 --2- 172.21.15.101:0/3961957545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fe670 0x7ff3800feac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 -- 172.21.15.101:0/3961957545 >> 172.21.15.101:0/3961957545 conn(0x7ff3800f8b40 msgr2=0x7ff3800fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 -- 172.21.15.101:0/3961957545 shutdown_connections 2022-07-03T10:37:25.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.151+0000 7ff385b9d700 1 -- 172.21.15.101:0/3961957545 wait complete. 2022-07-03T10:37:25.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.152+0000 7ff385b9d700 1 Processor -- start 2022-07-03T10:37:25.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 -- start start 2022-07-03T10:37:25.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800fe670 0x7ff380110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800ff000 0x7ff380117bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380118140 con 0x7ff3800ff000 2022-07-03T10:37:25.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380118280 con 0x7ff3800fe670 2022-07-03T10:37:25.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.153+0000 7ff385b9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380118440 con 0x7ff3800fd3f0 2022-07-03T10:37:25.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42340/0 (socket says 172.21.15.101:42340) 2022-07-03T10:37:25.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 -- 172.21.15.101:0/2101716527 learned_addr learned my addr 172.21.15.101:0/2101716527 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:25.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff37ffff700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800fe670 0x7ff380110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff38539c700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800ff000 0x7ff380117bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 -- 172.21.15.101:0/2101716527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800fe670 msgr2=0x7ff380110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800fe670 0x7ff380110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 -- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800ff000 msgr2=0x7ff380117bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800ff000 0x7ff380117bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.154+0000 7ff384b9b700 1 -- 172.21.15.101:0/2101716527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c009440 con 0x7ff3800fd3f0 2022-07-03T10:37:25.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.155+0000 7ff384b9b700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff37400d7e0 tx=0x7ff37400b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.155+0000 7ff37d7fa700 1 -- 172.21.15.101:0/2101716527 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374004020 con 0x7ff3800fd3f0 2022-07-03T10:37:25.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.155+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff380118720 con 0x7ff3800fd3f0 2022-07-03T10:37:25.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.155+0000 7ff37d7fa700 1 -- 172.21.15.101:0/2101716527 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff374004180 con 0x7ff3800fd3f0 2022-07-03T10:37:25.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.155+0000 7ff37d7fa700 1 -- 172.21.15.101:0/2101716527 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3740039d0 con 0x7ff3800fd3f0 2022-07-03T10:37:25.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.156+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380118d00 con 0x7ff3800fd3f0 2022-07-03T10:37:25.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.161+0000 7ff37d7fa700 1 -- 172.21.15.101:0/2101716527 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7ff37400c040 con 0x7ff3800fd3f0 2022-07-03T10:37:25.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.163+0000 7ff39ac97b80 1 Processor -- start 2022-07-03T10:37:25.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.163+0000 7ff37d7fa700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 0x7ff3700991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.163+0000 7ff37d7fa700 1 -- 172.21.15.101:0/2101716527 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff374016030 con 0x7ff3800fd3f0 2022-07-03T10:37:25.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.164+0000 7ff38539c700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 0x7ff3700991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.164+0000 7ff39ac97b80 1 -- start start 2022-07-03T10:37:25.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689da540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689daed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568688bd210 con 0x5568689da0d0 2022-07-03T10:37:25.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x556868926d00 con 0x5568689d9780 2022-07-03T10:37:25.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x556868920500 con 0x5568689daa80 2022-07-03T10:37:25.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff38539c700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 0x7ff3700991d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff37800ae70 tx=0x7ff378005f90 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff37ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689da540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff37ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689da540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:43770/0 (socket says 172.21.15.101:43770) 2022-07-03T10:37:25.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff37ffff700 1 -- 172.21.15.101:0/4281358905 learned_addr learned my addr 172.21.15.101:0/4281358905 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:25.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.165+0000 7ff384b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689daed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 -- 172.21.15.101:0/4281358905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 msgr2=0x5568689d9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689d9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 -- 172.21.15.101:0/4281358905 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 msgr2=0x5568689da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689da540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 -- 172.21.15.101:0/4281358905 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568687a93b0 con 0x5568689daa80 2022-07-03T10:37:25.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.166+0000 7ff38539c700 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689daed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff37800fee0 tx=0x7ff378016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b8e381dbddadd2 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.167+0000 7ff3621fe700 1 -- 172.21.15.101:0/4281358905 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378011e70 con 0x5568689daa80 2022-07-03T10:37:25.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.167+0000 7ff3621fe700 1 -- 172.21.15.101:0/4281358905 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff37801e900 con 0x5568689daa80 2022-07-03T10:37:25.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 -- 172.21.15.101:0/4281358905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 msgr2=0x5568689daed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689daed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff37800fee0 tx=0x7ff378016000 comp rx=0 tx=0).stop 2022-07-03T10:37:25.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 -- 172.21.15.101:0/4281358905 shutdown_connections 2022-07-03T10:37:25.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689daed0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689d9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/4281358905 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689da540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 -- 172.21.15.101:0/4281358905 >> 172.21.15.101:0/4281358905 conn(0x5568689c6800 msgr2=0x5568689cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 -- 172.21.15.101:0/4281358905 shutdown_connections 2022-07-03T10:37:25.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.168+0000 7ff39ac97b80 1 -- 172.21.15.101:0/4281358905 wait complete. 2022-07-03T10:37:25.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.170+0000 7ff39ac97b80 1 -- start start 2022-07-03T10:37:25.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.170+0000 7ff39ac97b80 1 Processor -- start 2022-07-03T10:37:25.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55686880ee90 con 0x5568689da0d0 2022-07-03T10:37:25.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689ea530 con 0x5568689d9780 2022-07-03T10:37:25.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689ea6f0 con 0x5568689daa80 2022-07-03T10:37:25.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff39ac97b80 10 client.0 _ll_register_callbacks cb 0x5568688cc1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:25.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.171+0000 7ff38539c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42372/0 (socket says 172.21.15.101:42372) 2022-07-03T10:37:25.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 -- 172.21.15.101:0/1622240289 learned_addr learned my addr 172.21.15.101:0/1622240289 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:25.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff37ffff700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 -- 172.21.15.101:0/1622240289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 msgr2=0x5568689e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 -- 172.21.15.101:0/1622240289 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 msgr2=0x5568689e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff38539c700 1 -- 172.21.15.101:0/1622240289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37800fb40 con 0x5568689daa80 2022-07-03T10:37:25.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.172+0000 7ff37ffff700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:25.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.173+0000 7ff38539c700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff37800f920 tx=0x7ff378016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2712623eb3970cf1 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.173+0000 7ff35b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:37:25.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.173+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37802d070 con 0x5568689daa80 2022-07-03T10:37:25.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.173+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff37801e730 con 0x5568689daa80 2022-07-03T10:37:25.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.173+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568689c3fc0 con 0x5568689daa80 2022-07-03T10:37:25.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.174+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff37801ec80 con 0x5568689daa80 2022-07-03T10:37:25.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.174+0000 7ff39ac97b80 10 client.25253 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:25.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.174+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556868755b60 con 0x5568689daa80 2022-07-03T10:37:25.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.174+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5568689c3fc0 con 0x5568689daa80 2022-07-03T10:37:25.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.174+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7ff378039070 con 0x5568689daa80 2022-07-03T10:37:25.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 10 client.25253 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:25.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 10 client.25253 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:25.190 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 20 client.25253 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:25.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689ee870 0x5568689f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 4 client.25253 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:25.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556868919a70 con 0x5568689ee870 2022-07-03T10:37:25.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.175+0000 7ff384b9b700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689ee870 0x5568689f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.176+0000 7ff35b7fe700 10 client.25253 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:25.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.176+0000 7ff384b9b700 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689ee870 0x5568689f0c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de9130a421dfb1d7 server_cookie=612e7e0ee1f3879e in_seq=0 out_seq=0 2022-07-03T10:37:25.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff35b7fe700 1 -- 172.21.15.101:0/1622240289 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x556868919a70 con 0x5568689ee870 2022-07-03T10:37:25.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff35b7fe700 10 client.25253 handle_command_reply: tid=0 2022-07-03T10:37:25.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 client.25253 shutdown 2022-07-03T10:37:25.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 msgr2=0x5568689e9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff37800f920 tx=0x7ff378016000 comp rx=0 tx=0).stop 2022-07-03T10:37:25.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 shutdown_connections 2022-07-03T10:37:25.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689ee870 0x5568689f0c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e9f80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/1622240289 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 >> 172.21.15.101:0/1622240289 conn(0x5568689c6800 msgr2=0x5568689ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.177+0000 7ff35b7fe700 0 client.25253 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:25.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.178+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 shutdown_connections 2022-07-03T10:37:25.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.178+0000 7ff39ac97b80 1 -- 172.21.15.101:0/1622240289 wait complete. 2022-07-03T10:37:25.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.178+0000 7ff39ac97b80 20 client.25253 trim_cache size 0 max 16384 2022-07-03T10:37:25.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 Processor -- start 2022-07-03T10:37:25.216 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 -- start start 2022-07-03T10:37:25.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.217 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689e1f90 con 0x5568689da0d0 2022-07-03T10:37:25.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689c9ea0 con 0x5568689d9780 2022-07-03T10:37:25.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.210+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689f5840 con 0x5568689daa80 2022-07-03T10:37:25.218 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55184/0 (socket says 172.21.15.101:55184) 2022-07-03T10:37:25.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff37ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 -- 172.21.15.101:0/3680737341 learned_addr learned my addr 172.21.15.101:0/3680737341 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:25.219 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff38539c700 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 -- 172.21.15.101:0/3680737341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 msgr2=0x5568689e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.211+0000 7ff384b9b700 1 -- 172.21.15.101:0/3680737341 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 msgr2=0x5568689e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.220 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff384b9b700 1 -- 172.21.15.101:0/3680737341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556868754030 con 0x5568689d9780 2022-07-03T10:37:25.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff37ffff700 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:25.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff3740906e0 tx=0x7ff3740bf040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a39309458641f391 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff3629ff700 1 -- 172.21.15.101:0/3680737341 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3740bb070 con 0x5568689d9780 2022-07-03T10:37:25.221 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff3629ff700 1 -- 172.21.15.101:0/3680737341 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff374096e40 con 0x5568689d9780 2022-07-03T10:37:25.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.212+0000 7ff3629ff700 1 -- 172.21.15.101:0/3680737341 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374004380 con 0x5568689d9780 2022-07-03T10:37:25.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.213+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3680737341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 msgr2=0x5568689dd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.213+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff3740906e0 tx=0x7ff3740bf040 comp rx=0 tx=0).stop 2022-07-03T10:37:25.222 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.213+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3680737341 shutdown_connections 2022-07-03T10:37:25.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689daa80 0x5568689e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689dd7d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.223 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3680737341 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689da0d0 0x5568689e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3680737341 >> 172.21.15.101:0/3680737341 conn(0x5568689c6800 msgr2=0x5568689ca100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3680737341 shutdown_connections 2022-07-03T10:37:25.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3680737341 wait complete. 2022-07-03T10:37:25.224 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.214+0000 7ff39ac97b80 1 -- start start 2022-07-03T10:37:25.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.215+0000 7ff39ac97b80 1 Processor -- start 2022-07-03T10:37:25.228 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.215+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.215+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 0x5568689e7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff39ac97b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 0x5568689d5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689e78e0 con 0x5568689daa80 2022-07-03T10:37:25.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689d62e0 con 0x5568689d9780 2022-07-03T10:37:25.229 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff39ac97b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568689d6470 con 0x5568689da0d0 2022-07-03T10:37:25.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff39ac97b80 10 client.0 _ll_register_callbacks cb 0x5568688cc1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:25.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff37ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 0x5568689e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff37ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 0x5568689e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42390/0 (socket says 172.21.15.101:42390) 2022-07-03T10:37:25.230 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff37ffff700 1 -- 172.21.15.101:0/3030478707 learned_addr learned my addr 172.21.15.101:0/3030478707 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:25.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff38539c700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 0x5568689d5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff384b9b700 1 -- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 msgr2=0x5568689e7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.231 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 0x5568689e7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.216+0000 7ff384b9b700 1 -- 172.21.15.101:0/3030478707 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 msgr2=0x5568689d5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:25.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 0x5568689d5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.232 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff384b9b700 1 -- 172.21.15.101:0/3030478707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3801061b0 con 0x5568689d9780 2022-07-03T10:37:25.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff38539c700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 0x5568689d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:37:25.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff384b9b700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e6cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff374091000 tx=0x7ff3740bf040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8fe3bc0d2157e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:25.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff35b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:37:25.233 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3740bb040 con 0x5568689d9780 2022-07-03T10:37:25.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7ff374004360 con 0x5568689d9780 2022-07-03T10:37:25.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.217+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff37801e900 con 0x5568689d9780 2022-07-03T10:37:25.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff37408fc40 con 0x5568689d9780 2022-07-03T10:37:25.234 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 10 client.25255 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:25.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7ff380118d00 con 0x5568689d9780 2022-07-03T10:37:25.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff37801e900 con 0x5568689d9780 2022-07-03T10:37:25.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7ff3740969b0 con 0x5568689d9780 2022-07-03T10:37:25.235 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 10 client.25255 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:25.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 10 client.25255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:25.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 20 client.25255 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:25.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689d7b60 0x5568689d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:25.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 4 client.25255 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:37:25.236 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.218+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556868919850 con 0x5568689d7b60 2022-07-03T10:37:25.237 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.219+0000 7ff37ffff700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689d7b60 0x5568689d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:25.237 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.219+0000 7ff37ffff700 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689d7b60 0x5568689d7f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d254d13eb39adfcd server_cookie=24b19aa87d7b9725 in_seq=0 out_seq=0 2022-07-03T10:37:25.237 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.219+0000 7ff35b7fe700 10 client.25255 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:25.237 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff35b7fe700 1 -- 172.21.15.101:0/3030478707 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556868919850 con 0x5568689d7b60 2022-07-03T10:37:25.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff35b7fe700 10 client.25255 handle_command_reply: tid=0 2022-07-03T10:37:25.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 client.25255 shutdown 2022-07-03T10:37:25.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 msgr2=0x5568689e6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.238 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e6cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff374091000 tx=0x7ff3740bf040 comp rx=0 tx=0).stop 2022-07-03T10:37:25.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 shutdown_connections 2022-07-03T10:37:25.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5568689d7b60 0x5568689d7f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5568689daa80 0x5568689d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5568689d9780 0x5568689e6cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.239 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 --2- 172.21.15.101:0/3030478707 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5568689da0d0 0x5568689e7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 >> 172.21.15.101:0/3030478707 conn(0x5568689c6800 msgr2=0x5568689e2940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff35b7fe700 0 client.25255 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:25.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.220+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 shutdown_connections 2022-07-03T10:37:25.240 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.221+0000 7ff39ac97b80 1 -- 172.21.15.101:0/3030478707 wait complete. 2022-07-03T10:37:25.241 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.221+0000 7ff39ac97b80 20 client.25255 trim_cache size 0 max 16384 2022-07-03T10:37:25.241 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:25.243 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "no active scrubs running", 2022-07-03T10:37:25.243 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": {} 2022-07-03T10:37:25.243 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:25.245 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 msgr2=0x7ff3700991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.245 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 0x7ff3700991d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff37800ae70 tx=0x7ff378005f90 comp rx=0 tx=0).stop 2022-07-03T10:37:25.245 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 msgr2=0x7ff380110610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:25.245 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff37400d7e0 tx=0x7ff37400b3a0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 shutdown_connections 2022-07-03T10:37:25.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7ff370096d20 0x7ff3700991d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.246 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff3800ff000 0x7ff380117bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3800fd3f0 0x7ff380110610 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 --2- 172.21.15.101:0/2101716527 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff3800fe670 0x7ff380110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:25.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 >> 172.21.15.101:0/2101716527 conn(0x7ff3800f8b40 msgr2=0x7ff3800faf10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:25.247 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 shutdown_connections 2022-07-03T10:37:25.248 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:25.222+0000 7ff385b9d700 1 -- 172.21.15.101:0/2101716527 wait complete. 2022-07-03T10:37:25.248 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:25 smithi101 ceph-mon[85126]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:25.269 INFO:tasks.cephfs.filesystem:all active scrubs completed 2022-07-03T10:37:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:25 smithi161 ceph-mon[91579]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:25 smithi188 ceph-mon[92425]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:37:26.270 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2022-07-03T10:37:26.271 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2022-07-03T10:37:26.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:37:26.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:26 smithi101 ceph-mon[85126]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:37:26.515 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 -- 172.21.15.101:0/1403627062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80ff000 msgr2=0x7fcbd81058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 --2- 172.21.15.101:0/1403627062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80ff000 0x7fcbd81058c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009bf0 tx=0x7fcbd0009460 comp rx=0 tx=0).stop 2022-07-03T10:37:26.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 -- 172.21.15.101:0/1403627062 shutdown_connections 2022-07-03T10:37:26.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 --2- 172.21.15.101:0/1403627062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80ff000 0x7fcbd81058c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.516 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 --2- 172.21.15.101:0/1403627062 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 0x7fcbd80fd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 --2- 172.21.15.101:0/1403627062 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80fe670 0x7fcbd80feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 -- 172.21.15.101:0/1403627062 >> 172.21.15.101:0/1403627062 conn(0x7fcbd80f8b40 msgr2=0x7fcbd80fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 -- 172.21.15.101:0/1403627062 shutdown_connections 2022-07-03T10:37:26.517 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.513+0000 7fcbde39d700 1 -- 172.21.15.101:0/1403627062 wait complete. 2022-07-03T10:37:26.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.514+0000 7fcbde39d700 1 Processor -- start 2022-07-03T10:37:26.521 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.514+0000 7fcbde39d700 1 -- start start 2022-07-03T10:37:26.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.514+0000 7fcbde39d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 0x7fcbd81103f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbde39d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbde39d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 0x7fcbd81179a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.522 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbde39d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8117f20 con 0x7fcbd80ff000 2022-07-03T10:37:26.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbde39d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8118060 con 0x7fcbd80fd3f0 2022-07-03T10:37:26.523 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbde39d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8118220 con 0x7fcbd80fe670 2022-07-03T10:37:26.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.524 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbddb9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 0x7fcbd81179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42406/0 (socket says 172.21.15.101:42406) 2022-07-03T10:37:26.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/4108065862 learned_addr learned my addr 172.21.15.101:0/4108065862 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.515+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 0x7fcbd81103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.525 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/4108065862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 msgr2=0x7fcbd81103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 0x7fcbd81103f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.526 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 msgr2=0x7fcbd81179a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 0x7fcbd81179a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/4108065862 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd0009700 con 0x7fcbd80fe670 2022-07-03T10:37:26.527 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 0x7fcbd81179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:26.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800ea80 tx=0x7fcbc8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.528 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbcdffb700 1 -- 172.21.15.101:0/4108065862 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8009e10 con 0x7fcbd80fe670 2022-07-03T10:37:26.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd8118500 con 0x7fcbd80fe670 2022-07-03T10:37:26.529 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbcdffb700 1 -- 172.21.15.101:0/4108065862 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcbc8009500 con 0x7fcbd80fe670 2022-07-03T10:37:26.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.516+0000 7fcbcdffb700 1 -- 172.21.15.101:0/4108065862 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8010e20 con 0x7fcbd80fe670 2022-07-03T10:37:26.530 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.517+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd8118ae0 con 0x7fcbd80fe670 2022-07-03T10:37:26.531 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.519+0000 7fcbcdffb700 1 -- 172.21.15.101:0/4108065862 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fcbc800c040 con 0x7fcbd80fe670 2022-07-03T10:37:26.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.521+0000 7fcbcdffb700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 0x7fcbc40991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.521+0000 7fcbf35fcb80 1 Processor -- start 2022-07-03T10:37:26.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.521+0000 7fcbcdffb700 1 -- 172.21.15.101:0/4108065862 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fcbc8014070 con 0x7fcbd80fe670 2022-07-03T10:37:26.569 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.521+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 0x7fcbc40991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.521+0000 7fcbf35fcb80 1 -- start start 2022-07-03T10:37:26.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 0x564876e93400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.570 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e94740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876d7f290 con 0x564876e92ff0 2022-07-03T10:37:26.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876de8d80 con 0x564876e93940 2022-07-03T10:37:26.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876de2580 con 0x564876e942f0 2022-07-03T10:37:26.571 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55216/0 (socket says 172.21.15.101:55216) 2022-07-03T10:37:26.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/1061233250 learned_addr learned my addr 172.21.15.101:0/1061233250 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 0x564876e93400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.572 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.522+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e94740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 0x7fcbc40991d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80fe4d0 tx=0x7fcbd40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/1061233250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 msgr2=0x564876e94740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e94740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.573 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/1061233250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 msgr2=0x564876e93400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 0x564876e93400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/1061233250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564876c6b430 con 0x564876e93940 2022-07-03T10:37:26.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 0x564876e93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:26.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80093d0 tx=0x7fcbc80bd670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3ee95a1998cb188 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.574 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbba9fe700 1 -- 172.21.15.101:0/1061233250 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc808cbb0 con 0x564876e93940 2022-07-03T10:37:26.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbba9fe700 1 -- 172.21.15.101:0/1061233250 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcbc8090f70 con 0x564876e93940 2022-07-03T10:37:26.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.523+0000 7fcbba9fe700 1 -- 172.21.15.101:0/1061233250 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8095330 con 0x564876e93940 2022-07-03T10:37:26.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1061233250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 msgr2=0x564876e93db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.575 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80093d0 tx=0x7fcbc80bd670 comp rx=0 tx=0).stop 2022-07-03T10:37:26.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1061233250 shutdown_connections 2022-07-03T10:37:26.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e92ff0 0x564876e93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e94740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.576 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1061233250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876e93db0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1061233250 >> 172.21.15.101:0/1061233250 conn(0x564876e88880 msgr2=0x564876e8d830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1061233250 shutdown_connections 2022-07-03T10:37:26.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.524+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1061233250 wait complete. 2022-07-03T10:37:26.577 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.525+0000 7fcbf35fcb80 1 -- start start 2022-07-03T10:37:26.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.525+0000 7fcbf35fcb80 1 Processor -- start 2022-07-03T10:37:26.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.525+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.525+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.578 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.525+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876ea3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876cd0f10 con 0x564876e93940 2022-07-03T10:37:26.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876ea4100 con 0x564876e92ff0 2022-07-03T10:37:26.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876ea42c0 con 0x564876e942f0 2022-07-03T10:37:26.579 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbf35fcb80 10 client.0 _ll_register_callbacks cb 0x564876d8e270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:26.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:43824/0 (socket says 172.21.15.101:43824) 2022-07-03T10:37:26.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/599409083 learned_addr learned my addr 172.21.15.101:0/599409083 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876ea3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.580 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 msgr2=0x564876ea3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876ea3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/599409083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 msgr2=0x564876e9c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.581 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.526+0000 7fcbdcb9a700 1 -- 172.21.15.101:0/599409083 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd8118ae0 con 0x564876e93940 2022-07-03T10:37:26.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8090520 tx=0x7fcbc80bd670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f928614baf79d5ae server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.582 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbb3fff700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:26.583 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8090870 con 0x564876e93940 2022-07-03T10:37:26.583 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564876e86040 con 0x564876e93940 2022-07-03T10:37:26.583 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fcbc8096f60 con 0x564876e93940 2022-07-03T10:37:26.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbc80952f0 con 0x564876e93940 2022-07-03T10:37:26.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbf35fcb80 10 client.15201 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:26.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564876c17be0 con 0x564876e93940 2022-07-03T10:37:26.584 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.527+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x564876e86040 con 0x564876e93940 2022-07-03T10:37:26.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fcbc808f840 con 0x564876e93940 2022-07-03T10:37:26.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 10 client.15201 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:26.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 10 client.15201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:26.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 20 client.15201 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:26.585 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876ea8250 0x564876eaa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 4 client.15201 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:26.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564876ddbaf0 con 0x564876ea8250 2022-07-03T10:37:26.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.528+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876ea8250 0x564876eaa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.586 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.529+0000 7fcbb3fff700 10 client.15201 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.529+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876ea8250 0x564876eaa660 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8860e53e81539700 server_cookie=e95a571daeb50886 in_seq=0 out_seq=0 2022-07-03T10:37:26.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.529+0000 7fcbb3fff700 1 -- 172.21.15.101:0/599409083 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x564876ddbaf0 con 0x564876ea8250 2022-07-03T10:37:26.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.529+0000 7fcbb3fff700 10 client.15201 handle_command_reply: tid=0 2022-07-03T10:37:26.587 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.529+0000 7fcbf35fcb80 1 client.15201 shutdown 2022-07-03T10:37:26.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 msgr2=0x564876e9cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8090520 tx=0x7fcbc80bd670 comp rx=0 tx=0).stop 2022-07-03T10:37:26.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 shutdown_connections 2022-07-03T10:37:26.588 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876ea8250 0x564876eaa660 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876ea3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/599409083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9cb10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.589 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 >> 172.21.15.101:0/599409083 conn(0x564876e88880 msgr2=0x564876e8c180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbb3fff700 0 client.15201 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 shutdown_connections 2022-07-03T10:37:26.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/599409083 wait complete. 2022-07-03T10:37:26.590 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.530+0000 7fcbf35fcb80 20 client.15201 trim_cache size 0 max 16384 2022-07-03T10:37:26.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.547+0000 7fcbf35fcb80 1 Processor -- start 2022-07-03T10:37:26.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.547+0000 7fcbf35fcb80 1 -- start start 2022-07-03T10:37:26.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.591 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e8cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876e9d2a0 con 0x564876e93940 2022-07-03T10:37:26.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876e8f230 con 0x564876e92ff0 2022-07-03T10:37:26.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876e8bef0 con 0x564876e942f0 2022-07-03T10:37:26.592 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbdcb9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbdd39b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbddb9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e8cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbdd39b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55230/0 (socket says 172.21.15.101:55230) 2022-07-03T10:37:26.593 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.548+0000 7fcbdd39b700 1 -- 172.21.15.101:0/1044528563 learned_addr learned my addr 172.21.15.101:0/1044528563 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 -- 172.21.15.101:0/1044528563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 msgr2=0x564876e9a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 -- 172.21.15.101:0/1044528563 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 msgr2=0x564876e9a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.594 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 -- 172.21.15.101:0/1044528563 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564876c160b0 con 0x564876e942f0 2022-07-03T10:37:26.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:26.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e8cf40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00096d0 tx=0x7fcbd001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d713bf07467c4df server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbbb1ff700 1 -- 172.21.15.101:0/1044528563 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd001d070 con 0x564876e942f0 2022-07-03T10:37:26.595 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbbb1ff700 1 -- 172.21.15.101:0/1044528563 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcbd001e040 con 0x564876e942f0 2022-07-03T10:37:26.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.549+0000 7fcbbb1ff700 1 -- 172.21.15.101:0/1044528563 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000bc70 con 0x564876e942f0 2022-07-03T10:37:26.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1044528563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 msgr2=0x564876e8cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e8cf40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcbd00096d0 tx=0x7fcbd001a040 comp rx=0 tx=0).stop 2022-07-03T10:37:26.596 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1044528563 shutdown_connections 2022-07-03T10:37:26.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e942f0 0x564876e8cf40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e92ff0 0x564876e9a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/1044528563 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e93940 0x564876e9a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.597 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1044528563 >> 172.21.15.101:0/1044528563 conn(0x564876e88880 msgr2=0x564876e9c590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1044528563 shutdown_connections 2022-07-03T10:37:26.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/1044528563 wait complete. 2022-07-03T10:37:26.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.550+0000 7fcbf35fcb80 1 -- start start 2022-07-03T10:37:26.598 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 Processor -- start 2022-07-03T10:37:26.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876ef2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e942f0 0x564876ef9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876efa0e0 con 0x564876e942f0 2022-07-03T10:37:26.599 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876efa250 con 0x564876e93940 2022-07-03T10:37:26.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x564876efa410 con 0x564876e92ff0 2022-07-03T10:37:26.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbf35fcb80 10 client.0 _ll_register_callbacks cb 0x564876d8e270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:26.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbdd39b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.600 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbdd39b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42448/0 (socket says 172.21.15.101:42448) 2022-07-03T10:37:26.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbdd39b700 1 -- 172.21.15.101:0/259231552 learned_addr learned my addr 172.21.15.101:0/259231552 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbddb9c700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e942f0 0x564876ef9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.551+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876ef2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 -- 172.21.15.101:0/259231552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 msgr2=0x564876ef2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.601 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876ef2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 -- 172.21.15.101:0/259231552 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e942f0 msgr2=0x564876ef9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e942f0 0x564876ef9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 -- 172.21.15.101:0/259231552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd4017070 con 0x564876e92ff0 2022-07-03T10:37:26.602 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbdd39b700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcbd402a9c0 tx=0x7fcbd401f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba07a2d7ae9edd3 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbb3fff700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:37:26.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4023040 con 0x564876e92ff0 2022-07-03T10:37:26.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.552+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbc8090f70 con 0x564876e92ff0 2022-07-03T10:37:26.603 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fcbd4030070 con 0x564876e92ff0 2022-07-03T10:37:26.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbd4004230 con 0x564876e92ff0 2022-07-03T10:37:26.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 10 client.25277 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:26.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7fcbd81061b0 con 0x564876e92ff0 2022-07-03T10:37:26.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcbc8090f70 con 0x564876e92ff0 2022-07-03T10:37:26.604 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fcbd402a040 con 0x564876e92ff0 2022-07-03T10:37:26.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 10 client.25277 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:26.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 10 client.25277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:26.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 20 client.25277 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:26.605 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876efbdf0 0x564876efe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 4 client.25277 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2022-07-03T10:37:26.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.553+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564876ddb8d0 con 0x564876efbdf0 2022-07-03T10:37:26.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.554+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876efbdf0 0x564876efe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.606 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.554+0000 7fcbb3fff700 10 client.25277 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.554+0000 7fcbdcb9a700 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876efbdf0 0x564876efe1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41930f99b57f58bb server_cookie=1a76fc4b897fed1f in_seq=0 out_seq=0 2022-07-03T10:37:26.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbb3fff700 1 -- 172.21.15.101:0/259231552 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564876ddb8d0 con 0x564876efbdf0 2022-07-03T10:37:26.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbb3fff700 10 client.25277 handle_command_reply: tid=0 2022-07-03T10:37:26.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 client.25277 shutdown 2022-07-03T10:37:26.607 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 msgr2=0x564876ef2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcbd402a9c0 tx=0x7fcbd401f040 comp rx=0 tx=0).stop 2022-07-03T10:37:26.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 shutdown_connections 2022-07-03T10:37:26.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x564876efbdf0 0x564876efe1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.608 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x564876e942f0 0x564876ef9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x564876e92ff0 0x564876ef2680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbb3fff700 0 client.25277 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 --2- 172.21.15.101:0/259231552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x564876e93940 0x564876ef2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.609 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.555+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 >> 172.21.15.101:0/259231552 conn(0x564876e88880 msgr2=0x564876e9a440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 shutdown_connections 2022-07-03T10:37:26.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbf35fcb80 1 -- 172.21.15.101:0/259231552 wait complete. 2022-07-03T10:37:26.610 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbf35fcb80 20 client.25277 trim_cache size 0 max 16384 2022-07-03T10:37:26.611 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:26.611 INFO:teuthology.orchestra.run.smithi101.stdout: "return_code": 0, 2022-07-03T10:37:26.611 INFO:teuthology.orchestra.run.smithi101.stdout: "scrub_tag": "4c77ba06-700d-4122-930f-70863cf9d1c4", 2022-07-03T10:37:26.611 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "asynchronous" 2022-07-03T10:37:26.612 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:26.613 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 msgr2=0x7fcbc40991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 0x7fcbc40991d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80fe4d0 tx=0x7fcbd40091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 msgr2=0x7fcbd8110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.556+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800ea80 tx=0x7fcbc8005ef0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.614 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 shutdown_connections 2022-07-03T10:37:26.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fcbc4096d20 0x7fcbc40991d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcbd80ff000 0x7fcbd81179a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcbd80fd3f0 0x7fcbd81103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.615 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 --2- 172.21.15.101:0/4108065862 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbd80fe670 0x7fcbd8110930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 >> 172.21.15.101:0/4108065862 conn(0x7fcbd80f8b40 msgr2=0x7fcbd80facf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 shutdown_connections 2022-07-03T10:37:26.616 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.557+0000 7fcbde39d700 1 -- 172.21.15.101:0/4108065862 wait complete. 2022-07-03T10:37:26.617 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:37:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:37:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:26 smithi161 ceph-mon[91579]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:37:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-07-03T10:37:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:26 smithi188 ceph-mon[92425]: from='mgr.14154 172.21.15.101:0/3645044299' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:37:26.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.855+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2400502653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 msgr2=0x7fa6c40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.858 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.855+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2400502653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c40712c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009900 tx=0x7fa6b40091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2400502653 shutdown_connections 2022-07-03T10:37:26.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2400502653 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4100ef0 0x7fa6c410bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2400502653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4072430 0x7fa6c4070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.859 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2400502653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c40712c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2400502653 >> 172.21.15.101:0/2400502653 conn(0x7fa6c40f8b40 msgr2=0x7fa6c40fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2400502653 shutdown_connections 2022-07-03T10:37:26.860 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.856+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2400502653 wait complete. 2022-07-03T10:37:26.862 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.857+0000 7fa6cb59d700 1 Processor -- start 2022-07-03T10:37:26.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.857+0000 7fa6cb59d700 1 -- start start 2022-07-03T10:37:26.863 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4072430 0x7fa6c4114f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.864 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4100ef0 0x7fa6c411bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c411c550 con 0x7fa6c4072430 2022-07-03T10:37:26.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c411c690 con 0x7fa6c4100ef0 2022-07-03T10:37:26.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cb59d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c411c850 con 0x7fa6c4070e50 2022-07-03T10:37:26.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.865 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42474/0 (socket says 172.21.15.101:42474) 2022-07-03T10:37:26.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2755825138 learned_addr learned my addr 172.21.15.101:0/2755825138 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4100ef0 0x7fa6c411bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.866 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.858+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4072430 0x7fa6c4114f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2755825138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4100ef0 msgr2=0x7fa6c411bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4100ef0 0x7fa6c411bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4072430 msgr2=0x7fa6c4114f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.867 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4072430 0x7fa6c4114f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2755825138 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c0009410 con 0x7fa6c4070e50 2022-07-03T10:37:26.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000ea80 tx=0x7fa6c0005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6baffd700 1 -- 172.21.15.101:0/2755825138 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0009690 con 0x7fa6c4070e50 2022-07-03T10:37:26.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b4009440 con 0x7fa6c4070e50 2022-07-03T10:37:26.868 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6baffd700 1 -- 172.21.15.101:0/2755825138 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa6c000c970 con 0x7fa6c4070e50 2022-07-03T10:37:26.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.859+0000 7fa6baffd700 1 -- 172.21.15.101:0/2755825138 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0003b40 con 0x7fa6c4070e50 2022-07-03T10:37:26.869 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.860+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c411cf30 con 0x7fa6c4070e50 2022-07-03T10:37:26.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.863+0000 7fa6baffd700 1 -- 172.21.15.101:0/2755825138 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7fa6c000c040 con 0x7fa6c4070e50 2022-07-03T10:37:26.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.865+0000 7fa6baffd700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 0x7fa6b00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.879 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.865+0000 7fa6baffd700 1 -- 172.21.15.101:0/2755825138 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa6c0014070 con 0x7fa6c4070e50 2022-07-03T10:37:26.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.865+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 0x7fa6b00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.866+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 0x7fa6b00991d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40092d0 tx=0x7fa6b40091e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.866+0000 7fa6e06eab80 1 Processor -- start 2022-07-03T10:37:26.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.866+0000 7fa6e06eab80 1 -- start start 2022-07-03T10:37:26.880 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745914380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5627459148c0 0x562745914d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 0x5627459156c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745800210 con 0x5627459148c0 2022-07-03T10:37:26.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745869d00 con 0x562745913f70 2022-07-03T10:37:26.881 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745863500 con 0x562745915270 2022-07-03T10:37:26.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745914380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6cad9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 0x5627459156c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745914380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55268/0 (socket says 172.21.15.101:55268) 2022-07-03T10:37:26.882 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6ca59b700 1 -- 172.21.15.101:0/693593176 learned_addr learned my addr 172.21.15.101:0/693593176 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.867+0000 7fa6c9d9a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5627459148c0 0x562745914d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 -- 172.21.15.101:0/693593176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 msgr2=0x562745914380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745914380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 -- 172.21.15.101:0/693593176 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5627459148c0 msgr2=0x562745914d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.883 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5627459148c0 0x562745914d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 -- 172.21.15.101:0/693593176 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627456ec3b0 con 0x562745915270 2022-07-03T10:37:26.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 0x5627459156c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005fd0 tx=0x7fa6bc009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a1144767f0822cc server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6a7fff700 1 -- 172.21.15.101:0/693593176 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc004a50 con 0x562745915270 2022-07-03T10:37:26.884 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6a7fff700 1 -- 172.21.15.101:0/693593176 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa6bc003d30 con 0x562745915270 2022-07-03T10:37:26.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.868+0000 7fa6a7fff700 1 -- 172.21.15.101:0/693593176 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc004650 con 0x562745915270 2022-07-03T10:37:26.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 -- 172.21.15.101:0/693593176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 msgr2=0x5627459156c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 0x5627459156c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005fd0 tx=0x7fa6bc009460 comp rx=0 tx=0).stop 2022-07-03T10:37:26.885 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 -- 172.21.15.101:0/693593176 shutdown_connections 2022-07-03T10:37:26.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745915270 0x5627459156c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745914380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/693593176 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5627459148c0 0x562745914d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.886 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.869+0000 7fa6e06eab80 1 -- 172.21.15.101:0/693593176 >> 172.21.15.101:0/693593176 conn(0x562745909800 msgr2=0x56274590e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.870+0000 7fa6e06eab80 1 -- 172.21.15.101:0/693593176 shutdown_connections 2022-07-03T10:37:26.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.870+0000 7fa6e06eab80 1 -- 172.21.15.101:0/693593176 wait complete. 2022-07-03T10:37:26.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.870+0000 7fa6e06eab80 1 -- start start 2022-07-03T10:37:26.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 Processor -- start 2022-07-03T10:37:26.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274591da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x562745924ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745751e90 con 0x562745915270 2022-07-03T10:37:26.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745925060 con 0x5627459148c0 2022-07-03T10:37:26.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.871+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x562745925220 con 0x562745913f70 2022-07-03T10:37:26.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6e06eab80 10 client.0 _ll_register_callbacks cb 0x56274580f1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:26.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42492/0 (socket says 172.21.15.101:42492) 2022-07-03T10:37:26.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 -- 172.21.15.101:0/3677663993 learned_addr learned my addr 172.21.15.101:0/3677663993 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274591da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x562745924ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 -- 172.21.15.101:0/3677663993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 msgr2=0x56274591da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274591da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 -- 172.21.15.101:0/3677663993 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 msgr2=0x562745924ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x562745924ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.872+0000 7fa6ca59b700 1 -- 172.21.15.101:0/3677663993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc009c60 con 0x562745913f70 2022-07-03T10:37:26.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008d750 tx=0x7fa6c0095040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70aeb309f852a2d server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.188:3300/0 2022-07-03T10:37:26.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0090d60 con 0x562745913f70 2022-07-03T10:37:26.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mon.2 v2:172.21.15.188:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa6c00973d0 con 0x562745913f70 2022-07-03T10:37:26.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562745906fc0 con 0x562745913f70 2022-07-03T10:37:26.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c00bd430 con 0x562745913f70 2022-07-03T10:37:26.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6e06eab80 10 client.25289 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:26.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562745698b60 con 0x562745913f70 2022-07-03T10:37:26.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mon.2 v2:172.21.15.188:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x562745906fc0 con 0x562745913f70 2022-07-03T10:37:26.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.873+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mon.2 v2:172.21.15.188:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fa6c0096f40 con 0x562745913f70 2022-07-03T10:37:26.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 10 client.25289 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:26.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 10 client.25289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:26.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 20 client.25289 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:26.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5627459295c0 0x56274592b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 4 client.25289 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:26.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56274585ca70 con 0x5627459295c0 2022-07-03T10:37:26.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.874+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5627459295c0 0x56274592b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.875+0000 7fa6a57fa700 10 client.25289 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.875+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5627459295c0 0x56274592b9d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f34d42e6dd97231 server_cookie=eb2237de7a80d26 in_seq=0 out_seq=0 2022-07-03T10:37:26.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6a57fa700 1 -- 172.21.15.101:0/3677663993 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x56274585ca70 con 0x5627459295c0 2022-07-03T10:37:26.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6a57fa700 10 client.25289 handle_command_reply: tid=0 2022-07-03T10:37:26.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 client.25289 shutdown 2022-07-03T10:37:26.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 msgr2=0x56274591d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008d750 tx=0x7fa6c0095040 comp rx=0 tx=0).stop 2022-07-03T10:37:26.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 shutdown_connections 2022-07-03T10:37:26.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x5627459295c0 0x56274592b9d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x562745924ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274591d530 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6a57fa700 0 client.25289 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/3677663993 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274591da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 >> 172.21.15.101:0/3677663993 conn(0x562745909800 msgr2=0x56274590d100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 shutdown_connections 2022-07-03T10:37:26.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.876+0000 7fa6e06eab80 1 -- 172.21.15.101:0/3677663993 wait complete. 2022-07-03T10:37:26.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.877+0000 7fa6e06eab80 20 client.25289 trim_cache size 0 max 16384 2022-07-03T10:37:26.906 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.894+0000 7fa6e06eab80 1 Processor -- start 2022-07-03T10:37:26.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 -- start start 2022-07-03T10:37:26.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274590ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274591cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.907 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627459305b0 con 0x562745915270 2022-07-03T10:37:26.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274591c200 con 0x5627459148c0 2022-07-03T10:37:26.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274590d8d0 con 0x562745913f70 2022-07-03T10:37:26.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42494/0 (socket says 172.21.15.101:42494) 2022-07-03T10:37:26.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6ca59b700 1 -- 172.21.15.101:0/46724083 learned_addr learned my addr 172.21.15.101:0/46724083 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274591cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.895+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274590ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 -- 172.21.15.101:0/46724083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 msgr2=0x56274590ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274590ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 -- 172.21.15.101:0/46724083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 msgr2=0x56274591cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274591cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 -- 172.21.15.101:0/46724083 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562745697030 con 0x562745913f70 2022-07-03T10:37:26.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6cad9c700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274591cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:26.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008f030 tx=0x7fa6c0095040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15a95cf7ed881f8f server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6b89ff700 1 -- 172.21.15.101:0/46724083 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00973d0 con 0x562745913f70 2022-07-03T10:37:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6b89ff700 1 -- 172.21.15.101:0/46724083 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa6c0090ce0 con 0x562745913f70 2022-07-03T10:37:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.896+0000 7fa6b89ff700 1 -- 172.21.15.101:0/46724083 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00911b0 con 0x562745913f70 2022-07-03T10:37:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 -- 172.21.15.101:0/46724083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 msgr2=0x56274590e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008f030 tx=0x7fa6c0095040 comp rx=0 tx=0).stop 2022-07-03T10:37:26.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 -- 172.21.15.101:0/46724083 shutdown_connections 2022-07-03T10:37:26.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274591cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x562745913f70 0x56274590e700 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/46724083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5627459148c0 0x56274590ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 -- 172.21.15.101:0/46724083 >> 172.21.15.101:0/46724083 conn(0x562745909800 msgr2=0x56274591b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 -- 172.21.15.101:0/46724083 shutdown_connections 2022-07-03T10:37:26.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.897+0000 7fa6e06eab80 1 -- 172.21.15.101:0/46724083 wait complete. 2022-07-03T10:37:26.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 -- start start 2022-07-03T10:37:26.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 Processor -- start 2022-07-03T10:37:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274597a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274597a720 con 0x562745915270 2022-07-03T10:37:26.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274597a890 con 0x562745913f70 2022-07-03T10:37:26.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56274597aa50 con 0x5627459148c0 2022-07-03T10:37:26.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.898+0000 7fa6e06eab80 10 client.0 _ll_register_callbacks cb 0x56274580f1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:26.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6c9d9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6cad9c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274597a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6c9d9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:42510/0 (socket says 172.21.15.101:42510) 2022-07-03T10:37:26.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:55294/0 (socket says 172.21.15.101:55294) 2022-07-03T10:37:26.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6c9d9a700 1 -- 172.21.15.101:0/2600619802 learned_addr learned my addr 172.21.15.101:0/2600619802 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:26.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 msgr2=0x5627459734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2600619802 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 msgr2=0x56274597a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:26.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274597a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 -- 172.21.15.101:0/2600619802 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562745698b60 con 0x562745913f70 2022-07-03T10:37:26.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:26.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6ca59b700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008d750 tx=0x7fa6c0095040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60d746aa27981ed server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:26.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.899+0000 7fa6a57fa700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:37:26.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00973d0 con 0x562745913f70 2022-07-03T10:37:26.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7fa6c008f080 con 0x562745913f70 2022-07-03T10:37:26.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6bc003d30 con 0x562745913f70 2022-07-03T10:37:26.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c0091150 con 0x562745913f70 2022-07-03T10:37:26.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 10 client.25291 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:26.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562745765800 con 0x562745913f70 2022-07-03T10:37:26.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa6c009b080 con 0x562745913f70 2022-07-03T10:37:26.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fa6c0005290 con 0x562745913f70 2022-07-03T10:37:26.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 10 client.25291 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:26.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 10 client.25291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:26.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 20 client.25291 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:26.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56274597bc70 0x56274597e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:26.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 4 client.25291 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:37:26.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.900+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56274585c850 con 0x56274597bc70 2022-07-03T10:37:26.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.901+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56274597bc70 0x56274597e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:26.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.901+0000 7fa6a57fa700 10 client.25291 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.901+0000 7fa6c9d9a700 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56274597bc70 0x56274597e050 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be6429944a3dd2e9 server_cookie=3f61455e989e6ec6 in_seq=0 out_seq=0 2022-07-03T10:37:26.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6a57fa700 1 -- 172.21.15.101:0/2600619802 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56274585c850 con 0x56274597bc70 2022-07-03T10:37:26.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6a57fa700 10 client.25291 handle_command_reply: tid=0 2022-07-03T10:37:26.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 client.25291 shutdown 2022-07-03T10:37:26.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 msgr2=0x562745972f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa6c008d750 tx=0x7fa6c0095040 comp rx=0 tx=0).stop 2022-07-03T10:37:26.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 shutdown_connections 2022-07-03T10:37:26.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56274597bc70 0x56274597e050 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x562745915270 0x56274597a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562745913f70 0x562745972f90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 --2- 172.21.15.101:0/2600619802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x5627459148c0 0x5627459734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 >> 172.21.15.101:0/2600619802 conn(0x562745909800 msgr2=0x56274590e700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6a57fa700 0 client.25291 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:26.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 shutdown_connections 2022-07-03T10:37:26.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 1 -- 172.21.15.101:0/2600619802 wait complete. 2022-07-03T10:37:26.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.902+0000 7fa6e06eab80 20 client.25291 trim_cache size 0 max 16384 2022-07-03T10:37:26.926 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:26.926 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "IDLE (waiting for more scrubs)", 2022-07-03T10:37:26.926 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": { 2022-07-03T10:37:26.926 INFO:teuthology.orchestra.run.smithi101.stdout: "4c77ba06-700d-4122-930f-70863cf9d1c4": { 2022-07-03T10:37:26.927 INFO:teuthology.orchestra.run.smithi101.stdout: "path": "/", 2022-07-03T10:37:26.927 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "4c77ba06-700d-4122-930f-70863cf9d1c4", 2022-07-03T10:37:26.927 INFO:teuthology.orchestra.run.smithi101.stdout: "options": "recursive,force" 2022-07-03T10:37:26.927 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:37:26.928 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:37:26.928 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:26.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 msgr2=0x7fa6b00991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 0x7fa6b00991d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40092d0 tx=0x7fa6b40091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 msgr2=0x7fa6c4114a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:26.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000ea80 tx=0x7fa6c0005ef0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 shutdown_connections 2022-07-03T10:37:26.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7fa6b0096d20 0x7fa6b00991d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6c4100ef0 0x7fa6c411bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6c4070e50 0x7fa6c4114a20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 --2- 172.21.15.101:0/2755825138 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa6c4072430 0x7fa6c4114f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:26.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 >> 172.21.15.101:0/2755825138 conn(0x7fa6c40f8b40 msgr2=0x7fa6c40faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:26.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 shutdown_connections 2022-07-03T10:37:26.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:26.903+0000 7fa6cb59d700 1 -- 172.21.15.101:0/2755825138 wait complete. 2022-07-03T10:37:26.932 INFO:tasks.cephfs.filesystem:scrub status for tag:4c77ba06-700d-4122-930f-70863cf9d1c4 - {'path': '/', 'tag': '4c77ba06-700d-4122-930f-70863cf9d1c4', 'options': 'recursive,force'} 2022-07-03T10:37:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:27 smithi161 ceph-mon[91579]: pgmap v342: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:27 smithi161 ceph-mon[91579]: scrub queued for path: / 2022-07-03T10:37:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:27 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:27 smithi161 ceph-mon[91579]: scrub summary: active paths [/] 2022-07-03T10:37:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:27 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:27 smithi188 ceph-mon[92425]: pgmap v342: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:27 smithi188 ceph-mon[92425]: scrub queued for path: / 2022-07-03T10:37:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:27 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:27 smithi188 ceph-mon[92425]: scrub summary: active paths [/] 2022-07-03T10:37:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:27 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:27.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:27 smithi101 ceph-mon[85126]: pgmap v342: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:27.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:27 smithi101 ceph-mon[85126]: scrub queued for path: / 2022-07-03T10:37:27.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:27 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:27.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:27 smithi101 ceph-mon[85126]: scrub summary: active paths [/] 2022-07-03T10:37:27.743 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:27 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:28.389 INFO:tasks.workunit.client.0.smithi101.stderr:Updating files: 71% (4833/6793) Updating files: 72% (4891/6793) Updating files: 73% (4959/6793) Updating files: 74% (5027/6793) Updating files: 75% (5095/6793) Updating files: 76% (5163/6793) Updating files: 77% (5231/6793) Updating files: 78% (5299/6793) Updating files: 79% (5367/6793) Updating files: 80% (5435/6793) Updating files: 81% (5503/6793) Updating files: 82% (5571/6793) Updating files: 83% (5639/6793) Updating files: 84% (5707/6793) Updating files: 85% (5775/6793) Updating files: 86% (5842/6793) Updating files: 87% (5910/6793) Updating files: 88% (5978/6793) Updating files: 89% (6046/6793) Updating files: 90% (6114/6793) Updating files: 91% (6182/6793) Updating files: 92% (6250/6793) Updating files: 93% (6318/6793) Updating files: 94% (6386/6793) Updating files: 95% (6454/6793) Updating files: 96% (6522/6793) Updating files: 97% (6590/6793) Updating files: 98% (6658/6793) Updating files: 99% (6726/6793) Updating files: 100% (6793/6793) Updating files: 100% (6793/6793), done. 2022-07-03T10:37:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:28 smithi161 ceph-mon[91579]: scrub summary: idle 2022-07-03T10:37:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:28 smithi188 ceph-mon[92425]: scrub summary: idle 2022-07-03T10:37:28.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:28 smithi101 ceph-mon[85126]: scrub summary: idle 2022-07-03T10:37:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:29 smithi161 ceph-mon[91579]: pgmap v343: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2022-07-03T10:37:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:29 smithi188 ceph-mon[92425]: pgmap v343: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2022-07-03T10:37:29.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:29 smithi101 ceph-mon[85126]: pgmap v343: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2022-07-03T10:37:31.109 INFO:tasks.workunit.client.0.smithi101.stderr:Note: switching to '61829bae536fb4126cfaa87ca353b9576e018e7d'. 2022-07-03T10:37:31.109 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.109 INFO:tasks.workunit.client.0.smithi101.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-07-03T10:37:31.110 INFO:tasks.workunit.client.0.smithi101.stderr:changes and commit them, and you can discard any commits you make in this 2022-07-03T10:37:31.110 INFO:tasks.workunit.client.0.smithi101.stderr:state without impacting any branches by switching back to a branch. 2022-07-03T10:37:31.110 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.110 INFO:tasks.workunit.client.0.smithi101.stderr:If you want to create a new branch to retain commits you create, you may 2022-07-03T10:37:31.111 INFO:tasks.workunit.client.0.smithi101.stderr:do so (now or later) by using -c with the switch command. Example: 2022-07-03T10:37:31.111 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.111 INFO:tasks.workunit.client.0.smithi101.stderr: git switch -c 2022-07-03T10:37:31.111 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.112 INFO:tasks.workunit.client.0.smithi101.stderr:Or undo this operation with: 2022-07-03T10:37:31.112 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.112 INFO:tasks.workunit.client.0.smithi101.stderr: git switch - 2022-07-03T10:37:31.112 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.112 INFO:tasks.workunit.client.0.smithi101.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-07-03T10:37:31.113 INFO:tasks.workunit.client.0.smithi101.stderr: 2022-07-03T10:37:31.113 INFO:tasks.workunit.client.0.smithi101.stderr:HEAD is now at 61829bae536 Merge remote-tracking branch 'ramana/fix-54108' into wip-rishabh-testing-20220702-163127 2022-07-03T10:37:31.158 DEBUG:teuthology.orchestra.run.smithi101:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-07-03T10:37:31.233 INFO:tasks.workunit.client.0.smithi101.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-07-03T10:37:31.238 INFO:tasks.workunit.client.0.smithi101.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-07-03T10:37:31.239 INFO:tasks.workunit.client.0.smithi101.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-07-03T10:37:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:31 smithi161 ceph-mon[91579]: pgmap v344: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:31 smithi188 ceph-mon[92425]: pgmap v344: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:31.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:31 smithi101 ceph-mon[85126]: pgmap v344: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:31.980 INFO:tasks.workunit.client.0.smithi101.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-07-03T10:37:32.084 INFO:tasks.workunit.client.0.smithi101.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-07-03T10:37:32.164 INFO:tasks.workunit.client.0.smithi101.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-07-03T10:37:32.169 INFO:tasks.workunit.client.0.smithi101.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-07-03T10:37:32.170 INFO:tasks.workunit.client.0.smithi101.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-07-03T10:37:32.248 INFO:tasks.workunit.client.0.smithi101.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-07-03T10:37:32.261 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2022-07-03T10:37:32.262 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-07-03T10:37:32.319 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2022-07-03T10:37:32.321 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2022-07-03T10:37:32.321 DEBUG:teuthology.orchestra.run.smithi101:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61829bae536fb4126cfaa87ca353b9576e018e7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2022-07-03T10:37:32.411 INFO:tasks.workunit.client.0.smithi101.stderr:Cloning into 'ffsb'... 2022-07-03T10:37:32.625 INFO:tasks.workunit.client.0.smithi101.stderr:error: file /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack is not a GIT packfile 2022-07-03T10:37:32.626 INFO:tasks.workunit.client.0.smithi101.stderr:error: file /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack is not a GIT packfile 2022-07-03T10:37:32.627 INFO:tasks.workunit.client.0.smithi101.stderr:error: file /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack is not a GIT packfile 2022-07-03T10:37:32.627 INFO:tasks.workunit.client.0.smithi101.stderr:error: file /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack is not a GIT packfile 2022-07-03T10:37:32.627 INFO:tasks.workunit.client.0.smithi101.stderr:error: file /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack is not a GIT packfile 2022-07-03T10:37:32.627 INFO:tasks.workunit.client.0.smithi101.stderr:error: packfile /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/.git/objects/pack/pack-3bf820d5c5ce8196f2871137170c23ed15435091.pack does not match index 2022-07-03T10:37:32.627 INFO:tasks.workunit.client.0.smithi101.stderr:fatal: bad object 07b2d777fdf23ac218fad57ae2b8fcfaf34d295d 2022-07-03T10:37:32.628 INFO:tasks.workunit.client.0.smithi101.stderr:fatal: remote did not send all necessary objects 2022-07-03T10:37:33.059 INFO:tasks.workunit.client.0.smithi101.stderr:Cloning into 'ffsb'... 2022-07-03T10:37:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 128 2022-07-03T10:37:33.417 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2022-07-03T10:37:33.418 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-07-03T10:37:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:33 smithi161 ceph-mon[91579]: pgmap v345: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:33 smithi188 ceph-mon[92425]: pgmap v345: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:33.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:33 smithi101 ceph-mon[85126]: pgmap v345: 97 pgs: 97 active+clean; 646 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 597 B/s wr, 2 op/s 2022-07-03T10:37:35.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-07-03T10:37:35.364 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi101 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61829bae536fb4126cfaa87ca353b9576e018e7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 438, in _run_tests args=args, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-07-03T10:37:35.480 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2e6088d17f364fc6896d3fefc86ebe43 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi101 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61829bae536fb4126cfaa87ca353b9576e018e7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/workunit.py", line 438, in _run_tests args=args, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0' 2022-07-03T10:37:35.484 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2022-07-03T10:37:35.495 DEBUG:tasks.check_counter:Getting stats from a 2022-07-03T10:37:35.496 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:61829bae536fb4126cfaa87ca353b9576e018e7d shell --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- ceph daemon mds.a perf dump 2022-07-03T10:37:35.549 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:35 smithi101 ceph-mon[85126]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 618 B/s wr, 2 op/s 2022-07-03T10:37:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:35 smithi161 ceph-mon[91579]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 618 B/s wr, 2 op/s 2022-07-03T10:37:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:35 smithi188 ceph-mon[92425]: pgmap v346: 97 pgs: 97 active+clean; 646 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 618 B/s wr, 2 op/s 2022-07-03T10:37:36.219 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/mon.a/config 2022-07-03T10:37:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:37 smithi161 ceph-mon[91579]: pgmap v347: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 6 op/s 2022-07-03T10:37:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:37 smithi188 ceph-mon[92425]: pgmap v347: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 6 op/s 2022-07-03T10:37:37.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:37 smithi101 ceph-mon[85126]: pgmap v347: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 6 op/s 2022-07-03T10:37:40.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:40 smithi101 ceph-mon[85126]: pgmap v348: 97 pgs: 97 active+clean; 1.3 MiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 5 op/s 2022-07-03T10:37:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:40 smithi161 ceph-mon[91579]: pgmap v348: 97 pgs: 97 active+clean; 1.3 MiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 5 op/s 2022-07-03T10:37:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:40 smithi188 ceph-mon[92425]: pgmap v348: 97 pgs: 97 active+clean; 1.3 MiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 66 KiB/s wr, 5 op/s 2022-07-03T10:37:42.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:42 smithi101 ceph-mon[85126]: pgmap v349: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 84 KiB/s wr, 6 op/s 2022-07-03T10:37:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:42 smithi161 ceph-mon[91579]: pgmap v349: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 84 KiB/s wr, 6 op/s 2022-07-03T10:37:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:42 smithi188 ceph-mon[92425]: pgmap v349: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 84 KiB/s wr, 6 op/s 2022-07-03T10:37:44.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:44 smithi101 ceph-mon[85126]: pgmap v350: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 85 KiB/s wr, 6 op/s 2022-07-03T10:37:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:44 smithi161 ceph-mon[91579]: pgmap v350: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 85 KiB/s wr, 6 op/s 2022-07-03T10:37:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:44 smithi188 ceph-mon[92425]: pgmap v350: 97 pgs: 97 active+clean; 1.5 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 85 KiB/s wr, 6 op/s 2022-07-03T10:37:45.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:45 smithi101 ceph-mon[85126]: pgmap v351: 97 pgs: 97 active+clean; 2.1 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 136 KiB/s wr, 11 op/s 2022-07-03T10:37:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:45 smithi161 ceph-mon[91579]: pgmap v351: 97 pgs: 97 active+clean; 2.1 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 136 KiB/s wr, 11 op/s 2022-07-03T10:37:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:45 smithi188 ceph-mon[92425]: pgmap v351: 97 pgs: 97 active+clean; 2.1 MiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 136 KiB/s wr, 11 op/s 2022-07-03T10:37:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:47 smithi161 ceph-mon[91579]: pgmap v352: 97 pgs: 97 active+clean; 2.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 166 KiB/s wr, 19 op/s 2022-07-03T10:37:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:47 smithi188 ceph-mon[92425]: pgmap v352: 97 pgs: 97 active+clean; 2.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 166 KiB/s wr, 19 op/s 2022-07-03T10:37:47.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:47 smithi101 ceph-mon[85126]: pgmap v352: 97 pgs: 97 active+clean; 2.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 166 KiB/s wr, 19 op/s 2022-07-03T10:37:50.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:50 smithi101 ceph-mon[85126]: pgmap v353: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 107 KiB/s wr, 17 op/s 2022-07-03T10:37:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:50 smithi161 ceph-mon[91579]: pgmap v353: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 107 KiB/s wr, 17 op/s 2022-07-03T10:37:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:50 smithi188 ceph-mon[92425]: pgmap v353: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 107 KiB/s wr, 17 op/s 2022-07-03T10:37:52.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:52 smithi101 ceph-mon[85126]: pgmap v354: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 109 KiB/s wr, 18 op/s 2022-07-03T10:37:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:52 smithi161 ceph-mon[91579]: pgmap v354: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 109 KiB/s wr, 18 op/s 2022-07-03T10:37:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:52 smithi188 ceph-mon[92425]: pgmap v354: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 109 KiB/s wr, 18 op/s 2022-07-03T10:37:54.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:54 smithi101 ceph-mon[85126]: pgmap v355: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 89 KiB/s wr, 16 op/s 2022-07-03T10:37:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:54 smithi161 ceph-mon[91579]: pgmap v355: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 89 KiB/s wr, 16 op/s 2022-07-03T10:37:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:54 smithi188 ceph-mon[92425]: pgmap v355: 97 pgs: 97 active+clean; 2.4 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 89 KiB/s wr, 16 op/s 2022-07-03T10:37:56.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:56 smithi101 ceph-mon[85126]: pgmap v356: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 18 op/s 2022-07-03T10:37:56.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:56 smithi161 ceph-mon[91579]: pgmap v356: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 18 op/s 2022-07-03T10:37:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:56 smithi188 ceph-mon[92425]: pgmap v356: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 18 op/s 2022-07-03T10:37:56.934 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:37:57.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- 172.21.15.101:0/271587625 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fc4a0 msgr2=0x7f40180fc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.095 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- 172.21.15.101:0/271587625 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fc4a0 0x7f40180fc8b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f400c009900 tx=0x7f400c0091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- 172.21.15.101:0/271587625 shutdown_connections 2022-07-03T10:37:57.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- 172.21.15.101:0/271587625 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fc4a0 0x7f40180fc8b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.096 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- 172.21.15.101:0/271587625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fdf10 0x7f4018104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- 172.21.15.101:0/271587625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fcdf0 0x7f40180fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- 172.21.15.101:0/271587625 >> 172.21.15.101:0/271587625 conn(0x7f40180f7c50 msgr2=0x7f40180fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.097 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- 172.21.15.101:0/271587625 shutdown_connections 2022-07-03T10:37:57.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- 172.21.15.101:0/271587625 wait complete. 2022-07-03T10:37:57.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 Processor -- start 2022-07-03T10:37:57.098 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- start start 2022-07-03T10:37:57.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fcdf0 0x7f4018115710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.099 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.090+0000 7f401fdff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018118230 con 0x7f40180fdf10 2022-07-03T10:37:57.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401fdff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181183a0 con 0x7f40180fcdf0 2022-07-03T10:37:57.100 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401fdff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018118560 con 0x7f40180fc4a0 2022-07-03T10:37:57.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401db9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.101 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401db9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53370/0 (socket says 172.21.15.101:53370) 2022-07-03T10:37:57.102 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401d39a700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fcdf0 0x7f4018115710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:33242/0 (socket says 172.21.15.101:33242) 2022-07-03T10:37:57.103 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401db9b700 1 -- 172.21.15.101:0/3394420722 learned_addr learned my addr 172.21.15.101:0/3394420722 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:57.104 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 -- 172.21.15.101:0/3394420722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 msgr2=0x7f40180661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.105 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 -- 172.21.15.101:0/3394420722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fcdf0 msgr2=0x7f4018115710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.106 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fcdf0 0x7f4018115710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 -- 172.21.15.101:0/3394420722 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c009440 con 0x7f40180fdf10 2022-07-03T10:37:57.107 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401db9b700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:57.108 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401e39c700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f401000ea70 tx=0x7f4010009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f400a7fc700 1 -- 172.21.15.101:0/3394420722 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401000c970 con 0x7f40180fdf10 2022-07-03T10:37:57.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f400a7fc700 1 -- 172.21.15.101:0/3394420722 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4010003880 con 0x7f40180fdf10 2022-07-03T10:37:57.109 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4018118840 con 0x7f40180fdf10 2022-07-03T10:37:57.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.091+0000 7f400a7fc700 1 -- 172.21.15.101:0/3394420722 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010004450 con 0x7f40180fdf10 2022-07-03T10:37:57.110 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.092+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018118dd0 con 0x7f40180fdf10 2022-07-03T10:37:57.131 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.094+0000 7f400a7fc700 1 -- 172.21.15.101:0/3394420722 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f4010020060 con 0x7f40180fdf10 2022-07-03T10:37:57.132 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.095+0000 7f400a7fc700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 0x7f40040991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.095+0000 7f400a7fc700 1 -- 172.21.15.101:0/3394420722 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f4010015070 con 0x7f40180fdf10 2022-07-03T10:37:57.133 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.095+0000 7f401db9b700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 0x7f40040991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.134 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.095+0000 7f4033c65b80 1 Processor -- start 2022-07-03T10:37:57.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 -- start start 2022-07-03T10:37:57.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 0x7f40040991d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f400c0092d0 tx=0x7f400c01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.135 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.136 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c5eba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 0x56048c5ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.137 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c4d7210 con 0x56048c5ec0a0 2022-07-03T10:37:57.138 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c540d00 con 0x56048c5ea810 2022-07-03T10:37:57.139 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f4033c65b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c53a500 con 0x56048c5eb160 2022-07-03T10:37:57.139 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401e39c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 0x56048c5ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.140 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:37184/0 (socket says 172.21.15.101:37184) 2022-07-03T10:37:57.141 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 -- 172.21.15.101:0/4078022141 learned_addr learned my addr 172.21.15.101:0/4078022141 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:57.142 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401d39a700 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c5eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.143 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 -- 172.21.15.101:0/4078022141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 msgr2=0x56048c5eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.143 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c5eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.144 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 -- 172.21.15.101:0/4078022141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 msgr2=0x56048c5ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.145 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 0x56048c5ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401db9b700 1 -- 172.21.15.101:0/4078022141 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56048c3c33b0 con 0x56048c5ea810 2022-07-03T10:37:57.146 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.096+0000 7f401e39c700 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 0x56048c5ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:57.147 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f401db9b700 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f400c005140 tx=0x7f400c023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba1af7e718a9eb82 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f3ffb1fe700 1 -- 172.21.15.101:0/4078022141 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c02b070 con 0x56048c5ea810 2022-07-03T10:37:57.148 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f3ffb1fe700 1 -- 172.21.15.101:0/4078022141 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f400c01e040 con 0x56048c5ea810 2022-07-03T10:37:57.149 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f3ffb1fe700 1 -- 172.21.15.101:0/4078022141 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00fde0 con 0x56048c5ea810 2022-07-03T10:37:57.150 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f4033c65b80 1 -- 172.21.15.101:0/4078022141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 msgr2=0x56048c5eac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.150 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f400c005140 tx=0x7f400c023000 comp rx=0 tx=0).stop 2022-07-03T10:37:57.151 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f4033c65b80 1 -- 172.21.15.101:0/4078022141 shutdown_connections 2022-07-03T10:37:57.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.097+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ec0a0 0x56048c5ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.152 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5eac20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.153 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4078022141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c5eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 -- 172.21.15.101:0/4078022141 >> 172.21.15.101:0/4078022141 conn(0x56048c5e0800 msgr2=0x56048c5e7190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.154 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 -- 172.21.15.101:0/4078022141 shutdown_connections 2022-07-03T10:37:57.155 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 -- 172.21.15.101:0/4078022141 wait complete. 2022-07-03T10:37:57.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 -- start start 2022-07-03T10:37:57.156 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.098+0000 7f4033c65b80 1 Processor -- start 2022-07-03T10:37:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5eb160 0x56048c5f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.158 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c428e90 con 0x56048c5ea810 2022-07-03T10:37:57.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c5fbca0 con 0x56048c5eb160 2022-07-03T10:37:57.159 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c5fbe60 con 0x56048c5ec0a0 2022-07-03T10:37:57.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f4033c65b80 10 client.0 _ll_register_callbacks cb 0x56048c4e61f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:57.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.160 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:33264/0 (socket says 172.21.15.101:33264) 2022-07-03T10:37:57.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 -- 172.21.15.101:0/1554652855 learned_addr learned my addr 172.21.15.101:0/1554652855 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:57.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401d39a700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5eb160 0x56048c5f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.161 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401e39c700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 -- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 msgr2=0x56048c5fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.162 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 -- 172.21.15.101:0/1554652855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5eb160 msgr2=0x56048c5f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5eb160 0x56048c5f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 -- 172.21.15.101:0/1554652855 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4018118dd0 con 0x56048c5ea810 2022-07-03T10:37:57.163 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f401db9b700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f400c0056f0 tx=0x7f400c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d675188d696ceddf server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.099+0000 7f3ff89f9700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c02b040 con 0x56048c5ea810 2022-07-03T10:37:57.164 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56048c5ddfc0 con 0x56048c5ea810 2022-07-03T10:37:57.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f400c00ba10 con 0x56048c5ea810 2022-07-03T10:37:57.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f400c00bde0 con 0x56048c5ea810 2022-07-03T10:37:57.165 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 10 client.15237 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:57.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x7f4018105250 con 0x56048c5ea810 2022-07-03T10:37:57.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f400c048070 con 0x56048c5ea810 2022-07-03T10:37:57.166 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f400c039720 con 0x56048c5ea810 2022-07-03T10:37:57.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 10 client.15237 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:57.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 10 client.15237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:57.167 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 20 client.15237 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:57.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c5ffd00 0x56048c602110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 4 client.15237 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:57.168 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56048c533a70 con 0x56048c5ffd00 2022-07-03T10:37:57.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.100+0000 7f401d39a700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c5ffd00 0x56048c602110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.101+0000 7f3ff89f9700 10 client.15237 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:57.169 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.101+0000 7f401d39a700 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c5ffd00 0x56048c602110 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b15f8b972a0f61c7 server_cookie=a12125b009e0a758 in_seq=0 out_seq=0 2022-07-03T10:37:57.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.101+0000 7f3ff89f9700 1 -- 172.21.15.101:0/1554652855 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x56048c533a70 con 0x56048c5ffd00 2022-07-03T10:37:57.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.101+0000 7f3ff89f9700 10 client.15237 handle_command_reply: tid=0 2022-07-03T10:37:57.170 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.101+0000 7f4033c65b80 1 client.15237 shutdown 2022-07-03T10:37:57.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 msgr2=0x56048c5f4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f400c0056f0 tx=0x7f400c023000 comp rx=0 tx=0).stop 2022-07-03T10:37:57.171 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 shutdown_connections 2022-07-03T10:37:57.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c5ffd00 0x56048c602110 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c5f4170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.172 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f3ff89f9700 0 client.15237 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:57.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 --2- 172.21.15.101:0/1554652855 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5eb160 0x56048c5f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.173 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 >> 172.21.15.101:0/1554652855 conn(0x56048c5e0800 msgr2=0x56048c5e6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 shutdown_connections 2022-07-03T10:37:57.174 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 1 -- 172.21.15.101:0/1554652855 wait complete. 2022-07-03T10:37:57.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.102+0000 7f4033c65b80 20 client.15237 trim_cache size 0 max 16384 2022-07-03T10:37:57.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 Processor -- start 2022-07-03T10:37:57.175 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 -- start start 2022-07-03T10:37:57.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.176 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.177 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c5e6a30 con 0x56048c5eb160 2022-07-03T10:37:57.177 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c5e3e70 con 0x56048c5ea810 2022-07-03T10:37:57.177 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f4033c65b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c5fbad0 con 0x56048c5ec0a0 2022-07-03T10:37:57.178 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f401d39a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.178 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f401d39a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:33266/0 (socket says 172.21.15.101:33266) 2022-07-03T10:37:57.178 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f401d39a700 1 -- 172.21.15.101:0/149515206 learned_addr learned my addr 172.21.15.101:0/149515206 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:57.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f401db9b700 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.118+0000 7f401e39c700 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.179 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 -- 172.21.15.101:0/149515206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 msgr2=0x56048c5f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 -- 172.21.15.101:0/149515206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 msgr2=0x56048c5f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.180 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 -- 172.21.15.101:0/149515206 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56048c36fb60 con 0x56048c5eb160 2022-07-03T10:37:57.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f401d39a700 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f40180fd1c0 tx=0x7f4014013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fff7543d8dc10e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.181 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f3ffb9ff700 1 -- 172.21.15.101:0/149515206 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4014016030 con 0x56048c5eb160 2022-07-03T10:37:57.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f3ffb9ff700 1 -- 172.21.15.101:0/149515206 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4014017040 con 0x56048c5eb160 2022-07-03T10:37:57.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f3ffb9ff700 1 -- 172.21.15.101:0/149515206 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4014003cc0 con 0x56048c5eb160 2022-07-03T10:37:57.182 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f4033c65b80 1 -- 172.21.15.101:0/149515206 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 msgr2=0x56048c5f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.119+0000 7f4033c65b80 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f40180fd1c0 tx=0x7f4014013040 comp rx=0 tx=0).stop 2022-07-03T10:37:57.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- 172.21.15.101:0/149515206 shutdown_connections 2022-07-03T10:37:57.183 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5ec0a0 0x56048c5f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ea810 0x56048c5f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- 172.21.15.101:0/149515206 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5eb160 0x56048c5f2620 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.184 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- 172.21.15.101:0/149515206 >> 172.21.15.101:0/149515206 conn(0x56048c5e0800 msgr2=0x56048c5eecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- 172.21.15.101:0/149515206 shutdown_connections 2022-07-03T10:37:57.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- 172.21.15.101:0/149515206 wait complete. 2022-07-03T10:37:57.185 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- start start 2022-07-03T10:37:57.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 Processor -- start 2022-07-03T10:37:57.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.186 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c649bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 0x56048c6507d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c650d50 con 0x56048c5ea810 2022-07-03T10:37:57.187 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c650ec0 con 0x56048c5ec0a0 2022-07-03T10:37:57.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048c651080 con 0x56048c5eb160 2022-07-03T10:37:57.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f4033c65b80 10 client.0 _ll_register_callbacks cb 0x56048c4e61f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:57.188 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.120+0000 7f401db9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:33272/0 (socket says 172.21.15.101:33272) 2022-07-03T10:37:57.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 -- 172.21.15.101:0/4013764690 learned_addr learned my addr 172.21.15.101:0/4013764690 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:57.189 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401d39a700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c649bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.190 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401e39c700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 0x56048c6507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.190 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 -- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 msgr2=0x56048c649bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.190 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c649bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 -- 172.21.15.101:0/4013764690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 msgr2=0x56048c6507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:57.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 0x56048c6507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.191 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 -- 172.21.15.101:0/4013764690 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401400b8d0 con 0x56048c5ea810 2022-07-03T10:37:57.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401d39a700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c649bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:57.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401e39c700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 0x56048c6507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:57.192 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f401db9b700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f400c039bd0 tx=0x7f400c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31259f2c758b1e server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:57.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f3ff89f9700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:57.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c02b070 con 0x56048c5ea810 2022-07-03T10:37:57.193 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f400c00b940 con 0x56048c5ea810 2022-07-03T10:37:57.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f400c01e040 con 0x56048c5ea810 2022-07-03T10:37:57.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f400c00beb0 con 0x56048c5ea810 2022-07-03T10:37:57.194 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f4033c65b80 10 client.15243 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:57.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.121+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56048c36e030 con 0x56048c5ea810 2022-07-03T10:37:57.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f400c01e040 con 0x56048c5ea810 2022-07-03T10:37:57.195 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f400c04d460 con 0x56048c5ea810 2022-07-03T10:37:57.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 10 client.15243 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:57.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 10 client.15243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:57.196 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 20 client.15243 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:57.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c652610 0x56048c654a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:57.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 4 client.15243 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:37:57.197 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56048c533850 con 0x56048c652610 2022-07-03T10:37:57.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.122+0000 7f401d39a700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c652610 0x56048c654a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:57.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f401d39a700 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c652610 0x56048c654a20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5412d3edefe569b server_cookie=7d9f8d54f573646 in_seq=0 out_seq=0 2022-07-03T10:37:57.198 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f3ff89f9700 10 client.15243 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:57.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f3ff89f9700 1 -- 172.21.15.101:0/4013764690 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56048c533850 con 0x56048c652610 2022-07-03T10:37:57.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f3ff89f9700 10 client.15243 handle_command_reply: tid=0 2022-07-03T10:37:57.199 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 client.15243 shutdown 2022-07-03T10:37:57.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 msgr2=0x56048c649690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f400c039bd0 tx=0x7f400c00b040 comp rx=0 tx=0).stop 2022-07-03T10:37:57.200 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 shutdown_connections 2022-07-03T10:37:57.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x56048c652610 0x56048c654a20 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x56048c5ea810 0x56048c649690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.201 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x56048c5ec0a0 0x56048c6507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f3ff89f9700 0 client.15243 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:57.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 --2- 172.21.15.101:0/4013764690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x56048c5eb160 0x56048c649bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 >> 172.21.15.101:0/4013764690 conn(0x56048c5e0800 msgr2=0x56048c5f20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.202 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 shutdown_connections 2022-07-03T10:37:57.203 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 1 -- 172.21.15.101:0/4013764690 wait complete. 2022-07-03T10:37:57.203 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.123+0000 7f4033c65b80 20 client.15243 trim_cache size 0 max 16384 2022-07-03T10:37:57.204 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:57.204 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "no active scrubs running", 2022-07-03T10:37:57.205 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": {} 2022-07-03T10:37:57.205 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:57.207 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 msgr2=0x7f40040991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 0x7f40040991d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f400c0092d0 tx=0x7f400c01a040 comp rx=0 tx=0).stop 2022-07-03T10:37:57.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 msgr2=0x7f4018117c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:57.208 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f401000ea70 tx=0x7f4010009640 comp rx=0 tx=0).stop 2022-07-03T10:37:57.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 shutdown_connections 2022-07-03T10:37:57.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4004096d20 0x7f40040991d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.209 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f40180fdf10 0x7f4018117c60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40180fc4a0 0x7f40180661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 --2- 172.21.15.101:0/3394420722 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40180fcdf0 0x7f4018115710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:57.210 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 >> 172.21.15.101:0/3394420722 conn(0x7f40180f7c50 msgr2=0x7f40180f9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:57.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 shutdown_connections 2022-07-03T10:37:57.211 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:57.124+0000 7f401fdff700 1 -- 172.21.15.101:0/3394420722 wait complete. 2022-07-03T10:37:57.212 INFO:tasks.cephfs.filesystem:all active scrubs completed 2022-07-03T10:37:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:57 smithi161 ceph-mon[91579]: pgmap v357: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-07-03T10:37:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:57 smithi188 ceph-mon[92425]: pgmap v357: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-07-03T10:37:57.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:57 smithi101 ceph-mon[85126]: pgmap v357: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-07-03T10:37:58.213 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2022-07-03T10:37:58.214 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2022-07-03T10:37:58.374 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.369+0000 7f4edefff700 1 -- 172.21.15.101:0/2550755073 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fdf10 msgr2=0x7f4ed8104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.369+0000 7f4edefff700 1 --2- 172.21.15.101:0/2550755073 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8104a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009ca0 tx=0x7f4ed0009520 comp rx=0 tx=0).stop 2022-07-03T10:37:58.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 -- 172.21.15.101:0/2550755073 shutdown_connections 2022-07-03T10:37:58.375 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 --2- 172.21.15.101:0/2550755073 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8104a10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 --2- 172.21.15.101:0/2550755073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fc4a0 0x7f4ed80fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 --2- 172.21.15.101:0/2550755073 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed80fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.376 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 -- 172.21.15.101:0/2550755073 >> 172.21.15.101:0/2550755073 conn(0x7f4ed80f7c50 msgr2=0x7f4ed80fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 -- 172.21.15.101:0/2550755073 shutdown_connections 2022-07-03T10:37:58.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 -- 172.21.15.101:0/2550755073 wait complete. 2022-07-03T10:37:58.377 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 Processor -- start 2022-07-03T10:37:58.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 -- start start 2022-07-03T10:37:58.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fc4a0 0x7f4ed810e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.378 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.370+0000 7f4edefff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edefff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edefff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8115db0 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.379 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edefff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8115ef0 con 0x7f4ed80fdf10 2022-07-03T10:37:58.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edefff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed81160b0 con 0x7f4ed80fc4a0 2022-07-03T10:37:58.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.380 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42460/0 (socket says 172.21.15.101:42460) 2022-07-03T10:37:58.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 -- 172.21.15.101:0/517862605 learned_addr learned my addr 172.21.15.101:0/517862605 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fc4a0 0x7f4ed810e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.381 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edd59c700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 -- 172.21.15.101:0/517862605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fc4a0 msgr2=0x7f4ed810e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fc4a0 0x7f4ed810e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.382 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 -- 172.21.15.101:0/517862605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fdf10 msgr2=0x7f4ed8115830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8115830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 -- 172.21.15.101:0/517862605 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00097e0 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.383 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e850 tx=0x7f4ec4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed57fa700 1 -- 172.21.15.101:0/517862605 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4005020 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed57fa700 1 -- 172.21.15.101:0/517862605 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4ec4010710 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed8116390 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.384 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.371+0000 7f4ed57fa700 1 -- 172.21.15.101:0/517862605 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4004a10 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.385 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.372+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed81168d0 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.374+0000 7f4ed57fa700 1 -- 172.21.15.101:0/517862605 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f4ec400c040 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.374+0000 7f4ed57fa700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 0x7f4ec80991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.392 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ed57fa700 1 -- 172.21.15.101:0/517862605 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f4ec4014070 con 0x7f4ed80fcdf0 2022-07-03T10:37:58.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 0x7f4ec80991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 0x7f4ec80991d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00b840 tx=0x7f4ecc00b3a0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.393 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 Processor -- start 2022-07-03T10:37:58.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 -- start start 2022-07-03T10:37:58.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0ae890 0x558edd0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.394 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcf9b290 con 0x558edd0af1e0 2022-07-03T10:37:58.395 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd004d80 con 0x558edd0b0120 2022-07-03T10:37:58.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.375+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edcffe580 con 0x558edd0ae890 2022-07-03T10:37:58.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.396 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42468/0 (socket says 172.21.15.101:42468) 2022-07-03T10:37:58.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4147256445 learned_addr learned my addr 172.21.15.101:0/4147256445 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0ae890 0x558edd0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.397 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4edd59c700 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4147256445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0ae890 msgr2=0x558edd0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0ae890 0x558edd0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.398 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4147256445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 msgr2=0x558edd0b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4147256445 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558edce87430 con 0x558edd0af1e0 2022-07-03T10:37:58.399 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400eef0 tx=0x7f4ec408d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854de95d670289 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4eba7fc700 1 -- 172.21.15.101:0/4147256445 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40915e0 con 0x558edd0af1e0 2022-07-03T10:37:58.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4eba7fc700 1 -- 172.21.15.101:0/4147256445 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4ec4090ee0 con 0x558edd0af1e0 2022-07-03T10:37:58.400 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.376+0000 7f4eba7fc700 1 -- 172.21.15.101:0/4147256445 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4010bf0 con 0x558edd0af1e0 2022-07-03T10:37:58.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4147256445 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 msgr2=0x558edd0afb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400eef0 tx=0x7f4ec408d350 comp rx=0 tx=0).stop 2022-07-03T10:37:58.401 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4147256445 shutdown_connections 2022-07-03T10:37:58.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0ae890 0x558edd0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.402 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4147256445 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd0afb00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4147256445 >> 172.21.15.101:0/4147256445 conn(0x558edd0a4880 msgr2=0x558edd0ab210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4147256445 shutdown_connections 2022-07-03T10:37:58.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4147256445 wait complete. 2022-07-03T10:37:58.403 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.377+0000 7f4ef2ec0b80 1 -- start start 2022-07-03T10:37:58.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 Processor -- start 2022-07-03T10:37:58.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd0b8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.404 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edceecf10 con 0x558edd0b0120 2022-07-03T10:37:58.405 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd0bfea0 con 0x558edd0ae890 2022-07-03T10:37:58.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd0c0060 con 0x558edd0af1e0 2022-07-03T10:37:58.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ef2ec0b80 10 client.0 _ll_register_callbacks cb 0x558edcfaa270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:58.406 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42478/0 (socket says 172.21.15.101:42478) 2022-07-03T10:37:58.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edcd9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd0b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.407 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 -- 172.21.15.101:0/2180926983 learned_addr learned my addr 172.21.15.101:0/2180926983 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 -- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 msgr2=0x558edd0b88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.408 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 -- 172.21.15.101:0/2180926983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 msgr2=0x558edd0b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd0b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.378+0000 7f4edd59c700 1 -- 172.21.15.101:0/2180926983 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed81168d0 con 0x558edd0b0120 2022-07-03T10:37:58.409 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4edd59c700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009ca0 tx=0x7f4ed0005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fecbc7bfee42e4 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:58.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed001d070 con 0x558edd0b0120 2022-07-03T10:37:58.410 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558edd0a2040 con 0x558edd0b0120 2022-07-03T10:37:58.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f4ed0005870 con 0x558edd0b0120 2022-07-03T10:37:58.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ed0003a90 con 0x558edd0b0120 2022-07-03T10:37:58.411 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 10 client.15255 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:58.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558edce33be0 con 0x558edd0b0120 2022-07-03T10:37:58.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4ed0031020 con 0x558edd0b0120 2022-07-03T10:37:58.412 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f4ed0015070 con 0x558edd0b0120 2022-07-03T10:37:58.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 10 client.15255 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:58.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 10 client.15255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:58.413 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 20 client.15255 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:58.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd0c4350 0x558edd0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 4 client.15255 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:58.414 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.379+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558edcff7af0 con 0x558edd0c4350 2022-07-03T10:37:58.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.380+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd0c4350 0x558edd0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.380+0000 7f4eb37fe700 10 client.15255 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.415 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.380+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd0c4350 0x558edd0c6760 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29247ff192063dcf server_cookie=91809dd0dff1f235 in_seq=0 out_seq=0 2022-07-03T10:37:58.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4eb37fe700 1 -- 172.21.15.101:0/2180926983 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x558edcff7af0 con 0x558edd0c4350 2022-07-03T10:37:58.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4eb37fe700 10 client.15255 handle_command_reply: tid=0 2022-07-03T10:37:58.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 client.15255 shutdown 2022-07-03T10:37:58.416 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 msgr2=0x558edd0bf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009ca0 tx=0x7f4ed0005040 comp rx=0 tx=0).stop 2022-07-03T10:37:58.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 shutdown_connections 2022-07-03T10:37:58.417 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd0c4350 0x558edd0c6760 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0b0120 0x558edd0bf8f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.418 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4eb37fe700 0 client.15255 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd0b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.419 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/2180926983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.420 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 >> 172.21.15.101:0/2180926983 conn(0x558edd0a4880 msgr2=0x558edd0aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 shutdown_connections 2022-07-03T10:37:58.421 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/2180926983 wait complete. 2022-07-03T10:37:58.422 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.381+0000 7f4ef2ec0b80 20 client.15255 trim_cache size 0 max 16384 2022-07-03T10:37:58.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.399+0000 7f4ef2ec0b80 1 Processor -- start 2022-07-03T10:37:58.430 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.399+0000 7f4ef2ec0b80 1 -- start start 2022-07-03T10:37:58.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.431 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd0b9040 con 0x558edd0ae890 2022-07-03T10:37:58.432 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd0af640 con 0x558edd0b0120 2022-07-03T10:37:58.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd0a7ef0 con 0x558edd0af1e0 2022-07-03T10:37:58.433 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42484/0 (socket says 172.21.15.101:42484) 2022-07-03T10:37:58.434 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 -- 172.21.15.101:0/873469190 learned_addr learned my addr 172.21.15.101:0/873469190 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.435 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edd59c700 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.436 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 -- 172.21.15.101:0/873469190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 msgr2=0x558edd0b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.437 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.438 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 -- 172.21.15.101:0/873469190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 msgr2=0x558edd0b84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.439 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 -- 172.21.15.101:0/873469190 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558edce320b0 con 0x558edd0ae890 2022-07-03T10:37:58.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc004110 tx=0x7f4ecc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1911d8e7615ee8e server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.440 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ebaffd700 1 -- 172.21.15.101:0/873469190 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc024040 con 0x558edd0ae890 2022-07-03T10:37:58.441 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ebaffd700 1 -- 172.21.15.101:0/873469190 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4ecc015070 con 0x558edd0ae890 2022-07-03T10:37:58.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.400+0000 7f4ebaffd700 1 -- 172.21.15.101:0/873469190 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc029b50 con 0x558edd0ae890 2022-07-03T10:37:58.442 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/873469190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 msgr2=0x558edd0a9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.443 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc004110 tx=0x7f4ecc015000 comp rx=0 tx=0).stop 2022-07-03T10:37:58.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/873469190 shutdown_connections 2022-07-03T10:37:58.444 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0ae890 0x558edd0a9d70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.445 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0b0120 0x558edd0b84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/873469190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0af1e0 0x558edd0b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.446 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/873469190 >> 172.21.15.101:0/873469190 conn(0x558edd0a4880 msgr2=0x558edd0b6230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/873469190 shutdown_connections 2022-07-03T10:37:58.447 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/873469190 wait complete. 2022-07-03T10:37:58.448 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.401+0000 7f4ef2ec0b80 1 -- start start 2022-07-03T10:37:58.449 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 Processor -- start 2022-07-03T10:37:58.449 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd112c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.450 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.451 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0b0120 0x558edd11a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd11a670 con 0x558edd0af1e0 2022-07-03T10:37:58.452 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd11a7e0 con 0x558edd0ae890 2022-07-03T10:37:58.453 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x558edd11a9a0 con 0x558edd0b0120 2022-07-03T10:37:58.454 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ef2ec0b80 10 client.0 _ll_register_callbacks cb 0x558edcfaa270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:58.455 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.456 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4edcd9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.456 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42490/0 (socket says 172.21.15.101:42490) 2022-07-03T10:37:58.457 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4255838803 learned_addr learned my addr 172.21.15.101:0/4255838803 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4edd59c700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0b0120 0x558edd11a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.458 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0b0120 msgr2=0x558edd11a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0b0120 0x558edd11a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.459 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4255838803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 msgr2=0x558edd112c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd112c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.460 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.402+0000 7f4ed7fff700 1 -- 172.21.15.101:0/4255838803 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ecc003df0 con 0x558edd0af1e0 2022-07-03T10:37:58.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ed7fff700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e820 tx=0x7f4ec40bf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92d05f8d115943c server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:58.461 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40975d0 con 0x558edd0af1e0 2022-07-03T10:37:58.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ec4090ee0 con 0x558edd0af1e0 2022-07-03T10:37:58.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f4ec4004b90 con 0x558edd0af1e0 2022-07-03T10:37:58.462 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec4010bf0 con 0x558edd0af1e0 2022-07-03T10:37:58.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 10 client.15261 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:58.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558edcf00880 con 0x558edd0af1e0 2022-07-03T10:37:58.463 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4ec4090ee0 con 0x558edd0af1e0 2022-07-03T10:37:58.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f4ec4097140 con 0x558edd0af1e0 2022-07-03T10:37:58.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 10 client.15261 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:58.464 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 10 client.15261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:58.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 20 client.15261 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:58.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd11c080 0x558edd11e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.465 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 4 client.15261 mds_command: new command op to 24569 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2022-07-03T10:37:58.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558edcff78d0 con 0x558edd11c080 2022-07-03T10:37:58.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.403+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd11c080 0x558edd11e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.466 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.404+0000 7f4eb37fe700 10 client.15261 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.404+0000 7f4edcd9b700 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd11c080 0x558edd11e490 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5167890b030f9847 server_cookie=f0e94fd6874fa0c1 in_seq=0 out_seq=0 2022-07-03T10:37:58.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4eb37fe700 1 -- 172.21.15.101:0/4255838803 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558edcff78d0 con 0x558edd11c080 2022-07-03T10:37:58.467 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4eb37fe700 10 client.15261 handle_command_reply: tid=0 2022-07-03T10:37:58.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 client.15261 shutdown 2022-07-03T10:37:58.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 msgr2=0x558edd113150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.468 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e820 tx=0x7f4ec40bf040 comp rx=0 tx=0).stop 2022-07-03T10:37:58.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 shutdown_connections 2022-07-03T10:37:58.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x558edd11c080 0x558edd11e490 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.469 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x558edd0b0120 0x558edd11a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x558edd0ae890 0x558edd112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 --2- 172.21.15.101:0/4255838803 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558edd0af1e0 0x558edd113150 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.470 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 >> 172.21.15.101:0/4255838803 conn(0x558edd0a4880 msgr2=0x558edd0b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4eb37fe700 0 client.15261 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 shutdown_connections 2022-07-03T10:37:58.471 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 1 -- 172.21.15.101:0/4255838803 wait complete. 2022-07-03T10:37:58.472 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.405+0000 7f4ef2ec0b80 20 client.15261 trim_cache size 0 max 16384 2022-07-03T10:37:58.472 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:58.473 INFO:teuthology.orchestra.run.smithi101.stdout: "return_code": 0, 2022-07-03T10:37:58.473 INFO:teuthology.orchestra.run.smithi101.stdout: "scrub_tag": "debfe181-0201-4e1a-8370-38f4437b2a0d", 2022-07-03T10:37:58.473 INFO:teuthology.orchestra.run.smithi101.stdout: "mode": "asynchronous" 2022-07-03T10:37:58.474 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:58.475 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 msgr2=0x7f4ec80991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 0x7f4ec80991d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00b840 tx=0x7f4ecc00b3a0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 msgr2=0x7f4ed810e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.476 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e850 tx=0x7f4ec4005ef0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 shutdown_connections 2022-07-03T10:37:58.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f4ec8096d20 0x7f4ec80991d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.477 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ed80fdf10 0x7f4ed8115830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4ed80fc4a0 0x7f4ed810e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 --2- 172.21.15.101:0/517862605 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ed80fcdf0 0x7f4ed810e7c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 >> 172.21.15.101:0/517862605 conn(0x7f4ed80f7c50 msgr2=0x7f4ed80f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.478 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 shutdown_connections 2022-07-03T10:37:58.479 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.406+0000 7f4edefff700 1 -- 172.21.15.101:0/517862605 wait complete. 2022-07-03T10:37:58.479 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:37:58.623 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 -- 172.21.15.101:0/3762185744 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fcdf0 msgr2=0x7f36040fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 --2- 172.21.15.101:0/3762185744 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fcdf0 0x7f36040fd7e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f35ec009900 tx=0x7f35ec0091e0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.624 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 -- 172.21.15.101:0/3762185744 shutdown_connections 2022-07-03T10:37:58.625 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 --2- 172.21.15.101:0/3762185744 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fdf10 0x7f3604104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.625 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 --2- 172.21.15.101:0/3762185744 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 0x7f36040fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.625 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 --2- 172.21.15.101:0/3762185744 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fcdf0 0x7f36040fd7e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 -- 172.21.15.101:0/3762185744 >> 172.21.15.101:0/3762185744 conn(0x7f36040f7c50 msgr2=0x7f36040fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 -- 172.21.15.101:0/3762185744 shutdown_connections 2022-07-03T10:37:58.626 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.619+0000 7f360a1ff700 1 -- 172.21.15.101:0/3762185744 wait complete. 2022-07-03T10:37:58.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 Processor -- start 2022-07-03T10:37:58.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 -- start start 2022-07-03T10:37:58.627 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 0x7f360410d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fcdf0 0x7f360410da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.628 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604115050 con 0x7f36040fdf10 2022-07-03T10:37:58.629 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604115190 con 0x7f36040fcdf0 2022-07-03T10:37:58.629 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f360a1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3604115350 con 0x7f36040fc4a0 2022-07-03T10:37:58.629 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f3603fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.630 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f3603fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42504/0 (socket says 172.21.15.101:42504) 2022-07-03T10:37:58.630 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f3603fff700 1 -- 172.21.15.101:0/2593812835 learned_addr learned my addr 172.21.15.101:0/2593812835 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.630 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f3602ffd700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fcdf0 0x7f360410da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.631 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.620+0000 7f36037fe700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 0x7f360410d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.631 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 -- 172.21.15.101:0/2593812835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 msgr2=0x7f360410d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.631 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 0x7f360410d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 -- 172.21.15.101:0/2593812835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fcdf0 msgr2=0x7f360410da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.632 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fcdf0 0x7f360410da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 -- 172.21.15.101:0/2593812835 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec009440 con 0x7f36040fdf10 2022-07-03T10:37:58.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f3603fff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00e830 tx=0x7f35fc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.633 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f35fbfff700 1 -- 172.21.15.101:0/2593812835 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc005020 con 0x7f36040fdf10 2022-07-03T10:37:58.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f35fbfff700 1 -- 172.21.15.101:0/2593812835 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f35fc010710 con 0x7f36040fdf10 2022-07-03T10:37:58.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3604115630 con 0x7f36040fdf10 2022-07-03T10:37:58.634 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f35fbfff700 1 -- 172.21.15.101:0/2593812835 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004980 con 0x7f36040fdf10 2022-07-03T10:37:58.635 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.621+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3604115bc0 con 0x7f36040fdf10 2022-07-03T10:37:58.642 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.623+0000 7f35fbfff700 1 -- 172.21.15.101:0/2593812835 <== mon.0 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7f35fc00c040 con 0x7f36040fdf10 2022-07-03T10:37:58.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.624+0000 7f35fbfff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 0x7f35f00991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.624+0000 7f35fbfff700 1 -- 172.21.15.101:0/2593812835 <== mon.0 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f35fc014070 con 0x7f36040fdf10 2022-07-03T10:37:58.643 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.624+0000 7f36037fe700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 0x7f35f00991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.644 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 0x7f35f00991d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005fd0 tx=0x7f35f4009460 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.644 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 Processor -- start 2022-07-03T10:37:58.644 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 -- start start 2022-07-03T10:37:58.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.645 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cfb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801be4210 con 0x55a801cf9a20 2022-07-03T10:37:58.646 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801c4dd00 con 0x55a801cfa370 2022-07-03T10:37:58.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801c47500 con 0x55a801cfb2b0 2022-07-03T10:37:58.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.647 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42514/0 (socket says 172.21.15.101:42514) 2022-07-03T10:37:58.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 -- 172.21.15.101:0/3836121295 learned_addr learned my addr 172.21.15.101:0/3836121295 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f3602ffd700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.648 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f3603fff700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cfb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 -- 172.21.15.101:0/3836121295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 msgr2=0x55a801cfb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.649 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cfb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 -- 172.21.15.101:0/3836121295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 msgr2=0x55a801cfac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.650 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f36037fe700 1 -- 172.21.15.101:0/3836121295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a801ad03b0 con 0x55a801cf9a20 2022-07-03T10:37:58.651 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.625+0000 7f3603fff700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cfb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-07-03T10:37:58.651 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f36037fe700 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f35f4003d40 tx=0x7f35f4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3e39da41995059 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.651 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f35f97fa700 1 -- 172.21.15.101:0/3836121295 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003930 con 0x55a801cf9a20 2022-07-03T10:37:58.652 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f35f97fa700 1 -- 172.21.15.101:0/3836121295 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f35f4003d70 con 0x55a801cf9a20 2022-07-03T10:37:58.652 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f35f97fa700 1 -- 172.21.15.101:0/3836121295 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4026670 con 0x55a801cf9a20 2022-07-03T10:37:58.652 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f361e1a2b80 1 -- 172.21.15.101:0/3836121295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 msgr2=0x55a801cf9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.626+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f35f4003d40 tx=0x7f35f4018000 comp rx=0 tx=0).stop 2022-07-03T10:37:58.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- 172.21.15.101:0/3836121295 shutdown_connections 2022-07-03T10:37:58.653 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cf9e30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cfb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.654 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/3836121295 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- 172.21.15.101:0/3836121295 >> 172.21.15.101:0/3836121295 conn(0x55a801ced800 msgr2=0x55a801cf4190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- 172.21.15.101:0/3836121295 shutdown_connections 2022-07-03T10:37:58.655 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- 172.21.15.101:0/3836121295 wait complete. 2022-07-03T10:37:58.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- start start 2022-07-03T10:37:58.656 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 Processor -- start 2022-07-03T10:37:58.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d03470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfa370 0x55a801d039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.657 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801b35e90 con 0x55a801cfb2b0 2022-07-03T10:37:58.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d0afa0 con 0x55a801cf9a20 2022-07-03T10:37:58.658 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d0b160 con 0x55a801cfa370 2022-07-03T10:37:58.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.627+0000 7f361e1a2b80 10 client.0 _ll_register_callbacks cb 0x55a801bf31f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:58.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.659 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42528/0 (socket says 172.21.15.101:42528) 2022-07-03T10:37:58.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 -- 172.21.15.101:0/2838836362 learned_addr learned my addr 172.21.15.101:0/2838836362 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f36037fe700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d03470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.660 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3602ffd700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfa370 0x55a801d039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 -- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfa370 msgr2=0x55a801d039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfa370 0x55a801d039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.661 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 -- 172.21.15.101:0/2838836362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 msgr2=0x55a801d03470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d03470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 -- 172.21.15.101:0/2838836362 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f4017070 con 0x55a801cfb2b0 2022-07-03T10:37:58.662 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f36037fe700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d03470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:37:58.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f3603fff700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00f010 tx=0x7f35fc090d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b62d6a5de6a51e9 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.663 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f35e27fc700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:58.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc090b50 con 0x55a801cfb2b0 2022-07-03T10:37:58.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a801ceafc0 con 0x55a801cfb2b0 2022-07-03T10:37:58.664 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.628+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f35fc010b80 con 0x55a801cfb2b0 2022-07-03T10:37:58.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35fc01f300 con 0x55a801cfb2b0 2022-07-03T10:37:58.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 10 client.15273 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:58.665 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a801a7cb60 con 0x55a801cfb2b0 2022-07-03T10:37:58.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55a801ceafc0 con 0x55a801cfb2b0 2022-07-03T10:37:58.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f35fc090390 con 0x55a801cfb2b0 2022-07-03T10:37:58.666 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 10 client.15273 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:58.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 10 client.15273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:58.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 20 client.15273 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:58.667 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d0f400 0x55a801d11810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 4 client.15273 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:37:58.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a801c40a70 con 0x55a801d0f400 2022-07-03T10:37:58.668 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.629+0000 7f36037fe700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d0f400 0x55a801d11810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.630+0000 7f35e27fc700 10 client.15273 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.630+0000 7f36037fe700 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d0f400 0x55a801d11810 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b045aa921a6ec634 server_cookie=3b93044573b0520 in_seq=0 out_seq=0 2022-07-03T10:37:58.669 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.630+0000 7f35e27fc700 1 -- 172.21.15.101:0/2838836362 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55a801c40a70 con 0x55a801d0f400 2022-07-03T10:37:58.670 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.630+0000 7f35e27fc700 10 client.15273 handle_command_reply: tid=0 2022-07-03T10:37:58.670 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.630+0000 7f361e1a2b80 1 client.15273 shutdown 2022-07-03T10:37:58.670 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 msgr2=0x55a801d0a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00f010 tx=0x7f35fc090d80 comp rx=0 tx=0).stop 2022-07-03T10:37:58.671 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 shutdown_connections 2022-07-03T10:37:58.672 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d0f400 0x55a801d11810 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.672 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfb2b0 0x55a801d0a9f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.672 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d03470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.673 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/2838836362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfa370 0x55a801d039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.673 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f35e27fc700 0 client.15273 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.673 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 >> 172.21.15.101:0/2838836362 conn(0x55a801ced800 msgr2=0x55a801cf3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.674 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 shutdown_connections 2022-07-03T10:37:58.674 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 1 -- 172.21.15.101:0/2838836362 wait complete. 2022-07-03T10:37:58.674 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.631+0000 7f361e1a2b80 20 client.15273 trim_cache size 0 max 16384 2022-07-03T10:37:58.682 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 Processor -- start 2022-07-03T10:37:58.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 -- start start 2022-07-03T10:37:58.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.683 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfe400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cf3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d04160 con 0x55a801cf9a20 2022-07-03T10:37:58.684 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d029c0 con 0x55a801cfa370 2022-07-03T10:37:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801cf3a10 con 0x55a801cfb2b0 2022-07-03T10:37:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.685 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42536/0 (socket says 172.21.15.101:42536) 2022-07-03T10:37:58.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 -- 172.21.15.101:0/1409452441 learned_addr learned my addr 172.21.15.101:0/1409452441 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.686 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f3602ffd700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfe400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f3603fff700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cf3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 -- 172.21.15.101:0/1409452441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 msgr2=0x55a801cf3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.687 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cf3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 -- 172.21.15.101:0/1409452441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 msgr2=0x55a801cfe400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfe400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.688 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f36037fe700 1 -- 172.21.15.101:0/1409452441 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a801a7b030 con 0x55a801cf9a20 2022-07-03T10:37:58.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.647+0000 7f3603fff700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cf3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:37:58.689 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f36037fe700 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005b70 tx=0x7f35f4028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cdaf0822095fcf server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f35f9ffb700 1 -- 172.21.15.101:0/1409452441 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f402b050 con 0x55a801cf9a20 2022-07-03T10:37:58.690 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f35f9ffb700 1 -- 172.21.15.101:0/1409452441 <== mon.0 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f35f402c040 con 0x55a801cf9a20 2022-07-03T10:37:58.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f35f9ffb700 1 -- 172.21.15.101:0/1409452441 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4000fb0 con 0x55a801cf9a20 2022-07-03T10:37:58.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1409452441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 msgr2=0x55a801cfdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.691 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005b70 tx=0x7f35f4028040 comp rx=0 tx=0).stop 2022-07-03T10:37:58.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1409452441 shutdown_connections 2022-07-03T10:37:58.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cf9a20 0x55a801cfdec0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.692 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801cf3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1409452441 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cfa370 0x55a801cfe400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1409452441 >> 172.21.15.101:0/1409452441 conn(0x55a801ced800 msgr2=0x55a801d03450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.693 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1409452441 shutdown_connections 2022-07-03T10:37:58.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.648+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1409452441 wait complete. 2022-07-03T10:37:58.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 -- start start 2022-07-03T10:37:58.694 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 Processor -- start 2022-07-03T10:37:58.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d06800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.695 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801d07cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d08230 con 0x55a801cfa370 2022-07-03T10:37:58.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d083a0 con 0x55a801cf9a20 2022-07-03T10:37:58.696 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a801d08560 con 0x55a801cfb2b0 2022-07-03T10:37:58.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f361e1a2b80 10 client.0 _ll_register_callbacks cb 0x55a801bf31f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:37:58.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f3602ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.697 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f3602ffd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:42538/0 (socket says 172.21.15.101:42538) 2022-07-03T10:37:58.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f3602ffd700 1 -- 172.21.15.101:0/1145822319 learned_addr learned my addr 172.21.15.101:0/1145822319 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:37:58.698 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f36037fe700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d06800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.649+0000 7f3603fff700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801d07cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.699 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 -- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 msgr2=0x55a801d07cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801d07cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 -- 172.21.15.101:0/1145822319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 msgr2=0x55a801d06800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:37:58.700 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d06800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 -- 172.21.15.101:0/1145822319 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a801ad03b0 con 0x55a801cfa370 2022-07-03T10:37:58.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f3602ffd700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f36040fd1c0 tx=0x7f35ec01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8664f2dc81775ea3 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:37:58.701 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 10 client.0 ms_handle_connect on v2:172.21.15.101:3300/0 2022-07-03T10:37:58.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mon.0 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec01d030 con 0x55a801cfa370 2022-07-03T10:37:58.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mon.0 v2:172.21.15.101:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7f35ec003fa0 con 0x55a801cfa370 2022-07-03T10:37:58.702 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35f4003d70 con 0x55a801cfa370 2022-07-03T10:37:58.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mon.0 v2:172.21.15.101:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35ec00b650 con 0x55a801cfa370 2022-07-03T10:37:58.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f361e1a2b80 10 client.15279 fetch_fsmap learned FSMap version 20 2022-07-03T10:37:58.703 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a801b49800 con 0x55a801cfa370 2022-07-03T10:37:58.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mon.0 v2:172.21.15.101:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f35f4003d70 con 0x55a801cfa370 2022-07-03T10:37:58.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.650+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mon.0 v2:172.21.15.101:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f35ec02faf0 con 0x55a801cfa370 2022-07-03T10:37:58.704 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 10 client.15279 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:37:58.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 10 client.15279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:37:58.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 20 client.15279 populate_metadata read hostname 'smithi101' 2022-07-03T10:37:58.705 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d640d0 0x55a801d664e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:37:58.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 4 client.15279 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:37:58.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a801c40850 con 0x55a801d640d0 2022-07-03T10:37:58.706 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f36037fe700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d640d0 0x55a801d664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:37:58.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f35e27fc700 10 client.15279 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.707 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.651+0000 7f36037fe700 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d640d0 0x55a801d664e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c353ce50d8d02d9d server_cookie=482fac0bc2d2ee77 in_seq=0 out_seq=0 2022-07-03T10:37:58.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f35e27fc700 1 -- 172.21.15.101:0/1145822319 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a801c40850 con 0x55a801d640d0 2022-07-03T10:37:58.708 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f35e27fc700 10 client.15279 handle_command_reply: tid=0 2022-07-03T10:37:58.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 client.15279 shutdown 2022-07-03T10:37:58.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 msgr2=0x55a801d06d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.709 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f36040fd1c0 tx=0x7f35ec01a040 comp rx=0 tx=0).stop 2022-07-03T10:37:58.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 shutdown_connections 2022-07-03T10:37:58.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55a801d640d0 0x55a801d664e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.710 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55a801cfb2b0 0x55a801d07cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f35e27fc700 0 client.15279 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:37:58.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55a801cf9a20 0x55a801d06800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.711 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 --2- 172.21.15.101:0/1145822319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55a801cfa370 0x55a801d06d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.712 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 >> 172.21.15.101:0/1145822319 conn(0x55a801ced800 msgr2=0x55a801cf3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.712 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 shutdown_connections 2022-07-03T10:37:58.712 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 1 -- 172.21.15.101:0/1145822319 wait complete. 2022-07-03T10:37:58.713 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f361e1a2b80 20 client.15279 trim_cache size 0 max 16384 2022-07-03T10:37:58.713 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:37:58.714 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "IDLE (waiting for more scrubs)", 2022-07-03T10:37:58.714 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": { 2022-07-03T10:37:58.715 INFO:teuthology.orchestra.run.smithi101.stdout: "debfe181-0201-4e1a-8370-38f4437b2a0d": { 2022-07-03T10:37:58.715 INFO:teuthology.orchestra.run.smithi101.stdout: "path": "/", 2022-07-03T10:37:58.715 INFO:teuthology.orchestra.run.smithi101.stdout: "tag": "debfe181-0201-4e1a-8370-38f4437b2a0d", 2022-07-03T10:37:58.716 INFO:teuthology.orchestra.run.smithi101.stdout: "options": "recursive,force" 2022-07-03T10:37:58.716 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:37:58.717 INFO:teuthology.orchestra.run.smithi101.stdout: } 2022-07-03T10:37:58.717 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:37:58.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 msgr2=0x7f35f00991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.719 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 0x7f35f00991d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35f4005fd0 tx=0x7f35f4009460 comp rx=0 tx=0).stop 2022-07-03T10:37:58.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 msgr2=0x7f3604114ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:37:58.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00e830 tx=0x7f35fc005ef0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.720 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 shutdown_connections 2022-07-03T10:37:58.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7f35f0096d20 0x7f35f00991d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36040fdf10 0x7f3604114ad0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.721 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f36040fc4a0 0x7f360410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 --2- 172.21.15.101:0/2593812835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f36040fcdf0 0x7f360410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:37:58.722 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.652+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 >> 172.21.15.101:0/2593812835 conn(0x7f36040f7c50 msgr2=0x7f3604100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:37:58.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.653+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 shutdown_connections 2022-07-03T10:37:58.723 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:37:58.653+0000 7f360a1ff700 1 -- 172.21.15.101:0/2593812835 wait complete. 2022-07-03T10:37:58.723 INFO:tasks.cephfs.filesystem:scrub status for tag:debfe181-0201-4e1a-8370-38f4437b2a0d - {'path': '/', 'tag': 'debfe181-0201-4e1a-8370-38f4437b2a0d', 'options': 'recursive,force'} 2022-07-03T10:37:59.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:59 smithi101 ceph-mon[85126]: scrub queued for path: / 2022-07-03T10:37:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:59 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:59 smithi101 ceph-mon[85126]: scrub summary: active paths [/] 2022-07-03T10:37:59.493 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:37:59 smithi101 ceph-mon[85126]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:59 smithi161 ceph-mon[91579]: scrub queued for path: / 2022-07-03T10:37:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:59 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:59 smithi161 ceph-mon[91579]: scrub summary: active paths [/] 2022-07-03T10:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:37:59 smithi161 ceph-mon[91579]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:59 smithi188 ceph-mon[92425]: scrub queued for path: / 2022-07-03T10:37:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:59 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:37:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:59 smithi188 ceph-mon[92425]: scrub summary: active paths [/] 2022-07-03T10:37:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:37:59 smithi188 ceph-mon[92425]: scrub summary: idle+waiting paths [/] 2022-07-03T10:38:00.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:00 smithi101 ceph-mon[85126]: pgmap v358: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 18 KiB/s wr, 5 op/s 2022-07-03T10:38:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:00 smithi161 ceph-mon[91579]: pgmap v358: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 18 KiB/s wr, 5 op/s 2022-07-03T10:38:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:00 smithi188 ceph-mon[92425]: pgmap v358: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 18 KiB/s wr, 5 op/s 2022-07-03T10:38:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:01 smithi161 ceph-mon[91579]: pgmap v359: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2022-07-03T10:38:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:01 smithi188 ceph-mon[92425]: pgmap v359: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2022-07-03T10:38:01.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:01 smithi101 ceph-mon[85126]: pgmap v359: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2022-07-03T10:38:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:02 smithi161 ceph-mon[91579]: scrub summary: idle 2022-07-03T10:38:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:02 smithi188 ceph-mon[92425]: scrub summary: idle 2022-07-03T10:38:03.242 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:02 smithi101 ceph-mon[85126]: scrub summary: idle 2022-07-03T10:38:04.492 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:04 smithi101 ceph-mon[85126]: pgmap v360: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2022-07-03T10:38:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:04 smithi161 ceph-mon[91579]: pgmap v360: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2022-07-03T10:38:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:04 smithi188 ceph-mon[92425]: pgmap v360: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2022-07-03T10:38:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:05 smithi161 ceph-mon[91579]: pgmap v361: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2022-07-03T10:38:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:05 smithi188 ceph-mon[92425]: pgmap v361: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2022-07-03T10:38:05.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:05 smithi101 ceph-mon[85126]: pgmap v361: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2022-07-03T10:38:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:07 smithi161 ceph-mon[91579]: pgmap v362: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-07-03T10:38:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:07 smithi188 ceph-mon[92425]: pgmap v362: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-07-03T10:38:07.742 INFO:journalctl@ceph.mon.a.smithi101.stdout:Jul 03 10:38:07 smithi101 ceph-mon[85126]: pgmap v362: 97 pgs: 97 active+clean; 2.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2022-07-03T10:38:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: mon.b calling monitor election 2022-07-03T10:38:24.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: mon.c calling monitor election 2022-07-03T10:38:24.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-07-03T10:38:24.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: monmap e3: 3 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} 2022-07-03T10:38:24.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:38:24.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: osdmap e82: 12 total, 12 up, 12 in 2022-07-03T10:38:24.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-07-03T10:38:24.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-07-03T10:38:24.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c 2022-07-03T10:38:24.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:38:24.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:24 smithi161 ceph-mon[91579]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:38:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: mon.b calling monitor election 2022-07-03T10:38:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: mon.c calling monitor election 2022-07-03T10:38:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-07-03T10:38:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: monmap e3: 3 mons at {a=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],b=[v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0],c=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} 2022-07-03T10:38:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:38:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: osdmap e82: 12 total, 12 up, 12 in 2022-07-03T10:38:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-07-03T10:38:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-07-03T10:38:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c 2022-07-03T10:38:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:38:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:24 smithi188 ceph-mon[92425]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:38:28.725 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2022-07-03T10:38:28.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 -- 172.21.15.101:0/1755810701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fcdf0 msgr2=0x7eff780fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.887 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/1755810701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fcdf0 0x7eff780fd7e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7eff64009860 tx=0x7eff64005ef0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 -- 172.21.15.101:0/1755810701 shutdown_connections 2022-07-03T10:38:28.888 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/1755810701 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff780fc4a0 0x7eff780fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/1755810701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fdf10 0x7eff78104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/1755810701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fcdf0 0x7eff780fd7e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 -- 172.21.15.101:0/1755810701 >> 172.21.15.101:0/1755810701 conn(0x7eff780f7c50 msgr2=0x7eff780fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.889 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 -- 172.21.15.101:0/1755810701 shutdown_connections 2022-07-03T10:38:28.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.880+0000 7eff7f1ff700 1 -- 172.21.15.101:0/1755810701 wait complete. 2022-07-03T10:38:28.890 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 Processor -- start 2022-07-03T10:38:28.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 -- start start 2022-07-03T10:38:28.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff780fcdf0 0x7eff7810d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.891 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fdf10 0x7eff78114a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78115030 con 0x7eff780fcdf0 2022-07-03T10:38:28.892 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78115170 con 0x7eff780fdf10 2022-07-03T10:38:28.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7f1ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff78115330 con 0x7eff780fc4a0 2022-07-03T10:38:28.893 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:53994/0 (socket says 172.21.15.101:53994) 2022-07-03T10:38:28.894 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.881+0000 7eff7cf9b700 1 -- 172.21.15.101:0/4003162161 learned_addr learned my addr 172.21.15.101:0/4003162161 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:38:28.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7d79c700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fdf10 0x7eff78114a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 -- 172.21.15.101:0/4003162161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fdf10 msgr2=0x7eff78114a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.895 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fdf10 0x7eff78114a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 -- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff780fcdf0 msgr2=0x7eff7810d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.896 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff780fcdf0 0x7eff7810d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 -- 172.21.15.101:0/4003162161 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64009410 con 0x7eff780fc4a0 2022-07-03T10:38:28.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00da20 tx=0x7eff6c00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:38:28.897 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff757fa700 1 -- 172.21.15.101:0/4003162161 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00ea00 con 0x7eff780fc4a0 2022-07-03T10:38:28.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff78065770 con 0x7eff780fc4a0 2022-07-03T10:38:28.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff757fa700 1 -- 172.21.15.101:0/4003162161 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7eff6c00ed70 con 0x7eff780fc4a0 2022-07-03T10:38:28.898 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff757fa700 1 -- 172.21.15.101:0/4003162161 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c01f4e0 con 0x7eff780fc4a0 2022-07-03T10:38:28.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.882+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff78065d50 con 0x7eff780fc4a0 2022-07-03T10:38:28.899 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.885+0000 7eff757fa700 1 -- 172.21.15.101:0/4003162161 <== mon.2 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 18) v1 ==== 125132+0+0 (secure 0 0 0) 0x7eff6c01d030 con 0x7eff780fc4a0 2022-07-03T10:38:28.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff757fa700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7eff68096d20 0x7eff680991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff757fa700 1 -- 172.21.15.101:0/4003162161 <== mon.2 v2:172.21.15.188:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7eff6c0c1000 con 0x7eff780fc4a0 2022-07-03T10:38:28.908 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 Processor -- start 2022-07-03T10:38:28.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 -- start start 2022-07-03T10:38:28.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.909 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0b9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bbfa4210 con 0x55d0bc0b8160 2022-07-03T10:38:28.910 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc00dd00 con 0x55d0bc0b90a0 2022-07-03T10:38:28.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff930deb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc007500 con 0x55d0bc0b7810 2022-07-03T10:38:28.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.911 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.886+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:54000/0 (socket says 172.21.15.101:54000) 2022-07-03T10:38:28.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2240090830 learned_addr learned my addr 172.21.15.101:0/2240090830 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:38:28.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7d79c700 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.912 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2240090830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 msgr2=0x55d0bc0b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2240090830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 msgr2=0x55d0bc0b8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.913 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2240090830 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0bbe903b0 con 0x55d0bc0b7810 2022-07-03T10:38:28.914 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00dfd0 tx=0x7eff6c0beca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4052cc90769e1a33 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:38:28.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff5a7fc700 1 -- 172.21.15.101:0/2240090830 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c092660 con 0x55d0bc0b7810 2022-07-03T10:38:28.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff5a7fc700 1 -- 172.21.15.101:0/2240090830 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7eff6c091f60 con 0x55d0bc0b7810 2022-07-03T10:38:28.915 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.887+0000 7eff5a7fc700 1 -- 172.21.15.101:0/2240090830 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c092430 con 0x55d0bc0b7810 2022-07-03T10:38:28.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- 172.21.15.101:0/2240090830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 msgr2=0x55d0bc0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00dfd0 tx=0x7eff6c0beca0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.916 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- 172.21.15.101:0/2240090830 shutdown_connections 2022-07-03T10:38:28.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0b7c20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.917 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 --2- 172.21.15.101:0/2240090830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- 172.21.15.101:0/2240090830 >> 172.21.15.101:0/2240090830 conn(0x55d0bc0ad800 msgr2=0x55d0bc0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- 172.21.15.101:0/2240090830 shutdown_connections 2022-07-03T10:38:28.918 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- 172.21.15.101:0/2240090830 wait complete. 2022-07-03T10:38:28.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 -- start start 2022-07-03T10:38:28.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.888+0000 7eff930deb80 1 Processor -- start 2022-07-03T10:38:28.919 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.920 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bbef5e90 con 0x55d0bc0b90a0 2022-07-03T10:38:28.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0c8f40 con 0x55d0bc0b7810 2022-07-03T10:38:28.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0c9100 con 0x55d0bc0b8160 2022-07-03T10:38:28.921 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff930deb80 10 client.0 _ll_register_callbacks cb 0x55d0bbfb31f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:38:28.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.101:50636/0 (socket says 172.21.15.101:50636) 2022-07-03T10:38:28.922 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff77fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2262647554 learned_addr learned my addr 172.21.15.101:0/2262647554 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:38:28.923 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b8160 msgr2=0x55d0bc0c1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2262647554 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b90a0 msgr2=0x55d0bc0c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.924 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 -- 172.21.15.101:0/2262647554 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff78065d50 con 0x55d0bc0b7810 2022-07-03T10:38:28.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00b620 tx=0x7eff6c0beca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b19cd57098910765 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:38:28.925 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff537fe700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:38:28.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.889+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c090300 con 0x55d0bc0b7810 2022-07-03T10:38:28.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0bc0aafc0 con 0x55d0bc0b7810 2022-07-03T10:38:28.926 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7eff6c0a8270 con 0x55d0bc0b7810 2022-07-03T10:38:28.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff6c0a8430 con 0x55d0bc0b7810 2022-07-03T10:38:28.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 10 client.25357 fetch_fsmap learned FSMap version 20 2022-07-03T10:38:28.927 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d0bbe3cb60 con 0x55d0bc0b7810 2022-07-03T10:38:28.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55d0bc0aafc0 con 0x55d0bc0b7810 2022-07-03T10:38:28.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7eff6c016600 con 0x55d0bc0b7810 2022-07-03T10:38:28.928 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 10 client.25357 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:38:28.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 10 client.25357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:38:28.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 20 client.25357 populate_metadata read hostname 'smithi101' 2022-07-03T10:38:28.929 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc0cd050 0x55d0bc0cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 4 client.25357 mds_command: new command op to 24569 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-03T10:38:28.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0bc000a70 con 0x55d0bc0cd050 2022-07-03T10:38:28.930 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.890+0000 7eff77fff700 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc0cd050 0x55d0bc0cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.891+0000 7eff537fe700 10 client.25357 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:38:28.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.891+0000 7eff77fff700 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc0cd050 0x55d0bc0cf460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e80f9f2c92fb8889 server_cookie=e73de27882cadfaa in_seq=0 out_seq=0 2022-07-03T10:38:28.931 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff537fe700 1 -- 172.21.15.101:0/2262647554 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55d0bc000a70 con 0x55d0bc0cd050 2022-07-03T10:38:28.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff537fe700 10 client.25357 handle_command_reply: tid=0 2022-07-03T10:38:28.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 client.25357 shutdown 2022-07-03T10:38:28.932 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 msgr2=0x55d0bc0c1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00b620 tx=0x7eff6c0beca0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.933 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 shutdown_connections 2022-07-03T10:38:28.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc0cd050 0x55d0bc0cf460 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.934 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff537fe700 0 client.25357 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:38:28.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c1410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 --2- 172.21.15.101:0/2262647554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.935 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 >> 172.21.15.101:0/2262647554 conn(0x55d0bc0ad800 msgr2=0x55d0bc0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 shutdown_connections 2022-07-03T10:38:28.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 1 -- 172.21.15.101:0/2262647554 wait complete. 2022-07-03T10:38:28.936 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.892+0000 7eff930deb80 20 client.25357 trim_cache size 0 max 16384 2022-07-03T10:38:28.943 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 Processor -- start 2022-07-03T10:38:28.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 -- start start 2022-07-03T10:38:28.944 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0bbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.945 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0d4010 con 0x55d0bc0b8160 2022-07-03T10:38:28.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0aae90 con 0x55d0bc0b90a0 2022-07-03T10:38:28.946 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff930deb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0b8640 con 0x55d0bc0b7810 2022-07-03T10:38:28.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff7cf9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:54012/0 (socket says 172.21.15.101:54012) 2022-07-03T10:38:28.947 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.908+0000 7eff7cf9b700 1 -- 172.21.15.101:0/3264909390 learned_addr learned my addr 172.21.15.101:0/3264909390 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:38:28.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7d79c700 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0bbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 -- 172.21.15.101:0/3264909390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 msgr2=0x55d0bc0bbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.948 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0bbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 -- 172.21.15.101:0/3264909390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 msgr2=0x55d0bc0b9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.949 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 -- 172.21.15.101:0/3264909390 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0bbe3b030 con 0x55d0bc0b7810 2022-07-03T10:38:28.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7d79c700 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0bbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-03T10:38:28.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff6c016dd0 tx=0x7eff6c0beca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e691f4d956dba1 server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:38:28.950 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff5affd700 1 -- 172.21.15.101:0/3264909390 <== mon.2 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c090300 con 0x55d0bc0b7810 2022-07-03T10:38:28.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff5affd700 1 -- 172.21.15.101:0/3264909390 <== mon.2 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7eff6c0a8270 con 0x55d0bc0b7810 2022-07-03T10:38:28.951 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.909+0000 7eff5affd700 1 -- 172.21.15.101:0/3264909390 <== mon.2 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c091130 con 0x55d0bc0b7810 2022-07-03T10:38:28.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- 172.21.15.101:0/3264909390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 msgr2=0x55d0bc0bf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff6c016dd0 tx=0x7eff6c0beca0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.952 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- 172.21.15.101:0/3264909390 shutdown_connections 2022-07-03T10:38:28.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0bbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.953 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b7810 0x55d0bc0bf850 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- 172.21.15.101:0/3264909390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b8160 0x55d0bc0b9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- 172.21.15.101:0/3264909390 >> 172.21.15.101:0/3264909390 conn(0x55d0bc0ad800 msgr2=0x55d0bc0c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.954 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- 172.21.15.101:0/3264909390 shutdown_connections 2022-07-03T10:38:28.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- 172.21.15.101:0/3264909390 wait complete. 2022-07-03T10:38:28.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- start start 2022-07-03T10:38:28.955 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 Processor -- start 2022-07-03T10:38:28.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.956 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.957 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0c5030 con 0x55d0bc0b7810 2022-07-03T10:38:28.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0c51a0 con 0x55d0bc0b8160 2022-07-03T10:38:28.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0bc0c5360 con 0x55d0bc0b90a0 2022-07-03T10:38:28.958 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.910+0000 7eff930deb80 10 client.0 _ll_register_callbacks cb 0x55d0bbfb31f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-03T10:38:28.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff7d79c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.959 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff7d79c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.101:54028/0 (socket says 172.21.15.101:54028) 2022-07-03T10:38:28.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff7d79c700 1 -- 172.21.15.101:0/123112568 learned_addr learned my addr 172.21.15.101:0/123112568 (peer_addr_for_me v2:172.21.15.101:0/0) 2022-07-03T10:38:28.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 -- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 msgr2=0x55d0bc0c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.960 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 -- 172.21.15.101:0/123112568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b7810 msgr2=0x55d0bc0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-03T10:38:28.961 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 -- 172.21.15.101:0/123112568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d0bbe3cb60 con 0x55d0bc0b8160 2022-07-03T10:38:28.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff7d79c700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-07-03T10:38:28.962 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff77fff700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c6710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7eff64009830 tx=0x7eff64019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d5c72a24b62aa3e server_cookie=0 in_seq=0 out_seq=0 2022-07-03T10:38:28.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff537fe700 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2022-07-03T10:38:28.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6402a030 con 0x55d0bc0b8160 2022-07-03T10:38:28.963 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff6c091f60 con 0x55d0bc0b8160 2022-07-03T10:38:28.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.911+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mon.1 v2:172.21.15.161:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8236+0+0 (secure 0 0 0) 0x7eff6402b070 con 0x55d0bc0b8160 2022-07-03T10:38:28.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff6400fb10 con 0x55d0bc0b8160 2022-07-03T10:38:28.964 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 10 client.25363 fetch_fsmap learned FSMap version 20 2022-07-03T10:38:28.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d0bbf09800 con 0x55d0bc0b8160 2022-07-03T10:38:28.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mon.1 v2:172.21.15.161:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7eff64035020 con 0x55d0bc0b8160 2022-07-03T10:38:28.965 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mon.1 v2:172.21.15.161:3300/0 5 ==== fsmap(e 20) v1 ==== 7448+0+0 (secure 0 0 0) 0x7eff64022070 con 0x55d0bc0b8160 2022-07-03T10:38:28.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 10 client.25363 fetch_fsmap finished waiting for FSMap version 20 2022-07-03T10:38:28.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 10 client.25363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.l 2022-07-03T10:38:28.966 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 20 client.25363 populate_metadata read hostname 'smithi101' 2022-07-03T10:38:28.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc11fc50 0x55d0bc122060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-07-03T10:38:28.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 4 client.25363 mds_command: new command op to 24569 tid=0[{"prefix": "scrub status"}] 2022-07-03T10:38:28.967 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 --> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d0bc000850 con 0x55d0bc11fc50 2022-07-03T10:38:28.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.912+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc11fc50 0x55d0bc122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-07-03T10:38:28.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff537fe700 10 client.25363 ms_handle_connect on v2:172.21.15.188:6838/204174115 2022-07-03T10:38:28.968 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff7cf9b700 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc11fc50 0x55d0bc122060 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7c4d8d17a1a887 server_cookie=2e6de169a61cd7eb in_seq=0 out_seq=0 2022-07-03T10:38:28.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff537fe700 1 -- 172.21.15.101:0/123112568 <== mds.0 v2:172.21.15.188:6838/204174115 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d0bc000850 con 0x55d0bc11fc50 2022-07-03T10:38:28.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff537fe700 10 client.25363 handle_command_reply: tid=0 2022-07-03T10:38:28.969 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 client.25363 shutdown 2022-07-03T10:38:28.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 msgr2=0x55d0bc0c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.970 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c6710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7eff64009830 tx=0x7eff64019040 comp rx=0 tx=0).stop 2022-07-03T10:38:28.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 shutdown_connections 2022-07-03T10:38:28.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] conn(0x55d0bc11fc50 0x55d0bc122060 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.971 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d0bc0b7810 0x55d0bc0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff537fe700 0 client.25363 ms_handle_reset on v2:172.21.15.188:6838/204174115 2022-07-03T10:38:28.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x55d0bc0b90a0 0x55d0bc0c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.972 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 --2- 172.21.15.101:0/123112568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55d0bc0b8160 0x55d0bc0c6710 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 >> 172.21.15.101:0/123112568 conn(0x55d0bc0ad800 msgr2=0x55d0bc0c0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.973 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 shutdown_connections 2022-07-03T10:38:28.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 1 -- 172.21.15.101:0/123112568 wait complete. 2022-07-03T10:38:28.974 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.913+0000 7eff930deb80 20 client.25363 trim_cache size 0 max 16384 2022-07-03T10:38:28.975 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2022-07-03T10:38:28.975 INFO:teuthology.orchestra.run.smithi101.stdout: "status": "no active scrubs running", 2022-07-03T10:38:28.975 INFO:teuthology.orchestra.run.smithi101.stdout: "scrubs": {} 2022-07-03T10:38:28.976 INFO:teuthology.orchestra.run.smithi101.stdout:} 2022-07-03T10:38:28.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7eff68096d20 msgr2=0x7eff680991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7eff68096d20 0x7eff680991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.978 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 msgr2=0x7eff7810d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-03T10:38:28.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00da20 tx=0x7eff6c00b3b0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.979 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 shutdown_connections 2022-07-03T10:38:28.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:6800/1865562803,v1:172.21.15.101:6801/1865562803] conn(0x7eff68096d20 0x7eff680991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff780fdf10 0x7eff78114a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.980 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff780fc4a0 0x7eff7810d470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 --2- 172.21.15.101:0/4003162161 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff780fcdf0 0x7eff7810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-07-03T10:38:28.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 >> 172.21.15.101:0/4003162161 conn(0x7eff780f7c50 msgr2=0x7eff78100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-03T10:38:28.981 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 shutdown_connections 2022-07-03T10:38:28.982 INFO:teuthology.orchestra.run.smithi101.stderr:2022-07-03T10:38:28.914+0000 7eff7f1ff700 1 -- 172.21.15.101:0/4003162161 wait complete. 2022-07-03T10:38:28.982 INFO:tasks.cephfs.filesystem:all active scrubs completed 2022-07-03T10:38:29.983 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2022-07-03T10:38:29.984 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2022-07-03T10:38:30.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2022-07-03T10:38:30.501 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/fwd_scrub.py", line 67, in _scrub out_json = self.fs.run_scrub(["start", path, ",".join(scrubopts)]) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1605, in run_scrub return self.rank_tell(["scrub"] + cmd, rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/cephfs/filesystem.py", line 1195, in rank_tell out = self.mon_manager.raw_cluster_cmd("tell", f"mds.{self.id}:{rank}", *command) File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/ceph_manager.py", line 1600, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_61829bae536fb4126cfaa87ca353b9576e018e7d/qa/tasks/ceph_manager.py", line 1591, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' 2022-07-03T10:38:56.802 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-07-03T10:38:56.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: osdmap e83: 12 total, 12 up, 12 in 2022-07-03T10:38:56.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: mgrmap e19: y(active, starting, since 0.0148961s), standbys: z 2022-07-03T10:38:56.803 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:38:56.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:38:56.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:38:56.804 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-07-03T10:38:56.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-07-03T10:38:56.805 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-07-03T10:38:56.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-07-03T10:38:56.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-07-03T10:38:56.806 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-07-03T10:38:56.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-07-03T10:38:56.807 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-07-03T10:38:56.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-07-03T10:38:56.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-07-03T10:38:56.808 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-07-03T10:38:56.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-07-03T10:38:56.809 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-07-03T10:38:56.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-07-03T10:38:56.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:38:56.810 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:38:56.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:38:56.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:38:56.811 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:38:56.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:38:56.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:38:56.812 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:38:56.813 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:38:56.813 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:38:56.813 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:38:56.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:38:56.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-07-03T10:38:56.814 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-07-03T10:38:56.815 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-07-03T10:38:56.815 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:56 smithi188 ceph-mon[92425]: Manager daemon y is now available 2022-07-03T10:38:56.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f725182b700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f725182b700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f725182b700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f725182b700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f725182b700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f724f827700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f724f827700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f724f827700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f724f827700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:38:56.626+0000 7f724f827700 -1 client.0 error registering admin socket command: (17) File exists 2022-07-03T10:38:56.836 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-07-03T10:38:56.836 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: osdmap e83: 12 total, 12 up, 12 in 2022-07-03T10:38:56.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: mgrmap e19: y(active, starting, since 0.0148961s), standbys: z 2022-07-03T10:38:56.837 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-07-03T10:38:56.838 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:38:56.838 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:38:56.838 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-07-03T10:38:56.839 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-07-03T10:38:56.839 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-07-03T10:38:56.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-07-03T10:38:56.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-07-03T10:38:56.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-07-03T10:38:56.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-07-03T10:38:56.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-07-03T10:38:56.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-07-03T10:38:56.842 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-07-03T10:38:56.842 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-07-03T10:38:56.843 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-07-03T10:38:56.843 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-07-03T10:38:56.843 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-07-03T10:38:56.844 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-07-03T10:38:56.844 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-07-03T10:38:56.845 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-07-03T10:38:56.845 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-07-03T10:38:56.846 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-07-03T10:38:56.846 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-07-03T10:38:56.846 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-07-03T10:38:56.847 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-07-03T10:38:56.847 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-07-03T10:38:56.847 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-07-03T10:38:56.848 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-07-03T10:38:56.848 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-07-03T10:38:56.848 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-07-03T10:38:56.849 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-07-03T10:38:56.849 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-07-03T10:38:56.850 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:56 smithi161 ceph-mon[91579]: Manager daemon y is now available 2022-07-03T10:38:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:38:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:38:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:38:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:38:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:38:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:38:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:38:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:38:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:57 smithi188 ceph-mon[92425]: mgrmap e20: y(active, since 1.01889s), standbys: z 2022-07-03T10:38:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:38:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:38:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:38:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:38:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:38:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:38:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:38:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:38:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:38:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:57 smithi161 ceph-mon[91579]: mgrmap e20: y(active, since 1.01889s), standbys: z 2022-07-03T10:38:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:58 smithi161 ceph-mon[91579]: [03/Jul/2022:10:38:57] ENGINE Bus STARTING 2022-07-03T10:38:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:58 smithi161 ceph-mon[91579]: [03/Jul/2022:10:38:57] ENGINE Serving on https://172.21.15.161:7150 2022-07-03T10:38:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:58 smithi161 ceph-mon[91579]: [03/Jul/2022:10:38:57] ENGINE Bus STARTED 2022-07-03T10:38:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:58 smithi188 ceph-mon[92425]: [03/Jul/2022:10:38:57] ENGINE Bus STARTING 2022-07-03T10:38:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:58 smithi188 ceph-mon[92425]: [03/Jul/2022:10:38:57] ENGINE Serving on https://172.21.15.161:7150 2022-07-03T10:38:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:58 smithi188 ceph-mon[92425]: [03/Jul/2022:10:38:57] ENGINE Bus STARTED 2022-07-03T10:38:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:38:59 smithi161 ceph-mon[91579]: mgrmap e21: y(active, since 2s), standbys: z 2022-07-03T10:38:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:38:59 smithi188 ceph-mon[92425]: mgrmap e21: y(active, since 2s), standbys: z 2022-07-03T10:39:01.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.120 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.121 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.121 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.122 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.123 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.124 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.125 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.127 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.127 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.128 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.129 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.129 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.129 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.174 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.175 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.176 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.176 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.177 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.178 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.178 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.179 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.179 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.179 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.180 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.180 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.180 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.181 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.181 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.181 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:01.182 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.182 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.182 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.183 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.183 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.184 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.184 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:01.184 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:39:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T10:39:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T10:39:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:39:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T10:39:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T10:39:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:39:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:39:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:39:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:39:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:39:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T10:39:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T10:39:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi188:/etc/ceph/ceph.conf 2022-07-03T10:39:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T10:39:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T10:39:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi161:/etc/ceph/ceph.conf 2022-07-03T10:39:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:39:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.conf 2022-07-03T10:39:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:39:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: Updating smithi161:/etc/ceph/ceph.client.admin.keyring 2022-07-03T10:39:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:39:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:03 smithi161 ceph-mon[91579]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:39:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:03 smithi161 ceph-mon[91579]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:39:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:03 smithi188 ceph-mon[92425]: Updating smithi188:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:39:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:03 smithi188 ceph-mon[92425]: Updating smithi161:/var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/config/ceph.client.admin.keyring 2022-07-03T10:39:16.630 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:16.269+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6806 osd.0 since back 2022-07-03T10:38:12.437738+0000 front 2022-07-03T10:38:12.437809+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:16.631 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:16.269+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6814 osd.3 since back 2022-07-03T10:38:12.437787+0000 front 2022-07-03T10:38:12.437698+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:16.631 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:16.269+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6822 osd.6 since back 2022-07-03T10:38:12.437845+0000 front 2022-07-03T10:38:12.437820+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:16.631 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:16.269+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6830 osd.9 since back 2022-07-03T10:38:12.437872+0000 front 2022-07-03T10:38:12.437921+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:16.632 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:39:16.505+0000 7f2c40033700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.101:6806 osd.0 since back 2022-07-03T10:38:12.417436+0000 front 2022-07-03T10:38:12.417422+0000 (oldest deadline 2022-07-03T10:39:15.917394+0000) 2022-07-03T10:39:16.632 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:39:16.505+0000 7f2c40033700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.101:6814 osd.3 since back 2022-07-03T10:38:12.417291+0000 front 2022-07-03T10:38:12.417280+0000 (oldest deadline 2022-07-03T10:39:15.917394+0000) 2022-07-03T10:39:16.633 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:39:16.505+0000 7f2c40033700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.101:6822 osd.6 since back 2022-07-03T10:38:12.417368+0000 front 2022-07-03T10:38:12.417358+0000 (oldest deadline 2022-07-03T10:39:15.917394+0000) 2022-07-03T10:39:16.633 INFO:journalctl@ceph.osd.10.smithi161.stdout:Jul 03 10:39:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-10[109881]: 2022-07-03T10:39:16.505+0000 7f2c40033700 -1 osd.10 83 heartbeat_check: no reply from 172.21.15.101:6830 osd.9 since back 2022-07-03T10:38:12.417449+0000 front 2022-07-03T10:38:12.417341+0000 (oldest deadline 2022-07-03T10:39:15.917394+0000) 2022-07-03T10:39:17.517 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.0 reported failed by osd.10 2022-07-03T10:39:17.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.3 reported failed by osd.10 2022-07-03T10:39:17.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.6 reported failed by osd.10 2022-07-03T10:39:17.518 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.9 reported failed by osd.10 2022-07-03T10:39:17.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: pgmap v4: 97 pgs: 63 active+clean, 34 unknown; 2.1 MiB data, 56 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:39:17.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.0 reported failed by osd.2 2022-07-03T10:39:17.519 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.0 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000277 >= grace 20.000000) 2022-07-03T10:39:17.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.3 reported failed by osd.2 2022-07-03T10:39:17.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.3 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000467 >= grace 20.000000) 2022-07-03T10:39:17.520 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.6 reported failed by osd.2 2022-07-03T10:39:17.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.6 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000631 >= grace 20.000000) 2022-07-03T10:39:17.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.9 reported failed by osd.2 2022-07-03T10:39:17.521 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.9 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000821 >= grace 20.000000) 2022-07-03T10:39:17.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.0 reported failed by osd.4 2022-07-03T10:39:17.522 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.3 reported failed by osd.4 2022-07-03T10:39:17.523 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:39:17.203+0000 7fcd28b4c700 -1 osd.2 83 heartbeat_check: no reply from 172.21.15.101:6806 osd.0 since back 2022-07-03T10:38:12.497651+0000 front 2022-07-03T10:38:12.497702+0000 (oldest deadline 2022-07-03T10:39:17.197798+0000) 2022-07-03T10:39:17.523 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:39:17.203+0000 7fcd28b4c700 -1 osd.2 83 heartbeat_check: no reply from 172.21.15.101:6814 osd.3 since back 2022-07-03T10:38:12.497766+0000 front 2022-07-03T10:38:12.497677+0000 (oldest deadline 2022-07-03T10:39:17.197798+0000) 2022-07-03T10:39:17.524 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:39:17.203+0000 7fcd28b4c700 -1 osd.2 83 heartbeat_check: no reply from 172.21.15.101:6822 osd.6 since back 2022-07-03T10:38:12.497926+0000 front 2022-07-03T10:38:12.497831+0000 (oldest deadline 2022-07-03T10:39:17.197798+0000) 2022-07-03T10:39:17.524 INFO:journalctl@ceph.osd.2.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-2[97208]: 2022-07-03T10:39:17.203+0000 7fcd28b4c700 -1 osd.2 83 heartbeat_check: no reply from 172.21.15.101:6830 osd.9 since back 2022-07-03T10:38:12.497879+0000 front 2022-07-03T10:38:12.497961+0000 (oldest deadline 2022-07-03T10:39:17.197798+0000) 2022-07-03T10:39:17.629 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:17.305+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6806 osd.0 since back 2022-07-03T10:38:12.437738+0000 front 2022-07-03T10:38:12.437809+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:17.630 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:17.305+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6814 osd.3 since back 2022-07-03T10:38:12.437787+0000 front 2022-07-03T10:38:12.437698+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:17.630 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:17.305+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6822 osd.6 since back 2022-07-03T10:38:12.437845+0000 front 2022-07-03T10:38:12.437820+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:17.631 INFO:journalctl@ceph.osd.4.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-osd-4[100667]: 2022-07-03T10:39:17.305+0000 7fddbdf59700 -1 osd.4 83 heartbeat_check: no reply from 172.21.15.101:6830 osd.9 since back 2022-07-03T10:38:12.437872+0000 front 2022-07-03T10:38:12.437921+0000 (oldest deadline 2022-07-03T10:39:15.937806+0000) 2022-07-03T10:39:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.0 reported failed by osd.10 2022-07-03T10:39:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.3 reported failed by osd.10 2022-07-03T10:39:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.6 reported failed by osd.10 2022-07-03T10:39:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.9 reported failed by osd.10 2022-07-03T10:39:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: pgmap v4: 97 pgs: 63 active+clean, 34 unknown; 2.1 MiB data, 56 MiB used, 715 GiB / 715 GiB avail 2022-07-03T10:39:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.0 reported failed by osd.2 2022-07-03T10:39:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.0 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000277 >= grace 20.000000) 2022-07-03T10:39:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.3 reported failed by osd.2 2022-07-03T10:39:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.3 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000467 >= grace 20.000000) 2022-07-03T10:39:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.6 reported failed by osd.2 2022-07-03T10:39:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.6 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000631 >= grace 20.000000) 2022-07-03T10:39:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.9 reported failed by osd.2 2022-07-03T10:39:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.9 failed (root=default,host=smithi101) (2 reporters from different osd after 64.000821 >= grace 20.000000) 2022-07-03T10:39:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.0 reported failed by osd.4 2022-07-03T10:39:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.3 reported failed by osd.4 2022-07-03T10:39:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.6 reported failed by osd.4 2022-07-03T10:39:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:17 smithi161 ceph-mon[91579]: osd.9 reported failed by osd.4 2022-07-03T10:39:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.6 reported failed by osd.4 2022-07-03T10:39:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:17 smithi188 ceph-mon[92425]: osd.9 reported failed by osd.4 2022-07-03T10:39:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:18 smithi161 ceph-mon[91579]: Health check failed: 4 osds down (OSD_DOWN) 2022-07-03T10:39:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:18 smithi161 ceph-mon[91579]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-07-03T10:39:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:18 smithi161 ceph-mon[91579]: osdmap e84: 12 total, 8 up, 12 in 2022-07-03T10:39:18.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:18 smithi161 ceph-mon[91579]: osdmap e85: 12 total, 8 up, 12 in 2022-07-03T10:39:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:18 smithi188 ceph-mon[92425]: Health check failed: 4 osds down (OSD_DOWN) 2022-07-03T10:39:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:18 smithi188 ceph-mon[92425]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-07-03T10:39:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:18 smithi188 ceph-mon[92425]: osdmap e84: 12 total, 8 up, 12 in 2022-07-03T10:39:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:18 smithi188 ceph-mon[92425]: osdmap e85: 12 total, 8 up, 12 in 2022-07-03T10:39:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:19 smithi161 ceph-mon[91579]: pgmap v7: 97 pgs: 54 peering, 5 unknown, 38 active+clean; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 258 KiB/s wr, 46 op/s; 0 B/s, 3 keys/s, 0 objects/s recovering 2022-07-03T10:39:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:19 smithi188 ceph-mon[92425]: pgmap v7: 97 pgs: 54 peering, 5 unknown, 38 active+clean; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 258 KiB/s wr, 46 op/s; 0 B/s, 3 keys/s, 0 objects/s recovering 2022-07-03T10:39:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:20 smithi161 ceph-mon[91579]: Health check failed: Reduced data availability: 7 pgs inactive, 29 pgs peering (PG_AVAILABILITY) 2022-07-03T10:39:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:20 smithi188 ceph-mon[92425]: Health check failed: Reduced data availability: 7 pgs inactive, 29 pgs peering (PG_AVAILABILITY) 2022-07-03T10:39:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:21 smithi161 ceph-mon[91579]: pgmap v8: 97 pgs: 54 peering, 5 unknown, 38 active+clean; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 129 KiB/s wr, 23 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:39:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:21 smithi188 ceph-mon[92425]: pgmap v8: 97 pgs: 54 peering, 5 unknown, 38 active+clean; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 129 KiB/s wr, 23 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:39:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:23 smithi161 ceph-mon[91579]: pgmap v9: 97 pgs: 33 active+clean, 3 active+undersized, 1 undersized+peered, 3 unknown, 43 peering, 4 undersized+degraded+peered, 10 active+undersized+degraded; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 88 KiB/s wr, 16 op/s; 23/330 objects degraded (6.970%); 0 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:39:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:23 smithi161 ceph-mon[91579]: Health check failed: Degraded data redundancy: 23/330 objects degraded (6.970%), 14 pgs degraded (PG_DEGRADED) 2022-07-03T10:39:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:23 smithi188 ceph-mon[92425]: pgmap v9: 97 pgs: 33 active+clean, 3 active+undersized, 1 undersized+peered, 3 unknown, 43 peering, 4 undersized+degraded+peered, 10 active+undersized+degraded; 2.5 MiB data, 56 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 88 KiB/s wr, 16 op/s; 23/330 objects degraded (6.970%); 0 B/s, 1 keys/s, 0 objects/s recovering 2022-07-03T10:39:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:23 smithi188 ceph-mon[92425]: Health check failed: Degraded data redundancy: 23/330 objects degraded (6.970%), 14 pgs degraded (PG_DEGRADED) 2022-07-03T10:39:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:24 smithi161 ceph-mon[91579]: Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:39:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:24 smithi188 ceph-mon[92425]: Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:39:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:25 smithi161 ceph-mon[91579]: pgmap v10: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 KiB/s wr, 13 op/s; 98/330 objects degraded (29.697%); 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:39:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:25 smithi188 ceph-mon[92425]: pgmap v10: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 KiB/s wr, 13 op/s; 98/330 objects degraded (29.697%); 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:39:27.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.060+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:39:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:39:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:39:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:39:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:39:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.061+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:39:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:39:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:39:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:39:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:39:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:39:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:39:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:27.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:39:27.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:27.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:39:27.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:27.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.063+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:27.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:27.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:39:27.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:27.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:27.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:27.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:39:27.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:27.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:39:27.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:27.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:39:27.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:27.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:39:27.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:27.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:39:27.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:27.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:27.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:27.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:27.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:27.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:27.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:27.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:39:27.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:27.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:39:27.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:27.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:39:27.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:27.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:39:27.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:27.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:27.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:27.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:39:27.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:27.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:39:27.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1068, in _write_client_files 2022-07-03T10:39:27.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-07-03T10:39:27.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 243, in write_remote_file 2022-07-03T10:39:27.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: host, path, content, mode, uid, gid, addr)) 2022-07-03T10:39:27.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:27.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:39:27.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:27.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:27.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:27.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:39:27.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:27.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:39:27.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 231, in _write_remote_file 2022-07-03T10:39:27.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:39:27.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-mon[91579]: pgmap v11: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 KiB/s wr, 11 op/s; 98/330 objects degraded (29.697%); 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:39:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:39:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:39:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:39:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:27 smithi188 ceph-mon[92425]: pgmap v11: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 KiB/s wr, 11 op/s; 98/330 objects degraded (29.697%); 0 B/s, 0 keys/s, 0 objects/s recovering 2022-07-03T10:39:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:39:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:39:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:39:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:39:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:39:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:39:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:39:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:39:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:39:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:39:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:28.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:28.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:28.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:28.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:39:28.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:28.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:28.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:28.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:28.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:39:28.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:39:28.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:39:28.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:28.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:28.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:28.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:28.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:28.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:28.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:28.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:39:28.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:28.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:28.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:28.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:28.923 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.923 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:28.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:39:28.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:28.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:39:28.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1068, in _write_client_files 2022-07-03T10:39:28.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-07-03T10:39:28.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 243, in write_remote_file 2022-07-03T10:39:28.926 INFO:journalctl@ceph.mon.b.smithi161.stdout: host, path, content, mode, uid, gid, addr)) 2022-07-03T10:39:28.926 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:28.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:39:28.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:28.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:28.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:28.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:39:28.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:28.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:39:28.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 231, in _write_remote_file 2022-07-03T10:39:28.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.930 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.930 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:28.930 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:28.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: Updating smithi101:/etc/ceph/ceph.conf 2022-07-03T10:39:28.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:39:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:39:28.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:39:28.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:39:28.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:39:28.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:39:28.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:28.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:28.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:28.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:28.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:28.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:28.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:28.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:39:28.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:28.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:28.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:28.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.950 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:28.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:28.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:39:28.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:28.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:28.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:28.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:39:28.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:28.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:39:28.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:28.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:39:28.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:28.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:28.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:28.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 209, in _write_remote_file 2022-07-03T10:39:28.958 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-07-03T10:39:28.958 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:28.958 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:28.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:28.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:28.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:28.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:39:28.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:28.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:28.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:28.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.962 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.962 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:28.962 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:28.963 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:28.963 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:28.963 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:39:28.964 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:28.964 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:39:28.964 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1068, in _write_client_files 2022-07-03T10:39:28.965 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-07-03T10:39:28.965 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 243, in write_remote_file 2022-07-03T10:39:28.965 INFO:journalctl@ceph.mon.c.smithi188.stdout: host, path, content, mode, uid, gid, addr)) 2022-07-03T10:39:28.966 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:28.966 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:39:28.966 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:28.967 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:28.967 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:28.967 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:39:28.968 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:28.968 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:39:28.968 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 231, in _write_remote_file 2022-07-03T10:39:28.969 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:28.969 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi101:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:28.969 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:28.970 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:29 smithi161 ceph-mon[91579]: pgmap v12: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 1.3 KiB/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:29 smithi188 ceph-mon[92425]: pgmap v12: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 1.3 KiB/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:31 smithi161 ceph-mon[91579]: pgmap v13: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1.1 KiB/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:31 smithi161 ceph-mon[91579]: Health check update: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded (PG_DEGRADED) 2022-07-03T10:39:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:31 smithi188 ceph-mon[92425]: pgmap v13: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1.1 KiB/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:31 smithi188 ceph-mon[92425]: Health check update: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded (PG_DEGRADED) 2022-07-03T10:39:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:33 smithi161 ceph-mon[91579]: pgmap v14: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:33 smithi188 ceph-mon[92425]: pgmap v14: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:35 smithi161 ceph-mon[91579]: pgmap v15: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:35 smithi188 ceph-mon[92425]: pgmap v15: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:37 smithi161 ceph-mon[91579]: pgmap v16: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:37 smithi188 ceph-mon[92425]: pgmap v16: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:39 smithi161 ceph-mon[91579]: pgmap v17: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:39 smithi188 ceph-mon[92425]: pgmap v17: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:41 smithi161 ceph-mon[91579]: pgmap v18: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:41 smithi188 ceph-mon[92425]: pgmap v18: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:43 smithi161 ceph-mon[91579]: pgmap v19: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:43 smithi188 ceph-mon[92425]: pgmap v19: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:45 smithi161 ceph-mon[91579]: pgmap v20: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:45 smithi188 ceph-mon[92425]: pgmap v20: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:47 smithi161 ceph-mon[91579]: pgmap v21: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:47 smithi188 ceph-mon[92425]: pgmap v21: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:49 smithi161 ceph-mon[91579]: pgmap v22: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:49 smithi188 ceph-mon[92425]: pgmap v22: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:51 smithi161 ceph-mon[91579]: pgmap v23: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:51 smithi188 ceph-mon[92425]: pgmap v23: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:53 smithi161 ceph-mon[91579]: pgmap v24: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:53 smithi188 ceph-mon[92425]: pgmap v24: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:55 smithi161 ceph-mon[91579]: pgmap v25: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:55 smithi188 ceph-mon[92425]: pgmap v25: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-mon[91579]: pgmap v26: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:39:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:39:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:39:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:39:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:57.509+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:39:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:39:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:39:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:39:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:39:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:57.510+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:39:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:39:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:39:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:39:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:39:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:39:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:39:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:39:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:39:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:39:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:39:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:39:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:39:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:39:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:39:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:39:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:39:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:57.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:57.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:57.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:39:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:39:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:57.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:39:57.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:39:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:57 smithi188 ceph-mon[92425]: pgmap v26: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:39:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:39:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:39:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:39:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:39:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:39:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:39:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:39:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:39:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:39:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:39:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:39:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:39:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:39:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:39:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:39:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:39:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:39:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:39:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:39:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:39:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:39:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:39:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:39:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:39:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:39:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:39:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:39:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:39:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:39:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:39:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:39:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:39:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:39:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:39:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:39:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:39:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:39:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:39:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:39:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:39:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:39:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:39:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:39:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:39:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:39:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:39:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:39:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:39:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:39:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:39:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:39:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:39:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:39:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:39:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:39:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:39:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:39:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:39:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:39:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:39:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:39:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:39:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:39:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:39:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:39:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:39:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:39:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:39:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:39:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:39:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:39:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:39:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:39:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:39:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:39:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:39:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:39:59 smithi161 ceph-mon[91579]: pgmap v27: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:39:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:39:59 smithi188 ceph-mon[92425]: pgmap v27: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 7 pgs inactive; Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded 2022-07-03T10:40:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:40:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:40:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: [WRN] OSD_DOWN: 4 osds down 2022-07-03T10:40:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: osd.0 (root=default,host=smithi101) is down 2022-07-03T10:40:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: osd.3 (root=default,host=smithi101) is down 2022-07-03T10:40:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: osd.6 (root=default,host=smithi101) is down 2022-07-03T10:40:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: osd.9 (root=default,host=smithi101) is down 2022-07-03T10:40:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-07-03T10:40:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: host smithi101 (root=default) (4 osds) is down 2022-07-03T10:40:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: [WRN] PG_AVAILABILITY: Reduced data availability: 7 pgs inactive 2022-07-03T10:40:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.1 is stuck inactive for 62s, current state unknown, last acting [] 2022-07-03T10:40:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.4 is stuck inactive for 4m, current state undersized+peered, last acting [2] 2022-07-03T10:40:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.9 is stuck inactive for 4m, current state undersized+peered, last acting [10] 2022-07-03T10:40:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.19 is stuck inactive for 4m, current state undersized+peered, last acting [4] 2022-07-03T10:40:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.1e is stuck inactive for 4m, current state undersized+peered, last acting [5] 2022-07-03T10:40:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.12 is stuck inactive for 4m, current state undersized+peered, last acting [7] 2022-07-03T10:40:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.18 is stuck inactive for 4m, current state undersized+peered, last acting [11] 2022-07-03T10:40:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: [WRN] PG_DEGRADED: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded 2022-07-03T10:40:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.6 is undersized+degraded+peered, acting [11] 2022-07-03T10:40:00.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.7 is active+undersized+degraded, acting [4,11] 2022-07-03T10:40:00.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.d is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.e is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.10 is active+undersized+degraded, acting [8,2] 2022-07-03T10:40:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.13 is active+undersized+degraded, acting [11,10] 2022-07-03T10:40:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.15 is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 2.1f is active+undersized+degraded, acting [7,8] 2022-07-03T10:40:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3 is active+undersized+degraded, acting [10,4] 2022-07-03T10:40:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.4 is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.5 is undersized+degraded+peered, acting [10] 2022-07-03T10:40:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.7 is active+undersized+degraded, acting [11,7] 2022-07-03T10:40:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.a is active+undersized+degraded, acting [7,1] 2022-07-03T10:40:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.c is active+undersized+degraded, acting [7,5] 2022-07-03T10:40:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.13 is active+undersized+degraded, acting [7,4] 2022-07-03T10:40:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.16 is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.17 is active+undersized+degraded, acting [4,5] 2022-07-03T10:40:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.19 is undersized+degraded+peered, acting [10] 2022-07-03T10:40:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1a is active+undersized+degraded, acting [8,5] 2022-07-03T10:40:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1b is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1c is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1e is active+undersized+degraded, acting [2,5] 2022-07-03T10:40:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.1f is undersized+degraded+peered, acting [1] 2022-07-03T10:40:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.21 is active+undersized+degraded, acting [2,4] 2022-07-03T10:40:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.22 is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.23 is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.24 is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.27 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.28 is undersized+degraded+peered, acting [5] 2022-07-03T10:40:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.29 is active+undersized+degraded, acting [2,11] 2022-07-03T10:40:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.2a is active+undersized+degraded, acting [1,7] 2022-07-03T10:40:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.2b is undersized+degraded+peered, acting [7] 2022-07-03T10:40:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.2d is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.2e is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.30 is active+undersized+degraded, acting [1,7] 2022-07-03T10:40:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.31 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.33 is active+undersized+degraded, acting [10,5] 2022-07-03T10:40:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.38 is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3a is active+undersized+degraded, acting [10,4] 2022-07-03T10:40:00.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3b is undersized+degraded+peered, acting [8] 2022-07-03T10:40:00.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3d is active+undersized+degraded, acting [2,1] 2022-07-03T10:40:00.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3e is active+undersized+degraded, acting [11,10] 2022-07-03T10:40:00.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:00 smithi161 ceph-mon[91579]: pg 3.3f is active+undersized+degraded, acting [11,1] 2022-07-03T10:40:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 7 pgs inactive; Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded 2022-07-03T10:40:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:40:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:40:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: [WRN] OSD_DOWN: 4 osds down 2022-07-03T10:40:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: osd.0 (root=default,host=smithi101) is down 2022-07-03T10:40:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: osd.3 (root=default,host=smithi101) is down 2022-07-03T10:40:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: osd.6 (root=default,host=smithi101) is down 2022-07-03T10:40:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: osd.9 (root=default,host=smithi101) is down 2022-07-03T10:40:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-07-03T10:40:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: host smithi101 (root=default) (4 osds) is down 2022-07-03T10:40:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: [WRN] PG_AVAILABILITY: Reduced data availability: 7 pgs inactive 2022-07-03T10:40:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.1 is stuck inactive for 62s, current state unknown, last acting [] 2022-07-03T10:40:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.4 is stuck inactive for 4m, current state undersized+peered, last acting [2] 2022-07-03T10:40:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.9 is stuck inactive for 4m, current state undersized+peered, last acting [10] 2022-07-03T10:40:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.19 is stuck inactive for 4m, current state undersized+peered, last acting [4] 2022-07-03T10:40:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.1e is stuck inactive for 4m, current state undersized+peered, last acting [5] 2022-07-03T10:40:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.12 is stuck inactive for 4m, current state undersized+peered, last acting [7] 2022-07-03T10:40:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.18 is stuck inactive for 4m, current state undersized+peered, last acting [11] 2022-07-03T10:40:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: [WRN] PG_DEGRADED: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded 2022-07-03T10:40:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.6 is undersized+degraded+peered, acting [11] 2022-07-03T10:40:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.7 is active+undersized+degraded, acting [4,11] 2022-07-03T10:40:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.d is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.e is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.10 is active+undersized+degraded, acting [8,2] 2022-07-03T10:40:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.13 is active+undersized+degraded, acting [11,10] 2022-07-03T10:40:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.15 is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 2.1f is active+undersized+degraded, acting [7,8] 2022-07-03T10:40:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3 is active+undersized+degraded, acting [10,4] 2022-07-03T10:40:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.4 is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.5 is undersized+degraded+peered, acting [10] 2022-07-03T10:40:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.7 is active+undersized+degraded, acting [11,7] 2022-07-03T10:40:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.a is active+undersized+degraded, acting [7,1] 2022-07-03T10:40:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.c is active+undersized+degraded, acting [7,5] 2022-07-03T10:40:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.13 is active+undersized+degraded, acting [7,4] 2022-07-03T10:40:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.16 is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.17 is active+undersized+degraded, acting [4,5] 2022-07-03T10:40:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.19 is undersized+degraded+peered, acting [10] 2022-07-03T10:40:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1a is active+undersized+degraded, acting [8,5] 2022-07-03T10:40:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1b is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1c is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1e is active+undersized+degraded, acting [2,5] 2022-07-03T10:40:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.1f is undersized+degraded+peered, acting [1] 2022-07-03T10:40:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.21 is active+undersized+degraded, acting [2,4] 2022-07-03T10:40:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.22 is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.23 is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.24 is undersized+degraded+peered, acting [4] 2022-07-03T10:40:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.27 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.28 is undersized+degraded+peered, acting [5] 2022-07-03T10:40:00.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.29 is active+undersized+degraded, acting [2,11] 2022-07-03T10:40:00.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.2a is active+undersized+degraded, acting [1,7] 2022-07-03T10:40:00.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.2b is undersized+degraded+peered, acting [7] 2022-07-03T10:40:00.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.2d is undersized+degraded+peered, acting [2] 2022-07-03T10:40:00.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.2e is active+undersized+degraded, acting [1,10] 2022-07-03T10:40:00.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.30 is active+undersized+degraded, acting [1,7] 2022-07-03T10:40:00.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.31 is active+undersized+degraded, acting [4,2] 2022-07-03T10:40:00.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.33 is active+undersized+degraded, acting [10,5] 2022-07-03T10:40:00.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.38 is active+undersized+degraded, acting [5,10] 2022-07-03T10:40:00.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3a is active+undersized+degraded, acting [10,4] 2022-07-03T10:40:00.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3b is undersized+degraded+peered, acting [8] 2022-07-03T10:40:00.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3d is active+undersized+degraded, acting [2,1] 2022-07-03T10:40:00.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3e is active+undersized+degraded, acting [11,10] 2022-07-03T10:40:00.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:00 smithi188 ceph-mon[92425]: pg 3.3f is active+undersized+degraded, acting [11,1] 2022-07-03T10:40:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:01 smithi161 ceph-mon[91579]: pgmap v28: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:01 smithi188 ceph-mon[92425]: pgmap v28: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:03 smithi161 ceph-mon[91579]: pgmap v29: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:03 smithi188 ceph-mon[92425]: pgmap v29: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:05 smithi161 ceph-mon[91579]: pgmap v30: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:05 smithi188 ceph-mon[92425]: pgmap v30: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:07 smithi161 ceph-mon[91579]: pgmap v31: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:07 smithi188 ceph-mon[92425]: pgmap v31: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:09 smithi161 ceph-mon[91579]: pgmap v32: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:09 smithi188 ceph-mon[92425]: pgmap v32: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:11 smithi161 ceph-mon[91579]: pgmap v33: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:11 smithi188 ceph-mon[92425]: pgmap v33: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:13 smithi161 ceph-mon[91579]: pgmap v34: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:13 smithi188 ceph-mon[92425]: pgmap v34: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:15 smithi161 ceph-mon[91579]: pgmap v35: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:15 smithi188 ceph-mon[92425]: pgmap v35: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:17 smithi161 ceph-mon[91579]: pgmap v36: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:17 smithi188 ceph-mon[92425]: pgmap v36: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:19 smithi161 ceph-mon[91579]: pgmap v37: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:19 smithi188 ceph-mon[92425]: pgmap v37: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:20 smithi161 ceph-mon[91579]: Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:40:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:20 smithi161 ceph-mon[91579]: Health check update: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2022-07-03T10:40:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:20 smithi188 ceph-mon[92425]: Health check update: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2022-07-03T10:40:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:20 smithi188 ceph-mon[92425]: Health check update: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded, 67 pgs undersized (PG_DEGRADED) 2022-07-03T10:40:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:21 smithi161 ceph-mon[91579]: pgmap v38: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:21 smithi188 ceph-mon[92425]: pgmap v38: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:23 smithi161 ceph-mon[91579]: pgmap v39: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:23 smithi188 ceph-mon[92425]: pgmap v39: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:25 smithi161 ceph-mon[91579]: pgmap v40: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:25 smithi188 ceph-mon[92425]: pgmap v40: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:27 smithi161 ceph-mon[91579]: pgmap v41: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:27 smithi188 ceph-mon[92425]: pgmap v41: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:29 smithi161 ceph-mon[91579]: mgrmap e22: y(active, since 92s), standbys: z 2022-07-03T10:40:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:29 smithi161 ceph-mon[91579]: pgmap v42: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:29 smithi188 ceph-mon[92425]: mgrmap e22: y(active, since 92s), standbys: z 2022-07-03T10:40:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:29 smithi188 ceph-mon[92425]: pgmap v42: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:31 smithi161 ceph-mon[91579]: pgmap v43: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:31 smithi188 ceph-mon[92425]: pgmap v43: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:33 smithi161 ceph-mon[91579]: pgmap v44: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:33 smithi188 ceph-mon[92425]: pgmap v44: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:35 smithi161 ceph-mon[91579]: pgmap v45: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:35 smithi188 ceph-mon[92425]: pgmap v45: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:37 smithi161 ceph-mon[91579]: pgmap v46: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:37 smithi188 ceph-mon[92425]: pgmap v46: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:39 smithi161 ceph-mon[91579]: pgmap v47: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:39 smithi188 ceph-mon[92425]: pgmap v47: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:41 smithi161 ceph-mon[91579]: pgmap v48: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:41 smithi188 ceph-mon[92425]: pgmap v48: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:43 smithi161 ceph-mon[91579]: pgmap v49: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:43 smithi188 ceph-mon[92425]: pgmap v49: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:45 smithi161 ceph-mon[91579]: pgmap v50: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:45 smithi188 ceph-mon[92425]: pgmap v50: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:47 smithi161 ceph-mon[91579]: pgmap v51: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:47 smithi188 ceph-mon[92425]: pgmap v51: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:49 smithi161 ceph-mon[91579]: pgmap v52: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:49 smithi188 ceph-mon[92425]: pgmap v52: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:51 smithi161 ceph-mon[91579]: pgmap v53: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:51 smithi188 ceph-mon[92425]: pgmap v53: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:53 smithi161 ceph-mon[91579]: pgmap v54: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:53 smithi188 ceph-mon[92425]: pgmap v54: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:55 smithi161 ceph-mon[91579]: pgmap v55: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:55 smithi188 ceph-mon[92425]: pgmap v55: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: pgmap v56: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:40:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:40:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:40:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:40:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:40:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:40:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:40:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: pgmap v56: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:40:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:40:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:40:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:40:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:40:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:40:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:40:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:40:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:40:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:40:59 smithi161 ceph-mon[91579]: pgmap v57: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:40:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:40:59 smithi188 ceph-mon[92425]: pgmap v57: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:01 smithi161 ceph-mon[91579]: pgmap v58: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:01 smithi188 ceph-mon[92425]: pgmap v58: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:03 smithi161 ceph-mon[91579]: pgmap v59: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:03 smithi188 ceph-mon[92425]: pgmap v59: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:04 smithi161 ceph-mon[91579]: pgmap v60: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:04 smithi188 ceph-mon[92425]: pgmap v60: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:07 smithi161 ceph-mon[91579]: pgmap v61: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:07 smithi188 ceph-mon[92425]: pgmap v61: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:09 smithi161 ceph-mon[91579]: pgmap v62: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:09 smithi188 ceph-mon[92425]: pgmap v62: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:11 smithi161 ceph-mon[91579]: pgmap v63: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:11 smithi188 ceph-mon[92425]: pgmap v63: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:13 smithi161 ceph-mon[91579]: pgmap v64: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:13 smithi188 ceph-mon[92425]: pgmap v64: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:14 smithi161 ceph-mon[91579]: pgmap v65: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:14 smithi188 ceph-mon[92425]: pgmap v65: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:17 smithi161 ceph-mon[91579]: pgmap v66: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:17 smithi188 ceph-mon[92425]: pgmap v66: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:19 smithi161 ceph-mon[91579]: pgmap v67: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:19 smithi188 ceph-mon[92425]: pgmap v67: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:21 smithi161 ceph-mon[91579]: pgmap v68: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:21 smithi188 ceph-mon[92425]: pgmap v68: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:22 smithi161 ceph-mon[91579]: pgmap v69: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:22 smithi188 ceph-mon[92425]: pgmap v69: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:24 smithi161 ceph-mon[91579]: pgmap v70: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:24 smithi188 ceph-mon[92425]: pgmap v70: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:27 smithi161 ceph-mon[91579]: pgmap v71: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:27 smithi188 ceph-mon[92425]: pgmap v71: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:41:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:41:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:29 smithi161 ceph-mon[91579]: pgmap v72: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:29 smithi188 ceph-mon[92425]: pgmap v72: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:30 smithi161 ceph-mon[91579]: pgmap v73: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:30 smithi188 ceph-mon[92425]: pgmap v73: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:33 smithi161 ceph-mon[91579]: pgmap v74: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:33 smithi188 ceph-mon[92425]: pgmap v74: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:35 smithi161 ceph-mon[91579]: pgmap v75: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:35 smithi188 ceph-mon[92425]: pgmap v75: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:37 smithi161 ceph-mon[91579]: pgmap v76: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:37 smithi188 ceph-mon[92425]: pgmap v76: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:39 smithi161 ceph-mon[91579]: pgmap v77: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:39 smithi188 ceph-mon[92425]: pgmap v77: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:40 smithi161 ceph-mon[91579]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-07-03T10:41:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:40 smithi188 ceph-mon[92425]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-07-03T10:41:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:41 smithi161 ceph-mon[91579]: pgmap v78: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:41 smithi188 ceph-mon[92425]: pgmap v78: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:43 smithi161 ceph-mon[91579]: pgmap v79: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:43 smithi188 ceph-mon[92425]: pgmap v79: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:44 smithi188 ceph-mon[92425]: pgmap v80: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:44 smithi161 ceph-mon[91579]: pgmap v80: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:47 smithi161 ceph-mon[91579]: pgmap v81: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:47 smithi188 ceph-mon[92425]: pgmap v81: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:49 smithi161 ceph-mon[91579]: pgmap v82: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:49 smithi188 ceph-mon[92425]: pgmap v82: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:50 smithi161 ceph-mon[91579]: pgmap v83: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:51.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:50 smithi188 ceph-mon[92425]: pgmap v83: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:52 smithi161 ceph-mon[91579]: pgmap v84: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:52 smithi188 ceph-mon[92425]: pgmap v84: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:54 smithi161 ceph-mon[91579]: pgmap v85: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:54 smithi188 ceph-mon[92425]: pgmap v85: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: pgmap v86: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:41:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:41:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:41:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:41:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:41:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:41:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: pgmap v86: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:41:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:41:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:41:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:41:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:41:57.550+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:41:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:41:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:41:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:41:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:41:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:41:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:41:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:41:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:41:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:41:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:41:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:41:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:41:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:41:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:41:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:41:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:41:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:41:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:41:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:41:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:41:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:41:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:41:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:41:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:41:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:41:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:41:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:41:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:41:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:41:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:41:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:41:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:41:57.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:41:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:41:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:41:57.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:41:57.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:41:57.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:41:57.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:41:57.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:41:57.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:41:57.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:41:57.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:41:57.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:41:57.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=7] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:41:57.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=7] Local address: 172.21.15.161, port 50746 2022-07-03T10:41:57.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=7] Peer address: 172.21.15.101, port 22 2022-07-03T10:41:57.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=7] Login timeout expired 2022-07-03T10:41:57.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=7] Aborting connection 2022-07-03T10:41:57.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:41:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:41:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:41:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:41:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:41:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:41:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:41:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:41:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:41:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:41:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:41:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:41:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:41:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:41:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:41:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:41:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:41:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:41:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:41:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:41:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:41:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:41:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:41:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:41:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:41:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:41:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:41:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:41:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:41:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:41:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:41:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:41:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:41:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:41:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:41:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:41:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:41:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:41:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:41:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:41:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:41:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:41:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:41:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:41:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:41:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:41:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:41:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=7] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:41:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=7] Local address: 172.21.15.161, port 50746 2022-07-03T10:41:58.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=7] Peer address: 172.21.15.101, port 22 2022-07-03T10:41:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=7] Login timeout expired 2022-07-03T10:41:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=7] Aborting connection 2022-07-03T10:41:58.910 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:58.910 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:41:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:41:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:41:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:41:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:41:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:41:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:41:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:41:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:41:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:41:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:41:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:41:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:41:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:41:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:41:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:41:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:41:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:41:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:41:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:41:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:41:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:41:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:41:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:41:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:41:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:41:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:41:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:41:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:41:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:41:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:41:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:41:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:41:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:41:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:41:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:41:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:41:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:41:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:41:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:41:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:41:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:41:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:41:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:41:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:41:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=7] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:41:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=7] Local address: 172.21.15.161, port 50746 2022-07-03T10:41:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=7] Peer address: 172.21.15.101, port 22 2022-07-03T10:41:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=7] Login timeout expired 2022-07-03T10:41:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=7] Aborting connection 2022-07-03T10:41:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:41:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:41:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:41:59 smithi161 ceph-mon[91579]: pgmap v87: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:41:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:41:59 smithi188 ceph-mon[92425]: pgmap v87: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:01 smithi161 ceph-mon[91579]: pgmap v88: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:01 smithi188 ceph-mon[92425]: pgmap v88: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:02 smithi161 ceph-mon[91579]: pgmap v89: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:02 smithi188 ceph-mon[92425]: pgmap v89: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:04 smithi188 ceph-mon[92425]: pgmap v90: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:04 smithi161 ceph-mon[91579]: pgmap v90: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:07 smithi161 ceph-mon[91579]: pgmap v91: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:07 smithi188 ceph-mon[92425]: pgmap v91: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:09 smithi161 ceph-mon[91579]: pgmap v92: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:09 smithi188 ceph-mon[92425]: pgmap v92: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:10 smithi161 ceph-mon[91579]: pgmap v93: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:10 smithi188 ceph-mon[92425]: pgmap v93: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:12 smithi161 ceph-mon[91579]: pgmap v94: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:12 smithi188 ceph-mon[92425]: pgmap v94: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:14 smithi161 ceph-mon[91579]: pgmap v95: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:14 smithi188 ceph-mon[92425]: pgmap v95: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:16 smithi161 ceph-mon[91579]: pgmap v96: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:16 smithi188 ceph-mon[92425]: pgmap v96: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:19 smithi161 ceph-mon[91579]: pgmap v97: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:19 smithi188 ceph-mon[92425]: pgmap v97: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:21 smithi161 ceph-mon[91579]: pgmap v98: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:21 smithi188 ceph-mon[92425]: pgmap v98: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:22 smithi188 ceph-mon[92425]: pgmap v99: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:22 smithi161 ceph-mon[91579]: pgmap v99: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:24 smithi188 ceph-mon[92425]: pgmap v100: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:24 smithi161 ceph-mon[91579]: pgmap v100: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:27 smithi161 ceph-mon[91579]: pgmap v101: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:27 smithi188 ceph-mon[92425]: pgmap v101: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:29 smithi161 ceph-mon[91579]: pgmap v102: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:29 smithi188 ceph-mon[92425]: pgmap v102: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:30 smithi161 ceph-mon[91579]: pgmap v103: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:30 smithi188 ceph-mon[92425]: pgmap v103: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:32 smithi161 ceph-mon[91579]: pgmap v104: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:32 smithi188 ceph-mon[92425]: pgmap v104: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:34 smithi161 ceph-mon[91579]: pgmap v105: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:34 smithi188 ceph-mon[92425]: pgmap v105: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:36 smithi161 ceph-mon[91579]: pgmap v106: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:36 smithi188 ceph-mon[92425]: pgmap v106: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:39 smithi161 ceph-mon[91579]: pgmap v107: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:39 smithi188 ceph-mon[92425]: pgmap v107: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:41 smithi161 ceph-mon[91579]: pgmap v108: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:41 smithi188 ceph-mon[92425]: pgmap v108: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:42 smithi188 ceph-mon[92425]: pgmap v109: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:42 smithi161 ceph-mon[91579]: pgmap v109: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:44 smithi161 ceph-mon[91579]: pgmap v110: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:44 smithi188 ceph-mon[92425]: pgmap v110: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:47 smithi161 ceph-mon[91579]: pgmap v111: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:47 smithi188 ceph-mon[92425]: pgmap v111: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:49 smithi161 ceph-mon[91579]: pgmap v112: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:49 smithi188 ceph-mon[92425]: pgmap v112: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:50 smithi161 ceph-mon[91579]: pgmap v113: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:50 smithi188 ceph-mon[92425]: pgmap v113: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:52 smithi161 ceph-mon[91579]: pgmap v114: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:52 smithi188 ceph-mon[92425]: pgmap v114: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:54 smithi161 ceph-mon[91579]: pgmap v115: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:54 smithi188 ceph-mon[92425]: pgmap v115: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:42:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:42:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:42:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:42:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:56 smithi161 ceph-mon[91579]: pgmap v116: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:42:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:42:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:42:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:42:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:42:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:42:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:56 smithi188 ceph-mon[92425]: pgmap v116: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:42:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:42:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:42:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:42:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:42:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:42:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:42:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:42:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:42:59 smithi161 ceph-mon[91579]: pgmap v117: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:42:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:42:59 smithi188 ceph-mon[92425]: pgmap v117: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:01 smithi161 ceph-mon[91579]: pgmap v118: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:01 smithi188 ceph-mon[92425]: pgmap v118: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:02 smithi188 ceph-mon[92425]: pgmap v119: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:02 smithi161 ceph-mon[91579]: pgmap v119: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:04 smithi161 ceph-mon[91579]: pgmap v120: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:04 smithi188 ceph-mon[92425]: pgmap v120: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:07 smithi161 ceph-mon[91579]: pgmap v121: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:07 smithi188 ceph-mon[92425]: pgmap v121: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:09 smithi161 ceph-mon[91579]: pgmap v122: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:09 smithi188 ceph-mon[92425]: pgmap v122: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:10 smithi161 ceph-mon[91579]: pgmap v123: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:10 smithi188 ceph-mon[92425]: pgmap v123: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:12 smithi161 ceph-mon[91579]: pgmap v124: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:12 smithi188 ceph-mon[92425]: pgmap v124: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:14 smithi161 ceph-mon[91579]: pgmap v125: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:14 smithi188 ceph-mon[92425]: pgmap v125: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:16 smithi161 ceph-mon[91579]: pgmap v126: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:16 smithi188 ceph-mon[92425]: pgmap v126: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:19 smithi161 ceph-mon[91579]: pgmap v127: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:19 smithi188 ceph-mon[92425]: pgmap v127: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:21 smithi161 ceph-mon[91579]: pgmap v128: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:21 smithi188 ceph-mon[92425]: pgmap v128: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:22 smithi188 ceph-mon[92425]: pgmap v129: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:22 smithi161 ceph-mon[91579]: pgmap v129: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:24 smithi161 ceph-mon[91579]: pgmap v130: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 10 up:standby 2022-07-03T10:43:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:24 smithi188 ceph-mon[92425]: pgmap v130: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:26 smithi161 ceph-mon[91579]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:26 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 9 up:standby 2022-07-03T10:43:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:26 smithi188 ceph-mon[92425]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:26 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 9 up:standby 2022-07-03T10:43:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:27 smithi161 ceph-mon[91579]: pgmap v131: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:27 smithi188 ceph-mon[92425]: pgmap v131: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:43:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:43:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:29 smithi161 ceph-mon[91579]: pgmap v132: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:29 smithi188 ceph-mon[92425]: pgmap v132: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:30 smithi161 ceph-mon[91579]: pgmap v133: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:30 smithi188 ceph-mon[92425]: pgmap v133: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:32.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:31 smithi161 ceph-mon[91579]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:31 smithi161 ceph-mon[91579]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:31 smithi161 ceph-mon[91579]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:31 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:31 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:31 smithi188 ceph-mon[92425]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:31 smithi188 ceph-mon[92425]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:31 smithi188 ceph-mon[92425]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-07-03T10:43:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:31 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:31 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:32 smithi161 ceph-mon[91579]: pgmap v134: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:32 smithi188 ceph-mon[92425]: pgmap v134: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:34 smithi161 ceph-mon[91579]: pgmap v135: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:34 smithi188 ceph-mon[92425]: pgmap v135: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:36 smithi161 ceph-mon[91579]: pgmap v136: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:36 smithi188 ceph-mon[92425]: pgmap v136: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:39 smithi161 ceph-mon[91579]: pgmap v137: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:39 smithi188 ceph-mon[92425]: pgmap v137: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:41 smithi161 ceph-mon[91579]: pgmap v138: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:41 smithi188 ceph-mon[92425]: pgmap v138: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:42 smithi161 ceph-mon[91579]: pgmap v139: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:42 smithi188 ceph-mon[92425]: pgmap v139: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:44 smithi161 ceph-mon[91579]: pgmap v140: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:44 smithi188 ceph-mon[92425]: pgmap v140: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:47 smithi161 ceph-mon[91579]: pgmap v141: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:47 smithi188 ceph-mon[92425]: pgmap v141: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:49 smithi161 ceph-mon[91579]: pgmap v142: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:49 smithi188 ceph-mon[92425]: pgmap v142: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:50 smithi161 ceph-mon[91579]: pgmap v143: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:50 smithi188 ceph-mon[92425]: pgmap v143: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:52 smithi161 ceph-mon[91579]: pgmap v144: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:52 smithi188 ceph-mon[92425]: pgmap v144: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:54 smithi161 ceph-mon[91579]: pgmap v145: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:54 smithi188 ceph-mon[92425]: pgmap v145: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:43:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:43:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:43:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:43:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:56 smithi161 ceph-mon[91579]: pgmap v146: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:43:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:43:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:43:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:43:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:43:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:43:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:56 smithi188 ceph-mon[92425]: pgmap v146: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:43:57.588+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:43:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:43:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:43:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:43:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:43:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:43:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:43:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:43:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:43:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:43:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:43:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:43:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:43:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:43:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:43:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:43:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:43:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:43:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:43:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:43:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:43:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:43:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:43:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:43:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:43:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:43:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:43:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:43:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:43:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:43:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:43:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:43:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:43:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:43:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:43:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:43:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:43:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:43:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:43:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:43:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:43:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:43:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:43:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:43:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:43:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=9] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:43:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=9] Local address: 172.21.15.161, port 50798 2022-07-03T10:43:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=9] Peer address: 172.21.15.101, port 22 2022-07-03T10:43:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=9] Login timeout expired 2022-07-03T10:43:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=9] Aborting connection 2022-07-03T10:43:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:43:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:43:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:43:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:43:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:43:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:43:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:43:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:43:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:43:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:43:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:43:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:43:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:43:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:43:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:43:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:43:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:43:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:43:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:43:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:43:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:43:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:43:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:43:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:43:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:43:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:43:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:43:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:43:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:43:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:43:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:43:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:43:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:43:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:43:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:43:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:43:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:43:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:43:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:43:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:43:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:43:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:43:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:43:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:43:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:43:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:43:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:43:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=9] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:43:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=9] Local address: 172.21.15.161, port 50798 2022-07-03T10:43:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=9] Peer address: 172.21.15.101, port 22 2022-07-03T10:43:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=9] Login timeout expired 2022-07-03T10:43:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=9] Aborting connection 2022-07-03T10:43:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:43:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:43:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:43:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:43:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:43:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:43:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:43:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:43:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:43:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:43:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:43:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:43:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:43:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:43:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:43:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:43:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:43:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:43:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:43:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:43:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:43:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:43:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:43:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:43:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:43:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:43:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:43:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:43:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:43:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:43:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:43:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:43:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:43:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:43:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:43:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:43:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:43:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:43:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:43:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:43:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:43:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:43:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:43:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:43:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:43:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=9] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:43:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=9] Local address: 172.21.15.161, port 50798 2022-07-03T10:43:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=9] Peer address: 172.21.15.101, port 22 2022-07-03T10:43:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=9] Login timeout expired 2022-07-03T10:43:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=9] Aborting connection 2022-07-03T10:43:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:43:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:43:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:43:59 smithi161 ceph-mon[91579]: pgmap v147: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:43:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:43:59 smithi188 ceph-mon[92425]: pgmap v147: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:01 smithi161 ceph-mon[91579]: pgmap v148: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:01 smithi188 ceph-mon[92425]: pgmap v148: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:02 smithi161 ceph-mon[91579]: pgmap v149: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:02 smithi188 ceph-mon[92425]: pgmap v149: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:04 smithi161 ceph-mon[91579]: pgmap v150: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:04 smithi188 ceph-mon[92425]: pgmap v150: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:07 smithi161 ceph-mon[91579]: pgmap v151: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:07 smithi188 ceph-mon[92425]: pgmap v151: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:09 smithi161 ceph-mon[91579]: pgmap v152: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:09 smithi188 ceph-mon[92425]: pgmap v152: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:10 smithi161 ceph-mon[91579]: pgmap v153: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:10 smithi188 ceph-mon[92425]: pgmap v153: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:12 smithi161 ceph-mon[91579]: pgmap v154: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:12 smithi188 ceph-mon[92425]: pgmap v154: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:15 smithi161 ceph-mon[91579]: pgmap v155: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:15 smithi188 ceph-mon[92425]: pgmap v155: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:18 smithi161 ceph-mon[91579]: pgmap v156: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:18 smithi188 ceph-mon[92425]: pgmap v156: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:20 smithi161 ceph-mon[91579]: pgmap v157: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:20 smithi188 ceph-mon[92425]: pgmap v157: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:22 smithi161 ceph-mon[91579]: pgmap v158: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:22 smithi188 ceph-mon[92425]: pgmap v158: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:24 smithi161 ceph-mon[91579]: pgmap v159: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:24 smithi188 ceph-mon[92425]: pgmap v159: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:26 smithi161 ceph-mon[91579]: pgmap v160: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:26 smithi188 ceph-mon[92425]: pgmap v160: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:28 smithi161 ceph-mon[91579]: pgmap v161: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:28 smithi188 ceph-mon[92425]: pgmap v161: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:30 smithi161 ceph-mon[91579]: pgmap v162: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:30 smithi188 ceph-mon[92425]: pgmap v162: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:32 smithi161 ceph-mon[91579]: pgmap v163: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:32 smithi188 ceph-mon[92425]: pgmap v163: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:34 smithi161 ceph-mon[91579]: pgmap v164: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:34 smithi188 ceph-mon[92425]: pgmap v164: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:36 smithi161 ceph-mon[91579]: pgmap v165: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:36 smithi188 ceph-mon[92425]: pgmap v165: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:38 smithi188 ceph-mon[92425]: pgmap v166: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:38 smithi161 ceph-mon[91579]: pgmap v166: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:40 smithi188 ceph-mon[92425]: pgmap v167: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:40 smithi161 ceph-mon[91579]: pgmap v167: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:42 smithi188 ceph-mon[92425]: pgmap v168: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:42 smithi161 ceph-mon[91579]: pgmap v168: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:44 smithi188 ceph-mon[92425]: pgmap v169: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:44 smithi161 ceph-mon[91579]: pgmap v169: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:46 smithi188 ceph-mon[92425]: pgmap v170: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:46 smithi161 ceph-mon[91579]: pgmap v170: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:48 smithi188 ceph-mon[92425]: pgmap v171: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:48 smithi161 ceph-mon[91579]: pgmap v171: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:49 smithi161 ceph-mon[91579]: pgmap v172: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:49 smithi188 ceph-mon[92425]: pgmap v172: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:52 smithi161 ceph-mon[91579]: pgmap v173: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:52 smithi188 ceph-mon[92425]: pgmap v173: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:54 smithi161 ceph-mon[91579]: pgmap v174: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:54 smithi188 ceph-mon[92425]: pgmap v174: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:56 smithi161 ceph-mon[91579]: pgmap v175: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:56 smithi188 ceph-mon[92425]: pgmap v175: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:44:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:44:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:44:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:44:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:44:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:44:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:44:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:44:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:44:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:44:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:44:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:44:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:58 smithi161 ceph-mon[91579]: pgmap v176: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:44:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:44:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:44:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:58 smithi188 ceph-mon[92425]: pgmap v176: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:44:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:44:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:44:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:44:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:00 smithi161 ceph-mon[91579]: pgmap v177: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:00 smithi188 ceph-mon[92425]: pgmap v177: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:02 smithi161 ceph-mon[91579]: pgmap v178: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:02 smithi188 ceph-mon[92425]: pgmap v178: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:04 smithi161 ceph-mon[91579]: pgmap v179: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:04 smithi188 ceph-mon[92425]: pgmap v179: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:06 smithi161 ceph-mon[91579]: pgmap v180: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:06 smithi188 ceph-mon[92425]: pgmap v180: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:08 smithi188 ceph-mon[92425]: pgmap v181: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:08 smithi161 ceph-mon[91579]: pgmap v181: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:10 smithi188 ceph-mon[92425]: pgmap v182: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:10 smithi161 ceph-mon[91579]: pgmap v182: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:12 smithi188 ceph-mon[92425]: pgmap v183: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:12 smithi161 ceph-mon[91579]: pgmap v183: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:14 smithi188 ceph-mon[92425]: pgmap v184: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:14 smithi161 ceph-mon[91579]: pgmap v184: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:16 smithi188 ceph-mon[92425]: pgmap v185: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:16 smithi161 ceph-mon[91579]: pgmap v185: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:18 smithi188 ceph-mon[92425]: pgmap v186: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:18 smithi161 ceph-mon[91579]: pgmap v186: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:19 smithi161 ceph-mon[91579]: pgmap v187: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:19 smithi188 ceph-mon[92425]: pgmap v187: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:22 smithi161 ceph-mon[91579]: pgmap v188: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:22 smithi188 ceph-mon[92425]: pgmap v188: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:24 smithi161 ceph-mon[91579]: pgmap v189: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:24 smithi188 ceph-mon[92425]: pgmap v189: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:26 smithi161 ceph-mon[91579]: pgmap v190: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:26 smithi188 ceph-mon[92425]: pgmap v190: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:28 smithi161 ceph-mon[91579]: pgmap v191: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:45:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:28 smithi188 ceph-mon[92425]: pgmap v191: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:45:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:30 smithi161 ceph-mon[91579]: pgmap v192: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:30 smithi188 ceph-mon[92425]: pgmap v192: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:32 smithi161 ceph-mon[91579]: pgmap v193: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:32 smithi188 ceph-mon[92425]: pgmap v193: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:34 smithi161 ceph-mon[91579]: pgmap v194: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:34 smithi188 ceph-mon[92425]: pgmap v194: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:36 smithi161 ceph-mon[91579]: pgmap v195: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:36 smithi188 ceph-mon[92425]: pgmap v195: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:38 smithi188 ceph-mon[92425]: pgmap v196: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:38 smithi161 ceph-mon[91579]: pgmap v196: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:40 smithi188 ceph-mon[92425]: pgmap v197: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:40 smithi161 ceph-mon[91579]: pgmap v197: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:42 smithi188 ceph-mon[92425]: pgmap v198: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:42 smithi161 ceph-mon[91579]: pgmap v198: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:44 smithi188 ceph-mon[92425]: pgmap v199: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:44 smithi161 ceph-mon[91579]: pgmap v199: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:46 smithi188 ceph-mon[92425]: pgmap v200: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:46 smithi161 ceph-mon[91579]: pgmap v200: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:48 smithi188 ceph-mon[92425]: pgmap v201: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:48 smithi161 ceph-mon[91579]: pgmap v201: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:49 smithi161 ceph-mon[91579]: pgmap v202: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:49 smithi188 ceph-mon[92425]: pgmap v202: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:52 smithi161 ceph-mon[91579]: pgmap v203: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:52 smithi188 ceph-mon[92425]: pgmap v203: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:54 smithi161 ceph-mon[91579]: pgmap v204: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:54 smithi188 ceph-mon[92425]: pgmap v204: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:56 smithi161 ceph-mon[91579]: pgmap v205: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:56 smithi188 ceph-mon[92425]: pgmap v205: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:45:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:45:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:45:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:45:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:45:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:45:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:45:57.641+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:45:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:45:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:45:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:45:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:45:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:45:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:45:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:45:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:45:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:45:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:45:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:45:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:45:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:45:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:45:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:45:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:45:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:45:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:45:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:45:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:45:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:45:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:45:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:45:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:45:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:45:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:45:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:45:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:45:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:45:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:45:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:45:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:45:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:45:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:45:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:45:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:45:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:45:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:45:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:45:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:45:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:45:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:45:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:45:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:45:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=11] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:45:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=11] Local address: 172.21.15.161, port 50850 2022-07-03T10:45:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=11] Peer address: 172.21.15.101, port 22 2022-07-03T10:45:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=11] Login timeout expired 2022-07-03T10:45:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=11] Aborting connection 2022-07-03T10:45:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:45:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:45:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:45:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:45:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:45:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:45:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:58 smithi161 ceph-mon[91579]: pgmap v206: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:45:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:45:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:45:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:45:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:45:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:45:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:45:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:45:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:45:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:45:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:45:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:45:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:45:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:45:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:45:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:45:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:45:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:45:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:45:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:45:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:45:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:45:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:45:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:45:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:45:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:45:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:45:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:45:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:45:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:45:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:45:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:45:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:45:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:45:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:45:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:45:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:45:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:45:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:45:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:45:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:45:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:45:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:45:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:45:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:45:58.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=11] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:45:58.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=11] Local address: 172.21.15.161, port 50850 2022-07-03T10:45:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=11] Peer address: 172.21.15.101, port 22 2022-07-03T10:45:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=11] Login timeout expired 2022-07-03T10:45:58.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=11] Aborting connection 2022-07-03T10:45:58.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:58.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:45:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:58 smithi188 ceph-mon[92425]: pgmap v206: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:45:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:45:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:45:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:45:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:45:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:45:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:45:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:45:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:45:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:45:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:45:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:45:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:45:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:45:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:45:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:45:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:45:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:45:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:45:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:45:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:45:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:45:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:45:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:45:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:45:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:45:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:45:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:45:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:45:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:45:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:45:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:45:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:45:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:45:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:45:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:45:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:45:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:45:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:45:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:45:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:45:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:45:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:45:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:45:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=11] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:45:58.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=11] Local address: 172.21.15.161, port 50850 2022-07-03T10:45:58.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=11] Peer address: 172.21.15.101, port 22 2022-07-03T10:45:58.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=11] Login timeout expired 2022-07-03T10:45:58.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=11] Aborting connection 2022-07-03T10:45:58.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:45:58.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:00 smithi161 ceph-mon[91579]: pgmap v207: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:00 smithi188 ceph-mon[92425]: pgmap v207: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:02 smithi161 ceph-mon[91579]: pgmap v208: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:02 smithi188 ceph-mon[92425]: pgmap v208: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:04 smithi188 ceph-mon[92425]: pgmap v209: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:04 smithi161 ceph-mon[91579]: pgmap v209: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:06 smithi188 ceph-mon[92425]: pgmap v210: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:06 smithi161 ceph-mon[91579]: pgmap v210: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:08 smithi188 ceph-mon[92425]: pgmap v211: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:08 smithi161 ceph-mon[91579]: pgmap v211: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:10 smithi188 ceph-mon[92425]: pgmap v212: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:10 smithi161 ceph-mon[91579]: pgmap v212: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:12 smithi188 ceph-mon[92425]: pgmap v213: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:12 smithi161 ceph-mon[91579]: pgmap v213: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:14 smithi188 ceph-mon[92425]: pgmap v214: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:14 smithi161 ceph-mon[91579]: pgmap v214: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:16 smithi161 ceph-mon[91579]: pgmap v215: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:16 smithi188 ceph-mon[92425]: pgmap v215: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:17 smithi161 ceph-mon[91579]: pgmap v216: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:17 smithi188 ceph-mon[92425]: pgmap v216: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:19 smithi161 ceph-mon[91579]: pgmap v217: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:19 smithi188 ceph-mon[92425]: pgmap v217: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:22 smithi161 ceph-mon[91579]: pgmap v218: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:22 smithi188 ceph-mon[92425]: pgmap v218: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:24 smithi161 ceph-mon[91579]: pgmap v219: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:24 smithi188 ceph-mon[92425]: pgmap v219: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:26 smithi161 ceph-mon[91579]: pgmap v220: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:26 smithi188 ceph-mon[92425]: pgmap v220: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:28 smithi161 ceph-mon[91579]: pgmap v221: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:28 smithi188 ceph-mon[92425]: pgmap v221: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:30 smithi161 ceph-mon[91579]: pgmap v222: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:30 smithi188 ceph-mon[92425]: pgmap v222: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:32 smithi188 ceph-mon[92425]: pgmap v223: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:33.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:32 smithi161 ceph-mon[91579]: pgmap v223: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:33.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:33.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:34 smithi188 ceph-mon[92425]: pgmap v224: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:34 smithi161 ceph-mon[91579]: pgmap v224: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:36 smithi188 ceph-mon[92425]: pgmap v225: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:36 smithi161 ceph-mon[91579]: pgmap v225: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:38 smithi188 ceph-mon[92425]: pgmap v226: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:38 smithi161 ceph-mon[91579]: pgmap v226: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:40 smithi188 ceph-mon[92425]: pgmap v227: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:40.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:40 smithi161 ceph-mon[91579]: pgmap v227: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:42 smithi188 ceph-mon[92425]: pgmap v228: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:42 smithi161 ceph-mon[91579]: pgmap v228: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:44 smithi161 ceph-mon[91579]: pgmap v229: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:44 smithi188 ceph-mon[92425]: pgmap v229: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:46 smithi161 ceph-mon[91579]: pgmap v230: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:46 smithi188 ceph-mon[92425]: pgmap v230: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:48 smithi161 ceph-mon[91579]: pgmap v231: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:48 smithi188 ceph-mon[92425]: pgmap v231: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:49 smithi161 ceph-mon[91579]: pgmap v232: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:49 smithi188 ceph-mon[92425]: pgmap v232: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:52 smithi161 ceph-mon[91579]: pgmap v233: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:52 smithi188 ceph-mon[92425]: pgmap v233: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:54 smithi161 ceph-mon[91579]: pgmap v234: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:54 smithi188 ceph-mon[92425]: pgmap v234: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: pgmap v235: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:46:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:46:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:46:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:46:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: pgmap v235: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:46:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:46:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:46:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:46:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:46:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:46:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: pgmap v236: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:46:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:46:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:46:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:46:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: pgmap v236: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:46:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:46:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:46:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:00 smithi188 ceph-mon[92425]: pgmap v237: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:00 smithi161 ceph-mon[91579]: pgmap v237: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:02 smithi188 ceph-mon[92425]: pgmap v238: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:02 smithi161 ceph-mon[91579]: pgmap v238: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:04 smithi188 ceph-mon[92425]: pgmap v239: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:04 smithi161 ceph-mon[91579]: pgmap v239: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:06 smithi188 ceph-mon[92425]: pgmap v240: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:06 smithi161 ceph-mon[91579]: pgmap v240: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:08 smithi161 ceph-mon[91579]: pgmap v241: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:08 smithi188 ceph-mon[92425]: pgmap v241: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:10 smithi161 ceph-mon[91579]: pgmap v242: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:10 smithi188 ceph-mon[92425]: pgmap v242: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:12 smithi161 ceph-mon[91579]: pgmap v243: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:12 smithi188 ceph-mon[92425]: pgmap v243: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:14.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:14.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:14 smithi161 ceph-mon[91579]: pgmap v244: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:14 smithi188 ceph-mon[92425]: pgmap v244: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:15 smithi161 ceph-mon[91579]: pgmap v245: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:15 smithi188 ceph-mon[92425]: pgmap v245: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:17 smithi161 ceph-mon[91579]: pgmap v246: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:17 smithi188 ceph-mon[92425]: pgmap v246: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:19 smithi161 ceph-mon[91579]: pgmap v247: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:19 smithi188 ceph-mon[92425]: pgmap v247: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:21.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:22 smithi161 ceph-mon[91579]: pgmap v248: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:22 smithi188 ceph-mon[92425]: pgmap v248: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:24 smithi161 ceph-mon[91579]: pgmap v249: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:24 smithi188 ceph-mon[92425]: pgmap v249: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:26 smithi161 ceph-mon[91579]: pgmap v250: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:26 smithi188 ceph-mon[92425]: pgmap v250: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:28.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:47:28.095+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:28.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:47:28.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:28.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:47:28.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:28.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:28.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:28.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:47:28.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:28.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:47:28.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:28.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:47:28.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:28.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:47:28.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:28.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:47:28.096+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:47:28.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:47:28.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:28.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:47:28.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:28.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:28.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:28.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:47:28.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:28.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:47:28.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:28.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:47:28.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:28.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:47:28.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:28.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:47:28.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:47:28.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:47:28.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:47:28.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:47:28.115 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:47:28.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:47:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:47:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:47:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:47:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:47:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:47:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:47:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:47:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:47:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:47:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:47:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:47:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:47:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:47:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:47:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:47:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:47:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:47:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:47:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:47:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:47:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:47:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:47:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:47:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-mon[91579]: pgmap v251: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:47:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:28 smithi188 ceph-mon[92425]: pgmap v251: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:47:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:29 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:47:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:47:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:47:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:47:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:47:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:47:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:47:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:47:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:47:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:47:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:47:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:47:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:47:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:47:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:47:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:47:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:47:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:47:29.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:47:29.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:47:29.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:47:29.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:47:29.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:47:29.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:47:29.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:47:29.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:47:29.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:47:29.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:47:29.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:47:29.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:47:29.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:47:29.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:47:29.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:47:29.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:47:29.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:47:29.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:47:29.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:47:29.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:47:29.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:47:29.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:47:29.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:47:29.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:47:29.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:47:29.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:29 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:47:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:47:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:47:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:47:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:47:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:47:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:47:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:47:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:47:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:47:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:47:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:47:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:47:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:47:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:47:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:47:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:47:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:47:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:47:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:47:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:47:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:47:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:47:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:47:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:47:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:47:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:47:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:47:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:47:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:47:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:47:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:47:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:47:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:47:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:47:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:47:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:47:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:47:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:47:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:47:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:47:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:47:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:47:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:47:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:47:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:47:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:47:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:47:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:47:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:47:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:47:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:47:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:30 smithi188 ceph-mon[92425]: pgmap v252: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:30 smithi161 ceph-mon[91579]: pgmap v252: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:32 smithi188 ceph-mon[92425]: pgmap v253: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:32 smithi161 ceph-mon[91579]: pgmap v253: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:34 smithi188 ceph-mon[92425]: pgmap v254: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:34 smithi161 ceph-mon[91579]: pgmap v254: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:36 smithi188 ceph-mon[92425]: pgmap v255: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:36 smithi161 ceph-mon[91579]: pgmap v255: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:38 smithi188 ceph-mon[92425]: pgmap v256: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:38 smithi161 ceph-mon[91579]: pgmap v256: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:40 smithi161 ceph-mon[91579]: pgmap v257: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:40 smithi188 ceph-mon[92425]: pgmap v257: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:42 smithi161 ceph-mon[91579]: pgmap v258: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:42 smithi188 ceph-mon[92425]: pgmap v258: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:44 smithi161 ceph-mon[91579]: pgmap v259: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:44 smithi188 ceph-mon[92425]: pgmap v259: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:45 smithi161 ceph-mon[91579]: pgmap v260: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:45 smithi188 ceph-mon[92425]: pgmap v260: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:48 smithi161 ceph-mon[91579]: pgmap v261: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:48 smithi188 ceph-mon[92425]: pgmap v261: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:50 smithi161 ceph-mon[91579]: pgmap v262: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:50 smithi188 ceph-mon[92425]: pgmap v262: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:52 smithi188 ceph-mon[92425]: pgmap v263: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:52 smithi161 ceph-mon[91579]: pgmap v263: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:54 smithi188 ceph-mon[92425]: pgmap v264: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:54 smithi161 ceph-mon[91579]: pgmap v264: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: pgmap v265: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:47:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:47:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:47:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:47:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: pgmap v265: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:47:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:47:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:47:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:47:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:47:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:47:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:58 smithi188 ceph-mon[92425]: pgmap v266: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:47:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:58 smithi161 ceph-mon[91579]: pgmap v266: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:47:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:47:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:00 smithi188 ceph-mon[92425]: pgmap v267: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:00 smithi161 ceph-mon[91579]: pgmap v267: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:02 smithi188 ceph-mon[92425]: pgmap v268: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:02 smithi161 ceph-mon[91579]: pgmap v268: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:04 smithi161 ceph-mon[91579]: pgmap v269: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:04 smithi188 ceph-mon[92425]: pgmap v269: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:06 smithi161 ceph-mon[91579]: pgmap v270: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:06 smithi188 ceph-mon[92425]: pgmap v270: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:08 smithi161 ceph-mon[91579]: pgmap v271: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:08 smithi188 ceph-mon[92425]: pgmap v271: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:10 smithi161 ceph-mon[91579]: pgmap v272: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:10 smithi188 ceph-mon[92425]: pgmap v272: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:12 smithi161 ceph-mon[91579]: pgmap v273: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:12 smithi188 ceph-mon[92425]: pgmap v273: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:14 smithi161 ceph-mon[91579]: pgmap v274: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:14 smithi188 ceph-mon[92425]: pgmap v274: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:15 smithi161 ceph-mon[91579]: pgmap v275: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:15 smithi188 ceph-mon[92425]: pgmap v275: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:17 smithi161 ceph-mon[91579]: pgmap v276: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:17 smithi188 ceph-mon[92425]: pgmap v276: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:19 smithi161 ceph-mon[91579]: pgmap v277: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:19 smithi188 ceph-mon[92425]: pgmap v277: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:22 smithi188 ceph-mon[92425]: pgmap v278: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:22 smithi161 ceph-mon[91579]: pgmap v278: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:24 smithi188 ceph-mon[92425]: pgmap v279: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:24 smithi161 ceph-mon[91579]: pgmap v279: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:26 smithi188 ceph-mon[92425]: pgmap v280: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:26 smithi161 ceph-mon[91579]: pgmap v280: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: pgmap v281: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:48:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:48:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:48:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: pgmap v281: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:48:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:48:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:48:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:30 smithi188 ceph-mon[92425]: pgmap v282: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:30 smithi161 ceph-mon[91579]: pgmap v282: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:32 smithi188 ceph-mon[92425]: pgmap v283: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:32 smithi161 ceph-mon[91579]: pgmap v283: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:34 smithi188 ceph-mon[92425]: pgmap v284: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:34 smithi161 ceph-mon[91579]: pgmap v284: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:36 smithi161 ceph-mon[91579]: pgmap v285: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:36 smithi188 ceph-mon[92425]: pgmap v285: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:38 smithi161 ceph-mon[91579]: pgmap v286: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:38 smithi188 ceph-mon[92425]: pgmap v286: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:40 smithi161 ceph-mon[91579]: pgmap v287: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:40 smithi188 ceph-mon[92425]: pgmap v287: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:42 smithi161 ceph-mon[91579]: pgmap v288: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:42 smithi188 ceph-mon[92425]: pgmap v288: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:44 smithi161 ceph-mon[91579]: pgmap v289: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:44 smithi188 ceph-mon[92425]: pgmap v289: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:45 smithi161 ceph-mon[91579]: pgmap v290: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:45 smithi188 ceph-mon[92425]: pgmap v290: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:48 smithi188 ceph-mon[92425]: pgmap v291: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:48 smithi161 ceph-mon[91579]: pgmap v291: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:50 smithi188 ceph-mon[92425]: pgmap v292: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:50 smithi161 ceph-mon[91579]: pgmap v292: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:52 smithi188 ceph-mon[92425]: pgmap v293: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:52 smithi161 ceph-mon[91579]: pgmap v293: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:54 smithi188 ceph-mon[92425]: pgmap v294: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:54 smithi161 ceph-mon[91579]: pgmap v294: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: pgmap v295: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:48:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:48:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:48:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:48:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: pgmap v295: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:48:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:48:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:48:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:48:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:48:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:48:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:58 smithi161 ceph-mon[91579]: pgmap v296: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:48:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:58 smithi188 ceph-mon[92425]: pgmap v296: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:48:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:48:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:48:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:49:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:48:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:49:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:00 smithi161 ceph-mon[91579]: pgmap v297: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:00 smithi188 ceph-mon[92425]: pgmap v297: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:02 smithi161 ceph-mon[91579]: pgmap v298: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:02 smithi188 ceph-mon[92425]: pgmap v298: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:04 smithi161 ceph-mon[91579]: pgmap v299: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:04 smithi188 ceph-mon[92425]: pgmap v299: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:06 smithi161 ceph-mon[91579]: pgmap v300: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:06 smithi188 ceph-mon[92425]: pgmap v300: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:08 smithi161 ceph-mon[91579]: pgmap v301: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:08 smithi188 ceph-mon[92425]: pgmap v301: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:10 smithi161 ceph-mon[91579]: pgmap v302: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:10 smithi188 ceph-mon[92425]: pgmap v302: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:12 smithi161 ceph-mon[91579]: pgmap v303: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:12 smithi188 ceph-mon[92425]: pgmap v303: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:13 smithi161 ceph-mon[91579]: pgmap v304: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:13 smithi188 ceph-mon[92425]: pgmap v304: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:15 smithi161 ceph-mon[91579]: pgmap v305: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:15 smithi188 ceph-mon[92425]: pgmap v305: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:16 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:16 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:16 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:16 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:18 smithi188 ceph-mon[92425]: pgmap v306: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:18 smithi161 ceph-mon[91579]: pgmap v306: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:20 smithi188 ceph-mon[92425]: pgmap v307: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:20 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:20 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:20 smithi161 ceph-mon[91579]: pgmap v307: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:20 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:20 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Marking osd.0 out (has been down for 604 seconds) 2022-07-03T10:49:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Marking osd.3 out (has been down for 604 seconds) 2022-07-03T10:49:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Marking osd.6 out (has been down for 604 seconds) 2022-07-03T10:49:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Marking osd.9 out (has been down for 604 seconds) 2022-07-03T10:49:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Health check cleared: OSD_DOWN (was: 4 osds down) 2022-07-03T10:49:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-07-03T10:49:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:21 smithi161 ceph-mon[91579]: osdmap e86: 12 total, 8 up, 8 in 2022-07-03T10:49:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Marking osd.0 out (has been down for 604 seconds) 2022-07-03T10:49:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Marking osd.3 out (has been down for 604 seconds) 2022-07-03T10:49:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Marking osd.6 out (has been down for 604 seconds) 2022-07-03T10:49:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Marking osd.9 out (has been down for 604 seconds) 2022-07-03T10:49:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Health check cleared: OSD_DOWN (was: 4 osds down) 2022-07-03T10:49:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-07-03T10:49:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:21 smithi188 ceph-mon[92425]: osdmap e86: 12 total, 8 up, 8 in 2022-07-03T10:49:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:22 smithi161 ceph-mon[91579]: pgmap v308: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:22 smithi161 ceph-mon[91579]: osdmap e87: 12 total, 8 up, 8 in 2022-07-03T10:49:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:22 smithi188 ceph-mon[92425]: pgmap v308: 97 pgs: 29 active+clean, 14 active+undersized, 9 undersized+peered, 1 unknown, 14 undersized+degraded+peered, 30 active+undersized+degraded; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 1 op/s; 98/330 objects degraded (29.697%) 2022-07-03T10:49:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:22 smithi188 ceph-mon[92425]: osdmap e87: 12 total, 8 up, 8 in 2022-07-03T10:49:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:24 smithi161 ceph-mon[91579]: pgmap v311: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:24 smithi161 ceph-mon[91579]: Health check update: Reduced data availability: 23 pgs inactive, 64 pgs peering (PG_AVAILABILITY) 2022-07-03T10:49:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:24 smithi161 ceph-mon[91579]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded, 67 pgs undersized) 2022-07-03T10:49:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:24 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:24 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:24 smithi188 ceph-mon[92425]: pgmap v311: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:24 smithi188 ceph-mon[92425]: Health check update: Reduced data availability: 23 pgs inactive, 64 pgs peering (PG_AVAILABILITY) 2022-07-03T10:49:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:24 smithi188 ceph-mon[92425]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 98/330 objects degraded (29.697%), 44 pgs degraded, 67 pgs undersized) 2022-07-03T10:49:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:24 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:24 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:26 smithi161 ceph-mon[91579]: pgmap v312: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:26 smithi188 ceph-mon[92425]: pgmap v312: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:49:28.134+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:49:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:49:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:49:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:49:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:49:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:49:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:49:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:49:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:49:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:49:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:49:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:49:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:49:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:49:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:49:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:49:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:49:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:49:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:49:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:49:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:49:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:49:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:49:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:49:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:49:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:49:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:49:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:49:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:49:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:49:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:49:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:49:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:49:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:49:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:49:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:49:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:49:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:49:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:49:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:49:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:49:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:49:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:49:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:49:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:49:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=15] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:49:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=15] Local address: 172.21.15.161, port 50944 2022-07-03T10:49:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=15] Peer address: 172.21.15.101, port 22 2022-07-03T10:49:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=15] Login timeout expired 2022-07-03T10:49:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=15] Aborting connection 2022-07-03T10:49:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:49:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-mon[91579]: pgmap v313: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s 2022-07-03T10:49:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:28 smithi188 ceph-mon[92425]: pgmap v313: 97 pgs: 67 peering, 29 active+clean, 1 unknown; 2.5 MiB data, 57 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1 op/s 2022-07-03T10:49:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:49:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:49:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:49:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:49:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:49:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:49:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:49:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:49:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:49:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:49:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:49:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:49:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:49:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:49:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:49:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:49:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:49:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:49:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:49:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:49:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:49:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:49:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:49:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:49:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:49:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:49:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:49:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:49:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:49:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:49:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:49:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:49:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:49:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:49:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:49:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:49:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:49:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:49:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:49:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:49:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:49:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:49:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:49:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:49:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:49:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=15] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:49:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=15] Local address: 172.21.15.161, port 50944 2022-07-03T10:49:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=15] Peer address: 172.21.15.101, port 22 2022-07-03T10:49:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=15] Login timeout expired 2022-07-03T10:49:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=15] Aborting connection 2022-07-03T10:49:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:49:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:49:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:49:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:49:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:49:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:49:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:49:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:49:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:49:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:49:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:49:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:49:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:49:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:49:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:49:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:49:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:49:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:49:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:49:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:49:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:49:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:49:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:49:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:49:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:49:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:49:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:49:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:49:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:49:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:49:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:49:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:49:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:49:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:49:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:49:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:49:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:49:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:49:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:49:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:49:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:49:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:49:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:49:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:49:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:49:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=15] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:49:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=15] Local address: 172.21.15.161, port 50944 2022-07-03T10:49:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=15] Peer address: 172.21.15.101, port 22 2022-07-03T10:49:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=15] Login timeout expired 2022-07-03T10:49:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=15] Aborting connection 2022-07-03T10:49:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:30 smithi161 ceph-mon[91579]: pgmap v314: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 84 KiB/s, 3 keys/s, 12 objects/s recovering 2022-07-03T10:49:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:30 smithi161 ceph-mon[91579]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-07-03T10:49:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:30 smithi188 ceph-mon[92425]: pgmap v314: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 84 KiB/s, 3 keys/s, 12 objects/s recovering 2022-07-03T10:49:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:30 smithi188 ceph-mon[92425]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-07-03T10:49:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:32 smithi161 ceph-mon[91579]: pgmap v315: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2 op/s; 74 KiB/s, 2 keys/s, 10 objects/s recovering 2022-07-03T10:49:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:32 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:32 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:32 smithi188 ceph-mon[92425]: pgmap v315: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2 op/s; 74 KiB/s, 2 keys/s, 10 objects/s recovering 2022-07-03T10:49:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:32 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:32 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:33 smithi161 ceph-mon[91579]: pgmap v316: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 2.6 KiB/s wr, 2 op/s; 67 KiB/s, 3 keys/s, 9 objects/s recovering 2022-07-03T10:49:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:33 smithi188 ceph-mon[92425]: pgmap v316: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1008 B/s rd, 2.6 KiB/s wr, 2 op/s; 67 KiB/s, 3 keys/s, 9 objects/s recovering 2022-07-03T10:49:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:35 smithi161 ceph-mon[91579]: pgmap v317: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 1 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:35 smithi188 ceph-mon[92425]: pgmap v317: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 1 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:36 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:36 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:36 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:36 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:38 smithi188 ceph-mon[92425]: pgmap v318: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 1 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:38 smithi161 ceph-mon[91579]: pgmap v318: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.2 KiB/s wr, 1 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:40 smithi188 ceph-mon[92425]: pgmap v319: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:40 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:40 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:40 smithi161 ceph-mon[91579]: pgmap v319: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 56 KiB/s, 3 keys/s, 8 objects/s recovering 2022-07-03T10:49:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:40 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:40 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:42 smithi188 ceph-mon[92425]: pgmap v320: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:42 smithi161 ceph-mon[91579]: pgmap v320: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:44 smithi161 ceph-mon[91579]: pgmap v321: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:44 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:44 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:44 smithi188 ceph-mon[92425]: pgmap v321: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:44 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:44 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:46 smithi161 ceph-mon[91579]: pgmap v322: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:46 smithi188 ceph-mon[92425]: pgmap v322: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:48 smithi161 ceph-mon[91579]: pgmap v323: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:48 smithi188 ceph-mon[92425]: pgmap v323: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:50 smithi161 ceph-mon[91579]: pgmap v324: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:50 smithi188 ceph-mon[92425]: pgmap v324: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:52 smithi161 ceph-mon[91579]: pgmap v325: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:52 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:52 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:52 smithi188 ceph-mon[92425]: pgmap v325: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:52 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:52 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:54 smithi161 ceph-mon[91579]: pgmap v326: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:54 smithi188 ceph-mon[92425]: pgmap v326: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:49:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: pgmap v327: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:49:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:49:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:49:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:49:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: pgmap v327: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:49:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:49:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:49:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:49:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:49:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:49:58.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:49:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:49:57.737+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T10:49:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:58 smithi161 ceph-mon[91579]: pgmap v328: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:49:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:58 smithi188 ceph-mon[92425]: pgmap v328: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:49:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T10:49:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: pgmap v329: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T10:50:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-07-03T10:50:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: mds.l(mds.0): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 680 secs 2022-07-03T10:50:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:50:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:50:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-07-03T10:50:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:00 smithi161 ceph-mon[91579]: pg 2.1 is stuck inactive for 11m, current state unknown, last acting [] 2022-07-03T10:50:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: pgmap v329: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T10:50:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-07-03T10:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: mds.l(mds.0): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 680 secs 2022-07-03T10:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T10:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T10:50:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-07-03T10:50:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:00 smithi188 ceph-mon[92425]: pg 2.1 is stuck inactive for 11m, current state unknown, last acting [] 2022-07-03T10:50:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:02 smithi188 ceph-mon[92425]: pgmap v330: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:02 smithi161 ceph-mon[91579]: pgmap v330: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:04 smithi188 ceph-mon[92425]: pgmap v331: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:04 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:04 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:04 smithi161 ceph-mon[91579]: pgmap v331: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:04 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:04 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:06 smithi161 ceph-mon[91579]: pgmap v332: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:06 smithi188 ceph-mon[92425]: pgmap v332: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:08 smithi161 ceph-mon[91579]: pgmap v333: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:08 smithi188 ceph-mon[92425]: pgmap v333: 97 pgs: 64 active+clean+wait, 32 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:10 smithi161 ceph-mon[91579]: pgmap v334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2022-07-03T10:50:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:10 smithi188 ceph-mon[92425]: pgmap v334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2022-07-03T10:50:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:11 smithi161 ceph-mon[91579]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 685 secs 2022-07-03T10:50:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:11 smithi188 ceph-mon[92425]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 685 secs 2022-07-03T10:50:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:12 smithi161 ceph-mon[91579]: pgmap v335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:50:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:12 smithi161 ceph-mon[91579]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2022-07-03T10:50:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:12 smithi161 ceph-mon[91579]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:12 smithi161 ceph-mon[91579]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:12 smithi188 ceph-mon[92425]: pgmap v335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:50:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:12 smithi188 ceph-mon[92425]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2022-07-03T10:50:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:12 smithi188 ceph-mon[92425]: mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115] up:active 2022-07-03T10:50:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:12 smithi188 ceph-mon[92425]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 6 up:standby 2022-07-03T10:50:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:14 smithi161 ceph-mon[91579]: pgmap v336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:14 smithi188 ceph-mon[92425]: pgmap v336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:16 smithi161 ceph-mon[91579]: pgmap v337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:16 smithi188 ceph-mon[92425]: pgmap v337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:18 smithi161 ceph-mon[91579]: pgmap v338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:18 smithi188 ceph-mon[92425]: pgmap v338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:20 smithi161 ceph-mon[91579]: pgmap v339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:20 smithi188 ceph-mon[92425]: pgmap v339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 2 op/s 2022-07-03T10:50:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:22 smithi161 ceph-mon[91579]: pgmap v340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:50:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:22 smithi188 ceph-mon[92425]: pgmap v340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2022-07-03T10:50:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:24 smithi161 ceph-mon[91579]: pgmap v341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2022-07-03T10:50:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:24 smithi188 ceph-mon[92425]: pgmap v341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2022-07-03T10:50:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:25 smithi161 ceph-mon[91579]: pgmap v342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:25 smithi188 ceph-mon[92425]: pgmap v342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: pgmap v343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:50:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:50:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:50:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: pgmap v343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:50:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:50:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:50:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:30 smithi188 ceph-mon[92425]: pgmap v344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:30 smithi161 ceph-mon[91579]: pgmap v344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: pgmap v345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: pgmap v345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T10:50:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T10:50:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:33 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T10:50:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:33 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T10:50:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:33 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T10:50:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:33 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T10:50:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:33 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T10:50:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:33 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T10:50:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:33 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T10:50:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:33 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T10:50:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:34 smithi161 ceph-mon[91579]: pgmap v346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:34 smithi188 ceph-mon[92425]: pgmap v346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:36 smithi161 ceph-mon[91579]: pgmap v347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:36 smithi188 ceph-mon[92425]: pgmap v347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:38 smithi161 ceph-mon[91579]: pgmap v348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:38 smithi188 ceph-mon[92425]: pgmap v348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:40 smithi161 ceph-mon[91579]: pgmap v349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:40 smithi188 ceph-mon[92425]: pgmap v349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:42 smithi161 ceph-mon[91579]: pgmap v350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:42 smithi188 ceph-mon[92425]: pgmap v350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:44 smithi161 ceph-mon[91579]: pgmap v351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:44 smithi188 ceph-mon[92425]: pgmap v351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:46 smithi161 ceph-mon[91579]: pgmap v352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:46 smithi188 ceph-mon[92425]: pgmap v352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:48 smithi161 ceph-mon[91579]: pgmap v353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:48 smithi188 ceph-mon[92425]: pgmap v353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:50 smithi161 ceph-mon[91579]: pgmap v354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:50 smithi188 ceph-mon[92425]: pgmap v354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:52 smithi161 ceph-mon[91579]: pgmap v355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:52 smithi188 ceph-mon[92425]: pgmap v355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:54 smithi161 ceph-mon[91579]: pgmap v356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:54 smithi188 ceph-mon[92425]: pgmap v356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:50:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:56 smithi161 ceph-mon[91579]: pgmap v357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:50:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:50:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:50:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:50:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:56 smithi188 ceph-mon[92425]: pgmap v357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:50:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:50:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:50:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:50:58.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:50:58.175+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:50:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:50:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:50:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:50:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:50:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:50:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:50:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:50:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:50:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:50:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:50:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:50:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:50:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:50:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:50:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:50:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:50:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:50:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:50:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:50:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:50:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:50:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:50:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:50:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:50:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:50:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:50:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:50:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:50:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:50:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:50:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:50:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:50:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:50:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:50:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:50:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:50:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:50:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:50:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:50:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:50:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:50:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:50:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:50:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:50:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=17] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:50:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=17] Local address: 172.21.15.161, port 50972 2022-07-03T10:50:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=17] Peer address: 172.21.15.101, port 22 2022-07-03T10:50:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=17] Connection lost 2022-07-03T10:50:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=17] Aborting connection 2022-07-03T10:50:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:50:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-mon[91579]: pgmap v358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:50:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:58 smithi188 ceph-mon[92425]: pgmap v358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:50:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:51:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:50:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:51:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:51:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:51:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:51:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:51:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:51:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:51:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:51:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:51:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:51:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:51:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:51:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:51:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:51:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:51:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:51:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:51:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:51:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:51:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:51:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:51:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:51:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:51:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:51:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:51:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:51:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:51:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:51:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:51:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:51:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:51:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:51:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:51:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:51:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:51:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:51:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:51:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:51:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:51:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:51:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:51:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:51:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:51:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:51:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:51:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=17] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:51:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=17] Local address: 172.21.15.161, port 50972 2022-07-03T10:51:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=17] Peer address: 172.21.15.101, port 22 2022-07-03T10:51:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=17] Connection lost 2022-07-03T10:51:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=17] Aborting connection 2022-07-03T10:51:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:51:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:50:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:51:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:51:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:51:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:51:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:51:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:51:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:51:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:51:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:51:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:51:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:51:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:51:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:51:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:51:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:51:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:51:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:51:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:51:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:51:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:51:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:51:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:51:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:51:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:51:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:51:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:51:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:51:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:51:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:51:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:51:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:51:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:51:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:51:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:51:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:51:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:51:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:51:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:51:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:51:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:51:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:51:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:51:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:51:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:51:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:51:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=17] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:51:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=17] Local address: 172.21.15.161, port 50972 2022-07-03T10:51:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=17] Peer address: 172.21.15.101, port 22 2022-07-03T10:51:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=17] Connection lost 2022-07-03T10:51:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=17] Aborting connection 2022-07-03T10:51:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:00 smithi161 ceph-mon[91579]: pgmap v359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:00 smithi188 ceph-mon[92425]: pgmap v359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:02 smithi161 ceph-mon[91579]: pgmap v360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:02 smithi188 ceph-mon[92425]: pgmap v360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:03.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:03 smithi188 ceph-mon[92425]: pgmap v361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:03.720 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:03 smithi161 ceph-mon[91579]: pgmap v361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:05 smithi188 ceph-mon[92425]: pgmap v362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:05 smithi161 ceph-mon[91579]: pgmap v362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:08 smithi161 ceph-mon[91579]: pgmap v363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:08 smithi188 ceph-mon[92425]: pgmap v363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:10 smithi161 ceph-mon[91579]: pgmap v364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:10 smithi188 ceph-mon[92425]: pgmap v364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:12 smithi161 ceph-mon[91579]: pgmap v365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:12 smithi188 ceph-mon[92425]: pgmap v365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:14 smithi161 ceph-mon[91579]: pgmap v366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:14 smithi188 ceph-mon[92425]: pgmap v366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:16 smithi161 ceph-mon[91579]: pgmap v367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:16 smithi188 ceph-mon[92425]: pgmap v367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:18 smithi161 ceph-mon[91579]: pgmap v368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:18 smithi188 ceph-mon[92425]: pgmap v368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:20 smithi161 ceph-mon[91579]: pgmap v369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:20 smithi188 ceph-mon[92425]: pgmap v369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:22 smithi161 ceph-mon[91579]: pgmap v370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:22 smithi188 ceph-mon[92425]: pgmap v370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:24 smithi161 ceph-mon[91579]: pgmap v371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:24 smithi188 ceph-mon[92425]: pgmap v371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:26 smithi161 ceph-mon[91579]: pgmap v372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:26 smithi188 ceph-mon[92425]: pgmap v372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:28 smithi161 ceph-mon[91579]: pgmap v373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:28 smithi188 ceph-mon[92425]: pgmap v373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:30 smithi161 ceph-mon[91579]: pgmap v374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:30 smithi188 ceph-mon[92425]: pgmap v374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:32 smithi161 ceph-mon[91579]: pgmap v375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:32 smithi188 ceph-mon[92425]: pgmap v375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:34 smithi161 ceph-mon[91579]: pgmap v376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:34 smithi188 ceph-mon[92425]: pgmap v376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:35 smithi161 ceph-mon[91579]: pgmap v377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:35 smithi188 ceph-mon[92425]: pgmap v377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:38 smithi161 ceph-mon[91579]: pgmap v378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:38 smithi188 ceph-mon[92425]: pgmap v378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:40 smithi161 ceph-mon[91579]: pgmap v379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:40 smithi188 ceph-mon[92425]: pgmap v379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:42 smithi161 ceph-mon[91579]: pgmap v380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:42 smithi188 ceph-mon[92425]: pgmap v380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:44 smithi161 ceph-mon[91579]: pgmap v381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:44 smithi188 ceph-mon[92425]: pgmap v381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:46 smithi161 ceph-mon[91579]: pgmap v382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:46 smithi188 ceph-mon[92425]: pgmap v382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:48 smithi161 ceph-mon[91579]: pgmap v383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:48 smithi188 ceph-mon[92425]: pgmap v383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T10:51:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:50 smithi161 ceph-mon[91579]: pgmap v384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:50 smithi188 ceph-mon[92425]: pgmap v384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:52 smithi161 ceph-mon[91579]: pgmap v385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:52 smithi188 ceph-mon[92425]: pgmap v385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:54 smithi161 ceph-mon[91579]: pgmap v386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:54 smithi188 ceph-mon[92425]: pgmap v386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:51:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:56 smithi161 ceph-mon[91579]: pgmap v387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:51:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:51:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:51:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:51:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:56 smithi188 ceph-mon[92425]: pgmap v387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:51:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:51:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:51:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:51:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-mon[91579]: pgmap v388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:51:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:51:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:51:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:51:58.606+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:51:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:51:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:51:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:51:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:51:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:51:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:51:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:51:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:51:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:51:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:51:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:51:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:51:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:51:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:51:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:51:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:58 smithi188 ceph-mon[92425]: pgmap v388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:51:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:51:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:51:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:51:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:52:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:59 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:52:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:52:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:52:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:52:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:52:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:52:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:52:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:52:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:52:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:52:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:52:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:52:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:52:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:52:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:52:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:51:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:52:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:59 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:52:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:52:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:52:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:52:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:52:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:52:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:52:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:52:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:52:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:52:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:52:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:52:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:52:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:52:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:52:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:51:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:52:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:00 smithi161 ceph-mon[91579]: pgmap v389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:00 smithi188 ceph-mon[92425]: pgmap v389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:02 smithi161 ceph-mon[91579]: pgmap v390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:02 smithi188 ceph-mon[92425]: pgmap v390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:03 smithi161 ceph-mon[91579]: pgmap v391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:03 smithi188 ceph-mon[92425]: pgmap v391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:05 smithi188 ceph-mon[92425]: pgmap v392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:05 smithi161 ceph-mon[91579]: pgmap v392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:08 smithi161 ceph-mon[91579]: pgmap v393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:08 smithi188 ceph-mon[92425]: pgmap v393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:10 smithi161 ceph-mon[91579]: pgmap v394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:10 smithi188 ceph-mon[92425]: pgmap v394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:12 smithi161 ceph-mon[91579]: pgmap v395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:12 smithi188 ceph-mon[92425]: pgmap v395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:14.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:14 smithi161 ceph-mon[91579]: pgmap v396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:14 smithi188 ceph-mon[92425]: pgmap v396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:16 smithi161 ceph-mon[91579]: pgmap v397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:16 smithi188 ceph-mon[92425]: pgmap v397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:18 smithi161 ceph-mon[91579]: pgmap v398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:18 smithi188 ceph-mon[92425]: pgmap v398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:20 smithi161 ceph-mon[91579]: pgmap v399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:20 smithi188 ceph-mon[92425]: pgmap v399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:22 smithi161 ceph-mon[91579]: pgmap v400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:22 smithi188 ceph-mon[92425]: pgmap v400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:24 smithi161 ceph-mon[91579]: pgmap v401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:24 smithi188 ceph-mon[92425]: pgmap v401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:26 smithi161 ceph-mon[91579]: pgmap v402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:26 smithi188 ceph-mon[92425]: pgmap v402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-mon[91579]: pgmap v403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:52:28.620+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:52:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:52:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:52:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:52:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:52:28.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:52:28.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:52:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:52:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:52:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:52:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:52:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:52:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:52:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:52:28.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:52:28.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:52:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:52:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:52:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:52:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:52:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:52:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:52:28.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:52:28.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:52:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:52:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:52:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:52:28.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:52:28.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:52:28.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:52:28.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:52:28.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:52:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:52:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:52:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:52:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:52:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:52:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:52:28.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:52:28.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:52:28.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:52:28.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:52:28.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:52:28.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:52:28.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:52:28.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=19] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:52:28.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=19] Local address: 172.21.15.161, port 50976 2022-07-03T10:52:28.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=19] Peer address: 172.21.15.101, port 22 2022-07-03T10:52:28.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=103] Received exit status 0 2022-07-03T10:52:28.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=103] Received channel close 2022-07-03T10:52:28.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=103] Channel closed 2022-07-03T10:52:28.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=103] Received exit status 0 2022-07-03T10:52:28.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=103] Received channel close 2022-07-03T10:52:28.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=103] Channel closed 2022-07-03T10:52:28.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=19] Login timeout expired 2022-07-03T10:52:28.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=19] Aborting connection 2022-07-03T10:52:28.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:52:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:52:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:28 smithi188 ceph-mon[92425]: pgmap v403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:52:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:52:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:52:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:52:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:52:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:52:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:52:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:52:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:52:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:52:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:52:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:52:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:52:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:52:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:52:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:52:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:52:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:52:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:52:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:52:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:52:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:52:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:52:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:52:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:52:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:52:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:52:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:52:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:52:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:52:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:52:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:52:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:52:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:52:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:52:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:52:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:52:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:52:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:52:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:52:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:52:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:52:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:52:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:52:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:52:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=19] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:52:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=19] Local address: 172.21.15.161, port 50976 2022-07-03T10:52:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=19] Peer address: 172.21.15.101, port 22 2022-07-03T10:52:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=103] Received exit status 0 2022-07-03T10:52:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=103] Received channel close 2022-07-03T10:52:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=103] Channel closed 2022-07-03T10:52:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=103] Received exit status 0 2022-07-03T10:52:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=103] Received channel close 2022-07-03T10:52:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=103] Channel closed 2022-07-03T10:52:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=19] Login timeout expired 2022-07-03T10:52:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=19] Aborting connection 2022-07-03T10:52:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:52:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:52:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:52:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:52:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:52:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:52:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:52:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:52:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:52:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:52:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:52:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:52:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:52:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:52:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:52:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:52:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:52:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:52:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:52:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:52:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:52:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:52:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:52:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:52:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:52:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:52:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:52:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:52:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:52:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:52:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:52:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:52:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:52:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:52:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:52:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:52:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:52:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:52:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:52:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:52:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:52:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:52:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:52:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:52:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:52:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=19] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:52:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=19] Local address: 172.21.15.161, port 50976 2022-07-03T10:52:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=19] Peer address: 172.21.15.101, port 22 2022-07-03T10:52:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=103] Received exit status 0 2022-07-03T10:52:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=103] Received channel close 2022-07-03T10:52:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=103] Channel closed 2022-07-03T10:52:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=103] Received exit status 0 2022-07-03T10:52:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=103] Received channel close 2022-07-03T10:52:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=103] Channel closed 2022-07-03T10:52:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=19] Login timeout expired 2022-07-03T10:52:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=19] Aborting connection 2022-07-03T10:52:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:30 smithi161 ceph-mon[91579]: pgmap v404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:30 smithi188 ceph-mon[92425]: pgmap v404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:32 smithi161 ceph-mon[91579]: pgmap v405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:32 smithi188 ceph-mon[92425]: pgmap v405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:34.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:34 smithi161 ceph-mon[91579]: pgmap v406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:34 smithi188 ceph-mon[92425]: pgmap v406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:36 smithi161 ceph-mon[91579]: pgmap v407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:36 smithi188 ceph-mon[92425]: pgmap v407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:38 smithi161 ceph-mon[91579]: pgmap v408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:38 smithi188 ceph-mon[92425]: pgmap v408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:40 smithi161 ceph-mon[91579]: pgmap v409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:40 smithi188 ceph-mon[92425]: pgmap v409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:42 smithi161 ceph-mon[91579]: pgmap v410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:42 smithi188 ceph-mon[92425]: pgmap v410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:44 smithi161 ceph-mon[91579]: pgmap v411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:44 smithi188 ceph-mon[92425]: pgmap v411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:46 smithi161 ceph-mon[91579]: pgmap v412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:46 smithi188 ceph-mon[92425]: pgmap v412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:48 smithi161 ceph-mon[91579]: pgmap v413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:48 smithi188 ceph-mon[92425]: pgmap v413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:50 smithi161 ceph-mon[91579]: pgmap v414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:50 smithi188 ceph-mon[92425]: pgmap v414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:52 smithi161 ceph-mon[91579]: pgmap v415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:52 smithi188 ceph-mon[92425]: pgmap v415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:54 smithi161 ceph-mon[91579]: pgmap v416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:54 smithi188 ceph-mon[92425]: pgmap v416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:52:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:55 smithi161 ceph-mon[91579]: pgmap v417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:55 smithi188 ceph-mon[92425]: pgmap v417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:52:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:52:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:52:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:52:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:52:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:52:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:52:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:52:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:57 smithi161 ceph-mon[91579]: pgmap v418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:57 smithi188 ceph-mon[92425]: pgmap v418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:52:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:52:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:52:59 smithi161 ceph-mon[91579]: pgmap v419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:52:59 smithi188 ceph-mon[92425]: pgmap v419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:02 smithi161 ceph-mon[91579]: pgmap v420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:02 smithi188 ceph-mon[92425]: pgmap v420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:04 smithi161 ceph-mon[91579]: pgmap v421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:04 smithi188 ceph-mon[92425]: pgmap v421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:06 smithi161 ceph-mon[91579]: pgmap v422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:06 smithi188 ceph-mon[92425]: pgmap v422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:08 smithi161 ceph-mon[91579]: pgmap v423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:08 smithi188 ceph-mon[92425]: pgmap v423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:10 smithi161 ceph-mon[91579]: pgmap v424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:10 smithi188 ceph-mon[92425]: pgmap v424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:12 smithi161 ceph-mon[91579]: pgmap v425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:12 smithi188 ceph-mon[92425]: pgmap v425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:14 smithi161 ceph-mon[91579]: pgmap v426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:14 smithi188 ceph-mon[92425]: pgmap v426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:16 smithi161 ceph-mon[91579]: pgmap v427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:16 smithi188 ceph-mon[92425]: pgmap v427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:18 smithi161 ceph-mon[91579]: pgmap v428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:18 smithi188 ceph-mon[92425]: pgmap v428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:20 smithi161 ceph-mon[91579]: pgmap v429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:20 smithi188 ceph-mon[92425]: pgmap v429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:22 smithi161 ceph-mon[91579]: pgmap v430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:22 smithi188 ceph-mon[92425]: pgmap v430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:24 smithi161 ceph-mon[91579]: pgmap v431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:24 smithi188 ceph-mon[92425]: pgmap v431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:25 smithi161 ceph-mon[91579]: pgmap v432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:25 smithi188 ceph-mon[92425]: pgmap v432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:29.062 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:28 smithi161 ceph-mon[91579]: pgmap v433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:29.062 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:29.063 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:53:29.065 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:53:29.065 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:53:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:28 smithi188 ceph-mon[92425]: pgmap v433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:53:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:53:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:53:29.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:53:29.060+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:53:29.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:53:29.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:29.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:53:29.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:29.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:29.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:29.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:53:29.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:29.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:53:29.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:29.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:53:29.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:53:29.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:53:29.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:53:29.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:53:29.073+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:53:29.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:53:29.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:29.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:53:29.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:29.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:29.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:29.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:53:29.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:29.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:53:29.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:29.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:53:29.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:53:29.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:53:29.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:53:29.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:53:29.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:53:29.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:53:29.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:53:29.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:53:29.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:53:29.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:53:29.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:53:29.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:53:29.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:53:29.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:53:29.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:53:29.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:53:29.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:53:29.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:53:29.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:53:29.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:53:29.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:53:29.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:53:29.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:53:29.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:53:29.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:53:29.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:53:29.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:53:29.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:53:29.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:53:29.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:53:29.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:53:29.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:53:29.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:53:29.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=21] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:53:29.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=21] Local address: 172.21.15.161, port 50980 2022-07-03T10:53:29.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=21] Peer address: 172.21.15.101, port 22 2022-07-03T10:53:29.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=21] Connection lost 2022-07-03T10:53:29.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=21] Aborting connection 2022-07-03T10:53:29.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:53:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:53:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:30 smithi161 ceph-mon[91579]: pgmap v434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:30 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:53:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:53:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:53:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:53:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:53:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:53:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:53:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:53:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:53:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:53:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:53:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:53:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:53:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:53:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:53:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:53:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:53:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:53:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:53:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:53:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:53:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:53:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:53:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:53:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:53:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:53:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:53:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:53:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:53:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:53:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:53:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:53:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:53:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:53:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:53:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:53:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:53:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:53:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:53:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:53:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:53:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:53:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:53:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:53:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:53:31.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:53:31.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=21] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:53:31.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=21] Local address: 172.21.15.161, port 50980 2022-07-03T10:53:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=21] Peer address: 172.21.15.101, port 22 2022-07-03T10:53:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=21] Connection lost 2022-07-03T10:53:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=21] Aborting connection 2022-07-03T10:53:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:30 smithi188 ceph-mon[92425]: pgmap v434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:30 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:53:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:53:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:53:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:53:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:53:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:53:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:53:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:53:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:53:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:53:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:53:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:53:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:53:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:53:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:53:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:53:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:53:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:53:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:53:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:53:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:53:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:53:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:53:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:53:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:53:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:53:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:53:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:53:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:53:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:53:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:53:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:53:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:53:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:53:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:53:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:53:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:53:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:53:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:53:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:53:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:53:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:53:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:53:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:53:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:53:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:53:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:53:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:53:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:53:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:53:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:53:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:53:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:53:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=21] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:53:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=21] Local address: 172.21.15.161, port 50980 2022-07-03T10:53:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=21] Peer address: 172.21.15.101, port 22 2022-07-03T10:53:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=21] Connection lost 2022-07-03T10:53:31.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=21] Aborting connection 2022-07-03T10:53:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:32 smithi161 ceph-mon[91579]: pgmap v435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:32 smithi188 ceph-mon[92425]: pgmap v435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:34 smithi161 ceph-mon[91579]: pgmap v436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:34 smithi188 ceph-mon[92425]: pgmap v436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:36 smithi161 ceph-mon[91579]: pgmap v437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:36 smithi188 ceph-mon[92425]: pgmap v437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:38 smithi161 ceph-mon[91579]: pgmap v438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:38 smithi188 ceph-mon[92425]: pgmap v438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:40 smithi161 ceph-mon[91579]: pgmap v439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:40 smithi188 ceph-mon[92425]: pgmap v439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:42 smithi161 ceph-mon[91579]: pgmap v440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:42 smithi188 ceph-mon[92425]: pgmap v440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:44 smithi161 ceph-mon[91579]: pgmap v441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:44 smithi188 ceph-mon[92425]: pgmap v441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:46 smithi161 ceph-mon[91579]: pgmap v442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:46 smithi188 ceph-mon[92425]: pgmap v442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:48 smithi161 ceph-mon[91579]: pgmap v443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:48 smithi188 ceph-mon[92425]: pgmap v443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:50 smithi161 ceph-mon[91579]: pgmap v444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:50 smithi188 ceph-mon[92425]: pgmap v444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:52 smithi161 ceph-mon[91579]: pgmap v445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:52 smithi188 ceph-mon[92425]: pgmap v445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:54 smithi161 ceph-mon[91579]: pgmap v446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:54 smithi188 ceph-mon[92425]: pgmap v446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:53:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:56 smithi161 ceph-mon[91579]: pgmap v447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:53:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:53:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:53:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:53:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:56 smithi188 ceph-mon[92425]: pgmap v447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:53:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:53:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:53:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:53:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:58 smithi161 ceph-mon[91579]: pgmap v448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:53:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:58 smithi188 ceph-mon[92425]: pgmap v448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:53:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:00 smithi161 ceph-mon[91579]: pgmap v449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:00 smithi188 ceph-mon[92425]: pgmap v449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:02 smithi161 ceph-mon[91579]: pgmap v450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:02 smithi188 ceph-mon[92425]: pgmap v450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:04 smithi161 ceph-mon[91579]: pgmap v451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:04 smithi188 ceph-mon[92425]: pgmap v451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:06 smithi161 ceph-mon[91579]: pgmap v452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:06 smithi188 ceph-mon[92425]: pgmap v452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:08 smithi161 ceph-mon[91579]: pgmap v453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:08 smithi188 ceph-mon[92425]: pgmap v453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:10 smithi161 ceph-mon[91579]: pgmap v454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:10 smithi188 ceph-mon[92425]: pgmap v454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:12 smithi161 ceph-mon[91579]: pgmap v455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:12 smithi188 ceph-mon[92425]: pgmap v455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:14 smithi161 ceph-mon[91579]: pgmap v456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:14 smithi188 ceph-mon[92425]: pgmap v456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:15 smithi161 ceph-mon[91579]: pgmap v457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:15 smithi188 ceph-mon[92425]: pgmap v457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:18 smithi161 ceph-mon[91579]: pgmap v458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:18 smithi188 ceph-mon[92425]: pgmap v458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:20 smithi161 ceph-mon[91579]: pgmap v459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:20 smithi188 ceph-mon[92425]: pgmap v459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:22 smithi161 ceph-mon[91579]: pgmap v460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:22 smithi188 ceph-mon[92425]: pgmap v460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:24 smithi161 ceph-mon[91579]: pgmap v461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:24 smithi188 ceph-mon[92425]: pgmap v461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:26 smithi161 ceph-mon[91579]: pgmap v462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:26 smithi188 ceph-mon[92425]: pgmap v462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:28 smithi161 ceph-mon[91579]: pgmap v463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:28 smithi188 ceph-mon[92425]: pgmap v463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:54:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:54:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:54:29.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:54:29.562+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:54:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:54:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:54:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:54:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:54:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:54:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:54:29.562+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:54:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:54:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:54:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:54:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:54:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:54:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T10:54:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:54:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:54:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:54:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:54:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:54:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:54:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:54:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:54:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:54:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:54:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:54:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:54:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:54:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:54:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:54:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:54:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:54:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:54:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:54:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:54:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:54:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:54:29.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:54:29.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:54:29.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:54:29.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:54:29.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:54:29.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:54:29.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:54:29.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:54:29.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:54:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:54:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:54:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:54:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:30 smithi161 ceph-mon[91579]: pgmap v464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:54:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:30 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:54:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:54:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:54:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:54:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:54:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:54:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:54:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:54:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:54:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:54:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:54:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:54:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:54:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:54:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:54:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:54:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:54:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:54:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:54:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:54:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:54:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:54:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:54:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:54:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:54:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:54:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:54:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:54:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:54:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:54:31.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:54:31.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:54:31.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:54:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:54:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:54:31.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:54:31.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:54:31.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:54:31.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:54:31.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:54:31.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:30 smithi188 ceph-mon[92425]: pgmap v464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:54:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:30 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:54:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:54:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:54:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:54:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:54:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:54:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:54:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:54:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:54:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:54:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:54:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:54:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:54:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:54:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:54:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:54:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T10:54:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T10:54:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T10:54:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:54:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:54:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:54:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:54:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:54:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:54:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:54:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:54:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:54:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:54:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:54:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:54:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:54:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:54:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:54:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:54:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:54:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:54:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:54:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:54:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:54:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:54:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:54:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:54:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:54:31.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:54:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:54:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:54:31.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T10:54:31.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:54:31.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T10:54:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:32 smithi161 ceph-mon[91579]: pgmap v465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:32 smithi188 ceph-mon[92425]: pgmap v465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:34 smithi161 ceph-mon[91579]: pgmap v466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:34 smithi188 ceph-mon[92425]: pgmap v466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:35 smithi161 ceph-mon[91579]: pgmap v467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:35 smithi188 ceph-mon[92425]: pgmap v467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:38 smithi161 ceph-mon[91579]: pgmap v468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:38 smithi188 ceph-mon[92425]: pgmap v468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:40 smithi161 ceph-mon[91579]: pgmap v469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:40 smithi188 ceph-mon[92425]: pgmap v469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:42 smithi161 ceph-mon[91579]: pgmap v470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:42 smithi188 ceph-mon[92425]: pgmap v470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:44 smithi161 ceph-mon[91579]: pgmap v471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:44 smithi188 ceph-mon[92425]: pgmap v471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:46 smithi161 ceph-mon[91579]: pgmap v472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:46 smithi188 ceph-mon[92425]: pgmap v472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:48 smithi161 ceph-mon[91579]: pgmap v473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:48 smithi188 ceph-mon[92425]: pgmap v473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:50 smithi161 ceph-mon[91579]: pgmap v474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:50 smithi188 ceph-mon[92425]: pgmap v474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:52 smithi161 ceph-mon[91579]: pgmap v475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:52 smithi188 ceph-mon[92425]: pgmap v475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:54 smithi161 ceph-mon[91579]: pgmap v476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:54 smithi188 ceph-mon[92425]: pgmap v476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:54:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:55 smithi161 ceph-mon[91579]: pgmap v477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:55 smithi188 ceph-mon[92425]: pgmap v477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:54:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:54:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:54:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:54:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:54:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:54:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:54:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:54:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:57 smithi161 ceph-mon[91579]: pgmap v478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:57 smithi188 ceph-mon[92425]: pgmap v478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:54:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:54:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:54:59 smithi161 ceph-mon[91579]: pgmap v479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:54:59 smithi188 ceph-mon[92425]: pgmap v479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:02 smithi161 ceph-mon[91579]: pgmap v480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:02 smithi188 ceph-mon[92425]: pgmap v480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:04 smithi161 ceph-mon[91579]: pgmap v481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:04 smithi188 ceph-mon[92425]: pgmap v481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:06 smithi161 ceph-mon[91579]: pgmap v482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:06 smithi188 ceph-mon[92425]: pgmap v482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:08 smithi161 ceph-mon[91579]: pgmap v483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:08 smithi188 ceph-mon[92425]: pgmap v483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:10 smithi161 ceph-mon[91579]: pgmap v484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:10 smithi188 ceph-mon[92425]: pgmap v484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:12 smithi161 ceph-mon[91579]: pgmap v485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:12 smithi188 ceph-mon[92425]: pgmap v485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:14 smithi161 ceph-mon[91579]: pgmap v486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:14 smithi188 ceph-mon[92425]: pgmap v486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:16 smithi161 ceph-mon[91579]: pgmap v487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:16 smithi188 ceph-mon[92425]: pgmap v487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:17 smithi161 ceph-mon[91579]: pgmap v488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:17 smithi188 ceph-mon[92425]: pgmap v488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:19 smithi161 ceph-mon[91579]: pgmap v489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:19 smithi188 ceph-mon[92425]: pgmap v489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:22 smithi161 ceph-mon[91579]: pgmap v490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:22 smithi188 ceph-mon[92425]: pgmap v490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:24 smithi161 ceph-mon[91579]: pgmap v491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:24 smithi188 ceph-mon[92425]: pgmap v491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:26 smithi161 ceph-mon[91579]: pgmap v492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:26 smithi188 ceph-mon[92425]: pgmap v492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:28 smithi161 ceph-mon[91579]: pgmap v493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:28 smithi188 ceph-mon[92425]: pgmap v493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:55:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:55:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:55:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:55:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:55:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:55:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:30 smithi161 ceph-mon[91579]: pgmap v494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:30 smithi188 ceph-mon[92425]: pgmap v494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:32 smithi161 ceph-mon[91579]: pgmap v495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:32 smithi188 ceph-mon[92425]: pgmap v495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:34 smithi161 ceph-mon[91579]: pgmap v496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:34 smithi188 ceph-mon[92425]: pgmap v496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:35 smithi161 ceph-mon[91579]: pgmap v497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:35 smithi188 ceph-mon[92425]: pgmap v497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:38 smithi161 ceph-mon[91579]: pgmap v498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:38 smithi188 ceph-mon[92425]: pgmap v498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:40 smithi161 ceph-mon[91579]: pgmap v499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:40 smithi188 ceph-mon[92425]: pgmap v499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:42 smithi161 ceph-mon[91579]: pgmap v500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:42 smithi188 ceph-mon[92425]: pgmap v500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:44 smithi161 ceph-mon[91579]: pgmap v501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:44 smithi188 ceph-mon[92425]: pgmap v501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:46 smithi161 ceph-mon[91579]: pgmap v502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:46 smithi188 ceph-mon[92425]: pgmap v502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:48 smithi161 ceph-mon[91579]: pgmap v503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:48 smithi188 ceph-mon[92425]: pgmap v503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:50 smithi161 ceph-mon[91579]: pgmap v504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:50 smithi188 ceph-mon[92425]: pgmap v504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:52 smithi161 ceph-mon[91579]: pgmap v505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:52 smithi188 ceph-mon[92425]: pgmap v505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:54 smithi161 ceph-mon[91579]: pgmap v506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:54 smithi188 ceph-mon[92425]: pgmap v506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:55:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:55 smithi161 ceph-mon[91579]: pgmap v507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:55 smithi188 ceph-mon[92425]: pgmap v507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:55:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:55:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:55:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:55:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:55:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:55:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:55:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:55:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:57 smithi161 ceph-mon[91579]: pgmap v508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:57 smithi188 ceph-mon[92425]: pgmap v508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:55:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:55:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:59 smithi161 ceph-mon[91579]: pgmap v509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:55:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:56:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:59 smithi188 ceph-mon[92425]: pgmap v509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:55:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:56:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:02 smithi161 ceph-mon[91579]: pgmap v510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:02 smithi188 ceph-mon[92425]: pgmap v510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:04 smithi161 ceph-mon[91579]: pgmap v511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:04 smithi188 ceph-mon[92425]: pgmap v511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:06 smithi161 ceph-mon[91579]: pgmap v512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:06 smithi188 ceph-mon[92425]: pgmap v512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:08 smithi161 ceph-mon[91579]: pgmap v513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:08 smithi188 ceph-mon[92425]: pgmap v513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:10 smithi161 ceph-mon[91579]: pgmap v514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:10 smithi188 ceph-mon[92425]: pgmap v514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:12 smithi161 ceph-mon[91579]: pgmap v515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:12 smithi188 ceph-mon[92425]: pgmap v515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:14 smithi161 ceph-mon[91579]: pgmap v516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:14 smithi188 ceph-mon[92425]: pgmap v516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:16 smithi161 ceph-mon[91579]: pgmap v517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:16 smithi188 ceph-mon[92425]: pgmap v517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:17 smithi161 ceph-mon[91579]: pgmap v518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:17 smithi188 ceph-mon[92425]: pgmap v518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:19 smithi161 ceph-mon[91579]: pgmap v519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:19 smithi188 ceph-mon[92425]: pgmap v519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:22 smithi161 ceph-mon[91579]: pgmap v520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:22 smithi188 ceph-mon[92425]: pgmap v520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:24 smithi161 ceph-mon[91579]: pgmap v521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:24 smithi188 ceph-mon[92425]: pgmap v521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:26 smithi161 ceph-mon[91579]: pgmap v522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:26 smithi188 ceph-mon[92425]: pgmap v522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:28 smithi161 ceph-mon[91579]: pgmap v523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:28 smithi188 ceph-mon[92425]: pgmap v523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:30.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:56:29.655+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:56:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:56:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:56:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:56:30.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:56:30.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:56:30.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:56:30.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:56:30.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:56:30.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:56:30.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:56:30.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:56:30.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:56:30.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:56:30.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:56:30.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:56:30.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:56:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:56:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:56:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:56:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:56:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:56:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:56:30.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:56:30.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:56:30.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:56:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:56:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:56:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:56:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:56:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:56:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:56:30.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:56:30.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:56:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:56:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:56:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:56:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:56:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:56:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:56:30.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:56:30.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:56:30.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:56:30.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:56:30.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:56:30.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=25] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:56:30.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=25] Local address: 172.21.15.161, port 51008 2022-07-03T10:56:30.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=25] Peer address: 172.21.15.101, port 22 2022-07-03T10:56:30.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=25] Login timeout expired 2022-07-03T10:56:30.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=25] Aborting connection 2022-07-03T10:56:30.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:56:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:56:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:30 smithi161 ceph-mon[91579]: pgmap v524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:56:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:56:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:56:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:56:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:56:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:56:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:56:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:56:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:56:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:56:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:56:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:56:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:56:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:56:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:56:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:56:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:56:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:56:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:56:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:56:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:56:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:56:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:56:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:56:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:56:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:56:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:56:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:56:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:56:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:56:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:56:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:56:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:56:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:56:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:56:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:56:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:56:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:56:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:56:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:56:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:56:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:56:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:56:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:56:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:56:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=25] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:56:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=25] Local address: 172.21.15.161, port 51008 2022-07-03T10:56:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=25] Peer address: 172.21.15.101, port 22 2022-07-03T10:56:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=25] Login timeout expired 2022-07-03T10:56:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=25] Aborting connection 2022-07-03T10:56:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:30 smithi188 ceph-mon[92425]: pgmap v524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:56:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:56:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:56:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:56:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:56:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:56:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:56:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:56:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:56:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:56:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:56:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:56:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:56:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:56:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:56:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:56:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:56:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:56:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:56:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:56:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:56:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:56:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:56:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:56:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:56:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:56:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:56:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:56:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:56:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:56:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:56:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:56:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:56:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:56:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:56:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:56:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:56:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:56:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:56:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:56:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:56:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:56:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:56:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:56:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:56:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=25] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:56:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=25] Local address: 172.21.15.161, port 51008 2022-07-03T10:56:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=25] Peer address: 172.21.15.101, port 22 2022-07-03T10:56:31.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=25] Login timeout expired 2022-07-03T10:56:31.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=25] Aborting connection 2022-07-03T10:56:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:32 smithi161 ceph-mon[91579]: pgmap v525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:32 smithi188 ceph-mon[92425]: pgmap v525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:34 smithi161 ceph-mon[91579]: pgmap v526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:34 smithi188 ceph-mon[92425]: pgmap v526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:35 smithi161 ceph-mon[91579]: pgmap v527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:35 smithi188 ceph-mon[92425]: pgmap v527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:38 smithi161 ceph-mon[91579]: pgmap v528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:38 smithi188 ceph-mon[92425]: pgmap v528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:40 smithi161 ceph-mon[91579]: pgmap v529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:40 smithi188 ceph-mon[92425]: pgmap v529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:42 smithi161 ceph-mon[91579]: pgmap v530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:42 smithi188 ceph-mon[92425]: pgmap v530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:44 smithi161 ceph-mon[91579]: pgmap v531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:44 smithi188 ceph-mon[92425]: pgmap v531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:46 smithi161 ceph-mon[91579]: pgmap v532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:46 smithi188 ceph-mon[92425]: pgmap v532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:48 smithi161 ceph-mon[91579]: pgmap v533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:48 smithi188 ceph-mon[92425]: pgmap v533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:50 smithi161 ceph-mon[91579]: pgmap v534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:50 smithi188 ceph-mon[92425]: pgmap v534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:52 smithi161 ceph-mon[91579]: pgmap v535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:52 smithi188 ceph-mon[92425]: pgmap v535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:54 smithi161 ceph-mon[91579]: pgmap v536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:54 smithi188 ceph-mon[92425]: pgmap v536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:56:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:55 smithi161 ceph-mon[91579]: pgmap v537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:55 smithi188 ceph-mon[92425]: pgmap v537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:56:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:56:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:56:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:56:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:56:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:56:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:56:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:56:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:57 smithi161 ceph-mon[91579]: pgmap v538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:57 smithi188 ceph-mon[92425]: pgmap v538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:56:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:56:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:56:59 smithi161 ceph-mon[91579]: pgmap v539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:56:59 smithi188 ceph-mon[92425]: pgmap v539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:02 smithi161 ceph-mon[91579]: pgmap v540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:02 smithi188 ceph-mon[92425]: pgmap v540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:04 smithi161 ceph-mon[91579]: pgmap v541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:04 smithi188 ceph-mon[92425]: pgmap v541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:06 smithi161 ceph-mon[91579]: pgmap v542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:06 smithi188 ceph-mon[92425]: pgmap v542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:08 smithi161 ceph-mon[91579]: pgmap v543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:08 smithi188 ceph-mon[92425]: pgmap v543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:10 smithi161 ceph-mon[91579]: pgmap v544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:10 smithi188 ceph-mon[92425]: pgmap v544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:12 smithi161 ceph-mon[91579]: pgmap v545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:12 smithi188 ceph-mon[92425]: pgmap v545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:14 smithi161 ceph-mon[91579]: pgmap v546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:14 smithi188 ceph-mon[92425]: pgmap v546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:15 smithi161 ceph-mon[91579]: pgmap v547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:15 smithi188 ceph-mon[92425]: pgmap v547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:18 smithi161 ceph-mon[91579]: pgmap v548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:18 smithi188 ceph-mon[92425]: pgmap v548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:20 smithi161 ceph-mon[91579]: pgmap v549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:20 smithi188 ceph-mon[92425]: pgmap v549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:22 smithi161 ceph-mon[91579]: pgmap v550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:22 smithi188 ceph-mon[92425]: pgmap v550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:24 smithi161 ceph-mon[91579]: pgmap v551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:24 smithi188 ceph-mon[92425]: pgmap v551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:26 smithi161 ceph-mon[91579]: pgmap v552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:26 smithi188 ceph-mon[92425]: pgmap v552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:28 smithi161 ceph-mon[91579]: pgmap v553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:28 smithi188 ceph-mon[92425]: pgmap v553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:57:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:57:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:57:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:57:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:57:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:57:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:30 smithi161 ceph-mon[91579]: pgmap v554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:57:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:30 smithi188 ceph-mon[92425]: pgmap v554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:57:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:32 smithi161 ceph-mon[91579]: pgmap v555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:32 smithi188 ceph-mon[92425]: pgmap v555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:34 smithi161 ceph-mon[91579]: pgmap v556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:34 smithi188 ceph-mon[92425]: pgmap v556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:36 smithi161 ceph-mon[91579]: pgmap v557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:36 smithi188 ceph-mon[92425]: pgmap v557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:38 smithi161 ceph-mon[91579]: pgmap v558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:38 smithi188 ceph-mon[92425]: pgmap v558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:40 smithi161 ceph-mon[91579]: pgmap v559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:40 smithi188 ceph-mon[92425]: pgmap v559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:42 smithi161 ceph-mon[91579]: pgmap v560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:42 smithi188 ceph-mon[92425]: pgmap v560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:44 smithi161 ceph-mon[91579]: pgmap v561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:44 smithi188 ceph-mon[92425]: pgmap v561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:45 smithi161 ceph-mon[91579]: pgmap v562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:45 smithi188 ceph-mon[92425]: pgmap v562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:48 smithi161 ceph-mon[91579]: pgmap v563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:48 smithi188 ceph-mon[92425]: pgmap v563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:50 smithi161 ceph-mon[91579]: pgmap v564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:50 smithi188 ceph-mon[92425]: pgmap v564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:52 smithi161 ceph-mon[91579]: pgmap v565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:52 smithi188 ceph-mon[92425]: pgmap v565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:54 smithi161 ceph-mon[91579]: pgmap v566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:54 smithi188 ceph-mon[92425]: pgmap v566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:57:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:56 smithi161 ceph-mon[91579]: pgmap v567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:57:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:57:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:57:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:57:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:56 smithi188 ceph-mon[92425]: pgmap v567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:57:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:57:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:57:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:57:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:58 smithi161 ceph-mon[91579]: pgmap v568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:57:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:58 smithi188 ceph-mon[92425]: pgmap v568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:57:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-mon[91579]: pgmap v569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:58:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:58:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:58:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:58:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:58:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:58:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:58:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:58:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:58:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:58:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:58:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:58:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:58:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:58:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:58:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:58:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:58:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:58:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:58:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:58:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:58:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:58:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:58:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:58:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:58:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:58:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:58:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:58:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:58:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:58:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:58:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:58:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:58:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:58:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:58:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:58:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:58:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:58:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:58:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:58:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:58:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:58:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:58:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:58:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:58:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=27] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:58:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=27] Local address: 172.21.15.161, port 51012 2022-07-03T10:58:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=27] Peer address: 172.21.15.101, port 22 2022-07-03T10:58:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=116] Received exit status 0 2022-07-03T10:58:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=116] Received channel close 2022-07-03T10:58:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=116] Channel closed 2022-07-03T10:58:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=116] Received exit status 0 2022-07-03T10:58:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=116] Received channel close 2022-07-03T10:58:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=116] Channel closed 2022-07-03T10:58:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=117] Requesting new SSH session 2022-07-03T10:58:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=117] Requesting new SSH session 2022-07-03T10:58:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=117] Command: which python3 2022-07-03T10:58:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=117] Command: which python3 2022-07-03T10:58:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=117] Received exit status 0 2022-07-03T10:58:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=117] Received channel close 2022-07-03T10:58:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=117] Channel closed 2022-07-03T10:58:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=118] Requesting new SSH session 2022-07-03T10:58:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=117] Received exit status 0 2022-07-03T10:58:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=117] Received channel close 2022-07-03T10:58:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=117] Channel closed 2022-07-03T10:58:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=118] Command: sudo true 2022-07-03T10:58:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=118] Requesting new SSH session 2022-07-03T10:58:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=118] Command: sudo true 2022-07-03T10:58:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=118] Received exit status 0 2022-07-03T10:58:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=118] Received channel close 2022-07-03T10:58:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=118] Channel closed 2022-07-03T10:58:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=119] Requesting new SSH session 2022-07-03T10:58:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=118] Received exit status 0 2022-07-03T10:58:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=118] Received channel close 2022-07-03T10:58:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=118] Channel closed 2022-07-03T10:58:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=119] Requesting new SSH session 2022-07-03T10:58:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=119] Received exit status 0 2022-07-03T10:58:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=119] Received channel close 2022-07-03T10:58:00.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=119] Channel closed 2022-07-03T10:58:00.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=119] Received exit status 0 2022-07-03T10:58:00.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=119] Received channel close 2022-07-03T10:58:00.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=119] Channel closed 2022-07-03T10:58:00.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=27] Login timeout expired 2022-07-03T10:58:00.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=27] Aborting connection 2022-07-03T10:58:00.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:57:59.690+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:58:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:58:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:58:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:58:00.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:58:00.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:58:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:58:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:58:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:58:00.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:58:00.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:58:00.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:58:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:58:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:58:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:58:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:58:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:58:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:58:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:58:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:58:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:58:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:58:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:58:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:58:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:58:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:58:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:58:00.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:58:00.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:58:00.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:58:00.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:58:00.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:58:00.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:58:00.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:58:00.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:58:00.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:58:00.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:58:00.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:58:00.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:58:00.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:58:00.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:58:00.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:58:00.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:58:00.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:58:00.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:58:00.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=27] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:58:00.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=27] Local address: 172.21.15.161, port 51012 2022-07-03T10:58:00.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=27] Peer address: 172.21.15.101, port 22 2022-07-03T10:58:00.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=116] Received exit status 0 2022-07-03T10:58:00.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=116] Received channel close 2022-07-03T10:58:00.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=116] Channel closed 2022-07-03T10:58:00.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=116] Received exit status 0 2022-07-03T10:58:00.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=116] Received channel close 2022-07-03T10:58:00.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=116] Channel closed 2022-07-03T10:58:00.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=117] Requesting new SSH session 2022-07-03T10:58:00.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=117] Requesting new SSH session 2022-07-03T10:58:00.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=117] Command: which python3 2022-07-03T10:58:00.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=117] Command: which python3 2022-07-03T10:58:00.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=117] Received exit status 0 2022-07-03T10:58:00.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=117] Received channel close 2022-07-03T10:58:00.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=117] Channel closed 2022-07-03T10:58:00.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=118] Requesting new SSH session 2022-07-03T10:58:00.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=117] Received exit status 0 2022-07-03T10:58:00.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=117] Received channel close 2022-07-03T10:58:00.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=117] Channel closed 2022-07-03T10:58:00.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=118] Command: sudo true 2022-07-03T10:58:00.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=118] Requesting new SSH session 2022-07-03T10:58:00.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=118] Command: sudo true 2022-07-03T10:58:00.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=118] Received exit status 0 2022-07-03T10:58:00.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=118] Received channel close 2022-07-03T10:58:00.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=118] Channel closed 2022-07-03T10:58:00.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=119] Requesting new SSH session 2022-07-03T10:58:00.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=118] Received exit status 0 2022-07-03T10:58:00.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=118] Received channel close 2022-07-03T10:58:00.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=118] Channel closed 2022-07-03T10:58:00.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=119] Requesting new SSH session 2022-07-03T10:58:00.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=119] Received exit status 0 2022-07-03T10:58:00.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=119] Received channel close 2022-07-03T10:58:00.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=119] Channel closed 2022-07-03T10:58:00.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=119] Received exit status 0 2022-07-03T10:58:00.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=119] Received channel close 2022-07-03T10:58:00.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=119] Channel closed 2022-07-03T10:58:00.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=27] Login timeout expired 2022-07-03T10:58:00.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=27] Aborting connection 2022-07-03T10:58:00.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:57:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:58:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:59 smithi188 ceph-mon[92425]: pgmap v569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:57:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:58:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:58:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:58:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:58:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:58:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:58:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:58:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:58:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:58:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:58:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:58:00.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:58:00.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T10:58:00.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:58:00.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:58:00.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:58:00.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:58:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:58:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:58:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:58:00.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:58:00.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:58:00.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:58:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:58:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:58:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:58:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:58:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:58:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:58:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:58:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:58:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:58:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:58:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:58:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:58:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:58:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:58:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:58:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:58:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:58:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:58:00.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:58:00.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:58:00.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T10:58:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:58:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=27] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:58:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=27] Local address: 172.21.15.161, port 51012 2022-07-03T10:58:00.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=27] Peer address: 172.21.15.101, port 22 2022-07-03T10:58:00.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=116] Received exit status 0 2022-07-03T10:58:00.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=116] Received channel close 2022-07-03T10:58:00.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=116] Channel closed 2022-07-03T10:58:00.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=116] Received exit status 0 2022-07-03T10:58:00.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=116] Received channel close 2022-07-03T10:58:00.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=116] Channel closed 2022-07-03T10:58:00.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=117] Requesting new SSH session 2022-07-03T10:58:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=117] Requesting new SSH session 2022-07-03T10:58:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=117] Command: which python3 2022-07-03T10:58:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=117] Command: which python3 2022-07-03T10:58:00.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=117] Received exit status 0 2022-07-03T10:58:00.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=117] Received channel close 2022-07-03T10:58:00.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=117] Channel closed 2022-07-03T10:58:00.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=118] Requesting new SSH session 2022-07-03T10:58:00.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=117] Received exit status 0 2022-07-03T10:58:00.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=117] Received channel close 2022-07-03T10:58:00.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=117] Channel closed 2022-07-03T10:58:00.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=118] Command: sudo true 2022-07-03T10:58:00.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=118] Requesting new SSH session 2022-07-03T10:58:00.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=118] Command: sudo true 2022-07-03T10:58:00.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=118] Received exit status 0 2022-07-03T10:58:00.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=118] Received channel close 2022-07-03T10:58:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=118] Channel closed 2022-07-03T10:58:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=119] Requesting new SSH session 2022-07-03T10:58:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=118] Received exit status 0 2022-07-03T10:58:00.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=118] Received channel close 2022-07-03T10:58:00.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=118] Channel closed 2022-07-03T10:58:00.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=119] Requesting new SSH session 2022-07-03T10:58:00.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=119] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T10:58:00.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=119] Received exit status 0 2022-07-03T10:58:00.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=119] Received channel close 2022-07-03T10:58:00.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=119] Channel closed 2022-07-03T10:58:00.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=119] Received exit status 0 2022-07-03T10:58:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=119] Received channel close 2022-07-03T10:58:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=119] Channel closed 2022-07-03T10:58:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=27] Login timeout expired 2022-07-03T10:58:00.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=27] Aborting connection 2022-07-03T10:58:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:02 smithi161 ceph-mon[91579]: pgmap v570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:02 smithi188 ceph-mon[92425]: pgmap v570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:04 smithi161 ceph-mon[91579]: pgmap v571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:04 smithi188 ceph-mon[92425]: pgmap v571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:06 smithi161 ceph-mon[91579]: pgmap v572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:06 smithi188 ceph-mon[92425]: pgmap v572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:08 smithi161 ceph-mon[91579]: pgmap v573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:08 smithi188 ceph-mon[92425]: pgmap v573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:10 smithi161 ceph-mon[91579]: pgmap v574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:10 smithi188 ceph-mon[92425]: pgmap v574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:12 smithi161 ceph-mon[91579]: pgmap v575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:12 smithi188 ceph-mon[92425]: pgmap v575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:14 smithi161 ceph-mon[91579]: pgmap v576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:14 smithi188 ceph-mon[92425]: pgmap v576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:15 smithi161 ceph-mon[91579]: pgmap v577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:15 smithi188 ceph-mon[92425]: pgmap v577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:18 smithi161 ceph-mon[91579]: pgmap v578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:18 smithi188 ceph-mon[92425]: pgmap v578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:20 smithi161 ceph-mon[91579]: pgmap v579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:20 smithi188 ceph-mon[92425]: pgmap v579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:22 smithi161 ceph-mon[91579]: pgmap v580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:22 smithi188 ceph-mon[92425]: pgmap v580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:24 smithi161 ceph-mon[91579]: pgmap v581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:24 smithi188 ceph-mon[92425]: pgmap v581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:26 smithi161 ceph-mon[91579]: pgmap v582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:26 smithi188 ceph-mon[92425]: pgmap v582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:27 smithi161 ceph-mon[91579]: pgmap v583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:27 smithi188 ceph-mon[92425]: pgmap v583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:29 smithi161 ceph-mon[91579]: pgmap v584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:29 smithi188 ceph-mon[92425]: pgmap v584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:32 smithi161 ceph-mon[91579]: pgmap v585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:32 smithi188 ceph-mon[92425]: pgmap v585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:34 smithi161 ceph-mon[91579]: pgmap v586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:34 smithi188 ceph-mon[92425]: pgmap v586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:36 smithi161 ceph-mon[91579]: pgmap v587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:36 smithi188 ceph-mon[92425]: pgmap v587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:38 smithi161 ceph-mon[91579]: pgmap v588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:38 smithi188 ceph-mon[92425]: pgmap v588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:40 smithi161 ceph-mon[91579]: pgmap v589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:40 smithi188 ceph-mon[92425]: pgmap v589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:42 smithi161 ceph-mon[91579]: pgmap v590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:42 smithi188 ceph-mon[92425]: pgmap v590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:44 smithi161 ceph-mon[91579]: pgmap v591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:44 smithi188 ceph-mon[92425]: pgmap v591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:45 smithi161 ceph-mon[91579]: pgmap v592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:45 smithi188 ceph-mon[92425]: pgmap v592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:48 smithi161 ceph-mon[91579]: pgmap v593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:48 smithi188 ceph-mon[92425]: pgmap v593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:50 smithi161 ceph-mon[91579]: pgmap v594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:50 smithi188 ceph-mon[92425]: pgmap v594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:52 smithi161 ceph-mon[91579]: pgmap v595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:52 smithi188 ceph-mon[92425]: pgmap v595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:54 smithi161 ceph-mon[91579]: pgmap v596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:54 smithi188 ceph-mon[92425]: pgmap v596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:58:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:56 smithi161 ceph-mon[91579]: pgmap v597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:58:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:58:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:58:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:58:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:56 smithi188 ceph-mon[92425]: pgmap v597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:58:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:58:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:58:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:58:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:58 smithi161 ceph-mon[91579]: pgmap v598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:58:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:58 smithi188 ceph-mon[92425]: pgmap v598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:58:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: pgmap v599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:59:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:59:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:59:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:59:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:59:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:59:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:59:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T10:59:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:59:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:59:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:59:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T10:59:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:59:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T10:59:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:59:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T10:59:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:59:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:59:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:59:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T10:59:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T10:59:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:59:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T10:59:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:59:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T10:59:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:59:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:59:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:59:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:59:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:59:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:59:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:59:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:59:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:59:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T10:59:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:59:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T10:59:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:59:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:59:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:59:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:59:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:59:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T10:59:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:59:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:59:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:59:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T10:59:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:59:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:59:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=29] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:59:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=29] Local address: 172.21.15.161, port 51016 2022-07-03T10:59:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=29] Peer address: 172.21.15.101, port 22 2022-07-03T10:59:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=29] Connection lost 2022-07-03T10:59:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=29] Aborting connection 2022-07-03T10:59:00.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:58:59.726+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:59:00.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:59:00.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:59:00.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T10:59:00.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:59:00.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:59:00.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:59:00.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T10:59:00.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:59:00.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T10:59:00.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:59:00.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T10:59:00.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:59:00.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:59:00.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T10:59:00.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:59:00.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T10:59:00.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:59:00.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T10:59:00.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:59:00.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T10:59:00.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:59:00.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:59:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:59:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:59:00.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:59:00.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T10:59:00.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:59:00.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:59:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:59:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T10:59:00.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:59:00.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T10:59:00.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:59:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:59:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:59:00.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T10:59:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:59:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T10:59:00.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:59:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T10:59:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:59:00.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T10:59:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:59:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:59:00.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=29] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:59:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=29] Local address: 172.21.15.161, port 51016 2022-07-03T10:59:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=29] Peer address: 172.21.15.101, port 22 2022-07-03T10:59:00.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=29] Connection lost 2022-07-03T10:59:00.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=29] Aborting connection 2022-07-03T10:59:00.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 10:58:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T10:59:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: pgmap v599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T10:59:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T10:59:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T10:59:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T10:59:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:58:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T10:59:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:59:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T10:59:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T10:59:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T10:59:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T10:59:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T10:59:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T10:59:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T10:59:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T10:59:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T10:59:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T10:59:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T10:59:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:59:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T10:59:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T10:59:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T10:59:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T10:59:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T10:59:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T10:59:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T10:59:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T10:59:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T10:59:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T10:59:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T10:59:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T10:59:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T10:59:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T10:59:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T10:59:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T10:59:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T10:59:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T10:59:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T10:59:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T10:59:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T10:59:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T10:59:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T10:59:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T10:59:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T10:59:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T10:59:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T10:59:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T10:59:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T10:59:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T10:59:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T10:59:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=29] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T10:59:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=29] Local address: 172.21.15.161, port 51016 2022-07-03T10:59:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=29] Peer address: 172.21.15.101, port 22 2022-07-03T10:59:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=29] Connection lost 2022-07-03T10:59:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=29] Aborting connection 2022-07-03T10:59:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:02 smithi161 ceph-mon[91579]: pgmap v600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:02 smithi188 ceph-mon[92425]: pgmap v600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:04 smithi161 ceph-mon[91579]: pgmap v601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:04 smithi188 ceph-mon[92425]: pgmap v601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:05 smithi161 ceph-mon[91579]: pgmap v602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:05 smithi188 ceph-mon[92425]: pgmap v602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:08 smithi161 ceph-mon[91579]: pgmap v603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:08 smithi188 ceph-mon[92425]: pgmap v603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:10 smithi161 ceph-mon[91579]: pgmap v604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:10 smithi188 ceph-mon[92425]: pgmap v604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:12 smithi161 ceph-mon[91579]: pgmap v605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:12 smithi188 ceph-mon[92425]: pgmap v605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:14 smithi161 ceph-mon[91579]: pgmap v606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:14 smithi188 ceph-mon[92425]: pgmap v606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:16 smithi161 ceph-mon[91579]: pgmap v607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:16 smithi188 ceph-mon[92425]: pgmap v607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:18 smithi161 ceph-mon[91579]: pgmap v608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:18 smithi188 ceph-mon[92425]: pgmap v608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:19 smithi161 ceph-mon[91579]: pgmap v609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:19 smithi188 ceph-mon[92425]: pgmap v609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:22 smithi161 ceph-mon[91579]: pgmap v610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:22 smithi188 ceph-mon[92425]: pgmap v610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:24 smithi161 ceph-mon[91579]: pgmap v611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:24 smithi188 ceph-mon[92425]: pgmap v611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:26 smithi161 ceph-mon[91579]: pgmap v612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:26 smithi188 ceph-mon[92425]: pgmap v612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:28 smithi161 ceph-mon[91579]: pgmap v613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:28 smithi188 ceph-mon[92425]: pgmap v613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:30 smithi161 ceph-mon[91579]: pgmap v614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:30 smithi188 ceph-mon[92425]: pgmap v614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:32 smithi161 ceph-mon[91579]: pgmap v615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:32 smithi188 ceph-mon[92425]: pgmap v615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:34 smithi161 ceph-mon[91579]: pgmap v616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:34 smithi188 ceph-mon[92425]: pgmap v616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:36 smithi161 ceph-mon[91579]: pgmap v617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:36 smithi188 ceph-mon[92425]: pgmap v617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:38 smithi161 ceph-mon[91579]: pgmap v618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:38 smithi188 ceph-mon[92425]: pgmap v618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:39 smithi161 ceph-mon[91579]: pgmap v619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:39 smithi188 ceph-mon[92425]: pgmap v619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:42 smithi161 ceph-mon[91579]: pgmap v620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:43.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:42 smithi188 ceph-mon[92425]: pgmap v620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:44 smithi161 ceph-mon[91579]: pgmap v621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:44 smithi188 ceph-mon[92425]: pgmap v621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:46 smithi161 ceph-mon[91579]: pgmap v622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:46 smithi188 ceph-mon[92425]: pgmap v622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:47 smithi161 ceph-mon[91579]: pgmap v623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:47 smithi188 ceph-mon[92425]: pgmap v623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:49 smithi161 ceph-mon[91579]: pgmap v624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:49 smithi188 ceph-mon[92425]: pgmap v624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:52 smithi161 ceph-mon[91579]: pgmap v625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:52 smithi188 ceph-mon[92425]: pgmap v625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:54 smithi161 ceph-mon[91579]: pgmap v626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:54 smithi188 ceph-mon[92425]: pgmap v626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T10:59:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:56 smithi161 ceph-mon[91579]: pgmap v627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:59:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:59:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:59:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:59:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:56 smithi188 ceph-mon[92425]: pgmap v627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:59:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:59:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T10:59:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T10:59:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:58 smithi161 ceph-mon[91579]: pgmap v628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T10:59:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:58 smithi188 ceph-mon[92425]: pgmap v628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T10:59:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 10:59:59 smithi161 ceph-mon[91579]: pgmap v629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 10:59:59 smithi188 ceph-mon[92425]: pgmap v629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:00:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T11:00:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T11:00:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-07-03T11:00:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: pg 2.1 is stuck inactive for 21m, current state unknown, last acting [] 2022-07-03T11:00:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:00:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:00:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:00:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:00:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-07-03T11:00:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: mon.a (rank 0) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2022-07-03T11:00:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-07-03T11:00:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: pg 2.1 is stuck inactive for 21m, current state unknown, last acting [] 2022-07-03T11:00:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:00:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:00:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:00:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:02 smithi161 ceph-mon[91579]: pgmap v630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:02 smithi188 ceph-mon[92425]: pgmap v630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:04 smithi161 ceph-mon[91579]: pgmap v631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:04 smithi188 ceph-mon[92425]: pgmap v631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:06 smithi161 ceph-mon[91579]: pgmap v632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:06 smithi188 ceph-mon[92425]: pgmap v632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:08 smithi161 ceph-mon[91579]: pgmap v633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:08 smithi188 ceph-mon[92425]: pgmap v633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:09 smithi161 ceph-mon[91579]: pgmap v634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:09 smithi188 ceph-mon[92425]: pgmap v634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:12 smithi161 ceph-mon[91579]: pgmap v635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:12 smithi188 ceph-mon[92425]: pgmap v635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:14 smithi161 ceph-mon[91579]: pgmap v636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:14 smithi188 ceph-mon[92425]: pgmap v636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:16 smithi161 ceph-mon[91579]: pgmap v637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:16 smithi188 ceph-mon[92425]: pgmap v637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:18 smithi161 ceph-mon[91579]: pgmap v638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:18 smithi188 ceph-mon[92425]: pgmap v638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:19 smithi161 ceph-mon[91579]: pgmap v639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:19 smithi188 ceph-mon[92425]: pgmap v639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:22 smithi161 ceph-mon[91579]: pgmap v640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:22 smithi188 ceph-mon[92425]: pgmap v640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:24 smithi161 ceph-mon[91579]: pgmap v641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:24 smithi188 ceph-mon[92425]: pgmap v641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:26 smithi161 ceph-mon[91579]: pgmap v642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:26 smithi188 ceph-mon[92425]: pgmap v642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:28 smithi161 ceph-mon[91579]: pgmap v643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:28 smithi188 ceph-mon[92425]: pgmap v643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:29 smithi161 ceph-mon[91579]: pgmap v644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:29 smithi188 ceph-mon[92425]: pgmap v644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:00:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:00:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:32 smithi161 ceph-mon[91579]: pgmap v645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:32 smithi188 ceph-mon[92425]: pgmap v645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:34 smithi161 ceph-mon[91579]: pgmap v646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:34 smithi188 ceph-mon[92425]: pgmap v646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:36 smithi161 ceph-mon[91579]: pgmap v647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:36 smithi188 ceph-mon[92425]: pgmap v647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:37 smithi161 ceph-mon[91579]: pgmap v648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:37 smithi188 ceph-mon[92425]: pgmap v648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:39 smithi161 ceph-mon[91579]: pgmap v649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:39 smithi188 ceph-mon[92425]: pgmap v649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:42 smithi161 ceph-mon[91579]: pgmap v650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:42 smithi188 ceph-mon[92425]: pgmap v650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:44 smithi161 ceph-mon[91579]: pgmap v651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:44 smithi188 ceph-mon[92425]: pgmap v651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:46 smithi161 ceph-mon[91579]: pgmap v652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:46 smithi188 ceph-mon[92425]: pgmap v652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:47 smithi161 ceph-mon[91579]: pgmap v653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:47 smithi188 ceph-mon[92425]: pgmap v653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:49 smithi161 ceph-mon[91579]: pgmap v654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:49 smithi188 ceph-mon[92425]: pgmap v654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:52 smithi161 ceph-mon[91579]: pgmap v655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:52 smithi188 ceph-mon[92425]: pgmap v655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:54 smithi161 ceph-mon[91579]: pgmap v656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:54 smithi188 ceph-mon[92425]: pgmap v656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:00:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:56 smithi161 ceph-mon[91579]: pgmap v657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:00:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:00:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:00:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:00:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:56 smithi188 ceph-mon[92425]: pgmap v657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:00:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:00:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:00:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:00:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:57 smithi161 ceph-mon[91579]: pgmap v658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:57 smithi188 ceph-mon[92425]: pgmap v658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:00:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:00:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:00:59 smithi161 ceph-mon[91579]: pgmap v659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:00:59 smithi188 ceph-mon[92425]: pgmap v659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:00.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:01:00.229+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:01:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:01:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:01:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:01:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:01:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:01:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:01:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:01:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:01:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:01:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:01:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:01:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:01:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:01:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:01:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:01:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:01:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:01:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:01:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:01:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:01:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:01:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:01:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:01:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:01:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:01:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:01:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:01:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:01:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:01:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:01:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:01:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:01:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:01:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:01:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:01:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:01:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:01:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:01:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:01:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:01:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:01:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:01:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:01:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:01:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=31] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:01:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=31] Local address: 172.21.15.161, port 51020 2022-07-03T11:01:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=31] Peer address: 172.21.15.101, port 22 2022-07-03T11:01:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=31] Login timeout expired 2022-07-03T11:01:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=31] Aborting connection 2022-07-03T11:01:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:01:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:01:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:01:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:01:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:01:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:01:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:01:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:01:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:01:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:01:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:01:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:01:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:01:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:01:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:01:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:01:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:01:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:01:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:01:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:01:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:01:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:01:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:01:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:01:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:01:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:01:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:01:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:01:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:01:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:01:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:01:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:01:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:01:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:01:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:01:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:01:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:01:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:01:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:01:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:01:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:01:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:01:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:01:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:01:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:01:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:01:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=31] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:01:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=31] Local address: 172.21.15.161, port 51020 2022-07-03T11:01:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=31] Peer address: 172.21.15.101, port 22 2022-07-03T11:01:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=31] Login timeout expired 2022-07-03T11:01:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=31] Aborting connection 2022-07-03T11:01:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:01:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:01:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:01:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:01:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:01:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:01:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:01:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:01:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:01:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:01:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:01:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:01:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:01:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:01:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:01:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:01:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:01:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:01:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:01:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:01:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:01:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:01:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:01:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:01:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:01:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:01:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:01:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:01:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:01:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:01:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:01:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:01:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:01:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:01:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:01:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:01:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:01:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:01:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:01:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:01:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:01:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:01:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:01:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:01:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:01:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=31] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:01:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=31] Local address: 172.21.15.161, port 51020 2022-07-03T11:01:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=31] Peer address: 172.21.15.101, port 22 2022-07-03T11:01:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=31] Login timeout expired 2022-07-03T11:01:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=31] Aborting connection 2022-07-03T11:01:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:02 smithi161 ceph-mon[91579]: pgmap v660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:02 smithi188 ceph-mon[92425]: pgmap v660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:04 smithi161 ceph-mon[91579]: pgmap v661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:04 smithi188 ceph-mon[92425]: pgmap v661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:06 smithi161 ceph-mon[91579]: pgmap v662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:06 smithi188 ceph-mon[92425]: pgmap v662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:07 smithi161 ceph-mon[91579]: pgmap v663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:07 smithi188 ceph-mon[92425]: pgmap v663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:09 smithi161 ceph-mon[91579]: pgmap v664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:09 smithi188 ceph-mon[92425]: pgmap v664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:12 smithi161 ceph-mon[91579]: pgmap v665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:12 smithi188 ceph-mon[92425]: pgmap v665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:14 smithi161 ceph-mon[91579]: pgmap v666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:14 smithi188 ceph-mon[92425]: pgmap v666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:16 smithi161 ceph-mon[91579]: pgmap v667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:16 smithi188 ceph-mon[92425]: pgmap v667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:17 smithi161 ceph-mon[91579]: pgmap v668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:17 smithi188 ceph-mon[92425]: pgmap v668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:19 smithi161 ceph-mon[91579]: pgmap v669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:19 smithi188 ceph-mon[92425]: pgmap v669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:22 smithi161 ceph-mon[91579]: pgmap v670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:22 smithi188 ceph-mon[92425]: pgmap v670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:24 smithi161 ceph-mon[91579]: pgmap v671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:24 smithi188 ceph-mon[92425]: pgmap v671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:25 smithi161 ceph-mon[91579]: pgmap v672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:25 smithi188 ceph-mon[92425]: pgmap v672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:28 smithi161 ceph-mon[91579]: pgmap v673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:28 smithi188 ceph-mon[92425]: pgmap v673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:30 smithi161 ceph-mon[91579]: pgmap v674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:30 smithi188 ceph-mon[92425]: pgmap v674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:32 smithi161 ceph-mon[91579]: pgmap v675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:32 smithi188 ceph-mon[92425]: pgmap v675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:33 smithi161 ceph-mon[91579]: pgmap v676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:33 smithi188 ceph-mon[92425]: pgmap v676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:36 smithi161 ceph-mon[91579]: pgmap v677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:36 smithi188 ceph-mon[92425]: pgmap v677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:38 smithi161 ceph-mon[91579]: pgmap v678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:38 smithi188 ceph-mon[92425]: pgmap v678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:40 smithi161 ceph-mon[91579]: pgmap v679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:40 smithi188 ceph-mon[92425]: pgmap v679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:42 smithi161 ceph-mon[91579]: pgmap v680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:42 smithi188 ceph-mon[92425]: pgmap v680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:44.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:44 smithi161 ceph-mon[91579]: pgmap v681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:44 smithi188 ceph-mon[92425]: pgmap v681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:45 smithi161 ceph-mon[91579]: pgmap v682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:45 smithi188 ceph-mon[92425]: pgmap v682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:48 smithi161 ceph-mon[91579]: pgmap v683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:48 smithi188 ceph-mon[92425]: pgmap v683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:50 smithi161 ceph-mon[91579]: pgmap v684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:50 smithi188 ceph-mon[92425]: pgmap v684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:52 smithi161 ceph-mon[91579]: pgmap v685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:52 smithi188 ceph-mon[92425]: pgmap v685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:54 smithi161 ceph-mon[91579]: pgmap v686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:54 smithi188 ceph-mon[92425]: pgmap v686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:01:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:55 smithi161 ceph-mon[91579]: pgmap v687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:55 smithi188 ceph-mon[92425]: pgmap v687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:01:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:01:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:01:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:01:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:01:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:01:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:01:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:01:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:57 smithi161 ceph-mon[91579]: pgmap v688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:57 smithi188 ceph-mon[92425]: pgmap v688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:01:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:01:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:01:59 smithi161 ceph-mon[91579]: pgmap v689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:01:59 smithi188 ceph-mon[92425]: pgmap v689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:02:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:02:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:02:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:02:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:02:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:02:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:02 smithi161 ceph-mon[91579]: pgmap v690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:02 smithi188 ceph-mon[92425]: pgmap v690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: pgmap v691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: pgmap v691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:02:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:02:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:02:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:02:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:02:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:02:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:02:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:02:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:05 smithi161 ceph-mon[91579]: pgmap v692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:05 smithi188 ceph-mon[92425]: pgmap v692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:08 smithi161 ceph-mon[91579]: pgmap v693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:08 smithi188 ceph-mon[92425]: pgmap v693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:09 smithi161 ceph-mon[91579]: pgmap v694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:09 smithi188 ceph-mon[92425]: pgmap v694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:12 smithi161 ceph-mon[91579]: pgmap v695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:12 smithi188 ceph-mon[92425]: pgmap v695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:14 smithi161 ceph-mon[91579]: pgmap v696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:14 smithi188 ceph-mon[92425]: pgmap v696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:15 smithi161 ceph-mon[91579]: pgmap v697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:15 smithi188 ceph-mon[92425]: pgmap v697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:18 smithi161 ceph-mon[91579]: pgmap v698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:18 smithi188 ceph-mon[92425]: pgmap v698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:19 smithi161 ceph-mon[91579]: pgmap v699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:19 smithi188 ceph-mon[92425]: pgmap v699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:22 smithi161 ceph-mon[91579]: pgmap v700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:22 smithi188 ceph-mon[92425]: pgmap v700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:02:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:24 smithi161 ceph-mon[91579]: pgmap v701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:24 smithi188 ceph-mon[92425]: pgmap v701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:25 smithi161 ceph-mon[91579]: pgmap v702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:25 smithi188 ceph-mon[92425]: pgmap v702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:28 smithi161 ceph-mon[91579]: pgmap v703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:28 smithi188 ceph-mon[92425]: pgmap v703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:29 smithi161 ceph-mon[91579]: pgmap v704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:29 smithi188 ceph-mon[92425]: pgmap v704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:02:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:02:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:32 smithi161 ceph-mon[91579]: pgmap v705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:32 smithi188 ceph-mon[92425]: pgmap v705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:34 smithi161 ceph-mon[91579]: pgmap v706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:34 smithi188 ceph-mon[92425]: pgmap v706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:35 smithi161 ceph-mon[91579]: pgmap v707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:35 smithi188 ceph-mon[92425]: pgmap v707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:38 smithi161 ceph-mon[91579]: pgmap v708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:38 smithi188 ceph-mon[92425]: pgmap v708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:39 smithi161 ceph-mon[91579]: pgmap v709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:39 smithi188 ceph-mon[92425]: pgmap v709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:42 smithi161 ceph-mon[91579]: pgmap v710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:42 smithi188 ceph-mon[92425]: pgmap v710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:44 smithi161 ceph-mon[91579]: pgmap v711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:44 smithi188 ceph-mon[92425]: pgmap v711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:45 smithi161 ceph-mon[91579]: pgmap v712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:45 smithi188 ceph-mon[92425]: pgmap v712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:48 smithi161 ceph-mon[91579]: pgmap v713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:48 smithi188 ceph-mon[92425]: pgmap v713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:49 smithi161 ceph-mon[91579]: pgmap v714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:49 smithi188 ceph-mon[92425]: pgmap v714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:52 smithi161 ceph-mon[91579]: pgmap v715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:52 smithi188 ceph-mon[92425]: pgmap v715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:53 smithi161 ceph-mon[91579]: pgmap v716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:53 smithi188 ceph-mon[92425]: pgmap v716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:02:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:56 smithi161 ceph-mon[91579]: pgmap v717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:56 smithi188 ceph-mon[92425]: pgmap v717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:02:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:02:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:02:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:02:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:02:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:02:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:02:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:02:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:58 smithi161 ceph-mon[91579]: pgmap v718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:58 smithi188 ceph-mon[92425]: pgmap v718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:02:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:02:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-mon[91579]: pgmap v719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:00.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:03:00.272+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:03:00.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:03:00.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:03:00.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:03:00.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:03:00.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:03:00.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:03:00.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:03:00.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:03:00.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:03:00.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:03:00.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:03:00.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:03:00.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:03:00.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:03:00.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:03:00.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:03:00.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:03:00.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:03:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:03:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:03:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:03:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:03:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:03:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:03:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:03:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:03:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:03:00.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:03:00.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:03:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:03:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:03:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:03:00.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:03:00.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:03:00.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:03:00.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:03:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:03:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:03:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:03:00.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:03:00.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:03:00.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:03:00.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:03:00.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:03:00.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=33] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:03:00.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=33] Local address: 172.21.15.161, port 51024 2022-07-03T11:03:00.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=33] Peer address: 172.21.15.101, port 22 2022-07-03T11:03:00.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=33] Login timeout expired 2022-07-03T11:03:00.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=33] Aborting connection 2022-07-03T11:03:00.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:03:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:03:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:00 smithi188 ceph-mon[92425]: pgmap v719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:03:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:03:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:03:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:03:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:03:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:03:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:03:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:03:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:03:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:03:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:03:01.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:03:01.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:03:01.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:03:01.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:03:01.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:03:01.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:03:01.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:03:01.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:03:01.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:03:01.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:03:01.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:03:01.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:03:01.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:03:01.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:03:01.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:03:01.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:03:01.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:03:01.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:03:01.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:03:01.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:03:01.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:03:01.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:03:01.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:03:01.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:03:01.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:03:01.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:03:01.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:03:01.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:03:01.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:03:01.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:03:01.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:03:01.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:03:01.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:03:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:03:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=33] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:03:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=33] Local address: 172.21.15.161, port 51024 2022-07-03T11:03:01.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=33] Peer address: 172.21.15.101, port 22 2022-07-03T11:03:01.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=33] Login timeout expired 2022-07-03T11:03:01.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=33] Aborting connection 2022-07-03T11:03:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:03:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:03:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:03:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:03:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:03:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:03:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:03:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:03:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:03:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:03:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:03:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:03:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:03:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:03:01.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:03:01.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:03:01.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:03:01.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:03:01.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:03:01.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:03:01.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:03:01.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:03:01.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:03:01.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:03:01.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:03:01.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:03:01.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:03:01.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:03:01.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:03:01.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:03:01.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:03:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:03:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:03:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:03:01.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:03:01.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:03:01.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:03:01.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:03:01.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:03:01.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:03:01.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:03:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:03:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:03:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:03:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:03:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=33] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:03:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=33] Local address: 172.21.15.161, port 51024 2022-07-03T11:03:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=33] Peer address: 172.21.15.101, port 22 2022-07-03T11:03:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=33] Login timeout expired 2022-07-03T11:03:01.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=33] Aborting connection 2022-07-03T11:03:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:02 smithi161 ceph-mon[91579]: pgmap v720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:02 smithi188 ceph-mon[92425]: pgmap v720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:03 smithi161 ceph-mon[91579]: pgmap v721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:03 smithi188 ceph-mon[92425]: pgmap v721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:06 smithi161 ceph-mon[91579]: pgmap v722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:06 smithi188 ceph-mon[92425]: pgmap v722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:07 smithi188 ceph-mon[92425]: pgmap v723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:07 smithi161 ceph-mon[91579]: pgmap v723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:09 smithi188 ceph-mon[92425]: pgmap v724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:09 smithi161 ceph-mon[91579]: pgmap v724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:12 smithi161 ceph-mon[91579]: pgmap v725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:12 smithi188 ceph-mon[92425]: pgmap v725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:13 smithi161 ceph-mon[91579]: pgmap v726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:13 smithi188 ceph-mon[92425]: pgmap v726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:16 smithi161 ceph-mon[91579]: pgmap v727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:16 smithi188 ceph-mon[92425]: pgmap v727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:17 smithi188 ceph-mon[92425]: pgmap v728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:17 smithi161 ceph-mon[91579]: pgmap v728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:19 smithi188 ceph-mon[92425]: pgmap v729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:19 smithi161 ceph-mon[91579]: pgmap v729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:22 smithi161 ceph-mon[91579]: pgmap v730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:22 smithi188 ceph-mon[92425]: pgmap v730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:24 smithi161 ceph-mon[91579]: pgmap v731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:24 smithi188 ceph-mon[92425]: pgmap v731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:26 smithi161 ceph-mon[91579]: pgmap v732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:26 smithi188 ceph-mon[92425]: pgmap v732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:27 smithi161 ceph-mon[91579]: pgmap v733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:27 smithi188 ceph-mon[92425]: pgmap v733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:29 smithi161 ceph-mon[91579]: pgmap v734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:29 smithi188 ceph-mon[92425]: pgmap v734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:32 smithi161 ceph-mon[91579]: pgmap v735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:32 smithi188 ceph-mon[92425]: pgmap v735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:34 smithi161 ceph-mon[91579]: pgmap v736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:34 smithi188 ceph-mon[92425]: pgmap v736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:36 smithi161 ceph-mon[91579]: pgmap v737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:36 smithi188 ceph-mon[92425]: pgmap v737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:37 smithi161 ceph-mon[91579]: pgmap v738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:37 smithi188 ceph-mon[92425]: pgmap v738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:39 smithi161 ceph-mon[91579]: pgmap v739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:39 smithi188 ceph-mon[92425]: pgmap v739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:42 smithi161 ceph-mon[91579]: pgmap v740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:42 smithi188 ceph-mon[92425]: pgmap v740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:44 smithi161 ceph-mon[91579]: pgmap v741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:44 smithi188 ceph-mon[92425]: pgmap v741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:46 smithi161 ceph-mon[91579]: pgmap v742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:46 smithi188 ceph-mon[92425]: pgmap v742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:47 smithi161 ceph-mon[91579]: pgmap v743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:47 smithi188 ceph-mon[92425]: pgmap v743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:49 smithi161 ceph-mon[91579]: pgmap v744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:49 smithi188 ceph-mon[92425]: pgmap v744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:52 smithi161 ceph-mon[91579]: pgmap v745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:52 smithi188 ceph-mon[92425]: pgmap v745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:54 smithi161 ceph-mon[91579]: pgmap v746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:54 smithi188 ceph-mon[92425]: pgmap v746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:03:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:56 smithi161 ceph-mon[91579]: pgmap v747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:56 smithi188 ceph-mon[92425]: pgmap v747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:03:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:03:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:03:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:03:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:03:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:03:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:58 smithi188 ceph-mon[92425]: pgmap v748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:58 smithi161 ceph-mon[91579]: pgmap v748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:03:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:03:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:00 smithi188 ceph-mon[92425]: pgmap v749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:00 smithi161 ceph-mon[91579]: pgmap v749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:04:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:04:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:04:01.606 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:04:01.607 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:04:01.607 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:04:02.358 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:02 smithi161 ceph-mon[91579]: pgmap v750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:02 smithi188 ceph-mon[92425]: pgmap v750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:04 smithi188 ceph-mon[92425]: pgmap v751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:04 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T11:04:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:04:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:04:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:04 smithi161 ceph-mon[91579]: pgmap v751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:04 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T11:04:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:04:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:04:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:06 smithi188 ceph-mon[92425]: pgmap v752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:06 smithi161 ceph-mon[91579]: pgmap v752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:07 smithi161 ceph-mon[91579]: pgmap v753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:07 smithi188 ceph-mon[92425]: pgmap v753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:09 smithi161 ceph-mon[91579]: pgmap v754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:09 smithi188 ceph-mon[92425]: pgmap v754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:12 smithi161 ceph-mon[91579]: pgmap v755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:12 smithi188 ceph-mon[92425]: pgmap v755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:14 smithi188 ceph-mon[92425]: pgmap v756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:14 smithi161 ceph-mon[91579]: pgmap v756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:16 smithi188 ceph-mon[92425]: pgmap v757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:16 smithi161 ceph-mon[91579]: pgmap v757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:17 smithi161 ceph-mon[91579]: pgmap v758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:17 smithi188 ceph-mon[92425]: pgmap v758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:19 smithi161 ceph-mon[91579]: pgmap v759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:19 smithi188 ceph-mon[92425]: pgmap v759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:22 smithi188 ceph-mon[92425]: pgmap v760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:22 smithi161 ceph-mon[91579]: pgmap v760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:24 smithi188 ceph-mon[92425]: pgmap v761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:24 smithi161 ceph-mon[91579]: pgmap v761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:26 smithi188 ceph-mon[92425]: pgmap v762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:26 smithi161 ceph-mon[91579]: pgmap v762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:27 smithi161 ceph-mon[91579]: pgmap v763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:27 smithi188 ceph-mon[92425]: pgmap v763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:29 smithi161 ceph-mon[91579]: pgmap v764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:29 smithi188 ceph-mon[92425]: pgmap v764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:30.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:04:30.307+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:04:30.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:04:30.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:04:30.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:04:30.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:04:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:04:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:04:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:04:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:04:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:04:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:04:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:04:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:04:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:04:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:04:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:04:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:04:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:04:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:04:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:04:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:04:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:04:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:04:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:04:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:04:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:04:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:04:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:04:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:04:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:04:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:04:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:04:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:04:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:04:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:04:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:04:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:04:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:04:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:04:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:04:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:04:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:04:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:04:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:04:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:04:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=35] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:04:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=35] Local address: 172.21.15.161, port 51028 2022-07-03T11:04:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=35] Peer address: 172.21.15.101, port 22 2022-07-03T11:04:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=35] Connection lost 2022-07-03T11:04:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=35] Aborting connection 2022-07-03T11:04:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:04:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:04:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:04:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:04:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:04:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:04:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:04:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:04:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:04:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:04:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:04:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:04:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:04:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:04:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:04:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:04:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:04:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:04:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:04:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:04:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:04:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:04:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:04:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:04:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:04:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:04:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:04:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:04:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:04:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:04:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:04:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:04:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:04:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:04:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:04:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:04:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:04:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:04:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:04:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:04:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:04:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:04:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:04:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:04:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:04:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:04:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:04:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=35] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:04:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=35] Local address: 172.21.15.161, port 51028 2022-07-03T11:04:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=35] Peer address: 172.21.15.101, port 22 2022-07-03T11:04:31.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=35] Connection lost 2022-07-03T11:04:31.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=35] Aborting connection 2022-07-03T11:04:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:04:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:04:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:04:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:04:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:04:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:04:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:04:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:04:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:04:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:04:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:04:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:04:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:04:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:04:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:04:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:04:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:04:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:04:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:04:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:04:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:04:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:04:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:04:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:04:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:04:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:04:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:04:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:04:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:04:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:04:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:04:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:04:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:04:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:04:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:04:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:04:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:04:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:04:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:04:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:04:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:04:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:04:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:04:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:04:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:04:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:04:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=35] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:04:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=35] Local address: 172.21.15.161, port 51028 2022-07-03T11:04:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=35] Peer address: 172.21.15.101, port 22 2022-07-03T11:04:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=35] Connection lost 2022-07-03T11:04:31.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=35] Aborting connection 2022-07-03T11:04:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:31 smithi161 ceph-mon[91579]: pgmap v765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:31 smithi188 ceph-mon[92425]: pgmap v765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:34 smithi188 ceph-mon[92425]: pgmap v766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:34 smithi161 ceph-mon[91579]: pgmap v766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:36 smithi188 ceph-mon[92425]: pgmap v767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:36 smithi161 ceph-mon[91579]: pgmap v767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:37 smithi161 ceph-mon[91579]: pgmap v768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:37 smithi188 ceph-mon[92425]: pgmap v768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:39 smithi161 ceph-mon[91579]: pgmap v769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:39 smithi188 ceph-mon[92425]: pgmap v769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:42 smithi188 ceph-mon[92425]: pgmap v770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:42 smithi161 ceph-mon[91579]: pgmap v770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:44 smithi188 ceph-mon[92425]: pgmap v771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:44 smithi161 ceph-mon[91579]: pgmap v771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:46 smithi188 ceph-mon[92425]: pgmap v772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:46 smithi161 ceph-mon[91579]: pgmap v772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:47 smithi161 ceph-mon[91579]: pgmap v773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:47 smithi188 ceph-mon[92425]: pgmap v773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:49 smithi161 ceph-mon[91579]: pgmap v774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:49 smithi188 ceph-mon[92425]: pgmap v774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:52 smithi188 ceph-mon[92425]: pgmap v775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:52 smithi161 ceph-mon[91579]: pgmap v775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:54 smithi188 ceph-mon[92425]: pgmap v776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:54 smithi161 ceph-mon[91579]: pgmap v776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:04:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:56 smithi188 ceph-mon[92425]: pgmap v777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:56 smithi161 ceph-mon[91579]: pgmap v777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:04:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:04:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:04:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:04:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:04:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:04:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:04:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:04:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:58 smithi188 ceph-mon[92425]: pgmap v778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:58 smithi161 ceph-mon[91579]: pgmap v778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:04:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:04:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:00 smithi188 ceph-mon[92425]: pgmap v779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:00 smithi161 ceph-mon[91579]: pgmap v779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:02 smithi188 ceph-mon[92425]: pgmap v780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:02 smithi161 ceph-mon[91579]: pgmap v780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:04 smithi161 ceph-mon[91579]: pgmap v781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:04 smithi188 ceph-mon[92425]: pgmap v781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:06 smithi161 ceph-mon[91579]: pgmap v782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:06 smithi188 ceph-mon[92425]: pgmap v782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:07 smithi161 ceph-mon[91579]: pgmap v783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:07 smithi188 ceph-mon[92425]: pgmap v783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:09 smithi161 ceph-mon[91579]: pgmap v784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:09 smithi188 ceph-mon[92425]: pgmap v784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:12 smithi188 ceph-mon[92425]: pgmap v785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:12 smithi161 ceph-mon[91579]: pgmap v785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:14 smithi188 ceph-mon[92425]: pgmap v786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:14 smithi161 ceph-mon[91579]: pgmap v786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:16 smithi188 ceph-mon[92425]: pgmap v787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:16 smithi161 ceph-mon[91579]: pgmap v787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:17 smithi161 ceph-mon[91579]: pgmap v788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:17 smithi188 ceph-mon[92425]: pgmap v788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:20.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:19 smithi161 ceph-mon[91579]: pgmap v789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:19 smithi188 ceph-mon[92425]: pgmap v789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:22 smithi188 ceph-mon[92425]: pgmap v790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:22 smithi161 ceph-mon[91579]: pgmap v790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:24 smithi188 ceph-mon[92425]: pgmap v791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:24 smithi161 ceph-mon[91579]: pgmap v791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:26 smithi188 ceph-mon[92425]: pgmap v792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:26 smithi161 ceph-mon[91579]: pgmap v792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:27 smithi161 ceph-mon[91579]: pgmap v793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:27 smithi188 ceph-mon[92425]: pgmap v793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:29 smithi161 ceph-mon[91579]: pgmap v794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:29 smithi188 ceph-mon[92425]: pgmap v794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:05:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:05:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:05:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:05:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:05:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:05:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:31 smithi188 ceph-mon[92425]: pgmap v795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:31 smithi161 ceph-mon[91579]: pgmap v795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:34 smithi188 ceph-mon[92425]: pgmap v796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:34 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T11:05:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:05:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:05:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:34 smithi161 ceph-mon[91579]: pgmap v796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:34 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T11:05:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:05:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:05:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:36 smithi188 ceph-mon[92425]: pgmap v797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:36 smithi161 ceph-mon[91579]: pgmap v797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:37 smithi161 ceph-mon[91579]: pgmap v798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:37 smithi188 ceph-mon[92425]: pgmap v798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:40.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:39 smithi161 ceph-mon[91579]: pgmap v799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:39 smithi188 ceph-mon[92425]: pgmap v799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:42 smithi188 ceph-mon[92425]: pgmap v800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:42 smithi161 ceph-mon[91579]: pgmap v800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:44 smithi188 ceph-mon[92425]: pgmap v801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:44 smithi161 ceph-mon[91579]: pgmap v801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:46 smithi188 ceph-mon[92425]: pgmap v802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:46 smithi161 ceph-mon[91579]: pgmap v802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:47 smithi161 ceph-mon[91579]: pgmap v803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:47 smithi188 ceph-mon[92425]: pgmap v803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:49.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:49 smithi188 ceph-mon[92425]: pgmap v804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:49 smithi161 ceph-mon[91579]: pgmap v804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:52 smithi188 ceph-mon[92425]: pgmap v805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:52 smithi161 ceph-mon[91579]: pgmap v805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:54 smithi188 ceph-mon[92425]: pgmap v806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:54 smithi161 ceph-mon[91579]: pgmap v806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:05:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:56 smithi161 ceph-mon[91579]: pgmap v807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:56 smithi188 ceph-mon[92425]: pgmap v807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:05:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:05:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:05:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:05:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:58 smithi161 ceph-mon[91579]: pgmap v808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:58 smithi188 ceph-mon[92425]: pgmap v808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:05:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:05:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:00 smithi161 ceph-mon[91579]: pgmap v809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:00 smithi188 ceph-mon[92425]: pgmap v809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:06:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:06:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:02 smithi161 ceph-mon[91579]: pgmap v810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:02 smithi188 ceph-mon[92425]: pgmap v810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:04 smithi161 ceph-mon[91579]: pgmap v811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:04 smithi188 ceph-mon[92425]: pgmap v811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:06 smithi161 ceph-mon[91579]: pgmap v812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:06 smithi188 ceph-mon[92425]: pgmap v812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:07 smithi161 ceph-mon[91579]: pgmap v813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:07 smithi188 ceph-mon[92425]: pgmap v813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:09 smithi188 ceph-mon[92425]: pgmap v814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:09 smithi161 ceph-mon[91579]: pgmap v814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:12 smithi161 ceph-mon[91579]: pgmap v815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:12 smithi188 ceph-mon[92425]: pgmap v815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:14 smithi161 ceph-mon[91579]: pgmap v816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:14 smithi188 ceph-mon[92425]: pgmap v816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:16 smithi161 ceph-mon[91579]: pgmap v817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:16 smithi188 ceph-mon[92425]: pgmap v817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:17 smithi188 ceph-mon[92425]: pgmap v818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:17 smithi161 ceph-mon[91579]: pgmap v818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:19 smithi188 ceph-mon[92425]: pgmap v819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:19 smithi161 ceph-mon[91579]: pgmap v819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:22 smithi161 ceph-mon[91579]: pgmap v820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:22 smithi188 ceph-mon[92425]: pgmap v820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:24 smithi161 ceph-mon[91579]: pgmap v821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:24 smithi188 ceph-mon[92425]: pgmap v821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:26 smithi161 ceph-mon[91579]: pgmap v822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:26 smithi188 ceph-mon[92425]: pgmap v822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:27 smithi188 ceph-mon[92425]: pgmap v823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:27 smithi161 ceph-mon[91579]: pgmap v823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:29 smithi188 ceph-mon[92425]: pgmap v824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:30.346 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:29 smithi161 ceph-mon[91579]: pgmap v824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:30.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:06:30.344+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:06:30.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:06:30.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:06:30.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:06:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:06:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:06:30.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:06:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:06:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:06:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:06:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:06:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:06:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:06:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:06:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:06:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:06:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:06:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:06:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:06:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:06:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:06:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:06:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:06:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:06:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:06:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:06:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:06:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:06:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:06:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:06:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:06:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:06:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:06:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:06:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:06:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:06:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:06:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:06:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:06:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:06:30.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:06:30.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:06:30.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:06:30.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:06:30.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:06:30.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=37] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:06:30.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=37] Local address: 172.21.15.161, port 51032 2022-07-03T11:06:30.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=37] Peer address: 172.21.15.101, port 22 2022-07-03T11:06:30.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=37] Login timeout expired 2022-07-03T11:06:30.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=37] Aborting connection 2022-07-03T11:06:30.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:06:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:06:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:06:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:06:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:06:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:06:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:06:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:06:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:06:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:06:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:06:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:06:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:06:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:06:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:06:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:06:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:06:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:06:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:06:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:06:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:06:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:06:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:06:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:06:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:06:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:06:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:06:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:06:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:06:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:06:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:06:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:06:31.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:06:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:06:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:06:31.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:06:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:06:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:06:31.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:06:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:06:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:06:31.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:06:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:06:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:06:31.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:06:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:06:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=37] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:06:31.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=37] Local address: 172.21.15.161, port 51032 2022-07-03T11:06:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=37] Peer address: 172.21.15.101, port 22 2022-07-03T11:06:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=37] Login timeout expired 2022-07-03T11:06:31.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=37] Aborting connection 2022-07-03T11:06:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:06:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:06:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:06:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:06:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:06:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:06:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:06:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:06:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:06:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:06:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:06:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:06:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:06:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:06:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:06:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:06:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:06:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:06:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:06:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:06:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:06:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:06:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:06:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:06:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:06:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:06:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:06:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:06:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:06:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:06:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:06:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:06:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:06:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:06:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:06:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:06:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:06:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:06:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:06:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:06:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:06:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:06:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:06:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:06:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:06:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=37] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:06:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=37] Local address: 172.21.15.161, port 51032 2022-07-03T11:06:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=37] Peer address: 172.21.15.101, port 22 2022-07-03T11:06:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=37] Login timeout expired 2022-07-03T11:06:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=37] Aborting connection 2022-07-03T11:06:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:31 smithi188 ceph-mon[92425]: pgmap v825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:31 smithi161 ceph-mon[91579]: pgmap v825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:34 smithi161 ceph-mon[91579]: pgmap v826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:34 smithi188 ceph-mon[92425]: pgmap v826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:36 smithi161 ceph-mon[91579]: pgmap v827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:36 smithi188 ceph-mon[92425]: pgmap v827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:37 smithi188 ceph-mon[92425]: pgmap v828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:37 smithi161 ceph-mon[91579]: pgmap v828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:39 smithi188 ceph-mon[92425]: pgmap v829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:39 smithi161 ceph-mon[91579]: pgmap v829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:42 smithi161 ceph-mon[91579]: pgmap v830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:42 smithi188 ceph-mon[92425]: pgmap v830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:44 smithi161 ceph-mon[91579]: pgmap v831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:44 smithi188 ceph-mon[92425]: pgmap v831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:46 smithi161 ceph-mon[91579]: pgmap v832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:46 smithi188 ceph-mon[92425]: pgmap v832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:47 smithi188 ceph-mon[92425]: pgmap v833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:47 smithi161 ceph-mon[91579]: pgmap v833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:49 smithi188 ceph-mon[92425]: pgmap v834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:49 smithi161 ceph-mon[91579]: pgmap v834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:52 smithi161 ceph-mon[91579]: pgmap v835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:52 smithi188 ceph-mon[92425]: pgmap v835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:54 smithi161 ceph-mon[91579]: pgmap v836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:54 smithi188 ceph-mon[92425]: pgmap v836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:06:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:56 smithi161 ceph-mon[91579]: pgmap v837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:56 smithi188 ceph-mon[92425]: pgmap v837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:06:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:06:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:06:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:06:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:06:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:06:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:58 smithi161 ceph-mon[91579]: pgmap v838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:58 smithi188 ceph-mon[92425]: pgmap v838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:06:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:06:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:00 smithi161 ceph-mon[91579]: pgmap v839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:00 smithi188 ceph-mon[92425]: pgmap v839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:02 smithi161 ceph-mon[91579]: pgmap v840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:02 smithi188 ceph-mon[92425]: pgmap v840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:04 smithi161 ceph-mon[91579]: pgmap v841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:04 smithi188 ceph-mon[92425]: pgmap v841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:06 smithi161 ceph-mon[91579]: pgmap v842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:06 smithi188 ceph-mon[92425]: pgmap v842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:07 smithi188 ceph-mon[92425]: pgmap v843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:07 smithi161 ceph-mon[91579]: pgmap v843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:09 smithi188 ceph-mon[92425]: pgmap v844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:09 smithi161 ceph-mon[91579]: pgmap v844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:12 smithi161 ceph-mon[91579]: pgmap v845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:12 smithi188 ceph-mon[92425]: pgmap v845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:14 smithi161 ceph-mon[91579]: pgmap v846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:14 smithi188 ceph-mon[92425]: pgmap v846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:16 smithi161 ceph-mon[91579]: pgmap v847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:16 smithi188 ceph-mon[92425]: pgmap v847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:17 smithi188 ceph-mon[92425]: pgmap v848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:17 smithi161 ceph-mon[91579]: pgmap v848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:19 smithi188 ceph-mon[92425]: pgmap v849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:19 smithi161 ceph-mon[91579]: pgmap v849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:22 smithi161 ceph-mon[91579]: pgmap v850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:22 smithi188 ceph-mon[92425]: pgmap v850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:24 smithi161 ceph-mon[91579]: pgmap v851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:24 smithi188 ceph-mon[92425]: pgmap v851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:26 smithi161 ceph-mon[91579]: pgmap v852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:26 smithi188 ceph-mon[92425]: pgmap v852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:27 smithi188 ceph-mon[92425]: pgmap v853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:27 smithi161 ceph-mon[91579]: pgmap v853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:29.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:29 smithi188 ceph-mon[92425]: pgmap v854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:29 smithi161 ceph-mon[91579]: pgmap v854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:07:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:07:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:07:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:07:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:07:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:07:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:31 smithi188 ceph-mon[92425]: pgmap v855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:31 smithi161 ceph-mon[91579]: pgmap v855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:34 smithi161 ceph-mon[91579]: pgmap v856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:34 smithi188 ceph-mon[92425]: pgmap v856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:36 smithi161 ceph-mon[91579]: pgmap v857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:36 smithi188 ceph-mon[92425]: pgmap v857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:37 smithi188 ceph-mon[92425]: pgmap v858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:37 smithi161 ceph-mon[91579]: pgmap v858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:39 smithi188 ceph-mon[92425]: pgmap v859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:39 smithi161 ceph-mon[91579]: pgmap v859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:42 smithi161 ceph-mon[91579]: pgmap v860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:42 smithi188 ceph-mon[92425]: pgmap v860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:44 smithi161 ceph-mon[91579]: pgmap v861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:44 smithi188 ceph-mon[92425]: pgmap v861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:46 smithi161 ceph-mon[91579]: pgmap v862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:46 smithi188 ceph-mon[92425]: pgmap v862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:47 smithi188 ceph-mon[92425]: pgmap v863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:47 smithi161 ceph-mon[91579]: pgmap v863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:49 smithi188 ceph-mon[92425]: pgmap v864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:50.394 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:49 smithi161 ceph-mon[91579]: pgmap v864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:52 smithi161 ceph-mon[91579]: pgmap v865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:52 smithi188 ceph-mon[92425]: pgmap v865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:54 smithi161 ceph-mon[91579]: pgmap v866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:54 smithi188 ceph-mon[92425]: pgmap v866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:07:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:56 smithi161 ceph-mon[91579]: pgmap v867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:56 smithi188 ceph-mon[92425]: pgmap v867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:07:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:07:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:07:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:07:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:07:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:07:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:07:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:07:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:58 smithi161 ceph-mon[91579]: pgmap v868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:58 smithi188 ceph-mon[92425]: pgmap v868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:07:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:07:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:00 smithi161 ceph-mon[91579]: pgmap v869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:00 smithi188 ceph-mon[92425]: pgmap v869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:08:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:08:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:02 smithi161 ceph-mon[91579]: pgmap v870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:02 smithi188 ceph-mon[92425]: pgmap v870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:04 smithi161 ceph-mon[91579]: pgmap v871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:04 smithi188 ceph-mon[92425]: pgmap v871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:06 smithi161 ceph-mon[91579]: pgmap v872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:06 smithi188 ceph-mon[92425]: pgmap v872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:07 smithi188 ceph-mon[92425]: pgmap v873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:07 smithi161 ceph-mon[91579]: pgmap v873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:09 smithi161 ceph-mon[91579]: pgmap v874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:09 smithi188 ceph-mon[92425]: pgmap v874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:12.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:12 smithi161 ceph-mon[91579]: pgmap v875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:12 smithi188 ceph-mon[92425]: pgmap v875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:14 smithi161 ceph-mon[91579]: pgmap v876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:14 smithi188 ceph-mon[92425]: pgmap v876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:16 smithi161 ceph-mon[91579]: pgmap v877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:16 smithi188 ceph-mon[92425]: pgmap v877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:17 smithi188 ceph-mon[92425]: pgmap v878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:17 smithi161 ceph-mon[91579]: pgmap v878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:19 smithi161 ceph-mon[91579]: pgmap v879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:19 smithi188 ceph-mon[92425]: pgmap v879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:22 smithi161 ceph-mon[91579]: pgmap v880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:22 smithi188 ceph-mon[92425]: pgmap v880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:24 smithi161 ceph-mon[91579]: pgmap v881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:24 smithi188 ceph-mon[92425]: pgmap v881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:26 smithi161 ceph-mon[91579]: pgmap v882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:26 smithi188 ceph-mon[92425]: pgmap v882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:27 smithi188 ceph-mon[92425]: pgmap v883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:27 smithi161 ceph-mon[91579]: pgmap v883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:29 smithi161 ceph-mon[91579]: pgmap v884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:29 smithi188 ceph-mon[92425]: pgmap v884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:08:30.383+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:08:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:08:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:08:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:08:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:08:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:08:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:08:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:08:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:08:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:08:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:08:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:08:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:08:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:08:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:08:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:08:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:08:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:08:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:08:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:08:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:08:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:08:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:08:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:08:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:08:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:08:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:08:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:08:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:08:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:08:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:08:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:08:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:08:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:08:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:08:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:08:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:08:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:08:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:08:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:08:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:08:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:08:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:08:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:08:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:08:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=39] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:08:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=39] Local address: 172.21.15.161, port 51036 2022-07-03T11:08:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=39] Peer address: 172.21.15.101, port 22 2022-07-03T11:08:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=39] Login timeout expired 2022-07-03T11:08:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=39] Aborting connection 2022-07-03T11:08:30.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:08:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:08:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:08:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:08:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:08:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:08:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:08:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:08:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:08:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:08:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:08:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:08:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:08:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:08:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:08:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:08:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:08:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:08:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:08:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:08:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:08:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:08:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:08:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:08:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:08:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:08:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:08:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:08:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:08:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:08:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:08:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:08:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:08:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:08:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:08:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:08:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:08:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:08:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:08:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:08:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:08:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:08:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:08:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:08:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:08:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:08:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=39] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:08:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=39] Local address: 172.21.15.161, port 51036 2022-07-03T11:08:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=39] Peer address: 172.21.15.101, port 22 2022-07-03T11:08:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=39] Login timeout expired 2022-07-03T11:08:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=39] Aborting connection 2022-07-03T11:08:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:08:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:08:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:08:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:08:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:08:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:08:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:08:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:08:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:08:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:08:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:08:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:08:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:08:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:08:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:08:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:08:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:08:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:08:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:08:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:08:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:08:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:08:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:08:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:08:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:08:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:08:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:08:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:08:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:08:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:08:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:08:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:08:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:08:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:08:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:08:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:08:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:08:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:08:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:08:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:08:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:08:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:08:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:08:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:08:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:08:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=39] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:08:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=39] Local address: 172.21.15.161, port 51036 2022-07-03T11:08:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=39] Peer address: 172.21.15.101, port 22 2022-07-03T11:08:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=39] Login timeout expired 2022-07-03T11:08:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=39] Aborting connection 2022-07-03T11:08:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:31 smithi161 ceph-mon[91579]: pgmap v885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:31 smithi188 ceph-mon[92425]: pgmap v885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:34 smithi161 ceph-mon[91579]: pgmap v886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:34 smithi188 ceph-mon[92425]: pgmap v886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:36 smithi161 ceph-mon[91579]: pgmap v887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:36 smithi188 ceph-mon[92425]: pgmap v887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:37 smithi188 ceph-mon[92425]: pgmap v888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:37 smithi161 ceph-mon[91579]: pgmap v888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:39 smithi161 ceph-mon[91579]: pgmap v889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:39 smithi188 ceph-mon[92425]: pgmap v889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:42 smithi161 ceph-mon[91579]: pgmap v890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:42 smithi188 ceph-mon[92425]: pgmap v890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:44 smithi161 ceph-mon[91579]: pgmap v891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:44 smithi188 ceph-mon[92425]: pgmap v891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:46 smithi161 ceph-mon[91579]: pgmap v892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:46 smithi188 ceph-mon[92425]: pgmap v892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:47 smithi161 ceph-mon[91579]: pgmap v893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:47 smithi188 ceph-mon[92425]: pgmap v893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:49 smithi161 ceph-mon[91579]: pgmap v894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:49 smithi188 ceph-mon[92425]: pgmap v894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:52 smithi161 ceph-mon[91579]: pgmap v895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:52 smithi188 ceph-mon[92425]: pgmap v895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:54 smithi161 ceph-mon[91579]: pgmap v896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:54 smithi188 ceph-mon[92425]: pgmap v896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:08:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:56 smithi161 ceph-mon[91579]: pgmap v897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:56 smithi188 ceph-mon[92425]: pgmap v897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:08:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:08:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:08:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:08:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:08:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:08:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:08:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:08:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:58 smithi161 ceph-mon[91579]: pgmap v898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:58 smithi188 ceph-mon[92425]: pgmap v898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:08:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:08:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:00 smithi161 ceph-mon[91579]: pgmap v899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:00 smithi188 ceph-mon[92425]: pgmap v899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:02 smithi161 ceph-mon[91579]: pgmap v900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:02 smithi188 ceph-mon[92425]: pgmap v900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:04 smithi161 ceph-mon[91579]: pgmap v901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:04 smithi188 ceph-mon[92425]: pgmap v901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:06 smithi161 ceph-mon[91579]: pgmap v902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:06 smithi188 ceph-mon[92425]: pgmap v902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:07 smithi161 ceph-mon[91579]: pgmap v903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:07 smithi188 ceph-mon[92425]: pgmap v903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:09 smithi161 ceph-mon[91579]: pgmap v904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:09 smithi188 ceph-mon[92425]: pgmap v904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:12 smithi161 ceph-mon[91579]: pgmap v905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:12 smithi188 ceph-mon[92425]: pgmap v905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:14 smithi161 ceph-mon[91579]: pgmap v906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:14 smithi188 ceph-mon[92425]: pgmap v906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:16 smithi161 ceph-mon[91579]: pgmap v907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:16 smithi188 ceph-mon[92425]: pgmap v907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:17 smithi161 ceph-mon[91579]: pgmap v908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:17 smithi188 ceph-mon[92425]: pgmap v908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:19 smithi161 ceph-mon[91579]: pgmap v909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:19 smithi188 ceph-mon[92425]: pgmap v909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:22.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:22 smithi161 ceph-mon[91579]: pgmap v910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:22 smithi188 ceph-mon[92425]: pgmap v910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:24 smithi161 ceph-mon[91579]: pgmap v911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:24 smithi188 ceph-mon[92425]: pgmap v911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:26 smithi161 ceph-mon[91579]: pgmap v912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:26 smithi188 ceph-mon[92425]: pgmap v912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:27 smithi161 ceph-mon[91579]: pgmap v913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:27 smithi188 ceph-mon[92425]: pgmap v913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:29 smithi161 ceph-mon[91579]: pgmap v914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:29 smithi188 ceph-mon[92425]: pgmap v914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:09:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:09:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:09:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:09:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:09:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:09:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:32 smithi161 ceph-mon[91579]: pgmap v915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:32 smithi188 ceph-mon[92425]: pgmap v915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:09:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:34 smithi161 ceph-mon[91579]: pgmap v916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:34 smithi188 ceph-mon[92425]: pgmap v916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:36 smithi161 ceph-mon[91579]: pgmap v917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:36 smithi188 ceph-mon[92425]: pgmap v917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:37 smithi161 ceph-mon[91579]: pgmap v918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:37 smithi188 ceph-mon[92425]: pgmap v918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:39 smithi161 ceph-mon[91579]: pgmap v919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:39 smithi188 ceph-mon[92425]: pgmap v919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:42 smithi161 ceph-mon[91579]: pgmap v920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:42 smithi188 ceph-mon[92425]: pgmap v920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:44 smithi161 ceph-mon[91579]: pgmap v921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:44 smithi188 ceph-mon[92425]: pgmap v921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:46 smithi161 ceph-mon[91579]: pgmap v922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:46 smithi188 ceph-mon[92425]: pgmap v922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:47 smithi161 ceph-mon[91579]: pgmap v923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:47 smithi188 ceph-mon[92425]: pgmap v923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:49 smithi161 ceph-mon[91579]: pgmap v924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:49 smithi188 ceph-mon[92425]: pgmap v924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:52 smithi161 ceph-mon[91579]: pgmap v925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:52 smithi188 ceph-mon[92425]: pgmap v925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:54 smithi161 ceph-mon[91579]: pgmap v926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:54 smithi188 ceph-mon[92425]: pgmap v926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:09:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:56 smithi161 ceph-mon[91579]: pgmap v927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:56 smithi188 ceph-mon[92425]: pgmap v927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:09:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:09:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:09:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:09:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:09:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:09:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:09:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:09:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:58 smithi161 ceph-mon[91579]: pgmap v928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:58 smithi188 ceph-mon[92425]: pgmap v928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:09:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:09:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-mon[91579]: pgmap v929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:10:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:10:00.418+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:10:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:10:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:10:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:10:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:10:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:10:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:10:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:10:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:10:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:10:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:10:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:10:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:10:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:10:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:10:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:10:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:10:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:10:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:10:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:10:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:10:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:10:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:10:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:10:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:10:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:10:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:10:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:10:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:10:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:10:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:10:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:10:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:10:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:10:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:10:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:10:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:10:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:10:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:10:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:10:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:10:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:10:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:10:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:10:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:10:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=41] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:10:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=41] Local address: 172.21.15.161, port 51066 2022-07-03T11:10:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=41] Peer address: 172.21.15.101, port 22 2022-07-03T11:10:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=41] Connection lost 2022-07-03T11:10:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=41] Aborting connection 2022-07-03T11:10:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:10:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:10:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:00 smithi188 ceph-mon[92425]: pgmap v929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:10:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:10:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:10:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:10:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:10:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:10:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:10:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:10:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:10:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:10:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:10:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:10:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:10:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:10:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:10:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:10:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:10:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:10:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:10:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:10:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:10:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:10:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:10:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:10:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:10:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:10:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:10:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:10:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:10:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:10:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:10:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:10:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:10:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:10:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:10:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:10:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:10:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:10:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:10:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:10:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:10:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:10:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:10:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:10:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:10:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:10:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:10:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=41] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:10:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=41] Local address: 172.21.15.161, port 51066 2022-07-03T11:10:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=41] Peer address: 172.21.15.101, port 22 2022-07-03T11:10:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=41] Connection lost 2022-07-03T11:10:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=41] Aborting connection 2022-07-03T11:10:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:10:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:10:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:10:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:10:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:10:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:10:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:10:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:10:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:10:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:10:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:10:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:10:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:10:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:10:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:10:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:10:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:10:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:10:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:10:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:10:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:10:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:10:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:10:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:10:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:10:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:10:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:10:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:10:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:10:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:10:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:10:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:10:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:10:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:10:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:10:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:10:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:10:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:10:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:10:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:10:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:10:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:10:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:10:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:10:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:10:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:10:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=41] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:10:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=41] Local address: 172.21.15.161, port 51066 2022-07-03T11:10:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=41] Peer address: 172.21.15.101, port 22 2022-07-03T11:10:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=41] Connection lost 2022-07-03T11:10:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=41] Aborting connection 2022-07-03T11:10:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:02 smithi161 ceph-mon[91579]: pgmap v930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:02 smithi188 ceph-mon[92425]: pgmap v930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:04 smithi161 ceph-mon[91579]: pgmap v931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:04 smithi188 ceph-mon[92425]: pgmap v931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:06 smithi161 ceph-mon[91579]: pgmap v932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:06 smithi188 ceph-mon[92425]: pgmap v932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:07 smithi161 ceph-mon[91579]: pgmap v933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:07 smithi188 ceph-mon[92425]: pgmap v933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:09 smithi161 ceph-mon[91579]: pgmap v934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:09 smithi188 ceph-mon[92425]: pgmap v934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:12 smithi161 ceph-mon[91579]: pgmap v935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:12 smithi188 ceph-mon[92425]: pgmap v935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:14 smithi161 ceph-mon[91579]: pgmap v936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:14 smithi188 ceph-mon[92425]: pgmap v936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:16 smithi161 ceph-mon[91579]: pgmap v937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:16 smithi188 ceph-mon[92425]: pgmap v937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:17 smithi161 ceph-mon[91579]: pgmap v938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:17 smithi188 ceph-mon[92425]: pgmap v938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:19 smithi161 ceph-mon[91579]: pgmap v939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:19 smithi188 ceph-mon[92425]: pgmap v939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:22 smithi161 ceph-mon[91579]: pgmap v940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:22 smithi188 ceph-mon[92425]: pgmap v940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:24 smithi161 ceph-mon[91579]: pgmap v941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:24 smithi188 ceph-mon[92425]: pgmap v941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:26 smithi161 ceph-mon[91579]: pgmap v942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:26 smithi188 ceph-mon[92425]: pgmap v942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:27 smithi161 ceph-mon[91579]: pgmap v943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:27 smithi188 ceph-mon[92425]: pgmap v943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:29 smithi161 ceph-mon[91579]: pgmap v944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:29 smithi188 ceph-mon[92425]: pgmap v944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:32 smithi161 ceph-mon[91579]: pgmap v945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:32 smithi188 ceph-mon[92425]: pgmap v945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:34 smithi161 ceph-mon[91579]: pgmap v946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:34 smithi188 ceph-mon[92425]: pgmap v946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:36 smithi161 ceph-mon[91579]: pgmap v947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:36 smithi188 ceph-mon[92425]: pgmap v947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:37 smithi161 ceph-mon[91579]: pgmap v948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:37 smithi188 ceph-mon[92425]: pgmap v948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:39 smithi161 ceph-mon[91579]: pgmap v949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:39 smithi188 ceph-mon[92425]: pgmap v949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:42 smithi161 ceph-mon[91579]: pgmap v950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:42 smithi188 ceph-mon[92425]: pgmap v950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:44 smithi161 ceph-mon[91579]: pgmap v951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:44 smithi188 ceph-mon[92425]: pgmap v951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:46 smithi161 ceph-mon[91579]: pgmap v952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:46 smithi188 ceph-mon[92425]: pgmap v952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:47 smithi161 ceph-mon[91579]: pgmap v953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:47 smithi188 ceph-mon[92425]: pgmap v953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:49 smithi161 ceph-mon[91579]: pgmap v954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:49 smithi188 ceph-mon[92425]: pgmap v954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:52 smithi161 ceph-mon[91579]: pgmap v955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:52 smithi188 ceph-mon[92425]: pgmap v955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:10:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:54 smithi161 ceph-mon[91579]: pgmap v956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:54 smithi188 ceph-mon[92425]: pgmap v956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:10:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:56 smithi161 ceph-mon[91579]: pgmap v957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:10:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:56 smithi188 ceph-mon[92425]: pgmap v957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:10:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:10:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:10:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:10:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:10:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:10:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:10:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:58 smithi161 ceph-mon[91579]: pgmap v958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:10:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:58 smithi188 ceph-mon[92425]: pgmap v958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:10:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:10:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:00.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:00 smithi188 ceph-mon[92425]: pgmap v959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:00 smithi161 ceph-mon[91579]: pgmap v959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:11:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:11:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:11:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:11:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:11:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:11:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:02 smithi161 ceph-mon[91579]: pgmap v960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:11:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:02 smithi188 ceph-mon[92425]: pgmap v960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:11:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:04 smithi161 ceph-mon[91579]: pgmap v961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:04 smithi188 ceph-mon[92425]: pgmap v961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:06 smithi161 ceph-mon[91579]: pgmap v962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:06 smithi188 ceph-mon[92425]: pgmap v962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:07 smithi161 ceph-mon[91579]: pgmap v963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:07 smithi188 ceph-mon[92425]: pgmap v963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:09 smithi161 ceph-mon[91579]: pgmap v964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:09 smithi188 ceph-mon[92425]: pgmap v964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:12 smithi161 ceph-mon[91579]: pgmap v965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:12 smithi188 ceph-mon[92425]: pgmap v965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:14 smithi161 ceph-mon[91579]: pgmap v966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:14 smithi188 ceph-mon[92425]: pgmap v966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:16 smithi161 ceph-mon[91579]: pgmap v967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:16 smithi188 ceph-mon[92425]: pgmap v967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:17 smithi161 ceph-mon[91579]: pgmap v968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:17 smithi188 ceph-mon[92425]: pgmap v968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:19 smithi161 ceph-mon[91579]: pgmap v969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:19 smithi188 ceph-mon[92425]: pgmap v969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:22 smithi161 ceph-mon[91579]: pgmap v970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:22 smithi188 ceph-mon[92425]: pgmap v970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:24 smithi161 ceph-mon[91579]: pgmap v971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:24 smithi188 ceph-mon[92425]: pgmap v971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:26 smithi161 ceph-mon[91579]: pgmap v972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:26 smithi188 ceph-mon[92425]: pgmap v972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:27 smithi161 ceph-mon[91579]: pgmap v973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:27 smithi188 ceph-mon[92425]: pgmap v973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:29 smithi161 ceph-mon[91579]: pgmap v974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:29 smithi188 ceph-mon[92425]: pgmap v974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:11:30.453+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:11:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:11:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:11:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:11:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:11:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:11:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:11:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:11:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:11:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:11:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:11:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:11:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:11:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:11:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:11:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:11:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:11:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:11:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:11:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:11:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:11:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:11:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:11:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:11:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:11:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:11:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:11:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:11:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:11:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:11:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:11:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:11:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:11:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:11:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:11:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:11:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:11:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:11:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:11:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:11:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:11:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:11:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:11:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:11:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:11:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=43] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:11:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=43] Local address: 172.21.15.161, port 51070 2022-07-03T11:11:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=43] Peer address: 172.21.15.101, port 22 2022-07-03T11:11:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=43] Connection lost 2022-07-03T11:11:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=43] Aborting connection 2022-07-03T11:11:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:11:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:11:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:11:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:11:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:11:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:11:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:11:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:11:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:11:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:11:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:11:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:11:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:11:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:11:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:11:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:11:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:11:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:11:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:11:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:11:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:11:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:11:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:11:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:11:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:11:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:11:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:11:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:11:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:11:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:11:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:11:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:11:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:11:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:11:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:11:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:11:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:11:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:11:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:11:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:11:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:11:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:11:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:11:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:11:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:11:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:11:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:11:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=43] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:11:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=43] Local address: 172.21.15.161, port 51070 2022-07-03T11:11:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=43] Peer address: 172.21.15.101, port 22 2022-07-03T11:11:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=43] Connection lost 2022-07-03T11:11:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=43] Aborting connection 2022-07-03T11:11:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:11:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:11:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:11:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:11:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:11:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:11:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:11:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:11:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:11:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:11:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:11:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:11:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:11:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:11:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:11:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:11:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:11:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:11:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:11:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:11:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:11:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:11:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:11:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:11:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:11:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:11:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:11:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:11:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:11:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:11:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:11:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:11:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:11:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:11:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:11:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:11:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:11:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:11:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:11:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:11:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:11:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:11:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:11:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:11:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:11:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:11:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=43] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:11:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=43] Local address: 172.21.15.161, port 51070 2022-07-03T11:11:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=43] Peer address: 172.21.15.101, port 22 2022-07-03T11:11:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=43] Connection lost 2022-07-03T11:11:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=43] Aborting connection 2022-07-03T11:11:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:31 smithi161 ceph-mon[91579]: pgmap v975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:31 smithi188 ceph-mon[92425]: pgmap v975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:34 smithi161 ceph-mon[91579]: pgmap v976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:34 smithi188 ceph-mon[92425]: pgmap v976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:36 smithi161 ceph-mon[91579]: pgmap v977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:36 smithi188 ceph-mon[92425]: pgmap v977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:37 smithi161 ceph-mon[91579]: pgmap v978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:37 smithi188 ceph-mon[92425]: pgmap v978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:39 smithi161 ceph-mon[91579]: pgmap v979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:39 smithi188 ceph-mon[92425]: pgmap v979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:42 smithi161 ceph-mon[91579]: pgmap v980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:42 smithi188 ceph-mon[92425]: pgmap v980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:44 smithi161 ceph-mon[91579]: pgmap v981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:44 smithi188 ceph-mon[92425]: pgmap v981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:46 smithi161 ceph-mon[91579]: pgmap v982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:46 smithi188 ceph-mon[92425]: pgmap v982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:47 smithi161 ceph-mon[91579]: pgmap v983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:47 smithi188 ceph-mon[92425]: pgmap v983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:49 smithi161 ceph-mon[91579]: pgmap v984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:49 smithi188 ceph-mon[92425]: pgmap v984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:52 smithi161 ceph-mon[91579]: pgmap v985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:52 smithi188 ceph-mon[92425]: pgmap v985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:54 smithi161 ceph-mon[91579]: pgmap v986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:54 smithi188 ceph-mon[92425]: pgmap v986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:11:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:56 smithi161 ceph-mon[91579]: pgmap v987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:56 smithi188 ceph-mon[92425]: pgmap v987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:11:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:11:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:11:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:11:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:11:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:58 smithi161 ceph-mon[91579]: pgmap v988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:58 smithi188 ceph-mon[92425]: pgmap v988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:11:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:11:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:00 smithi161 ceph-mon[91579]: pgmap v989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:00 smithi188 ceph-mon[92425]: pgmap v989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:02 smithi161 ceph-mon[91579]: pgmap v990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:02 smithi188 ceph-mon[92425]: pgmap v990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:04 smithi161 ceph-mon[91579]: pgmap v991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:04 smithi188 ceph-mon[92425]: pgmap v991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:06 smithi161 ceph-mon[91579]: pgmap v992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:06 smithi188 ceph-mon[92425]: pgmap v992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:07 smithi161 ceph-mon[91579]: pgmap v993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:07 smithi188 ceph-mon[92425]: pgmap v993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:09 smithi161 ceph-mon[91579]: pgmap v994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:09 smithi188 ceph-mon[92425]: pgmap v994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:12 smithi161 ceph-mon[91579]: pgmap v995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:12 smithi188 ceph-mon[92425]: pgmap v995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:14 smithi161 ceph-mon[91579]: pgmap v996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:14 smithi188 ceph-mon[92425]: pgmap v996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:16 smithi161 ceph-mon[91579]: pgmap v997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:16 smithi188 ceph-mon[92425]: pgmap v997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:17 smithi161 ceph-mon[91579]: pgmap v998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:17 smithi188 ceph-mon[92425]: pgmap v998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:19 smithi161 ceph-mon[91579]: pgmap v999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:19 smithi188 ceph-mon[92425]: pgmap v999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:22 smithi161 ceph-mon[91579]: pgmap v1000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:22 smithi188 ceph-mon[92425]: pgmap v1000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:24 smithi161 ceph-mon[91579]: pgmap v1001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:24 smithi188 ceph-mon[92425]: pgmap v1001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:26 smithi161 ceph-mon[91579]: pgmap v1002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:26 smithi188 ceph-mon[92425]: pgmap v1002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:27 smithi161 ceph-mon[91579]: pgmap v1003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:27 smithi188 ceph-mon[92425]: pgmap v1003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:29 smithi161 ceph-mon[91579]: pgmap v1004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:29 smithi188 ceph-mon[92425]: pgmap v1004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:12:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:12:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:12:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:12:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:12:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:12:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:12:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:12:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:31 smithi161 ceph-mon[91579]: pgmap v1005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:31 smithi188 ceph-mon[92425]: pgmap v1005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: pgmap v1006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: pgmap v1006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:12:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:12:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:35 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:12:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:35 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:12:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:35 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:12:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:35 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:12:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:35 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:12:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:35 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:12:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:35 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:12:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:35 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:12:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:36 smithi161 ceph-mon[91579]: pgmap v1007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:36 smithi188 ceph-mon[92425]: pgmap v1007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:37 smithi161 ceph-mon[91579]: pgmap v1008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:37 smithi188 ceph-mon[92425]: pgmap v1008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:39 smithi161 ceph-mon[91579]: pgmap v1009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:39 smithi188 ceph-mon[92425]: pgmap v1009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:42 smithi161 ceph-mon[91579]: pgmap v1010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:42 smithi188 ceph-mon[92425]: pgmap v1010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:44 smithi161 ceph-mon[91579]: pgmap v1011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:44 smithi188 ceph-mon[92425]: pgmap v1011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:46 smithi161 ceph-mon[91579]: pgmap v1012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:46 smithi188 ceph-mon[92425]: pgmap v1012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:47 smithi161 ceph-mon[91579]: pgmap v1013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:47 smithi188 ceph-mon[92425]: pgmap v1013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:49 smithi161 ceph-mon[91579]: pgmap v1014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:49 smithi188 ceph-mon[92425]: pgmap v1014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:52 smithi161 ceph-mon[91579]: pgmap v1015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:52 smithi188 ceph-mon[92425]: pgmap v1015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:54 smithi161 ceph-mon[91579]: pgmap v1016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:54 smithi188 ceph-mon[92425]: pgmap v1016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:12:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:56 smithi161 ceph-mon[91579]: pgmap v1017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:56 smithi188 ceph-mon[92425]: pgmap v1017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:12:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:12:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:12:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:12:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:12:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:12:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:12:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:12:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:58 smithi161 ceph-mon[91579]: pgmap v1018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:58 smithi188 ceph-mon[92425]: pgmap v1018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:12:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:12:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-mon[91579]: pgmap v1019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:13:00.495+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:13:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:13:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:13:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:13:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:13:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:13:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:13:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:13:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:13:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:13:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:13:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:13:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:13:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:13:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:13:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:13:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:13:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:13:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:13:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:13:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:13:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:13:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:13:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:13:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:13:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:13:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:13:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:13:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:13:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:13:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:13:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:13:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:13:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:13:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:13:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:13:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:13:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:13:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:13:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:13:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:13:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:13:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:13:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:13:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:13:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=45] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:13:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=45] Local address: 172.21.15.161, port 51074 2022-07-03T11:13:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=45] Peer address: 172.21.15.101, port 22 2022-07-03T11:13:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=168] Received exit status 0 2022-07-03T11:13:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=168] Received channel close 2022-07-03T11:13:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=168] Channel closed 2022-07-03T11:13:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=168] Received exit status 0 2022-07-03T11:13:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=168] Received channel close 2022-07-03T11:13:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=168] Channel closed 2022-07-03T11:13:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=169] Requesting new SSH session 2022-07-03T11:13:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=169] Requesting new SSH session 2022-07-03T11:13:00.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=169] Command: which python3 2022-07-03T11:13:00.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=169] Command: which python3 2022-07-03T11:13:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=169] Received exit status 0 2022-07-03T11:13:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=169] Received channel close 2022-07-03T11:13:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=169] Channel closed 2022-07-03T11:13:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=169] Received exit status 0 2022-07-03T11:13:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=169] Received channel close 2022-07-03T11:13:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=170] Requesting new SSH session 2022-07-03T11:13:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=169] Channel closed 2022-07-03T11:13:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=170] Requesting new SSH session 2022-07-03T11:13:00.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=170] Command: sudo true 2022-07-03T11:13:00.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=170] Command: sudo true 2022-07-03T11:13:00.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=170] Received exit status 0 2022-07-03T11:13:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=170] Received channel close 2022-07-03T11:13:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=170] Channel closed 2022-07-03T11:13:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=171] Requesting new SSH session 2022-07-03T11:13:00.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=170] Received exit status 0 2022-07-03T11:13:00.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=170] Received channel close 2022-07-03T11:13:00.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:00.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=170] Channel closed 2022-07-03T11:13:00.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=171] Requesting new SSH session 2022-07-03T11:13:00.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:00.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=171] Received exit status 0 2022-07-03T11:13:00.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=171] Received channel close 2022-07-03T11:13:00.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=171] Channel closed 2022-07-03T11:13:00.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=171] Received exit status 0 2022-07-03T11:13:00.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=171] Received channel close 2022-07-03T11:13:00.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=171] Channel closed 2022-07-03T11:13:00.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=172] Requesting new SSH session 2022-07-03T11:13:00.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=172] Command: sudo true 2022-07-03T11:13:00.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=172] Requesting new SSH session 2022-07-03T11:13:00.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=172] Command: sudo true 2022-07-03T11:13:00.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=172] Received exit status 0 2022-07-03T11:13:00.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=172] Received channel close 2022-07-03T11:13:00.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=172] Channel closed 2022-07-03T11:13:00.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=173] Requesting new SSH session 2022-07-03T11:13:00.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=173] Command: which python3 2022-07-03T11:13:00.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=172] Received exit status 0 2022-07-03T11:13:00.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=172] Received channel close 2022-07-03T11:13:00.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=172] Channel closed 2022-07-03T11:13:00.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=173] Requesting new SSH session 2022-07-03T11:13:00.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=173] Command: which python3 2022-07-03T11:13:00.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=173] Received exit status 0 2022-07-03T11:13:00.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=173] Received channel close 2022-07-03T11:13:00.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=173] Channel closed 2022-07-03T11:13:00.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=174] Requesting new SSH session 2022-07-03T11:13:00.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=174] Command: sudo true 2022-07-03T11:13:00.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=173] Received exit status 0 2022-07-03T11:13:00.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=173] Received channel close 2022-07-03T11:13:00.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=173] Channel closed 2022-07-03T11:13:00.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=174] Requesting new SSH session 2022-07-03T11:13:00.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=174] Command: sudo true 2022-07-03T11:13:00.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=174] Received exit status 0 2022-07-03T11:13:00.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=174] Received channel close 2022-07-03T11:13:00.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=174] Channel closed 2022-07-03T11:13:00.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=175] Requesting new SSH session 2022-07-03T11:13:00.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:00.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=174] Received exit status 0 2022-07-03T11:13:00.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=174] Received channel close 2022-07-03T11:13:00.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=174] Channel closed 2022-07-03T11:13:00.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=175] Requesting new SSH session 2022-07-03T11:13:00.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:00.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=175] Received exit status 0 2022-07-03T11:13:00.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=175] Received channel close 2022-07-03T11:13:00.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=175] Channel closed 2022-07-03T11:13:00.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=176] Requesting new SSH session 2022-07-03T11:13:00.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=176] Command: sudo true 2022-07-03T11:13:00.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=176] Received exit status 0 2022-07-03T11:13:00.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=176] Received channel close 2022-07-03T11:13:00.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=176] Channel closed 2022-07-03T11:13:00.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=177] Requesting new SSH session 2022-07-03T11:13:00.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=177] Command: which python3 2022-07-03T11:13:00.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=177] Received exit status 0 2022-07-03T11:13:00.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=177] Received channel close 2022-07-03T11:13:00.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=177] Channel closed 2022-07-03T11:13:00.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=178] Requesting new SSH session 2022-07-03T11:13:00.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=178] Command: sudo true 2022-07-03T11:13:00.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=178] Received exit status 0 2022-07-03T11:13:00.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=178] Received channel close 2022-07-03T11:13:00.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=178] Channel closed 2022-07-03T11:13:00.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=179] Requesting new SSH session 2022-07-03T11:13:00.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:00.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=175] Received exit status 0 2022-07-03T11:13:00.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=175] Received channel close 2022-07-03T11:13:00.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=175] Channel closed 2022-07-03T11:13:00.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=176] Requesting new SSH session 2022-07-03T11:13:00.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=176] Command: sudo true 2022-07-03T11:13:00.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=176] Received exit status 0 2022-07-03T11:13:00.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=176] Received channel close 2022-07-03T11:13:00.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=176] Channel closed 2022-07-03T11:13:00.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=177] Requesting new SSH session 2022-07-03T11:13:00.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=177] Command: which python3 2022-07-03T11:13:00.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=177] Received exit status 0 2022-07-03T11:13:00.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=177] Received channel close 2022-07-03T11:13:00.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=177] Channel closed 2022-07-03T11:13:00.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=178] Requesting new SSH session 2022-07-03T11:13:00.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=178] Command: sudo true 2022-07-03T11:13:00.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=178] Received exit status 0 2022-07-03T11:13:00.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=178] Received channel close 2022-07-03T11:13:00.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=178] Channel closed 2022-07-03T11:13:00.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=179] Requesting new SSH session 2022-07-03T11:13:00.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:00.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=179] Received exit status 0 2022-07-03T11:13:00.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=179] Received channel close 2022-07-03T11:13:00.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=179] Channel closed 2022-07-03T11:13:00.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=179] Received exit status 0 2022-07-03T11:13:00.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=179] Received channel close 2022-07-03T11:13:00.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=179] Channel closed 2022-07-03T11:13:00.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=45] Login timeout expired 2022-07-03T11:13:00.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=45] Aborting connection 2022-07-03T11:13:00.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:13:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:13:00.697 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:00 smithi188 ceph-mon[92425]: pgmap v1019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:13:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:13:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:13:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:13:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:13:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:13:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:13:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:13:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:13:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:13:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:13:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:13:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:13:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:13:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:13:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:13:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:13:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:13:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:13:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:13:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:13:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:13:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:13:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:13:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:13:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:13:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:13:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:13:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:13:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:13:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:13:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:13:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:13:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:13:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:13:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:13:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:13:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:13:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:13:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:13:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:13:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:13:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:13:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:13:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=45] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:13:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=45] Local address: 172.21.15.161, port 51074 2022-07-03T11:13:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=45] Peer address: 172.21.15.101, port 22 2022-07-03T11:13:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=168] Received exit status 0 2022-07-03T11:13:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=168] Received channel close 2022-07-03T11:13:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=168] Channel closed 2022-07-03T11:13:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=168] Received exit status 0 2022-07-03T11:13:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=168] Received channel close 2022-07-03T11:13:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=168] Channel closed 2022-07-03T11:13:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=169] Requesting new SSH session 2022-07-03T11:13:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=169] Requesting new SSH session 2022-07-03T11:13:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=169] Command: which python3 2022-07-03T11:13:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=169] Command: which python3 2022-07-03T11:13:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=169] Received exit status 0 2022-07-03T11:13:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=169] Received channel close 2022-07-03T11:13:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=169] Channel closed 2022-07-03T11:13:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=169] Received exit status 0 2022-07-03T11:13:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=169] Received channel close 2022-07-03T11:13:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=170] Requesting new SSH session 2022-07-03T11:13:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=169] Channel closed 2022-07-03T11:13:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=170] Requesting new SSH session 2022-07-03T11:13:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=170] Command: sudo true 2022-07-03T11:13:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=170] Command: sudo true 2022-07-03T11:13:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=170] Received exit status 0 2022-07-03T11:13:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=170] Received channel close 2022-07-03T11:13:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=170] Channel closed 2022-07-03T11:13:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=171] Requesting new SSH session 2022-07-03T11:13:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=170] Received exit status 0 2022-07-03T11:13:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=170] Received channel close 2022-07-03T11:13:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=170] Channel closed 2022-07-03T11:13:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=171] Requesting new SSH session 2022-07-03T11:13:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=171] Received exit status 0 2022-07-03T11:13:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=171] Received channel close 2022-07-03T11:13:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=171] Channel closed 2022-07-03T11:13:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=171] Received exit status 0 2022-07-03T11:13:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=171] Received channel close 2022-07-03T11:13:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=171] Channel closed 2022-07-03T11:13:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=172] Requesting new SSH session 2022-07-03T11:13:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=172] Command: sudo true 2022-07-03T11:13:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=172] Requesting new SSH session 2022-07-03T11:13:01.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=172] Command: sudo true 2022-07-03T11:13:01.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=172] Received exit status 0 2022-07-03T11:13:01.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=172] Received channel close 2022-07-03T11:13:01.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=172] Channel closed 2022-07-03T11:13:01.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=173] Requesting new SSH session 2022-07-03T11:13:01.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=173] Command: which python3 2022-07-03T11:13:01.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=172] Received exit status 0 2022-07-03T11:13:01.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=172] Received channel close 2022-07-03T11:13:01.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=172] Channel closed 2022-07-03T11:13:01.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=173] Requesting new SSH session 2022-07-03T11:13:01.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=173] Command: which python3 2022-07-03T11:13:01.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=173] Received exit status 0 2022-07-03T11:13:01.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=173] Received channel close 2022-07-03T11:13:01.666 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=173] Channel closed 2022-07-03T11:13:01.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=174] Requesting new SSH session 2022-07-03T11:13:01.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=174] Command: sudo true 2022-07-03T11:13:01.668 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=173] Received exit status 0 2022-07-03T11:13:01.668 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=173] Received channel close 2022-07-03T11:13:01.669 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=173] Channel closed 2022-07-03T11:13:01.670 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=174] Requesting new SSH session 2022-07-03T11:13:01.670 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=174] Command: sudo true 2022-07-03T11:13:01.670 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=174] Received exit status 0 2022-07-03T11:13:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=174] Received channel close 2022-07-03T11:13:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=174] Channel closed 2022-07-03T11:13:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=175] Requesting new SSH session 2022-07-03T11:13:01.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:01.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=174] Received exit status 0 2022-07-03T11:13:01.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=174] Received channel close 2022-07-03T11:13:01.673 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=174] Channel closed 2022-07-03T11:13:01.673 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=175] Requesting new SSH session 2022-07-03T11:13:01.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:01.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=175] Received exit status 0 2022-07-03T11:13:01.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=175] Received channel close 2022-07-03T11:13:01.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=175] Channel closed 2022-07-03T11:13:01.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=176] Requesting new SSH session 2022-07-03T11:13:01.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=176] Command: sudo true 2022-07-03T11:13:01.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=176] Received exit status 0 2022-07-03T11:13:01.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=176] Received channel close 2022-07-03T11:13:01.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=176] Channel closed 2022-07-03T11:13:01.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=177] Requesting new SSH session 2022-07-03T11:13:01.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=177] Command: which python3 2022-07-03T11:13:01.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=177] Received exit status 0 2022-07-03T11:13:01.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=177] Received channel close 2022-07-03T11:13:01.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=177] Channel closed 2022-07-03T11:13:01.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=178] Requesting new SSH session 2022-07-03T11:13:01.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=178] Command: sudo true 2022-07-03T11:13:01.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=178] Received exit status 0 2022-07-03T11:13:01.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=178] Received channel close 2022-07-03T11:13:01.680 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=178] Channel closed 2022-07-03T11:13:01.680 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=179] Requesting new SSH session 2022-07-03T11:13:01.680 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:01.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=175] Received exit status 0 2022-07-03T11:13:01.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=175] Received channel close 2022-07-03T11:13:01.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=175] Channel closed 2022-07-03T11:13:01.682 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=176] Requesting new SSH session 2022-07-03T11:13:01.682 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=176] Command: sudo true 2022-07-03T11:13:01.682 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=176] Received exit status 0 2022-07-03T11:13:01.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=176] Received channel close 2022-07-03T11:13:01.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=176] Channel closed 2022-07-03T11:13:01.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=177] Requesting new SSH session 2022-07-03T11:13:01.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=177] Command: which python3 2022-07-03T11:13:01.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=177] Received exit status 0 2022-07-03T11:13:01.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=177] Received channel close 2022-07-03T11:13:01.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=177] Channel closed 2022-07-03T11:13:01.685 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=178] Requesting new SSH session 2022-07-03T11:13:01.685 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=178] Command: sudo true 2022-07-03T11:13:01.685 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=178] Received exit status 0 2022-07-03T11:13:01.686 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=178] Received channel close 2022-07-03T11:13:01.686 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=178] Channel closed 2022-07-03T11:13:01.686 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=179] Requesting new SSH session 2022-07-03T11:13:01.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:01.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=179] Received exit status 0 2022-07-03T11:13:01.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=179] Received channel close 2022-07-03T11:13:01.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=179] Channel closed 2022-07-03T11:13:01.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=179] Received exit status 0 2022-07-03T11:13:01.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=179] Received channel close 2022-07-03T11:13:01.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=179] Channel closed 2022-07-03T11:13:01.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=45] Login timeout expired 2022-07-03T11:13:01.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=45] Aborting connection 2022-07-03T11:13:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:13:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:13:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:13:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:13:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:13:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:13:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:13:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:13:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:13:01.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:13:01.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:13:01.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:13:01.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:13:01.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:13:01.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:13:01.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:13:01.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:13:01.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:13:01.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:13:01.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:13:01.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:13:01.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:13:01.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:13:01.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:13:01.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:13:01.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:13:01.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:13:01.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:13:01.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:13:01.701 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:13:01.701 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:13:01.701 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:13:01.702 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:13:01.702 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:13:01.702 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:13:01.703 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:13:01.703 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:13:01.703 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:13:01.704 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:13:01.704 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:13:01.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:13:01.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:13:01.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:13:01.706 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:13:01.706 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:13:01.706 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=45] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:13:01.707 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=45] Local address: 172.21.15.161, port 51074 2022-07-03T11:13:01.707 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=45] Peer address: 172.21.15.101, port 22 2022-07-03T11:13:01.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=168] Received exit status 0 2022-07-03T11:13:01.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=168] Received channel close 2022-07-03T11:13:01.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=168] Channel closed 2022-07-03T11:13:01.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=168] Received exit status 0 2022-07-03T11:13:01.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=168] Received channel close 2022-07-03T11:13:01.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=168] Channel closed 2022-07-03T11:13:01.710 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=169] Requesting new SSH session 2022-07-03T11:13:01.710 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=169] Requesting new SSH session 2022-07-03T11:13:01.710 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=169] Command: which python3 2022-07-03T11:13:01.711 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=169] Command: which python3 2022-07-03T11:13:01.711 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=169] Received exit status 0 2022-07-03T11:13:01.711 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=169] Received channel close 2022-07-03T11:13:01.712 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=169] Channel closed 2022-07-03T11:13:01.712 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=169] Received exit status 0 2022-07-03T11:13:01.712 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=169] Received channel close 2022-07-03T11:13:01.713 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=170] Requesting new SSH session 2022-07-03T11:13:01.713 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=169] Channel closed 2022-07-03T11:13:01.713 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=170] Requesting new SSH session 2022-07-03T11:13:01.713 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=170] Command: sudo true 2022-07-03T11:13:01.714 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=170] Command: sudo true 2022-07-03T11:13:01.714 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=170] Received exit status 0 2022-07-03T11:13:01.715 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=170] Received channel close 2022-07-03T11:13:01.715 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=170] Channel closed 2022-07-03T11:13:01.715 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=171] Requesting new SSH session 2022-07-03T11:13:01.716 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=170] Received exit status 0 2022-07-03T11:13:01.716 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=170] Received channel close 2022-07-03T11:13:01.717 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:01.717 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=170] Channel closed 2022-07-03T11:13:01.718 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=171] Requesting new SSH session 2022-07-03T11:13:01.718 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=171] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T11:13:01.719 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=171] Received exit status 0 2022-07-03T11:13:01.719 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=171] Received channel close 2022-07-03T11:13:01.719 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=171] Channel closed 2022-07-03T11:13:01.720 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=171] Received exit status 0 2022-07-03T11:13:01.720 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=171] Received channel close 2022-07-03T11:13:01.720 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=171] Channel closed 2022-07-03T11:13:01.721 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=172] Requesting new SSH session 2022-07-03T11:13:01.721 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=172] Command: sudo true 2022-07-03T11:13:01.721 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=172] Requesting new SSH session 2022-07-03T11:13:01.722 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=172] Command: sudo true 2022-07-03T11:13:01.722 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=172] Received exit status 0 2022-07-03T11:13:01.722 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=172] Received channel close 2022-07-03T11:13:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=172] Channel closed 2022-07-03T11:13:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=173] Requesting new SSH session 2022-07-03T11:13:01.723 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=173] Command: which python3 2022-07-03T11:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=172] Received exit status 0 2022-07-03T11:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=172] Received channel close 2022-07-03T11:13:01.724 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=172] Channel closed 2022-07-03T11:13:01.725 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=173] Requesting new SSH session 2022-07-03T11:13:01.725 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=173] Command: which python3 2022-07-03T11:13:01.725 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=173] Received exit status 0 2022-07-03T11:13:01.725 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=173] Received channel close 2022-07-03T11:13:01.726 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=173] Channel closed 2022-07-03T11:13:01.726 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=174] Requesting new SSH session 2022-07-03T11:13:01.726 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=174] Command: sudo true 2022-07-03T11:13:01.727 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=173] Received exit status 0 2022-07-03T11:13:01.727 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=173] Received channel close 2022-07-03T11:13:01.727 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=173] Channel closed 2022-07-03T11:13:01.728 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=174] Requesting new SSH session 2022-07-03T11:13:01.728 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=174] Command: sudo true 2022-07-03T11:13:01.728 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=174] Received exit status 0 2022-07-03T11:13:01.729 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=174] Received channel close 2022-07-03T11:13:01.729 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=174] Channel closed 2022-07-03T11:13:01.730 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=175] Requesting new SSH session 2022-07-03T11:13:01.730 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:01.730 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=174] Received exit status 0 2022-07-03T11:13:01.731 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=174] Received channel close 2022-07-03T11:13:01.731 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=174] Channel closed 2022-07-03T11:13:01.731 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=175] Requesting new SSH session 2022-07-03T11:13:01.731 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=175] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:13:01.732 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=175] Received exit status 0 2022-07-03T11:13:01.732 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=175] Received channel close 2022-07-03T11:13:01.732 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=175] Channel closed 2022-07-03T11:13:01.733 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=176] Requesting new SSH session 2022-07-03T11:13:01.733 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=176] Command: sudo true 2022-07-03T11:13:01.733 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=176] Received exit status 0 2022-07-03T11:13:01.734 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=176] Received channel close 2022-07-03T11:13:01.734 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=176] Channel closed 2022-07-03T11:13:01.734 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=177] Requesting new SSH session 2022-07-03T11:13:01.735 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=177] Command: which python3 2022-07-03T11:13:01.735 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=177] Received exit status 0 2022-07-03T11:13:01.736 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=177] Received channel close 2022-07-03T11:13:01.736 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=177] Channel closed 2022-07-03T11:13:01.736 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=178] Requesting new SSH session 2022-07-03T11:13:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=178] Command: sudo true 2022-07-03T11:13:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=178] Received exit status 0 2022-07-03T11:13:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=178] Received channel close 2022-07-03T11:13:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=178] Channel closed 2022-07-03T11:13:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=179] Requesting new SSH session 2022-07-03T11:13:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:01.739 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=175] Received exit status 0 2022-07-03T11:13:01.739 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=175] Received channel close 2022-07-03T11:13:01.739 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=175] Channel closed 2022-07-03T11:13:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=176] Requesting new SSH session 2022-07-03T11:13:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=176] Command: sudo true 2022-07-03T11:13:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=176] Received exit status 0 2022-07-03T11:13:01.741 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=176] Received channel close 2022-07-03T11:13:01.741 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=176] Channel closed 2022-07-03T11:13:01.741 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=177] Requesting new SSH session 2022-07-03T11:13:01.742 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=177] Command: which python3 2022-07-03T11:13:01.742 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=177] Received exit status 0 2022-07-03T11:13:01.743 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=177] Received channel close 2022-07-03T11:13:01.743 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=177] Channel closed 2022-07-03T11:13:01.743 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=178] Requesting new SSH session 2022-07-03T11:13:01.743 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=178] Command: sudo true 2022-07-03T11:13:01.744 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=178] Received exit status 0 2022-07-03T11:13:01.744 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=178] Received channel close 2022-07-03T11:13:01.745 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=178] Channel closed 2022-07-03T11:13:01.745 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=179] Requesting new SSH session 2022-07-03T11:13:01.745 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=179] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:13:01.746 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=179] Received exit status 0 2022-07-03T11:13:01.746 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=179] Received channel close 2022-07-03T11:13:01.746 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=179] Channel closed 2022-07-03T11:13:01.747 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=179] Received exit status 0 2022-07-03T11:13:01.747 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=179] Received channel close 2022-07-03T11:13:01.747 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=179] Channel closed 2022-07-03T11:13:01.748 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=45] Login timeout expired 2022-07-03T11:13:01.748 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=45] Aborting connection 2022-07-03T11:13:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:02 smithi161 ceph-mon[91579]: pgmap v1020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:02 smithi188 ceph-mon[92425]: pgmap v1020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:04 smithi161 ceph-mon[91579]: pgmap v1021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:04 smithi188 ceph-mon[92425]: pgmap v1021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:06 smithi161 ceph-mon[91579]: pgmap v1022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:06 smithi188 ceph-mon[92425]: pgmap v1022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:07 smithi161 ceph-mon[91579]: pgmap v1023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:07 smithi188 ceph-mon[92425]: pgmap v1023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:09 smithi161 ceph-mon[91579]: pgmap v1024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:09 smithi188 ceph-mon[92425]: pgmap v1024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:12 smithi161 ceph-mon[91579]: pgmap v1025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:12 smithi188 ceph-mon[92425]: pgmap v1025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:14 smithi161 ceph-mon[91579]: pgmap v1026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:14 smithi188 ceph-mon[92425]: pgmap v1026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:16 smithi161 ceph-mon[91579]: pgmap v1027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:16 smithi188 ceph-mon[92425]: pgmap v1027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:17 smithi161 ceph-mon[91579]: pgmap v1028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:17 smithi188 ceph-mon[92425]: pgmap v1028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:19 smithi161 ceph-mon[91579]: pgmap v1029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:19 smithi188 ceph-mon[92425]: pgmap v1029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:22 smithi161 ceph-mon[91579]: pgmap v1030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:22 smithi188 ceph-mon[92425]: pgmap v1030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:24 smithi161 ceph-mon[91579]: pgmap v1031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:24 smithi188 ceph-mon[92425]: pgmap v1031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:26 smithi161 ceph-mon[91579]: pgmap v1032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:26 smithi188 ceph-mon[92425]: pgmap v1032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:27 smithi161 ceph-mon[91579]: pgmap v1033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:27 smithi188 ceph-mon[92425]: pgmap v1033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:29 smithi161 ceph-mon[91579]: pgmap v1034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:29 smithi188 ceph-mon[92425]: pgmap v1034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:32 smithi161 ceph-mon[91579]: pgmap v1035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:32 smithi188 ceph-mon[92425]: pgmap v1035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:34 smithi161 ceph-mon[91579]: pgmap v1036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:34 smithi188 ceph-mon[92425]: pgmap v1036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:36 smithi161 ceph-mon[91579]: pgmap v1037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:36 smithi188 ceph-mon[92425]: pgmap v1037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:37 smithi161 ceph-mon[91579]: pgmap v1038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:37 smithi188 ceph-mon[92425]: pgmap v1038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:39 smithi161 ceph-mon[91579]: pgmap v1039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:39 smithi188 ceph-mon[92425]: pgmap v1039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:42 smithi161 ceph-mon[91579]: pgmap v1040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:42 smithi188 ceph-mon[92425]: pgmap v1040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:44 smithi161 ceph-mon[91579]: pgmap v1041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:44 smithi188 ceph-mon[92425]: pgmap v1041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:46 smithi161 ceph-mon[91579]: pgmap v1042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:46 smithi188 ceph-mon[92425]: pgmap v1042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:47 smithi161 ceph-mon[91579]: pgmap v1043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:47 smithi188 ceph-mon[92425]: pgmap v1043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:49 smithi161 ceph-mon[91579]: pgmap v1044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:49 smithi188 ceph-mon[92425]: pgmap v1044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:52 smithi161 ceph-mon[91579]: pgmap v1045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:52 smithi188 ceph-mon[92425]: pgmap v1045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:54 smithi161 ceph-mon[91579]: pgmap v1046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:54 smithi188 ceph-mon[92425]: pgmap v1046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:13:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:56 smithi161 ceph-mon[91579]: pgmap v1047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:56 smithi188 ceph-mon[92425]: pgmap v1047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:13:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:13:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:13:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:13:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:13:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:13:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:13:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:13:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:58 smithi161 ceph-mon[91579]: pgmap v1048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:58 smithi188 ceph-mon[92425]: pgmap v1048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:13:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:13:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:00.588 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:00 smithi161 ceph-mon[91579]: pgmap v1049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:00.592 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:00 smithi188 ceph-mon[92425]: pgmap v1049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:14:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:14:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:14:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:14:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:14:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:02 smithi161 ceph-mon[91579]: pgmap v1050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:02 smithi188 ceph-mon[92425]: pgmap v1050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:04 smithi161 ceph-mon[91579]: pgmap v1051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:04 smithi188 ceph-mon[92425]: pgmap v1051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:06 smithi161 ceph-mon[91579]: pgmap v1052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:06 smithi188 ceph-mon[92425]: pgmap v1052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:07 smithi161 ceph-mon[91579]: pgmap v1053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:07 smithi188 ceph-mon[92425]: pgmap v1053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:09 smithi161 ceph-mon[91579]: pgmap v1054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:09 smithi188 ceph-mon[92425]: pgmap v1054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:12 smithi161 ceph-mon[91579]: pgmap v1055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:12 smithi188 ceph-mon[92425]: pgmap v1055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:14 smithi161 ceph-mon[91579]: pgmap v1056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:14 smithi188 ceph-mon[92425]: pgmap v1056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:16 smithi161 ceph-mon[91579]: pgmap v1057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:16 smithi188 ceph-mon[92425]: pgmap v1057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:17 smithi161 ceph-mon[91579]: pgmap v1058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:17 smithi188 ceph-mon[92425]: pgmap v1058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:19 smithi161 ceph-mon[91579]: pgmap v1059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:19 smithi188 ceph-mon[92425]: pgmap v1059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:22 smithi161 ceph-mon[91579]: pgmap v1060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:22 smithi188 ceph-mon[92425]: pgmap v1060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:24 smithi161 ceph-mon[91579]: pgmap v1061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:24 smithi188 ceph-mon[92425]: pgmap v1061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:26 smithi161 ceph-mon[91579]: pgmap v1062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:26 smithi188 ceph-mon[92425]: pgmap v1062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:27 smithi161 ceph-mon[91579]: pgmap v1063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:27 smithi188 ceph-mon[92425]: pgmap v1063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:29 smithi161 ceph-mon[91579]: pgmap v1064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:29 smithi188 ceph-mon[92425]: pgmap v1064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:14:30.534+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:14:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:14:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:14:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:14:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:14:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:14:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:14:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:14:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:14:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:14:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:14:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:14:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:14:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:14:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:14:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:14:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:14:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:14:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:14:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:14:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:14:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:14:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:14:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:14:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:14:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:14:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:14:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:14:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:14:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:14:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:14:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:14:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:14:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:14:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:14:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:14:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:14:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:14:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:14:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:14:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:14:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:14:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:14:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:14:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:14:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=47] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:14:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=47] Local address: 172.21.15.161, port 51078 2022-07-03T11:14:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=47] Peer address: 172.21.15.101, port 22 2022-07-03T11:14:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=180] Received exit status 0 2022-07-03T11:14:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=180] Received channel close 2022-07-03T11:14:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=180] Channel closed 2022-07-03T11:14:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=181] Requesting new SSH session 2022-07-03T11:14:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=180] Received exit status 0 2022-07-03T11:14:30.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=180] Received channel close 2022-07-03T11:14:30.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=181] Command: which python3 2022-07-03T11:14:30.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=180] Channel closed 2022-07-03T11:14:30.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=181] Requesting new SSH session 2022-07-03T11:14:30.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=181] Command: which python3 2022-07-03T11:14:30.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=181] Received exit status 0 2022-07-03T11:14:30.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=181] Received channel close 2022-07-03T11:14:30.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=181] Channel closed 2022-07-03T11:14:30.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=182] Requesting new SSH session 2022-07-03T11:14:30.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=182] Command: sudo true 2022-07-03T11:14:30.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=181] Received exit status 0 2022-07-03T11:14:30.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=181] Received channel close 2022-07-03T11:14:30.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=181] Channel closed 2022-07-03T11:14:30.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=182] Requesting new SSH session 2022-07-03T11:14:30.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=182] Command: sudo true 2022-07-03T11:14:30.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=182] Received exit status 0 2022-07-03T11:14:30.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=182] Received channel close 2022-07-03T11:14:30.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=182] Channel closed 2022-07-03T11:14:30.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=183] Requesting new SSH session 2022-07-03T11:14:30.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:30.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=182] Received exit status 0 2022-07-03T11:14:30.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=182] Received channel close 2022-07-03T11:14:30.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=182] Channel closed 2022-07-03T11:14:30.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=183] Requesting new SSH session 2022-07-03T11:14:30.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:30.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=183] Received exit status 0 2022-07-03T11:14:30.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=183] Received channel close 2022-07-03T11:14:30.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=183] Channel closed 2022-07-03T11:14:30.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=183] Received exit status 0 2022-07-03T11:14:30.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=183] Received channel close 2022-07-03T11:14:30.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=183] Channel closed 2022-07-03T11:14:30.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=47] Login timeout expired 2022-07-03T11:14:30.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=47] Aborting connection 2022-07-03T11:14:30.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:14:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:14:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:14:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:14:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:14:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:14:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:14:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:14:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:14:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:14:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:14:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:14:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:14:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:14:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:14:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:14:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:14:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:14:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:14:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:14:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:14:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:14:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:14:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:14:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:14:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:14:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:14:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:14:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:14:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:14:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:14:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:14:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:14:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:14:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:14:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:14:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:14:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:14:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:14:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:14:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:14:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:14:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:14:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:14:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:14:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:14:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:14:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=47] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:14:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=47] Local address: 172.21.15.161, port 51078 2022-07-03T11:14:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=47] Peer address: 172.21.15.101, port 22 2022-07-03T11:14:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=180] Received exit status 0 2022-07-03T11:14:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=180] Received channel close 2022-07-03T11:14:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=180] Channel closed 2022-07-03T11:14:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=181] Requesting new SSH session 2022-07-03T11:14:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=180] Received exit status 0 2022-07-03T11:14:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=180] Received channel close 2022-07-03T11:14:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=181] Command: which python3 2022-07-03T11:14:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=180] Channel closed 2022-07-03T11:14:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=181] Requesting new SSH session 2022-07-03T11:14:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=181] Command: which python3 2022-07-03T11:14:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=181] Received exit status 0 2022-07-03T11:14:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=181] Received channel close 2022-07-03T11:14:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=181] Channel closed 2022-07-03T11:14:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=182] Requesting new SSH session 2022-07-03T11:14:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=182] Command: sudo true 2022-07-03T11:14:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=181] Received exit status 0 2022-07-03T11:14:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=181] Received channel close 2022-07-03T11:14:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=181] Channel closed 2022-07-03T11:14:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=182] Requesting new SSH session 2022-07-03T11:14:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=182] Command: sudo true 2022-07-03T11:14:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=182] Received exit status 0 2022-07-03T11:14:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=182] Received channel close 2022-07-03T11:14:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=182] Channel closed 2022-07-03T11:14:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=183] Requesting new SSH session 2022-07-03T11:14:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=182] Received exit status 0 2022-07-03T11:14:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=182] Received channel close 2022-07-03T11:14:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=182] Channel closed 2022-07-03T11:14:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=183] Requesting new SSH session 2022-07-03T11:14:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=183] Received exit status 0 2022-07-03T11:14:31.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=183] Received channel close 2022-07-03T11:14:31.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=183] Channel closed 2022-07-03T11:14:31.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=183] Received exit status 0 2022-07-03T11:14:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=183] Received channel close 2022-07-03T11:14:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=183] Channel closed 2022-07-03T11:14:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=47] Login timeout expired 2022-07-03T11:14:31.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=47] Aborting connection 2022-07-03T11:14:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:14:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:14:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:14:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:14:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:14:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:14:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:14:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:14:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:14:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:14:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:14:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:14:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:14:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:14:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:14:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:14:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:14:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:14:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:14:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:14:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:14:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:14:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:14:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:14:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:14:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:14:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:14:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:14:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:14:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:14:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:14:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:14:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:14:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:14:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:14:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:14:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:14:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:14:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:14:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:14:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:14:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:14:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:14:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:14:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:14:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=47] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:14:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=47] Local address: 172.21.15.161, port 51078 2022-07-03T11:14:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=47] Peer address: 172.21.15.101, port 22 2022-07-03T11:14:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=180] Received exit status 0 2022-07-03T11:14:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=180] Received channel close 2022-07-03T11:14:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=180] Channel closed 2022-07-03T11:14:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=181] Requesting new SSH session 2022-07-03T11:14:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=180] Received exit status 0 2022-07-03T11:14:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=180] Received channel close 2022-07-03T11:14:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=181] Command: which python3 2022-07-03T11:14:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=180] Channel closed 2022-07-03T11:14:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=181] Requesting new SSH session 2022-07-03T11:14:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=181] Command: which python3 2022-07-03T11:14:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=181] Received exit status 0 2022-07-03T11:14:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=181] Received channel close 2022-07-03T11:14:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=181] Channel closed 2022-07-03T11:14:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=182] Requesting new SSH session 2022-07-03T11:14:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=182] Command: sudo true 2022-07-03T11:14:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=181] Received exit status 0 2022-07-03T11:14:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=181] Received channel close 2022-07-03T11:14:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=181] Channel closed 2022-07-03T11:14:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=182] Requesting new SSH session 2022-07-03T11:14:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=182] Command: sudo true 2022-07-03T11:14:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=182] Received exit status 0 2022-07-03T11:14:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=182] Received channel close 2022-07-03T11:14:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=182] Channel closed 2022-07-03T11:14:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=183] Requesting new SSH session 2022-07-03T11:14:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=182] Received exit status 0 2022-07-03T11:14:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=182] Received channel close 2022-07-03T11:14:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=182] Channel closed 2022-07-03T11:14:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=183] Requesting new SSH session 2022-07-03T11:14:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=183] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:14:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=183] Received exit status 0 2022-07-03T11:14:31.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=183] Received channel close 2022-07-03T11:14:31.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=183] Channel closed 2022-07-03T11:14:31.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=183] Received exit status 0 2022-07-03T11:14:31.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=183] Received channel close 2022-07-03T11:14:31.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=183] Channel closed 2022-07-03T11:14:31.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=47] Login timeout expired 2022-07-03T11:14:31.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=47] Aborting connection 2022-07-03T11:14:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:32 smithi161 ceph-mon[91579]: pgmap v1065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:32 smithi188 ceph-mon[92425]: pgmap v1065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:34 smithi161 ceph-mon[91579]: pgmap v1066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:34 smithi188 ceph-mon[92425]: pgmap v1066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:36 smithi161 ceph-mon[91579]: pgmap v1067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:36 smithi188 ceph-mon[92425]: pgmap v1067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:37 smithi161 ceph-mon[91579]: pgmap v1068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:37 smithi188 ceph-mon[92425]: pgmap v1068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:40 smithi161 ceph-mon[91579]: pgmap v1069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:40 smithi188 ceph-mon[92425]: pgmap v1069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:42 smithi161 ceph-mon[91579]: pgmap v1070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:42 smithi188 ceph-mon[92425]: pgmap v1070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:44 smithi161 ceph-mon[91579]: pgmap v1071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:44 smithi188 ceph-mon[92425]: pgmap v1071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:46 smithi161 ceph-mon[91579]: pgmap v1072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:46 smithi188 ceph-mon[92425]: pgmap v1072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:47 smithi161 ceph-mon[91579]: pgmap v1073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:47 smithi188 ceph-mon[92425]: pgmap v1073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:50 smithi161 ceph-mon[91579]: pgmap v1074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:50 smithi188 ceph-mon[92425]: pgmap v1074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:52 smithi161 ceph-mon[91579]: pgmap v1075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:52 smithi188 ceph-mon[92425]: pgmap v1075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:54 smithi161 ceph-mon[91579]: pgmap v1076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:54 smithi188 ceph-mon[92425]: pgmap v1076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:14:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:56 smithi161 ceph-mon[91579]: pgmap v1077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:56 smithi188 ceph-mon[92425]: pgmap v1077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:14:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:14:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:14:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:14:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:14:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:14:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:14:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:14:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:58 smithi161 ceph-mon[91579]: pgmap v1078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:58 smithi188 ceph-mon[92425]: pgmap v1078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:14:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:14:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:00 smithi161 ceph-mon[91579]: pgmap v1079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:00 smithi188 ceph-mon[92425]: pgmap v1079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:02 smithi161 ceph-mon[91579]: pgmap v1080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:02 smithi188 ceph-mon[92425]: pgmap v1080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:04 smithi161 ceph-mon[91579]: pgmap v1081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:04 smithi188 ceph-mon[92425]: pgmap v1081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:06 smithi161 ceph-mon[91579]: pgmap v1082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:06 smithi188 ceph-mon[92425]: pgmap v1082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:08 smithi161 ceph-mon[91579]: pgmap v1083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:08 smithi188 ceph-mon[92425]: pgmap v1083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:10 smithi161 ceph-mon[91579]: pgmap v1084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:10 smithi188 ceph-mon[92425]: pgmap v1084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:12 smithi161 ceph-mon[91579]: pgmap v1085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:12 smithi188 ceph-mon[92425]: pgmap v1085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:14 smithi161 ceph-mon[91579]: pgmap v1086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:14 smithi188 ceph-mon[92425]: pgmap v1086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:16 smithi161 ceph-mon[91579]: pgmap v1087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:16 smithi188 ceph-mon[92425]: pgmap v1087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:18 smithi161 ceph-mon[91579]: pgmap v1088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:18 smithi188 ceph-mon[92425]: pgmap v1088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:20 smithi161 ceph-mon[91579]: pgmap v1089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:20 smithi188 ceph-mon[92425]: pgmap v1089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:22 smithi161 ceph-mon[91579]: pgmap v1090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:22 smithi188 ceph-mon[92425]: pgmap v1090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:24 smithi161 ceph-mon[91579]: pgmap v1091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:24 smithi188 ceph-mon[92425]: pgmap v1091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:26 smithi161 ceph-mon[91579]: pgmap v1092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:26 smithi188 ceph-mon[92425]: pgmap v1092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:28 smithi161 ceph-mon[91579]: pgmap v1093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:28 smithi188 ceph-mon[92425]: pgmap v1093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:30 smithi161 ceph-mon[91579]: pgmap v1094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:30 smithi188 ceph-mon[92425]: pgmap v1094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:15:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:15:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:15:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:15:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:15:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:15:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:15:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:15:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:32 smithi161 ceph-mon[91579]: pgmap v1095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:32 smithi188 ceph-mon[92425]: pgmap v1095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:34 smithi161 ceph-mon[91579]: pgmap v1096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:34 smithi188 ceph-mon[92425]: pgmap v1096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:36 smithi161 ceph-mon[91579]: pgmap v1097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:36 smithi188 ceph-mon[92425]: pgmap v1097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:38 smithi161 ceph-mon[91579]: pgmap v1098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:38 smithi188 ceph-mon[92425]: pgmap v1098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:40 smithi161 ceph-mon[91579]: pgmap v1099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:40 smithi188 ceph-mon[92425]: pgmap v1099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:42 smithi161 ceph-mon[91579]: pgmap v1100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:42 smithi188 ceph-mon[92425]: pgmap v1100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:44 smithi161 ceph-mon[91579]: pgmap v1101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:44 smithi188 ceph-mon[92425]: pgmap v1101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:46 smithi161 ceph-mon[91579]: pgmap v1102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:46 smithi188 ceph-mon[92425]: pgmap v1102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:48.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:48 smithi161 ceph-mon[91579]: pgmap v1103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:48 smithi188 ceph-mon[92425]: pgmap v1103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:50 smithi161 ceph-mon[91579]: pgmap v1104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:50 smithi188 ceph-mon[92425]: pgmap v1104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:52 smithi161 ceph-mon[91579]: pgmap v1105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:52 smithi188 ceph-mon[92425]: pgmap v1105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:54 smithi161 ceph-mon[91579]: pgmap v1106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:54 smithi188 ceph-mon[92425]: pgmap v1106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:15:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:56 smithi161 ceph-mon[91579]: pgmap v1107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:56 smithi188 ceph-mon[92425]: pgmap v1107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:15:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:15:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:15:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:15:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:15:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:15:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:15:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:15:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:58 smithi161 ceph-mon[91579]: pgmap v1108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:58 smithi188 ceph-mon[92425]: pgmap v1108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:15:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:15:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-mon[91579]: pgmap v1109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:00.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:16:00.574+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:16:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:16:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:16:00.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:16:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:16:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:16:00.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:16:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:16:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:16:00.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:16:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:16:00.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:16:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:16:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:16:00.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:16:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:16:00.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:16:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:16:00.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:16:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:16:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:16:00.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:16:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:16:00.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:16:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:16:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:16:00.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:16:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:16:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:16:00.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:16:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:16:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:16:00.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:16:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:16:00.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:16:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:16:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:16:00.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:16:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:16:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:16:00.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:16:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:16:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:16:00.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:16:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:16:00.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=49] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:16:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=49] Local address: 172.21.15.161, port 51082 2022-07-03T11:16:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=49] Peer address: 172.21.15.101, port 22 2022-07-03T11:16:00.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=184] Received exit status 0 2022-07-03T11:16:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=184] Received channel close 2022-07-03T11:16:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=184] Channel closed 2022-07-03T11:16:00.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=184] Received exit status 0 2022-07-03T11:16:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=184] Received channel close 2022-07-03T11:16:00.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=184] Channel closed 2022-07-03T11:16:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=185] Requesting new SSH session 2022-07-03T11:16:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=185] Requesting new SSH session 2022-07-03T11:16:00.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=185] Command: which python3 2022-07-03T11:16:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=185] Command: which python3 2022-07-03T11:16:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=185] Received exit status 0 2022-07-03T11:16:00.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=185] Received channel close 2022-07-03T11:16:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=185] Channel closed 2022-07-03T11:16:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=186] Requesting new SSH session 2022-07-03T11:16:00.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=185] Received exit status 0 2022-07-03T11:16:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=185] Received channel close 2022-07-03T11:16:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=185] Channel closed 2022-07-03T11:16:00.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=186] Requesting new SSH session 2022-07-03T11:16:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=186] Command: sudo true 2022-07-03T11:16:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=186] Command: sudo true 2022-07-03T11:16:00.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=186] Received exit status 0 2022-07-03T11:16:00.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=186] Received channel close 2022-07-03T11:16:00.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=186] Channel closed 2022-07-03T11:16:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=186] Received exit status 0 2022-07-03T11:16:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=186] Received channel close 2022-07-03T11:16:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=186] Channel closed 2022-07-03T11:16:00.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=187] Requesting new SSH session 2022-07-03T11:16:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=187] Requesting new SSH session 2022-07-03T11:16:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:00.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=187] Received exit status 0 2022-07-03T11:16:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=187] Received channel close 2022-07-03T11:16:00.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=187] Channel closed 2022-07-03T11:16:00.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=187] Received exit status 0 2022-07-03T11:16:00.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=187] Received channel close 2022-07-03T11:16:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=187] Channel closed 2022-07-03T11:16:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=49] Login timeout expired 2022-07-03T11:16:00.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=49] Aborting connection 2022-07-03T11:16:00.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:16:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:00 smithi188 ceph-mon[92425]: pgmap v1109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:16:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:16:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:16:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:16:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:16:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:16:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:16:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:16:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:16:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:16:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:16:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:16:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:16:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:16:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:16:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:16:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:16:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:16:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:16:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:16:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:16:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:16:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:16:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:16:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:16:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:16:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:16:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:16:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:16:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:16:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:16:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:16:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:16:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:16:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:16:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:16:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:16:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:16:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:16:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:16:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:16:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:16:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:16:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:16:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:16:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=49] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:16:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=49] Local address: 172.21.15.161, port 51082 2022-07-03T11:16:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=49] Peer address: 172.21.15.101, port 22 2022-07-03T11:16:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=184] Received exit status 0 2022-07-03T11:16:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=184] Received channel close 2022-07-03T11:16:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=184] Channel closed 2022-07-03T11:16:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=184] Received exit status 0 2022-07-03T11:16:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=184] Received channel close 2022-07-03T11:16:01.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=184] Channel closed 2022-07-03T11:16:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=185] Requesting new SSH session 2022-07-03T11:16:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=185] Requesting new SSH session 2022-07-03T11:16:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=185] Command: which python3 2022-07-03T11:16:01.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=185] Command: which python3 2022-07-03T11:16:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=185] Received exit status 0 2022-07-03T11:16:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=185] Received channel close 2022-07-03T11:16:01.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=185] Channel closed 2022-07-03T11:16:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=186] Requesting new SSH session 2022-07-03T11:16:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=185] Received exit status 0 2022-07-03T11:16:01.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=185] Received channel close 2022-07-03T11:16:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=185] Channel closed 2022-07-03T11:16:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=186] Requesting new SSH session 2022-07-03T11:16:01.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=186] Command: sudo true 2022-07-03T11:16:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=186] Command: sudo true 2022-07-03T11:16:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=186] Received exit status 0 2022-07-03T11:16:01.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=186] Received channel close 2022-07-03T11:16:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=186] Channel closed 2022-07-03T11:16:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=186] Received exit status 0 2022-07-03T11:16:01.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=186] Received channel close 2022-07-03T11:16:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=186] Channel closed 2022-07-03T11:16:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=187] Requesting new SSH session 2022-07-03T11:16:01.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=187] Requesting new SSH session 2022-07-03T11:16:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:01.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=187] Received exit status 0 2022-07-03T11:16:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=187] Received channel close 2022-07-03T11:16:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=187] Channel closed 2022-07-03T11:16:01.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=187] Received exit status 0 2022-07-03T11:16:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=187] Received channel close 2022-07-03T11:16:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=187] Channel closed 2022-07-03T11:16:01.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=49] Login timeout expired 2022-07-03T11:16:01.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=49] Aborting connection 2022-07-03T11:16:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:16:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:16:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:16:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:16:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:16:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:16:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:16:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:16:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:16:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:16:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:16:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:16:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:16:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:16:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:16:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:16:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:16:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:16:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:16:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:16:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:16:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:16:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:16:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:16:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:16:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:16:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:16:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:16:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:16:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:16:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:16:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:16:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:16:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:16:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:16:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:16:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:16:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:16:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:16:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:16:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:16:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:16:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=49] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:16:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=49] Local address: 172.21.15.161, port 51082 2022-07-03T11:16:01.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=49] Peer address: 172.21.15.101, port 22 2022-07-03T11:16:01.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=184] Received exit status 0 2022-07-03T11:16:01.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=184] Received channel close 2022-07-03T11:16:01.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=184] Channel closed 2022-07-03T11:16:01.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=184] Received exit status 0 2022-07-03T11:16:01.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=184] Received channel close 2022-07-03T11:16:01.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=184] Channel closed 2022-07-03T11:16:01.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=185] Requesting new SSH session 2022-07-03T11:16:01.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=185] Requesting new SSH session 2022-07-03T11:16:01.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=185] Command: which python3 2022-07-03T11:16:01.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=185] Command: which python3 2022-07-03T11:16:01.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=185] Received exit status 0 2022-07-03T11:16:01.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=185] Received channel close 2022-07-03T11:16:01.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=185] Channel closed 2022-07-03T11:16:01.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=186] Requesting new SSH session 2022-07-03T11:16:01.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=185] Received exit status 0 2022-07-03T11:16:01.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=185] Received channel close 2022-07-03T11:16:01.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=185] Channel closed 2022-07-03T11:16:01.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=186] Requesting new SSH session 2022-07-03T11:16:01.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=186] Command: sudo true 2022-07-03T11:16:01.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=186] Command: sudo true 2022-07-03T11:16:01.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=186] Received exit status 0 2022-07-03T11:16:01.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=186] Received channel close 2022-07-03T11:16:01.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=186] Channel closed 2022-07-03T11:16:01.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=186] Received exit status 0 2022-07-03T11:16:01.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=186] Received channel close 2022-07-03T11:16:01.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=186] Channel closed 2022-07-03T11:16:01.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=187] Requesting new SSH session 2022-07-03T11:16:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=187] Requesting new SSH session 2022-07-03T11:16:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:01.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=187] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:16:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=187] Received exit status 0 2022-07-03T11:16:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=187] Received channel close 2022-07-03T11:16:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=187] Channel closed 2022-07-03T11:16:01.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=187] Received exit status 0 2022-07-03T11:16:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=187] Received channel close 2022-07-03T11:16:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=187] Channel closed 2022-07-03T11:16:01.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=49] Login timeout expired 2022-07-03T11:16:01.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=49] Aborting connection 2022-07-03T11:16:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:02 smithi161 ceph-mon[91579]: pgmap v1110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:02 smithi188 ceph-mon[92425]: pgmap v1110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:04 smithi161 ceph-mon[91579]: pgmap v1111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:04 smithi188 ceph-mon[92425]: pgmap v1111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:06 smithi161 ceph-mon[91579]: pgmap v1112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:06 smithi188 ceph-mon[92425]: pgmap v1112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:08 smithi161 ceph-mon[91579]: pgmap v1113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:08 smithi188 ceph-mon[92425]: pgmap v1113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:10 smithi161 ceph-mon[91579]: pgmap v1114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:10 smithi188 ceph-mon[92425]: pgmap v1114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:12 smithi161 ceph-mon[91579]: pgmap v1115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:12 smithi188 ceph-mon[92425]: pgmap v1115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:14 smithi161 ceph-mon[91579]: pgmap v1116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:14 smithi188 ceph-mon[92425]: pgmap v1116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:16 smithi161 ceph-mon[91579]: pgmap v1117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:16 smithi188 ceph-mon[92425]: pgmap v1117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:18 smithi161 ceph-mon[91579]: pgmap v1118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:18 smithi188 ceph-mon[92425]: pgmap v1118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:20 smithi161 ceph-mon[91579]: pgmap v1119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:20 smithi188 ceph-mon[92425]: pgmap v1119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:22 smithi161 ceph-mon[91579]: pgmap v1120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:22 smithi188 ceph-mon[92425]: pgmap v1120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:24 smithi161 ceph-mon[91579]: pgmap v1121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:24 smithi188 ceph-mon[92425]: pgmap v1121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:26 smithi161 ceph-mon[91579]: pgmap v1122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:26 smithi188 ceph-mon[92425]: pgmap v1122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:28 smithi161 ceph-mon[91579]: pgmap v1123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:28 smithi188 ceph-mon[92425]: pgmap v1123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:30 smithi161 ceph-mon[91579]: pgmap v1124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:30 smithi188 ceph-mon[92425]: pgmap v1124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:32 smithi161 ceph-mon[91579]: pgmap v1125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:32 smithi188 ceph-mon[92425]: pgmap v1125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:34 smithi161 ceph-mon[91579]: pgmap v1126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:34 smithi188 ceph-mon[92425]: pgmap v1126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:36 smithi161 ceph-mon[91579]: pgmap v1127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:36 smithi188 ceph-mon[92425]: pgmap v1127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:38 smithi161 ceph-mon[91579]: pgmap v1128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:38 smithi188 ceph-mon[92425]: pgmap v1128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:40 smithi161 ceph-mon[91579]: pgmap v1129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:40 smithi188 ceph-mon[92425]: pgmap v1129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:42 smithi161 ceph-mon[91579]: pgmap v1130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:42 smithi188 ceph-mon[92425]: pgmap v1130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:44 smithi161 ceph-mon[91579]: pgmap v1131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:44 smithi188 ceph-mon[92425]: pgmap v1131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:46 smithi161 ceph-mon[91579]: pgmap v1132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:46 smithi188 ceph-mon[92425]: pgmap v1132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:48 smithi161 ceph-mon[91579]: pgmap v1133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:48 smithi188 ceph-mon[92425]: pgmap v1133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:50 smithi161 ceph-mon[91579]: pgmap v1134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:50 smithi188 ceph-mon[92425]: pgmap v1134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:52 smithi161 ceph-mon[91579]: pgmap v1135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:52 smithi188 ceph-mon[92425]: pgmap v1135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:54 smithi161 ceph-mon[91579]: pgmap v1136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:54 smithi188 ceph-mon[92425]: pgmap v1136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:16:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:56 smithi161 ceph-mon[91579]: pgmap v1137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:56 smithi188 ceph-mon[92425]: pgmap v1137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:16:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:16:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:16:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:16:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:16:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:16:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:16:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:16:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:58 smithi161 ceph-mon[91579]: pgmap v1138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:58 smithi188 ceph-mon[92425]: pgmap v1138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:16:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:16:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:00 smithi161 ceph-mon[91579]: pgmap v1139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:00 smithi188 ceph-mon[92425]: pgmap v1139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:17:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:17:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:17:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:17:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:17:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:17:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:17:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:17:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:02 smithi188 ceph-mon[92425]: pgmap v1140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:02 smithi161 ceph-mon[91579]: pgmap v1140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:04 smithi188 ceph-mon[92425]: pgmap v1141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:04 smithi161 ceph-mon[91579]: pgmap v1141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:06 smithi188 ceph-mon[92425]: pgmap v1142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:06 smithi161 ceph-mon[91579]: pgmap v1142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:08 smithi161 ceph-mon[91579]: pgmap v1143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:08 smithi188 ceph-mon[92425]: pgmap v1143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:10 smithi161 ceph-mon[91579]: pgmap v1144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:10 smithi188 ceph-mon[92425]: pgmap v1144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:12 smithi161 ceph-mon[91579]: pgmap v1145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:12 smithi188 ceph-mon[92425]: pgmap v1145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:14 smithi161 ceph-mon[91579]: pgmap v1146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:14 smithi188 ceph-mon[92425]: pgmap v1146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:16 smithi188 ceph-mon[92425]: pgmap v1147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:16 smithi161 ceph-mon[91579]: pgmap v1147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:18 smithi161 ceph-mon[91579]: pgmap v1148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:18 smithi188 ceph-mon[92425]: pgmap v1148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:20 smithi161 ceph-mon[91579]: pgmap v1149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:20 smithi188 ceph-mon[92425]: pgmap v1149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:22 smithi161 ceph-mon[91579]: pgmap v1150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:22 smithi188 ceph-mon[92425]: pgmap v1150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:24 smithi161 ceph-mon[91579]: pgmap v1151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:24 smithi188 ceph-mon[92425]: pgmap v1151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:26 smithi161 ceph-mon[91579]: pgmap v1152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:26 smithi188 ceph-mon[92425]: pgmap v1152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:28 smithi161 ceph-mon[91579]: pgmap v1153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:28 smithi188 ceph-mon[92425]: pgmap v1153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-mon[91579]: pgmap v1154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:30 smithi188 ceph-mon[92425]: pgmap v1154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:17:30.611+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:17:30.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:17:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:17:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:17:30.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:17:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:17:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:17:30.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:17:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:17:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:17:30.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:17:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:17:30.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:17:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:17:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:17:30.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:17:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:17:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:17:30.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:17:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:17:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:17:30.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:17:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:17:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:17:30.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:17:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:17:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:17:30.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:17:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:17:30.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:17:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:17:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:17:30.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:17:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:17:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:17:30.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:17:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:17:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:17:30.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:17:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:17:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:17:30.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:17:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:17:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:17:30.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:17:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=51] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:17:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=51] Local address: 172.21.15.161, port 51086 2022-07-03T11:17:30.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=51] Peer address: 172.21.15.101, port 22 2022-07-03T11:17:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=188] Received exit status 0 2022-07-03T11:17:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=188] Received channel close 2022-07-03T11:17:30.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=188] Channel closed 2022-07-03T11:17:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=189] Requesting new SSH session 2022-07-03T11:17:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=189] Command: which python3 2022-07-03T11:17:30.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=188] Received exit status 0 2022-07-03T11:17:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=188] Received channel close 2022-07-03T11:17:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=188] Channel closed 2022-07-03T11:17:30.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=189] Requesting new SSH session 2022-07-03T11:17:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=189] Command: which python3 2022-07-03T11:17:30.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=189] Received exit status 0 2022-07-03T11:17:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=189] Received channel close 2022-07-03T11:17:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=189] Channel closed 2022-07-03T11:17:30.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=190] Requesting new SSH session 2022-07-03T11:17:30.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=190] Command: sudo true 2022-07-03T11:17:30.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=189] Received exit status 0 2022-07-03T11:17:30.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=189] Received channel close 2022-07-03T11:17:30.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=189] Channel closed 2022-07-03T11:17:30.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=190] Requesting new SSH session 2022-07-03T11:17:30.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=190] Command: sudo true 2022-07-03T11:17:30.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=190] Received exit status 0 2022-07-03T11:17:30.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=190] Received channel close 2022-07-03T11:17:30.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=190] Channel closed 2022-07-03T11:17:30.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=191] Requesting new SSH session 2022-07-03T11:17:30.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:30.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=190] Received exit status 0 2022-07-03T11:17:30.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=190] Received channel close 2022-07-03T11:17:30.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=190] Channel closed 2022-07-03T11:17:30.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=191] Requesting new SSH session 2022-07-03T11:17:30.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:30.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=191] Received exit status 0 2022-07-03T11:17:30.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=191] Received channel close 2022-07-03T11:17:30.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=191] Channel closed 2022-07-03T11:17:30.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=191] Received exit status 0 2022-07-03T11:17:30.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=191] Received channel close 2022-07-03T11:17:30.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=191] Channel closed 2022-07-03T11:17:30.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=51] Login timeout expired 2022-07-03T11:17:30.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=51] Aborting connection 2022-07-03T11:17:30.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:17:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:17:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:17:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:17:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:17:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:17:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:17:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:17:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:17:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:17:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:17:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:17:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:17:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:17:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:17:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:17:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:17:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:17:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:17:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:17:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:17:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:17:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:17:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:17:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:17:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:17:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:17:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:17:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:17:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:17:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:17:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:17:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:17:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:17:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:17:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:17:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:17:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:17:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:17:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:17:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:17:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:17:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:17:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:17:31.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:17:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:17:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:17:31.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=51] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:17:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=51] Local address: 172.21.15.161, port 51086 2022-07-03T11:17:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=51] Peer address: 172.21.15.101, port 22 2022-07-03T11:17:31.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=188] Received exit status 0 2022-07-03T11:17:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=188] Received channel close 2022-07-03T11:17:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=188] Channel closed 2022-07-03T11:17:31.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=189] Requesting new SSH session 2022-07-03T11:17:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=189] Command: which python3 2022-07-03T11:17:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=188] Received exit status 0 2022-07-03T11:17:31.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=188] Received channel close 2022-07-03T11:17:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=188] Channel closed 2022-07-03T11:17:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=189] Requesting new SSH session 2022-07-03T11:17:31.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=189] Command: which python3 2022-07-03T11:17:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=189] Received exit status 0 2022-07-03T11:17:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=189] Received channel close 2022-07-03T11:17:31.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=189] Channel closed 2022-07-03T11:17:31.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=190] Requesting new SSH session 2022-07-03T11:17:31.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=190] Command: sudo true 2022-07-03T11:17:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=189] Received exit status 0 2022-07-03T11:17:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=189] Received channel close 2022-07-03T11:17:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=189] Channel closed 2022-07-03T11:17:31.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=190] Requesting new SSH session 2022-07-03T11:17:31.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=190] Command: sudo true 2022-07-03T11:17:31.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=190] Received exit status 0 2022-07-03T11:17:31.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=190] Received channel close 2022-07-03T11:17:31.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=190] Channel closed 2022-07-03T11:17:31.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=191] Requesting new SSH session 2022-07-03T11:17:31.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:31.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=190] Received exit status 0 2022-07-03T11:17:31.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=190] Received channel close 2022-07-03T11:17:31.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=190] Channel closed 2022-07-03T11:17:31.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=191] Requesting new SSH session 2022-07-03T11:17:31.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:31.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=191] Received exit status 0 2022-07-03T11:17:31.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=191] Received channel close 2022-07-03T11:17:31.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=191] Channel closed 2022-07-03T11:17:31.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=191] Received exit status 0 2022-07-03T11:17:31.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=191] Received channel close 2022-07-03T11:17:31.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=191] Channel closed 2022-07-03T11:17:31.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=51] Login timeout expired 2022-07-03T11:17:31.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=51] Aborting connection 2022-07-03T11:17:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:17:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:17:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:17:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:17:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:17:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:17:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:17:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:17:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:17:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:17:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:17:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:17:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:17:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:17:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:17:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:17:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:17:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:17:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:17:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:17:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:17:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:17:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:17:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:17:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:17:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:17:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:17:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:17:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:17:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:17:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:17:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:17:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:17:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:17:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:17:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:17:31.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:17:31.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:17:31.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:17:31.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:17:31.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:17:31.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:17:31.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:17:31.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:17:31.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:17:31.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=51] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:17:31.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=51] Local address: 172.21.15.161, port 51086 2022-07-03T11:17:31.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=51] Peer address: 172.21.15.101, port 22 2022-07-03T11:17:31.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=188] Received exit status 0 2022-07-03T11:17:31.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=188] Received channel close 2022-07-03T11:17:31.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=188] Channel closed 2022-07-03T11:17:31.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=189] Requesting new SSH session 2022-07-03T11:17:31.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=189] Command: which python3 2022-07-03T11:17:31.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=188] Received exit status 0 2022-07-03T11:17:31.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=188] Received channel close 2022-07-03T11:17:31.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=188] Channel closed 2022-07-03T11:17:31.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=189] Requesting new SSH session 2022-07-03T11:17:31.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=189] Command: which python3 2022-07-03T11:17:31.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=189] Received exit status 0 2022-07-03T11:17:31.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=189] Received channel close 2022-07-03T11:17:31.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=189] Channel closed 2022-07-03T11:17:31.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=190] Requesting new SSH session 2022-07-03T11:17:31.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=190] Command: sudo true 2022-07-03T11:17:31.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=189] Received exit status 0 2022-07-03T11:17:31.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=189] Received channel close 2022-07-03T11:17:31.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=189] Channel closed 2022-07-03T11:17:31.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=190] Requesting new SSH session 2022-07-03T11:17:31.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=190] Command: sudo true 2022-07-03T11:17:31.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=190] Received exit status 0 2022-07-03T11:17:31.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=190] Received channel close 2022-07-03T11:17:31.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=190] Channel closed 2022-07-03T11:17:31.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=191] Requesting new SSH session 2022-07-03T11:17:31.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:31.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=190] Received exit status 0 2022-07-03T11:17:31.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=190] Received channel close 2022-07-03T11:17:31.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=190] Channel closed 2022-07-03T11:17:31.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=191] Requesting new SSH session 2022-07-03T11:17:31.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=191] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:17:31.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=191] Received exit status 0 2022-07-03T11:17:31.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=191] Received channel close 2022-07-03T11:17:31.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=191] Channel closed 2022-07-03T11:17:31.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=191] Received exit status 0 2022-07-03T11:17:31.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=191] Received channel close 2022-07-03T11:17:31.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=191] Channel closed 2022-07-03T11:17:31.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=51] Login timeout expired 2022-07-03T11:17:31.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=51] Aborting connection 2022-07-03T11:17:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:32 smithi161 ceph-mon[91579]: pgmap v1155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:32 smithi188 ceph-mon[92425]: pgmap v1155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:34 smithi161 ceph-mon[91579]: pgmap v1156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:34 smithi188 ceph-mon[92425]: pgmap v1156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:36 smithi161 ceph-mon[91579]: pgmap v1157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:36 smithi188 ceph-mon[92425]: pgmap v1157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:38 smithi161 ceph-mon[91579]: pgmap v1158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:38 smithi188 ceph-mon[92425]: pgmap v1158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:40 smithi161 ceph-mon[91579]: pgmap v1159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:40 smithi188 ceph-mon[92425]: pgmap v1159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:42 smithi161 ceph-mon[91579]: pgmap v1160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:42 smithi188 ceph-mon[92425]: pgmap v1160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:44 smithi188 ceph-mon[92425]: pgmap v1161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:44 smithi161 ceph-mon[91579]: pgmap v1161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:46 smithi188 ceph-mon[92425]: pgmap v1162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:46 smithi161 ceph-mon[91579]: pgmap v1162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:48 smithi161 ceph-mon[91579]: pgmap v1163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:48 smithi188 ceph-mon[92425]: pgmap v1163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:50 smithi161 ceph-mon[91579]: pgmap v1164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:50 smithi188 ceph-mon[92425]: pgmap v1164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:52 smithi188 ceph-mon[92425]: pgmap v1165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:52 smithi161 ceph-mon[91579]: pgmap v1165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:54 smithi188 ceph-mon[92425]: pgmap v1166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:54 smithi161 ceph-mon[91579]: pgmap v1166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:17:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:56 smithi188 ceph-mon[92425]: pgmap v1167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:56 smithi161 ceph-mon[91579]: pgmap v1167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:17:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:17:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:17:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:17:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:17:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:17:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:17:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:17:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:58 smithi188 ceph-mon[92425]: pgmap v1168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:58 smithi161 ceph-mon[91579]: pgmap v1168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:17:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:17:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:00 smithi188 ceph-mon[92425]: pgmap v1169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:00 smithi161 ceph-mon[91579]: pgmap v1169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:02 smithi188 ceph-mon[92425]: pgmap v1170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:02 smithi161 ceph-mon[91579]: pgmap v1170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:04 smithi188 ceph-mon[92425]: pgmap v1171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:04 smithi161 ceph-mon[91579]: pgmap v1171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:06 smithi188 ceph-mon[92425]: pgmap v1172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:06 smithi161 ceph-mon[91579]: pgmap v1172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:08 smithi161 ceph-mon[91579]: pgmap v1173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:08 smithi188 ceph-mon[92425]: pgmap v1173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:10 smithi161 ceph-mon[91579]: pgmap v1174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:10 smithi188 ceph-mon[92425]: pgmap v1174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:12 smithi188 ceph-mon[92425]: pgmap v1175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:12 smithi161 ceph-mon[91579]: pgmap v1175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:14 smithi188 ceph-mon[92425]: pgmap v1176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:14 smithi161 ceph-mon[91579]: pgmap v1176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:16 smithi188 ceph-mon[92425]: pgmap v1177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:16 smithi161 ceph-mon[91579]: pgmap v1177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:18 smithi161 ceph-mon[91579]: pgmap v1178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:18 smithi188 ceph-mon[92425]: pgmap v1178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:20 smithi161 ceph-mon[91579]: pgmap v1179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:20 smithi188 ceph-mon[92425]: pgmap v1179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:22 smithi188 ceph-mon[92425]: pgmap v1180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:22 smithi161 ceph-mon[91579]: pgmap v1180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:24 smithi188 ceph-mon[92425]: pgmap v1181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:24 smithi161 ceph-mon[91579]: pgmap v1181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:26 smithi188 ceph-mon[92425]: pgmap v1182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:26 smithi161 ceph-mon[91579]: pgmap v1182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:28 smithi161 ceph-mon[91579]: pgmap v1183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:28 smithi188 ceph-mon[92425]: pgmap v1183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:30 smithi161 ceph-mon[91579]: pgmap v1184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:30 smithi188 ceph-mon[92425]: pgmap v1184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:18:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:18:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:18:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:18:31.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:18:31.058+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:31.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:18:31.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:31.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:18:31.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:31.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:31.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:31.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:18:31.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:31.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:18:31.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:31.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:18:31.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:31.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:18:31.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:31.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:18:31.058+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:18:31.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:18:31.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:31.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:18:31.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:31.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:31.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:31.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:18:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:18:31.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:31.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:18:31.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:31.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:18:31.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:31.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:18:31.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:18:31.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:18:31.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:18:31.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:18:31.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:18:31.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:18:31.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:18:31.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:18:31.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:18:31.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:18:31.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:18:31.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:18:31.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:18:31.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:18:31.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:18:31.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:18:31.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:18:31.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:18:31.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:18:31.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:18:31.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:18:31.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:18:31.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:18:31.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:18:31.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:18:31.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:18:31.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:18:31.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:18:31.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:18:31.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:18:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:18:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:18:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:18:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:18:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:32 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:18:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:18:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:18:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:18:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:18:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:18:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:18:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:18:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:18:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:18:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:18:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:18:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:18:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:18:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:18:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:18:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:18:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:18:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:18:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:18:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:18:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:18:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:18:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:18:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:18:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:18:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:18:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:18:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:18:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:18:32.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:18:32.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:18:32.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:18:32.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:18:32.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:18:32.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:18:32.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:18:32.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:18:32.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:18:32.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:18:32.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:32.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:32 smithi161 ceph-mon[91579]: pgmap v1185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:32 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:18:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:18:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:18:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:18:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:18:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:18:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:18:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:18:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:18:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:18:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:18:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:18:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:18:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:18:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:18:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:18:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:18:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:18:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:18:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:18:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:18:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:18:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:18:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:18:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:18:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:18:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:18:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:18:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:18:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:18:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:18:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:18:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:18:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:18:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:18:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:18:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:18:32.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:18:32.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:18:32.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:18:32.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:18:32.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:18:32.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:18:32.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:18:32.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:18:32.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:18:32.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:18:32.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:18:32.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:18:32.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:18:32.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:32 smithi188 ceph-mon[92425]: pgmap v1185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:34 smithi188 ceph-mon[92425]: pgmap v1186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:34 smithi161 ceph-mon[91579]: pgmap v1186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:36 smithi188 ceph-mon[92425]: pgmap v1187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:36 smithi161 ceph-mon[91579]: pgmap v1187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:38 smithi161 ceph-mon[91579]: pgmap v1188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:38 smithi188 ceph-mon[92425]: pgmap v1188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:40 smithi161 ceph-mon[91579]: pgmap v1189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:40 smithi188 ceph-mon[92425]: pgmap v1189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:42 smithi188 ceph-mon[92425]: pgmap v1190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:42 smithi161 ceph-mon[91579]: pgmap v1190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:44 smithi188 ceph-mon[92425]: pgmap v1191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:44 smithi161 ceph-mon[91579]: pgmap v1191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:46 smithi188 ceph-mon[92425]: pgmap v1192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:46 smithi161 ceph-mon[91579]: pgmap v1192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:48 smithi161 ceph-mon[91579]: pgmap v1193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:48 smithi188 ceph-mon[92425]: pgmap v1193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:50 smithi161 ceph-mon[91579]: pgmap v1194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:50 smithi188 ceph-mon[92425]: pgmap v1194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:52 smithi188 ceph-mon[92425]: pgmap v1195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:52 smithi161 ceph-mon[91579]: pgmap v1195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:54 smithi188 ceph-mon[92425]: pgmap v1196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:54 smithi161 ceph-mon[91579]: pgmap v1196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:18:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:56 smithi188 ceph-mon[92425]: pgmap v1197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:56 smithi161 ceph-mon[91579]: pgmap v1197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:18:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:18:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:18:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:18:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:18:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:18:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:58 smithi188 ceph-mon[92425]: pgmap v1198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:58 smithi161 ceph-mon[91579]: pgmap v1198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:18:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:18:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:00 smithi161 ceph-mon[91579]: pgmap v1199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:00 smithi188 ceph-mon[92425]: pgmap v1199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:02 smithi161 ceph-mon[91579]: pgmap v1200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:02 smithi188 ceph-mon[92425]: pgmap v1200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:04 smithi161 ceph-mon[91579]: pgmap v1201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:04 smithi188 ceph-mon[92425]: pgmap v1201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:06 smithi161 ceph-mon[91579]: pgmap v1202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:06 smithi188 ceph-mon[92425]: pgmap v1202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:08 smithi161 ceph-mon[91579]: pgmap v1203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:08 smithi188 ceph-mon[92425]: pgmap v1203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:10 smithi161 ceph-mon[91579]: pgmap v1204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:10 smithi188 ceph-mon[92425]: pgmap v1204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:12 smithi188 ceph-mon[92425]: pgmap v1205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:12 smithi161 ceph-mon[91579]: pgmap v1205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:14 smithi188 ceph-mon[92425]: pgmap v1206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:14 smithi161 ceph-mon[91579]: pgmap v1206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:16 smithi188 ceph-mon[92425]: pgmap v1207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:16 smithi161 ceph-mon[91579]: pgmap v1207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:18 smithi161 ceph-mon[91579]: pgmap v1208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:18 smithi188 ceph-mon[92425]: pgmap v1208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:20 smithi161 ceph-mon[91579]: pgmap v1209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:20 smithi188 ceph-mon[92425]: pgmap v1209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:22 smithi188 ceph-mon[92425]: pgmap v1210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:22 smithi161 ceph-mon[91579]: pgmap v1210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:24 smithi188 ceph-mon[92425]: pgmap v1211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:24 smithi161 ceph-mon[91579]: pgmap v1211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:26 smithi161 ceph-mon[91579]: pgmap v1212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:26 smithi188 ceph-mon[92425]: pgmap v1212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:28 smithi161 ceph-mon[91579]: pgmap v1213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:28 smithi188 ceph-mon[92425]: pgmap v1213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:30 smithi161 ceph-mon[91579]: pgmap v1214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:30 smithi188 ceph-mon[92425]: pgmap v1214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:19:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:19:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:19:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:19:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:19:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:19:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:19:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:32 smithi161 ceph-mon[91579]: pgmap v1215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:19:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:32 smithi188 ceph-mon[92425]: pgmap v1215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:34 smithi188 ceph-mon[92425]: pgmap v1216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:34 smithi161 ceph-mon[91579]: pgmap v1216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:36 smithi161 ceph-mon[91579]: pgmap v1217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:36 smithi188 ceph-mon[92425]: pgmap v1217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:38 smithi161 ceph-mon[91579]: pgmap v1218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:38 smithi188 ceph-mon[92425]: pgmap v1218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:40 smithi161 ceph-mon[91579]: pgmap v1219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:40 smithi188 ceph-mon[92425]: pgmap v1219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:42 smithi161 ceph-mon[91579]: pgmap v1220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:42 smithi188 ceph-mon[92425]: pgmap v1220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:44 smithi161 ceph-mon[91579]: pgmap v1221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:44 smithi188 ceph-mon[92425]: pgmap v1221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:46 smithi161 ceph-mon[91579]: pgmap v1222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:46 smithi188 ceph-mon[92425]: pgmap v1222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:48 smithi161 ceph-mon[91579]: pgmap v1223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:48 smithi188 ceph-mon[92425]: pgmap v1223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:50 smithi161 ceph-mon[91579]: pgmap v1224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:50 smithi188 ceph-mon[92425]: pgmap v1224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:52 smithi161 ceph-mon[91579]: pgmap v1225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:52 smithi188 ceph-mon[92425]: pgmap v1225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:54 smithi161 ceph-mon[91579]: pgmap v1226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:54 smithi188 ceph-mon[92425]: pgmap v1226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:19:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:56 smithi161 ceph-mon[91579]: pgmap v1227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:56 smithi188 ceph-mon[92425]: pgmap v1227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:19:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:19:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:19:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:19:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:19:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:19:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:19:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:19:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:58 smithi161 ceph-mon[91579]: pgmap v1228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:58 smithi188 ceph-mon[92425]: pgmap v1228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:19:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:19:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:00 smithi161 ceph-mon[91579]: pgmap v1229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:20:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:00 smithi188 ceph-mon[92425]: pgmap v1229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:20:01.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:20:01.153+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:20:01.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:20:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:20:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:20:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:20:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:20:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:20:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:20:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:20:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:20:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:20:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:20:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:20:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:20:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:20:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:20:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:20:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:20:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:20:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:20:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:20:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:20:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:20:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:20:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:20:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:20:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:20:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:20:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:20:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:20:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:20:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:20:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:20:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:20:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:20:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:20:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:20:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:20:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:20:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:20:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:20:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:20:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:20:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:20:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:20:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=55] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:20:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=55] Local address: 172.21.15.161, port 51094 2022-07-03T11:20:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=55] Peer address: 172.21.15.101, port 22 2022-07-03T11:20:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=196] Received exit status 0 2022-07-03T11:20:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=196] Received channel close 2022-07-03T11:20:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=196] Channel closed 2022-07-03T11:20:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=196] Received exit status 0 2022-07-03T11:20:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=196] Received channel close 2022-07-03T11:20:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=196] Channel closed 2022-07-03T11:20:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=197] Requesting new SSH session 2022-07-03T11:20:01.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=197] Requesting new SSH session 2022-07-03T11:20:01.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=197] Command: which python3 2022-07-03T11:20:01.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=197] Command: which python3 2022-07-03T11:20:01.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=197] Received exit status 0 2022-07-03T11:20:01.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=197] Received channel close 2022-07-03T11:20:01.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=197] Channel closed 2022-07-03T11:20:01.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=198] Requesting new SSH session 2022-07-03T11:20:01.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=197] Received exit status 0 2022-07-03T11:20:01.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=197] Received channel close 2022-07-03T11:20:01.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=197] Channel closed 2022-07-03T11:20:01.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=198] Requesting new SSH session 2022-07-03T11:20:01.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=198] Command: sudo true 2022-07-03T11:20:01.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=198] Command: sudo true 2022-07-03T11:20:01.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=198] Received exit status 0 2022-07-03T11:20:01.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=198] Received channel close 2022-07-03T11:20:01.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=198] Channel closed 2022-07-03T11:20:01.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=199] Requesting new SSH session 2022-07-03T11:20:01.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=198] Received exit status 0 2022-07-03T11:20:01.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=198] Received channel close 2022-07-03T11:20:01.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=198] Channel closed 2022-07-03T11:20:01.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:01.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=199] Requesting new SSH session 2022-07-03T11:20:01.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:01.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=199] Received exit status 0 2022-07-03T11:20:01.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=199] Received channel close 2022-07-03T11:20:01.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=199] Channel closed 2022-07-03T11:20:01.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=199] Received exit status 0 2022-07-03T11:20:01.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=199] Received channel close 2022-07-03T11:20:01.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=199] Channel closed 2022-07-03T11:20:01.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=55] Login timeout expired 2022-07-03T11:20:01.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=55] Aborting connection 2022-07-03T11:20:01.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:20:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:20:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:20:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:20:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:20:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:20:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:20:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:20:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:20:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:20:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:20:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:20:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:20:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:20:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:20:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:20:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:20:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:20:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:20:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:20:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:20:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:20:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:20:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:20:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:20:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:20:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:20:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:20:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:20:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:20:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:20:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:20:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:20:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:20:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:20:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:20:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:20:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:20:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:20:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:20:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:20:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:20:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:20:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:20:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:20:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:20:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:20:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=55] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:20:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=55] Local address: 172.21.15.161, port 51094 2022-07-03T11:20:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=55] Peer address: 172.21.15.101, port 22 2022-07-03T11:20:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=196] Received exit status 0 2022-07-03T11:20:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=196] Received channel close 2022-07-03T11:20:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=196] Channel closed 2022-07-03T11:20:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=196] Received exit status 0 2022-07-03T11:20:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=196] Received channel close 2022-07-03T11:20:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=196] Channel closed 2022-07-03T11:20:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=197] Requesting new SSH session 2022-07-03T11:20:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=197] Requesting new SSH session 2022-07-03T11:20:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=197] Command: which python3 2022-07-03T11:20:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=197] Command: which python3 2022-07-03T11:20:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=197] Received exit status 0 2022-07-03T11:20:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=197] Received channel close 2022-07-03T11:20:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=197] Channel closed 2022-07-03T11:20:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=198] Requesting new SSH session 2022-07-03T11:20:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=197] Received exit status 0 2022-07-03T11:20:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=197] Received channel close 2022-07-03T11:20:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=197] Channel closed 2022-07-03T11:20:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=198] Requesting new SSH session 2022-07-03T11:20:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=198] Command: sudo true 2022-07-03T11:20:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=198] Command: sudo true 2022-07-03T11:20:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=198] Received exit status 0 2022-07-03T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=198] Received channel close 2022-07-03T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=198] Channel closed 2022-07-03T11:20:02.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=199] Requesting new SSH session 2022-07-03T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=198] Received exit status 0 2022-07-03T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=198] Received channel close 2022-07-03T11:20:02.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=198] Channel closed 2022-07-03T11:20:02.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:02.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=199] Requesting new SSH session 2022-07-03T11:20:02.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:02.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=199] Received exit status 0 2022-07-03T11:20:02.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=199] Received channel close 2022-07-03T11:20:02.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=199] Channel closed 2022-07-03T11:20:02.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=199] Received exit status 0 2022-07-03T11:20:02.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=199] Received channel close 2022-07-03T11:20:02.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=199] Channel closed 2022-07-03T11:20:02.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=55] Login timeout expired 2022-07-03T11:20:02.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=55] Aborting connection 2022-07-03T11:20:02.907 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:02 smithi161 ceph-mon[91579]: pgmap v1230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:20:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:20:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:20:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:20:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:20:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:20:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:20:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:20:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:20:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:20:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:20:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:20:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:20:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:20:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:20:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:20:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:20:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:20:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:20:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:20:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:20:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:20:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:20:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:20:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:20:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:20:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:20:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:20:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:20:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:20:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:20:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:20:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:20:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:20:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:20:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:20:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:20:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:20:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:20:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:20:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:20:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:20:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:20:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:20:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:20:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=55] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:20:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=55] Local address: 172.21.15.161, port 51094 2022-07-03T11:20:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=55] Peer address: 172.21.15.101, port 22 2022-07-03T11:20:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=196] Received exit status 0 2022-07-03T11:20:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=196] Received channel close 2022-07-03T11:20:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=196] Channel closed 2022-07-03T11:20:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=196] Received exit status 0 2022-07-03T11:20:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=196] Received channel close 2022-07-03T11:20:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=196] Channel closed 2022-07-03T11:20:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=197] Requesting new SSH session 2022-07-03T11:20:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=197] Requesting new SSH session 2022-07-03T11:20:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=197] Command: which python3 2022-07-03T11:20:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=197] Command: which python3 2022-07-03T11:20:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=197] Received exit status 0 2022-07-03T11:20:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=197] Received channel close 2022-07-03T11:20:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=197] Channel closed 2022-07-03T11:20:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=198] Requesting new SSH session 2022-07-03T11:20:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=197] Received exit status 0 2022-07-03T11:20:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=197] Received channel close 2022-07-03T11:20:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=197] Channel closed 2022-07-03T11:20:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=198] Requesting new SSH session 2022-07-03T11:20:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=198] Command: sudo true 2022-07-03T11:20:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=198] Command: sudo true 2022-07-03T11:20:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=198] Received exit status 0 2022-07-03T11:20:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=198] Received channel close 2022-07-03T11:20:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=198] Channel closed 2022-07-03T11:20:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=199] Requesting new SSH session 2022-07-03T11:20:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=198] Received exit status 0 2022-07-03T11:20:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=198] Received channel close 2022-07-03T11:20:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=198] Channel closed 2022-07-03T11:20:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=199] Requesting new SSH session 2022-07-03T11:20:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=199] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:20:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=199] Received exit status 0 2022-07-03T11:20:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=199] Received channel close 2022-07-03T11:20:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=199] Channel closed 2022-07-03T11:20:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=199] Received exit status 0 2022-07-03T11:20:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=199] Received channel close 2022-07-03T11:20:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=199] Channel closed 2022-07-03T11:20:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=55] Login timeout expired 2022-07-03T11:20:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=55] Aborting connection 2022-07-03T11:20:02.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:02 smithi188 ceph-mon[92425]: pgmap v1230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:04 smithi161 ceph-mon[91579]: pgmap v1231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:04 smithi188 ceph-mon[92425]: pgmap v1231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:06 smithi161 ceph-mon[91579]: pgmap v1232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:06 smithi188 ceph-mon[92425]: pgmap v1232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:08 smithi161 ceph-mon[91579]: pgmap v1233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:08 smithi188 ceph-mon[92425]: pgmap v1233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:10 smithi161 ceph-mon[91579]: pgmap v1234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:10 smithi188 ceph-mon[92425]: pgmap v1234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:12 smithi161 ceph-mon[91579]: pgmap v1235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:12 smithi188 ceph-mon[92425]: pgmap v1235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:14 smithi161 ceph-mon[91579]: pgmap v1236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:14 smithi188 ceph-mon[92425]: pgmap v1236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:16 smithi161 ceph-mon[91579]: pgmap v1237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:16 smithi188 ceph-mon[92425]: pgmap v1237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:18 smithi161 ceph-mon[91579]: pgmap v1238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:18 smithi188 ceph-mon[92425]: pgmap v1238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:20 smithi161 ceph-mon[91579]: pgmap v1239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:20 smithi188 ceph-mon[92425]: pgmap v1239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:22 smithi161 ceph-mon[91579]: pgmap v1240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:22 smithi188 ceph-mon[92425]: pgmap v1240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:24 smithi161 ceph-mon[91579]: pgmap v1241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:24 smithi188 ceph-mon[92425]: pgmap v1241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:26 smithi161 ceph-mon[91579]: pgmap v1242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:26 smithi188 ceph-mon[92425]: pgmap v1242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:28 smithi161 ceph-mon[91579]: pgmap v1243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:28 smithi188 ceph-mon[92425]: pgmap v1243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:30 smithi161 ceph-mon[91579]: pgmap v1244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:30 smithi188 ceph-mon[92425]: pgmap v1244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:32 smithi161 ceph-mon[91579]: pgmap v1245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:32 smithi188 ceph-mon[92425]: pgmap v1245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:34 smithi161 ceph-mon[91579]: pgmap v1246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:34 smithi188 ceph-mon[92425]: pgmap v1246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:36 smithi161 ceph-mon[91579]: pgmap v1247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:36 smithi188 ceph-mon[92425]: pgmap v1247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:38 smithi161 ceph-mon[91579]: pgmap v1248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:38 smithi188 ceph-mon[92425]: pgmap v1248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:40 smithi161 ceph-mon[91579]: pgmap v1249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:40 smithi188 ceph-mon[92425]: pgmap v1249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:42 smithi161 ceph-mon[91579]: pgmap v1250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:42 smithi188 ceph-mon[92425]: pgmap v1250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:44 smithi161 ceph-mon[91579]: pgmap v1251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:44 smithi188 ceph-mon[92425]: pgmap v1251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:46 smithi161 ceph-mon[91579]: pgmap v1252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:46 smithi188 ceph-mon[92425]: pgmap v1252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:48 smithi161 ceph-mon[91579]: pgmap v1253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:48.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:48 smithi188 ceph-mon[92425]: pgmap v1253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:50 smithi161 ceph-mon[91579]: pgmap v1254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:50 smithi188 ceph-mon[92425]: pgmap v1254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:52 smithi161 ceph-mon[91579]: pgmap v1255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:52 smithi188 ceph-mon[92425]: pgmap v1255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:54 smithi161 ceph-mon[91579]: pgmap v1256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:54 smithi188 ceph-mon[92425]: pgmap v1256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:20:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:56 smithi161 ceph-mon[91579]: pgmap v1257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:56 smithi188 ceph-mon[92425]: pgmap v1257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:20:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:20:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:20:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:20:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:20:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:58 smithi161 ceph-mon[91579]: pgmap v1258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:58 smithi188 ceph-mon[92425]: pgmap v1258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:20:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:20:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:20:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:00 smithi161 ceph-mon[91579]: pgmap v1259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:00 smithi188 ceph-mon[92425]: pgmap v1259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:01.459 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:21:01.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:21:01.460 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:21:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:21:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:21:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:21:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:02 smithi161 ceph-mon[91579]: pgmap v1260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:02 smithi188 ceph-mon[92425]: pgmap v1260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:04 smithi161 ceph-mon[91579]: pgmap v1261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:04 smithi188 ceph-mon[92425]: pgmap v1261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:06 smithi161 ceph-mon[91579]: pgmap v1262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:06 smithi188 ceph-mon[92425]: pgmap v1262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:08 smithi161 ceph-mon[91579]: pgmap v1263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:08 smithi188 ceph-mon[92425]: pgmap v1263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:10 smithi161 ceph-mon[91579]: pgmap v1264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:10 smithi188 ceph-mon[92425]: pgmap v1264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:12 smithi161 ceph-mon[91579]: pgmap v1265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:12 smithi188 ceph-mon[92425]: pgmap v1265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:14 smithi161 ceph-mon[91579]: pgmap v1266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:14 smithi188 ceph-mon[92425]: pgmap v1266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:16 smithi161 ceph-mon[91579]: pgmap v1267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:16 smithi188 ceph-mon[92425]: pgmap v1267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:18.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:21:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:21:17.766+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T11:21:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:18 smithi161 ceph-mon[91579]: pgmap v1268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:18 smithi188 ceph-mon[92425]: pgmap v1268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:21:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:20 smithi161 ceph-mon[91579]: pgmap v1269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:20 smithi188 ceph-mon[92425]: pgmap v1269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:22 smithi161 ceph-mon[91579]: pgmap v1270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:22 smithi188 ceph-mon[92425]: pgmap v1270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:24 smithi161 ceph-mon[91579]: pgmap v1271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:24 smithi188 ceph-mon[92425]: pgmap v1271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:26 smithi161 ceph-mon[91579]: pgmap v1272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:26 smithi188 ceph-mon[92425]: pgmap v1272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:28 smithi161 ceph-mon[91579]: pgmap v1273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:28 smithi188 ceph-mon[92425]: pgmap v1273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:30 smithi161 ceph-mon[91579]: pgmap v1274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:30 smithi188 ceph-mon[92425]: pgmap v1274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:21:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:32 smithi161 ceph-mon[91579]: pgmap v1275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:21:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:32 smithi188 ceph-mon[92425]: pgmap v1275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:34 smithi161 ceph-mon[91579]: pgmap v1276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:34 smithi188 ceph-mon[92425]: pgmap v1276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:36 smithi161 ceph-mon[91579]: pgmap v1277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:36 smithi188 ceph-mon[92425]: pgmap v1277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:38 smithi161 ceph-mon[91579]: pgmap v1278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:38 smithi188 ceph-mon[92425]: pgmap v1278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:40 smithi161 ceph-mon[91579]: pgmap v1279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:40 smithi188 ceph-mon[92425]: pgmap v1279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:42 smithi161 ceph-mon[91579]: pgmap v1280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:42 smithi188 ceph-mon[92425]: pgmap v1280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:21:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:44 smithi161 ceph-mon[91579]: pgmap v1281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:44 smithi188 ceph-mon[92425]: pgmap v1281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:46 smithi161 ceph-mon[91579]: pgmap v1282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:46 smithi188 ceph-mon[92425]: pgmap v1282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:48 smithi161 ceph-mon[91579]: pgmap v1283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:48 smithi188 ceph-mon[92425]: pgmap v1283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:50 smithi161 ceph-mon[91579]: pgmap v1284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:50 smithi188 ceph-mon[92425]: pgmap v1284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:52 smithi161 ceph-mon[91579]: pgmap v1285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:52 smithi188 ceph-mon[92425]: pgmap v1285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:54 smithi161 ceph-mon[91579]: pgmap v1286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:54 smithi188 ceph-mon[92425]: pgmap v1286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:21:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:56 smithi161 ceph-mon[91579]: pgmap v1287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:56 smithi188 ceph-mon[92425]: pgmap v1287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:21:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:21:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:21:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:21:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:21:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:21:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:21:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:21:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:58 smithi161 ceph-mon[91579]: pgmap v1288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:58 smithi188 ceph-mon[92425]: pgmap v1288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:21:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:21:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:00 smithi161 ceph-mon[91579]: pgmap v1289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:00 smithi188 ceph-mon[92425]: pgmap v1289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:01.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:22:01.190+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:22:01.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:22:01.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:22:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:22:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:22:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:22:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:22:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:22:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:22:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:22:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:22:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:22:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:22:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:22:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:22:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:22:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:22:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:22:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:22:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:22:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:22:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:22:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:22:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:22:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:22:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:22:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:22:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:22:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:22:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:22:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:22:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:22:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:22:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:22:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:22:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:22:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:22:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:22:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:22:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:22:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:22:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:22:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:22:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:22:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:22:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=57] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:22:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=57] Local address: 172.21.15.161, port 51132 2022-07-03T11:22:01.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=57] Peer address: 172.21.15.101, port 22 2022-07-03T11:22:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=57] Login timeout expired 2022-07-03T11:22:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=57] Aborting connection 2022-07-03T11:22:01.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:22:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:22:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:22:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:22:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:22:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:22:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:22:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:22:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:22:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:22:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:22:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:22:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:22:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:22:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:22:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:22:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:22:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:22:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:22:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:22:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:22:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:22:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:22:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:22:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:22:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:22:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:22:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:22:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:22:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:22:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:22:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:22:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:22:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:22:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:22:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:22:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:22:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:22:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:22:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:22:02.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:22:02.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:22:02.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:22:02.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:22:02.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:22:02.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:22:02.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:22:02.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=57] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:22:02.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=57] Local address: 172.21.15.161, port 51132 2022-07-03T11:22:02.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=57] Peer address: 172.21.15.101, port 22 2022-07-03T11:22:02.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=57] Login timeout expired 2022-07-03T11:22:02.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=57] Aborting connection 2022-07-03T11:22:02.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:02 smithi161 ceph-mon[91579]: pgmap v1290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:22:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:22:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:22:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:22:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:22:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:22:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:22:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:22:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:22:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:22:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:22:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:22:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:22:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:22:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:22:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:22:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:22:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:22:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:22:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:22:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:22:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:22:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:22:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:22:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:22:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:22:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:22:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:22:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:22:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:22:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:22:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:22:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:22:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:22:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:22:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:22:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:22:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:22:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:22:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:22:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:22:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:22:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:22:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:22:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:22:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=57] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:22:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=57] Local address: 172.21.15.161, port 51132 2022-07-03T11:22:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=57] Peer address: 172.21.15.101, port 22 2022-07-03T11:22:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=57] Login timeout expired 2022-07-03T11:22:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=57] Aborting connection 2022-07-03T11:22:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:02 smithi188 ceph-mon[92425]: pgmap v1290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:04 smithi161 ceph-mon[91579]: pgmap v1291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:04 smithi188 ceph-mon[92425]: pgmap v1291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:06 smithi161 ceph-mon[91579]: pgmap v1292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:06 smithi188 ceph-mon[92425]: pgmap v1292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:08 smithi161 ceph-mon[91579]: pgmap v1293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:08 smithi188 ceph-mon[92425]: pgmap v1293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:10 smithi161 ceph-mon[91579]: pgmap v1294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:10 smithi188 ceph-mon[92425]: pgmap v1294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:12 smithi161 ceph-mon[91579]: pgmap v1295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:12 smithi188 ceph-mon[92425]: pgmap v1295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:14 smithi161 ceph-mon[91579]: pgmap v1296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:14 smithi188 ceph-mon[92425]: pgmap v1296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:16 smithi161 ceph-mon[91579]: pgmap v1297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:16 smithi188 ceph-mon[92425]: pgmap v1297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:18 smithi161 ceph-mon[91579]: pgmap v1298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:18 smithi188 ceph-mon[92425]: pgmap v1298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:20 smithi161 ceph-mon[91579]: pgmap v1299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:20 smithi188 ceph-mon[92425]: pgmap v1299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:22 smithi161 ceph-mon[91579]: pgmap v1300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:22 smithi188 ceph-mon[92425]: pgmap v1300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:24 smithi161 ceph-mon[91579]: pgmap v1301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:24 smithi188 ceph-mon[92425]: pgmap v1301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:26 smithi161 ceph-mon[91579]: pgmap v1302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:26 smithi188 ceph-mon[92425]: pgmap v1302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:28 smithi161 ceph-mon[91579]: pgmap v1303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:28 smithi188 ceph-mon[92425]: pgmap v1303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:30 smithi161 ceph-mon[91579]: pgmap v1304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:30 smithi188 ceph-mon[92425]: pgmap v1304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:32 smithi161 ceph-mon[91579]: pgmap v1305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:32 smithi188 ceph-mon[92425]: pgmap v1305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:34 smithi161 ceph-mon[91579]: pgmap v1306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:34 smithi188 ceph-mon[92425]: pgmap v1306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:36 smithi161 ceph-mon[91579]: pgmap v1307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:36 smithi188 ceph-mon[92425]: pgmap v1307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:38 smithi161 ceph-mon[91579]: pgmap v1308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:38 smithi188 ceph-mon[92425]: pgmap v1308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:40 smithi161 ceph-mon[91579]: pgmap v1309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:40 smithi188 ceph-mon[92425]: pgmap v1309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:42 smithi161 ceph-mon[91579]: pgmap v1310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:42 smithi188 ceph-mon[92425]: pgmap v1310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:44 smithi161 ceph-mon[91579]: pgmap v1311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:44 smithi188 ceph-mon[92425]: pgmap v1311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:46 smithi161 ceph-mon[91579]: pgmap v1312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:46 smithi188 ceph-mon[92425]: pgmap v1312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:48 smithi161 ceph-mon[91579]: pgmap v1313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:48 smithi188 ceph-mon[92425]: pgmap v1313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:50 smithi161 ceph-mon[91579]: pgmap v1314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:50 smithi188 ceph-mon[92425]: pgmap v1314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:52 smithi161 ceph-mon[91579]: pgmap v1315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:52 smithi188 ceph-mon[92425]: pgmap v1315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:54 smithi161 ceph-mon[91579]: pgmap v1316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:54 smithi188 ceph-mon[92425]: pgmap v1316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:22:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:56 smithi161 ceph-mon[91579]: pgmap v1317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:56 smithi188 ceph-mon[92425]: pgmap v1317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:22:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:22:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:22:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:22:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:22:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:22:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:22:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:22:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:58 smithi161 ceph-mon[91579]: pgmap v1318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:58 smithi188 ceph-mon[92425]: pgmap v1318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:22:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:22:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:00 smithi161 ceph-mon[91579]: pgmap v1319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:00 smithi188 ceph-mon[92425]: pgmap v1319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:23:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:23:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:23:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:23:01.631+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:23:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:23:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:23:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:23:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:23:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:23:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:23:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:23:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:23:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:23:01.650+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:23:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:23:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:23:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:23:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:23:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:23:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:23:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:23:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:23:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:23:01.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:23:01.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:23:01.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:23:01.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:23:01.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:23:01.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:23:01.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:23:01.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:23:01.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:23:01.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:23:01.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:23:01.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:23:01.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:23:01.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:23:01.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:23:01.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:23:01.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:23:01.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:23:01.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:23:01.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:23:01.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:23:01.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:23:01.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:23:01.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:23:01.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:23:01.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:23:01.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:23:01.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:23:01.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:23:01.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=207] Received exit status 0 2022-07-03T11:23:01.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=207] Received channel close 2022-07-03T11:23:01.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=207] Received exit status 0 2022-07-03T11:23:01.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=207] Received channel close 2022-07-03T11:23:01.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=207] Channel closed 2022-07-03T11:23:01.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=207] Channel closed 2022-07-03T11:23:01.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=59] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:23:01.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=59] Local address: 172.21.15.161, port 51136 2022-07-03T11:23:01.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=59] Peer address: 172.21.15.101, port 22 2022-07-03T11:23:01.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=59] Connection lost 2022-07-03T11:23:01.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=59] Aborting connection 2022-07-03T11:23:01.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:23:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:23:01.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:23:01.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:23:01.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:23:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: pgmap v1320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:23:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:23:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:23:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:23:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:23:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:23:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:23:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:23:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:23:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:23:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:23:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:23:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:23:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:23:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:23:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:23:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:23:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:23:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:23:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:23:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:23:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:23:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:23:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:23:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:23:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:23:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:23:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:23:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:23:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:23:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:23:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:23:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:23:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:23:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:23:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:23:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:23:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:23:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:23:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:23:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:23:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:23:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:23:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:23:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:23:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:23:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:23:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:23:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:23:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=207] Received exit status 0 2022-07-03T11:23:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=207] Received channel close 2022-07-03T11:23:02.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=207] Received exit status 0 2022-07-03T11:23:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=207] Received channel close 2022-07-03T11:23:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=207] Channel closed 2022-07-03T11:23:02.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=207] Channel closed 2022-07-03T11:23:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=59] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:23:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=59] Local address: 172.21.15.161, port 51136 2022-07-03T11:23:02.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=59] Peer address: 172.21.15.101, port 22 2022-07-03T11:23:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=59] Connection lost 2022-07-03T11:23:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=59] Aborting connection 2022-07-03T11:23:02.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:02.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: pgmap v1320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:23:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:23:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:23:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:23:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:23:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:23:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:23:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:23:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:23:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:23:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:23:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:23:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:23:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:23:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:23:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:23:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:23:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:23:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:23:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:23:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:23:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:23:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:23:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:23:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:23:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:23:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:23:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:23:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:23:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:23:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:23:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:23:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:23:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:23:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:23:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:23:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:23:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:23:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:23:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:23:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:23:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:23:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:23:03.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:23:03.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:23:03.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:23:03.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:23:03.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:23:03.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:23:03.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:23:03.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:23:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:23:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=207] Received exit status 0 2022-07-03T11:23:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=207] Received channel close 2022-07-03T11:23:03.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=207] Received exit status 0 2022-07-03T11:23:03.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=207] Received channel close 2022-07-03T11:23:03.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=207] Channel closed 2022-07-03T11:23:03.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=207] Channel closed 2022-07-03T11:23:03.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=59] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:23:03.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=59] Local address: 172.21.15.161, port 51136 2022-07-03T11:23:03.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=59] Peer address: 172.21.15.101, port 22 2022-07-03T11:23:03.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=59] Connection lost 2022-07-03T11:23:03.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=59] Aborting connection 2022-07-03T11:23:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:03.157 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: pgmap v1321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: pgmap v1321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:23:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:23:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:23:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:23:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:05 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:23:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:23:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:23:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:23:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:23:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:05 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:23:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:06 smithi161 ceph-mon[91579]: pgmap v1322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:06 smithi188 ceph-mon[92425]: pgmap v1322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:08 smithi161 ceph-mon[91579]: pgmap v1323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:08 smithi188 ceph-mon[92425]: pgmap v1323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:10 smithi161 ceph-mon[91579]: pgmap v1324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:10 smithi188 ceph-mon[92425]: pgmap v1324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:12 smithi161 ceph-mon[91579]: pgmap v1325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:12 smithi188 ceph-mon[92425]: pgmap v1325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:14 smithi161 ceph-mon[91579]: pgmap v1326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:14 smithi188 ceph-mon[92425]: pgmap v1326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:16 smithi161 ceph-mon[91579]: pgmap v1327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:16 smithi188 ceph-mon[92425]: pgmap v1327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:18 smithi161 ceph-mon[91579]: pgmap v1328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:18 smithi188 ceph-mon[92425]: pgmap v1328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:20 smithi161 ceph-mon[91579]: pgmap v1329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:20 smithi188 ceph-mon[92425]: pgmap v1329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:22 smithi161 ceph-mon[91579]: pgmap v1330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:22 smithi188 ceph-mon[92425]: pgmap v1330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:24 smithi161 ceph-mon[91579]: pgmap v1331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:24 smithi188 ceph-mon[92425]: pgmap v1331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:26 smithi161 ceph-mon[91579]: pgmap v1332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:26 smithi188 ceph-mon[92425]: pgmap v1332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:28 smithi161 ceph-mon[91579]: pgmap v1333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:28 smithi188 ceph-mon[92425]: pgmap v1333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:30 smithi161 ceph-mon[91579]: pgmap v1334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:30 smithi188 ceph-mon[92425]: pgmap v1334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:32 smithi161 ceph-mon[91579]: pgmap v1335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:32 smithi188 ceph-mon[92425]: pgmap v1335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:34 smithi161 ceph-mon[91579]: pgmap v1336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:34 smithi188 ceph-mon[92425]: pgmap v1336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:36 smithi161 ceph-mon[91579]: pgmap v1337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:36 smithi188 ceph-mon[92425]: pgmap v1337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:38 smithi161 ceph-mon[91579]: pgmap v1338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:38 smithi188 ceph-mon[92425]: pgmap v1338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:40 smithi161 ceph-mon[91579]: pgmap v1339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:40 smithi188 ceph-mon[92425]: pgmap v1339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:42 smithi161 ceph-mon[91579]: pgmap v1340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:42 smithi188 ceph-mon[92425]: pgmap v1340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:44 smithi161 ceph-mon[91579]: pgmap v1341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:44 smithi188 ceph-mon[92425]: pgmap v1341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:46 smithi161 ceph-mon[91579]: pgmap v1342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:46 smithi188 ceph-mon[92425]: pgmap v1342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:48 smithi161 ceph-mon[91579]: pgmap v1343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:48 smithi188 ceph-mon[92425]: pgmap v1343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:50 smithi161 ceph-mon[91579]: pgmap v1344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:50 smithi188 ceph-mon[92425]: pgmap v1344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:52 smithi161 ceph-mon[91579]: pgmap v1345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:52 smithi188 ceph-mon[92425]: pgmap v1345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:54 smithi161 ceph-mon[91579]: pgmap v1346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:54 smithi188 ceph-mon[92425]: pgmap v1346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:23:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:56 smithi161 ceph-mon[91579]: pgmap v1347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:56 smithi188 ceph-mon[92425]: pgmap v1347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:23:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:23:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:58 smithi161 ceph-mon[91579]: pgmap v1348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:58 smithi188 ceph-mon[92425]: pgmap v1348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:23:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:23:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:00 smithi161 ceph-mon[91579]: pgmap v1349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:00 smithi188 ceph-mon[92425]: pgmap v1349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:02 smithi161 ceph-mon[91579]: pgmap v1350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:02 smithi188 ceph-mon[92425]: pgmap v1350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:04 smithi161 ceph-mon[91579]: pgmap v1351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:04 smithi188 ceph-mon[92425]: pgmap v1351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:24:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:24:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:24:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:24:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:24:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:24:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:06 smithi161 ceph-mon[91579]: pgmap v1352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:06 smithi188 ceph-mon[92425]: pgmap v1352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:08 smithi161 ceph-mon[91579]: pgmap v1353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:08 smithi188 ceph-mon[92425]: pgmap v1353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:10 smithi188 ceph-mon[92425]: pgmap v1354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:10 smithi161 ceph-mon[91579]: pgmap v1354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:12 smithi161 ceph-mon[91579]: pgmap v1355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:12 smithi188 ceph-mon[92425]: pgmap v1355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:14 smithi161 ceph-mon[91579]: pgmap v1356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:14 smithi188 ceph-mon[92425]: pgmap v1356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:16 smithi161 ceph-mon[91579]: pgmap v1357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:16 smithi188 ceph-mon[92425]: pgmap v1357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:18 smithi161 ceph-mon[91579]: pgmap v1358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:18 smithi188 ceph-mon[92425]: pgmap v1358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:20 smithi188 ceph-mon[92425]: pgmap v1359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:20 smithi161 ceph-mon[91579]: pgmap v1359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:22 smithi161 ceph-mon[91579]: pgmap v1360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:22 smithi188 ceph-mon[92425]: pgmap v1360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:24 smithi161 ceph-mon[91579]: pgmap v1361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:24 smithi188 ceph-mon[92425]: pgmap v1361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:26 smithi161 ceph-mon[91579]: pgmap v1362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:26 smithi188 ceph-mon[92425]: pgmap v1362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:28 smithi161 ceph-mon[91579]: pgmap v1363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:28 smithi188 ceph-mon[92425]: pgmap v1363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:30 smithi188 ceph-mon[92425]: pgmap v1364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:30 smithi161 ceph-mon[91579]: pgmap v1364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:32 smithi161 ceph-mon[91579]: pgmap v1365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:32 smithi188 ceph-mon[92425]: pgmap v1365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-mon[91579]: pgmap v1366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:34 smithi188 ceph-mon[92425]: pgmap v1366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:24:34.916+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:24:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:24:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:24:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:24:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:24:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:24:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:24:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:24:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:24:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:24:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:24:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:24:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:24:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=61] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:24:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=61] Local address: 172.21.15.161, port 51164 2022-07-03T11:24:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=61] Peer address: 172.21.15.101, port 22 2022-07-03T11:24:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=61] Connection lost 2022-07-03T11:24:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=61] Aborting connection 2022-07-03T11:24:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:24:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:24:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:24:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:24:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:24:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:24:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:24:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:24:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:24:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:24:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:24:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:24:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:24:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:24:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:24:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:24:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:24:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:24:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:24:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:24:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:24:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:24:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:24:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:24:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:24:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:24:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:24:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:24:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:24:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:24:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:24:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:24:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:24:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:24:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:24:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:24:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:24:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:24:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:24:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:24:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:24:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:24:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:24:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:24:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:24:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:24:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:24:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=61] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:24:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=61] Local address: 172.21.15.161, port 51164 2022-07-03T11:24:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=61] Peer address: 172.21.15.101, port 22 2022-07-03T11:24:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=61] Connection lost 2022-07-03T11:24:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=61] Aborting connection 2022-07-03T11:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:24:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:24:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:24:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:24:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:24:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:24:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:24:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:24:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:24:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:24:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:24:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:24:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:24:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:24:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:24:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:24:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:24:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:24:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:24:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:24:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:24:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:24:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:24:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:24:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:24:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:24:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:24:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:24:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:24:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:24:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:24:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:24:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:24:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:24:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:24:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:24:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:24:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:24:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:24:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:24:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:24:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:24:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:24:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:24:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:24:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=61] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:24:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=61] Local address: 172.21.15.161, port 51164 2022-07-03T11:24:35.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=61] Peer address: 172.21.15.101, port 22 2022-07-03T11:24:35.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=61] Connection lost 2022-07-03T11:24:35.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=61] Aborting connection 2022-07-03T11:24:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:36 smithi161 ceph-mon[91579]: pgmap v1367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:36 smithi188 ceph-mon[92425]: pgmap v1367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:38 smithi161 ceph-mon[91579]: pgmap v1368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:38 smithi188 ceph-mon[92425]: pgmap v1368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:40 smithi188 ceph-mon[92425]: pgmap v1369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:40 smithi161 ceph-mon[91579]: pgmap v1369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:42 smithi161 ceph-mon[91579]: pgmap v1370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:42 smithi188 ceph-mon[92425]: pgmap v1370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:44 smithi161 ceph-mon[91579]: pgmap v1371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:44 smithi188 ceph-mon[92425]: pgmap v1371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:46 smithi161 ceph-mon[91579]: pgmap v1372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:46 smithi188 ceph-mon[92425]: pgmap v1372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:48 smithi161 ceph-mon[91579]: pgmap v1373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:48 smithi188 ceph-mon[92425]: pgmap v1373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:50 smithi188 ceph-mon[92425]: pgmap v1374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:50 smithi161 ceph-mon[91579]: pgmap v1374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:52 smithi161 ceph-mon[91579]: pgmap v1375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:52 smithi188 ceph-mon[92425]: pgmap v1375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:54 smithi161 ceph-mon[91579]: pgmap v1376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:54 smithi188 ceph-mon[92425]: pgmap v1376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:24:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:56 smithi161 ceph-mon[91579]: pgmap v1377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:56 smithi188 ceph-mon[92425]: pgmap v1377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:24:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:24:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:24:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:24:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:24:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:24:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:58 smithi161 ceph-mon[91579]: pgmap v1378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:58 smithi188 ceph-mon[92425]: pgmap v1378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:24:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:24:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:00 smithi161 ceph-mon[91579]: pgmap v1379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:00 smithi188 ceph-mon[92425]: pgmap v1379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:02 smithi161 ceph-mon[91579]: pgmap v1380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:02 smithi188 ceph-mon[92425]: pgmap v1380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:04 smithi161 ceph-mon[91579]: pgmap v1381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:04 smithi188 ceph-mon[92425]: pgmap v1381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:06 smithi161 ceph-mon[91579]: pgmap v1382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:06 smithi188 ceph-mon[92425]: pgmap v1382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:08 smithi188 ceph-mon[92425]: pgmap v1383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:08 smithi161 ceph-mon[91579]: pgmap v1383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:10 smithi188 ceph-mon[92425]: pgmap v1384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:10 smithi161 ceph-mon[91579]: pgmap v1384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:12 smithi161 ceph-mon[91579]: pgmap v1385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:12 smithi188 ceph-mon[92425]: pgmap v1385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:14 smithi161 ceph-mon[91579]: pgmap v1386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:14 smithi188 ceph-mon[92425]: pgmap v1386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:16 smithi161 ceph-mon[91579]: pgmap v1387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:16 smithi188 ceph-mon[92425]: pgmap v1387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:18 smithi188 ceph-mon[92425]: pgmap v1388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:18 smithi161 ceph-mon[91579]: pgmap v1388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:20 smithi188 ceph-mon[92425]: pgmap v1389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:20 smithi161 ceph-mon[91579]: pgmap v1389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:22 smithi161 ceph-mon[91579]: pgmap v1390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:22 smithi188 ceph-mon[92425]: pgmap v1390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:24 smithi161 ceph-mon[91579]: pgmap v1391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:24 smithi188 ceph-mon[92425]: pgmap v1391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:26 smithi161 ceph-mon[91579]: pgmap v1392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:26 smithi188 ceph-mon[92425]: pgmap v1392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:28 smithi188 ceph-mon[92425]: pgmap v1393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:28 smithi161 ceph-mon[91579]: pgmap v1393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:30 smithi188 ceph-mon[92425]: pgmap v1394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:30 smithi161 ceph-mon[91579]: pgmap v1394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:32 smithi161 ceph-mon[91579]: pgmap v1395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:32 smithi188 ceph-mon[92425]: pgmap v1395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:34 smithi161 ceph-mon[91579]: pgmap v1396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:34 smithi188 ceph-mon[92425]: pgmap v1396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:25:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:25:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:25:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:36 smithi161 ceph-mon[91579]: pgmap v1397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:36 smithi188 ceph-mon[92425]: pgmap v1397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:38 smithi188 ceph-mon[92425]: pgmap v1398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:38 smithi161 ceph-mon[91579]: pgmap v1398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:40 smithi188 ceph-mon[92425]: pgmap v1399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:40 smithi161 ceph-mon[91579]: pgmap v1399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:42 smithi161 ceph-mon[91579]: pgmap v1400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:42 smithi188 ceph-mon[92425]: pgmap v1400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:44 smithi161 ceph-mon[91579]: pgmap v1401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:44 smithi188 ceph-mon[92425]: pgmap v1401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:46 smithi161 ceph-mon[91579]: pgmap v1402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:46 smithi188 ceph-mon[92425]: pgmap v1402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:48 smithi188 ceph-mon[92425]: pgmap v1403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:48 smithi161 ceph-mon[91579]: pgmap v1403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:50 smithi188 ceph-mon[92425]: pgmap v1404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:50 smithi161 ceph-mon[91579]: pgmap v1404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:52 smithi161 ceph-mon[91579]: pgmap v1405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:52 smithi188 ceph-mon[92425]: pgmap v1405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:54 smithi161 ceph-mon[91579]: pgmap v1406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:54 smithi188 ceph-mon[92425]: pgmap v1406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:25:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:56 smithi161 ceph-mon[91579]: pgmap v1407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:56 smithi188 ceph-mon[92425]: pgmap v1407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:25:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:25:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:25:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:25:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:25:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:25:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:25:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:25:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:58 smithi161 ceph-mon[91579]: pgmap v1408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:58 smithi188 ceph-mon[92425]: pgmap v1408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:25:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:25:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:00 smithi161 ceph-mon[91579]: pgmap v1409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:00 smithi188 ceph-mon[92425]: pgmap v1409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:02 smithi161 ceph-mon[91579]: pgmap v1410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:02 smithi188 ceph-mon[92425]: pgmap v1410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-mon[91579]: pgmap v1411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:04 smithi188 ceph-mon[92425]: pgmap v1411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:26:04.956+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:26:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:26:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:26:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:26:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:26:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:26:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:26:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:26:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:26:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:26:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:26:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:26:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:26:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:26:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:26:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:26:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:26:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:26:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:26:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:26:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:26:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:26:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:26:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:26:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:26:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:26:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:26:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:26:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:26:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:26:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:26:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:26:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:26:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:26:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:26:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:26:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:26:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:26:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:26:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:26:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:26:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:26:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:26:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:26:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:26:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=63] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:26:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=63] Local address: 172.21.15.161, port 51168 2022-07-03T11:26:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=63] Peer address: 172.21.15.101, port 22 2022-07-03T11:26:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=220] Received exit status 0 2022-07-03T11:26:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=220] Received channel close 2022-07-03T11:26:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=220] Channel closed 2022-07-03T11:26:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=220] Received exit status 0 2022-07-03T11:26:05.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=220] Received channel close 2022-07-03T11:26:05.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=220] Channel closed 2022-07-03T11:26:05.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=221] Requesting new SSH session 2022-07-03T11:26:05.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=221] Requesting new SSH session 2022-07-03T11:26:05.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=221] Command: which python3 2022-07-03T11:26:05.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=221] Command: which python3 2022-07-03T11:26:05.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=221] Received exit status 0 2022-07-03T11:26:05.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=221] Received channel close 2022-07-03T11:26:05.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=221] Channel closed 2022-07-03T11:26:05.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=221] Received exit status 0 2022-07-03T11:26:05.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=221] Received channel close 2022-07-03T11:26:05.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=222] Requesting new SSH session 2022-07-03T11:26:05.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=221] Channel closed 2022-07-03T11:26:05.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=222] Requesting new SSH session 2022-07-03T11:26:05.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=222] Command: sudo true 2022-07-03T11:26:05.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=222] Command: sudo true 2022-07-03T11:26:05.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=222] Received exit status 0 2022-07-03T11:26:05.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=222] Received channel close 2022-07-03T11:26:05.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=222] Channel closed 2022-07-03T11:26:05.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=222] Received exit status 0 2022-07-03T11:26:05.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=222] Received channel close 2022-07-03T11:26:05.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=222] Channel closed 2022-07-03T11:26:05.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=223] Requesting new SSH session 2022-07-03T11:26:05.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=223] Requesting new SSH session 2022-07-03T11:26:05.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=223] Received exit status 0 2022-07-03T11:26:05.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=223] Received channel close 2022-07-03T11:26:05.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=223] Channel closed 2022-07-03T11:26:05.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=223] Received exit status 0 2022-07-03T11:26:05.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=223] Received channel close 2022-07-03T11:26:05.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=223] Channel closed 2022-07-03T11:26:05.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=63] Login timeout expired 2022-07-03T11:26:05.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=63] Aborting connection 2022-07-03T11:26:05.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:26:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:26:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:26:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:26:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:26:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:26:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:26:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:26:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:26:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:26:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:26:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:26:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:26:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:26:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:26:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:26:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:26:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:26:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:26:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:26:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:26:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:26:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:26:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:26:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:26:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:26:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:26:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:26:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:26:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:26:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:26:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:26:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:26:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:26:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:26:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:26:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:26:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:26:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:26:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:26:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:26:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:26:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:26:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:26:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:26:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:26:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:26:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=63] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:26:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=63] Local address: 172.21.15.161, port 51168 2022-07-03T11:26:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=63] Peer address: 172.21.15.101, port 22 2022-07-03T11:26:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=220] Received exit status 0 2022-07-03T11:26:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=220] Received channel close 2022-07-03T11:26:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=220] Channel closed 2022-07-03T11:26:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=220] Received exit status 0 2022-07-03T11:26:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=220] Received channel close 2022-07-03T11:26:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=220] Channel closed 2022-07-03T11:26:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=221] Requesting new SSH session 2022-07-03T11:26:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=221] Requesting new SSH session 2022-07-03T11:26:05.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=221] Command: which python3 2022-07-03T11:26:05.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=221] Command: which python3 2022-07-03T11:26:05.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=221] Received exit status 0 2022-07-03T11:26:05.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=221] Received channel close 2022-07-03T11:26:05.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=221] Channel closed 2022-07-03T11:26:05.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=221] Received exit status 0 2022-07-03T11:26:05.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=221] Received channel close 2022-07-03T11:26:05.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=222] Requesting new SSH session 2022-07-03T11:26:05.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=221] Channel closed 2022-07-03T11:26:05.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=222] Requesting new SSH session 2022-07-03T11:26:05.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=222] Command: sudo true 2022-07-03T11:26:05.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=222] Command: sudo true 2022-07-03T11:26:05.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=222] Received exit status 0 2022-07-03T11:26:05.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=222] Received channel close 2022-07-03T11:26:05.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=222] Channel closed 2022-07-03T11:26:05.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=222] Received exit status 0 2022-07-03T11:26:05.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=222] Received channel close 2022-07-03T11:26:05.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=222] Channel closed 2022-07-03T11:26:05.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=223] Requesting new SSH session 2022-07-03T11:26:05.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=223] Requesting new SSH session 2022-07-03T11:26:05.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=223] Received exit status 0 2022-07-03T11:26:05.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=223] Received channel close 2022-07-03T11:26:05.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=223] Channel closed 2022-07-03T11:26:05.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=223] Received exit status 0 2022-07-03T11:26:05.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=223] Received channel close 2022-07-03T11:26:05.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=223] Channel closed 2022-07-03T11:26:05.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=63] Login timeout expired 2022-07-03T11:26:05.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=63] Aborting connection 2022-07-03T11:26:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:26:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:26:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:26:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:26:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:26:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:26:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:26:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:26:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:26:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:26:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:26:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:26:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:26:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:26:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:26:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:26:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:26:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:26:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:26:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:26:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:26:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:26:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:26:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:26:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:26:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:26:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:26:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:26:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:26:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:26:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:26:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:26:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:26:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:26:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:26:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:26:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:26:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:26:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:26:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:26:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:26:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:26:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:26:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:26:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:26:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=63] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:26:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=63] Local address: 172.21.15.161, port 51168 2022-07-03T11:26:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=63] Peer address: 172.21.15.101, port 22 2022-07-03T11:26:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=220] Received exit status 0 2022-07-03T11:26:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=220] Received channel close 2022-07-03T11:26:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=220] Channel closed 2022-07-03T11:26:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=220] Received exit status 0 2022-07-03T11:26:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=220] Received channel close 2022-07-03T11:26:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=220] Channel closed 2022-07-03T11:26:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=221] Requesting new SSH session 2022-07-03T11:26:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=221] Requesting new SSH session 2022-07-03T11:26:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=221] Command: which python3 2022-07-03T11:26:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=221] Command: which python3 2022-07-03T11:26:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=221] Received exit status 0 2022-07-03T11:26:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=221] Received channel close 2022-07-03T11:26:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=221] Channel closed 2022-07-03T11:26:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=221] Received exit status 0 2022-07-03T11:26:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=221] Received channel close 2022-07-03T11:26:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=222] Requesting new SSH session 2022-07-03T11:26:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=221] Channel closed 2022-07-03T11:26:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=222] Requesting new SSH session 2022-07-03T11:26:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=222] Command: sudo true 2022-07-03T11:26:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=222] Command: sudo true 2022-07-03T11:26:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=222] Received exit status 0 2022-07-03T11:26:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=222] Received channel close 2022-07-03T11:26:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=222] Channel closed 2022-07-03T11:26:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=222] Received exit status 0 2022-07-03T11:26:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=222] Received channel close 2022-07-03T11:26:05.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=222] Channel closed 2022-07-03T11:26:05.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=223] Requesting new SSH session 2022-07-03T11:26:05.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=223] Requesting new SSH session 2022-07-03T11:26:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=223] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:26:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=223] Received exit status 0 2022-07-03T11:26:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=223] Received channel close 2022-07-03T11:26:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=223] Channel closed 2022-07-03T11:26:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=223] Received exit status 0 2022-07-03T11:26:05.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=223] Received channel close 2022-07-03T11:26:05.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=223] Channel closed 2022-07-03T11:26:05.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=63] Login timeout expired 2022-07-03T11:26:05.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=63] Aborting connection 2022-07-03T11:26:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:06 smithi161 ceph-mon[91579]: pgmap v1412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:06 smithi188 ceph-mon[92425]: pgmap v1412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:08 smithi188 ceph-mon[92425]: pgmap v1413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:08 smithi161 ceph-mon[91579]: pgmap v1413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:10 smithi188 ceph-mon[92425]: pgmap v1414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:10 smithi161 ceph-mon[91579]: pgmap v1414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:12 smithi161 ceph-mon[91579]: pgmap v1415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:12 smithi188 ceph-mon[92425]: pgmap v1415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:14 smithi161 ceph-mon[91579]: pgmap v1416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:14 smithi188 ceph-mon[92425]: pgmap v1416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:16 smithi161 ceph-mon[91579]: pgmap v1417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:16 smithi188 ceph-mon[92425]: pgmap v1417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:18 smithi188 ceph-mon[92425]: pgmap v1418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:18 smithi161 ceph-mon[91579]: pgmap v1418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:20 smithi188 ceph-mon[92425]: pgmap v1419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:20 smithi161 ceph-mon[91579]: pgmap v1419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:22 smithi161 ceph-mon[91579]: pgmap v1420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:22 smithi188 ceph-mon[92425]: pgmap v1420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:24 smithi161 ceph-mon[91579]: pgmap v1421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:24 smithi188 ceph-mon[92425]: pgmap v1421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:26 smithi161 ceph-mon[91579]: pgmap v1422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:26 smithi188 ceph-mon[92425]: pgmap v1422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:28 smithi188 ceph-mon[92425]: pgmap v1423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:28 smithi161 ceph-mon[91579]: pgmap v1423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:30 smithi161 ceph-mon[91579]: pgmap v1424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:30 smithi188 ceph-mon[92425]: pgmap v1424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:32 smithi161 ceph-mon[91579]: pgmap v1425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:32 smithi188 ceph-mon[92425]: pgmap v1425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:34 smithi161 ceph-mon[91579]: pgmap v1426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:34 smithi188 ceph-mon[92425]: pgmap v1426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:36 smithi161 ceph-mon[91579]: pgmap v1427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:36 smithi188 ceph-mon[92425]: pgmap v1427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:38 smithi161 ceph-mon[91579]: pgmap v1428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:38 smithi188 ceph-mon[92425]: pgmap v1428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:40 smithi161 ceph-mon[91579]: pgmap v1429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:40 smithi188 ceph-mon[92425]: pgmap v1429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:42 smithi161 ceph-mon[91579]: pgmap v1430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:42 smithi188 ceph-mon[92425]: pgmap v1430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:44 smithi161 ceph-mon[91579]: pgmap v1431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:44 smithi188 ceph-mon[92425]: pgmap v1431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:46 smithi161 ceph-mon[91579]: pgmap v1432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:46 smithi188 ceph-mon[92425]: pgmap v1432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:48 smithi188 ceph-mon[92425]: pgmap v1433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:48 smithi161 ceph-mon[91579]: pgmap v1433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:50 smithi161 ceph-mon[91579]: pgmap v1434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:50 smithi188 ceph-mon[92425]: pgmap v1434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:52 smithi161 ceph-mon[91579]: pgmap v1435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:52 smithi188 ceph-mon[92425]: pgmap v1435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:54 smithi161 ceph-mon[91579]: pgmap v1436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:54 smithi188 ceph-mon[92425]: pgmap v1436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:26:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:56 smithi161 ceph-mon[91579]: pgmap v1437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:56 smithi188 ceph-mon[92425]: pgmap v1437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:26:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:26:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:26:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:26:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:26:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:26:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:26:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:26:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:58 smithi161 ceph-mon[91579]: pgmap v1438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:58 smithi188 ceph-mon[92425]: pgmap v1438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:26:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:26:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:00 smithi161 ceph-mon[91579]: pgmap v1439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:00 smithi188 ceph-mon[92425]: pgmap v1439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:02 smithi161 ceph-mon[91579]: pgmap v1440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:02 smithi188 ceph-mon[92425]: pgmap v1440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:04 smithi161 ceph-mon[91579]: pgmap v1441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:04 smithi188 ceph-mon[92425]: pgmap v1441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:27:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:27:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:27:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:27:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:27:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:27:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:06 smithi161 ceph-mon[91579]: pgmap v1442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:06 smithi188 ceph-mon[92425]: pgmap v1442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:08 smithi161 ceph-mon[91579]: pgmap v1443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:08 smithi188 ceph-mon[92425]: pgmap v1443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:10 smithi161 ceph-mon[91579]: pgmap v1444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:10 smithi188 ceph-mon[92425]: pgmap v1444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:12 smithi161 ceph-mon[91579]: pgmap v1445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:12 smithi188 ceph-mon[92425]: pgmap v1445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:14 smithi161 ceph-mon[91579]: pgmap v1446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:14 smithi188 ceph-mon[92425]: pgmap v1446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:16 smithi161 ceph-mon[91579]: pgmap v1447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:16 smithi188 ceph-mon[92425]: pgmap v1447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:18 smithi161 ceph-mon[91579]: pgmap v1448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:18 smithi188 ceph-mon[92425]: pgmap v1448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:20 smithi161 ceph-mon[91579]: pgmap v1449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:20 smithi188 ceph-mon[92425]: pgmap v1449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:22 smithi161 ceph-mon[91579]: pgmap v1450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:22 smithi188 ceph-mon[92425]: pgmap v1450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:24 smithi161 ceph-mon[91579]: pgmap v1451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:24 smithi188 ceph-mon[92425]: pgmap v1451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:26 smithi161 ceph-mon[91579]: pgmap v1452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:26 smithi188 ceph-mon[92425]: pgmap v1452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:28 smithi161 ceph-mon[91579]: pgmap v1453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:28 smithi188 ceph-mon[92425]: pgmap v1453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:30 smithi161 ceph-mon[91579]: pgmap v1454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:30 smithi188 ceph-mon[92425]: pgmap v1454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:32 smithi161 ceph-mon[91579]: pgmap v1455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:32 smithi188 ceph-mon[92425]: pgmap v1455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:34 smithi161 ceph-mon[91579]: pgmap v1456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:34 smithi188 ceph-mon[92425]: pgmap v1456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:27:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:27:35.405+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:27:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:27:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:27:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:27:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:27:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:27:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:27:35.406+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:27:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:27:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:27:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:27:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:27:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:27:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:27:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:27:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:27:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:27:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:27:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:27:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:27:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:27:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:27:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:27:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:27:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:27:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:27:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:27:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:27:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:27:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:27:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:27:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:27:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:27:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:27:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:27:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:27:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:27:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:27:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:27:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:27:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:27:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:27:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:27:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:27:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:27:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:27:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:36 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:27:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:27:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:27:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:27:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:27:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:27:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:27:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:27:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:27:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:27:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:27:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:27:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:27:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:27:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:27:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:27:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:27:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:27:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:27:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:27:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:27:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:27:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:27:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:27:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:27:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:27:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:27:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:27:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:27:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:27:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:27:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:27:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:27:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:27:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:27:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:27:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:27:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:27:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:27:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:27:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:36.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:36 smithi161 ceph-mon[91579]: pgmap v1457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:36 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:27:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:27:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:27:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:27:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:27:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:27:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:27:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:27:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:27:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:27:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:27:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:27:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:27:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:27:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:27:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:27:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:27:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:27:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:27:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:27:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:27:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:27:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:27:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:27:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:27:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:27:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:27:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:27:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:27:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:27:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:27:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:27:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:27:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:27:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:27:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:27:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:27:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:27:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:27:36.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:27:36.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:27:36.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:27:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:27:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:27:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:27:36.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:27:36.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:27:36.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:27:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:27:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:27:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:36 smithi188 ceph-mon[92425]: pgmap v1457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:38 smithi161 ceph-mon[91579]: pgmap v1458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:38 smithi188 ceph-mon[92425]: pgmap v1458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:40 smithi161 ceph-mon[91579]: pgmap v1459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:40 smithi188 ceph-mon[92425]: pgmap v1459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:42 smithi161 ceph-mon[91579]: pgmap v1460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:42 smithi188 ceph-mon[92425]: pgmap v1460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:44 smithi161 ceph-mon[91579]: pgmap v1461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:44 smithi188 ceph-mon[92425]: pgmap v1461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:46 smithi161 ceph-mon[91579]: pgmap v1462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:46 smithi188 ceph-mon[92425]: pgmap v1462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:48 smithi161 ceph-mon[91579]: pgmap v1463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:48 smithi188 ceph-mon[92425]: pgmap v1463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:50 smithi161 ceph-mon[91579]: pgmap v1464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:50 smithi188 ceph-mon[92425]: pgmap v1464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:52 smithi161 ceph-mon[91579]: pgmap v1465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:52 smithi188 ceph-mon[92425]: pgmap v1465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:54 smithi161 ceph-mon[91579]: pgmap v1466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:54 smithi188 ceph-mon[92425]: pgmap v1466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:27:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:56 smithi161 ceph-mon[91579]: pgmap v1467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:56 smithi188 ceph-mon[92425]: pgmap v1467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:27:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:27:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:27:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:27:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:27:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:27:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:27:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:27:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:58 smithi161 ceph-mon[91579]: pgmap v1468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:58 smithi188 ceph-mon[92425]: pgmap v1468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:27:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:27:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:00 smithi161 ceph-mon[91579]: pgmap v1469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:00 smithi188 ceph-mon[92425]: pgmap v1469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:02 smithi161 ceph-mon[91579]: pgmap v1470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:02 smithi188 ceph-mon[92425]: pgmap v1470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:04 smithi161 ceph-mon[91579]: pgmap v1471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:04 smithi188 ceph-mon[92425]: pgmap v1471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:06 smithi161 ceph-mon[91579]: pgmap v1472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:06 smithi188 ceph-mon[92425]: pgmap v1472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:08 smithi161 ceph-mon[91579]: pgmap v1473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:08 smithi188 ceph-mon[92425]: pgmap v1473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:10 smithi161 ceph-mon[91579]: pgmap v1474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:10 smithi188 ceph-mon[92425]: pgmap v1474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:12 smithi161 ceph-mon[91579]: pgmap v1475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:12 smithi188 ceph-mon[92425]: pgmap v1475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:14 smithi161 ceph-mon[91579]: pgmap v1476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:14 smithi188 ceph-mon[92425]: pgmap v1476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:16 smithi161 ceph-mon[91579]: pgmap v1477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:16 smithi188 ceph-mon[92425]: pgmap v1477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:18 smithi161 ceph-mon[91579]: pgmap v1478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:18 smithi188 ceph-mon[92425]: pgmap v1478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:20 smithi161 ceph-mon[91579]: pgmap v1479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:20 smithi188 ceph-mon[92425]: pgmap v1479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:22 smithi161 ceph-mon[91579]: pgmap v1480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:22 smithi188 ceph-mon[92425]: pgmap v1480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:24 smithi161 ceph-mon[91579]: pgmap v1481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:24 smithi188 ceph-mon[92425]: pgmap v1481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:26 smithi161 ceph-mon[91579]: pgmap v1482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:26 smithi188 ceph-mon[92425]: pgmap v1482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:28 smithi161 ceph-mon[91579]: pgmap v1483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:28 smithi188 ceph-mon[92425]: pgmap v1483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:30 smithi161 ceph-mon[91579]: pgmap v1484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:30 smithi188 ceph-mon[92425]: pgmap v1484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:32 smithi161 ceph-mon[91579]: pgmap v1485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:32 smithi188 ceph-mon[92425]: pgmap v1485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:34 smithi161 ceph-mon[91579]: pgmap v1486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:34 smithi188 ceph-mon[92425]: pgmap v1486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:35.834 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:28:35.835 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:28:35.835 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:28:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:28:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:28:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:28:36.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:28:35.832+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:28:36.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:28:36.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:28:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:28:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:28:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:28:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:28:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:28:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:28:36.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:28:36.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:28:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:28:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:28:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:28:36.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:28:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:28:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:36 smithi161 ceph-mon[91579]: pgmap v1487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:36 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:28:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:28:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:28:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:28:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:28:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:28:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:28:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:28:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:28:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:28:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:28:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:28:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:28:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:28:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:28:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:28:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:36 smithi188 ceph-mon[92425]: pgmap v1487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:36 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:28:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:28:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:28:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:28:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:28:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:28:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:28:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:28:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:28:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:28:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:28:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:28:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:28:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:28:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:28:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:28:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:38 smithi161 ceph-mon[91579]: pgmap v1488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:38 smithi188 ceph-mon[92425]: pgmap v1488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:40 smithi161 ceph-mon[91579]: pgmap v1489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:40 smithi188 ceph-mon[92425]: pgmap v1489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:42 smithi161 ceph-mon[91579]: pgmap v1490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:42 smithi188 ceph-mon[92425]: pgmap v1490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:44 smithi161 ceph-mon[91579]: pgmap v1491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:44 smithi188 ceph-mon[92425]: pgmap v1491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:46.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:46 smithi161 ceph-mon[91579]: pgmap v1492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:46 smithi188 ceph-mon[92425]: pgmap v1492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:48 smithi161 ceph-mon[91579]: pgmap v1493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:48 smithi188 ceph-mon[92425]: pgmap v1493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:50 smithi161 ceph-mon[91579]: pgmap v1494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:50 smithi188 ceph-mon[92425]: pgmap v1494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:52 smithi161 ceph-mon[91579]: pgmap v1495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:52 smithi188 ceph-mon[92425]: pgmap v1495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:54 smithi161 ceph-mon[91579]: pgmap v1496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:54 smithi188 ceph-mon[92425]: pgmap v1496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:28:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:56 smithi161 ceph-mon[91579]: pgmap v1497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:56 smithi188 ceph-mon[92425]: pgmap v1497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:28:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:28:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:28:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:28:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:28:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:28:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:28:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:28:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:58 smithi161 ceph-mon[91579]: pgmap v1498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:58 smithi188 ceph-mon[92425]: pgmap v1498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:28:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:28:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:00 smithi161 ceph-mon[91579]: pgmap v1499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:00 smithi188 ceph-mon[92425]: pgmap v1499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:02 smithi161 ceph-mon[91579]: pgmap v1500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:02 smithi188 ceph-mon[92425]: pgmap v1500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:04 smithi161 ceph-mon[91579]: pgmap v1501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:04 smithi188 ceph-mon[92425]: pgmap v1501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:06.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:29:05.849+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:29:06.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:29:06.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:29:06.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:29:06.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:29:06.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:29:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:29:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:29:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:29:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:29:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:29:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:29:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:29:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:29:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:29:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:29:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:29:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:29:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:29:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:29:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:29:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:29:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:29:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:29:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:29:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:29:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:29:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:29:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:29:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:29:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:29:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:29:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:29:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:29:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:29:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:29:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:29:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:29:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:29:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:29:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:29:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:29:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:29:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:29:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:29:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=67] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:29:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=67] Local address: 172.21.15.161, port 51176 2022-07-03T11:29:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=67] Peer address: 172.21.15.101, port 22 2022-07-03T11:29:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=231] Received exit status 0 2022-07-03T11:29:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=231] Received channel close 2022-07-03T11:29:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=231] Channel closed 2022-07-03T11:29:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=231] Received exit status 0 2022-07-03T11:29:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=231] Received channel close 2022-07-03T11:29:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=231] Channel closed 2022-07-03T11:29:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=67] Login timeout expired 2022-07-03T11:29:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=67] Aborting connection 2022-07-03T11:29:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:29:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:29:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:06 smithi161 ceph-mon[91579]: pgmap v1502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:29:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:29:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:29:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:29:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:29:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:29:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:29:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:29:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:29:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:29:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:29:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:29:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:29:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:29:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:29:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:29:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:29:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:29:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:29:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:29:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:29:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:29:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:29:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:29:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:29:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:29:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:29:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:29:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:29:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:29:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:29:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:29:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:29:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:29:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:29:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:29:06.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:29:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:29:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:29:06.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:29:06.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:29:06.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:29:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:29:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:29:06.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:29:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=67] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:29:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=67] Local address: 172.21.15.161, port 51176 2022-07-03T11:29:06.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=67] Peer address: 172.21.15.101, port 22 2022-07-03T11:29:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=231] Received exit status 0 2022-07-03T11:29:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=231] Received channel close 2022-07-03T11:29:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=231] Channel closed 2022-07-03T11:29:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=231] Received exit status 0 2022-07-03T11:29:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=231] Received channel close 2022-07-03T11:29:06.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=231] Channel closed 2022-07-03T11:29:06.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=67] Login timeout expired 2022-07-03T11:29:06.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=67] Aborting connection 2022-07-03T11:29:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:06 smithi188 ceph-mon[92425]: pgmap v1502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:29:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:29:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:29:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:29:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:29:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:29:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:29:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:29:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:29:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:29:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:29:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:29:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:29:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:29:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:29:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:29:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:29:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:29:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:29:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:29:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:29:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:29:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:29:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:29:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:29:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:29:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:29:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:29:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:29:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:29:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:29:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:29:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:29:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:29:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:29:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:29:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:29:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:29:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:29:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:29:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:29:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:29:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:29:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:29:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:29:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=67] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:29:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=67] Local address: 172.21.15.161, port 51176 2022-07-03T11:29:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=67] Peer address: 172.21.15.101, port 22 2022-07-03T11:29:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=231] Received exit status 0 2022-07-03T11:29:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=231] Received channel close 2022-07-03T11:29:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=231] Channel closed 2022-07-03T11:29:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=231] Received exit status 0 2022-07-03T11:29:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=231] Received channel close 2022-07-03T11:29:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=231] Channel closed 2022-07-03T11:29:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=67] Login timeout expired 2022-07-03T11:29:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=67] Aborting connection 2022-07-03T11:29:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:08 smithi161 ceph-mon[91579]: pgmap v1503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:08 smithi188 ceph-mon[92425]: pgmap v1503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:10 smithi161 ceph-mon[91579]: pgmap v1504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:10 smithi188 ceph-mon[92425]: pgmap v1504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:12 smithi161 ceph-mon[91579]: pgmap v1505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:12 smithi188 ceph-mon[92425]: pgmap v1505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:14 smithi161 ceph-mon[91579]: pgmap v1506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:14 smithi188 ceph-mon[92425]: pgmap v1506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:16 smithi161 ceph-mon[91579]: pgmap v1507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:16 smithi188 ceph-mon[92425]: pgmap v1507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:18 smithi161 ceph-mon[91579]: pgmap v1508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:18 smithi188 ceph-mon[92425]: pgmap v1508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:20 smithi161 ceph-mon[91579]: pgmap v1509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:20 smithi188 ceph-mon[92425]: pgmap v1509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:22 smithi161 ceph-mon[91579]: pgmap v1510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:22 smithi188 ceph-mon[92425]: pgmap v1510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:24 smithi161 ceph-mon[91579]: pgmap v1511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:24 smithi188 ceph-mon[92425]: pgmap v1511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:26 smithi161 ceph-mon[91579]: pgmap v1512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:26 smithi188 ceph-mon[92425]: pgmap v1512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:28 smithi161 ceph-mon[91579]: pgmap v1513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:28 smithi188 ceph-mon[92425]: pgmap v1513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:30 smithi161 ceph-mon[91579]: pgmap v1514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:30 smithi188 ceph-mon[92425]: pgmap v1514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:32 smithi161 ceph-mon[91579]: pgmap v1515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:32 smithi188 ceph-mon[92425]: pgmap v1515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:34 smithi161 ceph-mon[91579]: pgmap v1516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:34 smithi188 ceph-mon[92425]: pgmap v1516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:36 smithi161 ceph-mon[91579]: pgmap v1517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:36 smithi188 ceph-mon[92425]: pgmap v1517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:38 smithi161 ceph-mon[91579]: pgmap v1518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:38 smithi188 ceph-mon[92425]: pgmap v1518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:40 smithi161 ceph-mon[91579]: pgmap v1519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:40 smithi188 ceph-mon[92425]: pgmap v1519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:42 smithi161 ceph-mon[91579]: pgmap v1520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:42 smithi188 ceph-mon[92425]: pgmap v1520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:44 smithi161 ceph-mon[91579]: pgmap v1521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:44 smithi188 ceph-mon[92425]: pgmap v1521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:46 smithi161 ceph-mon[91579]: pgmap v1522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:46 smithi188 ceph-mon[92425]: pgmap v1522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:48 smithi161 ceph-mon[91579]: pgmap v1523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:48 smithi188 ceph-mon[92425]: pgmap v1523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:50 smithi161 ceph-mon[91579]: pgmap v1524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:50 smithi188 ceph-mon[92425]: pgmap v1524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:52 smithi161 ceph-mon[91579]: pgmap v1525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:52 smithi188 ceph-mon[92425]: pgmap v1525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:54 smithi161 ceph-mon[91579]: pgmap v1526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:54 smithi188 ceph-mon[92425]: pgmap v1526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:29:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:56 smithi161 ceph-mon[91579]: pgmap v1527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:56 smithi188 ceph-mon[92425]: pgmap v1527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:29:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:29:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:29:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:29:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:29:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:29:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:29:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:29:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:58 smithi161 ceph-mon[91579]: pgmap v1528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:58 smithi188 ceph-mon[92425]: pgmap v1528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:29:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:29:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:00 smithi161 ceph-mon[91579]: pgmap v1529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:30:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:00 smithi188 ceph-mon[92425]: pgmap v1529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:30:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:02 smithi161 ceph-mon[91579]: pgmap v1530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:02 smithi188 ceph-mon[92425]: pgmap v1530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:04 smithi188 ceph-mon[92425]: pgmap v1531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:04 smithi161 ceph-mon[91579]: pgmap v1531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:06 smithi188 ceph-mon[92425]: pgmap v1532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:30:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:30:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:30:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:06 smithi161 ceph-mon[91579]: pgmap v1532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:30:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:30:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:30:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:08 smithi161 ceph-mon[91579]: pgmap v1533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:08 smithi188 ceph-mon[92425]: pgmap v1533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:10 smithi161 ceph-mon[91579]: pgmap v1534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:10 smithi188 ceph-mon[92425]: pgmap v1534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:12 smithi161 ceph-mon[91579]: pgmap v1535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:12 smithi188 ceph-mon[92425]: pgmap v1535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:14 smithi161 ceph-mon[91579]: pgmap v1536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:14 smithi188 ceph-mon[92425]: pgmap v1536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:16 smithi161 ceph-mon[91579]: pgmap v1537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:16 smithi188 ceph-mon[92425]: pgmap v1537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:18 smithi161 ceph-mon[91579]: pgmap v1538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:18 smithi188 ceph-mon[92425]: pgmap v1538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:20 smithi161 ceph-mon[91579]: pgmap v1539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:20 smithi188 ceph-mon[92425]: pgmap v1539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:22 smithi161 ceph-mon[91579]: pgmap v1540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:22 smithi188 ceph-mon[92425]: pgmap v1540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:24 smithi161 ceph-mon[91579]: pgmap v1541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:24 smithi188 ceph-mon[92425]: pgmap v1541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:26 smithi161 ceph-mon[91579]: pgmap v1542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:26 smithi188 ceph-mon[92425]: pgmap v1542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:28 smithi161 ceph-mon[91579]: pgmap v1543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:28 smithi188 ceph-mon[92425]: pgmap v1543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:30 smithi161 ceph-mon[91579]: pgmap v1544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:30 smithi188 ceph-mon[92425]: pgmap v1544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:32 smithi161 ceph-mon[91579]: pgmap v1545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:32 smithi188 ceph-mon[92425]: pgmap v1545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:34 smithi161 ceph-mon[91579]: pgmap v1546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:34 smithi188 ceph-mon[92425]: pgmap v1546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:36.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:30:36.290+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:36.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:30:36.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:36.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:30:36.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:36.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:36.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:36.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:30:36.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:36.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:30:36.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:36.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:30:36.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:36.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:30:36.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:36.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:30:36.291+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:30:36.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:30:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:30:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:36.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:36.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:36.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:30:36.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:36.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:30:36.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:36.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:30:36.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:36.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:30:36.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:30:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:30:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:30:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:30:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:30:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:30:36.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:30:36.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:30:36.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:30:36.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:30:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:30:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:30:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:30:36.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:30:36.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:30:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:30:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:30:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:30:36.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:30:36.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:30:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:30:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:30:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:30:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:30:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:30:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:30:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:30:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:30:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:30:36.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:30:36.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-mon[91579]: pgmap v1547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:30:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:36 smithi188 ceph-mon[92425]: pgmap v1547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:30:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:37 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:30:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:30:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:30:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:30:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:30:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:30:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:30:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:30:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:30:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:30:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:30:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:30:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:30:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:30:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:30:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:30:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:30:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:30:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:30:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:30:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:30:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:30:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:30:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:30:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:30:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:30:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:30:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:30:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:30:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:30:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:30:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:30:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:30:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:30:37.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:30:37.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:30:37.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:30:37.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:30:37.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:30:37.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:30:37.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:37 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:30:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:30:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:30:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:30:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:30:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:30:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:30:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:30:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:30:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:30:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:30:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:30:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:30:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:30:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:30:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:30:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:30:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:30:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:30:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:30:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:30:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:30:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:30:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:30:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:30:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:30:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:30:37.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:30:37.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:30:37.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:30:37.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:30:37.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:30:37.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:30:37.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:30:37.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:30:37.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:30:37.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:30:37.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:30:37.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:30:37.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:30:37.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:30:37.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:30:37.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:30:37.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:30:37.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:30:37.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:30:37.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:30:37.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T11:30:37.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:30:37.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:30:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:38 smithi188 ceph-mon[92425]: pgmap v1548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:38 smithi161 ceph-mon[91579]: pgmap v1548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:40 smithi188 ceph-mon[92425]: pgmap v1549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:40 smithi161 ceph-mon[91579]: pgmap v1549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:42 smithi188 ceph-mon[92425]: pgmap v1550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:42 smithi161 ceph-mon[91579]: pgmap v1550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:44 smithi188 ceph-mon[92425]: pgmap v1551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:44 smithi161 ceph-mon[91579]: pgmap v1551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:46 smithi188 ceph-mon[92425]: pgmap v1552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:46 smithi161 ceph-mon[91579]: pgmap v1552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:48 smithi161 ceph-mon[91579]: pgmap v1553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:48 smithi188 ceph-mon[92425]: pgmap v1553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:50 smithi161 ceph-mon[91579]: pgmap v1554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:50 smithi188 ceph-mon[92425]: pgmap v1554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:52 smithi161 ceph-mon[91579]: pgmap v1555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:52 smithi188 ceph-mon[92425]: pgmap v1555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:54 smithi161 ceph-mon[91579]: pgmap v1556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:54 smithi188 ceph-mon[92425]: pgmap v1556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:30:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:30:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:56 smithi188 ceph-mon[92425]: pgmap v1557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:56 smithi161 ceph-mon[91579]: pgmap v1557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:30:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:30:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:30:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:30:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:30:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:30:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:30:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:30:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:58 smithi188 ceph-mon[92425]: pgmap v1558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:58 smithi161 ceph-mon[91579]: pgmap v1558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:30:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:00 smithi188 ceph-mon[92425]: pgmap v1559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:00 smithi161 ceph-mon[91579]: pgmap v1559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:02 smithi188 ceph-mon[92425]: pgmap v1560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:02 smithi161 ceph-mon[91579]: pgmap v1560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:04 smithi188 ceph-mon[92425]: pgmap v1561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:04 smithi161 ceph-mon[91579]: pgmap v1561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:06 smithi188 ceph-mon[92425]: pgmap v1562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:06 smithi161 ceph-mon[91579]: pgmap v1562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:08 smithi161 ceph-mon[91579]: pgmap v1563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:08 smithi188 ceph-mon[92425]: pgmap v1563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:10 smithi161 ceph-mon[91579]: pgmap v1564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:10 smithi188 ceph-mon[92425]: pgmap v1564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:12 smithi188 ceph-mon[92425]: pgmap v1565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:12 smithi161 ceph-mon[91579]: pgmap v1565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:14 smithi188 ceph-mon[92425]: pgmap v1566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:14 smithi161 ceph-mon[91579]: pgmap v1566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:16 smithi188 ceph-mon[92425]: pgmap v1567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:16 smithi161 ceph-mon[91579]: pgmap v1567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:18 smithi161 ceph-mon[91579]: pgmap v1568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:18 smithi188 ceph-mon[92425]: pgmap v1568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:20 smithi161 ceph-mon[91579]: pgmap v1569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:20 smithi188 ceph-mon[92425]: pgmap v1569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:22 smithi188 ceph-mon[92425]: pgmap v1570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:22 smithi161 ceph-mon[91579]: pgmap v1570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:24 smithi188 ceph-mon[92425]: pgmap v1571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:24 smithi161 ceph-mon[91579]: pgmap v1571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:26 smithi188 ceph-mon[92425]: pgmap v1572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:26 smithi161 ceph-mon[91579]: pgmap v1572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:28 smithi161 ceph-mon[91579]: pgmap v1573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:28 smithi188 ceph-mon[92425]: pgmap v1573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:30 smithi161 ceph-mon[91579]: pgmap v1574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:30 smithi188 ceph-mon[92425]: pgmap v1574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:32 smithi188 ceph-mon[92425]: pgmap v1575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:32 smithi161 ceph-mon[91579]: pgmap v1575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:34 smithi188 ceph-mon[92425]: pgmap v1576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:34 smithi161 ceph-mon[91579]: pgmap v1576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:36 smithi188 ceph-mon[92425]: pgmap v1577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:31:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:31:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:31:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:36 smithi161 ceph-mon[91579]: pgmap v1577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:31:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:31:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:31:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:38 smithi161 ceph-mon[91579]: pgmap v1578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:38 smithi188 ceph-mon[92425]: pgmap v1578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:40 smithi161 ceph-mon[91579]: pgmap v1579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:40 smithi188 ceph-mon[92425]: pgmap v1579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:42 smithi188 ceph-mon[92425]: pgmap v1580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:42 smithi161 ceph-mon[91579]: pgmap v1580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:44 smithi188 ceph-mon[92425]: pgmap v1581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:44 smithi161 ceph-mon[91579]: pgmap v1581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:46 smithi188 ceph-mon[92425]: pgmap v1582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:46 smithi161 ceph-mon[91579]: pgmap v1582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:48 smithi161 ceph-mon[91579]: pgmap v1583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:48 smithi188 ceph-mon[92425]: pgmap v1583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:50 smithi161 ceph-mon[91579]: pgmap v1584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:50 smithi188 ceph-mon[92425]: pgmap v1584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:52 smithi188 ceph-mon[92425]: pgmap v1585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:52 smithi161 ceph-mon[91579]: pgmap v1585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:54 smithi188 ceph-mon[92425]: pgmap v1586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:54 smithi161 ceph-mon[91579]: pgmap v1586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:31:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:31:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:56 smithi188 ceph-mon[92425]: pgmap v1587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:56 smithi161 ceph-mon[91579]: pgmap v1587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:31:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:31:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:31:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:31:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:31:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:58 smithi188 ceph-mon[92425]: pgmap v1588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:58 smithi161 ceph-mon[91579]: pgmap v1588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:31:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:00 smithi188 ceph-mon[92425]: pgmap v1589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:00 smithi161 ceph-mon[91579]: pgmap v1589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:02 smithi161 ceph-mon[91579]: pgmap v1590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:02 smithi188 ceph-mon[92425]: pgmap v1590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:04 smithi161 ceph-mon[91579]: pgmap v1591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:04 smithi188 ceph-mon[92425]: pgmap v1591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:06.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:32:06.326+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:32:06.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:32:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:32:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:32:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:32:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:32:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:32:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:32:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:32:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:32:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:32:06.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:32:06.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:32:06.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:32:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:32:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:32:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:32:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:32:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:32:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:32:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:32:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:32:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:32:06.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:32:06.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:32:06.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:32:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:32:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:32:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:32:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:32:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:32:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:32:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:32:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:32:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:32:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:32:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:32:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:32:06.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:32:06.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:32:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:32:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:32:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:32:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:32:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:32:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=71] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:32:06.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=71] Local address: 172.21.15.161, port 51184 2022-07-03T11:32:06.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=71] Peer address: 172.21.15.101, port 22 2022-07-03T11:32:06.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=71] Connection lost 2022-07-03T11:32:06.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=71] Aborting connection 2022-07-03T11:32:06.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:32:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:06 smithi161 ceph-mon[91579]: pgmap v1592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:06 smithi188 ceph-mon[92425]: pgmap v1592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:32:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:32:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:32:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:32:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:32:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:32:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:32:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:32:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:32:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:32:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:32:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:32:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:32:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:32:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:32:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:32:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:32:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:32:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:32:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:32:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:32:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:32:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:32:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:32:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:32:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:32:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:32:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:32:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:32:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:32:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:32:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:32:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:32:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:32:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:32:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:32:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:32:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:32:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:32:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:32:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:32:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:32:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:32:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:32:08.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:32:08.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:32:08.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=71] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:32:08.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=71] Local address: 172.21.15.161, port 51184 2022-07-03T11:32:08.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=71] Peer address: 172.21.15.101, port 22 2022-07-03T11:32:08.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=71] Connection lost 2022-07-03T11:32:08.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=71] Aborting connection 2022-07-03T11:32:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:32:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:32:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:32:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:32:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:32:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:32:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:32:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:32:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:32:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:32:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:32:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:32:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:32:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:32:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:32:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:32:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:32:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:32:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:32:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:32:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:32:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:32:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:32:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:32:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:32:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:32:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:32:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:32:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:32:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:32:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:32:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:32:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:32:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:32:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:32:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:32:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:32:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:32:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:32:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:32:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:32:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:32:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:32:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:32:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:32:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:32:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=71] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:32:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=71] Local address: 172.21.15.161, port 51184 2022-07-03T11:32:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=71] Peer address: 172.21.15.101, port 22 2022-07-03T11:32:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=71] Connection lost 2022-07-03T11:32:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=71] Aborting connection 2022-07-03T11:32:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:08 smithi161 ceph-mon[91579]: pgmap v1593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:08 smithi188 ceph-mon[92425]: pgmap v1593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:10 smithi161 ceph-mon[91579]: pgmap v1594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:10 smithi188 ceph-mon[92425]: pgmap v1594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:12 smithi161 ceph-mon[91579]: pgmap v1595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:12 smithi188 ceph-mon[92425]: pgmap v1595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:14 smithi161 ceph-mon[91579]: pgmap v1596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:14 smithi188 ceph-mon[92425]: pgmap v1596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:16 smithi161 ceph-mon[91579]: pgmap v1597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:16 smithi188 ceph-mon[92425]: pgmap v1597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:18 smithi161 ceph-mon[91579]: pgmap v1598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:18 smithi188 ceph-mon[92425]: pgmap v1598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:32:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:20 smithi161 ceph-mon[91579]: pgmap v1599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:20 smithi188 ceph-mon[92425]: pgmap v1599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:22 smithi188 ceph-mon[92425]: pgmap v1600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:22 smithi161 ceph-mon[91579]: pgmap v1600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:24 smithi188 ceph-mon[92425]: pgmap v1601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:24 smithi161 ceph-mon[91579]: pgmap v1601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:26 smithi188 ceph-mon[92425]: pgmap v1602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:26 smithi161 ceph-mon[91579]: pgmap v1602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:28 smithi161 ceph-mon[91579]: pgmap v1603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:28 smithi188 ceph-mon[92425]: pgmap v1603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:30 smithi161 ceph-mon[91579]: pgmap v1604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:30 smithi188 ceph-mon[92425]: pgmap v1604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:32 smithi188 ceph-mon[92425]: pgmap v1605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:32 smithi161 ceph-mon[91579]: pgmap v1605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:34 smithi161 ceph-mon[91579]: pgmap v1606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:34 smithi188 ceph-mon[92425]: pgmap v1606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:36 smithi161 ceph-mon[91579]: pgmap v1607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:36 smithi188 ceph-mon[92425]: pgmap v1607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:38 smithi161 ceph-mon[91579]: pgmap v1608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:38 smithi188 ceph-mon[92425]: pgmap v1608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:40 smithi161 ceph-mon[91579]: pgmap v1609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:40 smithi188 ceph-mon[92425]: pgmap v1609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:42 smithi188 ceph-mon[92425]: pgmap v1610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:42 smithi161 ceph-mon[91579]: pgmap v1610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:44 smithi188 ceph-mon[92425]: pgmap v1611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:44 smithi161 ceph-mon[91579]: pgmap v1611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:46 smithi161 ceph-mon[91579]: pgmap v1612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:46 smithi188 ceph-mon[92425]: pgmap v1612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:48 smithi161 ceph-mon[91579]: pgmap v1613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:48 smithi188 ceph-mon[92425]: pgmap v1613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:50 smithi161 ceph-mon[91579]: pgmap v1614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:50 smithi188 ceph-mon[92425]: pgmap v1614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:52 smithi161 ceph-mon[91579]: pgmap v1615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:52 smithi188 ceph-mon[92425]: pgmap v1615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:54 smithi161 ceph-mon[91579]: pgmap v1616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:54 smithi188 ceph-mon[92425]: pgmap v1616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:32:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:32:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:56 smithi161 ceph-mon[91579]: pgmap v1617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:56 smithi188 ceph-mon[92425]: pgmap v1617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:32:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:32:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:32:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:32:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:32:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:32:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:32:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:32:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:58 smithi161 ceph-mon[91579]: pgmap v1618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:32:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:58 smithi188 ceph-mon[92425]: pgmap v1618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:00 smithi161 ceph-mon[91579]: pgmap v1619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:00 smithi188 ceph-mon[92425]: pgmap v1619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:02 smithi161 ceph-mon[91579]: pgmap v1620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:02 smithi188 ceph-mon[92425]: pgmap v1620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:04 smithi161 ceph-mon[91579]: pgmap v1621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:04 smithi188 ceph-mon[92425]: pgmap v1621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: pgmap v1622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:33:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:33:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:33:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:33:06.767+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:33:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:33:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:33:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:33:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:33:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:07.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:33:07.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:33:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:33:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:33:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: pgmap v1622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:33:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:33:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:33:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:07 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:33:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:33:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:33:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:33:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:33:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:33:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:33:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:33:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:33:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:33:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:07 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:33:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:33:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:33:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:33:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:33:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:33:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:33:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:33:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:33:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:33:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:08 smithi161 ceph-mon[91579]: pgmap v1623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:08 smithi188 ceph-mon[92425]: pgmap v1623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:33:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: pgmap v1624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:33:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:33:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:33:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:33:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: pgmap v1624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:33:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:33:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:33:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:33:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:33:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:12 smithi161 ceph-mon[91579]: pgmap v1625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:12 smithi188 ceph-mon[92425]: pgmap v1625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:14 smithi161 ceph-mon[91579]: pgmap v1626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:14 smithi188 ceph-mon[92425]: pgmap v1626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:16 smithi161 ceph-mon[91579]: pgmap v1627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:16 smithi188 ceph-mon[92425]: pgmap v1627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:18 smithi161 ceph-mon[91579]: pgmap v1628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:18 smithi188 ceph-mon[92425]: pgmap v1628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:20 smithi161 ceph-mon[91579]: pgmap v1629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:20 smithi188 ceph-mon[92425]: pgmap v1629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:22 smithi161 ceph-mon[91579]: pgmap v1630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:22 smithi188 ceph-mon[92425]: pgmap v1630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:24 smithi161 ceph-mon[91579]: pgmap v1631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:24 smithi188 ceph-mon[92425]: pgmap v1631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:26 smithi161 ceph-mon[91579]: pgmap v1632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:26 smithi188 ceph-mon[92425]: pgmap v1632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:28 smithi161 ceph-mon[91579]: pgmap v1633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:28 smithi188 ceph-mon[92425]: pgmap v1633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:30 smithi161 ceph-mon[91579]: pgmap v1634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:30 smithi188 ceph-mon[92425]: pgmap v1634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:32 smithi161 ceph-mon[91579]: pgmap v1635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:33.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:32 smithi188 ceph-mon[92425]: pgmap v1635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:34 smithi161 ceph-mon[91579]: pgmap v1636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:34 smithi188 ceph-mon[92425]: pgmap v1636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-mon[91579]: pgmap v1637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:37.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:33:36.782+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:33:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:33:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:33:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:33:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:33:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:33:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:33:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:33:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:33:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:33:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:33:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:33:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:33:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:33:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:33:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:33:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:33:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:33:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:33:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:33:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:33:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:33:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:33:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:33:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:33:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:33:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:33:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:33:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:33:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:33:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:33:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:33:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:33:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:33:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:33:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:33:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:33:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:33:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:33:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=243] Received exit status 0 2022-07-03T11:33:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=243] Received channel close 2022-07-03T11:33:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=243] Channel closed 2022-07-03T11:33:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=73] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:33:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=73] Local address: 172.21.15.161, port 51188 2022-07-03T11:33:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=73] Peer address: 172.21.15.101, port 22 2022-07-03T11:33:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=244] Requesting new SSH session 2022-07-03T11:33:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=244] Command: sudo true 2022-07-03T11:33:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=244] Received exit status 0 2022-07-03T11:33:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=244] Received channel close 2022-07-03T11:33:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=244] Channel closed 2022-07-03T11:33:37.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=245] Requesting new SSH session 2022-07-03T11:33:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=245] Command: which python3 2022-07-03T11:33:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=245] Received exit status 0 2022-07-03T11:33:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=245] Received channel close 2022-07-03T11:33:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=245] Channel closed 2022-07-03T11:33:37.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=246] Requesting new SSH session 2022-07-03T11:33:37.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=246] Command: sudo true 2022-07-03T11:33:37.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=246] Received exit status 0 2022-07-03T11:33:37.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=246] Received channel close 2022-07-03T11:33:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=246] Channel closed 2022-07-03T11:33:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=247] Requesting new SSH session 2022-07-03T11:33:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=247] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:33:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=247] Received exit status 0 2022-07-03T11:33:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=247] Received channel close 2022-07-03T11:33:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=247] Channel closed 2022-07-03T11:33:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=248] Requesting new SSH session 2022-07-03T11:33:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=248] Command: sudo true 2022-07-03T11:33:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=248] Received exit status 0 2022-07-03T11:33:37.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=248] Received channel close 2022-07-03T11:33:37.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=248] Channel closed 2022-07-03T11:33:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=249] Requesting new SSH session 2022-07-03T11:33:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=249] Command: which python3 2022-07-03T11:33:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=249] Received exit status 0 2022-07-03T11:33:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=249] Received channel close 2022-07-03T11:33:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=249] Channel closed 2022-07-03T11:33:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=250] Requesting new SSH session 2022-07-03T11:33:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=250] Command: sudo true 2022-07-03T11:33:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=250] Received exit status 0 2022-07-03T11:33:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=250] Received channel close 2022-07-03T11:33:37.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=250] Channel closed 2022-07-03T11:33:37.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=251] Requesting new SSH session 2022-07-03T11:33:37.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:37.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=247] Received exit status 0 2022-07-03T11:33:37.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=247] Received channel close 2022-07-03T11:33:37.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=247] Channel closed 2022-07-03T11:33:37.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=248] Requesting new SSH session 2022-07-03T11:33:37.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=248] Command: sudo true 2022-07-03T11:33:37.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=248] Received exit status 0 2022-07-03T11:33:37.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=248] Received channel close 2022-07-03T11:33:37.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=248] Channel closed 2022-07-03T11:33:37.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=249] Requesting new SSH session 2022-07-03T11:33:37.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=249] Command: which python3 2022-07-03T11:33:37.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=249] Received exit status 0 2022-07-03T11:33:37.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=249] Received channel close 2022-07-03T11:33:37.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=249] Channel closed 2022-07-03T11:33:37.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=250] Requesting new SSH session 2022-07-03T11:33:37.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=250] Command: sudo true 2022-07-03T11:33:37.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=250] Received exit status 0 2022-07-03T11:33:37.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=250] Received channel close 2022-07-03T11:33:37.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=250] Channel closed 2022-07-03T11:33:37.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=251] Requesting new SSH session 2022-07-03T11:33:37.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:37.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=251] Received exit status 0 2022-07-03T11:33:37.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=251] Received channel close 2022-07-03T11:33:37.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=251] Channel closed 2022-07-03T11:33:37.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=251] Received exit status 0 2022-07-03T11:33:37.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=251] Received channel close 2022-07-03T11:33:37.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=251] Channel closed 2022-07-03T11:33:37.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=73] Login timeout expired 2022-07-03T11:33:37.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=73] Aborting connection 2022-07-03T11:33:37.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:33:37.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:36 smithi188 ceph-mon[92425]: pgmap v1637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:33:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:33:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:33:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:33:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:33:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:33:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:33:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:33:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:33:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:33:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:33:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:33:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:33:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:33:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:33:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:33:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:33:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:33:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=243] Received exit status 0 2022-07-03T11:33:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=243] Received channel close 2022-07-03T11:33:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=243] Channel closed 2022-07-03T11:33:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=73] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:33:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=73] Local address: 172.21.15.161, port 51188 2022-07-03T11:33:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=73] Peer address: 172.21.15.101, port 22 2022-07-03T11:33:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=244] Requesting new SSH session 2022-07-03T11:33:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=244] Command: sudo true 2022-07-03T11:33:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=244] Received exit status 0 2022-07-03T11:33:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=244] Received channel close 2022-07-03T11:33:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=244] Channel closed 2022-07-03T11:33:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=245] Requesting new SSH session 2022-07-03T11:33:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=245] Command: which python3 2022-07-03T11:33:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=245] Received exit status 0 2022-07-03T11:33:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=245] Received channel close 2022-07-03T11:33:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=245] Channel closed 2022-07-03T11:33:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=246] Requesting new SSH session 2022-07-03T11:33:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=246] Command: sudo true 2022-07-03T11:33:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=246] Received exit status 0 2022-07-03T11:33:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=246] Received channel close 2022-07-03T11:33:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=246] Channel closed 2022-07-03T11:33:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=247] Requesting new SSH session 2022-07-03T11:33:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=247] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:33:38.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=247] Received exit status 0 2022-07-03T11:33:38.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=247] Received channel close 2022-07-03T11:33:38.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=247] Channel closed 2022-07-03T11:33:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=248] Requesting new SSH session 2022-07-03T11:33:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=248] Command: sudo true 2022-07-03T11:33:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=248] Received exit status 0 2022-07-03T11:33:38.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=248] Received channel close 2022-07-03T11:33:38.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=248] Channel closed 2022-07-03T11:33:38.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=249] Requesting new SSH session 2022-07-03T11:33:38.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=249] Command: which python3 2022-07-03T11:33:38.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=249] Received exit status 0 2022-07-03T11:33:38.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=249] Received channel close 2022-07-03T11:33:38.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=249] Channel closed 2022-07-03T11:33:38.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=250] Requesting new SSH session 2022-07-03T11:33:38.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=250] Command: sudo true 2022-07-03T11:33:38.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=250] Received exit status 0 2022-07-03T11:33:38.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=250] Received channel close 2022-07-03T11:33:38.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=250] Channel closed 2022-07-03T11:33:38.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=251] Requesting new SSH session 2022-07-03T11:33:38.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:38.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=247] Received exit status 0 2022-07-03T11:33:38.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=247] Received channel close 2022-07-03T11:33:38.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=247] Channel closed 2022-07-03T11:33:38.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=248] Requesting new SSH session 2022-07-03T11:33:38.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=248] Command: sudo true 2022-07-03T11:33:38.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=248] Received exit status 0 2022-07-03T11:33:38.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=248] Received channel close 2022-07-03T11:33:38.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=248] Channel closed 2022-07-03T11:33:38.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=249] Requesting new SSH session 2022-07-03T11:33:38.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=249] Command: which python3 2022-07-03T11:33:38.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=249] Received exit status 0 2022-07-03T11:33:38.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=249] Received channel close 2022-07-03T11:33:38.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=249] Channel closed 2022-07-03T11:33:38.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=250] Requesting new SSH session 2022-07-03T11:33:38.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=250] Command: sudo true 2022-07-03T11:33:38.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=250] Received exit status 0 2022-07-03T11:33:38.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=250] Received channel close 2022-07-03T11:33:38.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=250] Channel closed 2022-07-03T11:33:38.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=251] Requesting new SSH session 2022-07-03T11:33:38.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:38.176 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=251] Received exit status 0 2022-07-03T11:33:38.176 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=251] Received channel close 2022-07-03T11:33:38.177 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=251] Channel closed 2022-07-03T11:33:38.177 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=251] Received exit status 0 2022-07-03T11:33:38.178 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=251] Received channel close 2022-07-03T11:33:38.179 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=251] Channel closed 2022-07-03T11:33:38.179 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=73] Login timeout expired 2022-07-03T11:33:38.180 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=73] Aborting connection 2022-07-03T11:33:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:33:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:33:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:33:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:33:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:33:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:33:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:33:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:33:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:33:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:33:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:33:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:33:38.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:33:38.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:33:38.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:33:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:33:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:33:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:33:38.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:33:38.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:33:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:33:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:33:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:33:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:33:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:33:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:33:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:33:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:33:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:33:38.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:33:38.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:33:38.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:33:38.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:33:38.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:33:38.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:33:38.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:33:38.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:33:38.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:33:38.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:33:38.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:33:38.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:33:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:33:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:33:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:33:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:33:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=243] Received exit status 0 2022-07-03T11:33:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=243] Received channel close 2022-07-03T11:33:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=243] Channel closed 2022-07-03T11:33:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=73] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:33:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=73] Local address: 172.21.15.161, port 51188 2022-07-03T11:33:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=73] Peer address: 172.21.15.101, port 22 2022-07-03T11:33:38.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=244] Requesting new SSH session 2022-07-03T11:33:38.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=244] Command: sudo true 2022-07-03T11:33:38.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=244] Received exit status 0 2022-07-03T11:33:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=244] Received channel close 2022-07-03T11:33:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=244] Channel closed 2022-07-03T11:33:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=245] Requesting new SSH session 2022-07-03T11:33:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=245] Command: which python3 2022-07-03T11:33:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=245] Received exit status 0 2022-07-03T11:33:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=245] Received channel close 2022-07-03T11:33:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=245] Channel closed 2022-07-03T11:33:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=246] Requesting new SSH session 2022-07-03T11:33:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=246] Command: sudo true 2022-07-03T11:33:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=246] Received exit status 0 2022-07-03T11:33:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=246] Received channel close 2022-07-03T11:33:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=246] Channel closed 2022-07-03T11:33:38.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=247] Requesting new SSH session 2022-07-03T11:33:38.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=247] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T11:33:38.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=247] Received exit status 0 2022-07-03T11:33:38.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=247] Received channel close 2022-07-03T11:33:38.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=247] Channel closed 2022-07-03T11:33:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=248] Requesting new SSH session 2022-07-03T11:33:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=248] Command: sudo true 2022-07-03T11:33:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=248] Received exit status 0 2022-07-03T11:33:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=248] Received channel close 2022-07-03T11:33:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=248] Channel closed 2022-07-03T11:33:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=249] Requesting new SSH session 2022-07-03T11:33:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=249] Command: which python3 2022-07-03T11:33:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=249] Received exit status 0 2022-07-03T11:33:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=249] Received channel close 2022-07-03T11:33:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=249] Channel closed 2022-07-03T11:33:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=250] Requesting new SSH session 2022-07-03T11:33:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=250] Command: sudo true 2022-07-03T11:33:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=250] Received exit status 0 2022-07-03T11:33:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=250] Received channel close 2022-07-03T11:33:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=250] Channel closed 2022-07-03T11:33:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=251] Requesting new SSH session 2022-07-03T11:33:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=247] Received exit status 0 2022-07-03T11:33:38.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=247] Received channel close 2022-07-03T11:33:38.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=247] Channel closed 2022-07-03T11:33:38.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=248] Requesting new SSH session 2022-07-03T11:33:38.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=248] Command: sudo true 2022-07-03T11:33:38.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=248] Received exit status 0 2022-07-03T11:33:38.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=248] Received channel close 2022-07-03T11:33:38.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=248] Channel closed 2022-07-03T11:33:38.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=249] Requesting new SSH session 2022-07-03T11:33:38.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=249] Command: which python3 2022-07-03T11:33:38.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=249] Received exit status 0 2022-07-03T11:33:38.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=249] Received channel close 2022-07-03T11:33:38.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=249] Channel closed 2022-07-03T11:33:38.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=250] Requesting new SSH session 2022-07-03T11:33:38.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=250] Command: sudo true 2022-07-03T11:33:38.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=250] Received exit status 0 2022-07-03T11:33:38.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=250] Received channel close 2022-07-03T11:33:38.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=250] Channel closed 2022-07-03T11:33:38.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=251] Requesting new SSH session 2022-07-03T11:33:38.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=251] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:33:38.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=251] Received exit status 0 2022-07-03T11:33:38.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=251] Received channel close 2022-07-03T11:33:38.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=251] Channel closed 2022-07-03T11:33:38.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=251] Received exit status 0 2022-07-03T11:33:38.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=251] Received channel close 2022-07-03T11:33:38.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=251] Channel closed 2022-07-03T11:33:38.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=73] Login timeout expired 2022-07-03T11:33:38.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=73] Aborting connection 2022-07-03T11:33:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:38 smithi161 ceph-mon[91579]: pgmap v1638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:38 smithi188 ceph-mon[92425]: pgmap v1638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:40 smithi161 ceph-mon[91579]: pgmap v1639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:40 smithi188 ceph-mon[92425]: pgmap v1639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:42 smithi161 ceph-mon[91579]: pgmap v1640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:42 smithi188 ceph-mon[92425]: pgmap v1640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:44 smithi161 ceph-mon[91579]: pgmap v1641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:44 smithi188 ceph-mon[92425]: pgmap v1641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:46 smithi161 ceph-mon[91579]: pgmap v1642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:46 smithi188 ceph-mon[92425]: pgmap v1642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:48 smithi161 ceph-mon[91579]: pgmap v1643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:48 smithi188 ceph-mon[92425]: pgmap v1643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:50 smithi161 ceph-mon[91579]: pgmap v1644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:50 smithi188 ceph-mon[92425]: pgmap v1644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:52 smithi161 ceph-mon[91579]: pgmap v1645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:52 smithi188 ceph-mon[92425]: pgmap v1645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:54 smithi161 ceph-mon[91579]: pgmap v1646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:54 smithi188 ceph-mon[92425]: pgmap v1646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:33:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:33:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:56 smithi161 ceph-mon[91579]: pgmap v1647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:56 smithi188 ceph-mon[92425]: pgmap v1647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:33:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:33:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:33:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:33:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:33:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:33:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:33:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:33:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:58 smithi161 ceph-mon[91579]: pgmap v1648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:33:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:58 smithi188 ceph-mon[92425]: pgmap v1648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:00 smithi161 ceph-mon[91579]: pgmap v1649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:00 smithi188 ceph-mon[92425]: pgmap v1649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:02 smithi161 ceph-mon[91579]: pgmap v1650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:02 smithi188 ceph-mon[92425]: pgmap v1650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:04 smithi161 ceph-mon[91579]: pgmap v1651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:04 smithi188 ceph-mon[92425]: pgmap v1651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:06 smithi161 ceph-mon[91579]: pgmap v1652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:06 smithi188 ceph-mon[92425]: pgmap v1652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:08 smithi161 ceph-mon[91579]: pgmap v1653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:08 smithi188 ceph-mon[92425]: pgmap v1653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:10 smithi161 ceph-mon[91579]: pgmap v1654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:10 smithi188 ceph-mon[92425]: pgmap v1654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:12 smithi161 ceph-mon[91579]: pgmap v1655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:12 smithi188 ceph-mon[92425]: pgmap v1655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:14 smithi161 ceph-mon[91579]: pgmap v1656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:15.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:14 smithi188 ceph-mon[92425]: pgmap v1656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:16 smithi161 ceph-mon[91579]: pgmap v1657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:16 smithi188 ceph-mon[92425]: pgmap v1657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:18 smithi161 ceph-mon[91579]: pgmap v1658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:18 smithi188 ceph-mon[92425]: pgmap v1658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:20 smithi161 ceph-mon[91579]: pgmap v1659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:20 smithi188 ceph-mon[92425]: pgmap v1659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:22 smithi161 ceph-mon[91579]: pgmap v1660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:22 smithi188 ceph-mon[92425]: pgmap v1660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:24 smithi161 ceph-mon[91579]: pgmap v1661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:24 smithi188 ceph-mon[92425]: pgmap v1661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:26 smithi161 ceph-mon[91579]: pgmap v1662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:26 smithi188 ceph-mon[92425]: pgmap v1662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:28 smithi161 ceph-mon[91579]: pgmap v1663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:28 smithi188 ceph-mon[92425]: pgmap v1663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:30 smithi161 ceph-mon[91579]: pgmap v1664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:30 smithi188 ceph-mon[92425]: pgmap v1664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:32 smithi161 ceph-mon[91579]: pgmap v1665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:32 smithi188 ceph-mon[92425]: pgmap v1665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:34 smithi161 ceph-mon[91579]: pgmap v1666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:34 smithi188 ceph-mon[92425]: pgmap v1666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:36 smithi161 ceph-mon[91579]: pgmap v1667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:36 smithi188 ceph-mon[92425]: pgmap v1667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:34:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:34:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:34:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:34:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:34:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:34:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:38 smithi161 ceph-mon[91579]: pgmap v1668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:38 smithi188 ceph-mon[92425]: pgmap v1668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:39.974 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:40 smithi161 ceph-mon[91579]: pgmap v1669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:34:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:34:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:40 smithi188 ceph-mon[92425]: pgmap v1669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:34:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:34:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:41 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T11:34:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:41 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T11:34:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:42 smithi161 ceph-mon[91579]: pgmap v1670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:42 smithi188 ceph-mon[92425]: pgmap v1670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:44 smithi161 ceph-mon[91579]: pgmap v1671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:44 smithi188 ceph-mon[92425]: pgmap v1671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:45.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:46 smithi161 ceph-mon[91579]: pgmap v1672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:46 smithi188 ceph-mon[92425]: pgmap v1672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:48 smithi161 ceph-mon[91579]: pgmap v1673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:48 smithi188 ceph-mon[92425]: pgmap v1673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:50 smithi161 ceph-mon[91579]: pgmap v1674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:50 smithi188 ceph-mon[92425]: pgmap v1674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:52 smithi161 ceph-mon[91579]: pgmap v1675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:52 smithi188 ceph-mon[92425]: pgmap v1675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:54 smithi161 ceph-mon[91579]: pgmap v1676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:54 smithi188 ceph-mon[92425]: pgmap v1676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:34:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:34:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:56 smithi161 ceph-mon[91579]: pgmap v1677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:56 smithi188 ceph-mon[92425]: pgmap v1677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:34:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:34:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:34:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:34:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:34:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:34:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:34:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:34:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:58 smithi161 ceph-mon[91579]: pgmap v1678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:34:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:58 smithi188 ceph-mon[92425]: pgmap v1678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:00 smithi161 ceph-mon[91579]: pgmap v1679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:00 smithi188 ceph-mon[92425]: pgmap v1679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:02 smithi161 ceph-mon[91579]: pgmap v1680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:02 smithi188 ceph-mon[92425]: pgmap v1680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:04 smithi161 ceph-mon[91579]: pgmap v1681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:04 smithi188 ceph-mon[92425]: pgmap v1681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:06 smithi161 ceph-mon[91579]: pgmap v1682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:06 smithi188 ceph-mon[92425]: pgmap v1682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:35:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:35:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:08 smithi161 ceph-mon[91579]: pgmap v1683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:08 smithi188 ceph-mon[92425]: pgmap v1683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:10 smithi161 ceph-mon[91579]: pgmap v1684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:10 smithi188 ceph-mon[92425]: pgmap v1684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:12 smithi161 ceph-mon[91579]: pgmap v1685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:12 smithi188 ceph-mon[92425]: pgmap v1685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:14 smithi161 ceph-mon[91579]: pgmap v1686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:14 smithi188 ceph-mon[92425]: pgmap v1686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:16 smithi161 ceph-mon[91579]: pgmap v1687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:16 smithi188 ceph-mon[92425]: pgmap v1687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:18 smithi161 ceph-mon[91579]: pgmap v1688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:18 smithi188 ceph-mon[92425]: pgmap v1688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:20 smithi161 ceph-mon[91579]: pgmap v1689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:20 smithi188 ceph-mon[92425]: pgmap v1689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:22 smithi161 ceph-mon[91579]: pgmap v1690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:22 smithi188 ceph-mon[92425]: pgmap v1690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:24 smithi161 ceph-mon[91579]: pgmap v1691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:24 smithi188 ceph-mon[92425]: pgmap v1691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:26 smithi161 ceph-mon[91579]: pgmap v1692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:26 smithi188 ceph-mon[92425]: pgmap v1692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:28 smithi161 ceph-mon[91579]: pgmap v1693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:28 smithi188 ceph-mon[92425]: pgmap v1693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:30 smithi161 ceph-mon[91579]: pgmap v1694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:30 smithi188 ceph-mon[92425]: pgmap v1694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:32 smithi161 ceph-mon[91579]: pgmap v1695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:32 smithi188 ceph-mon[92425]: pgmap v1695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:34 smithi161 ceph-mon[91579]: pgmap v1696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:34 smithi188 ceph-mon[92425]: pgmap v1696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-mon[91579]: pgmap v1697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:37.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:35:36.820+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:35:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:35:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:35:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:35:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:35:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:35:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:35:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:35:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:35:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:35:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:35:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:35:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:35:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:35:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:35:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:35:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:35:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:35:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:35:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:35:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:35:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:35:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:35:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:35:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:35:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:35:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:35:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:35:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:35:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:35:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:35:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:35:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:35:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:35:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:35:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:35:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:35:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:35:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:35:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:35:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:35:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:35:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:35:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:35:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:35:37.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=75] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:35:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=75] Local address: 172.21.15.161, port 51192 2022-07-03T11:35:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=75] Peer address: 172.21.15.101, port 22 2022-07-03T11:35:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=75] Login timeout expired 2022-07-03T11:35:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=75] Aborting connection 2022-07-03T11:35:37.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:35:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:35:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:36 smithi188 ceph-mon[92425]: pgmap v1697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:35:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:35:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:35:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:35:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:35:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:35:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:35:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:35:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:35:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:35:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:35:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:35:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:35:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:35:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:35:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:35:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:35:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:35:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:35:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:35:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:35:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:35:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:35:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:35:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:35:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:35:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:35:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:35:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:35:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:35:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:35:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:35:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:35:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:35:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:35:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:35:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:35:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:35:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:35:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:35:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:35:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:35:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:35:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:35:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:35:38.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=75] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:35:38.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=75] Local address: 172.21.15.161, port 51192 2022-07-03T11:35:38.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=75] Peer address: 172.21.15.101, port 22 2022-07-03T11:35:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=75] Login timeout expired 2022-07-03T11:35:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=75] Aborting connection 2022-07-03T11:35:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:35:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:35:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:35:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:35:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:35:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:35:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:35:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:35:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:35:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:35:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:35:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:35:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:35:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:35:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:35:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:35:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:35:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:35:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:35:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:35:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:35:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:35:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:35:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:35:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:35:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:35:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:35:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:35:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:35:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:35:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:35:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:35:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:35:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:35:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:35:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:35:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:35:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:35:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:35:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:35:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:35:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:35:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:35:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:35:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:35:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=75] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:35:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=75] Local address: 172.21.15.161, port 51192 2022-07-03T11:35:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=75] Peer address: 172.21.15.101, port 22 2022-07-03T11:35:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=75] Login timeout expired 2022-07-03T11:35:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=75] Aborting connection 2022-07-03T11:35:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:38 smithi161 ceph-mon[91579]: pgmap v1698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:38 smithi188 ceph-mon[92425]: pgmap v1698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:40 smithi161 ceph-mon[91579]: pgmap v1699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:40 smithi188 ceph-mon[92425]: pgmap v1699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:42 smithi161 ceph-mon[91579]: pgmap v1700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:42 smithi188 ceph-mon[92425]: pgmap v1700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:44 smithi161 ceph-mon[91579]: pgmap v1701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:44 smithi188 ceph-mon[92425]: pgmap v1701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:46 smithi161 ceph-mon[91579]: pgmap v1702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:46 smithi188 ceph-mon[92425]: pgmap v1702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:48 smithi161 ceph-mon[91579]: pgmap v1703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:48 smithi188 ceph-mon[92425]: pgmap v1703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:50 smithi161 ceph-mon[91579]: pgmap v1704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:50 smithi188 ceph-mon[92425]: pgmap v1704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:52 smithi161 ceph-mon[91579]: pgmap v1705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:52 smithi188 ceph-mon[92425]: pgmap v1705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:54 smithi161 ceph-mon[91579]: pgmap v1706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:54 smithi188 ceph-mon[92425]: pgmap v1706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:35:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:35:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:56 smithi161 ceph-mon[91579]: pgmap v1707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:56 smithi188 ceph-mon[92425]: pgmap v1707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:35:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:35:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:35:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:35:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:35:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:35:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:35:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:35:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:58 smithi161 ceph-mon[91579]: pgmap v1708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:35:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:58 smithi188 ceph-mon[92425]: pgmap v1708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:00 smithi161 ceph-mon[91579]: pgmap v1709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:00 smithi188 ceph-mon[92425]: pgmap v1709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:02 smithi161 ceph-mon[91579]: pgmap v1710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:02 smithi188 ceph-mon[92425]: pgmap v1710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:04 smithi161 ceph-mon[91579]: pgmap v1711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:04 smithi188 ceph-mon[92425]: pgmap v1711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:06 smithi161 ceph-mon[91579]: pgmap v1712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:06 smithi188 ceph-mon[92425]: pgmap v1712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:08 smithi161 ceph-mon[91579]: pgmap v1713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:08 smithi188 ceph-mon[92425]: pgmap v1713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:10 smithi161 ceph-mon[91579]: pgmap v1714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:10 smithi188 ceph-mon[92425]: pgmap v1714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:12 smithi161 ceph-mon[91579]: pgmap v1715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:12 smithi188 ceph-mon[92425]: pgmap v1715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:14 smithi161 ceph-mon[91579]: pgmap v1716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:14 smithi188 ceph-mon[92425]: pgmap v1716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:16 smithi161 ceph-mon[91579]: pgmap v1717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:16 smithi188 ceph-mon[92425]: pgmap v1717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:18 smithi161 ceph-mon[91579]: pgmap v1718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:18 smithi188 ceph-mon[92425]: pgmap v1718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:20 smithi161 ceph-mon[91579]: pgmap v1719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:20 smithi188 ceph-mon[92425]: pgmap v1719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:22 smithi161 ceph-mon[91579]: pgmap v1720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:22 smithi188 ceph-mon[92425]: pgmap v1720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:24 smithi161 ceph-mon[91579]: pgmap v1721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:24 smithi188 ceph-mon[92425]: pgmap v1721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:26 smithi161 ceph-mon[91579]: pgmap v1722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:26 smithi188 ceph-mon[92425]: pgmap v1722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:28 smithi161 ceph-mon[91579]: pgmap v1723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:28 smithi188 ceph-mon[92425]: pgmap v1723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:30 smithi161 ceph-mon[91579]: pgmap v1724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:30 smithi188 ceph-mon[92425]: pgmap v1724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:32 smithi161 ceph-mon[91579]: pgmap v1725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:32 smithi188 ceph-mon[92425]: pgmap v1725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:34 smithi161 ceph-mon[91579]: pgmap v1726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:34 smithi188 ceph-mon[92425]: pgmap v1726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:36 smithi161 ceph-mon[91579]: pgmap v1727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:36 smithi188 ceph-mon[92425]: pgmap v1727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:36:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:36:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:36:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:36:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:36:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:36:39.012 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:38 smithi188 ceph-mon[92425]: pgmap v1728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:39.013 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:38 smithi161 ceph-mon[91579]: pgmap v1728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:40 smithi161 ceph-mon[91579]: pgmap v1729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:36:41.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:36:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:40 smithi188 ceph-mon[92425]: pgmap v1729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:36:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:36:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:41 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T11:36:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:41 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T11:36:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:42 smithi161 ceph-mon[91579]: pgmap v1730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:43.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:42 smithi188 ceph-mon[92425]: pgmap v1730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:44 smithi161 ceph-mon[91579]: pgmap v1731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:44 smithi188 ceph-mon[92425]: pgmap v1731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:46 smithi161 ceph-mon[91579]: pgmap v1732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:47.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:46 smithi188 ceph-mon[92425]: pgmap v1732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:48 smithi161 ceph-mon[91579]: pgmap v1733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:48 smithi188 ceph-mon[92425]: pgmap v1733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:50 smithi161 ceph-mon[91579]: pgmap v1734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:50 smithi188 ceph-mon[92425]: pgmap v1734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:52 smithi161 ceph-mon[91579]: pgmap v1735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:52 smithi188 ceph-mon[92425]: pgmap v1735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:54 smithi161 ceph-mon[91579]: pgmap v1736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:54 smithi188 ceph-mon[92425]: pgmap v1736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:36:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:56 smithi161 ceph-mon[91579]: pgmap v1737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:56 smithi188 ceph-mon[92425]: pgmap v1737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:36:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:36:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:36:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:36:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:36:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:36:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:36:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:36:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:36:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:58 smithi161 ceph-mon[91579]: pgmap v1738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:36:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:58 smithi188 ceph-mon[92425]: pgmap v1738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:36:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:00 smithi161 ceph-mon[91579]: pgmap v1739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:00 smithi188 ceph-mon[92425]: pgmap v1739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:02 smithi161 ceph-mon[91579]: pgmap v1740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:02 smithi188 ceph-mon[92425]: pgmap v1740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:04 smithi161 ceph-mon[91579]: pgmap v1741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:04 smithi188 ceph-mon[92425]: pgmap v1741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-mon[91579]: pgmap v1742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:07.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:37:06.855+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:37:07.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:37:07.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:37:07.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:37:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:37:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:37:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:37:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:37:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:37:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:37:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:37:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:37:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:37:07.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:37:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:37:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:37:07.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:37:07.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:37:07.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:37:07.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:37:07.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:37:07.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:37:07.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:37:07.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:37:07.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:37:07.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:37:07.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:37:07.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:37:07.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:37:07.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:37:07.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:37:07.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:37:07.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:37:07.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:37:07.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:37:07.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:37:07.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:37:07.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:37:07.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:37:07.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:37:07.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:37:07.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:37:07.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=77] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:37:07.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=77] Local address: 172.21.15.161, port 51216 2022-07-03T11:37:07.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=77] Peer address: 172.21.15.101, port 22 2022-07-03T11:37:07.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=77] Connection lost 2022-07-03T11:37:07.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=77] Aborting connection 2022-07-03T11:37:07.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:37:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:06 smithi188 ceph-mon[92425]: pgmap v1742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:37:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:37:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:37:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:37:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:37:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:37:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:37:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:37:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:37:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:37:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:37:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:37:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:37:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:37:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:37:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:37:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:37:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:37:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:37:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=77] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:37:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=77] Local address: 172.21.15.161, port 51216 2022-07-03T11:37:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=77] Peer address: 172.21.15.101, port 22 2022-07-03T11:37:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=77] Connection lost 2022-07-03T11:37:08.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=77] Aborting connection 2022-07-03T11:37:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:37:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:37:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:37:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:37:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:37:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:37:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:37:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:37:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:37:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:37:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:37:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:37:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:37:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:37:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:37:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:37:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:37:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:37:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:37:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:37:08.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:37:08.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:37:08.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:37:08.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=77] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:37:08.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=77] Local address: 172.21.15.161, port 51216 2022-07-03T11:37:08.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=77] Peer address: 172.21.15.101, port 22 2022-07-03T11:37:08.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=77] Connection lost 2022-07-03T11:37:08.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=77] Aborting connection 2022-07-03T11:37:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:08 smithi161 ceph-mon[91579]: pgmap v1743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:08 smithi188 ceph-mon[92425]: pgmap v1743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:10 smithi161 ceph-mon[91579]: pgmap v1744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:10 smithi188 ceph-mon[92425]: pgmap v1744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:12 smithi161 ceph-mon[91579]: pgmap v1745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:12 smithi188 ceph-mon[92425]: pgmap v1745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:14 smithi161 ceph-mon[91579]: pgmap v1746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:14 smithi188 ceph-mon[92425]: pgmap v1746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:17.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:16 smithi161 ceph-mon[91579]: pgmap v1747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:16 smithi188 ceph-mon[92425]: pgmap v1747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:18 smithi161 ceph-mon[91579]: pgmap v1748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:18 smithi188 ceph-mon[92425]: pgmap v1748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:20 smithi161 ceph-mon[91579]: pgmap v1749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:20 smithi188 ceph-mon[92425]: pgmap v1749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:22 smithi161 ceph-mon[91579]: pgmap v1750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:22 smithi188 ceph-mon[92425]: pgmap v1750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:24 smithi161 ceph-mon[91579]: pgmap v1751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:24 smithi188 ceph-mon[92425]: pgmap v1751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:26 smithi161 ceph-mon[91579]: pgmap v1752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:26 smithi188 ceph-mon[92425]: pgmap v1752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:28 smithi161 ceph-mon[91579]: pgmap v1753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:28 smithi188 ceph-mon[92425]: pgmap v1753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:30 smithi161 ceph-mon[91579]: pgmap v1754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:30 smithi188 ceph-mon[92425]: pgmap v1754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:32 smithi161 ceph-mon[91579]: pgmap v1755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:32 smithi188 ceph-mon[92425]: pgmap v1755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:34 smithi161 ceph-mon[91579]: pgmap v1756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:34 smithi188 ceph-mon[92425]: pgmap v1756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:36 smithi161 ceph-mon[91579]: pgmap v1757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:36 smithi188 ceph-mon[92425]: pgmap v1757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:38 smithi161 ceph-mon[91579]: pgmap v1758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:38 smithi188 ceph-mon[92425]: pgmap v1758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:40 smithi161 ceph-mon[91579]: pgmap v1759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:40 smithi188 ceph-mon[92425]: pgmap v1759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:42 smithi161 ceph-mon[91579]: pgmap v1760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:42 smithi188 ceph-mon[92425]: pgmap v1760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:44 smithi161 ceph-mon[91579]: pgmap v1761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:44 smithi188 ceph-mon[92425]: pgmap v1761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:46 smithi161 ceph-mon[91579]: pgmap v1762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:46 smithi188 ceph-mon[92425]: pgmap v1762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:48 smithi161 ceph-mon[91579]: pgmap v1763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:48 smithi188 ceph-mon[92425]: pgmap v1763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:50 smithi161 ceph-mon[91579]: pgmap v1764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:50 smithi188 ceph-mon[92425]: pgmap v1764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:52 smithi161 ceph-mon[91579]: pgmap v1765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:52 smithi188 ceph-mon[92425]: pgmap v1765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:54 smithi161 ceph-mon[91579]: pgmap v1766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:54 smithi188 ceph-mon[92425]: pgmap v1766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:37:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:56 smithi161 ceph-mon[91579]: pgmap v1767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:56 smithi188 ceph-mon[92425]: pgmap v1767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:37:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:37:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:37:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:37:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:37:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:37:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:37:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:37:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:37:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:58 smithi161 ceph-mon[91579]: pgmap v1768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:37:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:58 smithi188 ceph-mon[92425]: pgmap v1768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:37:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:00 smithi161 ceph-mon[91579]: pgmap v1769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:00 smithi188 ceph-mon[92425]: pgmap v1769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:02 smithi161 ceph-mon[91579]: pgmap v1770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:02 smithi188 ceph-mon[92425]: pgmap v1770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:04 smithi161 ceph-mon[91579]: pgmap v1771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:04 smithi188 ceph-mon[92425]: pgmap v1771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:06 smithi161 ceph-mon[91579]: pgmap v1772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:06 smithi188 ceph-mon[92425]: pgmap v1772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:38:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:38:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:38:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:38:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:38:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:38:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:38:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:38:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:08 smithi161 ceph-mon[91579]: pgmap v1773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:08 smithi188 ceph-mon[92425]: pgmap v1773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:10 smithi161 ceph-mon[91579]: pgmap v1774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:10 smithi188 ceph-mon[92425]: pgmap v1774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:12 smithi161 ceph-mon[91579]: pgmap v1775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:12 smithi188 ceph-mon[92425]: pgmap v1775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:14 smithi161 ceph-mon[91579]: pgmap v1776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:14 smithi188 ceph-mon[92425]: pgmap v1776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:16 smithi161 ceph-mon[91579]: pgmap v1777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:16 smithi188 ceph-mon[92425]: pgmap v1777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:18 smithi161 ceph-mon[91579]: pgmap v1778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:18 smithi188 ceph-mon[92425]: pgmap v1778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:19.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:20 smithi161 ceph-mon[91579]: pgmap v1779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:20.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:20 smithi188 ceph-mon[92425]: pgmap v1779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:22 smithi161 ceph-mon[91579]: pgmap v1780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:22 smithi188 ceph-mon[92425]: pgmap v1780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:24 smithi161 ceph-mon[91579]: pgmap v1781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:24 smithi188 ceph-mon[92425]: pgmap v1781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:26 smithi161 ceph-mon[91579]: pgmap v1782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:26 smithi188 ceph-mon[92425]: pgmap v1782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:28 smithi161 ceph-mon[91579]: pgmap v1783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:28 smithi188 ceph-mon[92425]: pgmap v1783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:30 smithi161 ceph-mon[91579]: pgmap v1784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:30 smithi188 ceph-mon[92425]: pgmap v1784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:32 smithi161 ceph-mon[91579]: pgmap v1785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:32 smithi188 ceph-mon[92425]: pgmap v1785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:34 smithi161 ceph-mon[91579]: pgmap v1786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:34 smithi188 ceph-mon[92425]: pgmap v1786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-mon[91579]: pgmap v1787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:38:36.892+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:38:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:38:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:38:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:38:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:38:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:38:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:38:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:38:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:38:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:38:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:38:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:38:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:38:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:38:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:38:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:38:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:38:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:38:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:38:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:38:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:38:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:38:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:38:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:38:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:38:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:38:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:38:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:38:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:38:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:38:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:38:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:38:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:38:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:38:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:38:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:38:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:38:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:38:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:38:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:38:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:38:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:38:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:38:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:38:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:38:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=79] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:38:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=79] Local address: 172.21.15.161, port 51244 2022-07-03T11:38:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=79] Peer address: 172.21.15.101, port 22 2022-07-03T11:38:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=264] Received exit status 0 2022-07-03T11:38:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=264] Received channel close 2022-07-03T11:38:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=264] Channel closed 2022-07-03T11:38:37.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=265] Requesting new SSH session 2022-07-03T11:38:37.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=265] Command: which python3 2022-07-03T11:38:37.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=264] Received exit status 0 2022-07-03T11:38:37.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=264] Received channel close 2022-07-03T11:38:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=264] Channel closed 2022-07-03T11:38:37.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=265] Requesting new SSH session 2022-07-03T11:38:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=265] Command: which python3 2022-07-03T11:38:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=265] Received exit status 0 2022-07-03T11:38:37.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=265] Received channel close 2022-07-03T11:38:37.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=265] Channel closed 2022-07-03T11:38:37.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=266] Requesting new SSH session 2022-07-03T11:38:37.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=266] Command: sudo true 2022-07-03T11:38:37.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=265] Received exit status 0 2022-07-03T11:38:37.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=265] Received channel close 2022-07-03T11:38:37.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=265] Channel closed 2022-07-03T11:38:37.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=266] Requesting new SSH session 2022-07-03T11:38:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=266] Command: sudo true 2022-07-03T11:38:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=266] Received exit status 0 2022-07-03T11:38:37.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=266] Received channel close 2022-07-03T11:38:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=266] Channel closed 2022-07-03T11:38:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=267] Requesting new SSH session 2022-07-03T11:38:37.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=266] Received exit status 0 2022-07-03T11:38:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=266] Received channel close 2022-07-03T11:38:37.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=266] Channel closed 2022-07-03T11:38:37.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=267] Requesting new SSH session 2022-07-03T11:38:37.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=267] Received exit status 0 2022-07-03T11:38:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=267] Received channel close 2022-07-03T11:38:37.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=267] Channel closed 2022-07-03T11:38:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=267] Received exit status 0 2022-07-03T11:38:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=267] Received channel close 2022-07-03T11:38:37.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=267] Channel closed 2022-07-03T11:38:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=79] Login timeout expired 2022-07-03T11:38:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=79] Aborting connection 2022-07-03T11:38:37.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:38:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:38:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:36 smithi188 ceph-mon[92425]: pgmap v1787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:38:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:38:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:38:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:38:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:38:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:38:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:38:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:38:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:38:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:38:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:38:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:38:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:38:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:38:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:38:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:38:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:38:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:38:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:38:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:38:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:38:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:38:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:38:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:38:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:38:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:38:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:38:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:38:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:38:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:38:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:38:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:38:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:38:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:38:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:38:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:38:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:38:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:38:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:38:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:38:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:38:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:38:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:38:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:38:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:38:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=79] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:38:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=79] Local address: 172.21.15.161, port 51244 2022-07-03T11:38:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=79] Peer address: 172.21.15.101, port 22 2022-07-03T11:38:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=264] Received exit status 0 2022-07-03T11:38:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=264] Received channel close 2022-07-03T11:38:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=264] Channel closed 2022-07-03T11:38:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=265] Requesting new SSH session 2022-07-03T11:38:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=265] Command: which python3 2022-07-03T11:38:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=264] Received exit status 0 2022-07-03T11:38:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=264] Received channel close 2022-07-03T11:38:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=264] Channel closed 2022-07-03T11:38:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=265] Requesting new SSH session 2022-07-03T11:38:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=265] Command: which python3 2022-07-03T11:38:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=265] Received exit status 0 2022-07-03T11:38:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=265] Received channel close 2022-07-03T11:38:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=265] Channel closed 2022-07-03T11:38:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=266] Requesting new SSH session 2022-07-03T11:38:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=266] Command: sudo true 2022-07-03T11:38:38.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=265] Received exit status 0 2022-07-03T11:38:38.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=265] Received channel close 2022-07-03T11:38:38.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=265] Channel closed 2022-07-03T11:38:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=266] Requesting new SSH session 2022-07-03T11:38:38.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=266] Command: sudo true 2022-07-03T11:38:38.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=266] Received exit status 0 2022-07-03T11:38:38.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=266] Received channel close 2022-07-03T11:38:38.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=266] Channel closed 2022-07-03T11:38:38.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=267] Requesting new SSH session 2022-07-03T11:38:38.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:38.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=266] Received exit status 0 2022-07-03T11:38:38.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=266] Received channel close 2022-07-03T11:38:38.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=266] Channel closed 2022-07-03T11:38:38.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=267] Requesting new SSH session 2022-07-03T11:38:38.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:38.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=267] Received exit status 0 2022-07-03T11:38:38.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=267] Received channel close 2022-07-03T11:38:38.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=267] Channel closed 2022-07-03T11:38:38.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=267] Received exit status 0 2022-07-03T11:38:38.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=267] Received channel close 2022-07-03T11:38:38.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=267] Channel closed 2022-07-03T11:38:38.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=79] Login timeout expired 2022-07-03T11:38:38.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=79] Aborting connection 2022-07-03T11:38:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:38:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:38:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:38:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:38:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:38:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:38:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:38:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:38:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:38:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:38:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:38:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:38:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:38:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:38:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:38:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:38:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:38:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:38:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:38:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:38:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:38:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:38:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:38:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:38:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:38:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:38:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:38:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:38:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:38:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:38:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:38:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:38:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:38:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:38:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:38:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:38:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:38:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:38:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:38:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:38:38.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:38:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:38:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:38:38.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:38:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:38:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=79] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:38:38.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=79] Local address: 172.21.15.161, port 51244 2022-07-03T11:38:38.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=79] Peer address: 172.21.15.101, port 22 2022-07-03T11:38:38.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=264] Received exit status 0 2022-07-03T11:38:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=264] Received channel close 2022-07-03T11:38:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=264] Channel closed 2022-07-03T11:38:38.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=265] Requesting new SSH session 2022-07-03T11:38:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=265] Command: which python3 2022-07-03T11:38:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=264] Received exit status 0 2022-07-03T11:38:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=264] Received channel close 2022-07-03T11:38:38.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=264] Channel closed 2022-07-03T11:38:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=265] Requesting new SSH session 2022-07-03T11:38:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=265] Command: which python3 2022-07-03T11:38:38.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=265] Received exit status 0 2022-07-03T11:38:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=265] Received channel close 2022-07-03T11:38:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=265] Channel closed 2022-07-03T11:38:38.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=266] Requesting new SSH session 2022-07-03T11:38:38.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=266] Command: sudo true 2022-07-03T11:38:38.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=265] Received exit status 0 2022-07-03T11:38:38.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=265] Received channel close 2022-07-03T11:38:38.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=265] Channel closed 2022-07-03T11:38:38.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=266] Requesting new SSH session 2022-07-03T11:38:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=266] Command: sudo true 2022-07-03T11:38:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=266] Received exit status 0 2022-07-03T11:38:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=266] Received channel close 2022-07-03T11:38:38.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=266] Channel closed 2022-07-03T11:38:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=267] Requesting new SSH session 2022-07-03T11:38:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:38.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=266] Received exit status 0 2022-07-03T11:38:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=266] Received channel close 2022-07-03T11:38:38.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=266] Channel closed 2022-07-03T11:38:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=267] Requesting new SSH session 2022-07-03T11:38:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=267] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:38:38.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=267] Received exit status 0 2022-07-03T11:38:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=267] Received channel close 2022-07-03T11:38:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=267] Channel closed 2022-07-03T11:38:38.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=267] Received exit status 0 2022-07-03T11:38:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=267] Received channel close 2022-07-03T11:38:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=267] Channel closed 2022-07-03T11:38:38.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=79] Login timeout expired 2022-07-03T11:38:38.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=79] Aborting connection 2022-07-03T11:38:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:38 smithi161 ceph-mon[91579]: pgmap v1788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:38 smithi188 ceph-mon[92425]: pgmap v1788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:40 smithi161 ceph-mon[91579]: pgmap v1789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:40 smithi188 ceph-mon[92425]: pgmap v1789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:42 smithi161 ceph-mon[91579]: pgmap v1790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:42 smithi188 ceph-mon[92425]: pgmap v1790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:44 smithi161 ceph-mon[91579]: pgmap v1791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:44 smithi188 ceph-mon[92425]: pgmap v1791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:46 smithi161 ceph-mon[91579]: pgmap v1792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:46 smithi188 ceph-mon[92425]: pgmap v1792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:48 smithi161 ceph-mon[91579]: pgmap v1793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:48 smithi188 ceph-mon[92425]: pgmap v1793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:50 smithi161 ceph-mon[91579]: pgmap v1794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:50 smithi188 ceph-mon[92425]: pgmap v1794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:52 smithi161 ceph-mon[91579]: pgmap v1795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:52 smithi188 ceph-mon[92425]: pgmap v1795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:54 smithi161 ceph-mon[91579]: pgmap v1796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:54 smithi188 ceph-mon[92425]: pgmap v1796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:38:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:56 smithi161 ceph-mon[91579]: pgmap v1797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:56 smithi188 ceph-mon[92425]: pgmap v1797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:38:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:38:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:38:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:38:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:38:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:38:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:38:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:38:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:38:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:58 smithi161 ceph-mon[91579]: pgmap v1798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:38:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:58 smithi188 ceph-mon[92425]: pgmap v1798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:38:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:00 smithi161 ceph-mon[91579]: pgmap v1799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:00 smithi188 ceph-mon[92425]: pgmap v1799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:02 smithi161 ceph-mon[91579]: pgmap v1800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:02 smithi188 ceph-mon[92425]: pgmap v1800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:04 smithi161 ceph-mon[91579]: pgmap v1801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:04 smithi188 ceph-mon[92425]: pgmap v1801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:06 smithi161 ceph-mon[91579]: pgmap v1802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:06 smithi188 ceph-mon[92425]: pgmap v1802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:08 smithi161 ceph-mon[91579]: pgmap v1803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:08 smithi188 ceph-mon[92425]: pgmap v1803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:10 smithi161 ceph-mon[91579]: pgmap v1804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:10 smithi188 ceph-mon[92425]: pgmap v1804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:12 smithi161 ceph-mon[91579]: pgmap v1805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:12 smithi188 ceph-mon[92425]: pgmap v1805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:14 smithi161 ceph-mon[91579]: pgmap v1806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:14 smithi188 ceph-mon[92425]: pgmap v1806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:16 smithi161 ceph-mon[91579]: pgmap v1807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:16 smithi188 ceph-mon[92425]: pgmap v1807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:18 smithi161 ceph-mon[91579]: pgmap v1808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:18 smithi188 ceph-mon[92425]: pgmap v1808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:20 smithi161 ceph-mon[91579]: pgmap v1809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:20 smithi188 ceph-mon[92425]: pgmap v1809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:22 smithi161 ceph-mon[91579]: pgmap v1810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:22 smithi188 ceph-mon[92425]: pgmap v1810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:24 smithi161 ceph-mon[91579]: pgmap v1811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:24 smithi188 ceph-mon[92425]: pgmap v1811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:26 smithi161 ceph-mon[91579]: pgmap v1812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:26 smithi188 ceph-mon[92425]: pgmap v1812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:28 smithi161 ceph-mon[91579]: pgmap v1813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:28 smithi188 ceph-mon[92425]: pgmap v1813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:30 smithi161 ceph-mon[91579]: pgmap v1814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:30 smithi188 ceph-mon[92425]: pgmap v1814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:32 smithi161 ceph-mon[91579]: pgmap v1815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:32 smithi188 ceph-mon[92425]: pgmap v1815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:34 smithi161 ceph-mon[91579]: pgmap v1816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:34 smithi188 ceph-mon[92425]: pgmap v1816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-mon[91579]: pgmap v1817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:37.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:39:36.930+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:39:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:39:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:39:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:39:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:39:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:39:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:39:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:39:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:39:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:39:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:39:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:39:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:39:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:39:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:39:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:39:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:39:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:39:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:39:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:39:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:39:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:39:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:39:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:39:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:39:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:39:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:39:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:39:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:39:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:39:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:39:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:39:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:39:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:39:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:39:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:39:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:39:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:39:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:39:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:39:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:39:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:39:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:39:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:39:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:39:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=81] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:39:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=81] Local address: 172.21.15.161, port 51314 2022-07-03T11:39:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=81] Peer address: 172.21.15.101, port 22 2022-07-03T11:39:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=81] Connection lost 2022-07-03T11:39:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=81] Aborting connection 2022-07-03T11:39:37.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:39:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:39:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:36 smithi188 ceph-mon[92425]: pgmap v1817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:39:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:39:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:39:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:39:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:39:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:39:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:39:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:39:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:39:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:39:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:39:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:39:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:39:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:39:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:39:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:39:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:39:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:39:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:39:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:39:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:39:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:39:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:39:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:39:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:39:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:39:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:39:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:39:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:39:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:39:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:39:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:39:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:39:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:39:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:39:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:39:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:39:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:39:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:39:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:39:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:39:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:39:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:39:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:39:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:39:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:39:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:39:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:39:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:39:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=81] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:39:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=81] Local address: 172.21.15.161, port 51314 2022-07-03T11:39:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=81] Peer address: 172.21.15.101, port 22 2022-07-03T11:39:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=81] Connection lost 2022-07-03T11:39:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=81] Aborting connection 2022-07-03T11:39:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:39:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:39:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:39:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:39:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:39:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:39:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:39:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:39:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:39:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:39:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:39:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:39:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:39:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:39:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:39:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:39:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:39:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:39:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:39:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:39:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:39:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:39:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:39:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:39:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:39:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:39:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:39:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:39:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:39:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:39:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:39:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:39:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:39:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:39:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:39:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:39:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:39:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:39:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:39:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:39:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:39:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:39:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:39:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:39:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:39:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:39:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:39:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:39:38.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:39:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=81] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:39:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=81] Local address: 172.21.15.161, port 51314 2022-07-03T11:39:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=81] Peer address: 172.21.15.101, port 22 2022-07-03T11:39:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=81] Connection lost 2022-07-03T11:39:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=81] Aborting connection 2022-07-03T11:39:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: pgmap v1818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: pgmap v1818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:39:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:40 smithi161 ceph-mon[91579]: pgmap v1819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:40 smithi188 ceph-mon[92425]: pgmap v1819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:42 smithi161 ceph-mon[91579]: pgmap v1820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:42 smithi188 ceph-mon[92425]: pgmap v1820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:44 smithi161 ceph-mon[91579]: pgmap v1821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:44 smithi188 ceph-mon[92425]: pgmap v1821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:46 smithi161 ceph-mon[91579]: pgmap v1822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:46 smithi188 ceph-mon[92425]: pgmap v1822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:48 smithi161 ceph-mon[91579]: pgmap v1823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:48 smithi188 ceph-mon[92425]: pgmap v1823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:50 smithi161 ceph-mon[91579]: pgmap v1824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:50 smithi188 ceph-mon[92425]: pgmap v1824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:52 smithi161 ceph-mon[91579]: pgmap v1825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:52 smithi188 ceph-mon[92425]: pgmap v1825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:54 smithi161 ceph-mon[91579]: pgmap v1826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:54 smithi188 ceph-mon[92425]: pgmap v1826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:39:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:56 smithi161 ceph-mon[91579]: pgmap v1827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:56 smithi188 ceph-mon[92425]: pgmap v1827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:39:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:39:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:39:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:39:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:39:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:39:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:39:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:39:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:39:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:58 smithi161 ceph-mon[91579]: pgmap v1828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:39:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:58 smithi188 ceph-mon[92425]: pgmap v1828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:39:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:00 smithi161 ceph-mon[91579]: pgmap v1829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:40:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:00 smithi188 ceph-mon[92425]: pgmap v1829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:40:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:02 smithi161 ceph-mon[91579]: pgmap v1830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:02 smithi188 ceph-mon[92425]: pgmap v1830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:04 smithi161 ceph-mon[91579]: pgmap v1831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:04 smithi188 ceph-mon[92425]: pgmap v1831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:06 smithi161 ceph-mon[91579]: pgmap v1832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:06 smithi188 ceph-mon[92425]: pgmap v1832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:08 smithi161 ceph-mon[91579]: pgmap v1833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:08 smithi188 ceph-mon[92425]: pgmap v1833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:10 smithi161 ceph-mon[91579]: pgmap v1834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:10 smithi188 ceph-mon[92425]: pgmap v1834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:12 smithi161 ceph-mon[91579]: pgmap v1835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:12 smithi188 ceph-mon[92425]: pgmap v1835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:14 smithi161 ceph-mon[91579]: pgmap v1836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:14 smithi188 ceph-mon[92425]: pgmap v1836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:16 smithi161 ceph-mon[91579]: pgmap v1837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:16 smithi188 ceph-mon[92425]: pgmap v1837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:18 smithi161 ceph-mon[91579]: pgmap v1838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:18 smithi188 ceph-mon[92425]: pgmap v1838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:20 smithi161 ceph-mon[91579]: pgmap v1839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:20 smithi188 ceph-mon[92425]: pgmap v1839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:22 smithi161 ceph-mon[91579]: pgmap v1840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:22 smithi188 ceph-mon[92425]: pgmap v1840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:24 smithi161 ceph-mon[91579]: pgmap v1841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:24 smithi188 ceph-mon[92425]: pgmap v1841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:26 smithi161 ceph-mon[91579]: pgmap v1842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:26 smithi188 ceph-mon[92425]: pgmap v1842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:28 smithi161 ceph-mon[91579]: pgmap v1843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:28 smithi188 ceph-mon[92425]: pgmap v1843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:30 smithi161 ceph-mon[91579]: pgmap v1844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:30 smithi188 ceph-mon[92425]: pgmap v1844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:32 smithi161 ceph-mon[91579]: pgmap v1845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:32 smithi188 ceph-mon[92425]: pgmap v1845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:34 smithi161 ceph-mon[91579]: pgmap v1846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:34 smithi188 ceph-mon[92425]: pgmap v1846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:36 smithi161 ceph-mon[91579]: pgmap v1847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:36 smithi188 ceph-mon[92425]: pgmap v1847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:38 smithi161 ceph-mon[91579]: pgmap v1848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:40:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:40:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:40:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:38 smithi188 ceph-mon[92425]: pgmap v1848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:40:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:40:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:40:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:40 smithi161 ceph-mon[91579]: pgmap v1849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:40 smithi188 ceph-mon[92425]: pgmap v1849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:42 smithi161 ceph-mon[91579]: pgmap v1850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:42 smithi188 ceph-mon[92425]: pgmap v1850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:44 smithi161 ceph-mon[91579]: pgmap v1851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:44 smithi188 ceph-mon[92425]: pgmap v1851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:46 smithi161 ceph-mon[91579]: pgmap v1852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:46 smithi188 ceph-mon[92425]: pgmap v1852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:48 smithi161 ceph-mon[91579]: pgmap v1853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:48 smithi188 ceph-mon[92425]: pgmap v1853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:50 smithi161 ceph-mon[91579]: pgmap v1854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:50 smithi188 ceph-mon[92425]: pgmap v1854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:52 smithi161 ceph-mon[91579]: pgmap v1855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:52 smithi188 ceph-mon[92425]: pgmap v1855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:40:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:54 smithi161 ceph-mon[91579]: pgmap v1856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:54 smithi188 ceph-mon[92425]: pgmap v1856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:40:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:56 smithi161 ceph-mon[91579]: pgmap v1857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:40:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:56 smithi188 ceph-mon[92425]: pgmap v1857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:40:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:40:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:40:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:40:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:40:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:40:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:40:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:40:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:40:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:58 smithi161 ceph-mon[91579]: pgmap v1858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:40:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:58 smithi188 ceph-mon[92425]: pgmap v1858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:40:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:00 smithi161 ceph-mon[91579]: pgmap v1859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:41:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:00 smithi188 ceph-mon[92425]: pgmap v1859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:41:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:02 smithi161 ceph-mon[91579]: pgmap v1860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:41:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:02 smithi188 ceph-mon[92425]: pgmap v1860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:41:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:04 smithi161 ceph-mon[91579]: pgmap v1861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:04 smithi188 ceph-mon[92425]: pgmap v1861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:06 smithi161 ceph-mon[91579]: pgmap v1862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:06 smithi188 ceph-mon[92425]: pgmap v1862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:08.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:41:07.843+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:41:08.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:41:08.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:41:08.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:41:08.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:41:08.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:41:08.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:41:08.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:41:08.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:41:08.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:41:08.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:41:08.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:41:08.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:41:08.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:41:08.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:41:08.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:41:08.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:41:08.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:41:08.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:41:08.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:41:08.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:41:08.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:41:08.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:41:08.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:41:08.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:41:08.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:41:08.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:41:08.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:41:08.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:41:08.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:41:08.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:41:08.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:41:08.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:41:08.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:41:08.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:41:08.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:41:08.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:41:08.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:41:08.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:41:08.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:41:08.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:41:08.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:41:08.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:41:08.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:41:08.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:41:08.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=83] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:41:08.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=83] Local address: 172.21.15.161, port 51354 2022-07-03T11:41:08.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=83] Peer address: 172.21.15.101, port 22 2022-07-03T11:41:08.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=83] Connection lost 2022-07-03T11:41:08.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=83] Aborting connection 2022-07-03T11:41:08.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:41:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:41:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:08 smithi161 ceph-mon[91579]: pgmap v1863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:41:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:41:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:41:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:41:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:41:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:41:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:41:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:41:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:41:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:41:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:41:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:41:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:41:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:41:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:41:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:41:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:41:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:41:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:41:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:41:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:41:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:41:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:41:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:41:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:41:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:41:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:41:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:41:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:41:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:41:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:41:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:41:08.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:41:08.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:41:08.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:41:08.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:41:08.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:41:08.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:41:08.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:41:08.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:41:08.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:41:08.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:41:08.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:41:08.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:41:08.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:41:08.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:41:08.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=83] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:41:08.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=83] Local address: 172.21.15.161, port 51354 2022-07-03T11:41:08.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=83] Peer address: 172.21.15.101, port 22 2022-07-03T11:41:08.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=83] Connection lost 2022-07-03T11:41:08.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=83] Aborting connection 2022-07-03T11:41:08.662 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:08 smithi188 ceph-mon[92425]: pgmap v1863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:41:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:41:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:41:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:41:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:41:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:41:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:41:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:41:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:41:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:41:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:41:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:41:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:41:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:41:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:41:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:41:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:41:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:41:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:41:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:41:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:41:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:41:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:41:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:41:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:41:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:41:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:41:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:41:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:41:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:41:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:41:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:41:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:41:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:41:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:41:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:41:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:41:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:41:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:41:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:41:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:41:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:41:08.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:41:08.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:41:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:41:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:41:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=83] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:41:08.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=83] Local address: 172.21.15.161, port 51354 2022-07-03T11:41:08.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=83] Peer address: 172.21.15.101, port 22 2022-07-03T11:41:08.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=83] Connection lost 2022-07-03T11:41:08.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=83] Aborting connection 2022-07-03T11:41:08.694 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:10 smithi161 ceph-mon[91579]: pgmap v1864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:10 smithi188 ceph-mon[92425]: pgmap v1864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:12 smithi161 ceph-mon[91579]: pgmap v1865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:12 smithi188 ceph-mon[92425]: pgmap v1865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:14 smithi161 ceph-mon[91579]: pgmap v1866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:14 smithi188 ceph-mon[92425]: pgmap v1866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:16 smithi161 ceph-mon[91579]: pgmap v1867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:16 smithi188 ceph-mon[92425]: pgmap v1867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:18 smithi161 ceph-mon[91579]: pgmap v1868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:18 smithi188 ceph-mon[92425]: pgmap v1868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:20 smithi161 ceph-mon[91579]: pgmap v1869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:20 smithi188 ceph-mon[92425]: pgmap v1869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:22 smithi161 ceph-mon[91579]: pgmap v1870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:22 smithi188 ceph-mon[92425]: pgmap v1870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:24 smithi161 ceph-mon[91579]: pgmap v1871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:24 smithi188 ceph-mon[92425]: pgmap v1871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:26 smithi161 ceph-mon[91579]: pgmap v1872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:26 smithi188 ceph-mon[92425]: pgmap v1872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:28 smithi161 ceph-mon[91579]: pgmap v1873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:28 smithi188 ceph-mon[92425]: pgmap v1873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:30 smithi161 ceph-mon[91579]: pgmap v1874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:30 smithi188 ceph-mon[92425]: pgmap v1874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:32 smithi161 ceph-mon[91579]: pgmap v1875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:32 smithi188 ceph-mon[92425]: pgmap v1875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:34 smithi161 ceph-mon[91579]: pgmap v1876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:34 smithi188 ceph-mon[92425]: pgmap v1876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:36 smithi161 ceph-mon[91579]: pgmap v1877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:36 smithi188 ceph-mon[92425]: pgmap v1877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:38 smithi161 ceph-mon[91579]: pgmap v1878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:38 smithi188 ceph-mon[92425]: pgmap v1878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:40 smithi161 ceph-mon[91579]: pgmap v1879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:40 smithi188 ceph-mon[92425]: pgmap v1879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:42 smithi161 ceph-mon[91579]: pgmap v1880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:42 smithi188 ceph-mon[92425]: pgmap v1880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:44 smithi161 ceph-mon[91579]: pgmap v1881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:44 smithi188 ceph-mon[92425]: pgmap v1881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:46 smithi161 ceph-mon[91579]: pgmap v1882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:46 smithi188 ceph-mon[92425]: pgmap v1882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:48 smithi161 ceph-mon[91579]: pgmap v1883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:48 smithi188 ceph-mon[92425]: pgmap v1883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:50 smithi161 ceph-mon[91579]: pgmap v1884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:50 smithi188 ceph-mon[92425]: pgmap v1884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:52 smithi161 ceph-mon[91579]: pgmap v1885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:52 smithi188 ceph-mon[92425]: pgmap v1885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:54 smithi161 ceph-mon[91579]: pgmap v1886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:54 smithi188 ceph-mon[92425]: pgmap v1886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:41:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:56 smithi161 ceph-mon[91579]: pgmap v1887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:56 smithi188 ceph-mon[92425]: pgmap v1887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:41:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:41:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:41:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:41:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:41:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:41:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:41:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:41:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:41:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:58 smithi161 ceph-mon[91579]: pgmap v1888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:41:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:58 smithi188 ceph-mon[92425]: pgmap v1888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:41:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:00 smithi161 ceph-mon[91579]: pgmap v1889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:00 smithi188 ceph-mon[92425]: pgmap v1889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:02 smithi161 ceph-mon[91579]: pgmap v1890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:02 smithi188 ceph-mon[92425]: pgmap v1890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:04 smithi161 ceph-mon[91579]: pgmap v1891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:04 smithi188 ceph-mon[92425]: pgmap v1891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:06 smithi161 ceph-mon[91579]: pgmap v1892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:06 smithi188 ceph-mon[92425]: pgmap v1892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:08.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:42:07.880+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:42:08.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:42:08.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:42:08.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:42:08.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:42:08.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:42:08.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:42:08.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:42:08.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:42:08.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:42:08.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:42:08.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:42:08.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:42:08.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:42:08.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:42:08.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:42:08.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:42:08.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:42:08.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:42:08.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:42:08.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:42:08.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:42:08.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:42:08.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:42:08.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:42:08.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:42:08.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:42:08.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:42:08.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:42:08.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:42:08.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:42:08.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:42:08.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:42:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:42:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:42:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:42:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:42:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:42:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:42:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:42:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:42:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:42:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:42:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:42:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:42:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=85] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:42:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=85] Local address: 172.21.15.161, port 51382 2022-07-03T11:42:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=85] Peer address: 172.21.15.101, port 22 2022-07-03T11:42:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=85] Connection lost 2022-07-03T11:42:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=85] Aborting connection 2022-07-03T11:42:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:42:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:42:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: pgmap v1893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:42:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:42:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:42:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:42:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:42:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:42:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:42:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:42:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:42:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:42:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:42:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:42:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:42:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:42:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:42:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:42:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:42:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:42:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:42:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:42:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:42:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:42:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:42:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:42:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:42:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:42:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:42:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:42:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:42:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:42:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:42:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:42:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:42:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:42:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:42:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:42:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:42:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:42:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:42:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:42:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:42:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:42:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:42:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:42:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:42:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:42:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:42:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:42:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:42:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=85] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:42:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=85] Local address: 172.21.15.161, port 51382 2022-07-03T11:42:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=85] Peer address: 172.21.15.101, port 22 2022-07-03T11:42:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=85] Connection lost 2022-07-03T11:42:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=85] Aborting connection 2022-07-03T11:42:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: pgmap v1893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:42:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:42:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:42:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:42:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:42:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:42:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:42:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:42:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:42:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:42:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:42:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:42:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:42:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:42:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:42:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:42:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:42:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:42:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:42:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:42:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:42:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:42:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:42:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:42:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:42:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:42:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:42:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:42:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:42:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:42:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:42:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:42:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:42:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:42:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:42:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:42:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:42:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:42:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:42:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:42:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:42:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:42:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:42:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:42:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:42:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:42:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:42:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:42:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:42:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=85] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:42:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=85] Local address: 172.21.15.161, port 51382 2022-07-03T11:42:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=85] Peer address: 172.21.15.101, port 22 2022-07-03T11:42:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=85] Connection lost 2022-07-03T11:42:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=85] Aborting connection 2022-07-03T11:42:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:10 smithi161 ceph-mon[91579]: pgmap v1894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:10 smithi188 ceph-mon[92425]: pgmap v1894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:12 smithi161 ceph-mon[91579]: pgmap v1895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:12 smithi188 ceph-mon[92425]: pgmap v1895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:14 smithi161 ceph-mon[91579]: pgmap v1896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:14 smithi188 ceph-mon[92425]: pgmap v1896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:16 smithi161 ceph-mon[91579]: pgmap v1897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:16 smithi188 ceph-mon[92425]: pgmap v1897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:18 smithi161 ceph-mon[91579]: pgmap v1898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:18 smithi188 ceph-mon[92425]: pgmap v1898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:20 smithi188 ceph-mon[92425]: pgmap v1899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:20 smithi161 ceph-mon[91579]: pgmap v1899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:22 smithi161 ceph-mon[91579]: pgmap v1900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:22 smithi188 ceph-mon[92425]: pgmap v1900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:24 smithi161 ceph-mon[91579]: pgmap v1901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:24 smithi188 ceph-mon[92425]: pgmap v1901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:26 smithi161 ceph-mon[91579]: pgmap v1902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:26 smithi188 ceph-mon[92425]: pgmap v1902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:28 smithi161 ceph-mon[91579]: pgmap v1903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:28 smithi188 ceph-mon[92425]: pgmap v1903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:30 smithi161 ceph-mon[91579]: pgmap v1904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:30 smithi188 ceph-mon[92425]: pgmap v1904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:32 smithi161 ceph-mon[91579]: pgmap v1905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:32 smithi188 ceph-mon[92425]: pgmap v1905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:34 smithi161 ceph-mon[91579]: pgmap v1906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:34 smithi188 ceph-mon[92425]: pgmap v1906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:36 smithi161 ceph-mon[91579]: pgmap v1907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:36 smithi188 ceph-mon[92425]: pgmap v1907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:38 smithi161 ceph-mon[91579]: pgmap v1908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:38 smithi188 ceph-mon[92425]: pgmap v1908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:40 smithi188 ceph-mon[92425]: pgmap v1909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:40 smithi161 ceph-mon[91579]: pgmap v1909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:42 smithi161 ceph-mon[91579]: pgmap v1910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:42 smithi188 ceph-mon[92425]: pgmap v1910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:44 smithi161 ceph-mon[91579]: pgmap v1911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:44 smithi188 ceph-mon[92425]: pgmap v1911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:46 smithi161 ceph-mon[91579]: pgmap v1912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:46 smithi188 ceph-mon[92425]: pgmap v1912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:48 smithi161 ceph-mon[91579]: pgmap v1913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:48 smithi188 ceph-mon[92425]: pgmap v1913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:50 smithi188 ceph-mon[92425]: pgmap v1914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:50 smithi161 ceph-mon[91579]: pgmap v1914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:52 smithi161 ceph-mon[91579]: pgmap v1915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:52 smithi188 ceph-mon[92425]: pgmap v1915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:54 smithi161 ceph-mon[91579]: pgmap v1916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:54 smithi188 ceph-mon[92425]: pgmap v1916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:42:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:56 smithi161 ceph-mon[91579]: pgmap v1917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:56 smithi188 ceph-mon[92425]: pgmap v1917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:42:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:42:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:42:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:42:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:42:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:42:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:42:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:42:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:42:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:58 smithi161 ceph-mon[91579]: pgmap v1918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:42:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:58 smithi188 ceph-mon[92425]: pgmap v1918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:42:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:00 smithi161 ceph-mon[91579]: pgmap v1919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:00 smithi188 ceph-mon[92425]: pgmap v1919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:02 smithi161 ceph-mon[91579]: pgmap v1920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:02 smithi188 ceph-mon[92425]: pgmap v1920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:04 smithi188 ceph-mon[92425]: pgmap v1921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:04 smithi161 ceph-mon[91579]: pgmap v1921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:06 smithi188 ceph-mon[92425]: pgmap v1922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:06 smithi161 ceph-mon[91579]: pgmap v1922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:08 smithi188 ceph-mon[92425]: pgmap v1923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:43:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:43:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:43:08.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-mon[91579]: pgmap v1923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:08.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:08.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:43:08.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:43:08.842 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:43:08.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:43:08.394+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:43:08.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:43:08.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:43:08.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:43:08.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:43:08.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:43:08.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:43:08.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:43:08.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:43:08.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:43:08.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:43:08.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:43:08.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:43:08.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:43:08.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:43:08.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:43:08.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:43:08.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:43:08.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:43:08.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:43:08.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:43:08.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:43:08.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:43:08.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:43:08.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:43:08.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:43:08.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:43:08.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:43:08.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:43:08.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:43:08.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:43:08.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:43:08.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:43:08.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:43:08.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:43:08.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:43:08.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:43:08.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:43:08.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:43:08.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:43:08.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:43:08.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:43:08.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:43:08.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:43:08.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:43:08.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=87] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:43:08.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=87] Local address: 172.21.15.161, port 51412 2022-07-03T11:43:08.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=87] Peer address: 172.21.15.101, port 22 2022-07-03T11:43:08.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=87] Connection lost 2022-07-03T11:43:08.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=87] Aborting connection 2022-07-03T11:43:08.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:43:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:43:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:43:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:43:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:43:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:43:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:43:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:43:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:43:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:43:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:43:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:43:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:43:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:43:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:43:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:43:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:43:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:43:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:43:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:43:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:43:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:43:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:43:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:43:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:43:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:43:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:43:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:43:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:43:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:43:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:43:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:43:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:43:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:43:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:43:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:43:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:43:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:43:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:43:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:43:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:43:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:43:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:43:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:43:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:43:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:43:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:43:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:43:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=87] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:43:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=87] Local address: 172.21.15.161, port 51412 2022-07-03T11:43:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=87] Peer address: 172.21.15.101, port 22 2022-07-03T11:43:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=87] Connection lost 2022-07-03T11:43:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=87] Aborting connection 2022-07-03T11:43:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:43:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:43:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:43:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:43:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:43:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:43:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:43:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:43:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:43:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:43:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:43:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:43:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:43:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:43:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:43:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:43:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:43:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:43:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:43:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:43:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:43:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:43:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:43:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:43:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:43:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:43:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:43:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:43:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:43:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:43:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:43:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:43:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:43:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:43:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:43:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:43:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:43:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:43:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:43:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:43:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:43:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:43:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:43:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:43:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:43:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:43:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=87] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:43:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=87] Local address: 172.21.15.161, port 51412 2022-07-03T11:43:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=87] Peer address: 172.21.15.101, port 22 2022-07-03T11:43:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=87] Connection lost 2022-07-03T11:43:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=87] Aborting connection 2022-07-03T11:43:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:43:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:10 smithi161 ceph-mon[91579]: pgmap v1924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:10 smithi188 ceph-mon[92425]: pgmap v1924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:12 smithi161 ceph-mon[91579]: pgmap v1925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:12 smithi188 ceph-mon[92425]: pgmap v1925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:14 smithi161 ceph-mon[91579]: pgmap v1926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:14 smithi188 ceph-mon[92425]: pgmap v1926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:16 smithi161 ceph-mon[91579]: pgmap v1927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:16 smithi188 ceph-mon[92425]: pgmap v1927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:18 smithi188 ceph-mon[92425]: pgmap v1928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:18 smithi161 ceph-mon[91579]: pgmap v1928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:20 smithi188 ceph-mon[92425]: pgmap v1929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:20 smithi161 ceph-mon[91579]: pgmap v1929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:22 smithi161 ceph-mon[91579]: pgmap v1930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:22 smithi188 ceph-mon[92425]: pgmap v1930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:24 smithi161 ceph-mon[91579]: pgmap v1931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:24 smithi188 ceph-mon[92425]: pgmap v1931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:26 smithi161 ceph-mon[91579]: pgmap v1932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:26 smithi188 ceph-mon[92425]: pgmap v1932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:28 smithi188 ceph-mon[92425]: pgmap v1933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:28 smithi161 ceph-mon[91579]: pgmap v1933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:30 smithi188 ceph-mon[92425]: pgmap v1934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:30 smithi161 ceph-mon[91579]: pgmap v1934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:32 smithi161 ceph-mon[91579]: pgmap v1935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:32 smithi188 ceph-mon[92425]: pgmap v1935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:34 smithi161 ceph-mon[91579]: pgmap v1936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:34 smithi188 ceph-mon[92425]: pgmap v1936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:36 smithi161 ceph-mon[91579]: pgmap v1937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:36 smithi188 ceph-mon[92425]: pgmap v1937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:38 smithi188 ceph-mon[92425]: pgmap v1938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:38 smithi161 ceph-mon[91579]: pgmap v1938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:40 smithi188 ceph-mon[92425]: pgmap v1939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:40 smithi161 ceph-mon[91579]: pgmap v1939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:42 smithi161 ceph-mon[91579]: pgmap v1940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:42 smithi188 ceph-mon[92425]: pgmap v1940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:44 smithi161 ceph-mon[91579]: pgmap v1941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:44 smithi188 ceph-mon[92425]: pgmap v1941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:46 smithi161 ceph-mon[91579]: pgmap v1942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:46 smithi188 ceph-mon[92425]: pgmap v1942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:48 smithi188 ceph-mon[92425]: pgmap v1943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:48 smithi161 ceph-mon[91579]: pgmap v1943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:50 smithi188 ceph-mon[92425]: pgmap v1944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:50 smithi161 ceph-mon[91579]: pgmap v1944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:52 smithi161 ceph-mon[91579]: pgmap v1945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:52 smithi188 ceph-mon[92425]: pgmap v1945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:54 smithi161 ceph-mon[91579]: pgmap v1946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:54 smithi188 ceph-mon[92425]: pgmap v1946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:43:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:56 smithi161 ceph-mon[91579]: pgmap v1947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:56 smithi188 ceph-mon[92425]: pgmap v1947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:43:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:43:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:43:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:43:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:43:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:43:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:43:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:43:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:43:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:58 smithi188 ceph-mon[92425]: pgmap v1948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:43:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:58 smithi161 ceph-mon[91579]: pgmap v1948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:43:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:00 smithi188 ceph-mon[92425]: pgmap v1949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:00 smithi161 ceph-mon[91579]: pgmap v1949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:02 smithi188 ceph-mon[92425]: pgmap v1950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:02 smithi161 ceph-mon[91579]: pgmap v1950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:04 smithi188 ceph-mon[92425]: pgmap v1951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:04 smithi161 ceph-mon[91579]: pgmap v1951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:06 smithi188 ceph-mon[92425]: pgmap v1952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:06 smithi161 ceph-mon[91579]: pgmap v1952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:08 smithi188 ceph-mon[92425]: pgmap v1953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:08 smithi161 ceph-mon[91579]: pgmap v1953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:44:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:44:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:44:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:09.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:44:09.318+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:44:09.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:44:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:44:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:44:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:44:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:44:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:44:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:44:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:44:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:44:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:44:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:44:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:44:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:44:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:44:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:44:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:44:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:44:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:44:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:44:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:44:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:44:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:44:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:44:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:44:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:44:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:44:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:44:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:44:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:44:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:44:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:44:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:44:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:44:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:44:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:44:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:44:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:44:09.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:44:09.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:44:09.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:44:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:44:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:44:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:44:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:44:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=89] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:44:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=89] Local address: 172.21.15.161, port 51440 2022-07-03T11:44:09.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=89] Peer address: 172.21.15.101, port 22 2022-07-03T11:44:09.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=89] Connection lost 2022-07-03T11:44:09.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=89] Aborting connection 2022-07-03T11:44:09.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:44:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:44:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:44:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:44:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:44:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:44:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:44:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:44:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:44:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:44:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:44:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:44:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:44:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:44:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:44:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:44:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:44:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:44:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:44:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:44:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:44:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:44:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:44:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:44:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:44:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:44:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:44:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:44:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:44:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:44:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:44:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:44:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:44:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:44:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:44:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:44:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:44:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:44:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:44:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:44:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:44:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:44:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:44:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:44:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:44:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:44:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:44:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:44:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:44:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:44:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:44:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=89] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:44:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=89] Local address: 172.21.15.161, port 51440 2022-07-03T11:44:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=89] Peer address: 172.21.15.101, port 22 2022-07-03T11:44:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=89] Connection lost 2022-07-03T11:44:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=89] Aborting connection 2022-07-03T11:44:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:10 smithi188 ceph-mon[92425]: pgmap v1954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:44:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:44:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:44:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:44:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:44:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:44:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:44:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:44:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:44:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:44:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:44:10.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:44:10.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:44:10.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:44:10.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:44:10.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:44:10.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:44:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:44:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:44:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:44:10.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:44:10.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:44:10.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:44:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:44:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:44:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:44:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:44:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:44:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:44:10.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:44:10.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:44:10.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:44:10.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:44:10.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:44:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:44:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:44:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:44:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:44:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:44:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:44:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:44:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:44:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:44:10.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:44:10.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:44:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:44:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=89] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:44:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=89] Local address: 172.21.15.161, port 51440 2022-07-03T11:44:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=89] Peer address: 172.21.15.101, port 22 2022-07-03T11:44:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=89] Connection lost 2022-07-03T11:44:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=89] Aborting connection 2022-07-03T11:44:10.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:10 smithi161 ceph-mon[91579]: pgmap v1954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: pgmap v1955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: pgmap v1955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:44:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:44:13.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:13 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:44:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:13 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:44:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:13 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:44:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:13 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:44:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:13 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:44:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:13 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:44:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:13 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:44:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:13 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:44:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:14 smithi161 ceph-mon[91579]: pgmap v1956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:14 smithi188 ceph-mon[92425]: pgmap v1956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:16 smithi161 ceph-mon[91579]: pgmap v1957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:16 smithi188 ceph-mon[92425]: pgmap v1957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:18 smithi188 ceph-mon[92425]: pgmap v1958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:18 smithi161 ceph-mon[91579]: pgmap v1958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:20 smithi188 ceph-mon[92425]: pgmap v1959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:20 smithi161 ceph-mon[91579]: pgmap v1959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:22 smithi161 ceph-mon[91579]: pgmap v1960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:22 smithi188 ceph-mon[92425]: pgmap v1960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:24 smithi188 ceph-mon[92425]: pgmap v1961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:24 smithi161 ceph-mon[91579]: pgmap v1961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:26 smithi188 ceph-mon[92425]: pgmap v1962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:26 smithi161 ceph-mon[91579]: pgmap v1962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:28 smithi188 ceph-mon[92425]: pgmap v1963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:28 smithi161 ceph-mon[91579]: pgmap v1963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:30 smithi188 ceph-mon[92425]: pgmap v1964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:30 smithi161 ceph-mon[91579]: pgmap v1964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:32 smithi188 ceph-mon[92425]: pgmap v1965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:32 smithi161 ceph-mon[91579]: pgmap v1965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:34 smithi188 ceph-mon[92425]: pgmap v1966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:34 smithi161 ceph-mon[91579]: pgmap v1966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:36 smithi188 ceph-mon[92425]: pgmap v1967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:36 smithi161 ceph-mon[91579]: pgmap v1967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:38 smithi188 ceph-mon[92425]: pgmap v1968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:38 smithi161 ceph-mon[91579]: pgmap v1968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:40 smithi188 ceph-mon[92425]: pgmap v1969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:40 smithi161 ceph-mon[91579]: pgmap v1969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:42 smithi188 ceph-mon[92425]: pgmap v1970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:42 smithi161 ceph-mon[91579]: pgmap v1970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:44 smithi188 ceph-mon[92425]: pgmap v1971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:44 smithi161 ceph-mon[91579]: pgmap v1971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:46 smithi188 ceph-mon[92425]: pgmap v1972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:46 smithi161 ceph-mon[91579]: pgmap v1972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:48 smithi188 ceph-mon[92425]: pgmap v1973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:48 smithi161 ceph-mon[91579]: pgmap v1973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:50 smithi161 ceph-mon[91579]: pgmap v1974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:50 smithi188 ceph-mon[92425]: pgmap v1974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:52 smithi188 ceph-mon[92425]: pgmap v1975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:52 smithi161 ceph-mon[91579]: pgmap v1975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:54 smithi188 ceph-mon[92425]: pgmap v1976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:54 smithi161 ceph-mon[91579]: pgmap v1976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:44:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:56 smithi188 ceph-mon[92425]: pgmap v1977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:56 smithi161 ceph-mon[91579]: pgmap v1977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:44:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:44:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:44:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:44:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:44:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:44:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:44:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:44:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:44:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:58 smithi188 ceph-mon[92425]: pgmap v1978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:44:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:58 smithi161 ceph-mon[91579]: pgmap v1978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:44:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:00 smithi188 ceph-mon[92425]: pgmap v1979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:00 smithi161 ceph-mon[91579]: pgmap v1979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:02 smithi188 ceph-mon[92425]: pgmap v1980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:02 smithi161 ceph-mon[91579]: pgmap v1980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:04 smithi161 ceph-mon[91579]: pgmap v1981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:04 smithi188 ceph-mon[92425]: pgmap v1981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:06 smithi161 ceph-mon[91579]: pgmap v1982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:06 smithi188 ceph-mon[92425]: pgmap v1982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:08 smithi188 ceph-mon[92425]: pgmap v1983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:08 smithi161 ceph-mon[91579]: pgmap v1983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:10 smithi161 ceph-mon[91579]: pgmap v1984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:10 smithi188 ceph-mon[92425]: pgmap v1984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-mon[91579]: pgmap v1985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:45:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:45:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:45:13.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:45:12.670+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:45:13.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:45:13.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:45:13.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:45:13.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:45:13.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:45:13.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:45:13.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:45:13.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:45:13.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:45:13.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:45:13.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:45:13.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:45:13.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:45:13.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:45:13.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:45:13.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:45:13.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:45:13.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:45:13.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:45:13.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:45:13.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:45:13.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:45:13.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:45:13.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:45:13.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:45:13.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:45:13.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:45:13.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:45:13.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:45:13.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:45:13.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:45:13.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:45:13.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:45:13.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:45:13.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:45:13.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:45:13.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:45:13.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:45:13.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:45:13.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:45:13.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:45:13.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:45:13.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:45:13.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:45:13.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=91] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:45:13.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=91] Local address: 172.21.15.161, port 51468 2022-07-03T11:45:13.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=91] Peer address: 172.21.15.101, port 22 2022-07-03T11:45:13.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=91] Connection lost 2022-07-03T11:45:13.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=91] Aborting connection 2022-07-03T11:45:13.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:45:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:45:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:12 smithi188 ceph-mon[92425]: pgmap v1985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:45:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:45:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:45:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:13 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:45:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:13 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:45:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:45:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:45:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:45:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:45:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:45:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:45:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:45:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:45:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:45:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:45:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:45:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:45:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:45:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:45:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:45:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:45:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:45:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:45:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:45:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:45:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:45:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:45:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:45:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:45:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:45:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:45:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:45:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:45:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:45:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:45:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:45:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:45:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:45:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:45:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:45:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:45:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:45:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:45:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:45:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:45:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:45:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:45:14.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:45:14.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:45:14.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=91] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:45:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=91] Local address: 172.21.15.161, port 51468 2022-07-03T11:45:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=91] Peer address: 172.21.15.101, port 22 2022-07-03T11:45:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=91] Connection lost 2022-07-03T11:45:14.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=91] Aborting connection 2022-07-03T11:45:14.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:13 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:45:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:13 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:45:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:45:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:45:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:45:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:45:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:45:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:45:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:45:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:45:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:45:14.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:45:14.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:45:14.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:45:14.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:45:14.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:45:14.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:45:14.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:45:14.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:45:14.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:45:14.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:45:14.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:45:14.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:45:14.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:45:14.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:45:14.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:45:14.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:45:14.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:45:14.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:45:14.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:45:14.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:45:14.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:45:14.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:45:14.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:45:14.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:45:14.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:45:14.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:45:14.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:45:14.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:45:14.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:45:14.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:45:14.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:45:14.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:45:14.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:45:14.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:45:14.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:45:14.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=91] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:45:14.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=91] Local address: 172.21.15.161, port 51468 2022-07-03T11:45:14.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=91] Peer address: 172.21.15.101, port 22 2022-07-03T11:45:14.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=91] Connection lost 2022-07-03T11:45:14.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=91] Aborting connection 2022-07-03T11:45:14.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:14 smithi188 ceph-mon[92425]: pgmap v1986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:14 smithi161 ceph-mon[91579]: pgmap v1986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:16 smithi188 ceph-mon[92425]: pgmap v1987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:16 smithi161 ceph-mon[91579]: pgmap v1987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:18 smithi161 ceph-mon[91579]: pgmap v1988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:18 smithi188 ceph-mon[92425]: pgmap v1988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:20 smithi161 ceph-mon[91579]: pgmap v1989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:20 smithi188 ceph-mon[92425]: pgmap v1989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:22 smithi188 ceph-mon[92425]: pgmap v1990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:22 smithi161 ceph-mon[91579]: pgmap v1990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:24 smithi188 ceph-mon[92425]: pgmap v1991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:24 smithi161 ceph-mon[91579]: pgmap v1991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:26 smithi188 ceph-mon[92425]: pgmap v1992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:26 smithi161 ceph-mon[91579]: pgmap v1992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:28 smithi161 ceph-mon[91579]: pgmap v1993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:28 smithi188 ceph-mon[92425]: pgmap v1993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:30 smithi161 ceph-mon[91579]: pgmap v1994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:30 smithi188 ceph-mon[92425]: pgmap v1994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:32 smithi188 ceph-mon[92425]: pgmap v1995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:32 smithi161 ceph-mon[91579]: pgmap v1995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:34 smithi188 ceph-mon[92425]: pgmap v1996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:34 smithi161 ceph-mon[91579]: pgmap v1996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:36 smithi188 ceph-mon[92425]: pgmap v1997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:36 smithi161 ceph-mon[91579]: pgmap v1997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:38 smithi161 ceph-mon[91579]: pgmap v1998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:38 smithi188 ceph-mon[92425]: pgmap v1998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:40 smithi161 ceph-mon[91579]: pgmap v1999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:40 smithi188 ceph-mon[92425]: pgmap v1999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:42 smithi188 ceph-mon[92425]: pgmap v2000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:42 smithi161 ceph-mon[91579]: pgmap v2000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:44 smithi188 ceph-mon[92425]: pgmap v2001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:44 smithi161 ceph-mon[91579]: pgmap v2001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:46 smithi188 ceph-mon[92425]: pgmap v2002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:46 smithi161 ceph-mon[91579]: pgmap v2002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:48 smithi161 ceph-mon[91579]: pgmap v2003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:48 smithi188 ceph-mon[92425]: pgmap v2003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:50 smithi161 ceph-mon[91579]: pgmap v2004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:50 smithi188 ceph-mon[92425]: pgmap v2004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:52 smithi188 ceph-mon[92425]: pgmap v2005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:52 smithi161 ceph-mon[91579]: pgmap v2005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:54 smithi188 ceph-mon[92425]: pgmap v2006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:54 smithi161 ceph-mon[91579]: pgmap v2006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:56 smithi188 ceph-mon[92425]: pgmap v2007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:45:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:45:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:45:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:45:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:56 smithi161 ceph-mon[91579]: pgmap v2007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:45:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:45:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:45:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:45:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:45:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:58 smithi161 ceph-mon[91579]: pgmap v2008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:58 smithi188 ceph-mon[92425]: pgmap v2008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:45:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:45:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:45:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:00 smithi161 ceph-mon[91579]: pgmap v2009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:00 smithi188 ceph-mon[92425]: pgmap v2009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:02 smithi188 ceph-mon[92425]: pgmap v2010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:02 smithi161 ceph-mon[91579]: pgmap v2010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:04 smithi161 ceph-mon[91579]: pgmap v2011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:04 smithi188 ceph-mon[92425]: pgmap v2011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:06 smithi161 ceph-mon[91579]: pgmap v2012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:06 smithi188 ceph-mon[92425]: pgmap v2012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:08 smithi161 ceph-mon[91579]: pgmap v2013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:08 smithi188 ceph-mon[92425]: pgmap v2013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:10 smithi161 ceph-mon[91579]: pgmap v2014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:10 smithi188 ceph-mon[92425]: pgmap v2014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:12 smithi188 ceph-mon[92425]: pgmap v2015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:13.182 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:12 smithi161 ceph-mon[91579]: pgmap v2015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:13.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:46:13.570+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:46:13.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:46:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:46:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:46:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:46:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:46:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:46:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:46:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:46:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:46:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:46:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:46:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:46:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:46:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:46:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:14 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:46:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:46:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:46:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:46:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:46:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:46:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:46:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:46:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:46:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:46:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:14 smithi161 ceph-mon[91579]: pgmap v2016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:14 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:46:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:46:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:46:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:46:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:46:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:46:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:46:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:46:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:46:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:46:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:14 smithi188 ceph-mon[92425]: pgmap v2016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:16 smithi161 ceph-mon[91579]: pgmap v2017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:16 smithi188 ceph-mon[92425]: pgmap v2017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:18 smithi161 ceph-mon[91579]: pgmap v2018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:18 smithi188 ceph-mon[92425]: pgmap v2018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:20 smithi161 ceph-mon[91579]: pgmap v2019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:20 smithi188 ceph-mon[92425]: pgmap v2019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:22 smithi161 ceph-mon[91579]: pgmap v2020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:22 smithi188 ceph-mon[92425]: pgmap v2020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:24 smithi161 ceph-mon[91579]: pgmap v2021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:24 smithi188 ceph-mon[92425]: pgmap v2021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:26 smithi161 ceph-mon[91579]: pgmap v2022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:26 smithi188 ceph-mon[92425]: pgmap v2022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:28 smithi161 ceph-mon[91579]: pgmap v2023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:28 smithi188 ceph-mon[92425]: pgmap v2023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:30 smithi161 ceph-mon[91579]: pgmap v2024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:30 smithi188 ceph-mon[92425]: pgmap v2024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:32 smithi161 ceph-mon[91579]: pgmap v2025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:32 smithi188 ceph-mon[92425]: pgmap v2025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:34 smithi161 ceph-mon[91579]: pgmap v2026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:34 smithi188 ceph-mon[92425]: pgmap v2026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:36 smithi161 ceph-mon[91579]: pgmap v2027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:36 smithi188 ceph-mon[92425]: pgmap v2027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:38 smithi161 ceph-mon[91579]: pgmap v2028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:38 smithi188 ceph-mon[92425]: pgmap v2028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:40 smithi161 ceph-mon[91579]: pgmap v2029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:40 smithi188 ceph-mon[92425]: pgmap v2029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:42 smithi161 ceph-mon[91579]: pgmap v2030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:42 smithi188 ceph-mon[92425]: pgmap v2030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:43.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:46:43.585+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:46:43.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:46:43.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:43.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:46:43.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:43.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:43.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:43.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:46:43.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:43.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:46:43.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:43.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:46:43.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:46:43.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:46:43.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:46:43.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:46:43.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:46:43.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:46:43.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:46:43.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:46:43.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:46:43.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:46:43.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:46:43.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:46:43.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:46:43.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:46:43.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:46:43.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:46:43.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:46:43.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:46:43.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:46:43.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:46:43.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:46:43.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:46:43.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:46:43.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:46:43.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:46:43.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:46:43.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:46:43.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:46:43.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:46:43.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:46:43.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:46:43.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:46:43.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:46:43.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=93] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:46:43.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=93] Local address: 172.21.15.161, port 51498 2022-07-03T11:46:43.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=93] Peer address: 172.21.15.101, port 22 2022-07-03T11:46:43.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=307] Received exit status 0 2022-07-03T11:46:43.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=307] Received channel close 2022-07-03T11:46:43.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=307] Channel closed 2022-07-03T11:46:43.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=307] Received exit status 0 2022-07-03T11:46:43.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=307] Received channel close 2022-07-03T11:46:43.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=307] Channel closed 2022-07-03T11:46:43.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=93] Login timeout expired 2022-07-03T11:46:43.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=93] Aborting connection 2022-07-03T11:46:43.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:46:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:46:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:46:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:46:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:46:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:46:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:46:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:46:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:46:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:46:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:46:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:46:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:46:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:46:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:46:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:46:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:46:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:46:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:46:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:46:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:46:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:46:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:46:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:46:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:46:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:46:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:46:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:46:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:46:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:46:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:46:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:46:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:46:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:46:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:46:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:46:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:46:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:46:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:46:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:46:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=93] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:46:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=93] Local address: 172.21.15.161, port 51498 2022-07-03T11:46:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=93] Peer address: 172.21.15.101, port 22 2022-07-03T11:46:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=307] Received exit status 0 2022-07-03T11:46:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=307] Received channel close 2022-07-03T11:46:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=307] Channel closed 2022-07-03T11:46:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=307] Received exit status 0 2022-07-03T11:46:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=307] Received channel close 2022-07-03T11:46:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=307] Channel closed 2022-07-03T11:46:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=93] Login timeout expired 2022-07-03T11:46:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=93] Aborting connection 2022-07-03T11:46:45.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:44 smithi161 ceph-mon[91579]: pgmap v2031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:45.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:46:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:46:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:46:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:46:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:46:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:46:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:46:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:46:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:46:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:46:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:46:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:46:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:46:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:46:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:46:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:46:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:46:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:46:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:46:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:46:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:46:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:46:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:46:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:46:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:46:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:46:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:46:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:46:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:46:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:46:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:46:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:46:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:46:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:46:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:46:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:46:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:46:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:46:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:46:45.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:46:45.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:46:45.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:46:45.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:46:45.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:46:45.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:46:45.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=93] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:46:45.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=93] Local address: 172.21.15.161, port 51498 2022-07-03T11:46:45.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=93] Peer address: 172.21.15.101, port 22 2022-07-03T11:46:45.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=307] Received exit status 0 2022-07-03T11:46:45.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=307] Received channel close 2022-07-03T11:46:45.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=307] Channel closed 2022-07-03T11:46:45.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=307] Received exit status 0 2022-07-03T11:46:45.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=307] Received channel close 2022-07-03T11:46:45.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=307] Channel closed 2022-07-03T11:46:45.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=93] Login timeout expired 2022-07-03T11:46:45.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=93] Aborting connection 2022-07-03T11:46:45.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:44 smithi188 ceph-mon[92425]: pgmap v2031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:45.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:46 smithi161 ceph-mon[91579]: pgmap v2032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:46 smithi188 ceph-mon[92425]: pgmap v2032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:48 smithi161 ceph-mon[91579]: pgmap v2033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:48 smithi188 ceph-mon[92425]: pgmap v2033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:50 smithi161 ceph-mon[91579]: pgmap v2034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:50 smithi188 ceph-mon[92425]: pgmap v2034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:52 smithi161 ceph-mon[91579]: pgmap v2035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:52 smithi188 ceph-mon[92425]: pgmap v2035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:54 smithi161 ceph-mon[91579]: pgmap v2036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:54 smithi188 ceph-mon[92425]: pgmap v2036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:56 smithi161 ceph-mon[91579]: pgmap v2037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:46:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:46:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:46:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:46:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:56 smithi188 ceph-mon[92425]: pgmap v2037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:46:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:46:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:46:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:46:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:46:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:58 smithi161 ceph-mon[91579]: pgmap v2038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:58 smithi188 ceph-mon[92425]: pgmap v2038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:46:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:46:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:46:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:00 smithi161 ceph-mon[91579]: pgmap v2039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:00 smithi188 ceph-mon[92425]: pgmap v2039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:02 smithi161 ceph-mon[91579]: pgmap v2040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:02 smithi188 ceph-mon[92425]: pgmap v2040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:04 smithi161 ceph-mon[91579]: pgmap v2041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:04 smithi188 ceph-mon[92425]: pgmap v2041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:06 smithi161 ceph-mon[91579]: pgmap v2042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:06 smithi188 ceph-mon[92425]: pgmap v2042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:08 smithi161 ceph-mon[91579]: pgmap v2043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:08 smithi188 ceph-mon[92425]: pgmap v2043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:10 smithi161 ceph-mon[91579]: pgmap v2044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:10 smithi188 ceph-mon[92425]: pgmap v2044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:12 smithi161 ceph-mon[91579]: pgmap v2045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:12 smithi188 ceph-mon[92425]: pgmap v2045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:14 smithi161 ceph-mon[91579]: pgmap v2046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:14 smithi188 ceph-mon[92425]: pgmap v2046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:16 smithi161 ceph-mon[91579]: pgmap v2047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:16 smithi188 ceph-mon[92425]: pgmap v2047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:18 smithi161 ceph-mon[91579]: pgmap v2048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:18 smithi188 ceph-mon[92425]: pgmap v2048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:20 smithi161 ceph-mon[91579]: pgmap v2049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:20 smithi188 ceph-mon[92425]: pgmap v2049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:22 smithi161 ceph-mon[91579]: pgmap v2050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:22 smithi188 ceph-mon[92425]: pgmap v2050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:24 smithi161 ceph-mon[91579]: pgmap v2051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:24 smithi188 ceph-mon[92425]: pgmap v2051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:26 smithi161 ceph-mon[91579]: pgmap v2052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:26 smithi188 ceph-mon[92425]: pgmap v2052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:28 smithi161 ceph-mon[91579]: pgmap v2053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:28 smithi188 ceph-mon[92425]: pgmap v2053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:30 smithi161 ceph-mon[91579]: pgmap v2054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:30 smithi188 ceph-mon[92425]: pgmap v2054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:32 smithi161 ceph-mon[91579]: pgmap v2055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:32 smithi188 ceph-mon[92425]: pgmap v2055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:34 smithi161 ceph-mon[91579]: pgmap v2056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:34 smithi188 ceph-mon[92425]: pgmap v2056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:36 smithi161 ceph-mon[91579]: pgmap v2057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:36 smithi188 ceph-mon[92425]: pgmap v2057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:38 smithi161 ceph-mon[91579]: pgmap v2058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:38 smithi188 ceph-mon[92425]: pgmap v2058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:40 smithi161 ceph-mon[91579]: pgmap v2059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:40 smithi188 ceph-mon[92425]: pgmap v2059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:42 smithi161 ceph-mon[91579]: pgmap v2060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:42 smithi188 ceph-mon[92425]: pgmap v2060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:47:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:47:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:47:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:47:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:47:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:47:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:44 smithi161 ceph-mon[91579]: pgmap v2061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:44 smithi188 ceph-mon[92425]: pgmap v2061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:46 smithi161 ceph-mon[91579]: pgmap v2062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:46 smithi188 ceph-mon[92425]: pgmap v2062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:48 smithi161 ceph-mon[91579]: pgmap v2063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:48 smithi188 ceph-mon[92425]: pgmap v2063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:50 smithi161 ceph-mon[91579]: pgmap v2064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:50 smithi188 ceph-mon[92425]: pgmap v2064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:52 smithi161 ceph-mon[91579]: pgmap v2065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:52 smithi188 ceph-mon[92425]: pgmap v2065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:54 smithi161 ceph-mon[91579]: pgmap v2066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:54 smithi188 ceph-mon[92425]: pgmap v2066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:56 smithi161 ceph-mon[91579]: pgmap v2067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:47:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:47:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:47:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:47:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:56 smithi188 ceph-mon[92425]: pgmap v2067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:47:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:47:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:47:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:47:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:47:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:58 smithi161 ceph-mon[91579]: pgmap v2068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:58 smithi188 ceph-mon[92425]: pgmap v2068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:47:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:47:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:47:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:00 smithi161 ceph-mon[91579]: pgmap v2069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:00 smithi188 ceph-mon[92425]: pgmap v2069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:02 smithi161 ceph-mon[91579]: pgmap v2070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:02 smithi188 ceph-mon[92425]: pgmap v2070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:04 smithi161 ceph-mon[91579]: pgmap v2071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:04 smithi188 ceph-mon[92425]: pgmap v2071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:06 smithi161 ceph-mon[91579]: pgmap v2072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:06 smithi188 ceph-mon[92425]: pgmap v2072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:08 smithi161 ceph-mon[91579]: pgmap v2073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:08 smithi188 ceph-mon[92425]: pgmap v2073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:10 smithi161 ceph-mon[91579]: pgmap v2074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:10 smithi188 ceph-mon[92425]: pgmap v2074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:12 smithi161 ceph-mon[91579]: pgmap v2075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:12 smithi188 ceph-mon[92425]: pgmap v2075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:13.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:48:13.619+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:48:13.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:48:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:48:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:48:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:48:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:48:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:48:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:48:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:48:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:48:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:48:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:48:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:48:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:48:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:48:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:48:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:48:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:48:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:48:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:48:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:48:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:48:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:48:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:48:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:48:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:48:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:48:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:48:13.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:48:13.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:48:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:48:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:48:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:48:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:48:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:48:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:48:13.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:48:13.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:48:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:48:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:48:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:48:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:48:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:48:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:48:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:48:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=95] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:48:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=95] Local address: 172.21.15.161, port 51550 2022-07-03T11:48:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=95] Peer address: 172.21.15.101, port 22 2022-07-03T11:48:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=95] Connection lost 2022-07-03T11:48:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=95] Aborting connection 2022-07-03T11:48:13.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:48:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:48:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:48:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:48:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:48:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:48:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:48:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:48:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:48:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:48:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:48:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:48:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:48:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:48:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:48:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:48:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:48:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:48:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:48:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:48:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:48:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:48:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:48:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:48:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:48:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:48:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:48:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:48:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:48:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:48:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:48:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:48:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:48:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:48:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:48:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:48:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:48:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:48:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:48:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:48:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:48:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:48:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:48:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:48:15.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:48:15.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:48:15.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:48:15.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=95] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:48:15.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=95] Local address: 172.21.15.161, port 51550 2022-07-03T11:48:15.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=95] Peer address: 172.21.15.101, port 22 2022-07-03T11:48:15.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=95] Connection lost 2022-07-03T11:48:15.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=95] Aborting connection 2022-07-03T11:48:15.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:14 smithi161 ceph-mon[91579]: pgmap v2076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:15.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:48:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:48:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:48:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:48:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:48:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:48:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:48:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:48:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:48:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:48:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:48:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:48:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:48:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:48:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:48:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:48:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:48:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:48:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:48:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:48:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:48:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:48:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:48:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:48:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:48:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:48:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:48:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:48:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:48:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:48:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:48:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:48:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:48:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:48:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:48:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:48:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:48:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:48:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:48:15.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:48:15.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:48:15.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:48:15.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:48:15.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:48:15.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:48:15.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:48:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=95] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:48:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=95] Local address: 172.21.15.161, port 51550 2022-07-03T11:48:15.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=95] Peer address: 172.21.15.101, port 22 2022-07-03T11:48:15.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=95] Connection lost 2022-07-03T11:48:15.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=95] Aborting connection 2022-07-03T11:48:15.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:14 smithi188 ceph-mon[92425]: pgmap v2076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:15.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:16 smithi161 ceph-mon[91579]: pgmap v2077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:16 smithi188 ceph-mon[92425]: pgmap v2077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:18 smithi161 ceph-mon[91579]: pgmap v2078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:18 smithi188 ceph-mon[92425]: pgmap v2078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:20 smithi161 ceph-mon[91579]: pgmap v2079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:20 smithi188 ceph-mon[92425]: pgmap v2079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:22 smithi161 ceph-mon[91579]: pgmap v2080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:22 smithi188 ceph-mon[92425]: pgmap v2080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:24 smithi161 ceph-mon[91579]: pgmap v2081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:24 smithi188 ceph-mon[92425]: pgmap v2081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:26 smithi161 ceph-mon[91579]: pgmap v2082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:26 smithi188 ceph-mon[92425]: pgmap v2082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:28 smithi161 ceph-mon[91579]: pgmap v2083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:28 smithi188 ceph-mon[92425]: pgmap v2083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:30 smithi161 ceph-mon[91579]: pgmap v2084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:30 smithi188 ceph-mon[92425]: pgmap v2084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:32 smithi161 ceph-mon[91579]: pgmap v2085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:32 smithi188 ceph-mon[92425]: pgmap v2085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:34 smithi161 ceph-mon[91579]: pgmap v2086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:34 smithi188 ceph-mon[92425]: pgmap v2086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:36 smithi161 ceph-mon[91579]: pgmap v2087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:36 smithi188 ceph-mon[92425]: pgmap v2087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:38 smithi161 ceph-mon[91579]: pgmap v2088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:38 smithi188 ceph-mon[92425]: pgmap v2088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:40 smithi161 ceph-mon[91579]: pgmap v2089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:40 smithi188 ceph-mon[92425]: pgmap v2089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:42 smithi161 ceph-mon[91579]: pgmap v2090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:42 smithi188 ceph-mon[92425]: pgmap v2090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:44 smithi161 ceph-mon[91579]: pgmap v2091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:44 smithi188 ceph-mon[92425]: pgmap v2091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:46 smithi161 ceph-mon[91579]: pgmap v2092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:46 smithi188 ceph-mon[92425]: pgmap v2092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:48 smithi161 ceph-mon[91579]: pgmap v2093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:48 smithi188 ceph-mon[92425]: pgmap v2093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:50 smithi161 ceph-mon[91579]: pgmap v2094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:50 smithi188 ceph-mon[92425]: pgmap v2094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:52 smithi161 ceph-mon[91579]: pgmap v2095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:52 smithi188 ceph-mon[92425]: pgmap v2095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:54 smithi161 ceph-mon[91579]: pgmap v2096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:54 smithi188 ceph-mon[92425]: pgmap v2096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:56 smithi161 ceph-mon[91579]: pgmap v2097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:48:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:48:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:48:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:48:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:56 smithi188 ceph-mon[92425]: pgmap v2097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:48:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:48:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:48:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:48:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:48:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:58 smithi161 ceph-mon[91579]: pgmap v2098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:58 smithi188 ceph-mon[92425]: pgmap v2098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:48:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:48:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:48:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:00 smithi161 ceph-mon[91579]: pgmap v2099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:00 smithi188 ceph-mon[92425]: pgmap v2099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:02 smithi161 ceph-mon[91579]: pgmap v2100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:02 smithi188 ceph-mon[92425]: pgmap v2100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:04 smithi161 ceph-mon[91579]: pgmap v2101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:04 smithi188 ceph-mon[92425]: pgmap v2101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:06 smithi161 ceph-mon[91579]: pgmap v2102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:06 smithi188 ceph-mon[92425]: pgmap v2102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:08 smithi161 ceph-mon[91579]: pgmap v2103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:08 smithi188 ceph-mon[92425]: pgmap v2103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:10 smithi161 ceph-mon[91579]: pgmap v2104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:10 smithi188 ceph-mon[92425]: pgmap v2104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:12 smithi161 ceph-mon[91579]: pgmap v2105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:12 smithi188 ceph-mon[92425]: pgmap v2105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:13.968 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:49:13.658+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:49:13.968 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:49:13.969 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:49:13.969 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:49:13.969 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:49:13.970 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:49:13.970 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:49:13.970 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:49:13.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:49:13.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:49:13.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:49:13.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:49:13.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:49:13.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:49:13.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:49:13.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:49:13.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:49:13.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:49:13.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:49:13.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:49:13.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:49:13.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:49:13.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:49:13.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:49:13.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:49:13.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:49:13.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:49:13.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:49:13.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:49:13.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:49:13.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:49:13.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:49:13.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:49:13.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:49:13.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:49:13.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:49:13.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:49:13.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:49:13.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:49:13.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:49:13.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:49:13.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:49:13.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:49:13.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:49:13.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:49:13.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=97] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:49:13.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=97] Local address: 172.21.15.161, port 51578 2022-07-03T11:49:13.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=97] Peer address: 172.21.15.101, port 22 2022-07-03T11:49:13.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=97] Connection lost 2022-07-03T11:49:13.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=97] Aborting connection 2022-07-03T11:49:13.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:49:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:49:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:49:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:49:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:49:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:49:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:49:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:49:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:49:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:49:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:49:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:49:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:49:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:49:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:49:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:49:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:49:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:49:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:49:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:49:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:49:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:49:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:49:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:49:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:49:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:49:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:49:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:49:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:49:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:49:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:49:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:49:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:49:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:49:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:49:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:49:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:49:15.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:49:15.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:49:15.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:49:15.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:49:15.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:49:15.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:49:15.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:49:15.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:49:15.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:49:15.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:49:15.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:49:15.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:49:15.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:49:15.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:49:15.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:49:15.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:49:15.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=97] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:49:15.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=97] Local address: 172.21.15.161, port 51578 2022-07-03T11:49:15.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=97] Peer address: 172.21.15.101, port 22 2022-07-03T11:49:15.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=97] Connection lost 2022-07-03T11:49:15.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=97] Aborting connection 2022-07-03T11:49:15.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:14 smithi161 ceph-mon[91579]: pgmap v2106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:15.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:49:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:49:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:49:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:49:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:49:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:49:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:49:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:49:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:49:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:49:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:49:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:49:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:49:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:49:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:49:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:49:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:49:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:49:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:49:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:49:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:49:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:49:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:49:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:49:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:49:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:49:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:49:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:49:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:49:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:49:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:49:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:49:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:49:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:49:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:49:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:49:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:49:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:49:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:49:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:49:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:49:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:49:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:49:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:49:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:49:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:49:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=97] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:49:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=97] Local address: 172.21.15.161, port 51578 2022-07-03T11:49:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=97] Peer address: 172.21.15.101, port 22 2022-07-03T11:49:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=97] Connection lost 2022-07-03T11:49:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=97] Aborting connection 2022-07-03T11:49:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:14 smithi188 ceph-mon[92425]: pgmap v2106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:16 smithi161 ceph-mon[91579]: pgmap v2107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:16 smithi188 ceph-mon[92425]: pgmap v2107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:18 smithi161 ceph-mon[91579]: pgmap v2108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:18 smithi188 ceph-mon[92425]: pgmap v2108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:20 smithi161 ceph-mon[91579]: pgmap v2109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:20 smithi188 ceph-mon[92425]: pgmap v2109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:22 smithi161 ceph-mon[91579]: pgmap v2110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:22 smithi188 ceph-mon[92425]: pgmap v2110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:24 smithi161 ceph-mon[91579]: pgmap v2111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:24 smithi188 ceph-mon[92425]: pgmap v2111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:26 smithi161 ceph-mon[91579]: pgmap v2112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:26 smithi188 ceph-mon[92425]: pgmap v2112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:28 smithi161 ceph-mon[91579]: pgmap v2113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:28 smithi188 ceph-mon[92425]: pgmap v2113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:30 smithi161 ceph-mon[91579]: pgmap v2114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:30 smithi188 ceph-mon[92425]: pgmap v2114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:32 smithi161 ceph-mon[91579]: pgmap v2115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:32 smithi188 ceph-mon[92425]: pgmap v2115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:34 smithi161 ceph-mon[91579]: pgmap v2116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:34 smithi188 ceph-mon[92425]: pgmap v2116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:36 smithi161 ceph-mon[91579]: pgmap v2117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:36 smithi188 ceph-mon[92425]: pgmap v2117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:38 smithi161 ceph-mon[91579]: pgmap v2118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:38 smithi188 ceph-mon[92425]: pgmap v2118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:40 smithi161 ceph-mon[91579]: pgmap v2119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:40 smithi188 ceph-mon[92425]: pgmap v2119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:42 smithi161 ceph-mon[91579]: pgmap v2120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:42 smithi188 ceph-mon[92425]: pgmap v2120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:44 smithi161 ceph-mon[91579]: pgmap v2121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:44 smithi188 ceph-mon[92425]: pgmap v2121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:46 smithi161 ceph-mon[91579]: pgmap v2122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:46 smithi188 ceph-mon[92425]: pgmap v2122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:48 smithi161 ceph-mon[91579]: pgmap v2123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:48 smithi188 ceph-mon[92425]: pgmap v2123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:50 smithi161 ceph-mon[91579]: pgmap v2124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:50 smithi188 ceph-mon[92425]: pgmap v2124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:52 smithi161 ceph-mon[91579]: pgmap v2125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:52 smithi188 ceph-mon[92425]: pgmap v2125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:54 smithi161 ceph-mon[91579]: pgmap v2126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:54 smithi188 ceph-mon[92425]: pgmap v2126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:56 smithi161 ceph-mon[91579]: pgmap v2127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:49:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:49:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:49:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:49:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:56 smithi188 ceph-mon[92425]: pgmap v2127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:49:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:49:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:49:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:49:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:49:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:58 smithi161 ceph-mon[91579]: pgmap v2128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:58 smithi188 ceph-mon[92425]: pgmap v2128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:49:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:49:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:49:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:00 smithi161 ceph-mon[91579]: pgmap v2129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:50:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:00 smithi188 ceph-mon[92425]: pgmap v2129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T11:50:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:02 smithi161 ceph-mon[91579]: pgmap v2130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:02 smithi188 ceph-mon[92425]: pgmap v2130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:04 smithi161 ceph-mon[91579]: pgmap v2131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:04 smithi188 ceph-mon[92425]: pgmap v2131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:06 smithi161 ceph-mon[91579]: pgmap v2132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:06 smithi188 ceph-mon[92425]: pgmap v2132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:08 smithi161 ceph-mon[91579]: pgmap v2133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:08 smithi188 ceph-mon[92425]: pgmap v2133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:10 smithi161 ceph-mon[91579]: pgmap v2134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:10 smithi188 ceph-mon[92425]: pgmap v2134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:12 smithi161 ceph-mon[91579]: pgmap v2135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:12 smithi188 ceph-mon[92425]: pgmap v2135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:14.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:50:14.154+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:50:14.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:50:14.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:50:14.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:50:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:50:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:50:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:50:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:50:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:50:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:50:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:50:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:50:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:50:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:50:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:50:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:50:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:50:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:50:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:50:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:50:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:50:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:50:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:50:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:50:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:50:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:50:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:50:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:50:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:50:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:50:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:50:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:50:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:50:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:50:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:50:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:50:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:50:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:50:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:50:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:50:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:50:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:50:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:50:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:50:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:50:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=99] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:50:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=99] Local address: 172.21.15.161, port 51606 2022-07-03T11:50:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=99] Peer address: 172.21.15.101, port 22 2022-07-03T11:50:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=99] Connection lost 2022-07-03T11:50:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=99] Aborting connection 2022-07-03T11:50:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:50:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: pgmap v2136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:50:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:50:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:50:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:50:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:50:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:50:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:50:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:50:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:50:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:50:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:50:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:50:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:50:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:50:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:50:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:50:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:50:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:50:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:50:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:50:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:50:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:50:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:50:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:50:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:50:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:50:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:50:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:50:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:50:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:50:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:50:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:50:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:50:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:50:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:50:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:50:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:50:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:50:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:50:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:50:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:50:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:50:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:50:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:50:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:50:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:50:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:50:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:50:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:50:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=99] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:50:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=99] Local address: 172.21.15.161, port 51606 2022-07-03T11:50:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=99] Peer address: 172.21.15.101, port 22 2022-07-03T11:50:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=99] Connection lost 2022-07-03T11:50:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=99] Aborting connection 2022-07-03T11:50:15.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: pgmap v2136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:50:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:50:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:50:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:50:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:50:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:50:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:50:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:50:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:50:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:50:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:50:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:50:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:50:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:50:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:50:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:50:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:50:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:50:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:50:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:50:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:50:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:50:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:50:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:50:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:50:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:50:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:50:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:50:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:50:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:50:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:50:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:50:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:50:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:50:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:50:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:50:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:50:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:50:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:50:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:50:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:50:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:50:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:50:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:50:15.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:50:15.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:50:15.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:50:15.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:50:15.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:50:15.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=99] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:50:15.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=99] Local address: 172.21.15.161, port 51606 2022-07-03T11:50:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=99] Peer address: 172.21.15.101, port 22 2022-07-03T11:50:15.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=99] Connection lost 2022-07-03T11:50:15.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=99] Aborting connection 2022-07-03T11:50:15.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:16 smithi161 ceph-mon[91579]: pgmap v2137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:16 smithi188 ceph-mon[92425]: pgmap v2137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:18 smithi161 ceph-mon[91579]: pgmap v2138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:18 smithi188 ceph-mon[92425]: pgmap v2138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:20 smithi161 ceph-mon[91579]: pgmap v2139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:20 smithi188 ceph-mon[92425]: pgmap v2139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:22 smithi161 ceph-mon[91579]: pgmap v2140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:22 smithi188 ceph-mon[92425]: pgmap v2140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:24 smithi161 ceph-mon[91579]: pgmap v2141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:24 smithi188 ceph-mon[92425]: pgmap v2141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:26 smithi161 ceph-mon[91579]: pgmap v2142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:26 smithi188 ceph-mon[92425]: pgmap v2142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:28 smithi161 ceph-mon[91579]: pgmap v2143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:28 smithi188 ceph-mon[92425]: pgmap v2143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:30 smithi161 ceph-mon[91579]: pgmap v2144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:30 smithi188 ceph-mon[92425]: pgmap v2144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:32 smithi161 ceph-mon[91579]: pgmap v2145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:32 smithi188 ceph-mon[92425]: pgmap v2145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:35 smithi161 ceph-mon[91579]: pgmap v2146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:35 smithi188 ceph-mon[92425]: pgmap v2146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:37 smithi161 ceph-mon[91579]: pgmap v2147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:37 smithi188 ceph-mon[92425]: pgmap v2147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:38 smithi161 ceph-mon[91579]: pgmap v2148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:38 smithi188 ceph-mon[92425]: pgmap v2148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:40 smithi161 ceph-mon[91579]: pgmap v2149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:40 smithi188 ceph-mon[92425]: pgmap v2149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:42 smithi161 ceph-mon[91579]: pgmap v2150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:42 smithi188 ceph-mon[92425]: pgmap v2150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:45 smithi161 ceph-mon[91579]: pgmap v2151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:45 smithi188 ceph-mon[92425]: pgmap v2151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:47 smithi161 ceph-mon[91579]: pgmap v2152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:47 smithi188 ceph-mon[92425]: pgmap v2152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:48 smithi161 ceph-mon[91579]: pgmap v2153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:48 smithi188 ceph-mon[92425]: pgmap v2153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:50 smithi161 ceph-mon[91579]: pgmap v2154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:50 smithi188 ceph-mon[92425]: pgmap v2154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:52 smithi161 ceph-mon[91579]: pgmap v2155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:52 smithi188 ceph-mon[92425]: pgmap v2155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:55 smithi161 ceph-mon[91579]: pgmap v2156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:55 smithi188 ceph-mon[92425]: pgmap v2156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:57 smithi161 ceph-mon[91579]: pgmap v2157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:50:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:50:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:50:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:50:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:57 smithi188 ceph-mon[92425]: pgmap v2157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:50:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:50:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:50:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:50:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:50:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:58 smithi161 ceph-mon[91579]: pgmap v2158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:58 smithi188 ceph-mon[92425]: pgmap v2158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:50:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:50:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:00 smithi161 ceph-mon[91579]: pgmap v2159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:00 smithi188 ceph-mon[92425]: pgmap v2159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:02 smithi161 ceph-mon[91579]: pgmap v2160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:02 smithi188 ceph-mon[92425]: pgmap v2160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:05 smithi161 ceph-mon[91579]: pgmap v2161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:05 smithi188 ceph-mon[92425]: pgmap v2161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:07 smithi161 ceph-mon[91579]: pgmap v2162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:07 smithi188 ceph-mon[92425]: pgmap v2162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:08 smithi161 ceph-mon[91579]: pgmap v2163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:08 smithi188 ceph-mon[92425]: pgmap v2163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:10 smithi161 ceph-mon[91579]: pgmap v2164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:10 smithi188 ceph-mon[92425]: pgmap v2164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:13 smithi161 ceph-mon[91579]: pgmap v2165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:13 smithi188 ceph-mon[92425]: pgmap v2165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:15 smithi161 ceph-mon[91579]: pgmap v2166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:51:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:51:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:51:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:15 smithi188 ceph-mon[92425]: pgmap v2166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:51:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:51:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:51:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:51:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:51:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:17 smithi161 ceph-mon[91579]: pgmap v2167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:17.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:17 smithi188 ceph-mon[92425]: pgmap v2167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:18 smithi161 ceph-mon[91579]: pgmap v2168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:18 smithi188 ceph-mon[92425]: pgmap v2168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:20 smithi161 ceph-mon[91579]: pgmap v2169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:20 smithi188 ceph-mon[92425]: pgmap v2169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:23 smithi161 ceph-mon[91579]: pgmap v2170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:23 smithi188 ceph-mon[92425]: pgmap v2170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:25 smithi161 ceph-mon[91579]: pgmap v2171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:25 smithi188 ceph-mon[92425]: pgmap v2171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:27 smithi161 ceph-mon[91579]: pgmap v2172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:27 smithi188 ceph-mon[92425]: pgmap v2172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:28 smithi161 ceph-mon[91579]: pgmap v2173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:28 smithi188 ceph-mon[92425]: pgmap v2173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:30 smithi161 ceph-mon[91579]: pgmap v2174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:30 smithi188 ceph-mon[92425]: pgmap v2174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:33 smithi161 ceph-mon[91579]: pgmap v2175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:33 smithi188 ceph-mon[92425]: pgmap v2175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:35 smithi161 ceph-mon[91579]: pgmap v2176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:35 smithi188 ceph-mon[92425]: pgmap v2176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:37 smithi161 ceph-mon[91579]: pgmap v2177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:37 smithi188 ceph-mon[92425]: pgmap v2177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T11:51:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:38 smithi161 ceph-mon[91579]: pgmap v2178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:38 smithi188 ceph-mon[92425]: pgmap v2178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:40 smithi161 ceph-mon[91579]: pgmap v2179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:40 smithi188 ceph-mon[92425]: pgmap v2179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:43 smithi161 ceph-mon[91579]: pgmap v2180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:43 smithi188 ceph-mon[92425]: pgmap v2180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:45.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:51:44.663+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:51:45.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:51:45.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:51:45.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:51:45.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:51:45.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:51:45.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:51:45.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:51:45.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:51:45.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:51:45.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:51:45.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:51:45.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:51:45.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:51:45.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:51:45.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:51:45.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:51:45.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:51:45.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:51:45.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:51:45.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:51:45.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:51:45.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:51:45.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:51:45.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:51:45.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:51:45.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:51:45.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:51:45.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:51:45.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:51:45.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:51:45.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:51:45.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:51:45.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:51:45.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:51:45.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:51:45.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:51:45.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:51:45.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:51:45.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:51:45.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:51:45.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:51:45.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:51:45.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:51:45.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:51:45.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:51:45.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=101] Local address: 172.21.15.161, port 51634 2022-07-03T11:51:45.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=101] Peer address: 172.21.15.101, port 22 2022-07-03T11:51:45.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=320] Received exit status 0 2022-07-03T11:51:45.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=320] Received channel close 2022-07-03T11:51:45.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=320] Channel closed 2022-07-03T11:51:45.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=320] Received exit status 0 2022-07-03T11:51:45.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=320] Received channel close 2022-07-03T11:51:45.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=320] Channel closed 2022-07-03T11:51:45.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=321] Requesting new SSH session 2022-07-03T11:51:45.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=321] Requesting new SSH session 2022-07-03T11:51:45.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=321] Command: which python3 2022-07-03T11:51:45.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=321] Command: which python3 2022-07-03T11:51:45.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=321] Received exit status 0 2022-07-03T11:51:45.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=321] Received channel close 2022-07-03T11:51:45.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=321] Channel closed 2022-07-03T11:51:45.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=322] Requesting new SSH session 2022-07-03T11:51:45.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=321] Received exit status 0 2022-07-03T11:51:45.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=321] Received channel close 2022-07-03T11:51:45.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=321] Channel closed 2022-07-03T11:51:45.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=322] Requesting new SSH session 2022-07-03T11:51:45.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=322] Command: sudo true 2022-07-03T11:51:45.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=322] Command: sudo true 2022-07-03T11:51:45.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=322] Received exit status 0 2022-07-03T11:51:45.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=322] Received channel close 2022-07-03T11:51:45.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=322] Channel closed 2022-07-03T11:51:45.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=323] Requesting new SSH session 2022-07-03T11:51:45.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:45.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=322] Received exit status 0 2022-07-03T11:51:45.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=322] Received channel close 2022-07-03T11:51:45.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=322] Channel closed 2022-07-03T11:51:45.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=323] Requesting new SSH session 2022-07-03T11:51:45.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:45.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=323] Received exit status 0 2022-07-03T11:51:45.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=323] Received channel close 2022-07-03T11:51:45.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=323] Channel closed 2022-07-03T11:51:45.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=323] Received exit status 0 2022-07-03T11:51:45.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=323] Received channel close 2022-07-03T11:51:45.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=323] Channel closed 2022-07-03T11:51:45.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=101] Login timeout expired 2022-07-03T11:51:45.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=101] Aborting connection 2022-07-03T11:51:45.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:51:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:51:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:45 smithi161 ceph-mon[91579]: pgmap v2181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:45 smithi188 ceph-mon[92425]: pgmap v2181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:51:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:51:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:51:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:51:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:51:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:51:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:51:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:51:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:51:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:51:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:51:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:51:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:51:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:51:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:51:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:51:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:51:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:51:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:51:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:51:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:51:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:51:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:51:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:51:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:51:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:51:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:51:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:51:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:51:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:51:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:51:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:51:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:51:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:51:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:51:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:51:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:51:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:51:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:51:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:51:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:51:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:51:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:51:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:51:46.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:51:46.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:51:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=101] Local address: 172.21.15.161, port 51634 2022-07-03T11:51:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=101] Peer address: 172.21.15.101, port 22 2022-07-03T11:51:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=320] Received exit status 0 2022-07-03T11:51:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=320] Received channel close 2022-07-03T11:51:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=320] Channel closed 2022-07-03T11:51:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=320] Received exit status 0 2022-07-03T11:51:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=320] Received channel close 2022-07-03T11:51:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=320] Channel closed 2022-07-03T11:51:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=321] Requesting new SSH session 2022-07-03T11:51:46.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=321] Requesting new SSH session 2022-07-03T11:51:46.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=321] Command: which python3 2022-07-03T11:51:46.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=321] Command: which python3 2022-07-03T11:51:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=321] Received exit status 0 2022-07-03T11:51:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=321] Received channel close 2022-07-03T11:51:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=321] Channel closed 2022-07-03T11:51:46.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=322] Requesting new SSH session 2022-07-03T11:51:46.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=321] Received exit status 0 2022-07-03T11:51:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=321] Received channel close 2022-07-03T11:51:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=321] Channel closed 2022-07-03T11:51:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=322] Requesting new SSH session 2022-07-03T11:51:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=322] Command: sudo true 2022-07-03T11:51:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=322] Command: sudo true 2022-07-03T11:51:46.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=322] Received exit status 0 2022-07-03T11:51:46.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=322] Received channel close 2022-07-03T11:51:46.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=322] Channel closed 2022-07-03T11:51:46.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=323] Requesting new SSH session 2022-07-03T11:51:46.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:46.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=322] Received exit status 0 2022-07-03T11:51:46.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=322] Received channel close 2022-07-03T11:51:46.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=322] Channel closed 2022-07-03T11:51:46.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=323] Requesting new SSH session 2022-07-03T11:51:46.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:46.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=323] Received exit status 0 2022-07-03T11:51:46.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=323] Received channel close 2022-07-03T11:51:46.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=323] Channel closed 2022-07-03T11:51:46.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=323] Received exit status 0 2022-07-03T11:51:46.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=323] Received channel close 2022-07-03T11:51:46.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=323] Channel closed 2022-07-03T11:51:46.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=101] Login timeout expired 2022-07-03T11:51:46.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=101] Aborting connection 2022-07-03T11:51:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:51:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:51:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:51:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:51:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:51:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:51:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:51:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:51:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:51:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:51:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:51:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:51:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:51:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:51:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:51:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:51:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:51:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:51:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:51:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:51:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:51:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:51:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:51:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:51:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:51:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:51:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:51:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:51:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:51:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:51:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:51:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:51:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:51:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:51:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:51:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:51:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:51:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:51:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:51:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:51:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:51:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:51:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:51:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:51:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:51:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:51:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=101] Local address: 172.21.15.161, port 51634 2022-07-03T11:51:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=101] Peer address: 172.21.15.101, port 22 2022-07-03T11:51:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=320] Received exit status 0 2022-07-03T11:51:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=320] Received channel close 2022-07-03T11:51:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=320] Channel closed 2022-07-03T11:51:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=320] Received exit status 0 2022-07-03T11:51:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=320] Received channel close 2022-07-03T11:51:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=320] Channel closed 2022-07-03T11:51:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=321] Requesting new SSH session 2022-07-03T11:51:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=321] Requesting new SSH session 2022-07-03T11:51:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=321] Command: which python3 2022-07-03T11:51:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=321] Command: which python3 2022-07-03T11:51:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=321] Received exit status 0 2022-07-03T11:51:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=321] Received channel close 2022-07-03T11:51:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=321] Channel closed 2022-07-03T11:51:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=322] Requesting new SSH session 2022-07-03T11:51:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=321] Received exit status 0 2022-07-03T11:51:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=321] Received channel close 2022-07-03T11:51:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=321] Channel closed 2022-07-03T11:51:46.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=322] Requesting new SSH session 2022-07-03T11:51:46.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=322] Command: sudo true 2022-07-03T11:51:46.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=322] Command: sudo true 2022-07-03T11:51:46.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=322] Received exit status 0 2022-07-03T11:51:46.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=322] Received channel close 2022-07-03T11:51:46.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=322] Channel closed 2022-07-03T11:51:46.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=323] Requesting new SSH session 2022-07-03T11:51:46.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:46.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=322] Received exit status 0 2022-07-03T11:51:46.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=322] Received channel close 2022-07-03T11:51:46.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=322] Channel closed 2022-07-03T11:51:46.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=323] Requesting new SSH session 2022-07-03T11:51:46.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=323] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:51:46.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=323] Received exit status 0 2022-07-03T11:51:46.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=323] Received channel close 2022-07-03T11:51:46.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=323] Channel closed 2022-07-03T11:51:46.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=323] Received exit status 0 2022-07-03T11:51:46.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=323] Received channel close 2022-07-03T11:51:46.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=323] Channel closed 2022-07-03T11:51:46.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=101] Login timeout expired 2022-07-03T11:51:46.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=101] Aborting connection 2022-07-03T11:51:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:47 smithi161 ceph-mon[91579]: pgmap v2182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:47 smithi188 ceph-mon[92425]: pgmap v2182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:48 smithi161 ceph-mon[91579]: pgmap v2183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:48 smithi188 ceph-mon[92425]: pgmap v2183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:50 smithi161 ceph-mon[91579]: pgmap v2184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:50 smithi188 ceph-mon[92425]: pgmap v2184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:53 smithi161 ceph-mon[91579]: pgmap v2185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:53 smithi188 ceph-mon[92425]: pgmap v2185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:55 smithi161 ceph-mon[91579]: pgmap v2186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:55 smithi188 ceph-mon[92425]: pgmap v2186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:57 smithi161 ceph-mon[91579]: pgmap v2187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:51:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:51:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:51:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:51:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:57 smithi188 ceph-mon[92425]: pgmap v2187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:51:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:51:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:51:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:51:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:51:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:58 smithi161 ceph-mon[91579]: pgmap v2188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:58 smithi188 ceph-mon[92425]: pgmap v2188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:51:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:51:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:51:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:00 smithi161 ceph-mon[91579]: pgmap v2189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:00 smithi188 ceph-mon[92425]: pgmap v2189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:03 smithi161 ceph-mon[91579]: pgmap v2190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:03 smithi188 ceph-mon[92425]: pgmap v2190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:05 smithi161 ceph-mon[91579]: pgmap v2191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:05 smithi188 ceph-mon[92425]: pgmap v2191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:07 smithi161 ceph-mon[91579]: pgmap v2192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:07 smithi188 ceph-mon[92425]: pgmap v2192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:08 smithi161 ceph-mon[91579]: pgmap v2193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:08.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:08 smithi188 ceph-mon[92425]: pgmap v2193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:10 smithi161 ceph-mon[91579]: pgmap v2194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:10 smithi188 ceph-mon[92425]: pgmap v2194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:13 smithi161 ceph-mon[91579]: pgmap v2195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:13 smithi188 ceph-mon[92425]: pgmap v2195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:15 smithi161 ceph-mon[91579]: pgmap v2196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:15 smithi188 ceph-mon[92425]: pgmap v2196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:17 smithi161 ceph-mon[91579]: pgmap v2197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:17 smithi188 ceph-mon[92425]: pgmap v2197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:18 smithi161 ceph-mon[91579]: pgmap v2198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:18 smithi188 ceph-mon[92425]: pgmap v2198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:20 smithi161 ceph-mon[91579]: pgmap v2199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:20 smithi188 ceph-mon[92425]: pgmap v2199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:23 smithi161 ceph-mon[91579]: pgmap v2200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:23 smithi188 ceph-mon[92425]: pgmap v2200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:25 smithi161 ceph-mon[91579]: pgmap v2201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:25 smithi188 ceph-mon[92425]: pgmap v2201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:27 smithi161 ceph-mon[91579]: pgmap v2202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:27 smithi188 ceph-mon[92425]: pgmap v2202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:28 smithi161 ceph-mon[91579]: pgmap v2203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:28 smithi188 ceph-mon[92425]: pgmap v2203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:30 smithi161 ceph-mon[91579]: pgmap v2204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:30 smithi188 ceph-mon[92425]: pgmap v2204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:33 smithi161 ceph-mon[91579]: pgmap v2205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:33 smithi188 ceph-mon[92425]: pgmap v2205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:35 smithi161 ceph-mon[91579]: pgmap v2206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:35 smithi188 ceph-mon[92425]: pgmap v2206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:37 smithi161 ceph-mon[91579]: pgmap v2207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:37 smithi188 ceph-mon[92425]: pgmap v2207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:38 smithi161 ceph-mon[91579]: pgmap v2208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:38.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:38 smithi188 ceph-mon[92425]: pgmap v2208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:38.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:40 smithi161 ceph-mon[91579]: pgmap v2209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:40 smithi188 ceph-mon[92425]: pgmap v2209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:43 smithi161 ceph-mon[91579]: pgmap v2210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:43 smithi188 ceph-mon[92425]: pgmap v2210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:45.036 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:45 smithi161 ceph-mon[91579]: pgmap v2211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:52:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:52:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:52:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:45 smithi188 ceph-mon[92425]: pgmap v2211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:52:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:52:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:52:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:52:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:52:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:47 smithi161 ceph-mon[91579]: pgmap v2212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:47 smithi188 ceph-mon[92425]: pgmap v2212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:48 smithi161 ceph-mon[91579]: pgmap v2213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:48 smithi188 ceph-mon[92425]: pgmap v2213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:50 smithi188 ceph-mon[92425]: pgmap v2214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:50 smithi161 ceph-mon[91579]: pgmap v2214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:53 smithi161 ceph-mon[91579]: pgmap v2215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:53 smithi188 ceph-mon[92425]: pgmap v2215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:55 smithi161 ceph-mon[91579]: pgmap v2216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:55 smithi188 ceph-mon[92425]: pgmap v2216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:57 smithi161 ceph-mon[91579]: pgmap v2217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:52:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:52:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:52:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:52:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:57 smithi188 ceph-mon[92425]: pgmap v2217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:52:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:52:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:52:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:52:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:52:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:58 smithi161 ceph-mon[91579]: pgmap v2218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:58 smithi188 ceph-mon[92425]: pgmap v2218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:52:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:52:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:52:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:00 smithi188 ceph-mon[92425]: pgmap v2219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:00 smithi161 ceph-mon[91579]: pgmap v2219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:03 smithi161 ceph-mon[91579]: pgmap v2220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:03 smithi188 ceph-mon[92425]: pgmap v2220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:05 smithi161 ceph-mon[91579]: pgmap v2221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:05 smithi188 ceph-mon[92425]: pgmap v2221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:07 smithi161 ceph-mon[91579]: pgmap v2222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:07 smithi188 ceph-mon[92425]: pgmap v2222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:08 smithi161 ceph-mon[91579]: pgmap v2223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:08 smithi188 ceph-mon[92425]: pgmap v2223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:10 smithi188 ceph-mon[92425]: pgmap v2224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:10 smithi161 ceph-mon[91579]: pgmap v2224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:13 smithi161 ceph-mon[91579]: pgmap v2225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:13 smithi188 ceph-mon[92425]: pgmap v2225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:15.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:53:14.701+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:53:15.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:53:15.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:53:15.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:53:15.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:53:15.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:53:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:53:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:53:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:53:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:53:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:53:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:53:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:53:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:53:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:53:15.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:53:15.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:53:15.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:53:15.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:53:15.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:53:15.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:53:15.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:53:15.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:53:15.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:53:15.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:53:15.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:53:15.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:53:15.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:53:15.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:53:15.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:53:15.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:53:15.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:53:15.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:53:15.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:53:15.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:53:15.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:53:15.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:53:15.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:53:15.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:53:15.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:53:15.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:53:15.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:53:15.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:53:15.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:53:15.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:53:15.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:53:15.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=103] Local address: 172.21.15.161, port 51674 2022-07-03T11:53:15.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=103] Peer address: 172.21.15.101, port 22 2022-07-03T11:53:15.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=324] Received exit status 0 2022-07-03T11:53:15.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=324] Received channel close 2022-07-03T11:53:15.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=324] Channel closed 2022-07-03T11:53:15.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=324] Received exit status 0 2022-07-03T11:53:15.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=324] Received channel close 2022-07-03T11:53:15.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=324] Channel closed 2022-07-03T11:53:15.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=325] Requesting new SSH session 2022-07-03T11:53:15.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=325] Requesting new SSH session 2022-07-03T11:53:15.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=325] Command: which python3 2022-07-03T11:53:15.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=325] Command: which python3 2022-07-03T11:53:15.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=325] Received exit status 0 2022-07-03T11:53:15.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=325] Received channel close 2022-07-03T11:53:15.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=325] Channel closed 2022-07-03T11:53:15.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=325] Received exit status 0 2022-07-03T11:53:15.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=325] Received channel close 2022-07-03T11:53:15.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=326] Requesting new SSH session 2022-07-03T11:53:15.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=325] Channel closed 2022-07-03T11:53:15.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=326] Requesting new SSH session 2022-07-03T11:53:15.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=326] Command: sudo true 2022-07-03T11:53:15.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=326] Command: sudo true 2022-07-03T11:53:15.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=326] Received exit status 0 2022-07-03T11:53:15.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=326] Received channel close 2022-07-03T11:53:15.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=326] Channel closed 2022-07-03T11:53:15.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=327] Requesting new SSH session 2022-07-03T11:53:15.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:15.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=326] Received exit status 0 2022-07-03T11:53:15.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=326] Received channel close 2022-07-03T11:53:15.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=326] Channel closed 2022-07-03T11:53:15.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=327] Requesting new SSH session 2022-07-03T11:53:15.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:15.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=327] Received exit status 0 2022-07-03T11:53:15.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=327] Received channel close 2022-07-03T11:53:15.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=327] Channel closed 2022-07-03T11:53:15.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=327] Received exit status 0 2022-07-03T11:53:15.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=327] Received channel close 2022-07-03T11:53:15.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=327] Channel closed 2022-07-03T11:53:15.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=103] Login timeout expired 2022-07-03T11:53:15.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=103] Aborting connection 2022-07-03T11:53:15.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:53:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:53:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:15 smithi161 ceph-mon[91579]: pgmap v2226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:15 smithi188 ceph-mon[92425]: pgmap v2226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:53:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:53:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:53:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:53:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:53:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:53:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:53:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:53:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:53:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:53:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:53:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:53:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:53:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:53:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:53:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:53:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:53:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:53:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:53:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:53:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:53:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:53:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:53:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:53:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:53:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:53:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:53:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:53:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:53:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:53:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:53:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:53:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:53:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:53:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:53:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:53:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:53:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:53:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:53:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:53:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:53:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:53:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:53:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:53:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:53:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:53:16.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=103] Local address: 172.21.15.161, port 51674 2022-07-03T11:53:16.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=103] Peer address: 172.21.15.101, port 22 2022-07-03T11:53:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=324] Received exit status 0 2022-07-03T11:53:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=324] Received channel close 2022-07-03T11:53:16.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=324] Channel closed 2022-07-03T11:53:16.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=324] Received exit status 0 2022-07-03T11:53:16.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=324] Received channel close 2022-07-03T11:53:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=324] Channel closed 2022-07-03T11:53:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=325] Requesting new SSH session 2022-07-03T11:53:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=325] Requesting new SSH session 2022-07-03T11:53:16.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=325] Command: which python3 2022-07-03T11:53:16.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=325] Command: which python3 2022-07-03T11:53:16.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=325] Received exit status 0 2022-07-03T11:53:16.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=325] Received channel close 2022-07-03T11:53:16.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=325] Channel closed 2022-07-03T11:53:16.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=325] Received exit status 0 2022-07-03T11:53:16.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=325] Received channel close 2022-07-03T11:53:16.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=326] Requesting new SSH session 2022-07-03T11:53:16.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=325] Channel closed 2022-07-03T11:53:16.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=326] Requesting new SSH session 2022-07-03T11:53:16.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=326] Command: sudo true 2022-07-03T11:53:16.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=326] Command: sudo true 2022-07-03T11:53:16.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=326] Received exit status 0 2022-07-03T11:53:16.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=326] Received channel close 2022-07-03T11:53:16.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=326] Channel closed 2022-07-03T11:53:16.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=327] Requesting new SSH session 2022-07-03T11:53:16.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:16.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=326] Received exit status 0 2022-07-03T11:53:16.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=326] Received channel close 2022-07-03T11:53:16.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=326] Channel closed 2022-07-03T11:53:16.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=327] Requesting new SSH session 2022-07-03T11:53:16.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:16.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=327] Received exit status 0 2022-07-03T11:53:16.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=327] Received channel close 2022-07-03T11:53:16.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=327] Channel closed 2022-07-03T11:53:16.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=327] Received exit status 0 2022-07-03T11:53:16.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=327] Received channel close 2022-07-03T11:53:16.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=327] Channel closed 2022-07-03T11:53:16.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=103] Login timeout expired 2022-07-03T11:53:16.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=103] Aborting connection 2022-07-03T11:53:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:53:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:53:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:53:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:53:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:53:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:53:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:53:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:53:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:53:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:53:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:53:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:53:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:53:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:53:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:53:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:53:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:53:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:53:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:53:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:53:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:53:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:53:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:53:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:53:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:53:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:53:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:53:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:53:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:53:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:53:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:53:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:53:16.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:53:16.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:53:16.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:53:16.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:53:16.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:53:16.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:53:16.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:53:16.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:53:16.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:53:16.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:53:16.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:53:16.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:53:16.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:53:16.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:53:16.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=103] Local address: 172.21.15.161, port 51674 2022-07-03T11:53:16.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=103] Peer address: 172.21.15.101, port 22 2022-07-03T11:53:16.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=324] Received exit status 0 2022-07-03T11:53:16.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=324] Received channel close 2022-07-03T11:53:16.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=324] Channel closed 2022-07-03T11:53:16.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=324] Received exit status 0 2022-07-03T11:53:16.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=324] Received channel close 2022-07-03T11:53:16.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=324] Channel closed 2022-07-03T11:53:16.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=325] Requesting new SSH session 2022-07-03T11:53:16.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=325] Requesting new SSH session 2022-07-03T11:53:16.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=325] Command: which python3 2022-07-03T11:53:16.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=325] Command: which python3 2022-07-03T11:53:16.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=325] Received exit status 0 2022-07-03T11:53:16.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=325] Received channel close 2022-07-03T11:53:16.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=325] Channel closed 2022-07-03T11:53:16.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=325] Received exit status 0 2022-07-03T11:53:16.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=325] Received channel close 2022-07-03T11:53:16.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=326] Requesting new SSH session 2022-07-03T11:53:16.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=325] Channel closed 2022-07-03T11:53:16.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=326] Requesting new SSH session 2022-07-03T11:53:16.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=326] Command: sudo true 2022-07-03T11:53:16.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=326] Command: sudo true 2022-07-03T11:53:16.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=326] Received exit status 0 2022-07-03T11:53:16.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=326] Received channel close 2022-07-03T11:53:16.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=326] Channel closed 2022-07-03T11:53:16.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=327] Requesting new SSH session 2022-07-03T11:53:16.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:16.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=326] Received exit status 0 2022-07-03T11:53:16.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=326] Received channel close 2022-07-03T11:53:16.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=326] Channel closed 2022-07-03T11:53:16.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=327] Requesting new SSH session 2022-07-03T11:53:16.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=327] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T11:53:16.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=327] Received exit status 0 2022-07-03T11:53:16.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=327] Received channel close 2022-07-03T11:53:16.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=327] Channel closed 2022-07-03T11:53:16.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=327] Received exit status 0 2022-07-03T11:53:16.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=327] Received channel close 2022-07-03T11:53:16.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=327] Channel closed 2022-07-03T11:53:16.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=103] Login timeout expired 2022-07-03T11:53:16.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=103] Aborting connection 2022-07-03T11:53:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:17 smithi161 ceph-mon[91579]: pgmap v2227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:17 smithi188 ceph-mon[92425]: pgmap v2227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:18 smithi161 ceph-mon[91579]: pgmap v2228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:18 smithi188 ceph-mon[92425]: pgmap v2228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:20 smithi188 ceph-mon[92425]: pgmap v2229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:20 smithi161 ceph-mon[91579]: pgmap v2229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:23 smithi161 ceph-mon[91579]: pgmap v2230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:23 smithi188 ceph-mon[92425]: pgmap v2230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:25 smithi161 ceph-mon[91579]: pgmap v2231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:25 smithi188 ceph-mon[92425]: pgmap v2231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:27 smithi161 ceph-mon[91579]: pgmap v2232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:27 smithi188 ceph-mon[92425]: pgmap v2232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:28 smithi161 ceph-mon[91579]: pgmap v2233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:28 smithi188 ceph-mon[92425]: pgmap v2233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:30 smithi188 ceph-mon[92425]: pgmap v2234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:30 smithi161 ceph-mon[91579]: pgmap v2234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:33 smithi161 ceph-mon[91579]: pgmap v2235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:33 smithi188 ceph-mon[92425]: pgmap v2235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:35 smithi161 ceph-mon[91579]: pgmap v2236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:35 smithi188 ceph-mon[92425]: pgmap v2236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:37 smithi161 ceph-mon[91579]: pgmap v2237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:37 smithi188 ceph-mon[92425]: pgmap v2237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:38.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:38 smithi161 ceph-mon[91579]: pgmap v2238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:38.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:38 smithi188 ceph-mon[92425]: pgmap v2238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:40 smithi188 ceph-mon[92425]: pgmap v2239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:40 smithi161 ceph-mon[91579]: pgmap v2239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:43 smithi161 ceph-mon[91579]: pgmap v2240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:43 smithi188 ceph-mon[92425]: pgmap v2240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:45 smithi161 ceph-mon[91579]: pgmap v2241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:45 smithi188 ceph-mon[92425]: pgmap v2241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:47 smithi161 ceph-mon[91579]: pgmap v2242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:47 smithi188 ceph-mon[92425]: pgmap v2242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:48 smithi161 ceph-mon[91579]: pgmap v2243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:48 smithi188 ceph-mon[92425]: pgmap v2243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:50 smithi188 ceph-mon[92425]: pgmap v2244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:50 smithi161 ceph-mon[91579]: pgmap v2244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:53 smithi161 ceph-mon[91579]: pgmap v2245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:53 smithi188 ceph-mon[92425]: pgmap v2245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:55 smithi161 ceph-mon[91579]: pgmap v2246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:55 smithi188 ceph-mon[92425]: pgmap v2246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:53:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:57 smithi161 ceph-mon[91579]: pgmap v2247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:53:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:53:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:53:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:53:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:57 smithi188 ceph-mon[92425]: pgmap v2247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:53:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:53:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:53:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:53:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:53:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:58 smithi161 ceph-mon[91579]: pgmap v2248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:58 smithi188 ceph-mon[92425]: pgmap v2248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:53:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:53:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:00 smithi188 ceph-mon[92425]: pgmap v2249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:00 smithi161 ceph-mon[91579]: pgmap v2249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:03 smithi161 ceph-mon[91579]: pgmap v2250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:03 smithi188 ceph-mon[92425]: pgmap v2250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:05 smithi161 ceph-mon[91579]: pgmap v2251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:05 smithi188 ceph-mon[92425]: pgmap v2251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:07 smithi161 ceph-mon[91579]: pgmap v2252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:07 smithi188 ceph-mon[92425]: pgmap v2252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:08 smithi161 ceph-mon[91579]: pgmap v2253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:08 smithi188 ceph-mon[92425]: pgmap v2253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:10 smithi161 ceph-mon[91579]: pgmap v2254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:10 smithi188 ceph-mon[92425]: pgmap v2254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:13 smithi161 ceph-mon[91579]: pgmap v2255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:13 smithi188 ceph-mon[92425]: pgmap v2255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:15 smithi161 ceph-mon[91579]: pgmap v2256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:54:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:54:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:54:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:15 smithi188 ceph-mon[92425]: pgmap v2256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:54:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:54:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:54:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:17 smithi161 ceph-mon[91579]: pgmap v2257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:17 smithi188 ceph-mon[92425]: pgmap v2257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: pgmap v2258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:54:19.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:54:19.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T11:54:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: pgmap v2258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T11:54:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T11:54:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T11:54:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:19 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:54:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:19 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:54:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:19 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T11:54:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:19 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T11:54:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:20 smithi161 ceph-mon[91579]: pgmap v2259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:20 smithi188 ceph-mon[92425]: pgmap v2259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:23 smithi161 ceph-mon[91579]: pgmap v2260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:23 smithi188 ceph-mon[92425]: pgmap v2260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:25 smithi161 ceph-mon[91579]: pgmap v2261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:25 smithi188 ceph-mon[92425]: pgmap v2261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:27 smithi161 ceph-mon[91579]: pgmap v2262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:27 smithi188 ceph-mon[92425]: pgmap v2262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:28 smithi161 ceph-mon[91579]: pgmap v2263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:28 smithi188 ceph-mon[92425]: pgmap v2263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:30 smithi161 ceph-mon[91579]: pgmap v2264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:30 smithi188 ceph-mon[92425]: pgmap v2264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:33 smithi161 ceph-mon[91579]: pgmap v2265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:33 smithi188 ceph-mon[92425]: pgmap v2265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:35 smithi161 ceph-mon[91579]: pgmap v2266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:35 smithi188 ceph-mon[92425]: pgmap v2266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:37 smithi161 ceph-mon[91579]: pgmap v2267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:37 smithi188 ceph-mon[92425]: pgmap v2267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:38 smithi161 ceph-mon[91579]: pgmap v2268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:38 smithi188 ceph-mon[92425]: pgmap v2268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:40 smithi161 ceph-mon[91579]: pgmap v2269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:40 smithi188 ceph-mon[92425]: pgmap v2269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:43 smithi161 ceph-mon[91579]: pgmap v2270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:43 smithi188 ceph-mon[92425]: pgmap v2270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:45.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:54:44.740+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:54:45.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:54:45.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:54:45.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:54:45.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:54:45.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:54:45.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:54:45.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:54:45.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:54:45.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:54:45.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:54:45.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:54:45.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:54:45.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:54:45.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:54:45.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:54:45.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:54:45.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:54:45.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:54:45.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:54:45.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:54:45.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:54:45.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:54:45.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:54:45.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:54:45.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:54:45.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:54:45.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:54:45.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:54:45.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:54:45.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:54:45.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:54:45.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:54:45.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:54:45.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:54:45.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:54:45.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:54:45.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:54:45.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:54:45.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:54:45.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:54:45.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:54:45.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:54:45.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:54:45.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:54:45.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:54:45.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=105] Local address: 172.21.15.161, port 51752 2022-07-03T11:54:45.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=105] Peer address: 172.21.15.101, port 22 2022-07-03T11:54:45.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=105] Connection lost 2022-07-03T11:54:45.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=105] Aborting connection 2022-07-03T11:54:45.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:54:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:54:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:45 smithi161 ceph-mon[91579]: pgmap v2271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:45 smithi188 ceph-mon[92425]: pgmap v2271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:54:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:54:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:54:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:54:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:54:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:54:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:54:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:54:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:54:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:54:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:54:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:54:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:54:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:54:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:54:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:54:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:54:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:54:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:54:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:54:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:54:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:54:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:54:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:54:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:54:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:54:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:54:46.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:54:46.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:54:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:54:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:54:46.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:54:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:54:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:54:46.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:54:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:54:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:54:46.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:54:46.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:54:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:54:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:54:46.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:54:46.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:54:46.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:54:46.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:54:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:54:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:54:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=105] Local address: 172.21.15.161, port 51752 2022-07-03T11:54:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=105] Peer address: 172.21.15.101, port 22 2022-07-03T11:54:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=105] Connection lost 2022-07-03T11:54:46.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=105] Aborting connection 2022-07-03T11:54:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:54:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:54:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:54:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:54:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:54:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:54:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:54:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:54:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:54:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:54:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:54:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:54:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:54:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:54:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:54:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:54:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:54:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:54:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:54:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:54:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:54:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:54:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:54:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:54:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:54:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:54:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:54:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:54:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:54:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:54:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:54:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:54:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:54:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:54:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:54:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:54:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:54:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:54:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:54:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:54:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:54:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:54:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:54:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:54:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:54:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:54:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:54:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=105] Local address: 172.21.15.161, port 51752 2022-07-03T11:54:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=105] Peer address: 172.21.15.101, port 22 2022-07-03T11:54:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=105] Connection lost 2022-07-03T11:54:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=105] Aborting connection 2022-07-03T11:54:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:47 smithi161 ceph-mon[91579]: pgmap v2272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:47 smithi188 ceph-mon[92425]: pgmap v2272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:48 smithi161 ceph-mon[91579]: pgmap v2273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:48 smithi188 ceph-mon[92425]: pgmap v2273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:50 smithi161 ceph-mon[91579]: pgmap v2274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:50 smithi188 ceph-mon[92425]: pgmap v2274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:53 smithi161 ceph-mon[91579]: pgmap v2275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:53 smithi188 ceph-mon[92425]: pgmap v2275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:55 smithi161 ceph-mon[91579]: pgmap v2276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:55 smithi188 ceph-mon[92425]: pgmap v2276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:54:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:57 smithi161 ceph-mon[91579]: pgmap v2277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:54:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:54:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:54:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:54:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:57 smithi188 ceph-mon[92425]: pgmap v2277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:54:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:54:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:54:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:54:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:54:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:58 smithi161 ceph-mon[91579]: pgmap v2278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:54:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:58 smithi188 ceph-mon[92425]: pgmap v2278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:54:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:00 smithi161 ceph-mon[91579]: pgmap v2279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:00 smithi188 ceph-mon[92425]: pgmap v2279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:03 smithi161 ceph-mon[91579]: pgmap v2280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:03 smithi188 ceph-mon[92425]: pgmap v2280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:05 smithi161 ceph-mon[91579]: pgmap v2281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:05 smithi188 ceph-mon[92425]: pgmap v2281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:07 smithi161 ceph-mon[91579]: pgmap v2282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:07 smithi188 ceph-mon[92425]: pgmap v2282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:08 smithi161 ceph-mon[91579]: pgmap v2283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:08 smithi188 ceph-mon[92425]: pgmap v2283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:10 smithi161 ceph-mon[91579]: pgmap v2284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:10 smithi188 ceph-mon[92425]: pgmap v2284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:13 smithi161 ceph-mon[91579]: pgmap v2285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:13 smithi188 ceph-mon[92425]: pgmap v2285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:15 smithi161 ceph-mon[91579]: pgmap v2286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:15 smithi188 ceph-mon[92425]: pgmap v2286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:17 smithi161 ceph-mon[91579]: pgmap v2287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:17 smithi188 ceph-mon[92425]: pgmap v2287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:18 smithi161 ceph-mon[91579]: pgmap v2288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:18 smithi188 ceph-mon[92425]: pgmap v2288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:20 smithi161 ceph-mon[91579]: pgmap v2289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:20 smithi188 ceph-mon[92425]: pgmap v2289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:23 smithi161 ceph-mon[91579]: pgmap v2290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:23 smithi188 ceph-mon[92425]: pgmap v2290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:25 smithi161 ceph-mon[91579]: pgmap v2291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:25 smithi188 ceph-mon[92425]: pgmap v2291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:27 smithi161 ceph-mon[91579]: pgmap v2292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:27 smithi188 ceph-mon[92425]: pgmap v2292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:28 smithi161 ceph-mon[91579]: pgmap v2293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:28 smithi188 ceph-mon[92425]: pgmap v2293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:30 smithi161 ceph-mon[91579]: pgmap v2294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:30 smithi188 ceph-mon[92425]: pgmap v2294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:33 smithi161 ceph-mon[91579]: pgmap v2295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:33 smithi188 ceph-mon[92425]: pgmap v2295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:35 smithi161 ceph-mon[91579]: pgmap v2296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:35 smithi188 ceph-mon[92425]: pgmap v2296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:37 smithi161 ceph-mon[91579]: pgmap v2297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:37 smithi188 ceph-mon[92425]: pgmap v2297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:38 smithi161 ceph-mon[91579]: pgmap v2298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:38 smithi188 ceph-mon[92425]: pgmap v2298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:40 smithi161 ceph-mon[91579]: pgmap v2299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:40 smithi188 ceph-mon[92425]: pgmap v2299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:43 smithi161 ceph-mon[91579]: pgmap v2300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:43 smithi188 ceph-mon[92425]: pgmap v2300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:45.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:55:44.775+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:55:45.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:55:45.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:55:45.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:55:45.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:55:45.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:55:45.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:55:45.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:55:45.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:55:45.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:55:45.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:55:45.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:55:45.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:55:45.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:55:45.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:55:45.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:55:45.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:55:45.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:55:45.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:55:45.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:55:45.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:55:45.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:55:45.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:55:45.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:55:45.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:55:45.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:55:45.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:55:45.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:55:45.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:55:45.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:55:45.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:55:45.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:55:45.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:55:45.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:55:45.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:55:45.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:55:45.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:55:45.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:55:45.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:55:45.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:55:45.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:55:45.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:55:45.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:55:45.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:55:45.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:55:45.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:55:45.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=107] Local address: 172.21.15.161, port 51780 2022-07-03T11:55:45.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=107] Peer address: 172.21.15.101, port 22 2022-07-03T11:55:45.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=107] Connection lost 2022-07-03T11:55:45.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=107] Aborting connection 2022-07-03T11:55:45.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:55:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:55:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:45 smithi161 ceph-mon[91579]: pgmap v2301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:55:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:55:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:55:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:45 smithi188 ceph-mon[92425]: pgmap v2301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:55:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:55:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:55:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:55:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:55:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:55:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:55:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:55:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:55:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:55:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:55:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:55:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:55:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:55:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:55:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:55:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:55:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:55:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:55:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:55:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:55:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:55:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:55:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:55:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:55:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:55:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:55:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:55:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:55:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:55:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:55:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:55:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:55:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:55:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:55:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:55:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:55:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:55:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:55:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:55:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:55:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:55:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:55:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:55:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:55:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:55:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:55:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:55:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:55:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:55:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=107] Local address: 172.21.15.161, port 51780 2022-07-03T11:55:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=107] Peer address: 172.21.15.101, port 22 2022-07-03T11:55:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=107] Connection lost 2022-07-03T11:55:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=107] Aborting connection 2022-07-03T11:55:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:55:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:55:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:55:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:55:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:55:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:55:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:55:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:55:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:55:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:55:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:55:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:55:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:55:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:55:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:55:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:55:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:55:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:55:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:55:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:55:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:55:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:55:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:55:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:55:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:55:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:55:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:55:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:55:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:55:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:55:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:55:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:55:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:55:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:55:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:55:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:55:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:55:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:55:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:55:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:55:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:55:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:55:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:55:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:55:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:55:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:55:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:55:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=107] Local address: 172.21.15.161, port 51780 2022-07-03T11:55:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=107] Peer address: 172.21.15.101, port 22 2022-07-03T11:55:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=107] Connection lost 2022-07-03T11:55:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=107] Aborting connection 2022-07-03T11:55:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:47 smithi161 ceph-mon[91579]: pgmap v2302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:47 smithi188 ceph-mon[92425]: pgmap v2302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:48 smithi161 ceph-mon[91579]: pgmap v2303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:48 smithi188 ceph-mon[92425]: pgmap v2303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:50 smithi161 ceph-mon[91579]: pgmap v2304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:50 smithi188 ceph-mon[92425]: pgmap v2304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:53 smithi161 ceph-mon[91579]: pgmap v2305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:53 smithi188 ceph-mon[92425]: pgmap v2305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:55 smithi161 ceph-mon[91579]: pgmap v2306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:55 smithi188 ceph-mon[92425]: pgmap v2306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:55:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:57 smithi161 ceph-mon[91579]: pgmap v2307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:55:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:55:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:55:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:55:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:57 smithi188 ceph-mon[92425]: pgmap v2307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:55:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:55:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:55:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:55:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:55:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:58 smithi161 ceph-mon[91579]: pgmap v2308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:55:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:58 smithi188 ceph-mon[92425]: pgmap v2308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:55:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:00 smithi161 ceph-mon[91579]: pgmap v2309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:00 smithi188 ceph-mon[92425]: pgmap v2309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:03 smithi161 ceph-mon[91579]: pgmap v2310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:03 smithi188 ceph-mon[92425]: pgmap v2310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:05 smithi161 ceph-mon[91579]: pgmap v2311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:05 smithi188 ceph-mon[92425]: pgmap v2311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:07 smithi161 ceph-mon[91579]: pgmap v2312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:07 smithi188 ceph-mon[92425]: pgmap v2312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:08 smithi161 ceph-mon[91579]: pgmap v2313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:08 smithi188 ceph-mon[92425]: pgmap v2313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:10 smithi161 ceph-mon[91579]: pgmap v2314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:10 smithi188 ceph-mon[92425]: pgmap v2314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:13 smithi161 ceph-mon[91579]: pgmap v2315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:13 smithi188 ceph-mon[92425]: pgmap v2315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:15 smithi161 ceph-mon[91579]: pgmap v2316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:15 smithi188 ceph-mon[92425]: pgmap v2316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:17 smithi161 ceph-mon[91579]: pgmap v2317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:17 smithi188 ceph-mon[92425]: pgmap v2317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:18 smithi161 ceph-mon[91579]: pgmap v2318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:18 smithi188 ceph-mon[92425]: pgmap v2318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:20 smithi161 ceph-mon[91579]: pgmap v2319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:20 smithi188 ceph-mon[92425]: pgmap v2319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:23 smithi161 ceph-mon[91579]: pgmap v2320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:23 smithi188 ceph-mon[92425]: pgmap v2320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:25 smithi161 ceph-mon[91579]: pgmap v2321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:25 smithi188 ceph-mon[92425]: pgmap v2321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:27 smithi161 ceph-mon[91579]: pgmap v2322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:27 smithi188 ceph-mon[92425]: pgmap v2322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:28 smithi161 ceph-mon[91579]: pgmap v2323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:28 smithi188 ceph-mon[92425]: pgmap v2323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:30 smithi161 ceph-mon[91579]: pgmap v2324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:30 smithi188 ceph-mon[92425]: pgmap v2324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:33 smithi161 ceph-mon[91579]: pgmap v2325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:33 smithi188 ceph-mon[92425]: pgmap v2325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:35 smithi161 ceph-mon[91579]: pgmap v2326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:35 smithi188 ceph-mon[92425]: pgmap v2326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:37 smithi161 ceph-mon[91579]: pgmap v2327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:37 smithi188 ceph-mon[92425]: pgmap v2327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:38 smithi161 ceph-mon[91579]: pgmap v2328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:38 smithi188 ceph-mon[92425]: pgmap v2328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:40 smithi161 ceph-mon[91579]: pgmap v2329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:40 smithi188 ceph-mon[92425]: pgmap v2329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:43 smithi161 ceph-mon[91579]: pgmap v2330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:43 smithi188 ceph-mon[92425]: pgmap v2330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:45 smithi161 ceph-mon[91579]: pgmap v2331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:45 smithi188 ceph-mon[92425]: pgmap v2331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:56:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:56:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:56:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:56:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:56:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:56:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:47 smithi161 ceph-mon[91579]: pgmap v2332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:47 smithi188 ceph-mon[92425]: pgmap v2332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:48 smithi161 ceph-mon[91579]: pgmap v2333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:48 smithi188 ceph-mon[92425]: pgmap v2333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:50 smithi161 ceph-mon[91579]: pgmap v2334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:50 smithi188 ceph-mon[92425]: pgmap v2334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:53 smithi161 ceph-mon[91579]: pgmap v2335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:53 smithi188 ceph-mon[92425]: pgmap v2335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:55 smithi161 ceph-mon[91579]: pgmap v2336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:55 smithi188 ceph-mon[92425]: pgmap v2336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:56:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:57 smithi161 ceph-mon[91579]: pgmap v2337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:56:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:56:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:56:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:56:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:57 smithi188 ceph-mon[92425]: pgmap v2337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:56:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:56:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:56:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:56:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:56:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:58 smithi161 ceph-mon[91579]: pgmap v2338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:56:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:58 smithi188 ceph-mon[92425]: pgmap v2338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:56:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:00 smithi161 ceph-mon[91579]: pgmap v2339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:00 smithi188 ceph-mon[92425]: pgmap v2339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:03 smithi161 ceph-mon[91579]: pgmap v2340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:03 smithi188 ceph-mon[92425]: pgmap v2340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:05 smithi161 ceph-mon[91579]: pgmap v2341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:05 smithi188 ceph-mon[92425]: pgmap v2341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:07 smithi161 ceph-mon[91579]: pgmap v2342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:07 smithi188 ceph-mon[92425]: pgmap v2342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:08 smithi161 ceph-mon[91579]: pgmap v2343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:08 smithi188 ceph-mon[92425]: pgmap v2343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:10 smithi161 ceph-mon[91579]: pgmap v2344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:10 smithi188 ceph-mon[92425]: pgmap v2344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:13 smithi161 ceph-mon[91579]: pgmap v2345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:13 smithi188 ceph-mon[92425]: pgmap v2345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:15 smithi161 ceph-mon[91579]: pgmap v2346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:15 smithi188 ceph-mon[92425]: pgmap v2346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:57:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:57:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:17 smithi161 ceph-mon[91579]: pgmap v2347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:17 smithi188 ceph-mon[92425]: pgmap v2347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:18 smithi161 ceph-mon[91579]: pgmap v2348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:18 smithi188 ceph-mon[92425]: pgmap v2348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:20 smithi161 ceph-mon[91579]: pgmap v2349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:20 smithi188 ceph-mon[92425]: pgmap v2349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:23 smithi161 ceph-mon[91579]: pgmap v2350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:23 smithi188 ceph-mon[92425]: pgmap v2350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:25 smithi161 ceph-mon[91579]: pgmap v2351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:25 smithi188 ceph-mon[92425]: pgmap v2351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:27 smithi161 ceph-mon[91579]: pgmap v2352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:27 smithi188 ceph-mon[92425]: pgmap v2352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:28 smithi161 ceph-mon[91579]: pgmap v2353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:28 smithi188 ceph-mon[92425]: pgmap v2353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:30 smithi161 ceph-mon[91579]: pgmap v2354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:30 smithi188 ceph-mon[92425]: pgmap v2354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:33 smithi161 ceph-mon[91579]: pgmap v2355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:33 smithi188 ceph-mon[92425]: pgmap v2355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:35 smithi161 ceph-mon[91579]: pgmap v2356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:35 smithi188 ceph-mon[92425]: pgmap v2356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:37 smithi188 ceph-mon[92425]: pgmap v2357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:37 smithi161 ceph-mon[91579]: pgmap v2357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:38 smithi161 ceph-mon[91579]: pgmap v2358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:38 smithi188 ceph-mon[92425]: pgmap v2358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:40 smithi161 ceph-mon[91579]: pgmap v2359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:40 smithi188 ceph-mon[92425]: pgmap v2359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:43 smithi161 ceph-mon[91579]: pgmap v2360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:43 smithi188 ceph-mon[92425]: pgmap v2360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-mon[91579]: pgmap v2361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:45.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:57:45.282+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:57:45.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:57:45.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:57:45.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:57:45.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:57:45.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:57:45.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:57:45.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:57:45.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:57:45.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:57:45.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:57:45.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:57:45.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:57:45.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:57:45.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:57:45.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:57:45.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:57:45.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:57:45.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:57:45.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:57:45.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:57:45.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:57:45.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:57:45.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:57:45.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:57:45.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:57:45.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:57:45.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:57:45.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:57:45.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:57:45.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:57:45.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:57:45.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:57:45.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:57:45.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:57:45.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:57:45.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:57:45.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:57:45.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:57:45.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:57:45.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:57:45.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:57:45.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:57:45.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:57:45.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:57:45.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:57:45.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=109] Local address: 172.21.15.161, port 51820 2022-07-03T11:57:45.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=109] Peer address: 172.21.15.101, port 22 2022-07-03T11:57:45.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=109] Login timeout expired 2022-07-03T11:57:45.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=109] Aborting connection 2022-07-03T11:57:45.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:57:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:57:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:45 smithi188 ceph-mon[92425]: pgmap v2361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:57:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:57:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:57:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:57:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:57:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:57:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:57:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:57:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:57:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:57:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:57:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:57:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:57:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:57:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:57:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:57:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:57:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:57:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:57:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:57:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:57:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:57:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:57:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:57:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:57:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:57:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:57:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:57:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:57:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:57:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:57:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:57:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:57:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:57:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:57:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:57:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:57:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:57:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:57:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:57:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:57:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:57:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:57:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:57:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:57:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:57:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=109] Local address: 172.21.15.161, port 51820 2022-07-03T11:57:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=109] Peer address: 172.21.15.101, port 22 2022-07-03T11:57:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=109] Login timeout expired 2022-07-03T11:57:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=109] Aborting connection 2022-07-03T11:57:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:57:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:57:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:57:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:57:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:57:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:57:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:57:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:57:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:57:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:57:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:57:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:57:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T11:57:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:57:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:57:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:57:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:57:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:57:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:57:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:57:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:57:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:57:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:57:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:57:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:57:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:57:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:57:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:57:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:57:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:57:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:57:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:57:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:57:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:57:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:57:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:57:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:57:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:57:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:57:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:57:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:57:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:57:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:57:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T11:57:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:57:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:57:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=109] Local address: 172.21.15.161, port 51820 2022-07-03T11:57:46.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=109] Peer address: 172.21.15.101, port 22 2022-07-03T11:57:46.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=109] Login timeout expired 2022-07-03T11:57:46.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=109] Aborting connection 2022-07-03T11:57:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:47 smithi188 ceph-mon[92425]: pgmap v2362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:47 smithi161 ceph-mon[91579]: pgmap v2362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:48 smithi161 ceph-mon[91579]: pgmap v2363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:48 smithi188 ceph-mon[92425]: pgmap v2363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:50 smithi161 ceph-mon[91579]: pgmap v2364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:50 smithi188 ceph-mon[92425]: pgmap v2364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:53 smithi161 ceph-mon[91579]: pgmap v2365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:53 smithi188 ceph-mon[92425]: pgmap v2365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:55 smithi188 ceph-mon[92425]: pgmap v2366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:55 smithi161 ceph-mon[91579]: pgmap v2366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:57:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:57 smithi188 ceph-mon[92425]: pgmap v2367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:57:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:57:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:57:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:57:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:57 smithi161 ceph-mon[91579]: pgmap v2367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:57:57.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:57:57.672 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:57:57.673 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:57:57.673 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:57:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:58 smithi161 ceph-mon[91579]: pgmap v2368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:57:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:58 smithi188 ceph-mon[92425]: pgmap v2368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:57:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:00 smithi161 ceph-mon[91579]: pgmap v2369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:00 smithi188 ceph-mon[92425]: pgmap v2369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:03 smithi161 ceph-mon[91579]: pgmap v2370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:03 smithi188 ceph-mon[92425]: pgmap v2370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:05 smithi188 ceph-mon[92425]: pgmap v2371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:05 smithi161 ceph-mon[91579]: pgmap v2371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:07 smithi188 ceph-mon[92425]: pgmap v2372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:07 smithi161 ceph-mon[91579]: pgmap v2372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:08 smithi161 ceph-mon[91579]: pgmap v2373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:08.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:08 smithi188 ceph-mon[92425]: pgmap v2373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:08.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:10 smithi161 ceph-mon[91579]: pgmap v2374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:10 smithi188 ceph-mon[92425]: pgmap v2374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:13 smithi188 ceph-mon[92425]: pgmap v2375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:13 smithi161 ceph-mon[91579]: pgmap v2375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:15 smithi188 ceph-mon[92425]: pgmap v2376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:15 smithi161 ceph-mon[91579]: pgmap v2376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:17 smithi188 ceph-mon[92425]: pgmap v2377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:17 smithi161 ceph-mon[91579]: pgmap v2377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:18 smithi161 ceph-mon[91579]: pgmap v2378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:18 smithi188 ceph-mon[92425]: pgmap v2378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:20 smithi161 ceph-mon[91579]: pgmap v2379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:20 smithi188 ceph-mon[92425]: pgmap v2379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:23 smithi188 ceph-mon[92425]: pgmap v2380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:23 smithi161 ceph-mon[91579]: pgmap v2380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:24.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:25 smithi188 ceph-mon[92425]: pgmap v2381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:25 smithi161 ceph-mon[91579]: pgmap v2381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:27 smithi188 ceph-mon[92425]: pgmap v2382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:27 smithi161 ceph-mon[91579]: pgmap v2382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:28 smithi161 ceph-mon[91579]: pgmap v2383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:28 smithi188 ceph-mon[92425]: pgmap v2383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:30 smithi161 ceph-mon[91579]: pgmap v2384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:30 smithi188 ceph-mon[92425]: pgmap v2384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:33 smithi188 ceph-mon[92425]: pgmap v2385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:33 smithi161 ceph-mon[91579]: pgmap v2385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:35 smithi188 ceph-mon[92425]: pgmap v2386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:35 smithi161 ceph-mon[91579]: pgmap v2386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:37 smithi188 ceph-mon[92425]: pgmap v2387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:37 smithi161 ceph-mon[91579]: pgmap v2387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:38 smithi161 ceph-mon[91579]: pgmap v2388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:38 smithi188 ceph-mon[92425]: pgmap v2388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:40 smithi161 ceph-mon[91579]: pgmap v2389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:40 smithi188 ceph-mon[92425]: pgmap v2389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:43 smithi188 ceph-mon[92425]: pgmap v2390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:43 smithi161 ceph-mon[91579]: pgmap v2390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:45 smithi188 ceph-mon[92425]: pgmap v2391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-mon[91579]: pgmap v2391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:46.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:58:45.716+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:58:46.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:58:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:58:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:58:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:58:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:58:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:58:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:58:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:58:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:58:45.727+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:58:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:58:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:58:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:58:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:58:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:58:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:58:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:58:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T11:58:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:58:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:58:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:58:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T11:58:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:58:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T11:58:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:58:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:58:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:58:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:58:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:58:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T11:58:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:58:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:58:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:58:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T11:58:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:58:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T11:58:46.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:58:46.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:58:46.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:58:46.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T11:58:46.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:58:46.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T11:58:46.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:58:46.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T11:58:46.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:58:46.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T11:58:46.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:58:46.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:58:46.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:58:46.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=111] Local address: 172.21.15.161, port 51860 2022-07-03T11:58:46.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=111] Peer address: 172.21.15.101, port 22 2022-07-03T11:58:46.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=111] Connection lost 2022-07-03T11:58:46.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=111] Aborting connection 2022-07-03T11:58:46.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:58:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:58:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:58:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:58:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:58:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:58:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:58:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:58:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:47 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:58:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:58:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:58:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:58:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:58:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:58:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:58:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:58:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:58:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:58:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:58:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:58:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:58:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:58:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:58:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:58:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:58:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T11:58:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:58:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T11:58:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:58:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T11:58:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:58:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:58:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:58:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:58:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:58:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:58:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:58:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:58:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:58:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T11:58:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:58:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T11:58:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:58:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:58:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:58:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:58:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:58:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T11:58:47.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:58:47.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:58:47.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:58:47.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T11:58:47.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:58:47.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:58:47.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:58:47.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=111] Local address: 172.21.15.161, port 51860 2022-07-03T11:58:47.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=111] Peer address: 172.21.15.101, port 22 2022-07-03T11:58:47.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=111] Connection lost 2022-07-03T11:58:47.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=111] Aborting connection 2022-07-03T11:58:47.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:47 smithi188 ceph-mon[92425]: pgmap v2392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:47 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:58:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:58:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:58:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:47.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:58:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:58:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:58:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:58:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:58:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:58:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T11:58:47.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:58:47.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:58:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:58:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:58:47.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:58:47.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:58:47.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:58:47.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:58:47.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:58:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:58:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T11:58:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:58:47.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T11:58:47.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T11:58:47.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:58:47.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T11:58:47.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T11:58:47.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T11:58:47.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T11:58:47.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T11:58:47.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T11:58:47.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T11:58:47.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T11:58:47.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T11:58:47.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T11:58:47.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T11:58:47.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T11:58:47.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T11:58:47.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T11:58:47.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T11:58:47.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T11:58:47.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T11:58:47.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T11:58:47.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T11:58:47.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T11:58:47.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T11:58:47.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T11:58:47.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T11:58:47.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T11:58:47.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T11:58:47.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T11:58:47.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T11:58:47.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T11:58:47.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T11:58:47.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=111] Local address: 172.21.15.161, port 51860 2022-07-03T11:58:47.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=111] Peer address: 172.21.15.101, port 22 2022-07-03T11:58:47.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=111] Connection lost 2022-07-03T11:58:47.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=111] Aborting connection 2022-07-03T11:58:47.665 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:47 smithi161 ceph-mon[91579]: pgmap v2392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:48 smithi161 ceph-mon[91579]: pgmap v2393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:48 smithi188 ceph-mon[92425]: pgmap v2393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:50 smithi161 ceph-mon[91579]: pgmap v2394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:50 smithi188 ceph-mon[92425]: pgmap v2394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:53 smithi188 ceph-mon[92425]: pgmap v2395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:53 smithi161 ceph-mon[91579]: pgmap v2395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:55 smithi188 ceph-mon[92425]: pgmap v2396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:55 smithi161 ceph-mon[91579]: pgmap v2396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:58:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:57 smithi188 ceph-mon[92425]: pgmap v2397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:58:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:58:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:58:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:58:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:57 smithi161 ceph-mon[91579]: pgmap v2397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:58:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:58:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:58:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:58:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:58:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:58 smithi161 ceph-mon[91579]: pgmap v2398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:58:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:58 smithi188 ceph-mon[92425]: pgmap v2398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:58:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:00 smithi161 ceph-mon[91579]: pgmap v2399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:00 smithi188 ceph-mon[92425]: pgmap v2399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:03 smithi188 ceph-mon[92425]: pgmap v2400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:03 smithi161 ceph-mon[91579]: pgmap v2400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:05 smithi188 ceph-mon[92425]: pgmap v2401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:05 smithi161 ceph-mon[91579]: pgmap v2401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:07 smithi188 ceph-mon[92425]: pgmap v2402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:07 smithi161 ceph-mon[91579]: pgmap v2402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:08 smithi161 ceph-mon[91579]: pgmap v2403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:08 smithi188 ceph-mon[92425]: pgmap v2403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:10 smithi161 ceph-mon[91579]: pgmap v2404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:10 smithi188 ceph-mon[92425]: pgmap v2404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:13 smithi188 ceph-mon[92425]: pgmap v2405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:13 smithi161 ceph-mon[91579]: pgmap v2405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:15 smithi188 ceph-mon[92425]: pgmap v2406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:15 smithi161 ceph-mon[91579]: pgmap v2406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:17 smithi188 ceph-mon[92425]: pgmap v2407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:17 smithi161 ceph-mon[91579]: pgmap v2407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:18 smithi161 ceph-mon[91579]: pgmap v2408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:18 smithi188 ceph-mon[92425]: pgmap v2408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:20.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:20 smithi161 ceph-mon[91579]: pgmap v2409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:20 smithi188 ceph-mon[92425]: pgmap v2409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:23 smithi188 ceph-mon[92425]: pgmap v2410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:23 smithi161 ceph-mon[91579]: pgmap v2410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:25 smithi188 ceph-mon[92425]: pgmap v2411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:25 smithi161 ceph-mon[91579]: pgmap v2411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:27 smithi188 ceph-mon[92425]: pgmap v2412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:27 smithi161 ceph-mon[91579]: pgmap v2412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:28 smithi161 ceph-mon[91579]: pgmap v2413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:28 smithi188 ceph-mon[92425]: pgmap v2413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:30 smithi161 ceph-mon[91579]: pgmap v2414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:30 smithi188 ceph-mon[92425]: pgmap v2414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:33 smithi188 ceph-mon[92425]: pgmap v2415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:33 smithi161 ceph-mon[91579]: pgmap v2415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:35 smithi188 ceph-mon[92425]: pgmap v2416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:35 smithi161 ceph-mon[91579]: pgmap v2416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:37 smithi161 ceph-mon[91579]: pgmap v2417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:37 smithi188 ceph-mon[92425]: pgmap v2417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:38 smithi161 ceph-mon[91579]: pgmap v2418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:38 smithi188 ceph-mon[92425]: pgmap v2418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:40 smithi161 ceph-mon[91579]: pgmap v2419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:40 smithi188 ceph-mon[92425]: pgmap v2419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:43 smithi188 ceph-mon[92425]: pgmap v2420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:43 smithi161 ceph-mon[91579]: pgmap v2420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:45 smithi161 ceph-mon[91579]: pgmap v2421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:45 smithi188 ceph-mon[92425]: pgmap v2421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:59:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:59:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:59:46.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T11:59:46.221+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:59:46.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T11:59:46.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:59:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T11:59:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:59:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:59:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:59:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T11:59:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:59:46.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T11:59:46.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:59:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T11:59:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:59:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T11:59:46.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 11:59:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:59:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T11:59:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T11:59:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T11:59:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:47 smithi161 ceph-mon[91579]: pgmap v2422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:47 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:59:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T11:59:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:59:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T11:59:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:59:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:59:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:59:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T11:59:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:59:47.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T11:59:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:59:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T11:59:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:59:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:59:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:59:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:59:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:47 smithi188 ceph-mon[92425]: pgmap v2422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:47 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T11:59:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T11:59:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T11:59:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T11:59:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T11:59:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T11:59:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T11:59:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T11:59:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T11:59:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T11:59:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T11:59:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T11:59:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T11:59:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T11:59:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T11:59:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T11:59:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:48 smithi161 ceph-mon[91579]: pgmap v2423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:48 smithi188 ceph-mon[92425]: pgmap v2423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:50 smithi161 ceph-mon[91579]: pgmap v2424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:50 smithi188 ceph-mon[92425]: pgmap v2424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:53 smithi188 ceph-mon[92425]: pgmap v2425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:53 smithi161 ceph-mon[91579]: pgmap v2425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:55 smithi161 ceph-mon[91579]: pgmap v2426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:55 smithi188 ceph-mon[92425]: pgmap v2426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T11:59:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:57 smithi161 ceph-mon[91579]: pgmap v2427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:59:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:59:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:59:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:59:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:57 smithi188 ceph-mon[92425]: pgmap v2427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T11:59:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:59:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:59:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T11:59:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T11:59:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:58 smithi161 ceph-mon[91579]: pgmap v2428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T11:59:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:58 smithi188 ceph-mon[92425]: pgmap v2428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T11:59:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 11:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 11:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:00:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:00 smithi161 ceph-mon[91579]: pgmap v2429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:00:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:00 smithi188 ceph-mon[92425]: pgmap v2429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:03 smithi161 ceph-mon[91579]: pgmap v2430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:03 smithi188 ceph-mon[92425]: pgmap v2430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:05 smithi161 ceph-mon[91579]: pgmap v2431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:05.662 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:05 smithi188 ceph-mon[92425]: pgmap v2431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:07 smithi161 ceph-mon[91579]: pgmap v2432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:07 smithi188 ceph-mon[92425]: pgmap v2432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:08 smithi161 ceph-mon[91579]: pgmap v2433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:08.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:08 smithi188 ceph-mon[92425]: pgmap v2433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:08.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:10 smithi161 ceph-mon[91579]: pgmap v2434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:10 smithi188 ceph-mon[92425]: pgmap v2434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:13 smithi161 ceph-mon[91579]: pgmap v2435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:13 smithi188 ceph-mon[92425]: pgmap v2435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:15 smithi161 ceph-mon[91579]: pgmap v2436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:15 smithi188 ceph-mon[92425]: pgmap v2436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:16.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:00:16.235+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:00:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:00:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:00:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:00:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:00:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:00:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:00:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:00:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:00:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:00:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:00:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:00:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:00:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:00:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:00:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:00:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:00:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:00:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:00:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:00:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:00:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:00:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:00:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:00:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:00:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:00:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:00:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:00:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:00:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:00:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:00:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:00:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:00:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:00:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:00:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:00:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:00:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:00:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:00:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:00:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:00:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:00:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:00:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:00:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:00:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:00:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=113] Local address: 172.21.15.161, port 51888 2022-07-03T12:00:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=113] Peer address: 172.21.15.101, port 22 2022-07-03T12:00:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=355] Received exit status 0 2022-07-03T12:00:16.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=355] Received channel close 2022-07-03T12:00:16.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=355] Channel closed 2022-07-03T12:00:16.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=355] Received exit status 0 2022-07-03T12:00:16.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=355] Received channel close 2022-07-03T12:00:16.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=355] Channel closed 2022-07-03T12:00:16.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=113] Login timeout expired 2022-07-03T12:00:16.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=113] Aborting connection 2022-07-03T12:00:16.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:00:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:00:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:17 smithi161 ceph-mon[91579]: pgmap v2437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:00:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:00:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:00:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:00:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:00:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:00:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:00:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:00:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:00:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:00:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:00:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:00:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:00:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:00:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:00:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:00:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:00:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:00:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:00:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:00:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:00:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:00:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:00:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:00:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:00:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:00:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:00:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:00:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:00:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:00:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:00:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:00:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:00:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:00:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:00:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:00:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:00:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:00:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:00:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:00:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:00:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:00:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:00:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:00:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:00:17.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:00:17.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=113] Local address: 172.21.15.161, port 51888 2022-07-03T12:00:17.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=113] Peer address: 172.21.15.101, port 22 2022-07-03T12:00:17.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=355] Received exit status 0 2022-07-03T12:00:17.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=355] Received channel close 2022-07-03T12:00:17.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=355] Channel closed 2022-07-03T12:00:17.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=355] Received exit status 0 2022-07-03T12:00:17.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=355] Received channel close 2022-07-03T12:00:17.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=355] Channel closed 2022-07-03T12:00:17.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=113] Login timeout expired 2022-07-03T12:00:17.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=113] Aborting connection 2022-07-03T12:00:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:17 smithi188 ceph-mon[92425]: pgmap v2437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:00:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:00:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:00:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:00:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:00:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:00:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:00:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:00:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:00:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:00:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:00:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:00:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:00:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:00:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:00:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:00:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:00:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:00:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:00:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:00:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:00:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:00:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:00:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:00:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:00:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:00:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:00:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:00:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:00:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:00:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:00:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:00:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:00:17.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:00:17.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:00:17.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:00:17.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:00:17.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:00:17.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:00:17.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:00:17.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:00:17.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:00:17.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:00:17.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:00:17.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:00:17.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:00:17.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=113] Local address: 172.21.15.161, port 51888 2022-07-03T12:00:17.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=113] Peer address: 172.21.15.101, port 22 2022-07-03T12:00:17.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=355] Received exit status 0 2022-07-03T12:00:17.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=355] Received channel close 2022-07-03T12:00:17.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=355] Channel closed 2022-07-03T12:00:17.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=355] Received exit status 0 2022-07-03T12:00:17.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=355] Received channel close 2022-07-03T12:00:17.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=355] Channel closed 2022-07-03T12:00:17.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=113] Login timeout expired 2022-07-03T12:00:17.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=113] Aborting connection 2022-07-03T12:00:18.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:18 smithi161 ceph-mon[91579]: pgmap v2438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:18.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:18 smithi188 ceph-mon[92425]: pgmap v2438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:20 smithi161 ceph-mon[91579]: pgmap v2439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:20 smithi188 ceph-mon[92425]: pgmap v2439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:23 smithi161 ceph-mon[91579]: pgmap v2440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:23 smithi188 ceph-mon[92425]: pgmap v2440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:25 smithi161 ceph-mon[91579]: pgmap v2441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:25 smithi188 ceph-mon[92425]: pgmap v2441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:27 smithi161 ceph-mon[91579]: pgmap v2442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:27 smithi188 ceph-mon[92425]: pgmap v2442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:28 smithi161 ceph-mon[91579]: pgmap v2443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:28 smithi188 ceph-mon[92425]: pgmap v2443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:30 smithi161 ceph-mon[91579]: pgmap v2444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:30 smithi188 ceph-mon[92425]: pgmap v2444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:33 smithi161 ceph-mon[91579]: pgmap v2445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:33 smithi188 ceph-mon[92425]: pgmap v2445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:35 smithi161 ceph-mon[91579]: pgmap v2446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:35 smithi188 ceph-mon[92425]: pgmap v2446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:37 smithi161 ceph-mon[91579]: pgmap v2447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:37 smithi188 ceph-mon[92425]: pgmap v2447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:38 smithi161 ceph-mon[91579]: pgmap v2448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:38 smithi188 ceph-mon[92425]: pgmap v2448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:40.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:40.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:40 smithi161 ceph-mon[91579]: pgmap v2449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:40 smithi188 ceph-mon[92425]: pgmap v2449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:43 smithi161 ceph-mon[91579]: pgmap v2450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:43 smithi188 ceph-mon[92425]: pgmap v2450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:45 smithi161 ceph-mon[91579]: pgmap v2451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:45 smithi188 ceph-mon[92425]: pgmap v2451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:47 smithi161 ceph-mon[91579]: pgmap v2452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:47 smithi188 ceph-mon[92425]: pgmap v2452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:48 smithi161 ceph-mon[91579]: pgmap v2453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:48 smithi188 ceph-mon[92425]: pgmap v2453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:50 smithi161 ceph-mon[91579]: pgmap v2454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:50 smithi188 ceph-mon[92425]: pgmap v2454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:53 smithi161 ceph-mon[91579]: pgmap v2455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:53 smithi188 ceph-mon[92425]: pgmap v2455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:55 smithi161 ceph-mon[91579]: pgmap v2456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:55 smithi188 ceph-mon[92425]: pgmap v2456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:00:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:57 smithi161 ceph-mon[91579]: pgmap v2457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:00:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:00:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:00:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:00:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:57 smithi188 ceph-mon[92425]: pgmap v2457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:00:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:00:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:00:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:00:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:00:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:58 smithi188 ceph-mon[92425]: pgmap v2458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:00:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:58 smithi161 ceph-mon[91579]: pgmap v2458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:00:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:00 smithi161 ceph-mon[91579]: pgmap v2459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:00 smithi188 ceph-mon[92425]: pgmap v2459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:03 smithi161 ceph-mon[91579]: pgmap v2460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:03 smithi188 ceph-mon[92425]: pgmap v2460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:05 smithi161 ceph-mon[91579]: pgmap v2461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:05 smithi188 ceph-mon[92425]: pgmap v2461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:07 smithi161 ceph-mon[91579]: pgmap v2462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:07 smithi188 ceph-mon[92425]: pgmap v2462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:08 smithi188 ceph-mon[92425]: pgmap v2463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:08 smithi161 ceph-mon[91579]: pgmap v2463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:10 smithi161 ceph-mon[91579]: pgmap v2464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:10 smithi188 ceph-mon[92425]: pgmap v2464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:13 smithi161 ceph-mon[91579]: pgmap v2465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:13 smithi188 ceph-mon[92425]: pgmap v2465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:15 smithi161 ceph-mon[91579]: pgmap v2466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:15 smithi188 ceph-mon[92425]: pgmap v2466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:01:16.274+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:01:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:01:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:01:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:01:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:01:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:01:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:01:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:01:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:01:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:01:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:01:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:01:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:01:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:01:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:01:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:01:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:01:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:01:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:01:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:01:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:01:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:01:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:01:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:01:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:01:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:01:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:01:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:01:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:01:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:01:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:01:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:01:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:01:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:01:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:01:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:01:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:01:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:01:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:01:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:01:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:01:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:01:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:01:16.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:01:16.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:01:16.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:01:16.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=115] Local address: 172.21.15.161, port 51928 2022-07-03T12:01:16.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=115] Peer address: 172.21.15.101, port 22 2022-07-03T12:01:16.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=115] Connection lost 2022-07-03T12:01:16.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=115] Aborting connection 2022-07-03T12:01:16.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:01:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:01:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: pgmap v2467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:01:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:01:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:01:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:01:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:01:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:01:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:01:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:01:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:01:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:01:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:01:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:01:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:01:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:01:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:01:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:01:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:01:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:01:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:01:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:01:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:01:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:01:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:01:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:01:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:01:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:01:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:01:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:01:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:01:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:01:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:01:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:01:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:01:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:01:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:01:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:01:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:01:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:01:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:01:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:01:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:01:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:01:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:01:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:01:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:01:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:01:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:01:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:01:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:01:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:01:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=115] Local address: 172.21.15.161, port 51928 2022-07-03T12:01:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=115] Peer address: 172.21.15.101, port 22 2022-07-03T12:01:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=115] Connection lost 2022-07-03T12:01:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=115] Aborting connection 2022-07-03T12:01:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: pgmap v2467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:01:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:01:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:01:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:01:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:01:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:01:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:01:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:01:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:01:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:01:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:01:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:01:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:01:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:01:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:01:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:01:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:01:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:01:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:01:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:01:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:01:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:01:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:01:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:01:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:01:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:01:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:01:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:01:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:01:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:01:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:01:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:01:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:01:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:01:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:01:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:01:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:01:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:01:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:01:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:01:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:01:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:01:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:01:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:01:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:01:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:01:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:01:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:01:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:01:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:01:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=115] Local address: 172.21.15.161, port 51928 2022-07-03T12:01:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=115] Peer address: 172.21.15.101, port 22 2022-07-03T12:01:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=115] Connection lost 2022-07-03T12:01:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=115] Aborting connection 2022-07-03T12:01:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:18 smithi188 ceph-mon[92425]: pgmap v2468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:18 smithi161 ceph-mon[91579]: pgmap v2468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:20 smithi161 ceph-mon[91579]: pgmap v2469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:20 smithi188 ceph-mon[92425]: pgmap v2469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:23 smithi161 ceph-mon[91579]: pgmap v2470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:23 smithi188 ceph-mon[92425]: pgmap v2470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:01:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:25 smithi161 ceph-mon[91579]: pgmap v2471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:25 smithi188 ceph-mon[92425]: pgmap v2471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:27 smithi161 ceph-mon[91579]: pgmap v2472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:27 smithi188 ceph-mon[92425]: pgmap v2472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:28 smithi188 ceph-mon[92425]: pgmap v2473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:28 smithi161 ceph-mon[91579]: pgmap v2473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:30 smithi161 ceph-mon[91579]: pgmap v2474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:30 smithi188 ceph-mon[92425]: pgmap v2474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:33 smithi161 ceph-mon[91579]: pgmap v2475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:33 smithi188 ceph-mon[92425]: pgmap v2475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:35 smithi161 ceph-mon[91579]: pgmap v2476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:35 smithi188 ceph-mon[92425]: pgmap v2476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:37 smithi161 ceph-mon[91579]: pgmap v2477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:37 smithi188 ceph-mon[92425]: pgmap v2477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:38 smithi188 ceph-mon[92425]: pgmap v2478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:38 smithi161 ceph-mon[91579]: pgmap v2478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:40 smithi188 ceph-mon[92425]: pgmap v2479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:40 smithi161 ceph-mon[91579]: pgmap v2479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:43 smithi161 ceph-mon[91579]: pgmap v2480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:43 smithi188 ceph-mon[92425]: pgmap v2480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:45 smithi161 ceph-mon[91579]: pgmap v2481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:45 smithi188 ceph-mon[92425]: pgmap v2481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:47 smithi161 ceph-mon[91579]: pgmap v2482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:47 smithi188 ceph-mon[92425]: pgmap v2482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:48 smithi188 ceph-mon[92425]: pgmap v2483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:48 smithi161 ceph-mon[91579]: pgmap v2483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:50.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:50 smithi188 ceph-mon[92425]: pgmap v2484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:50 smithi161 ceph-mon[91579]: pgmap v2484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:53 smithi161 ceph-mon[91579]: pgmap v2485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:53.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:53 smithi188 ceph-mon[92425]: pgmap v2485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:55 smithi161 ceph-mon[91579]: pgmap v2486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:55 smithi188 ceph-mon[92425]: pgmap v2486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:01:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:57 smithi161 ceph-mon[91579]: pgmap v2487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:01:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:01:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:01:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:01:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:57 smithi188 ceph-mon[92425]: pgmap v2487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:01:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:01:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:01:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:01:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:01:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:58 smithi188 ceph-mon[92425]: pgmap v2488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:01:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:58 smithi161 ceph-mon[91579]: pgmap v2488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:01:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:00 smithi188 ceph-mon[92425]: pgmap v2489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:00 smithi161 ceph-mon[91579]: pgmap v2489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:03 smithi161 ceph-mon[91579]: pgmap v2490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:03 smithi188 ceph-mon[92425]: pgmap v2490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:05 smithi161 ceph-mon[91579]: pgmap v2491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:05 smithi188 ceph-mon[92425]: pgmap v2491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:07 smithi161 ceph-mon[91579]: pgmap v2492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:07 smithi188 ceph-mon[92425]: pgmap v2492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:08 smithi188 ceph-mon[92425]: pgmap v2493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:08 smithi161 ceph-mon[91579]: pgmap v2493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:10 smithi188 ceph-mon[92425]: pgmap v2494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:10 smithi161 ceph-mon[91579]: pgmap v2494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:13 smithi161 ceph-mon[91579]: pgmap v2495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:13 smithi188 ceph-mon[92425]: pgmap v2495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:15 smithi161 ceph-mon[91579]: pgmap v2496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:15 smithi188 ceph-mon[92425]: pgmap v2496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:17.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:02:16.780+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:02:17.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:02:17.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:02:17.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:02:17.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:02:17.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:02:17.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:02:17.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:02:17.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:02:17.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:02:17.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:02:17.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:02:17.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:02:17.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:02:17.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:02:17.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:02:17.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:02:17.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:02:17.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:02:17.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:02:17.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:02:17.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:02:17.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:02:17.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:02:17.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:02:17.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:02:17.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:02:17.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:02:17.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:02:17.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:02:17.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:02:17.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:02:17.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:02:17.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:02:17.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:02:17.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:02:17.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:02:17.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:02:17.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:02:17.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:02:17.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:02:17.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:02:17.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:02:17.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:02:17.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:02:17.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:02:17.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=117] Local address: 172.21.15.161, port 51956 2022-07-03T12:02:17.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=117] Peer address: 172.21.15.101, port 22 2022-07-03T12:02:17.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=117] Connection lost 2022-07-03T12:02:17.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=117] Aborting connection 2022-07-03T12:02:17.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:02:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:02:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:17 smithi161 ceph-mon[91579]: pgmap v2497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:02:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:02:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:02:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:17 smithi188 ceph-mon[92425]: pgmap v2497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:02:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:02:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:02:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:02:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:18 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:02:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:02:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:02:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:02:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:02:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:02:18.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:02:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:02:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:02:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:02:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:02:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:02:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:02:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:02:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:02:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:02:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:02:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:02:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:02:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:02:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:02:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:02:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:02:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:02:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:02:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:02:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:02:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:02:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:02:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:02:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:02:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:02:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:02:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:02:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:02:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:02:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:02:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:02:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:02:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:02:18.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:02:18.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:02:18.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:02:18.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:02:18.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:02:18.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:02:18.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=117] Local address: 172.21.15.161, port 51956 2022-07-03T12:02:18.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=117] Peer address: 172.21.15.101, port 22 2022-07-03T12:02:18.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=117] Connection lost 2022-07-03T12:02:18.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=117] Aborting connection 2022-07-03T12:02:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:02:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:18 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:02:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:02:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:02:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:02:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:02:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:02:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:02:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:02:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:02:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:02:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:02:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:02:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:02:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:02:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:02:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:02:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:02:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:02:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:02:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:02:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:02:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:02:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:02:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:02:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:02:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:02:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:02:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:02:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:02:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:02:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:02:18.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:02:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:02:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:02:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:02:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:02:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:02:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:02:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:02:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:02:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:02:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:02:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:02:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:02:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:02:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:02:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=117] Local address: 172.21.15.161, port 51956 2022-07-03T12:02:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=117] Peer address: 172.21.15.101, port 22 2022-07-03T12:02:18.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=117] Connection lost 2022-07-03T12:02:18.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=117] Aborting connection 2022-07-03T12:02:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:19 smithi161 ceph-mon[91579]: pgmap v2498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:19 smithi188 ceph-mon[92425]: pgmap v2498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:21 smithi161 ceph-mon[91579]: pgmap v2499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:21 smithi188 ceph-mon[92425]: pgmap v2499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:23 smithi161 ceph-mon[91579]: pgmap v2500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:23 smithi188 ceph-mon[92425]: pgmap v2500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:25 smithi161 ceph-mon[91579]: pgmap v2501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:25 smithi188 ceph-mon[92425]: pgmap v2501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:27 smithi161 ceph-mon[91579]: pgmap v2502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:27 smithi188 ceph-mon[92425]: pgmap v2502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:28 smithi188 ceph-mon[92425]: pgmap v2503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:28 smithi161 ceph-mon[91579]: pgmap v2503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:30 smithi188 ceph-mon[92425]: pgmap v2504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:30 smithi161 ceph-mon[91579]: pgmap v2504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:33 smithi161 ceph-mon[91579]: pgmap v2505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:33 smithi188 ceph-mon[92425]: pgmap v2505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:35 smithi161 ceph-mon[91579]: pgmap v2506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:35 smithi188 ceph-mon[92425]: pgmap v2506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:37 smithi161 ceph-mon[91579]: pgmap v2507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:37 smithi188 ceph-mon[92425]: pgmap v2507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:38 smithi161 ceph-mon[91579]: pgmap v2508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:38 smithi188 ceph-mon[92425]: pgmap v2508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:40 smithi188 ceph-mon[92425]: pgmap v2509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:40 smithi161 ceph-mon[91579]: pgmap v2509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:43 smithi161 ceph-mon[91579]: pgmap v2510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:43 smithi188 ceph-mon[92425]: pgmap v2510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:45 smithi161 ceph-mon[91579]: pgmap v2511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:45 smithi188 ceph-mon[92425]: pgmap v2511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:47 smithi161 ceph-mon[91579]: pgmap v2512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:47 smithi188 ceph-mon[92425]: pgmap v2512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:48 smithi188 ceph-mon[92425]: pgmap v2513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:48 smithi161 ceph-mon[91579]: pgmap v2513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:50 smithi161 ceph-mon[91579]: pgmap v2514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:50 smithi188 ceph-mon[92425]: pgmap v2514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:53 smithi161 ceph-mon[91579]: pgmap v2515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:53 smithi188 ceph-mon[92425]: pgmap v2515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:55 smithi161 ceph-mon[91579]: pgmap v2516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:55 smithi188 ceph-mon[92425]: pgmap v2516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:02:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:57 smithi161 ceph-mon[91579]: pgmap v2517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:02:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:02:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:02:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:02:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:57 smithi188 ceph-mon[92425]: pgmap v2517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:02:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:02:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:02:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:02:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:02:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:58 smithi161 ceph-mon[91579]: pgmap v2518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:02:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:58 smithi188 ceph-mon[92425]: pgmap v2518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:02:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:00 smithi161 ceph-mon[91579]: pgmap v2519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:00 smithi188 ceph-mon[92425]: pgmap v2519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:03 smithi161 ceph-mon[91579]: pgmap v2520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:03 smithi188 ceph-mon[92425]: pgmap v2520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:05 smithi161 ceph-mon[91579]: pgmap v2521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:05 smithi188 ceph-mon[92425]: pgmap v2521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:07 smithi161 ceph-mon[91579]: pgmap v2522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:07 smithi188 ceph-mon[92425]: pgmap v2522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:08 smithi161 ceph-mon[91579]: pgmap v2523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:08 smithi188 ceph-mon[92425]: pgmap v2523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:10 smithi161 ceph-mon[91579]: pgmap v2524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:10 smithi188 ceph-mon[92425]: pgmap v2524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:13 smithi161 ceph-mon[91579]: pgmap v2525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:13 smithi188 ceph-mon[92425]: pgmap v2525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:15 smithi161 ceph-mon[91579]: pgmap v2526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:15 smithi188 ceph-mon[92425]: pgmap v2526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:17 smithi161 ceph-mon[91579]: pgmap v2527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:17 smithi188 ceph-mon[92425]: pgmap v2527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:03:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:03:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:03:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:03:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:03:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:03:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:03:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:03:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:19 smithi161 ceph-mon[91579]: pgmap v2528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:19 smithi188 ceph-mon[92425]: pgmap v2528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:20.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:21 smithi161 ceph-mon[91579]: pgmap v2529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:21 smithi188 ceph-mon[92425]: pgmap v2529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:23 smithi161 ceph-mon[91579]: pgmap v2530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:23 smithi188 ceph-mon[92425]: pgmap v2530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:25 smithi161 ceph-mon[91579]: pgmap v2531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:25 smithi188 ceph-mon[92425]: pgmap v2531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:27 smithi161 ceph-mon[91579]: pgmap v2532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:27 smithi188 ceph-mon[92425]: pgmap v2532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:28 smithi161 ceph-mon[91579]: pgmap v2533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:28 smithi188 ceph-mon[92425]: pgmap v2533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:30 smithi161 ceph-mon[91579]: pgmap v2534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:30 smithi188 ceph-mon[92425]: pgmap v2534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:33 smithi161 ceph-mon[91579]: pgmap v2535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:33 smithi188 ceph-mon[92425]: pgmap v2535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:35 smithi161 ceph-mon[91579]: pgmap v2536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:35 smithi188 ceph-mon[92425]: pgmap v2536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:37 smithi161 ceph-mon[91579]: pgmap v2537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:37 smithi188 ceph-mon[92425]: pgmap v2537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:38 smithi161 ceph-mon[91579]: pgmap v2538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:38 smithi188 ceph-mon[92425]: pgmap v2538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:40 smithi161 ceph-mon[91579]: pgmap v2539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:40 smithi188 ceph-mon[92425]: pgmap v2539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:43 smithi161 ceph-mon[91579]: pgmap v2540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:43 smithi188 ceph-mon[92425]: pgmap v2540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:45 smithi161 ceph-mon[91579]: pgmap v2541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:45 smithi188 ceph-mon[92425]: pgmap v2541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-mon[91579]: pgmap v2542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:47.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:03:47.281+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:03:47.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:03:47.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:03:47.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:03:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:03:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:03:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:03:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:03:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:03:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:03:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:03:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:03:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:03:47.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:03:47.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:03:47.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:03:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:03:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:03:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:03:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:03:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:03:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:03:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:03:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:03:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:03:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:03:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:03:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:03:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:03:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:03:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:03:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:03:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:03:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:03:47.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:03:47.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:03:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:03:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:03:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:03:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:03:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:03:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:03:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:03:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:03:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:03:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:03:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=119] Local address: 172.21.15.161, port 51984 2022-07-03T12:03:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=119] Peer address: 172.21.15.101, port 22 2022-07-03T12:03:47.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=364] Received exit status 0 2022-07-03T12:03:47.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=364] Received channel close 2022-07-03T12:03:47.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=364] Channel closed 2022-07-03T12:03:47.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=364] Received exit status 0 2022-07-03T12:03:47.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=364] Received channel close 2022-07-03T12:03:47.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=364] Channel closed 2022-07-03T12:03:47.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=365] Requesting new SSH session 2022-07-03T12:03:47.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=365] Requesting new SSH session 2022-07-03T12:03:47.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=365] Command: which python3 2022-07-03T12:03:47.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=365] Command: which python3 2022-07-03T12:03:47.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=365] Received exit status 0 2022-07-03T12:03:47.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=365] Received channel close 2022-07-03T12:03:47.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=365] Channel closed 2022-07-03T12:03:47.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=366] Requesting new SSH session 2022-07-03T12:03:47.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=365] Received exit status 0 2022-07-03T12:03:47.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=365] Received channel close 2022-07-03T12:03:47.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=365] Channel closed 2022-07-03T12:03:47.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=366] Command: sudo true 2022-07-03T12:03:47.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=366] Requesting new SSH session 2022-07-03T12:03:47.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=366] Command: sudo true 2022-07-03T12:03:47.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=366] Received exit status 0 2022-07-03T12:03:47.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=366] Received channel close 2022-07-03T12:03:47.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=366] Channel closed 2022-07-03T12:03:47.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=367] Requesting new SSH session 2022-07-03T12:03:47.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:47.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=366] Received exit status 0 2022-07-03T12:03:47.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=366] Received channel close 2022-07-03T12:03:47.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=366] Channel closed 2022-07-03T12:03:47.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=367] Requesting new SSH session 2022-07-03T12:03:47.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:47.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=367] Received exit status 0 2022-07-03T12:03:47.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=367] Received channel close 2022-07-03T12:03:47.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=367] Channel closed 2022-07-03T12:03:47.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=367] Received exit status 0 2022-07-03T12:03:47.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=367] Received channel close 2022-07-03T12:03:47.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=367] Channel closed 2022-07-03T12:03:47.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=119] Login timeout expired 2022-07-03T12:03:47.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=119] Aborting connection 2022-07-03T12:03:47.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:03:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:03:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:47 smithi188 ceph-mon[92425]: pgmap v2542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:03:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:03:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:03:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:03:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:03:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:03:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:03:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:03:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:03:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:03:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:03:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:03:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:03:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:03:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:03:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:03:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:03:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:03:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:03:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:03:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:03:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:03:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:03:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:03:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:03:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:03:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:03:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:03:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:03:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:03:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:03:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:03:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:03:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:03:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:03:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:03:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:03:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:03:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:03:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:03:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:03:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:03:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:03:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:03:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:03:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:03:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=119] Local address: 172.21.15.161, port 51984 2022-07-03T12:03:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=119] Peer address: 172.21.15.101, port 22 2022-07-03T12:03:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=364] Received exit status 0 2022-07-03T12:03:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=364] Received channel close 2022-07-03T12:03:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=364] Channel closed 2022-07-03T12:03:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=364] Received exit status 0 2022-07-03T12:03:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=364] Received channel close 2022-07-03T12:03:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=364] Channel closed 2022-07-03T12:03:48.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=365] Requesting new SSH session 2022-07-03T12:03:48.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=365] Requesting new SSH session 2022-07-03T12:03:48.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=365] Command: which python3 2022-07-03T12:03:48.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=365] Command: which python3 2022-07-03T12:03:48.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=365] Received exit status 0 2022-07-03T12:03:48.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=365] Received channel close 2022-07-03T12:03:48.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=365] Channel closed 2022-07-03T12:03:48.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=366] Requesting new SSH session 2022-07-03T12:03:48.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=365] Received exit status 0 2022-07-03T12:03:48.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=365] Received channel close 2022-07-03T12:03:48.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=365] Channel closed 2022-07-03T12:03:48.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=366] Command: sudo true 2022-07-03T12:03:48.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=366] Requesting new SSH session 2022-07-03T12:03:48.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=366] Command: sudo true 2022-07-03T12:03:48.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=366] Received exit status 0 2022-07-03T12:03:48.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=366] Received channel close 2022-07-03T12:03:48.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=366] Channel closed 2022-07-03T12:03:48.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=367] Requesting new SSH session 2022-07-03T12:03:48.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:48.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=366] Received exit status 0 2022-07-03T12:03:48.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=366] Received channel close 2022-07-03T12:03:48.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=366] Channel closed 2022-07-03T12:03:48.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=367] Requesting new SSH session 2022-07-03T12:03:48.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:48.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=367] Received exit status 0 2022-07-03T12:03:48.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=367] Received channel close 2022-07-03T12:03:48.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=367] Channel closed 2022-07-03T12:03:48.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=367] Received exit status 0 2022-07-03T12:03:48.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=367] Received channel close 2022-07-03T12:03:48.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=367] Channel closed 2022-07-03T12:03:48.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=119] Login timeout expired 2022-07-03T12:03:48.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=119] Aborting connection 2022-07-03T12:03:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:03:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:03:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:03:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:03:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:03:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:03:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:03:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:03:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:03:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:03:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:03:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:03:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:03:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:03:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:03:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:03:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:03:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:03:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:03:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:03:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:03:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:03:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:03:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:03:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:03:48.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:03:48.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:03:48.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:03:48.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:03:48.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:03:48.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:03:48.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:03:48.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:03:48.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:03:48.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:03:48.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:03:48.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:03:48.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:03:48.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:03:48.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:03:48.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:03:48.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:03:48.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:03:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:03:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:03:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:03:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=119] Local address: 172.21.15.161, port 51984 2022-07-03T12:03:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=119] Peer address: 172.21.15.101, port 22 2022-07-03T12:03:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=364] Received exit status 0 2022-07-03T12:03:48.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=364] Received channel close 2022-07-03T12:03:48.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=364] Channel closed 2022-07-03T12:03:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=364] Received exit status 0 2022-07-03T12:03:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=364] Received channel close 2022-07-03T12:03:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=364] Channel closed 2022-07-03T12:03:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=365] Requesting new SSH session 2022-07-03T12:03:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=365] Requesting new SSH session 2022-07-03T12:03:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=365] Command: which python3 2022-07-03T12:03:48.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=365] Command: which python3 2022-07-03T12:03:48.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=365] Received exit status 0 2022-07-03T12:03:48.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=365] Received channel close 2022-07-03T12:03:48.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=365] Channel closed 2022-07-03T12:03:48.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=366] Requesting new SSH session 2022-07-03T12:03:48.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=365] Received exit status 0 2022-07-03T12:03:48.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=365] Received channel close 2022-07-03T12:03:48.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=365] Channel closed 2022-07-03T12:03:48.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=366] Command: sudo true 2022-07-03T12:03:48.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=366] Requesting new SSH session 2022-07-03T12:03:48.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=366] Command: sudo true 2022-07-03T12:03:48.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=366] Received exit status 0 2022-07-03T12:03:48.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=366] Received channel close 2022-07-03T12:03:48.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=366] Channel closed 2022-07-03T12:03:48.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=367] Requesting new SSH session 2022-07-03T12:03:48.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:48.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=366] Received exit status 0 2022-07-03T12:03:48.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=366] Received channel close 2022-07-03T12:03:48.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=366] Channel closed 2022-07-03T12:03:48.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=367] Requesting new SSH session 2022-07-03T12:03:48.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=367] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:03:48.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=367] Received exit status 0 2022-07-03T12:03:48.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=367] Received channel close 2022-07-03T12:03:48.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=367] Channel closed 2022-07-03T12:03:48.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=367] Received exit status 0 2022-07-03T12:03:48.699 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=367] Received channel close 2022-07-03T12:03:48.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=367] Channel closed 2022-07-03T12:03:48.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=119] Login timeout expired 2022-07-03T12:03:48.700 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=119] Aborting connection 2022-07-03T12:03:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:49 smithi161 ceph-mon[91579]: pgmap v2543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:49 smithi188 ceph-mon[92425]: pgmap v2543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:51 smithi161 ceph-mon[91579]: pgmap v2544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:51 smithi188 ceph-mon[92425]: pgmap v2544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:53 smithi161 ceph-mon[91579]: pgmap v2545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:53 smithi188 ceph-mon[92425]: pgmap v2545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:55 smithi161 ceph-mon[91579]: pgmap v2546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:55 smithi188 ceph-mon[92425]: pgmap v2546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:03:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:57 smithi161 ceph-mon[91579]: pgmap v2547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:03:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:03:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:57 smithi188 ceph-mon[92425]: pgmap v2547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:03:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:03:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:03:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:03:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:03:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:58 smithi161 ceph-mon[91579]: pgmap v2548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:03:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:58 smithi188 ceph-mon[92425]: pgmap v2548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:03:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:00 smithi161 ceph-mon[91579]: pgmap v2549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:00 smithi188 ceph-mon[92425]: pgmap v2549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:03 smithi161 ceph-mon[91579]: pgmap v2550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:03 smithi188 ceph-mon[92425]: pgmap v2550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:05 smithi161 ceph-mon[91579]: pgmap v2551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:05 smithi188 ceph-mon[92425]: pgmap v2551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:07 smithi161 ceph-mon[91579]: pgmap v2552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:07 smithi188 ceph-mon[92425]: pgmap v2552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:08 smithi161 ceph-mon[91579]: pgmap v2553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:08 smithi188 ceph-mon[92425]: pgmap v2553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:10 smithi161 ceph-mon[91579]: pgmap v2554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:10 smithi188 ceph-mon[92425]: pgmap v2554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:13 smithi161 ceph-mon[91579]: pgmap v2555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:13 smithi188 ceph-mon[92425]: pgmap v2555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:15 smithi161 ceph-mon[91579]: pgmap v2556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:15 smithi188 ceph-mon[92425]: pgmap v2556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:17 smithi161 ceph-mon[91579]: pgmap v2557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:17 smithi188 ceph-mon[92425]: pgmap v2557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:18 smithi161 ceph-mon[91579]: pgmap v2558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:18 smithi188 ceph-mon[92425]: pgmap v2558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:20 smithi161 ceph-mon[91579]: pgmap v2559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:20 smithi188 ceph-mon[92425]: pgmap v2559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:23 smithi161 ceph-mon[91579]: pgmap v2560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:23 smithi188 ceph-mon[92425]: pgmap v2560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:25 smithi161 ceph-mon[91579]: pgmap v2561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:25 smithi188 ceph-mon[92425]: pgmap v2561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:27 smithi161 ceph-mon[91579]: pgmap v2562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:27 smithi188 ceph-mon[92425]: pgmap v2562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:28 smithi161 ceph-mon[91579]: pgmap v2563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:28 smithi188 ceph-mon[92425]: pgmap v2563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:30 smithi161 ceph-mon[91579]: pgmap v2564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:30 smithi188 ceph-mon[92425]: pgmap v2564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:33 smithi161 ceph-mon[91579]: pgmap v2565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:33 smithi188 ceph-mon[92425]: pgmap v2565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:35 smithi161 ceph-mon[91579]: pgmap v2566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:35 smithi188 ceph-mon[92425]: pgmap v2566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:37 smithi161 ceph-mon[91579]: pgmap v2567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:37 smithi188 ceph-mon[92425]: pgmap v2567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:38 smithi161 ceph-mon[91579]: pgmap v2568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:38 smithi188 ceph-mon[92425]: pgmap v2568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:40 smithi161 ceph-mon[91579]: pgmap v2569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:40 smithi188 ceph-mon[92425]: pgmap v2569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:43 smithi161 ceph-mon[91579]: pgmap v2570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:43 smithi188 ceph-mon[92425]: pgmap v2570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:45 smithi161 ceph-mon[91579]: pgmap v2571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:45 smithi188 ceph-mon[92425]: pgmap v2571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-mon[91579]: pgmap v2572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:47.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:04:47.319+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:04:47.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:04:47.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:04:47.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:04:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:04:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:04:47.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:04:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:04:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:04:47.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:04:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:04:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:04:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:04:47.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:04:47.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:04:47.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:04:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:04:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:04:47.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:04:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:04:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:04:47.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:04:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:04:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:04:47.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:04:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:04:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:04:47.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:04:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:04:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:04:47.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:04:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:04:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:04:47.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:04:47.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:04:47.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:04:47.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:04:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:04:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:04:47.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:04:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:04:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:04:47.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:04:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:04:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:04:47.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:04:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=121] Local address: 172.21.15.161, port 52024 2022-07-03T12:04:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=121] Peer address: 172.21.15.101, port 22 2022-07-03T12:04:47.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=121] Connection lost 2022-07-03T12:04:47.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=121] Aborting connection 2022-07-03T12:04:47.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:04:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:04:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:47 smithi188 ceph-mon[92425]: pgmap v2572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:04:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:04:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:04:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:04:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:04:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:04:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:04:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:04:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:04:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:04:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:04:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:04:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:04:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:04:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:04:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:04:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:04:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:04:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:04:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:04:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:04:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:04:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:04:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:04:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:04:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:04:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:04:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:04:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:04:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:04:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:04:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:04:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:04:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:04:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:04:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:04:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:04:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:04:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:04:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:04:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:04:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:04:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:04:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:04:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:04:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:04:48.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:04:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:04:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:04:48.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:04:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=121] Local address: 172.21.15.161, port 52024 2022-07-03T12:04:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=121] Peer address: 172.21.15.101, port 22 2022-07-03T12:04:48.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=121] Connection lost 2022-07-03T12:04:48.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=121] Aborting connection 2022-07-03T12:04:48.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:04:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:04:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:04:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:04:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:04:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:04:48.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:04:48.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:04:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:04:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:04:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:04:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:04:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:04:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:04:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:04:48.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:04:48.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:04:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:04:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:04:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:04:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:04:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:04:48.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:04:48.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:04:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:04:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:04:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:04:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:04:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:04:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:04:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:04:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:04:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:04:48.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:04:48.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:04:48.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:04:48.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:04:48.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:04:48.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:04:48.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:04:48.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:04:48.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:04:48.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:04:48.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:04:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:04:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:04:48.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:04:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:04:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:04:48.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:04:48.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=121] Local address: 172.21.15.161, port 52024 2022-07-03T12:04:48.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=121] Peer address: 172.21.15.101, port 22 2022-07-03T12:04:48.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=121] Connection lost 2022-07-03T12:04:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=121] Aborting connection 2022-07-03T12:04:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:48.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:49 smithi161 ceph-mon[91579]: pgmap v2573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:49 smithi188 ceph-mon[92425]: pgmap v2573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:50.525 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: pgmap v2574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.684 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: pgmap v2574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:51.685 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.685 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.686 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.686 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:51.687 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.687 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.688 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.688 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.689 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.690 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.690 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:51.691 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:52 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:04:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:52 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:04:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:52 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:04:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:52 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:04:53.561 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:53 smithi161 ceph-mon[91579]: pgmap v2575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:53.562 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:53 smithi188 ceph-mon[92425]: pgmap v2575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T12:04:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:04:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:04:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:54 smithi188 ceph-mon[92425]: pgmap v2576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T12:04:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:04:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:04:55.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:04:55.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:04:55.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:54 smithi161 ceph-mon[91579]: pgmap v2576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:04:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:56 smithi188 ceph-mon[92425]: pgmap v2577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:56 smithi161 ceph-mon[91579]: pgmap v2577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:04:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:04:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:04:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:04:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:04:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:04:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:04:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:04:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:04:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:58 smithi161 ceph-mon[91579]: pgmap v2578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:04:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:58 smithi188 ceph-mon[92425]: pgmap v2578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:04:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:00 smithi161 ceph-mon[91579]: pgmap v2579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:00 smithi188 ceph-mon[92425]: pgmap v2579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:03 smithi161 ceph-mon[91579]: pgmap v2580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:03 smithi188 ceph-mon[92425]: pgmap v2580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:05 smithi161 ceph-mon[91579]: pgmap v2581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:05 smithi188 ceph-mon[92425]: pgmap v2581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:07 smithi161 ceph-mon[91579]: pgmap v2582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:07 smithi188 ceph-mon[92425]: pgmap v2582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:08 smithi161 ceph-mon[91579]: pgmap v2583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:08 smithi188 ceph-mon[92425]: pgmap v2583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:10 smithi161 ceph-mon[91579]: pgmap v2584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:10 smithi188 ceph-mon[92425]: pgmap v2584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:13 smithi161 ceph-mon[91579]: pgmap v2585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:13 smithi188 ceph-mon[92425]: pgmap v2585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:15 smithi161 ceph-mon[91579]: pgmap v2586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:15 smithi188 ceph-mon[92425]: pgmap v2586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:15.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:17 smithi161 ceph-mon[91579]: pgmap v2587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:17 smithi188 ceph-mon[92425]: pgmap v2587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:18 smithi161 ceph-mon[91579]: pgmap v2588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:18 smithi188 ceph-mon[92425]: pgmap v2588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:20 smithi161 ceph-mon[91579]: pgmap v2589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:20 smithi188 ceph-mon[92425]: pgmap v2589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:23 smithi161 ceph-mon[91579]: pgmap v2590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:23 smithi188 ceph-mon[92425]: pgmap v2590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:25 smithi161 ceph-mon[91579]: pgmap v2591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:25 smithi188 ceph-mon[92425]: pgmap v2591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:27 smithi161 ceph-mon[91579]: pgmap v2592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:27 smithi188 ceph-mon[92425]: pgmap v2592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:28 smithi161 ceph-mon[91579]: pgmap v2593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:28 smithi188 ceph-mon[92425]: pgmap v2593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:30 smithi161 ceph-mon[91579]: pgmap v2594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:30 smithi188 ceph-mon[92425]: pgmap v2594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:33 smithi161 ceph-mon[91579]: pgmap v2595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:33 smithi188 ceph-mon[92425]: pgmap v2595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:35 smithi161 ceph-mon[91579]: pgmap v2596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:35 smithi188 ceph-mon[92425]: pgmap v2596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:37 smithi161 ceph-mon[91579]: pgmap v2597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:37 smithi188 ceph-mon[92425]: pgmap v2597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:38 smithi161 ceph-mon[91579]: pgmap v2598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:38 smithi188 ceph-mon[92425]: pgmap v2598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:40 smithi161 ceph-mon[91579]: pgmap v2599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:40 smithi188 ceph-mon[92425]: pgmap v2599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:43 smithi161 ceph-mon[91579]: pgmap v2600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:43 smithi188 ceph-mon[92425]: pgmap v2600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:45 smithi161 ceph-mon[91579]: pgmap v2601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:45 smithi188 ceph-mon[92425]: pgmap v2601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:47 smithi161 ceph-mon[91579]: pgmap v2602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:47 smithi188 ceph-mon[92425]: pgmap v2602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:48 smithi161 ceph-mon[91579]: pgmap v2603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:48 smithi188 ceph-mon[92425]: pgmap v2603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:51 smithi161 ceph-mon[91579]: pgmap v2604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:51 smithi188 ceph-mon[92425]: pgmap v2604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:53 smithi161 ceph-mon[91579]: pgmap v2605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:53 smithi188 ceph-mon[92425]: pgmap v2605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:05:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:05:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:05:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:05:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:05:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:05:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:55 smithi161 ceph-mon[91579]: pgmap v2606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:55 smithi188 ceph-mon[92425]: pgmap v2606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:05:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:57 smithi161 ceph-mon[91579]: pgmap v2607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:05:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:05:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:57 smithi188 ceph-mon[92425]: pgmap v2607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:05:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:05:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:58 smithi161 ceph-mon[91579]: pgmap v2608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:05:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:58 smithi188 ceph-mon[92425]: pgmap v2608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:05:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:01 smithi161 ceph-mon[91579]: pgmap v2609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:01 smithi188 ceph-mon[92425]: pgmap v2609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:03 smithi161 ceph-mon[91579]: pgmap v2610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:03 smithi188 ceph-mon[92425]: pgmap v2610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:05 smithi161 ceph-mon[91579]: pgmap v2611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:05 smithi188 ceph-mon[92425]: pgmap v2611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:07 smithi161 ceph-mon[91579]: pgmap v2612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:07 smithi188 ceph-mon[92425]: pgmap v2612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:08 smithi161 ceph-mon[91579]: pgmap v2613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:08 smithi188 ceph-mon[92425]: pgmap v2613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:09.320 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:11 smithi161 ceph-mon[91579]: pgmap v2614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:11 smithi188 ceph-mon[92425]: pgmap v2614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:13 smithi161 ceph-mon[91579]: pgmap v2615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:13 smithi188 ceph-mon[92425]: pgmap v2615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:15 smithi161 ceph-mon[91579]: pgmap v2616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:15 smithi188 ceph-mon[92425]: pgmap v2616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:17 smithi161 ceph-mon[91579]: pgmap v2617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:17 smithi188 ceph-mon[92425]: pgmap v2617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:18 smithi161 ceph-mon[91579]: pgmap v2618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:18 smithi188 ceph-mon[92425]: pgmap v2618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:21 smithi161 ceph-mon[91579]: pgmap v2619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:21 smithi188 ceph-mon[92425]: pgmap v2619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-mon[91579]: pgmap v2620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:23 smithi188 ceph-mon[92425]: pgmap v2620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:06:23.690+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:06:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:06:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:06:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:06:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:06:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:06:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:06:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:06:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:06:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:06:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:06:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:06:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:06:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:06:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:06:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:06:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:06:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:06:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:06:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:06:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:06:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:06:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:06:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:06:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:06:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:06:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:06:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:06:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:06:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:06:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:06:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:06:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:06:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:06:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:06:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:06:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:06:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:06:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:06:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:06:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:06:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:06:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:06:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:06:24.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:06:24.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:06:24.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=123] Local address: 172.21.15.161, port 52068 2022-07-03T12:06:24.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=123] Peer address: 172.21.15.101, port 22 2022-07-03T12:06:24.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=123] Connection lost 2022-07-03T12:06:24.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=123] Aborting connection 2022-07-03T12:06:24.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:06:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:06:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:06:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:06:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:06:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:06:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:06:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:06:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:06:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:06:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:06:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:06:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:06:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:06:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:06:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:06:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:06:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:06:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:06:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:06:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:06:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:06:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:06:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:06:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:06:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:06:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:06:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:06:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:06:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:06:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:06:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:06:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:06:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:06:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:06:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:06:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:06:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:06:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:06:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:06:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:06:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:06:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:06:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:06:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:06:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:06:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:06:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:06:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:06:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=123] Local address: 172.21.15.161, port 52068 2022-07-03T12:06:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=123] Peer address: 172.21.15.101, port 22 2022-07-03T12:06:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=123] Connection lost 2022-07-03T12:06:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=123] Aborting connection 2022-07-03T12:06:25.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:25 smithi161 ceph-mon[91579]: pgmap v2621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:25.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:06:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:06:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:06:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:06:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:06:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:06:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:06:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:06:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:06:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:06:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:06:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:06:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:06:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:06:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:06:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:06:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:06:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:06:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:06:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:06:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:06:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:06:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:06:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:06:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:06:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:06:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:06:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:06:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:06:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:06:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:06:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:06:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:06:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:06:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:06:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:06:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:06:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:06:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:06:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:06:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:06:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:06:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:06:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:06:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:06:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:06:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:06:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=123] Local address: 172.21.15.161, port 52068 2022-07-03T12:06:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=123] Peer address: 172.21.15.101, port 22 2022-07-03T12:06:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=123] Connection lost 2022-07-03T12:06:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=123] Aborting connection 2022-07-03T12:06:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:25 smithi188 ceph-mon[92425]: pgmap v2621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:27 smithi161 ceph-mon[91579]: pgmap v2622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:27 smithi188 ceph-mon[92425]: pgmap v2622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:28 smithi161 ceph-mon[91579]: pgmap v2623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:28 smithi188 ceph-mon[92425]: pgmap v2623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:31 smithi161 ceph-mon[91579]: pgmap v2624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:31 smithi188 ceph-mon[92425]: pgmap v2624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:33 smithi161 ceph-mon[91579]: pgmap v2625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:33 smithi188 ceph-mon[92425]: pgmap v2625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:35 smithi161 ceph-mon[91579]: pgmap v2626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:35 smithi188 ceph-mon[92425]: pgmap v2626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:37 smithi161 ceph-mon[91579]: pgmap v2627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:37 smithi188 ceph-mon[92425]: pgmap v2627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:38 smithi161 ceph-mon[91579]: pgmap v2628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:38 smithi188 ceph-mon[92425]: pgmap v2628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:41 smithi161 ceph-mon[91579]: pgmap v2629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:41 smithi188 ceph-mon[92425]: pgmap v2629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:43 smithi161 ceph-mon[91579]: pgmap v2630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:43 smithi188 ceph-mon[92425]: pgmap v2630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:45 smithi161 ceph-mon[91579]: pgmap v2631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:45 smithi188 ceph-mon[92425]: pgmap v2631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:47 smithi161 ceph-mon[91579]: pgmap v2632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:47 smithi188 ceph-mon[92425]: pgmap v2632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:48 smithi161 ceph-mon[91579]: pgmap v2633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:48 smithi188 ceph-mon[92425]: pgmap v2633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:51 smithi161 ceph-mon[91579]: pgmap v2634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:51 smithi188 ceph-mon[92425]: pgmap v2634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:53 smithi161 ceph-mon[91579]: pgmap v2635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:53 smithi188 ceph-mon[92425]: pgmap v2635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:55 smithi161 ceph-mon[91579]: pgmap v2636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:55 smithi188 ceph-mon[92425]: pgmap v2636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:06:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:57 smithi161 ceph-mon[91579]: pgmap v2637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:06:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:06:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:57 smithi188 ceph-mon[92425]: pgmap v2637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:06:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:06:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:06:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:06:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:06:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:58 smithi161 ceph-mon[91579]: pgmap v2638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:06:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:58 smithi188 ceph-mon[92425]: pgmap v2638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:06:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:01 smithi161 ceph-mon[91579]: pgmap v2639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:01 smithi188 ceph-mon[92425]: pgmap v2639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:03 smithi161 ceph-mon[91579]: pgmap v2640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:03 smithi188 ceph-mon[92425]: pgmap v2640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:05 smithi161 ceph-mon[91579]: pgmap v2641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:05 smithi188 ceph-mon[92425]: pgmap v2641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:07 smithi161 ceph-mon[91579]: pgmap v2642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:07 smithi188 ceph-mon[92425]: pgmap v2642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:08 smithi161 ceph-mon[91579]: pgmap v2643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:08 smithi188 ceph-mon[92425]: pgmap v2643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:11 smithi161 ceph-mon[91579]: pgmap v2644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:11 smithi188 ceph-mon[92425]: pgmap v2644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:13 smithi161 ceph-mon[91579]: pgmap v2645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:13 smithi188 ceph-mon[92425]: pgmap v2645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:15 smithi161 ceph-mon[91579]: pgmap v2646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:15 smithi188 ceph-mon[92425]: pgmap v2646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:17 smithi161 ceph-mon[91579]: pgmap v2647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:17 smithi188 ceph-mon[92425]: pgmap v2647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:18 smithi161 ceph-mon[91579]: pgmap v2648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:18 smithi188 ceph-mon[92425]: pgmap v2648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:21 smithi161 ceph-mon[91579]: pgmap v2649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:21 smithi188 ceph-mon[92425]: pgmap v2649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-mon[91579]: pgmap v2650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:23 smithi188 ceph-mon[92425]: pgmap v2650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:07:23.723+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:07:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:07:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:07:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:07:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:07:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:07:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:07:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:07:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:07:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:07:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:07:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:07:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:07:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:07:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:07:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:07:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:07:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:07:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:07:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:07:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:07:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:07:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:07:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:07:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:07:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:07:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:07:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:07:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:07:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:07:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:07:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:07:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:07:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:07:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:07:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:07:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:07:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:07:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:07:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:07:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:07:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:07:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:07:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:07:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:07:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:07:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=125] Local address: 172.21.15.161, port 52096 2022-07-03T12:07:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=125] Peer address: 172.21.15.101, port 22 2022-07-03T12:07:24.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=125] Connection lost 2022-07-03T12:07:24.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=125] Aborting connection 2022-07-03T12:07:24.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:07:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:07:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:07:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:07:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:07:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:07:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:07:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:07:25.573 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:07:25.574 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:07:25.574 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:07:25.575 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:07:25.575 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:07:25.575 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:07:25.576 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:07:25.576 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:07:25.576 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:07:25.577 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:07:25.577 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:07:25.577 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:07:25.578 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:07:25.578 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:07:25.578 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:07:25.579 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:07:25.579 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:07:25.579 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:07:25.580 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:07:25.580 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:07:25.580 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:07:25.581 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:07:25.581 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:07:25.581 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:07:25.582 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:07:25.582 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:07:25.583 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:07:25.583 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:07:25.583 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:07:25.584 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:07:25.584 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:07:25.585 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:07:25.585 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:07:25.585 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:07:25.586 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:07:25.586 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:07:25.586 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:07:25.587 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:07:25.587 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:07:25.587 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:07:25.588 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:07:25.588 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:07:25.588 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:07:25.589 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:07:25.589 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:07:25.589 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:07:25.590 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:07:25.590 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=125] Local address: 172.21.15.161, port 52096 2022-07-03T12:07:25.590 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=125] Peer address: 172.21.15.101, port 22 2022-07-03T12:07:25.591 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=125] Connection lost 2022-07-03T12:07:25.591 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=125] Aborting connection 2022-07-03T12:07:25.591 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:25 smithi188 ceph-mon[92425]: pgmap v2651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:25.592 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:07:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:07:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:07:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:07:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:07:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:07:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:07:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:07:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:07:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:07:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:07:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:07:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:07:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:07:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:07:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:07:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:07:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:07:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:07:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:07:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:07:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:07:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:07:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:07:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:07:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:07:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:07:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:07:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:07:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:07:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:07:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:07:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:07:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:07:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:07:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:07:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:07:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:07:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:07:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:07:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:07:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:07:25.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:07:25.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:07:25.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:07:25.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:07:25.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:07:25.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=125] Local address: 172.21.15.161, port 52096 2022-07-03T12:07:25.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=125] Peer address: 172.21.15.101, port 22 2022-07-03T12:07:25.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=125] Connection lost 2022-07-03T12:07:25.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=125] Aborting connection 2022-07-03T12:07:25.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:25 smithi161 ceph-mon[91579]: pgmap v2651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:25.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:27 smithi188 ceph-mon[92425]: pgmap v2652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:07:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:07:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:27 smithi161 ceph-mon[91579]: pgmap v2652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:07:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:07:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:28 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T12:07:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:28 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T12:07:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:29 smithi161 ceph-mon[91579]: pgmap v2653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:29 smithi188 ceph-mon[92425]: pgmap v2653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:31 smithi161 ceph-mon[91579]: pgmap v2654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:31 smithi188 ceph-mon[92425]: pgmap v2654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:33 smithi161 ceph-mon[91579]: pgmap v2655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:33 smithi188 ceph-mon[92425]: pgmap v2655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:35 smithi161 ceph-mon[91579]: pgmap v2656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:35 smithi188 ceph-mon[92425]: pgmap v2656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:37 smithi161 ceph-mon[91579]: pgmap v2657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:37 smithi188 ceph-mon[92425]: pgmap v2657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:38 smithi161 ceph-mon[91579]: pgmap v2658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:38 smithi188 ceph-mon[92425]: pgmap v2658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:41 smithi161 ceph-mon[91579]: pgmap v2659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:41 smithi188 ceph-mon[92425]: pgmap v2659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:43 smithi161 ceph-mon[91579]: pgmap v2660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:43 smithi188 ceph-mon[92425]: pgmap v2660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:45 smithi161 ceph-mon[91579]: pgmap v2661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:45 smithi188 ceph-mon[92425]: pgmap v2661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:47 smithi161 ceph-mon[91579]: pgmap v2662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:47 smithi188 ceph-mon[92425]: pgmap v2662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:48 smithi161 ceph-mon[91579]: pgmap v2663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:48 smithi188 ceph-mon[92425]: pgmap v2663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:50.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:51 smithi161 ceph-mon[91579]: pgmap v2664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:51 smithi188 ceph-mon[92425]: pgmap v2664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:53 smithi161 ceph-mon[91579]: pgmap v2665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:53 smithi188 ceph-mon[92425]: pgmap v2665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:55 smithi161 ceph-mon[91579]: pgmap v2666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:55 smithi188 ceph-mon[92425]: pgmap v2666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:07:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:57 smithi161 ceph-mon[91579]: pgmap v2667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:07:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:07:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:07:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:07:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:57 smithi188 ceph-mon[92425]: pgmap v2667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:07:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:07:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:07:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:07:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:07:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:58 smithi161 ceph-mon[91579]: pgmap v2668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:07:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:58 smithi188 ceph-mon[92425]: pgmap v2668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:07:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:01 smithi161 ceph-mon[91579]: pgmap v2669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:01 smithi188 ceph-mon[92425]: pgmap v2669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:03 smithi161 ceph-mon[91579]: pgmap v2670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:03 smithi188 ceph-mon[92425]: pgmap v2670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:05 smithi161 ceph-mon[91579]: pgmap v2671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:05 smithi188 ceph-mon[92425]: pgmap v2671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:07 smithi161 ceph-mon[91579]: pgmap v2672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:07 smithi188 ceph-mon[92425]: pgmap v2672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:08 smithi161 ceph-mon[91579]: pgmap v2673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:08 smithi188 ceph-mon[92425]: pgmap v2673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:11 smithi161 ceph-mon[91579]: pgmap v2674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:11 smithi188 ceph-mon[92425]: pgmap v2674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:13 smithi161 ceph-mon[91579]: pgmap v2675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:13 smithi188 ceph-mon[92425]: pgmap v2675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:15 smithi161 ceph-mon[91579]: pgmap v2676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:15 smithi188 ceph-mon[92425]: pgmap v2676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:17 smithi161 ceph-mon[91579]: pgmap v2677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:17 smithi188 ceph-mon[92425]: pgmap v2677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:18 smithi161 ceph-mon[91579]: pgmap v2678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:18 smithi188 ceph-mon[92425]: pgmap v2678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:21 smithi161 ceph-mon[91579]: pgmap v2679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:21 smithi188 ceph-mon[92425]: pgmap v2679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:23 smithi161 ceph-mon[91579]: pgmap v2680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:23 smithi188 ceph-mon[92425]: pgmap v2680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:25 smithi161 ceph-mon[91579]: pgmap v2681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:25 smithi188 ceph-mon[92425]: pgmap v2681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:27.597 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-mon[91579]: pgmap v2682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:27.598 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:08:27.599 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:08:27.600 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:08:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:27 smithi188 ceph-mon[92425]: pgmap v2682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:08:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:08:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:08:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:08:27.596+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:08:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:08:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:08:27.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:27.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:27.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:08:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:08:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:08:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:08:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T12:08:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:08:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:08:27.607+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:08:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:08:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:08:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:08:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:08:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:08:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:08:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:08:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:08:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:08:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:08:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:08:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:08:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:08:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:08:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:08:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:08:27.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:08:27.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:08:27.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:08:27.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:08:27.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:08:27.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:08:27.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:08:27.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:08:27.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:08:27.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:08:27.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:08:27.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:08:27.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:08:27.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:08:27.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:08:27.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:08:27.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:08:27.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:08:27.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:08:27.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:08:27.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:08:27.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:08:27.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:08:27.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=127] Local address: 172.21.15.161, port 52124 2022-07-03T12:08:27.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=127] Peer address: 172.21.15.101, port 22 2022-07-03T12:08:27.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=127] Connection lost 2022-07-03T12:08:27.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=127] Aborting connection 2022-07-03T12:08:27.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:08:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:08:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:28 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:08:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:08:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:08:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:08:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:08:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:08:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:08:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:08:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:08:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:08:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:08:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:08:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:08:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:08:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:08:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:08:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:08:29.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:08:29.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:08:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:08:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:08:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:08:29.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:08:29.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:08:29.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:08:29.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:08:29.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:08:29.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:08:29.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:08:29.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:08:29.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:08:29.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:08:29.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:08:29.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:08:29.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:08:29.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:08:29.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:08:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:08:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:08:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:08:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:08:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:08:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:08:29.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:08:29.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:08:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:08:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:08:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=127] Local address: 172.21.15.161, port 52124 2022-07-03T12:08:29.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=127] Peer address: 172.21.15.101, port 22 2022-07-03T12:08:29.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=127] Connection lost 2022-07-03T12:08:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=127] Aborting connection 2022-07-03T12:08:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:28 smithi161 ceph-mon[91579]: pgmap v2683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:28 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:08:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:08:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:08:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:08:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:08:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:08:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:08:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:08:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:08:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:08:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:08:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:08:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:08:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:08:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:08:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:08:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:08:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:08:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:08:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:08:29.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:08:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:08:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:08:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:08:29.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:08:29.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:08:29.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:08:29.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:08:29.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:08:29.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:08:29.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:08:29.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:08:29.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:08:29.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:08:29.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:08:29.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:08:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:08:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:08:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:08:29.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:08:29.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:08:29.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:08:29.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:08:29.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:08:29.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:08:29.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:08:29.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:08:29.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:08:29.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:08:29.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:08:29.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:08:29.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:08:29.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:08:29.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:08:29.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=127] Local address: 172.21.15.161, port 52124 2022-07-03T12:08:29.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=127] Peer address: 172.21.15.101, port 22 2022-07-03T12:08:29.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=127] Connection lost 2022-07-03T12:08:29.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=127] Aborting connection 2022-07-03T12:08:29.201 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:28 smithi188 ceph-mon[92425]: pgmap v2683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:29.201 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:31 smithi161 ceph-mon[91579]: pgmap v2684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:31 smithi188 ceph-mon[92425]: pgmap v2684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:33 smithi161 ceph-mon[91579]: pgmap v2685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:33 smithi188 ceph-mon[92425]: pgmap v2685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:35 smithi161 ceph-mon[91579]: pgmap v2686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:35 smithi188 ceph-mon[92425]: pgmap v2686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:37 smithi161 ceph-mon[91579]: pgmap v2687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:37 smithi188 ceph-mon[92425]: pgmap v2687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:38 smithi161 ceph-mon[91579]: pgmap v2688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:38 smithi188 ceph-mon[92425]: pgmap v2688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:41 smithi161 ceph-mon[91579]: pgmap v2689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:41 smithi188 ceph-mon[92425]: pgmap v2689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:43 smithi161 ceph-mon[91579]: pgmap v2690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:43 smithi188 ceph-mon[92425]: pgmap v2690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:45 smithi161 ceph-mon[91579]: pgmap v2691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:45 smithi188 ceph-mon[92425]: pgmap v2691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:47 smithi161 ceph-mon[91579]: pgmap v2692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:47 smithi188 ceph-mon[92425]: pgmap v2692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:48 smithi161 ceph-mon[91579]: pgmap v2693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:48 smithi188 ceph-mon[92425]: pgmap v2693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:51 smithi161 ceph-mon[91579]: pgmap v2694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:51 smithi188 ceph-mon[92425]: pgmap v2694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:53 smithi161 ceph-mon[91579]: pgmap v2695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:53 smithi188 ceph-mon[92425]: pgmap v2695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:55 smithi161 ceph-mon[91579]: pgmap v2696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:55 smithi188 ceph-mon[92425]: pgmap v2696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:08:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:57 smithi161 ceph-mon[91579]: pgmap v2697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:08:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:08:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:08:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:08:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:57 smithi188 ceph-mon[92425]: pgmap v2697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:08:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:08:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:08:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:08:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:08:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:58 smithi161 ceph-mon[91579]: pgmap v2698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:08:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:58 smithi188 ceph-mon[92425]: pgmap v2698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:08:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:01 smithi161 ceph-mon[91579]: pgmap v2699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:01 smithi188 ceph-mon[92425]: pgmap v2699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:03 smithi161 ceph-mon[91579]: pgmap v2700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:03 smithi188 ceph-mon[92425]: pgmap v2700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:05 smithi161 ceph-mon[91579]: pgmap v2701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:05 smithi188 ceph-mon[92425]: pgmap v2701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:07 smithi161 ceph-mon[91579]: pgmap v2702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:07 smithi188 ceph-mon[92425]: pgmap v2702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:08 smithi161 ceph-mon[91579]: pgmap v2703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:08 smithi188 ceph-mon[92425]: pgmap v2703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:11 smithi161 ceph-mon[91579]: pgmap v2704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:11 smithi188 ceph-mon[92425]: pgmap v2704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:13 smithi161 ceph-mon[91579]: pgmap v2705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:13 smithi188 ceph-mon[92425]: pgmap v2705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:15 smithi161 ceph-mon[91579]: pgmap v2706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:15 smithi188 ceph-mon[92425]: pgmap v2706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:17 smithi161 ceph-mon[91579]: pgmap v2707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:17 smithi188 ceph-mon[92425]: pgmap v2707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:18 smithi161 ceph-mon[91579]: pgmap v2708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:18 smithi188 ceph-mon[92425]: pgmap v2708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:21 smithi161 ceph-mon[91579]: pgmap v2709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:21 smithi188 ceph-mon[92425]: pgmap v2709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:23 smithi161 ceph-mon[91579]: pgmap v2710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:23 smithi188 ceph-mon[92425]: pgmap v2710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:25 smithi161 ceph-mon[91579]: pgmap v2711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:25 smithi188 ceph-mon[92425]: pgmap v2711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-mon[91579]: pgmap v2712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:27 smithi188 ceph-mon[92425]: pgmap v2712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:28.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:09:27.709+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:09:28.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:09:28.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:09:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:09:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:09:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:09:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:09:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:09:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:09:28.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:09:28.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:09:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:09:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:09:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:09:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:09:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:09:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:09:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:09:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:09:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:09:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:09:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:09:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:09:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:09:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:09:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:09:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:09:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:09:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:09:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:09:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:09:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:09:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:09:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:09:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:09:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:09:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:09:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:09:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:09:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:09:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:09:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:09:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:09:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:09:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:09:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:09:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=129] Local address: 172.21.15.161, port 52180 2022-07-03T12:09:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=129] Peer address: 172.21.15.101, port 22 2022-07-03T12:09:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=129] Connection lost 2022-07-03T12:09:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=129] Aborting connection 2022-07-03T12:09:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:09:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:09:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:09:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:09:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:09:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:09:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:09:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:09:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:09:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:09:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:09:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:09:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:09:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:09:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:09:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:09:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:09:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:09:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:09:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:09:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:09:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:09:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:09:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:09:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:09:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:09:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:09:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:09:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:09:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:09:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:09:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:09:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:09:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:09:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:09:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:09:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:09:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:09:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:09:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:09:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:09:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:09:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:09:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:09:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:09:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:09:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:09:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:09:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:09:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:09:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:09:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:09:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:09:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:09:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:09:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=129] Local address: 172.21.15.161, port 52180 2022-07-03T12:09:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=129] Peer address: 172.21.15.101, port 22 2022-07-03T12:09:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=129] Connection lost 2022-07-03T12:09:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=129] Aborting connection 2022-07-03T12:09:29.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:29 smithi161 ceph-mon[91579]: pgmap v2713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:29.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:09:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:09:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:09:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:09:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:09:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:09:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:09:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:09:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:09:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:09:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:09:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:09:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:09:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:09:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:09:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:09:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:09:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:09:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:09:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:09:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:09:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:09:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:09:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:09:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:09:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:09:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:09:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:09:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:09:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:09:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:09:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:09:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:09:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:09:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:09:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:09:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:09:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:09:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:09:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:09:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:09:29.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:09:29.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:09:29.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:09:29.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:09:29.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:09:29.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:09:29.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=129] Local address: 172.21.15.161, port 52180 2022-07-03T12:09:29.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=129] Peer address: 172.21.15.101, port 22 2022-07-03T12:09:29.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=129] Connection lost 2022-07-03T12:09:29.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=129] Aborting connection 2022-07-03T12:09:29.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:29 smithi188 ceph-mon[92425]: pgmap v2713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:29.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:31 smithi161 ceph-mon[91579]: pgmap v2714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:31 smithi188 ceph-mon[92425]: pgmap v2714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:33 smithi161 ceph-mon[91579]: pgmap v2715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:33 smithi188 ceph-mon[92425]: pgmap v2715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:35 smithi161 ceph-mon[91579]: pgmap v2716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:35 smithi188 ceph-mon[92425]: pgmap v2716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:37 smithi161 ceph-mon[91579]: pgmap v2717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:37 smithi188 ceph-mon[92425]: pgmap v2717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:38 smithi161 ceph-mon[91579]: pgmap v2718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:38 smithi188 ceph-mon[92425]: pgmap v2718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:41 smithi161 ceph-mon[91579]: pgmap v2719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:41 smithi188 ceph-mon[92425]: pgmap v2719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:43 smithi161 ceph-mon[91579]: pgmap v2720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:43 smithi188 ceph-mon[92425]: pgmap v2720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:45 smithi161 ceph-mon[91579]: pgmap v2721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:45 smithi188 ceph-mon[92425]: pgmap v2721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:47 smithi161 ceph-mon[91579]: pgmap v2722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:47 smithi188 ceph-mon[92425]: pgmap v2722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:48 smithi161 ceph-mon[91579]: pgmap v2723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:48 smithi188 ceph-mon[92425]: pgmap v2723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:51 smithi161 ceph-mon[91579]: pgmap v2724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:51 smithi188 ceph-mon[92425]: pgmap v2724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:53 smithi161 ceph-mon[91579]: pgmap v2725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:53 smithi188 ceph-mon[92425]: pgmap v2725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:55 smithi161 ceph-mon[91579]: pgmap v2726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:55 smithi188 ceph-mon[92425]: pgmap v2726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:09:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:57 smithi161 ceph-mon[91579]: pgmap v2727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:09:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:09:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:09:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:09:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:57 smithi188 ceph-mon[92425]: pgmap v2727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:09:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:09:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:09:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:09:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:09:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:58 smithi161 ceph-mon[91579]: pgmap v2728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:09:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:58 smithi188 ceph-mon[92425]: pgmap v2728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:09:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:10:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:10:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:01 smithi161 ceph-mon[91579]: pgmap v2729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:01 smithi188 ceph-mon[92425]: pgmap v2729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:03 smithi161 ceph-mon[91579]: pgmap v2730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:03 smithi188 ceph-mon[92425]: pgmap v2730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:05 smithi161 ceph-mon[91579]: pgmap v2731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:05 smithi188 ceph-mon[92425]: pgmap v2731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:07 smithi161 ceph-mon[91579]: pgmap v2732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:07 smithi188 ceph-mon[92425]: pgmap v2732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:08 smithi161 ceph-mon[91579]: pgmap v2733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:08 smithi188 ceph-mon[92425]: pgmap v2733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:11 smithi161 ceph-mon[91579]: pgmap v2734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:11 smithi188 ceph-mon[92425]: pgmap v2734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:13 smithi161 ceph-mon[91579]: pgmap v2735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:13 smithi188 ceph-mon[92425]: pgmap v2735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:15 smithi161 ceph-mon[91579]: pgmap v2736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:15 smithi188 ceph-mon[92425]: pgmap v2736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:17 smithi161 ceph-mon[91579]: pgmap v2737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:17 smithi188 ceph-mon[92425]: pgmap v2737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:18 smithi161 ceph-mon[91579]: pgmap v2738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:18 smithi188 ceph-mon[92425]: pgmap v2738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:21 smithi188 ceph-mon[92425]: pgmap v2739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:21 smithi161 ceph-mon[91579]: pgmap v2739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:23 smithi161 ceph-mon[91579]: pgmap v2740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:23 smithi188 ceph-mon[92425]: pgmap v2740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:25 smithi161 ceph-mon[91579]: pgmap v2741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:25 smithi188 ceph-mon[92425]: pgmap v2741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:27 smithi161 ceph-mon[91579]: pgmap v2742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:27 smithi188 ceph-mon[92425]: pgmap v2742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:28.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:10:28.206+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:10:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:10:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:10:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:10:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:10:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:10:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:10:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:10:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:10:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:10:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:10:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:10:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:10:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:10:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:10:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:10:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:10:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:10:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:10:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:10:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:10:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:10:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:10:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:10:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:10:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:10:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:10:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:10:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:10:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:10:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:10:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:10:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:10:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:10:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:10:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:10:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:10:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:10:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:10:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:10:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:10:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:10:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:10:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:10:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:10:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:10:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=131] Local address: 172.21.15.161, port 52208 2022-07-03T12:10:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=131] Peer address: 172.21.15.101, port 22 2022-07-03T12:10:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=131] Connection lost 2022-07-03T12:10:28.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=131] Aborting connection 2022-07-03T12:10:28.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:10:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:10:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:10:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:10:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:10:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:10:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:10:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:10:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:10:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:10:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:10:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:10:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:10:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:10:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:10:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:10:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:10:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:10:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:10:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:10:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:10:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:10:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:10:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:10:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:10:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:10:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:10:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:10:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:10:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:10:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:10:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:10:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:10:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:10:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:10:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:10:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:10:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:10:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:10:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:10:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:10:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:10:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:10:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:10:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:10:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:10:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:10:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:10:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:10:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:10:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:10:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=131] Local address: 172.21.15.161, port 52208 2022-07-03T12:10:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=131] Peer address: 172.21.15.101, port 22 2022-07-03T12:10:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=131] Connection lost 2022-07-03T12:10:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=131] Aborting connection 2022-07-03T12:10:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: pgmap v2743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:10:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:10:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:10:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:10:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:10:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:10:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:10:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:10:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:10:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:10:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:10:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:10:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:10:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:10:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:10:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:10:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:10:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:10:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:10:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:10:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:10:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:10:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:10:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:10:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:10:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:10:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:10:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:10:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:10:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:10:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:10:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:10:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:10:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:10:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:10:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:10:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:10:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:10:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:10:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:10:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:10:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:10:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:10:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:10:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:10:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:10:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:10:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:10:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:10:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:10:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=131] Local address: 172.21.15.161, port 52208 2022-07-03T12:10:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=131] Peer address: 172.21.15.101, port 22 2022-07-03T12:10:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=131] Connection lost 2022-07-03T12:10:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=131] Aborting connection 2022-07-03T12:10:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: pgmap v2743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:10:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:31 smithi161 ceph-mon[91579]: pgmap v2744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:31 smithi188 ceph-mon[92425]: pgmap v2744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:33 smithi161 ceph-mon[91579]: pgmap v2745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:33 smithi188 ceph-mon[92425]: pgmap v2745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:35 smithi161 ceph-mon[91579]: pgmap v2746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:35 smithi188 ceph-mon[92425]: pgmap v2746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:37 smithi161 ceph-mon[91579]: pgmap v2747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:37 smithi188 ceph-mon[92425]: pgmap v2747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:38 smithi161 ceph-mon[91579]: pgmap v2748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:38 smithi188 ceph-mon[92425]: pgmap v2748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:41 smithi188 ceph-mon[92425]: pgmap v2749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:41 smithi161 ceph-mon[91579]: pgmap v2749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:43 smithi161 ceph-mon[91579]: pgmap v2750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:43 smithi188 ceph-mon[92425]: pgmap v2750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:45 smithi161 ceph-mon[91579]: pgmap v2751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:45 smithi188 ceph-mon[92425]: pgmap v2751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:45.717 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:47 smithi161 ceph-mon[91579]: pgmap v2752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:47 smithi188 ceph-mon[92425]: pgmap v2752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:48 smithi161 ceph-mon[91579]: pgmap v2753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:49.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:48 smithi188 ceph-mon[92425]: pgmap v2753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:49.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:51 smithi188 ceph-mon[92425]: pgmap v2754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:51 smithi161 ceph-mon[91579]: pgmap v2754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:53 smithi161 ceph-mon[91579]: pgmap v2755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:53 smithi188 ceph-mon[92425]: pgmap v2755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:55 smithi161 ceph-mon[91579]: pgmap v2756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:55 smithi188 ceph-mon[92425]: pgmap v2756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:10:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:57 smithi161 ceph-mon[91579]: pgmap v2757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:10:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:10:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:10:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:10:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:57 smithi188 ceph-mon[92425]: pgmap v2757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:10:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:10:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:58 smithi161 ceph-mon[91579]: pgmap v2758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:10:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:58 smithi188 ceph-mon[92425]: pgmap v2758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:10:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:01 smithi188 ceph-mon[92425]: pgmap v2759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:01 smithi161 ceph-mon[91579]: pgmap v2759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:03 smithi161 ceph-mon[91579]: pgmap v2760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:03 smithi188 ceph-mon[92425]: pgmap v2760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:05 smithi161 ceph-mon[91579]: pgmap v2761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:05 smithi188 ceph-mon[92425]: pgmap v2761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:07 smithi188 ceph-mon[92425]: pgmap v2762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:07 smithi161 ceph-mon[91579]: pgmap v2762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:08 smithi161 ceph-mon[91579]: pgmap v2763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:08 smithi188 ceph-mon[92425]: pgmap v2763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:11 smithi188 ceph-mon[92425]: pgmap v2764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:11 smithi161 ceph-mon[91579]: pgmap v2764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:13 smithi161 ceph-mon[91579]: pgmap v2765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:13 smithi188 ceph-mon[92425]: pgmap v2765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:15 smithi188 ceph-mon[92425]: pgmap v2766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:15 smithi161 ceph-mon[91579]: pgmap v2766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:17 smithi188 ceph-mon[92425]: pgmap v2767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:17 smithi161 ceph-mon[91579]: pgmap v2767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:18 smithi161 ceph-mon[91579]: pgmap v2768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:18 smithi188 ceph-mon[92425]: pgmap v2768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:21 smithi188 ceph-mon[92425]: pgmap v2769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:21 smithi161 ceph-mon[91579]: pgmap v2769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:23 smithi161 ceph-mon[91579]: pgmap v2770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:23 smithi188 ceph-mon[92425]: pgmap v2770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:25 smithi188 ceph-mon[92425]: pgmap v2771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:25 smithi161 ceph-mon[91579]: pgmap v2771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:27 smithi188 ceph-mon[92425]: pgmap v2772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:27 smithi161 ceph-mon[91579]: pgmap v2772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:28 smithi161 ceph-mon[91579]: pgmap v2773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:29.136 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:28 smithi188 ceph-mon[92425]: pgmap v2773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:29.137 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:11:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:11:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:11:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:11:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:11:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:11:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:11:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:11:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:30 smithi161 ceph-mon[91579]: pgmap v2774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:30 smithi188 ceph-mon[92425]: pgmap v2774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:33 smithi188 ceph-mon[92425]: pgmap v2775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:33 smithi161 ceph-mon[91579]: pgmap v2775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:35 smithi188 ceph-mon[92425]: pgmap v2776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:35 smithi161 ceph-mon[91579]: pgmap v2776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:37 smithi188 ceph-mon[92425]: pgmap v2777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:37 smithi161 ceph-mon[91579]: pgmap v2777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:38 smithi161 ceph-mon[91579]: pgmap v2778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:38 smithi188 ceph-mon[92425]: pgmap v2778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:41 smithi161 ceph-mon[91579]: pgmap v2779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:41 smithi188 ceph-mon[92425]: pgmap v2779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:43 smithi188 ceph-mon[92425]: pgmap v2780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:43 smithi161 ceph-mon[91579]: pgmap v2780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:45 smithi188 ceph-mon[92425]: pgmap v2781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:45 smithi161 ceph-mon[91579]: pgmap v2781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:47 smithi188 ceph-mon[92425]: pgmap v2782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:47 smithi161 ceph-mon[91579]: pgmap v2782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:11:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:48 smithi161 ceph-mon[91579]: pgmap v2783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:48 smithi188 ceph-mon[92425]: pgmap v2783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:51 smithi161 ceph-mon[91579]: pgmap v2784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:51 smithi188 ceph-mon[92425]: pgmap v2784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:53 smithi188 ceph-mon[92425]: pgmap v2785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:53 smithi161 ceph-mon[91579]: pgmap v2785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:55 smithi188 ceph-mon[92425]: pgmap v2786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:55 smithi161 ceph-mon[91579]: pgmap v2786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:11:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:57 smithi188 ceph-mon[92425]: pgmap v2787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:11:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:11:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:11:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:57 smithi161 ceph-mon[91579]: pgmap v2787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:11:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:11:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:11:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:11:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:11:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:58 smithi161 ceph-mon[91579]: pgmap v2788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:58 smithi188 ceph-mon[92425]: pgmap v2788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:11:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:11:59.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:11:59.128+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:11:59.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:11:59.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:11:59.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:11:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:11:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:11:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:11:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:11:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:11:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:11:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:11:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:11:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:11:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:11:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:11:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:11:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:11:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:11:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:11:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:11:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:11:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:11:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:11:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:11:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:11:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:11:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:11:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:11:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:11:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:11:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:11:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:11:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:11:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:11:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:11:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:11:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:11:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:11:59.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:11:59.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:11:59.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:11:59.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:11:59.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:11:59.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:11:59.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:11:59.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:11:59.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=133] Local address: 172.21.15.161, port 52238 2022-07-03T12:11:59.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=133] Peer address: 172.21.15.101, port 22 2022-07-03T12:11:59.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=408] Received exit status 0 2022-07-03T12:11:59.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=408] Received channel close 2022-07-03T12:11:59.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=408] Channel closed 2022-07-03T12:11:59.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=409] Requesting new SSH session 2022-07-03T12:11:59.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=408] Received exit status 0 2022-07-03T12:11:59.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=408] Received channel close 2022-07-03T12:11:59.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=408] Channel closed 2022-07-03T12:11:59.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=409] Command: which python3 2022-07-03T12:11:59.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=409] Requesting new SSH session 2022-07-03T12:11:59.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=409] Command: which python3 2022-07-03T12:11:59.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=409] Received exit status 0 2022-07-03T12:11:59.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=409] Received channel close 2022-07-03T12:11:59.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=409] Channel closed 2022-07-03T12:11:59.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=410] Requesting new SSH session 2022-07-03T12:11:59.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=410] Command: sudo true 2022-07-03T12:11:59.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=409] Received exit status 0 2022-07-03T12:11:59.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=409] Received channel close 2022-07-03T12:11:59.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=409] Channel closed 2022-07-03T12:11:59.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=410] Requesting new SSH session 2022-07-03T12:11:59.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=410] Command: sudo true 2022-07-03T12:11:59.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=410] Received exit status 0 2022-07-03T12:11:59.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=410] Received channel close 2022-07-03T12:11:59.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=410] Channel closed 2022-07-03T12:11:59.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=411] Requesting new SSH session 2022-07-03T12:11:59.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:11:59.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=410] Received exit status 0 2022-07-03T12:11:59.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=410] Received channel close 2022-07-03T12:11:59.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=410] Channel closed 2022-07-03T12:11:59.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=411] Requesting new SSH session 2022-07-03T12:11:59.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:11:59.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=411] Received exit status 0 2022-07-03T12:11:59.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=411] Received channel close 2022-07-03T12:11:59.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=411] Channel closed 2022-07-03T12:11:59.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=411] Received exit status 0 2022-07-03T12:11:59.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=411] Received channel close 2022-07-03T12:11:59.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=411] Channel closed 2022-07-03T12:11:59.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=133] Login timeout expired 2022-07-03T12:11:59.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=133] Aborting connection 2022-07-03T12:11:59.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:12:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:11:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:12:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:12:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:12:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:12:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:12:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:12:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:12:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:12:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:12:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:12:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:12:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:12:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:12:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:12:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:12:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:12:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:12:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:12:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:12:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:12:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:12:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:12:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:12:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:12:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:12:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:12:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:12:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:12:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:12:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:12:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:12:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:12:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:12:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:12:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:12:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:12:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:12:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:12:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:12:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:12:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:12:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:12:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:12:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:12:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:12:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:12:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=133] Local address: 172.21.15.161, port 52238 2022-07-03T12:12:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=133] Peer address: 172.21.15.101, port 22 2022-07-03T12:12:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=408] Received exit status 0 2022-07-03T12:12:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=408] Received channel close 2022-07-03T12:12:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=408] Channel closed 2022-07-03T12:12:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=409] Requesting new SSH session 2022-07-03T12:12:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=408] Received exit status 0 2022-07-03T12:12:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=408] Received channel close 2022-07-03T12:12:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=408] Channel closed 2022-07-03T12:12:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=409] Command: which python3 2022-07-03T12:12:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=409] Requesting new SSH session 2022-07-03T12:12:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=409] Command: which python3 2022-07-03T12:12:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=409] Received exit status 0 2022-07-03T12:12:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=409] Received channel close 2022-07-03T12:12:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=409] Channel closed 2022-07-03T12:12:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=410] Requesting new SSH session 2022-07-03T12:12:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=410] Command: sudo true 2022-07-03T12:12:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=409] Received exit status 0 2022-07-03T12:12:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=409] Received channel close 2022-07-03T12:12:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=409] Channel closed 2022-07-03T12:12:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=410] Requesting new SSH session 2022-07-03T12:12:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=410] Command: sudo true 2022-07-03T12:12:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=410] Received exit status 0 2022-07-03T12:12:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=410] Received channel close 2022-07-03T12:12:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=410] Channel closed 2022-07-03T12:12:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=411] Requesting new SSH session 2022-07-03T12:12:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:12:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=410] Received exit status 0 2022-07-03T12:12:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=410] Received channel close 2022-07-03T12:12:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=410] Channel closed 2022-07-03T12:12:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=411] Requesting new SSH session 2022-07-03T12:12:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:12:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=411] Received exit status 0 2022-07-03T12:12:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=411] Received channel close 2022-07-03T12:12:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=411] Channel closed 2022-07-03T12:12:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=411] Received exit status 0 2022-07-03T12:12:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=411] Received channel close 2022-07-03T12:12:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=411] Channel closed 2022-07-03T12:12:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=133] Login timeout expired 2022-07-03T12:12:00.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=133] Aborting connection 2022-07-03T12:12:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:11:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:12:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:12:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:12:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:12:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:12:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:12:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:12:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:12:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:12:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:12:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:12:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:12:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:12:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:12:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:12:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:12:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:12:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:12:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:12:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:12:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:12:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:12:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:12:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:12:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:12:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:12:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:12:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:12:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:12:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:12:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:12:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:12:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:12:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:12:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:12:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:12:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:12:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:12:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:12:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:12:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:12:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:12:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:12:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:12:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:12:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:12:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=133] Local address: 172.21.15.161, port 52238 2022-07-03T12:12:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=133] Peer address: 172.21.15.101, port 22 2022-07-03T12:12:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=408] Received exit status 0 2022-07-03T12:12:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=408] Received channel close 2022-07-03T12:12:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=408] Channel closed 2022-07-03T12:12:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=409] Requesting new SSH session 2022-07-03T12:12:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=408] Received exit status 0 2022-07-03T12:12:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=408] Received channel close 2022-07-03T12:12:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=408] Channel closed 2022-07-03T12:12:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=409] Command: which python3 2022-07-03T12:12:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=409] Requesting new SSH session 2022-07-03T12:12:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=409] Command: which python3 2022-07-03T12:12:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=409] Received exit status 0 2022-07-03T12:12:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=409] Received channel close 2022-07-03T12:12:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=409] Channel closed 2022-07-03T12:12:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=410] Requesting new SSH session 2022-07-03T12:12:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=410] Command: sudo true 2022-07-03T12:12:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=409] Received exit status 0 2022-07-03T12:12:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=409] Received channel close 2022-07-03T12:12:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=409] Channel closed 2022-07-03T12:12:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=410] Requesting new SSH session 2022-07-03T12:12:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=410] Command: sudo true 2022-07-03T12:12:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=410] Received exit status 0 2022-07-03T12:12:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=410] Received channel close 2022-07-03T12:12:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=410] Channel closed 2022-07-03T12:12:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=411] Requesting new SSH session 2022-07-03T12:12:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:12:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=410] Received exit status 0 2022-07-03T12:12:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=410] Received channel close 2022-07-03T12:12:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=410] Channel closed 2022-07-03T12:12:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=411] Requesting new SSH session 2022-07-03T12:12:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=411] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:12:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=411] Received exit status 0 2022-07-03T12:12:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=411] Received channel close 2022-07-03T12:12:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=411] Channel closed 2022-07-03T12:12:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=411] Received exit status 0 2022-07-03T12:12:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=411] Received channel close 2022-07-03T12:12:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=411] Channel closed 2022-07-03T12:12:00.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=133] Login timeout expired 2022-07-03T12:12:00.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=133] Aborting connection 2022-07-03T12:12:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:00 smithi161 ceph-mon[91579]: pgmap v2789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:00 smithi188 ceph-mon[92425]: pgmap v2789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:03 smithi188 ceph-mon[92425]: pgmap v2790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:03 smithi161 ceph-mon[91579]: pgmap v2790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:05 smithi188 ceph-mon[92425]: pgmap v2791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:05 smithi161 ceph-mon[91579]: pgmap v2791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:07 smithi188 ceph-mon[92425]: pgmap v2792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:07 smithi161 ceph-mon[91579]: pgmap v2792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:08 smithi161 ceph-mon[91579]: pgmap v2793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:08 smithi188 ceph-mon[92425]: pgmap v2793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:11 smithi161 ceph-mon[91579]: pgmap v2794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:11 smithi188 ceph-mon[92425]: pgmap v2794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:13 smithi188 ceph-mon[92425]: pgmap v2795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:13 smithi161 ceph-mon[91579]: pgmap v2795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:15 smithi188 ceph-mon[92425]: pgmap v2796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:15 smithi161 ceph-mon[91579]: pgmap v2796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:17 smithi188 ceph-mon[92425]: pgmap v2797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:17 smithi161 ceph-mon[91579]: pgmap v2797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:18 smithi161 ceph-mon[91579]: pgmap v2798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:18 smithi188 ceph-mon[92425]: pgmap v2798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:21 smithi161 ceph-mon[91579]: pgmap v2799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:21 smithi188 ceph-mon[92425]: pgmap v2799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:23 smithi188 ceph-mon[92425]: pgmap v2800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:23 smithi161 ceph-mon[91579]: pgmap v2800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:25 smithi188 ceph-mon[92425]: pgmap v2801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:25 smithi161 ceph-mon[91579]: pgmap v2801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:27 smithi188 ceph-mon[92425]: pgmap v2802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:27 smithi161 ceph-mon[91579]: pgmap v2802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:28 smithi161 ceph-mon[91579]: pgmap v2803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:28 smithi188 ceph-mon[92425]: pgmap v2803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:31 smithi161 ceph-mon[91579]: pgmap v2804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:31 smithi188 ceph-mon[92425]: pgmap v2804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:33 smithi188 ceph-mon[92425]: pgmap v2805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:33 smithi161 ceph-mon[91579]: pgmap v2805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:35 smithi188 ceph-mon[92425]: pgmap v2806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:35 smithi161 ceph-mon[91579]: pgmap v2806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:37 smithi188 ceph-mon[92425]: pgmap v2807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:37 smithi161 ceph-mon[91579]: pgmap v2807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:38 smithi161 ceph-mon[91579]: pgmap v2808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:38 smithi188 ceph-mon[92425]: pgmap v2808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:41 smithi161 ceph-mon[91579]: pgmap v2809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:41 smithi188 ceph-mon[92425]: pgmap v2809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:43 smithi188 ceph-mon[92425]: pgmap v2810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:43 smithi161 ceph-mon[91579]: pgmap v2810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:45 smithi188 ceph-mon[92425]: pgmap v2811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:45 smithi161 ceph-mon[91579]: pgmap v2811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:47 smithi188 ceph-mon[92425]: pgmap v2812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:47 smithi161 ceph-mon[91579]: pgmap v2812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:48 smithi161 ceph-mon[91579]: pgmap v2813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:48 smithi188 ceph-mon[92425]: pgmap v2813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:51 smithi161 ceph-mon[91579]: pgmap v2814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:51 smithi188 ceph-mon[92425]: pgmap v2814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:53 smithi188 ceph-mon[92425]: pgmap v2815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:53 smithi161 ceph-mon[91579]: pgmap v2815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:55 smithi188 ceph-mon[92425]: pgmap v2816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:55 smithi161 ceph-mon[91579]: pgmap v2816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:12:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:57 smithi161 ceph-mon[91579]: pgmap v2817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:12:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:12:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:12:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:12:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:57 smithi188 ceph-mon[92425]: pgmap v2817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:12:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:12:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:12:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:12:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:12:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:58 smithi161 ceph-mon[91579]: pgmap v2818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:12:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:58 smithi188 ceph-mon[92425]: pgmap v2818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:12:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:13:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:13:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:13:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:13:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:13:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:13:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:00 smithi161 ceph-mon[91579]: pgmap v2819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:00 smithi188 ceph-mon[92425]: pgmap v2819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:03 smithi188 ceph-mon[92425]: pgmap v2820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:03 smithi161 ceph-mon[91579]: pgmap v2820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:05 smithi161 ceph-mon[91579]: pgmap v2821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:05 smithi188 ceph-mon[92425]: pgmap v2821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:07 smithi161 ceph-mon[91579]: pgmap v2822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:07 smithi188 ceph-mon[92425]: pgmap v2822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:08 smithi161 ceph-mon[91579]: pgmap v2823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:08 smithi188 ceph-mon[92425]: pgmap v2823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:11 smithi161 ceph-mon[91579]: pgmap v2824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:11 smithi188 ceph-mon[92425]: pgmap v2824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:13 smithi188 ceph-mon[92425]: pgmap v2825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:13 smithi161 ceph-mon[91579]: pgmap v2825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:15 smithi161 ceph-mon[91579]: pgmap v2826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:15 smithi188 ceph-mon[92425]: pgmap v2826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:17 smithi161 ceph-mon[91579]: pgmap v2827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:17 smithi188 ceph-mon[92425]: pgmap v2827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:18 smithi161 ceph-mon[91579]: pgmap v2828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:18 smithi188 ceph-mon[92425]: pgmap v2828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:21 smithi161 ceph-mon[91579]: pgmap v2829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:21 smithi188 ceph-mon[92425]: pgmap v2829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:23 smithi161 ceph-mon[91579]: pgmap v2830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:23 smithi188 ceph-mon[92425]: pgmap v2830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:25 smithi161 ceph-mon[91579]: pgmap v2831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:25 smithi188 ceph-mon[92425]: pgmap v2831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:27 smithi161 ceph-mon[91579]: pgmap v2832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:27 smithi188 ceph-mon[92425]: pgmap v2832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:28 smithi161 ceph-mon[91579]: pgmap v2833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:28 smithi188 ceph-mon[92425]: pgmap v2833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:29.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:13:29.164+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:13:29.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:13:29.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:13:29.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:13:29.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:13:29.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:13:29.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:13:29.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:13:29.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:13:29.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:13:29.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:13:29.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:13:29.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:13:29.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:13:29.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:13:29.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:13:29.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:13:29.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:13:29.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:13:29.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:13:29.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:13:29.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:13:29.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:13:29.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:13:29.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:13:29.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:13:29.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:13:29.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:13:29.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:13:29.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:13:29.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:13:29.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:13:29.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:13:29.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:13:29.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:13:29.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:13:29.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:13:29.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:13:29.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:13:29.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:13:29.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:13:29.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:13:29.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:13:29.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:13:29.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:13:29.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:13:29.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=135] Local address: 172.21.15.161, port 52290 2022-07-03T12:13:29.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=135] Peer address: 172.21.15.101, port 22 2022-07-03T12:13:29.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=135] Connection lost 2022-07-03T12:13:29.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=135] Aborting connection 2022-07-03T12:13:29.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:13:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:13:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:13:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:13:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:13:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:13:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:13:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:13:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:13:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:13:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:13:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:13:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:13:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:13:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:13:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:13:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:13:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:13:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:13:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:13:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:13:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:13:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:13:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:13:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:13:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:13:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:13:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:13:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:13:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:13:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:13:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:13:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:13:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:13:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:13:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:13:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:13:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:13:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:13:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:13:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:13:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:13:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:13:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:13:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:13:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:13:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:13:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:13:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=135] Local address: 172.21.15.161, port 52290 2022-07-03T12:13:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=135] Peer address: 172.21.15.101, port 22 2022-07-03T12:13:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=135] Connection lost 2022-07-03T12:13:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=135] Aborting connection 2022-07-03T12:13:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:13:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:13:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:13:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:13:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:13:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:13:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:13:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:13:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:13:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:13:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:13:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:13:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:13:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:13:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:13:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:13:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:13:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:13:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:13:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:13:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:13:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:13:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:13:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:13:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:13:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:13:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:13:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:13:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:13:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:13:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:13:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:13:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:13:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:13:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:13:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:13:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:13:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:13:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:13:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:13:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:13:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:13:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:13:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:13:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:13:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:13:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:13:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=135] Local address: 172.21.15.161, port 52290 2022-07-03T12:13:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=135] Peer address: 172.21.15.101, port 22 2022-07-03T12:13:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=135] Connection lost 2022-07-03T12:13:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=135] Aborting connection 2022-07-03T12:13:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:30 smithi161 ceph-mon[91579]: pgmap v2834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:30 smithi188 ceph-mon[92425]: pgmap v2834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:33 smithi161 ceph-mon[91579]: pgmap v2835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:33 smithi188 ceph-mon[92425]: pgmap v2835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:35 smithi161 ceph-mon[91579]: pgmap v2836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:35 smithi188 ceph-mon[92425]: pgmap v2836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:37 smithi161 ceph-mon[91579]: pgmap v2837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:37 smithi188 ceph-mon[92425]: pgmap v2837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:38 smithi161 ceph-mon[91579]: pgmap v2838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:38 smithi188 ceph-mon[92425]: pgmap v2838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:41 smithi161 ceph-mon[91579]: pgmap v2839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:41 smithi188 ceph-mon[92425]: pgmap v2839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:43 smithi161 ceph-mon[91579]: pgmap v2840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:43 smithi188 ceph-mon[92425]: pgmap v2840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:45 smithi161 ceph-mon[91579]: pgmap v2841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:45 smithi188 ceph-mon[92425]: pgmap v2841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:47 smithi161 ceph-mon[91579]: pgmap v2842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:47 smithi188 ceph-mon[92425]: pgmap v2842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:48 smithi161 ceph-mon[91579]: pgmap v2843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:48 smithi188 ceph-mon[92425]: pgmap v2843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:51 smithi161 ceph-mon[91579]: pgmap v2844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:51 smithi188 ceph-mon[92425]: pgmap v2844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:53 smithi161 ceph-mon[91579]: pgmap v2845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:53 smithi188 ceph-mon[92425]: pgmap v2845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:55 smithi161 ceph-mon[91579]: pgmap v2846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:55 smithi188 ceph-mon[92425]: pgmap v2846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:13:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:57 smithi161 ceph-mon[91579]: pgmap v2847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:13:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:13:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:13:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:13:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:57 smithi188 ceph-mon[92425]: pgmap v2847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:13:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:13:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:13:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:13:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:13:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:58 smithi161 ceph-mon[91579]: pgmap v2848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:13:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:58 smithi188 ceph-mon[92425]: pgmap v2848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:13:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:01 smithi161 ceph-mon[91579]: pgmap v2849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:01 smithi188 ceph-mon[92425]: pgmap v2849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:03 smithi161 ceph-mon[91579]: pgmap v2850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:03 smithi188 ceph-mon[92425]: pgmap v2850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:05 smithi161 ceph-mon[91579]: pgmap v2851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:05 smithi188 ceph-mon[92425]: pgmap v2851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:07 smithi161 ceph-mon[91579]: pgmap v2852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:07 smithi188 ceph-mon[92425]: pgmap v2852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:08 smithi161 ceph-mon[91579]: pgmap v2853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:08 smithi188 ceph-mon[92425]: pgmap v2853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:11 smithi161 ceph-mon[91579]: pgmap v2854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:11 smithi188 ceph-mon[92425]: pgmap v2854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:13 smithi161 ceph-mon[91579]: pgmap v2855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:13 smithi188 ceph-mon[92425]: pgmap v2855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:15 smithi161 ceph-mon[91579]: pgmap v2856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:15 smithi188 ceph-mon[92425]: pgmap v2856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:17 smithi161 ceph-mon[91579]: pgmap v2857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:17 smithi188 ceph-mon[92425]: pgmap v2857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:18 smithi161 ceph-mon[91579]: pgmap v2858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:18 smithi188 ceph-mon[92425]: pgmap v2858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:21 smithi161 ceph-mon[91579]: pgmap v2859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:21 smithi188 ceph-mon[92425]: pgmap v2859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:23 smithi161 ceph-mon[91579]: pgmap v2860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:23 smithi188 ceph-mon[92425]: pgmap v2860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:25 smithi161 ceph-mon[91579]: pgmap v2861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:25 smithi188 ceph-mon[92425]: pgmap v2861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:27 smithi161 ceph-mon[91579]: pgmap v2862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:27 smithi188 ceph-mon[92425]: pgmap v2862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:28 smithi161 ceph-mon[91579]: pgmap v2863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:28 smithi188 ceph-mon[92425]: pgmap v2863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:14:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:14:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:14:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:14:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:14:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:14:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:30 smithi161 ceph-mon[91579]: pgmap v2864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:30 smithi188 ceph-mon[92425]: pgmap v2864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:33 smithi161 ceph-mon[91579]: pgmap v2865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:33 smithi188 ceph-mon[92425]: pgmap v2865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:35 smithi161 ceph-mon[91579]: pgmap v2866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:35 smithi188 ceph-mon[92425]: pgmap v2866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:37 smithi161 ceph-mon[91579]: pgmap v2867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:37 smithi188 ceph-mon[92425]: pgmap v2867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:38 smithi161 ceph-mon[91579]: pgmap v2868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:38 smithi188 ceph-mon[92425]: pgmap v2868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:41 smithi161 ceph-mon[91579]: pgmap v2869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:41 smithi188 ceph-mon[92425]: pgmap v2869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:43 smithi161 ceph-mon[91579]: pgmap v2870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:43 smithi188 ceph-mon[92425]: pgmap v2870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:45 smithi161 ceph-mon[91579]: pgmap v2871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:45 smithi188 ceph-mon[92425]: pgmap v2871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:47 smithi161 ceph-mon[91579]: pgmap v2872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:47 smithi188 ceph-mon[92425]: pgmap v2872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:48 smithi161 ceph-mon[91579]: pgmap v2873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:48 smithi188 ceph-mon[92425]: pgmap v2873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:51 smithi161 ceph-mon[91579]: pgmap v2874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:51 smithi188 ceph-mon[92425]: pgmap v2874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:53 smithi161 ceph-mon[91579]: pgmap v2875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:53 smithi188 ceph-mon[92425]: pgmap v2875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:55 smithi161 ceph-mon[91579]: pgmap v2876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:55 smithi188 ceph-mon[92425]: pgmap v2876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:14:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:57 smithi161 ceph-mon[91579]: pgmap v2877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:14:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:14:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:14:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:14:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:57 smithi188 ceph-mon[92425]: pgmap v2877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:14:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:14:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:14:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:14:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:14:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:58 smithi161 ceph-mon[91579]: pgmap v2878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:58 smithi188 ceph-mon[92425]: pgmap v2878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:14:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:14:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:14:59.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:14:59.614+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:14:59.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:14:59.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:14:59.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:14:59.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:14:59.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:14:59.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:14:59.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:14:59.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:14:59.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:14:59.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:14:59.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:14:59.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:14:59.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T12:14:59.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:14:59.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:14:59.615+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:14:59.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:14:59.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:14:59.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:14:59.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:14:59.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:14:59.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:14:59.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:14:59.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:14:59.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:14:59.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:14:59.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:14:59.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:14:59.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T12:14:59.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:14:59.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:14:59.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:14:59.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:14:59.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:14:59.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:14:59.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:14:59.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:14:59.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:14:59.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:14:59.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:14:59.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:14:59.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:14:59.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:14:59.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:14:59.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:14:59.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:14:59.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:14:59.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:14:59.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:14:59.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:14:59.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:14:59.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:14:59.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:14:59.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:14:59.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:14:59.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:14:59.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:14:59.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:14:59.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:14:59.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:14:59.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:14:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:15:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:14:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:15:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:00 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:15:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:15:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:15:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:15:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:15:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:15:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:15:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:15:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:15:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:15:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:15:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:15:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:15:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:15:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:15:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:15:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:15:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:15:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:15:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:15:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:15:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:15:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:15:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:15:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:15:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:15:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:15:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:15:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:15:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:15:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:15:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:15:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:15:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:15:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:15:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:15:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:15:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:15:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:15:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:15:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:15:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:15:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:15:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:15:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:15:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:15:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:15:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:15:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:15:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:15:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:15:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:15:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:15:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:15:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:15:01.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:15:01.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:15:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:15:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:00 smithi161 ceph-mon[91579]: pgmap v2879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:00 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:15:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:15:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:15:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:15:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:15:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:15:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:15:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:15:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:15:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:15:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:15:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:15:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:15:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:15:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:15:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:15:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:15:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:15:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:15:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:15:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:15:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:15:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:15:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:15:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:15:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:15:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:15:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:15:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:15:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:15:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:15:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:15:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:15:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:15:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:15:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:15:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:15:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:15:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:15:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:15:01.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:15:01.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:15:01.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:15:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:15:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:15:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:15:01.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:15:01.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:15:01.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:15:01.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:15:01.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:15:01.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:15:01.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:15:01.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:15:01.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:15:01.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:15:01.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:15:01.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:15:01.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:15:01.191 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:01.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:00 smithi188 ceph-mon[92425]: pgmap v2879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:03 smithi161 ceph-mon[91579]: pgmap v2880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:03 smithi188 ceph-mon[92425]: pgmap v2880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:05 smithi161 ceph-mon[91579]: pgmap v2881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:05 smithi188 ceph-mon[92425]: pgmap v2881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:07 smithi161 ceph-mon[91579]: pgmap v2882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:07 smithi188 ceph-mon[92425]: pgmap v2882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:08 smithi161 ceph-mon[91579]: pgmap v2883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:08 smithi188 ceph-mon[92425]: pgmap v2883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:11 smithi161 ceph-mon[91579]: pgmap v2884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:11 smithi188 ceph-mon[92425]: pgmap v2884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:13 smithi161 ceph-mon[91579]: pgmap v2885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:13 smithi188 ceph-mon[92425]: pgmap v2885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:15 smithi161 ceph-mon[91579]: pgmap v2886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:15 smithi188 ceph-mon[92425]: pgmap v2886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:17 smithi161 ceph-mon[91579]: pgmap v2887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:17 smithi188 ceph-mon[92425]: pgmap v2887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:18 smithi161 ceph-mon[91579]: pgmap v2888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:18 smithi188 ceph-mon[92425]: pgmap v2888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:21 smithi161 ceph-mon[91579]: pgmap v2889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:21 smithi188 ceph-mon[92425]: pgmap v2889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:23 smithi161 ceph-mon[91579]: pgmap v2890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:23 smithi188 ceph-mon[92425]: pgmap v2890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:25 smithi161 ceph-mon[91579]: pgmap v2891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:25 smithi188 ceph-mon[92425]: pgmap v2891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:27 smithi161 ceph-mon[91579]: pgmap v2892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:27 smithi188 ceph-mon[92425]: pgmap v2892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:28 smithi161 ceph-mon[91579]: pgmap v2893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:28 smithi188 ceph-mon[92425]: pgmap v2893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:31 smithi161 ceph-mon[91579]: pgmap v2894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:31 smithi188 ceph-mon[92425]: pgmap v2894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:33 smithi161 ceph-mon[91579]: pgmap v2895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:33 smithi188 ceph-mon[92425]: pgmap v2895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:35 smithi161 ceph-mon[91579]: pgmap v2896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:35 smithi188 ceph-mon[92425]: pgmap v2896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:37 smithi161 ceph-mon[91579]: pgmap v2897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:37 smithi188 ceph-mon[92425]: pgmap v2897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:38 smithi161 ceph-mon[91579]: pgmap v2898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:38 smithi188 ceph-mon[92425]: pgmap v2898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:41 smithi161 ceph-mon[91579]: pgmap v2899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:41 smithi188 ceph-mon[92425]: pgmap v2899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:43 smithi161 ceph-mon[91579]: pgmap v2900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:43 smithi188 ceph-mon[92425]: pgmap v2900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:45 smithi161 ceph-mon[91579]: pgmap v2901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:45 smithi188 ceph-mon[92425]: pgmap v2901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:47 smithi161 ceph-mon[91579]: pgmap v2902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:47 smithi188 ceph-mon[92425]: pgmap v2902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:48 smithi161 ceph-mon[91579]: pgmap v2903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:48 smithi188 ceph-mon[92425]: pgmap v2903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:51 smithi161 ceph-mon[91579]: pgmap v2904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:51 smithi188 ceph-mon[92425]: pgmap v2904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:53 smithi161 ceph-mon[91579]: pgmap v2905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:53 smithi188 ceph-mon[92425]: pgmap v2905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:55 smithi161 ceph-mon[91579]: pgmap v2906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:55 smithi188 ceph-mon[92425]: pgmap v2906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:15:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:57 smithi161 ceph-mon[91579]: pgmap v2907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:15:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:15:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:15:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:15:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:57 smithi188 ceph-mon[92425]: pgmap v2907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:15:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:15:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:15:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:15:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:15:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:58 smithi161 ceph-mon[91579]: pgmap v2908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:15:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:58 smithi188 ceph-mon[92425]: pgmap v2908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:15:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:00.064 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:16:00.064 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:16:00.065 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:16:00.116 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:16:00.116 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:16:00.117 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:16:00.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:16:00.062+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:00.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:16:00.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:00.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:16:00.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:00.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:00.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:00.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:16:00.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:00.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:16:00.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:00.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:16:00.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:00.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T12:16:00.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:00.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:16:00.063+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:16:00.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:16:00.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:00.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:16:00.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:00.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:00.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:00.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:16:00.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:00.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:16:00.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:00.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:16:00.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:00.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T12:16:00.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:00.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:16:00.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:16:00.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:16:00.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:16:00.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:16:00.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:16:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:16:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:16:00.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:16:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:16:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:16:00.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:16:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:16:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:16:00.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:16:00.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:16:00.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:16:00.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:16:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:16:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:16:00.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:16:00.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:16:00.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:16:00.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:16:00.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:16:00.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:16:00.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:16:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:16:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:16:00.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:16:00.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:16:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:16:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:16:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:16:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:01.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:16:01.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:01.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:16:01.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:01.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:16:01.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:01.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:16:01.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:01.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:16:01.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:01.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:01.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:01.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:16:01.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:01.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:16:01.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:01.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:16:01.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:01.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:16:01.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:01.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:16:01.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:16:01.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:16:01.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:16:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:16:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:16:01.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:16:01.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:16:01.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:16:01.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:16:01.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:16:01.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:16:01.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:16:01.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:16:01.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:16:01.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:16:01.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:16:01.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:16:01.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:16:01.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:16:01.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:16:01.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:16:01.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:16:01.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:16:01.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:16:01.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:16:01.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:16:01.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:16:01.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:16:01.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:01.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:01.409 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:01 smithi161 ceph-mon[91579]: pgmap v2909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:16:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:16:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:16:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:16:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:01.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:16:01.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:01.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:16:01.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:01.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:16:01.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:16:01.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:16:01.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:16:01.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:16:01.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:16:01.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:16:01.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:16:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:16:01.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:16:01.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:16:01.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T12:16:01.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T12:16:01.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T12:16:01.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:16:01.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:16:01.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:16:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:16:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:16:01.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:16:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:16:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:16:01.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:16:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:16:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:16:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:16:01.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:16:01.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:16:01.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:16:01.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:16:01.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:16:01.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:16:01.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:16:01.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:16:01.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:16:01.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:16:01.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:16:01.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:16:01.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:16:01.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:16:01.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:16:01.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:16:01.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T12:16:01.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:16:01.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T12:16:01.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.440 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:01.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:01.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:01 smithi188 ceph-mon[92425]: pgmap v2909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: pgmap v2910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: pgmap v2910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:16:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:16:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:16:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:16:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:16:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:16:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:16:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:16:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:16:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:16:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:05 smithi161 ceph-mon[91579]: pgmap v2911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:05 smithi188 ceph-mon[92425]: pgmap v2911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:07 smithi161 ceph-mon[91579]: pgmap v2912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:07 smithi188 ceph-mon[92425]: pgmap v2912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:08 smithi161 ceph-mon[91579]: pgmap v2913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:08 smithi188 ceph-mon[92425]: pgmap v2913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:11 smithi161 ceph-mon[91579]: pgmap v2914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:11 smithi188 ceph-mon[92425]: pgmap v2914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:13 smithi161 ceph-mon[91579]: pgmap v2915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:13 smithi188 ceph-mon[92425]: pgmap v2915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:15 smithi161 ceph-mon[91579]: pgmap v2916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:15 smithi188 ceph-mon[92425]: pgmap v2916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:17 smithi161 ceph-mon[91579]: pgmap v2917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:17 smithi188 ceph-mon[92425]: pgmap v2917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:18 smithi161 ceph-mon[91579]: pgmap v2918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:18 smithi188 ceph-mon[92425]: pgmap v2918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:21 smithi161 ceph-mon[91579]: pgmap v2919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:21 smithi188 ceph-mon[92425]: pgmap v2919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:23 smithi161 ceph-mon[91579]: pgmap v2920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:23 smithi188 ceph-mon[92425]: pgmap v2920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:25 smithi161 ceph-mon[91579]: pgmap v2921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:25 smithi188 ceph-mon[92425]: pgmap v2921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:27 smithi161 ceph-mon[91579]: pgmap v2922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:27 smithi188 ceph-mon[92425]: pgmap v2922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:28 smithi161 ceph-mon[91579]: pgmap v2923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:28 smithi188 ceph-mon[92425]: pgmap v2923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:31 smithi161 ceph-mon[91579]: pgmap v2924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:31 smithi188 ceph-mon[92425]: pgmap v2924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:33 smithi161 ceph-mon[91579]: pgmap v2925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:33 smithi188 ceph-mon[92425]: pgmap v2925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:35 smithi161 ceph-mon[91579]: pgmap v2926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:35 smithi188 ceph-mon[92425]: pgmap v2926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:37 smithi161 ceph-mon[91579]: pgmap v2927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:37 smithi188 ceph-mon[92425]: pgmap v2927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:38 smithi161 ceph-mon[91579]: pgmap v2928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:38 smithi188 ceph-mon[92425]: pgmap v2928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:41 smithi161 ceph-mon[91579]: pgmap v2929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:41 smithi188 ceph-mon[92425]: pgmap v2929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:43 smithi161 ceph-mon[91579]: pgmap v2930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:43 smithi188 ceph-mon[92425]: pgmap v2930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:45 smithi161 ceph-mon[91579]: pgmap v2931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:45 smithi188 ceph-mon[92425]: pgmap v2931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:47 smithi161 ceph-mon[91579]: pgmap v2932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:47 smithi188 ceph-mon[92425]: pgmap v2932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:48 smithi161 ceph-mon[91579]: pgmap v2933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:48 smithi188 ceph-mon[92425]: pgmap v2933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:51 smithi161 ceph-mon[91579]: pgmap v2934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:51 smithi188 ceph-mon[92425]: pgmap v2934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:53 smithi161 ceph-mon[91579]: pgmap v2935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:53 smithi188 ceph-mon[92425]: pgmap v2935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:55 smithi161 ceph-mon[91579]: pgmap v2936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:55 smithi188 ceph-mon[92425]: pgmap v2936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:16:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:57 smithi161 ceph-mon[91579]: pgmap v2937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:16:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:16:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:16:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:16:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:57 smithi188 ceph-mon[92425]: pgmap v2937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:16:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:16:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:16:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:16:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:16:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:58 smithi161 ceph-mon[91579]: pgmap v2938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:16:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:58 smithi188 ceph-mon[92425]: pgmap v2938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:16:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:01 smithi161 ceph-mon[91579]: pgmap v2939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:01 smithi188 ceph-mon[92425]: pgmap v2939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:03 smithi161 ceph-mon[91579]: pgmap v2940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:17:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:17:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:17:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:03 smithi188 ceph-mon[92425]: pgmap v2940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:17:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:17:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:17:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:17:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:17:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:05 smithi161 ceph-mon[91579]: pgmap v2941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:05 smithi188 ceph-mon[92425]: pgmap v2941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:07 smithi161 ceph-mon[91579]: pgmap v2942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:07 smithi188 ceph-mon[92425]: pgmap v2942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:08 smithi161 ceph-mon[91579]: pgmap v2943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:08 smithi188 ceph-mon[92425]: pgmap v2943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:11 smithi161 ceph-mon[91579]: pgmap v2944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:11 smithi188 ceph-mon[92425]: pgmap v2944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:13 smithi161 ceph-mon[91579]: pgmap v2945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:13 smithi188 ceph-mon[92425]: pgmap v2945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:15 smithi161 ceph-mon[91579]: pgmap v2946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:15 smithi188 ceph-mon[92425]: pgmap v2946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:17 smithi161 ceph-mon[91579]: pgmap v2947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:17 smithi188 ceph-mon[92425]: pgmap v2947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:18 smithi161 ceph-mon[91579]: pgmap v2948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:18 smithi188 ceph-mon[92425]: pgmap v2948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:21 smithi161 ceph-mon[91579]: pgmap v2949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:21 smithi188 ceph-mon[92425]: pgmap v2949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:23 smithi161 ceph-mon[91579]: pgmap v2950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:23 smithi188 ceph-mon[92425]: pgmap v2950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:25 smithi161 ceph-mon[91579]: pgmap v2951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:25 smithi188 ceph-mon[92425]: pgmap v2951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:27 smithi161 ceph-mon[91579]: pgmap v2952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:27 smithi188 ceph-mon[92425]: pgmap v2952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:28 smithi161 ceph-mon[91579]: pgmap v2953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:28 smithi188 ceph-mon[92425]: pgmap v2953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:31 smithi161 ceph-mon[91579]: pgmap v2954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:31 smithi188 ceph-mon[92425]: pgmap v2954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-mon[91579]: pgmap v2955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:33.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:17:33.349+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:17:33.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:17:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:17:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:17:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:17:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:17:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:17:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:17:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:17:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:17:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:17:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:17:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:17:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:17:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:17:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:17:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:17:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:17:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:17:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:17:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:17:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:17:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:17:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:17:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:17:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:17:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:17:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:17:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:17:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:17:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:17:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:17:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:17:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:17:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:17:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:17:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:17:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:17:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:17:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:17:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:17:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:17:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:17:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:17:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:17:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:17:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=141] Local address: 172.21.15.161, port 52386 2022-07-03T12:17:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=141] Peer address: 172.21.15.101, port 22 2022-07-03T12:17:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=432] Received exit status 0 2022-07-03T12:17:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=432] Received channel close 2022-07-03T12:17:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=432] Channel closed 2022-07-03T12:17:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=433] Requesting new SSH session 2022-07-03T12:17:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=433] Command: which python3 2022-07-03T12:17:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=432] Received exit status 0 2022-07-03T12:17:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=432] Received channel close 2022-07-03T12:17:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=432] Channel closed 2022-07-03T12:17:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=433] Requesting new SSH session 2022-07-03T12:17:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=433] Command: which python3 2022-07-03T12:17:33.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=433] Received exit status 0 2022-07-03T12:17:33.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=433] Received channel close 2022-07-03T12:17:33.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=433] Channel closed 2022-07-03T12:17:33.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=434] Requesting new SSH session 2022-07-03T12:17:33.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=434] Command: sudo true 2022-07-03T12:17:33.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=433] Received exit status 0 2022-07-03T12:17:33.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=433] Received channel close 2022-07-03T12:17:33.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=433] Channel closed 2022-07-03T12:17:33.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=434] Requesting new SSH session 2022-07-03T12:17:33.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=434] Command: sudo true 2022-07-03T12:17:33.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=434] Received exit status 0 2022-07-03T12:17:33.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=434] Received channel close 2022-07-03T12:17:33.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=434] Channel closed 2022-07-03T12:17:33.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=435] Requesting new SSH session 2022-07-03T12:17:33.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:33.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=434] Received exit status 0 2022-07-03T12:17:33.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=434] Received channel close 2022-07-03T12:17:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=434] Channel closed 2022-07-03T12:17:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=435] Requesting new SSH session 2022-07-03T12:17:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=435] Received exit status 0 2022-07-03T12:17:33.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=435] Received channel close 2022-07-03T12:17:33.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=435] Channel closed 2022-07-03T12:17:33.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=435] Received exit status 0 2022-07-03T12:17:33.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=435] Received channel close 2022-07-03T12:17:33.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=435] Channel closed 2022-07-03T12:17:33.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=141] Login timeout expired 2022-07-03T12:17:33.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=141] Aborting connection 2022-07-03T12:17:33.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:17:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:17:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:33 smithi188 ceph-mon[92425]: pgmap v2955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:17:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:17:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:17:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:17:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:17:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:17:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:17:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:17:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:17:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:17:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:17:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:17:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:17:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:17:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:17:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:17:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:17:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:17:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:17:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:17:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:17:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:17:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:17:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:17:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:17:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:17:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:17:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:17:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:17:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:17:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:17:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:17:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:17:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:17:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:17:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:17:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:17:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:17:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:17:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:17:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:17:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:17:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:17:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:17:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:17:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:17:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=141] Local address: 172.21.15.161, port 52386 2022-07-03T12:17:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=141] Peer address: 172.21.15.101, port 22 2022-07-03T12:17:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=432] Received exit status 0 2022-07-03T12:17:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=432] Received channel close 2022-07-03T12:17:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=432] Channel closed 2022-07-03T12:17:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=433] Requesting new SSH session 2022-07-03T12:17:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=433] Command: which python3 2022-07-03T12:17:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=432] Received exit status 0 2022-07-03T12:17:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=432] Received channel close 2022-07-03T12:17:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=432] Channel closed 2022-07-03T12:17:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=433] Requesting new SSH session 2022-07-03T12:17:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=433] Command: which python3 2022-07-03T12:17:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=433] Received exit status 0 2022-07-03T12:17:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=433] Received channel close 2022-07-03T12:17:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=433] Channel closed 2022-07-03T12:17:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=434] Requesting new SSH session 2022-07-03T12:17:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=434] Command: sudo true 2022-07-03T12:17:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=433] Received exit status 0 2022-07-03T12:17:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=433] Received channel close 2022-07-03T12:17:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=433] Channel closed 2022-07-03T12:17:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=434] Requesting new SSH session 2022-07-03T12:17:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=434] Command: sudo true 2022-07-03T12:17:34.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=434] Received exit status 0 2022-07-03T12:17:34.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=434] Received channel close 2022-07-03T12:17:34.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=434] Channel closed 2022-07-03T12:17:34.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=435] Requesting new SSH session 2022-07-03T12:17:34.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:34.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=434] Received exit status 0 2022-07-03T12:17:34.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=434] Received channel close 2022-07-03T12:17:34.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=434] Channel closed 2022-07-03T12:17:34.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=435] Requesting new SSH session 2022-07-03T12:17:34.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:34.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=435] Received exit status 0 2022-07-03T12:17:34.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=435] Received channel close 2022-07-03T12:17:34.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=435] Channel closed 2022-07-03T12:17:34.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=435] Received exit status 0 2022-07-03T12:17:34.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=435] Received channel close 2022-07-03T12:17:34.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=435] Channel closed 2022-07-03T12:17:34.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=141] Login timeout expired 2022-07-03T12:17:34.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=141] Aborting connection 2022-07-03T12:17:34.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:17:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:17:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:17:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:17:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:17:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:17:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:17:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:17:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:17:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:17:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:17:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:17:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:17:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:17:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:17:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:17:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:17:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:17:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:17:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:17:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:17:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:17:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:17:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:17:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:17:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:17:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:17:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:17:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:17:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:17:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:17:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:17:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:17:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:17:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:17:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:17:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:17:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:17:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:17:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:17:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:17:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:17:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:17:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:17:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:17:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:17:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=141] Local address: 172.21.15.161, port 52386 2022-07-03T12:17:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=141] Peer address: 172.21.15.101, port 22 2022-07-03T12:17:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=432] Received exit status 0 2022-07-03T12:17:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=432] Received channel close 2022-07-03T12:17:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=432] Channel closed 2022-07-03T12:17:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=433] Requesting new SSH session 2022-07-03T12:17:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=433] Command: which python3 2022-07-03T12:17:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=432] Received exit status 0 2022-07-03T12:17:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=432] Received channel close 2022-07-03T12:17:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=432] Channel closed 2022-07-03T12:17:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=433] Requesting new SSH session 2022-07-03T12:17:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=433] Command: which python3 2022-07-03T12:17:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=433] Received exit status 0 2022-07-03T12:17:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=433] Received channel close 2022-07-03T12:17:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=433] Channel closed 2022-07-03T12:17:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=434] Requesting new SSH session 2022-07-03T12:17:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=434] Command: sudo true 2022-07-03T12:17:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=433] Received exit status 0 2022-07-03T12:17:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=433] Received channel close 2022-07-03T12:17:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=433] Channel closed 2022-07-03T12:17:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=434] Requesting new SSH session 2022-07-03T12:17:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=434] Command: sudo true 2022-07-03T12:17:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=434] Received exit status 0 2022-07-03T12:17:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=434] Received channel close 2022-07-03T12:17:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=434] Channel closed 2022-07-03T12:17:34.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=435] Requesting new SSH session 2022-07-03T12:17:34.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:34.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=434] Received exit status 0 2022-07-03T12:17:34.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=434] Received channel close 2022-07-03T12:17:34.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=434] Channel closed 2022-07-03T12:17:34.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=435] Requesting new SSH session 2022-07-03T12:17:34.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=435] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:17:34.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=435] Received exit status 0 2022-07-03T12:17:34.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=435] Received channel close 2022-07-03T12:17:34.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=435] Channel closed 2022-07-03T12:17:34.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=435] Received exit status 0 2022-07-03T12:17:34.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=435] Received channel close 2022-07-03T12:17:34.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=435] Channel closed 2022-07-03T12:17:34.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=141] Login timeout expired 2022-07-03T12:17:34.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=141] Aborting connection 2022-07-03T12:17:34.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:35 smithi161 ceph-mon[91579]: pgmap v2956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:35 smithi188 ceph-mon[92425]: pgmap v2956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:37 smithi161 ceph-mon[91579]: pgmap v2957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:37 smithi188 ceph-mon[92425]: pgmap v2957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:38 smithi161 ceph-mon[91579]: pgmap v2958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:38 smithi188 ceph-mon[92425]: pgmap v2958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:41 smithi161 ceph-mon[91579]: pgmap v2959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:41 smithi188 ceph-mon[92425]: pgmap v2959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:43 smithi161 ceph-mon[91579]: pgmap v2960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:43 smithi188 ceph-mon[92425]: pgmap v2960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:45 smithi161 ceph-mon[91579]: pgmap v2961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:45 smithi188 ceph-mon[92425]: pgmap v2961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:47 smithi161 ceph-mon[91579]: pgmap v2962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:47 smithi188 ceph-mon[92425]: pgmap v2962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:48 smithi161 ceph-mon[91579]: pgmap v2963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:48 smithi188 ceph-mon[92425]: pgmap v2963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:51 smithi161 ceph-mon[91579]: pgmap v2964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:51 smithi188 ceph-mon[92425]: pgmap v2964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:53 smithi161 ceph-mon[91579]: pgmap v2965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:53 smithi188 ceph-mon[92425]: pgmap v2965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:55 smithi161 ceph-mon[91579]: pgmap v2966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:55 smithi188 ceph-mon[92425]: pgmap v2966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:17:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:57 smithi161 ceph-mon[91579]: pgmap v2967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:17:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:17:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:17:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:17:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:57 smithi188 ceph-mon[92425]: pgmap v2967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:17:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:17:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:17:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:17:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:17:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:58 smithi188 ceph-mon[92425]: pgmap v2968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:17:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:58 smithi161 ceph-mon[91579]: pgmap v2968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:17:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:01 smithi161 ceph-mon[91579]: pgmap v2969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:01 smithi188 ceph-mon[92425]: pgmap v2969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:03 smithi161 ceph-mon[91579]: pgmap v2970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:03 smithi188 ceph-mon[92425]: pgmap v2970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:05 smithi161 ceph-mon[91579]: pgmap v2971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:05 smithi188 ceph-mon[92425]: pgmap v2971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:07 smithi161 ceph-mon[91579]: pgmap v2972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:07 smithi188 ceph-mon[92425]: pgmap v2972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:09.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:08 smithi188 ceph-mon[92425]: pgmap v2973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:09.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:08 smithi161 ceph-mon[91579]: pgmap v2973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:11 smithi161 ceph-mon[91579]: pgmap v2974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:11 smithi188 ceph-mon[92425]: pgmap v2974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:13 smithi161 ceph-mon[91579]: pgmap v2975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:13 smithi188 ceph-mon[92425]: pgmap v2975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:15 smithi161 ceph-mon[91579]: pgmap v2976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:15 smithi188 ceph-mon[92425]: pgmap v2976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:17 smithi161 ceph-mon[91579]: pgmap v2977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:17 smithi188 ceph-mon[92425]: pgmap v2977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:18 smithi188 ceph-mon[92425]: pgmap v2978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:18 smithi161 ceph-mon[91579]: pgmap v2978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:21 smithi161 ceph-mon[91579]: pgmap v2979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:21 smithi188 ceph-mon[92425]: pgmap v2979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:23 smithi161 ceph-mon[91579]: pgmap v2980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:23 smithi188 ceph-mon[92425]: pgmap v2980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:25 smithi161 ceph-mon[91579]: pgmap v2981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:25 smithi188 ceph-mon[92425]: pgmap v2981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:27 smithi161 ceph-mon[91579]: pgmap v2982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:27 smithi188 ceph-mon[92425]: pgmap v2982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:28 smithi188 ceph-mon[92425]: pgmap v2983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:28 smithi161 ceph-mon[91579]: pgmap v2983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:31 smithi161 ceph-mon[91579]: pgmap v2984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:31 smithi188 ceph-mon[92425]: pgmap v2984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:33 smithi188 ceph-mon[92425]: pgmap v2985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:18:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:18:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:18:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:33 smithi161 ceph-mon[91579]: pgmap v2985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:18:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:18:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:18:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:35 smithi161 ceph-mon[91579]: pgmap v2986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:35 smithi188 ceph-mon[92425]: pgmap v2986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:37 smithi161 ceph-mon[91579]: pgmap v2987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:37 smithi188 ceph-mon[92425]: pgmap v2987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:38 smithi188 ceph-mon[92425]: pgmap v2988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:38 smithi161 ceph-mon[91579]: pgmap v2988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:41 smithi161 ceph-mon[91579]: pgmap v2989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:41 smithi188 ceph-mon[92425]: pgmap v2989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:43 smithi161 ceph-mon[91579]: pgmap v2990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:43 smithi188 ceph-mon[92425]: pgmap v2990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:45 smithi161 ceph-mon[91579]: pgmap v2991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:45 smithi188 ceph-mon[92425]: pgmap v2991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:47 smithi161 ceph-mon[91579]: pgmap v2992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:47 smithi188 ceph-mon[92425]: pgmap v2992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:48 smithi188 ceph-mon[92425]: pgmap v2993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:48 smithi161 ceph-mon[91579]: pgmap v2993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:51 smithi161 ceph-mon[91579]: pgmap v2994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:51 smithi188 ceph-mon[92425]: pgmap v2994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:53 smithi161 ceph-mon[91579]: pgmap v2995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:53 smithi188 ceph-mon[92425]: pgmap v2995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:55 smithi161 ceph-mon[91579]: pgmap v2996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:55 smithi188 ceph-mon[92425]: pgmap v2996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:18:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:57 smithi161 ceph-mon[91579]: pgmap v2997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:18:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:18:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:57 smithi188 ceph-mon[92425]: pgmap v2997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:18:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:18:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:18:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:18:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:18:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:58 smithi188 ceph-mon[92425]: pgmap v2998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:18:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:58 smithi161 ceph-mon[91579]: pgmap v2998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:18:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:01 smithi161 ceph-mon[91579]: pgmap v2999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:01 smithi188 ceph-mon[92425]: pgmap v2999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-mon[91579]: pgmap v3000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:03.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:19:03.385+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:19:03.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:19:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:19:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:19:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:19:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:19:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:19:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:19:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:19:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:19:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:19:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:19:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:19:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:19:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:19:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:19:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:19:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:19:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:19:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:19:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:19:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:19:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:19:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:19:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:19:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:19:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:19:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:19:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:19:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:19:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:19:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:19:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:19:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:19:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:19:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:19:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:19:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:19:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:19:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:19:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:19:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:19:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:19:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:19:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:19:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:19:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=143] Local address: 172.21.15.161, port 52438 2022-07-03T12:19:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=143] Peer address: 172.21.15.101, port 22 2022-07-03T12:19:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=143] Connection lost 2022-07-03T12:19:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=143] Aborting connection 2022-07-03T12:19:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:19:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:19:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:03 smithi188 ceph-mon[92425]: pgmap v3000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:19:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:19:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:19:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:19:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:19:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:19:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:19:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:19:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:19:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:19:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:19:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:19:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:19:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:19:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:19:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:19:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:19:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:19:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:19:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:19:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:19:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:19:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:19:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:19:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:19:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:19:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:19:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:19:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:19:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:19:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:19:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:19:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:19:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:19:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:19:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:19:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:19:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:19:04.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:19:04.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:19:04.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:19:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:19:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:19:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:19:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:19:04.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:19:04.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:19:04.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=143] Local address: 172.21.15.161, port 52438 2022-07-03T12:19:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=143] Peer address: 172.21.15.101, port 22 2022-07-03T12:19:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=143] Connection lost 2022-07-03T12:19:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=143] Aborting connection 2022-07-03T12:19:04.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:19:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:19:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:19:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:19:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:19:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:19:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:19:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:19:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:19:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:19:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:19:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:19:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:19:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:19:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:19:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:19:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:19:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:19:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:19:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:19:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:19:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:19:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:19:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:19:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:19:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:19:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:19:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:19:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:19:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:19:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:19:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:19:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:19:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:19:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:19:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:19:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:19:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:19:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:19:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:19:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:19:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:19:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:19:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:19:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:19:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:19:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:19:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=143] Local address: 172.21.15.161, port 52438 2022-07-03T12:19:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=143] Peer address: 172.21.15.101, port 22 2022-07-03T12:19:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=143] Connection lost 2022-07-03T12:19:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=143] Aborting connection 2022-07-03T12:19:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:05 smithi161 ceph-mon[91579]: pgmap v3001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:05 smithi188 ceph-mon[92425]: pgmap v3001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:07 smithi161 ceph-mon[91579]: pgmap v3002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:07 smithi188 ceph-mon[92425]: pgmap v3002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:08 smithi188 ceph-mon[92425]: pgmap v3003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:08 smithi161 ceph-mon[91579]: pgmap v3003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:11 smithi188 ceph-mon[92425]: pgmap v3004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:11 smithi161 ceph-mon[91579]: pgmap v3004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:13 smithi161 ceph-mon[91579]: pgmap v3005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:13 smithi188 ceph-mon[92425]: pgmap v3005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:15 smithi161 ceph-mon[91579]: pgmap v3006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:15 smithi188 ceph-mon[92425]: pgmap v3006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:17 smithi161 ceph-mon[91579]: pgmap v3007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:17 smithi188 ceph-mon[92425]: pgmap v3007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:18 smithi188 ceph-mon[92425]: pgmap v3008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:18 smithi161 ceph-mon[91579]: pgmap v3008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:21 smithi188 ceph-mon[92425]: pgmap v3009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:21 smithi161 ceph-mon[91579]: pgmap v3009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:23 smithi161 ceph-mon[91579]: pgmap v3010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:23 smithi188 ceph-mon[92425]: pgmap v3010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:25 smithi161 ceph-mon[91579]: pgmap v3011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:25 smithi188 ceph-mon[92425]: pgmap v3011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:27 smithi161 ceph-mon[91579]: pgmap v3012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:27 smithi188 ceph-mon[92425]: pgmap v3012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:28 smithi188 ceph-mon[92425]: pgmap v3013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:28 smithi161 ceph-mon[91579]: pgmap v3013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:31 smithi188 ceph-mon[92425]: pgmap v3014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:31 smithi161 ceph-mon[91579]: pgmap v3014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:33 smithi161 ceph-mon[91579]: pgmap v3015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:33 smithi188 ceph-mon[92425]: pgmap v3015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:35 smithi161 ceph-mon[91579]: pgmap v3016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:35 smithi188 ceph-mon[92425]: pgmap v3016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:37 smithi161 ceph-mon[91579]: pgmap v3017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:37 smithi188 ceph-mon[92425]: pgmap v3017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:38 smithi188 ceph-mon[92425]: pgmap v3018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:38 smithi161 ceph-mon[91579]: pgmap v3018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:41 smithi188 ceph-mon[92425]: pgmap v3019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:41 smithi161 ceph-mon[91579]: pgmap v3019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:43 smithi161 ceph-mon[91579]: pgmap v3020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:43 smithi188 ceph-mon[92425]: pgmap v3020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:45 smithi161 ceph-mon[91579]: pgmap v3021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:45 smithi188 ceph-mon[92425]: pgmap v3021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:47 smithi161 ceph-mon[91579]: pgmap v3022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:47 smithi188 ceph-mon[92425]: pgmap v3022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:48 smithi161 ceph-mon[91579]: pgmap v3023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:48 smithi188 ceph-mon[92425]: pgmap v3023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:51 smithi188 ceph-mon[92425]: pgmap v3024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:51 smithi161 ceph-mon[91579]: pgmap v3024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:53 smithi161 ceph-mon[91579]: pgmap v3025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:53 smithi188 ceph-mon[92425]: pgmap v3025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:55 smithi161 ceph-mon[91579]: pgmap v3026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:55 smithi188 ceph-mon[92425]: pgmap v3026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:19:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:57 smithi161 ceph-mon[91579]: pgmap v3027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:19:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:19:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:19:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:19:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:57 smithi188 ceph-mon[92425]: pgmap v3027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:19:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:19:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:19:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:19:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:19:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:58 smithi188 ceph-mon[92425]: pgmap v3028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:19:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:58 smithi161 ceph-mon[91579]: pgmap v3028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:19:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:20:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:20:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:01 smithi161 ceph-mon[91579]: pgmap v3029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:01 smithi188 ceph-mon[92425]: pgmap v3029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-mon[91579]: pgmap v3030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:20:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:20:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:20:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:20:03.423+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:20:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:20:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:20:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:20:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:20:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:20:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:20:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:20:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:20:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:20:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:20:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:20:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:20:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:20:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:20:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:20:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:20:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:20:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:20:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:20:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:20:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:20:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:20:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:20:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:20:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:20:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:20:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:20:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:20:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:20:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:20:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:20:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:20:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:20:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:20:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:20:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:20:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:20:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:20:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:20:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:20:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:20:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:20:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:20:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:20:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:20:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=145] Local address: 172.21.15.161, port 52466 2022-07-03T12:20:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=145] Peer address: 172.21.15.101, port 22 2022-07-03T12:20:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=145] Connection lost 2022-07-03T12:20:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=145] Aborting connection 2022-07-03T12:20:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:20:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:20:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:03 smithi188 ceph-mon[92425]: pgmap v3030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:20:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:20:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:20:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:20:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:20:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:20:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:20:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:20:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:20:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:20:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:20:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:20:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:20:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:20:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:20:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:20:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:20:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:20:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:20:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:20:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:20:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:20:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:20:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:20:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:20:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:20:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:20:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:20:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:20:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:20:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:20:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:20:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:20:04.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:20:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:20:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:20:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:20:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:20:04.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:20:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:20:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:20:04.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:20:04.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:20:04.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:20:04.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:20:04.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:20:04.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:20:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:20:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:20:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:20:04.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=145] Local address: 172.21.15.161, port 52466 2022-07-03T12:20:04.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=145] Peer address: 172.21.15.101, port 22 2022-07-03T12:20:04.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=145] Connection lost 2022-07-03T12:20:04.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=145] Aborting connection 2022-07-03T12:20:04.906 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:20:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:20:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:20:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:20:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:20:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:20:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:20:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:20:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:20:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:20:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:20:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:20:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:20:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:20:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:20:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:20:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:20:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:20:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:20:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:20:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:20:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:20:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:20:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:20:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:20:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:20:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:20:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:20:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:20:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:20:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:20:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:20:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:20:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:20:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:20:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:20:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:20:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:20:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:20:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:20:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:20:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:20:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:20:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:20:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:20:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:20:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:20:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=145] Local address: 172.21.15.161, port 52466 2022-07-03T12:20:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=145] Peer address: 172.21.15.101, port 22 2022-07-03T12:20:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=145] Connection lost 2022-07-03T12:20:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=145] Aborting connection 2022-07-03T12:20:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:05 smithi161 ceph-mon[91579]: pgmap v3031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:05 smithi188 ceph-mon[92425]: pgmap v3031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:07 smithi161 ceph-mon[91579]: pgmap v3032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:07 smithi188 ceph-mon[92425]: pgmap v3032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:08 smithi161 ceph-mon[91579]: pgmap v3033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:08 smithi188 ceph-mon[92425]: pgmap v3033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:11 smithi188 ceph-mon[92425]: pgmap v3034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:11 smithi161 ceph-mon[91579]: pgmap v3034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:13 smithi161 ceph-mon[91579]: pgmap v3035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:13.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:13 smithi188 ceph-mon[92425]: pgmap v3035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:15 smithi161 ceph-mon[91579]: pgmap v3036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:15 smithi188 ceph-mon[92425]: pgmap v3036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:17 smithi161 ceph-mon[91579]: pgmap v3037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:17 smithi188 ceph-mon[92425]: pgmap v3037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:18 smithi161 ceph-mon[91579]: pgmap v3038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:18 smithi188 ceph-mon[92425]: pgmap v3038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:21 smithi161 ceph-mon[91579]: pgmap v3039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:21 smithi188 ceph-mon[92425]: pgmap v3039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:23 smithi161 ceph-mon[91579]: pgmap v3040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:23 smithi188 ceph-mon[92425]: pgmap v3040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:25 smithi161 ceph-mon[91579]: pgmap v3041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:25 smithi188 ceph-mon[92425]: pgmap v3041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:27 smithi161 ceph-mon[91579]: pgmap v3042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:27 smithi188 ceph-mon[92425]: pgmap v3042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:28 smithi161 ceph-mon[91579]: pgmap v3043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:28 smithi188 ceph-mon[92425]: pgmap v3043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:31 smithi161 ceph-mon[91579]: pgmap v3044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:31 smithi188 ceph-mon[92425]: pgmap v3044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:33 smithi161 ceph-mon[91579]: pgmap v3045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:33 smithi188 ceph-mon[92425]: pgmap v3045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:35 smithi161 ceph-mon[91579]: pgmap v3046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:35 smithi188 ceph-mon[92425]: pgmap v3046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:37 smithi161 ceph-mon[91579]: pgmap v3047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:37 smithi188 ceph-mon[92425]: pgmap v3047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:38 smithi161 ceph-mon[91579]: pgmap v3048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:38 smithi188 ceph-mon[92425]: pgmap v3048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:41 smithi161 ceph-mon[91579]: pgmap v3049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:41 smithi188 ceph-mon[92425]: pgmap v3049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:43 smithi161 ceph-mon[91579]: pgmap v3050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:43 smithi188 ceph-mon[92425]: pgmap v3050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:45 smithi161 ceph-mon[91579]: pgmap v3051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:45 smithi188 ceph-mon[92425]: pgmap v3051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:47 smithi161 ceph-mon[91579]: pgmap v3052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:47 smithi188 ceph-mon[92425]: pgmap v3052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:48 smithi161 ceph-mon[91579]: pgmap v3053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:48 smithi188 ceph-mon[92425]: pgmap v3053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:51 smithi161 ceph-mon[91579]: pgmap v3054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:51 smithi188 ceph-mon[92425]: pgmap v3054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:53 smithi161 ceph-mon[91579]: pgmap v3055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:53 smithi188 ceph-mon[92425]: pgmap v3055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:55 smithi161 ceph-mon[91579]: pgmap v3056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:55 smithi188 ceph-mon[92425]: pgmap v3056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:20:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:57 smithi161 ceph-mon[91579]: pgmap v3057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:20:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:20:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:20:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:20:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:57 smithi188 ceph-mon[92425]: pgmap v3057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:20:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:20:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:20:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:58 smithi161 ceph-mon[91579]: pgmap v3058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:20:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:58 smithi188 ceph-mon[92425]: pgmap v3058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:20:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:01 smithi161 ceph-mon[91579]: pgmap v3059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:01 smithi188 ceph-mon[92425]: pgmap v3059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-mon[91579]: pgmap v3060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:03 smithi188 ceph-mon[92425]: pgmap v3060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:04.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:21:03.927+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:21:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:21:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:21:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:21:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:21:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:21:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:21:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:21:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:21:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:21:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:21:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:21:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:21:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:21:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:21:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:21:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:21:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:21:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:21:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:21:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:21:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:21:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:21:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:21:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:21:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:21:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:21:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:21:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:21:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:21:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:21:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:21:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:21:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:21:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:21:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:21:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:21:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:21:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:21:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:21:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:21:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:21:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:21:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:21:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:21:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:21:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=147] Local address: 172.21.15.161, port 52496 2022-07-03T12:21:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=147] Peer address: 172.21.15.101, port 22 2022-07-03T12:21:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=147] Connection lost 2022-07-03T12:21:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=147] Aborting connection 2022-07-03T12:21:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:21:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:21:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:21:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:21:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:21:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:21:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:21:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:21:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:21:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:21:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:21:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:21:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:21:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:21:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:21:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:21:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:21:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:21:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:21:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:21:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:21:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:21:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:21:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:21:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:21:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:21:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:21:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:21:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:21:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:21:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:21:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:21:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:21:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:21:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:21:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:21:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:21:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:21:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:21:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:21:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:21:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:21:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:21:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:21:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:21:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:21:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:21:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:21:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:21:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:21:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:21:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:21:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:21:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:21:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:21:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=147] Local address: 172.21.15.161, port 52496 2022-07-03T12:21:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=147] Peer address: 172.21.15.101, port 22 2022-07-03T12:21:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=147] Connection lost 2022-07-03T12:21:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=147] Aborting connection 2022-07-03T12:21:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:05 smithi161 ceph-mon[91579]: pgmap v3061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:21:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:21:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:21:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:21:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:21:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:21:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:21:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:21:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:21:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:21:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:21:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:21:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:21:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:21:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:21:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:21:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:21:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:21:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:21:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:21:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:21:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:21:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:21:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:21:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:21:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:21:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:21:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:21:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:21:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:21:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:21:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:21:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:21:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:21:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:21:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:21:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:21:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:21:05.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:21:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:21:05.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:21:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:21:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:21:05.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:21:05.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:21:05.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:21:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:21:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=147] Local address: 172.21.15.161, port 52496 2022-07-03T12:21:05.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=147] Peer address: 172.21.15.101, port 22 2022-07-03T12:21:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=147] Connection lost 2022-07-03T12:21:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=147] Aborting connection 2022-07-03T12:21:05.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:05 smithi188 ceph-mon[92425]: pgmap v3061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:05.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:07 smithi161 ceph-mon[91579]: pgmap v3062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:07 smithi188 ceph-mon[92425]: pgmap v3062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:08 smithi161 ceph-mon[91579]: pgmap v3063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:08 smithi188 ceph-mon[92425]: pgmap v3063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:11 smithi161 ceph-mon[91579]: pgmap v3064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:11 smithi188 ceph-mon[92425]: pgmap v3064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:13 smithi161 ceph-mon[91579]: pgmap v3065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:13 smithi188 ceph-mon[92425]: pgmap v3065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:15 smithi161 ceph-mon[91579]: pgmap v3066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:15 smithi188 ceph-mon[92425]: pgmap v3066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:17 smithi161 ceph-mon[91579]: pgmap v3067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:17 smithi188 ceph-mon[92425]: pgmap v3067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:18 smithi161 ceph-mon[91579]: pgmap v3068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:18 smithi188 ceph-mon[92425]: pgmap v3068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:21 smithi161 ceph-mon[91579]: pgmap v3069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:21 smithi188 ceph-mon[92425]: pgmap v3069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:23 smithi161 ceph-mon[91579]: pgmap v3070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:23 smithi188 ceph-mon[92425]: pgmap v3070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:25 smithi161 ceph-mon[91579]: pgmap v3071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:25 smithi188 ceph-mon[92425]: pgmap v3071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:27 smithi161 ceph-mon[91579]: pgmap v3072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:27 smithi188 ceph-mon[92425]: pgmap v3072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:28 smithi161 ceph-mon[91579]: pgmap v3073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:28 smithi188 ceph-mon[92425]: pgmap v3073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:31 smithi161 ceph-mon[91579]: pgmap v3074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:31 smithi188 ceph-mon[92425]: pgmap v3074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:33 smithi161 ceph-mon[91579]: pgmap v3075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:33 smithi188 ceph-mon[92425]: pgmap v3075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:35 smithi161 ceph-mon[91579]: pgmap v3076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:35 smithi188 ceph-mon[92425]: pgmap v3076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:37 smithi161 ceph-mon[91579]: pgmap v3077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:37 smithi188 ceph-mon[92425]: pgmap v3077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:38 smithi161 ceph-mon[91579]: pgmap v3078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:38 smithi188 ceph-mon[92425]: pgmap v3078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:41 smithi161 ceph-mon[91579]: pgmap v3079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:41 smithi188 ceph-mon[92425]: pgmap v3079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:43 smithi161 ceph-mon[91579]: pgmap v3080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:43 smithi188 ceph-mon[92425]: pgmap v3080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:45 smithi161 ceph-mon[91579]: pgmap v3081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:45 smithi188 ceph-mon[92425]: pgmap v3081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:47 smithi161 ceph-mon[91579]: pgmap v3082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:47 smithi188 ceph-mon[92425]: pgmap v3082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:48 smithi161 ceph-mon[91579]: pgmap v3083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:48 smithi188 ceph-mon[92425]: pgmap v3083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:51 smithi161 ceph-mon[91579]: pgmap v3084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:51 smithi188 ceph-mon[92425]: pgmap v3084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:53 smithi161 ceph-mon[91579]: pgmap v3085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:53 smithi188 ceph-mon[92425]: pgmap v3085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:55 smithi161 ceph-mon[91579]: pgmap v3086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:55 smithi188 ceph-mon[92425]: pgmap v3086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:21:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:57 smithi161 ceph-mon[91579]: pgmap v3087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:21:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:21:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:21:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:21:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:57 smithi188 ceph-mon[92425]: pgmap v3087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:21:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:21:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:21:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:21:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:21:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:58 smithi161 ceph-mon[91579]: pgmap v3088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:21:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:58 smithi188 ceph-mon[92425]: pgmap v3088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:21:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:01 smithi161 ceph-mon[91579]: pgmap v3089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:01 smithi188 ceph-mon[92425]: pgmap v3089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:03 smithi161 ceph-mon[91579]: pgmap v3090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:03 smithi188 ceph-mon[92425]: pgmap v3090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:22:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:22:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:22:04.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:22:04.441+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:22:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:22:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:22:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:22:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:22:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:22:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:22:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:22:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:22:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:22:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:22:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:22:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:22:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:22:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:22:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:22:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:22:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:22:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:22:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:22:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:22:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:22:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:22:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:22:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:22:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:22:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:22:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:22:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:22:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:22:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:22:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:22:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:22:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:22:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:22:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:22:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:22:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:22:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:22:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:22:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:22:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:22:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:22:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:22:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:22:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:22:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=149] Local address: 172.21.15.161, port 52524 2022-07-03T12:22:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=149] Peer address: 172.21.15.101, port 22 2022-07-03T12:22:04.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=149] Connection lost 2022-07-03T12:22:04.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=149] Aborting connection 2022-07-03T12:22:04.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:22:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:22:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:22:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:22:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:22:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:22:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:22:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:22:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:22:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:22:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:22:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:22:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:22:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:22:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:22:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:22:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:22:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:22:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:22:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:22:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:22:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:22:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:22:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:22:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:22:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:22:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:22:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:22:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:22:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:22:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:22:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:22:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:22:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:22:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:22:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:22:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:22:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:22:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:22:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:22:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:22:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:22:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:22:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:22:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:22:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:22:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:22:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:22:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:22:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:22:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:22:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:22:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=149] Local address: 172.21.15.161, port 52524 2022-07-03T12:22:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=149] Peer address: 172.21.15.101, port 22 2022-07-03T12:22:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=149] Connection lost 2022-07-03T12:22:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=149] Aborting connection 2022-07-03T12:22:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:05 smithi161 ceph-mon[91579]: pgmap v3091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:22:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:22:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:22:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:22:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:22:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:22:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:22:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:22:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:22:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:22:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:22:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:22:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:22:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:22:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:22:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:22:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:22:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:22:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:22:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:22:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:22:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:22:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:22:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:22:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:22:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:22:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:22:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:22:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:22:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:22:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:22:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:22:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:22:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:22:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:22:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:22:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:22:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:22:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:22:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:22:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:22:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:22:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:22:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:22:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:22:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:22:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:22:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=149] Local address: 172.21.15.161, port 52524 2022-07-03T12:22:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=149] Peer address: 172.21.15.101, port 22 2022-07-03T12:22:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=149] Connection lost 2022-07-03T12:22:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=149] Aborting connection 2022-07-03T12:22:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:05 smithi188 ceph-mon[92425]: pgmap v3091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:05.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:07 smithi161 ceph-mon[91579]: pgmap v3092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:07 smithi188 ceph-mon[92425]: pgmap v3092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:08 smithi161 ceph-mon[91579]: pgmap v3093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:08 smithi188 ceph-mon[92425]: pgmap v3093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:11 smithi161 ceph-mon[91579]: pgmap v3094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:11 smithi188 ceph-mon[92425]: pgmap v3094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:13 smithi161 ceph-mon[91579]: pgmap v3095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:13 smithi188 ceph-mon[92425]: pgmap v3095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:15 smithi161 ceph-mon[91579]: pgmap v3096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:15 smithi188 ceph-mon[92425]: pgmap v3096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:17 smithi161 ceph-mon[91579]: pgmap v3097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:17 smithi188 ceph-mon[92425]: pgmap v3097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:18 smithi161 ceph-mon[91579]: pgmap v3098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:18 smithi188 ceph-mon[92425]: pgmap v3098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:21 smithi161 ceph-mon[91579]: pgmap v3099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:21 smithi188 ceph-mon[92425]: pgmap v3099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:23 smithi161 ceph-mon[91579]: pgmap v3100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:23 smithi188 ceph-mon[92425]: pgmap v3100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:25 smithi161 ceph-mon[91579]: pgmap v3101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:25 smithi188 ceph-mon[92425]: pgmap v3101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:27 smithi161 ceph-mon[91579]: pgmap v3102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:27 smithi188 ceph-mon[92425]: pgmap v3102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:28 smithi161 ceph-mon[91579]: pgmap v3103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:28 smithi188 ceph-mon[92425]: pgmap v3103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:31 smithi161 ceph-mon[91579]: pgmap v3104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:31 smithi188 ceph-mon[92425]: pgmap v3104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:33 smithi161 ceph-mon[91579]: pgmap v3105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:33 smithi188 ceph-mon[92425]: pgmap v3105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:35 smithi161 ceph-mon[91579]: pgmap v3106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:35 smithi188 ceph-mon[92425]: pgmap v3106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:37 smithi161 ceph-mon[91579]: pgmap v3107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:37 smithi188 ceph-mon[92425]: pgmap v3107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:22:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:38 smithi161 ceph-mon[91579]: pgmap v3108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:38 smithi188 ceph-mon[92425]: pgmap v3108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:41 smithi161 ceph-mon[91579]: pgmap v3109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:41 smithi188 ceph-mon[92425]: pgmap v3109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:43 smithi161 ceph-mon[91579]: pgmap v3110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:43 smithi188 ceph-mon[92425]: pgmap v3110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:45 smithi161 ceph-mon[91579]: pgmap v3111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:45 smithi188 ceph-mon[92425]: pgmap v3111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:47 smithi161 ceph-mon[91579]: pgmap v3112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:47 smithi188 ceph-mon[92425]: pgmap v3112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:48 smithi161 ceph-mon[91579]: pgmap v3113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:48 smithi188 ceph-mon[92425]: pgmap v3113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:51 smithi161 ceph-mon[91579]: pgmap v3114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:51 smithi188 ceph-mon[92425]: pgmap v3114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:53 smithi161 ceph-mon[91579]: pgmap v3115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:53 smithi188 ceph-mon[92425]: pgmap v3115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:55 smithi161 ceph-mon[91579]: pgmap v3116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:55 smithi188 ceph-mon[92425]: pgmap v3116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:22:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:57 smithi161 ceph-mon[91579]: pgmap v3117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:22:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:22:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:22:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:22:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:57 smithi188 ceph-mon[92425]: pgmap v3117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:22:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:22:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:22:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:22:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:22:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:58 smithi161 ceph-mon[91579]: pgmap v3118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:22:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:58 smithi188 ceph-mon[92425]: pgmap v3118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:22:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:01 smithi161 ceph-mon[91579]: pgmap v3119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:01 smithi188 ceph-mon[92425]: pgmap v3119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:03 smithi161 ceph-mon[91579]: pgmap v3120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:03 smithi188 ceph-mon[92425]: pgmap v3120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:05 smithi161 ceph-mon[91579]: pgmap v3121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:23:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:23:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:23:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:05 smithi188 ceph-mon[92425]: pgmap v3121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:23:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:07 smithi161 ceph-mon[91579]: pgmap v3122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:07 smithi188 ceph-mon[92425]: pgmap v3122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:08 smithi161 ceph-mon[91579]: pgmap v3123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:08 smithi188 ceph-mon[92425]: pgmap v3123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:11 smithi161 ceph-mon[91579]: pgmap v3124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:11 smithi188 ceph-mon[92425]: pgmap v3124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:13 smithi161 ceph-mon[91579]: pgmap v3125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:13 smithi188 ceph-mon[92425]: pgmap v3125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:15 smithi161 ceph-mon[91579]: pgmap v3126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:15 smithi188 ceph-mon[92425]: pgmap v3126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:17 smithi161 ceph-mon[91579]: pgmap v3127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:17 smithi188 ceph-mon[92425]: pgmap v3127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:18 smithi161 ceph-mon[91579]: pgmap v3128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:18 smithi188 ceph-mon[92425]: pgmap v3128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:21 smithi161 ceph-mon[91579]: pgmap v3129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:21 smithi188 ceph-mon[92425]: pgmap v3129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:23 smithi161 ceph-mon[91579]: pgmap v3130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:23 smithi188 ceph-mon[92425]: pgmap v3130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:25 smithi161 ceph-mon[91579]: pgmap v3131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:25 smithi188 ceph-mon[92425]: pgmap v3131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:27 smithi161 ceph-mon[91579]: pgmap v3132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:27 smithi188 ceph-mon[92425]: pgmap v3132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:28 smithi161 ceph-mon[91579]: pgmap v3133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:28 smithi188 ceph-mon[92425]: pgmap v3133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:31 smithi161 ceph-mon[91579]: pgmap v3134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:31 smithi188 ceph-mon[92425]: pgmap v3134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:33 smithi161 ceph-mon[91579]: pgmap v3135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:33 smithi188 ceph-mon[92425]: pgmap v3135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:23:34.941+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:23:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:23:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:23:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:23:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:23:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:23:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:23:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:23:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:23:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:23:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:23:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:23:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:23:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:23:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:23:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:23:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:23:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:23:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:23:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:23:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:23:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:23:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:23:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:23:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:23:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:23:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:23:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:23:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:23:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:23:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:23:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:23:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:23:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:23:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:23:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:23:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:23:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:23:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:23:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:23:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:23:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:23:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:23:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:23:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:23:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:23:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=151] Local address: 172.21.15.161, port 52564 2022-07-03T12:23:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=151] Peer address: 172.21.15.101, port 22 2022-07-03T12:23:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=151] Connection lost 2022-07-03T12:23:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=151] Aborting connection 2022-07-03T12:23:35.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:23:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:23:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:35 smithi161 ceph-mon[91579]: pgmap v3136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:35 smithi188 ceph-mon[92425]: pgmap v3136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:23:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:23:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:23:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:23:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:23:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:23:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:23:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:23:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:23:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:23:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:23:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:23:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:23:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:23:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:23:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:23:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:23:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:23:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:23:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:23:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:23:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:23:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:23:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:23:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:23:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:23:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:23:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:23:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:23:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:23:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:23:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:23:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:23:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:23:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:23:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:23:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:23:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:23:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:23:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:23:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:23:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:23:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:23:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:23:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:23:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:23:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:23:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=151] Local address: 172.21.15.161, port 52564 2022-07-03T12:23:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=151] Peer address: 172.21.15.101, port 22 2022-07-03T12:23:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=151] Connection lost 2022-07-03T12:23:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=151] Aborting connection 2022-07-03T12:23:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:23:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:23:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:23:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:23:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:23:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:23:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:23:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:23:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:23:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:23:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:23:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:23:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:23:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:23:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:23:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:23:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:23:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:23:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:23:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:23:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:23:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:23:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:23:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:23:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:23:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:23:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:23:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:23:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:23:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:23:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:23:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:23:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:23:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:23:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:23:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:23:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:23:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:23:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:23:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:23:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:23:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:23:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:23:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:23:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:23:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:23:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:23:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=151] Local address: 172.21.15.161, port 52564 2022-07-03T12:23:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=151] Peer address: 172.21.15.101, port 22 2022-07-03T12:23:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=151] Connection lost 2022-07-03T12:23:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=151] Aborting connection 2022-07-03T12:23:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:37 smithi161 ceph-mon[91579]: pgmap v3137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:37 smithi188 ceph-mon[92425]: pgmap v3137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:38 smithi161 ceph-mon[91579]: pgmap v3138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:38 smithi188 ceph-mon[92425]: pgmap v3138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:41 smithi161 ceph-mon[91579]: pgmap v3139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:41 smithi188 ceph-mon[92425]: pgmap v3139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:43 smithi161 ceph-mon[91579]: pgmap v3140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:43 smithi188 ceph-mon[92425]: pgmap v3140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:45 smithi161 ceph-mon[91579]: pgmap v3141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:45 smithi188 ceph-mon[92425]: pgmap v3141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:47 smithi161 ceph-mon[91579]: pgmap v3142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:47 smithi188 ceph-mon[92425]: pgmap v3142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:48 smithi161 ceph-mon[91579]: pgmap v3143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:48 smithi188 ceph-mon[92425]: pgmap v3143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:51 smithi161 ceph-mon[91579]: pgmap v3144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:51 smithi188 ceph-mon[92425]: pgmap v3144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:53 smithi161 ceph-mon[91579]: pgmap v3145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:53 smithi188 ceph-mon[92425]: pgmap v3145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:55 smithi161 ceph-mon[91579]: pgmap v3146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:55 smithi188 ceph-mon[92425]: pgmap v3146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:23:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:57 smithi161 ceph-mon[91579]: pgmap v3147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:23:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:23:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:23:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:23:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:57 smithi188 ceph-mon[92425]: pgmap v3147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:23:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:23:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:58 smithi161 ceph-mon[91579]: pgmap v3148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:23:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:58 smithi188 ceph-mon[92425]: pgmap v3148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:23:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:01 smithi161 ceph-mon[91579]: pgmap v3149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:01 smithi188 ceph-mon[92425]: pgmap v3149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:03 smithi161 ceph-mon[91579]: pgmap v3150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:03 smithi188 ceph-mon[92425]: pgmap v3150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:05 smithi161 ceph-mon[91579]: pgmap v3151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:05 smithi188 ceph-mon[92425]: pgmap v3151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:07 smithi161 ceph-mon[91579]: pgmap v3152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:07 smithi188 ceph-mon[92425]: pgmap v3152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:08 smithi161 ceph-mon[91579]: pgmap v3153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:08 smithi188 ceph-mon[92425]: pgmap v3153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:11 smithi161 ceph-mon[91579]: pgmap v3154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:11 smithi188 ceph-mon[92425]: pgmap v3154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:13 smithi161 ceph-mon[91579]: pgmap v3155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:13 smithi188 ceph-mon[92425]: pgmap v3155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:15 smithi161 ceph-mon[91579]: pgmap v3156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:15 smithi188 ceph-mon[92425]: pgmap v3156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:17 smithi161 ceph-mon[91579]: pgmap v3157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:17 smithi188 ceph-mon[92425]: pgmap v3157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:18 smithi161 ceph-mon[91579]: pgmap v3158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:18 smithi188 ceph-mon[92425]: pgmap v3158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:21 smithi161 ceph-mon[91579]: pgmap v3159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:21 smithi188 ceph-mon[92425]: pgmap v3159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:23 smithi161 ceph-mon[91579]: pgmap v3160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:23 smithi188 ceph-mon[92425]: pgmap v3160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:25 smithi161 ceph-mon[91579]: pgmap v3161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:25 smithi188 ceph-mon[92425]: pgmap v3161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:27 smithi161 ceph-mon[91579]: pgmap v3162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:27 smithi188 ceph-mon[92425]: pgmap v3162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:28 smithi161 ceph-mon[91579]: pgmap v3163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:28 smithi188 ceph-mon[92425]: pgmap v3163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:31 smithi161 ceph-mon[91579]: pgmap v3164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:31 smithi188 ceph-mon[92425]: pgmap v3164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:33 smithi161 ceph-mon[91579]: pgmap v3165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:33 smithi188 ceph-mon[92425]: pgmap v3165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:35.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:24:34.978+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:24:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:24:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:24:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:24:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:24:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:24:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:24:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:24:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:24:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:24:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:24:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:24:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:24:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:24:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:24:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:24:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:24:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:24:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:24:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:24:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:24:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:24:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:24:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=153] Local address: 172.21.15.161, port 52620 2022-07-03T12:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=153] Peer address: 172.21.15.101, port 22 2022-07-03T12:24:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=153] Connection lost 2022-07-03T12:24:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=153] Aborting connection 2022-07-03T12:24:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:24:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:24:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:35 smithi188 ceph-mon[92425]: pgmap v3166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:24:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:24:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:35 smithi161 ceph-mon[91579]: pgmap v3166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:24:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:24:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:24:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:24:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:24:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:24:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:24:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:24:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:24:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:24:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:24:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:24:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:24:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:24:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:24:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:24:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:24:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:24:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:24:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:24:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:24:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:24:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:24:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:24:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:24:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:24:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:24:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:24:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:24:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:24:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:24:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:24:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:24:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:24:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:24:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:24:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:24:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:24:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:24:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:24:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:24:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:24:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:24:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:24:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:24:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:24:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:24:36.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:24:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:24:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:24:36.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=153] Local address: 172.21.15.161, port 52620 2022-07-03T12:24:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=153] Peer address: 172.21.15.101, port 22 2022-07-03T12:24:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=153] Connection lost 2022-07-03T12:24:36.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=153] Aborting connection 2022-07-03T12:24:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:24:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:24:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:24:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:24:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:24:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:24:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:24:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:24:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:24:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:24:37.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:24:37.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:24:37.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:24:37.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:24:37.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:24:37.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:24:37.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:24:37.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:24:37.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:24:37.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:24:37.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:24:37.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:24:37.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:24:37.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:24:37.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:24:37.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:24:37.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:24:37.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:24:37.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:24:37.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:24:37.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:24:37.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:24:37.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:24:37.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:24:37.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:24:37.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:24:37.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:24:37.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:24:37.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:24:37.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:24:37.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:24:37.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:24:37.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:24:37.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:24:37.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:24:37.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:24:37.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:24:37.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=153] Local address: 172.21.15.161, port 52620 2022-07-03T12:24:37.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=153] Peer address: 172.21.15.101, port 22 2022-07-03T12:24:37.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=153] Connection lost 2022-07-03T12:24:37.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=153] Aborting connection 2022-07-03T12:24:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:37 smithi188 ceph-mon[92425]: pgmap v3167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:37 smithi161 ceph-mon[91579]: pgmap v3167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:38 smithi161 ceph-mon[91579]: pgmap v3168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:38 smithi188 ceph-mon[92425]: pgmap v3168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:41 smithi161 ceph-mon[91579]: pgmap v3169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:41 smithi188 ceph-mon[92425]: pgmap v3169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:43 smithi161 ceph-mon[91579]: pgmap v3170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:43 smithi188 ceph-mon[92425]: pgmap v3170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:45 smithi188 ceph-mon[92425]: pgmap v3171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:45 smithi161 ceph-mon[91579]: pgmap v3171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:47 smithi188 ceph-mon[92425]: pgmap v3172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:47 smithi161 ceph-mon[91579]: pgmap v3172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:48 smithi161 ceph-mon[91579]: pgmap v3173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:48 smithi188 ceph-mon[92425]: pgmap v3173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:51 smithi161 ceph-mon[91579]: pgmap v3174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:51 smithi188 ceph-mon[92425]: pgmap v3174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:53 smithi161 ceph-mon[91579]: pgmap v3175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:53 smithi188 ceph-mon[92425]: pgmap v3175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:55 smithi188 ceph-mon[92425]: pgmap v3176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:55 smithi161 ceph-mon[91579]: pgmap v3176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:24:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:57 smithi188 ceph-mon[92425]: pgmap v3177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:24:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:24:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:57 smithi161 ceph-mon[91579]: pgmap v3177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:24:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:24:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:58 smithi161 ceph-mon[91579]: pgmap v3178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:24:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:58 smithi188 ceph-mon[92425]: pgmap v3178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:24:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:01 smithi161 ceph-mon[91579]: pgmap v3179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:01 smithi188 ceph-mon[92425]: pgmap v3179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:03 smithi188 ceph-mon[92425]: pgmap v3180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:03 smithi161 ceph-mon[91579]: pgmap v3180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:05 smithi188 ceph-mon[92425]: pgmap v3181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:05 smithi161 ceph-mon[91579]: pgmap v3181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:07 smithi188 ceph-mon[92425]: pgmap v3182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:07 smithi161 ceph-mon[91579]: pgmap v3182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:08 smithi161 ceph-mon[91579]: pgmap v3183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:08 smithi188 ceph-mon[92425]: pgmap v3183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:11 smithi161 ceph-mon[91579]: pgmap v3184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:11 smithi188 ceph-mon[92425]: pgmap v3184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:13 smithi188 ceph-mon[92425]: pgmap v3185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:13 smithi161 ceph-mon[91579]: pgmap v3185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:15 smithi188 ceph-mon[92425]: pgmap v3186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:15 smithi161 ceph-mon[91579]: pgmap v3186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:17 smithi188 ceph-mon[92425]: pgmap v3187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:17 smithi161 ceph-mon[91579]: pgmap v3187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:18 smithi161 ceph-mon[91579]: pgmap v3188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:18 smithi188 ceph-mon[92425]: pgmap v3188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:21 smithi161 ceph-mon[91579]: pgmap v3189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:21 smithi188 ceph-mon[92425]: pgmap v3189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:23 smithi188 ceph-mon[92425]: pgmap v3190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:23 smithi161 ceph-mon[91579]: pgmap v3190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:25 smithi188 ceph-mon[92425]: pgmap v3191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:25 smithi161 ceph-mon[91579]: pgmap v3191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:27 smithi188 ceph-mon[92425]: pgmap v3192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:27 smithi161 ceph-mon[91579]: pgmap v3192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:28 smithi161 ceph-mon[91579]: pgmap v3193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:28 smithi188 ceph-mon[92425]: pgmap v3193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:31 smithi161 ceph-mon[91579]: pgmap v3194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:31 smithi188 ceph-mon[92425]: pgmap v3194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:33 smithi188 ceph-mon[92425]: pgmap v3195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:33 smithi161 ceph-mon[91579]: pgmap v3195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:35 smithi161 ceph-mon[91579]: pgmap v3196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:25:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:35 smithi188 ceph-mon[92425]: pgmap v3196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:25:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:25:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:37 smithi188 ceph-mon[92425]: pgmap v3197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:37 smithi161 ceph-mon[91579]: pgmap v3197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:38 smithi161 ceph-mon[91579]: pgmap v3198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:38 smithi188 ceph-mon[92425]: pgmap v3198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:41 smithi161 ceph-mon[91579]: pgmap v3199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:41 smithi188 ceph-mon[92425]: pgmap v3199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:43 smithi188 ceph-mon[92425]: pgmap v3200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:43 smithi161 ceph-mon[91579]: pgmap v3200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:45 smithi188 ceph-mon[92425]: pgmap v3201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:45 smithi161 ceph-mon[91579]: pgmap v3201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:47 smithi188 ceph-mon[92425]: pgmap v3202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:47 smithi161 ceph-mon[91579]: pgmap v3202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:48 smithi161 ceph-mon[91579]: pgmap v3203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:48 smithi188 ceph-mon[92425]: pgmap v3203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:51 smithi161 ceph-mon[91579]: pgmap v3204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:51 smithi188 ceph-mon[92425]: pgmap v3204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:53 smithi188 ceph-mon[92425]: pgmap v3205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:53 smithi161 ceph-mon[91579]: pgmap v3205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:55 smithi188 ceph-mon[92425]: pgmap v3206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:55 smithi161 ceph-mon[91579]: pgmap v3206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:25:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:57 smithi188 ceph-mon[92425]: pgmap v3207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:25:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:25:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:25:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:25:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:57 smithi161 ceph-mon[91579]: pgmap v3207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:25:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:25:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:25:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:25:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:25:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:58 smithi161 ceph-mon[91579]: pgmap v3208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:25:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:58 smithi188 ceph-mon[92425]: pgmap v3208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:25:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:01 smithi161 ceph-mon[91579]: pgmap v3209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:01 smithi188 ceph-mon[92425]: pgmap v3209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:03 smithi188 ceph-mon[92425]: pgmap v3210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:03 smithi161 ceph-mon[91579]: pgmap v3210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-mon[91579]: pgmap v3211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:26:05.473+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:26:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:26:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:26:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:26:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:26:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:26:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:26:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:26:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:26:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:26:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:26:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:26:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:26:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:26:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:26:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:26:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:26:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:26:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:26:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:26:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:26:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:26:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:26:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:26:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:26:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:26:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:26:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:26:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:26:05.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:26:05.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:26:05.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:26:05.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:26:05.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:26:05.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:26:05.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:26:05.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:26:05.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:26:05.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:26:05.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:26:05.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:26:05.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:26:05.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:26:05.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:26:05.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:26:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:26:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=155] Local address: 172.21.15.161, port 52648 2022-07-03T12:26:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=155] Peer address: 172.21.15.101, port 22 2022-07-03T12:26:05.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=460] Received exit status 0 2022-07-03T12:26:05.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=460] Received channel close 2022-07-03T12:26:05.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=460] Channel closed 2022-07-03T12:26:05.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=460] Received exit status 0 2022-07-03T12:26:05.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=460] Received channel close 2022-07-03T12:26:05.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=460] Channel closed 2022-07-03T12:26:05.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=461] Requesting new SSH session 2022-07-03T12:26:05.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=461] Requesting new SSH session 2022-07-03T12:26:05.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=461] Command: which python3 2022-07-03T12:26:05.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=461] Command: which python3 2022-07-03T12:26:05.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=461] Received exit status 0 2022-07-03T12:26:05.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=461] Received channel close 2022-07-03T12:26:05.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=461] Channel closed 2022-07-03T12:26:05.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=461] Received exit status 0 2022-07-03T12:26:05.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=461] Received channel close 2022-07-03T12:26:05.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=462] Requesting new SSH session 2022-07-03T12:26:05.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=461] Channel closed 2022-07-03T12:26:05.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=462] Requesting new SSH session 2022-07-03T12:26:05.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=462] Command: sudo true 2022-07-03T12:26:05.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=462] Command: sudo true 2022-07-03T12:26:05.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=462] Received exit status 0 2022-07-03T12:26:05.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=462] Received channel close 2022-07-03T12:26:05.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=462] Channel closed 2022-07-03T12:26:05.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=463] Requesting new SSH session 2022-07-03T12:26:05.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:05.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=462] Received exit status 0 2022-07-03T12:26:05.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=462] Received channel close 2022-07-03T12:26:05.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=462] Channel closed 2022-07-03T12:26:05.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=463] Requesting new SSH session 2022-07-03T12:26:05.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:05.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=463] Received exit status 0 2022-07-03T12:26:05.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=463] Received channel close 2022-07-03T12:26:05.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=463] Channel closed 2022-07-03T12:26:05.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=463] Received exit status 0 2022-07-03T12:26:05.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=463] Received channel close 2022-07-03T12:26:05.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=463] Channel closed 2022-07-03T12:26:05.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=155] Login timeout expired 2022-07-03T12:26:05.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=155] Aborting connection 2022-07-03T12:26:05.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:26:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:26:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:05 smithi188 ceph-mon[92425]: pgmap v3211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:26:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:26:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:26:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:26:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:26:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:26:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:26:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:26:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:26:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:26:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:26:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:26:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:26:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:26:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:26:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:26:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:26:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:26:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:26:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:26:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:26:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:26:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:26:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:26:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:26:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:26:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:26:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:26:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:26:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:26:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:26:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:26:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:26:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:26:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:26:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:26:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:26:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:26:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:26:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:26:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:26:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:26:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:26:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:26:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:26:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:26:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=155] Local address: 172.21.15.161, port 52648 2022-07-03T12:26:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=155] Peer address: 172.21.15.101, port 22 2022-07-03T12:26:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=460] Received exit status 0 2022-07-03T12:26:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=460] Received channel close 2022-07-03T12:26:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=460] Channel closed 2022-07-03T12:26:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=460] Received exit status 0 2022-07-03T12:26:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=460] Received channel close 2022-07-03T12:26:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=460] Channel closed 2022-07-03T12:26:06.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=461] Requesting new SSH session 2022-07-03T12:26:06.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=461] Requesting new SSH session 2022-07-03T12:26:06.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=461] Command: which python3 2022-07-03T12:26:06.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=461] Command: which python3 2022-07-03T12:26:06.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=461] Received exit status 0 2022-07-03T12:26:06.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=461] Received channel close 2022-07-03T12:26:06.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=461] Channel closed 2022-07-03T12:26:06.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=461] Received exit status 0 2022-07-03T12:26:06.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=461] Received channel close 2022-07-03T12:26:06.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=462] Requesting new SSH session 2022-07-03T12:26:06.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=461] Channel closed 2022-07-03T12:26:06.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=462] Requesting new SSH session 2022-07-03T12:26:06.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=462] Command: sudo true 2022-07-03T12:26:06.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=462] Command: sudo true 2022-07-03T12:26:06.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=462] Received exit status 0 2022-07-03T12:26:06.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=462] Received channel close 2022-07-03T12:26:06.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=462] Channel closed 2022-07-03T12:26:06.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=463] Requesting new SSH session 2022-07-03T12:26:06.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:06.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=462] Received exit status 0 2022-07-03T12:26:06.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=462] Received channel close 2022-07-03T12:26:06.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=462] Channel closed 2022-07-03T12:26:06.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=463] Requesting new SSH session 2022-07-03T12:26:06.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:06.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=463] Received exit status 0 2022-07-03T12:26:06.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=463] Received channel close 2022-07-03T12:26:06.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=463] Channel closed 2022-07-03T12:26:06.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=463] Received exit status 0 2022-07-03T12:26:06.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=463] Received channel close 2022-07-03T12:26:06.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=463] Channel closed 2022-07-03T12:26:06.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=155] Login timeout expired 2022-07-03T12:26:06.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=155] Aborting connection 2022-07-03T12:26:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:26:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:26:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:26:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:26:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:26:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:26:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:26:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:26:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:26:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:26:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:26:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:26:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:26:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:26:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:26:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:26:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:26:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:26:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:26:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:26:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:26:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:26:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:26:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:26:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:26:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:26:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:26:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:26:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:26:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:26:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:26:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:26:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:26:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:26:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:26:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:26:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:26:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:26:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:26:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:26:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:26:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:26:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:26:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:26:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:26:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:26:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=155] Local address: 172.21.15.161, port 52648 2022-07-03T12:26:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=155] Peer address: 172.21.15.101, port 22 2022-07-03T12:26:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=460] Received exit status 0 2022-07-03T12:26:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=460] Received channel close 2022-07-03T12:26:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=460] Channel closed 2022-07-03T12:26:07.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=460] Received exit status 0 2022-07-03T12:26:07.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=460] Received channel close 2022-07-03T12:26:07.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=460] Channel closed 2022-07-03T12:26:07.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=461] Requesting new SSH session 2022-07-03T12:26:07.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=461] Requesting new SSH session 2022-07-03T12:26:07.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=461] Command: which python3 2022-07-03T12:26:07.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=461] Command: which python3 2022-07-03T12:26:07.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=461] Received exit status 0 2022-07-03T12:26:07.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=461] Received channel close 2022-07-03T12:26:07.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=461] Channel closed 2022-07-03T12:26:07.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=461] Received exit status 0 2022-07-03T12:26:07.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=461] Received channel close 2022-07-03T12:26:07.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=462] Requesting new SSH session 2022-07-03T12:26:07.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=461] Channel closed 2022-07-03T12:26:07.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=462] Requesting new SSH session 2022-07-03T12:26:07.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=462] Command: sudo true 2022-07-03T12:26:07.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=462] Command: sudo true 2022-07-03T12:26:07.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=462] Received exit status 0 2022-07-03T12:26:07.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=462] Received channel close 2022-07-03T12:26:07.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=462] Channel closed 2022-07-03T12:26:07.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=463] Requesting new SSH session 2022-07-03T12:26:07.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:07.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=462] Received exit status 0 2022-07-03T12:26:07.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=462] Received channel close 2022-07-03T12:26:07.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=462] Channel closed 2022-07-03T12:26:07.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=463] Requesting new SSH session 2022-07-03T12:26:07.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=463] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:26:07.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=463] Received exit status 0 2022-07-03T12:26:07.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=463] Received channel close 2022-07-03T12:26:07.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=463] Channel closed 2022-07-03T12:26:07.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=463] Received exit status 0 2022-07-03T12:26:07.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=463] Received channel close 2022-07-03T12:26:07.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=463] Channel closed 2022-07-03T12:26:07.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=155] Login timeout expired 2022-07-03T12:26:07.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=155] Aborting connection 2022-07-03T12:26:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:07 smithi161 ceph-mon[91579]: pgmap v3212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:07 smithi188 ceph-mon[92425]: pgmap v3212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:08 smithi161 ceph-mon[91579]: pgmap v3213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:08 smithi188 ceph-mon[92425]: pgmap v3213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:11 smithi161 ceph-mon[91579]: pgmap v3214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:11 smithi188 ceph-mon[92425]: pgmap v3214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:13 smithi188 ceph-mon[92425]: pgmap v3215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:13 smithi161 ceph-mon[91579]: pgmap v3215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:15 smithi188 ceph-mon[92425]: pgmap v3216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:15 smithi161 ceph-mon[91579]: pgmap v3216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:17 smithi188 ceph-mon[92425]: pgmap v3217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:17 smithi161 ceph-mon[91579]: pgmap v3217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:18 smithi161 ceph-mon[91579]: pgmap v3218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:19 smithi188 ceph-mon[92425]: pgmap v3218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:21 smithi161 ceph-mon[91579]: pgmap v3219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:21 smithi188 ceph-mon[92425]: pgmap v3219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:23 smithi188 ceph-mon[92425]: pgmap v3220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:23 smithi161 ceph-mon[91579]: pgmap v3220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:25 smithi188 ceph-mon[92425]: pgmap v3221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:25 smithi161 ceph-mon[91579]: pgmap v3221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:27 smithi161 ceph-mon[91579]: pgmap v3222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:27 smithi188 ceph-mon[92425]: pgmap v3222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:29 smithi161 ceph-mon[91579]: pgmap v3223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:29 smithi188 ceph-mon[92425]: pgmap v3223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:31 smithi161 ceph-mon[91579]: pgmap v3224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:31 smithi188 ceph-mon[92425]: pgmap v3224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:33 smithi188 ceph-mon[92425]: pgmap v3225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:33 smithi161 ceph-mon[91579]: pgmap v3225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:35 smithi161 ceph-mon[91579]: pgmap v3226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:35 smithi188 ceph-mon[92425]: pgmap v3226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:37 smithi161 ceph-mon[91579]: pgmap v3227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:37 smithi188 ceph-mon[92425]: pgmap v3227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:39 smithi161 ceph-mon[91579]: pgmap v3228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:39 smithi188 ceph-mon[92425]: pgmap v3228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:41 smithi161 ceph-mon[91579]: pgmap v3229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:41 smithi188 ceph-mon[92425]: pgmap v3229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:43 smithi188 ceph-mon[92425]: pgmap v3230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:43 smithi161 ceph-mon[91579]: pgmap v3230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:45 smithi161 ceph-mon[91579]: pgmap v3231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:45 smithi188 ceph-mon[92425]: pgmap v3231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:47 smithi161 ceph-mon[91579]: pgmap v3232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:47 smithi188 ceph-mon[92425]: pgmap v3232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:49 smithi161 ceph-mon[91579]: pgmap v3233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:49 smithi188 ceph-mon[92425]: pgmap v3233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:51 smithi161 ceph-mon[91579]: pgmap v3234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:51 smithi188 ceph-mon[92425]: pgmap v3234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:53 smithi161 ceph-mon[91579]: pgmap v3235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:53 smithi188 ceph-mon[92425]: pgmap v3235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:55 smithi161 ceph-mon[91579]: pgmap v3236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:55 smithi188 ceph-mon[92425]: pgmap v3236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:26:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:57 smithi161 ceph-mon[91579]: pgmap v3237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:26:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:26:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:26:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:26:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:57 smithi188 ceph-mon[92425]: pgmap v3237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:26:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:26:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:59 smithi161 ceph-mon[91579]: pgmap v3238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:26:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:59 smithi188 ceph-mon[92425]: pgmap v3238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:26:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:01 smithi161 ceph-mon[91579]: pgmap v3239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:01 smithi188 ceph-mon[92425]: pgmap v3239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:03 smithi161 ceph-mon[91579]: pgmap v3240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:03 smithi188 ceph-mon[92425]: pgmap v3240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-mon[91579]: pgmap v3241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:27:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:27:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:27:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:27:05.510+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:27:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:27:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:27:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:27:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:27:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:27:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:27:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:27:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:27:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:27:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:27:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:27:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:27:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:27:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:27:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:27:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:27:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:27:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:27:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:27:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:27:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:27:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:27:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:27:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:27:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:27:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:27:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:27:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:27:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:27:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:27:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:27:05.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:27:05.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:27:05.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:27:05.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:27:05.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:27:05.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:27:05.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:27:05.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:27:05.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:27:05.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:27:05.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:27:05.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:27:05.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:27:05.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:27:05.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=157] Local address: 172.21.15.161, port 52688 2022-07-03T12:27:05.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=157] Peer address: 172.21.15.101, port 22 2022-07-03T12:27:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=157] Connection lost 2022-07-03T12:27:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=157] Aborting connection 2022-07-03T12:27:05.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:27:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:27:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:05 smithi188 ceph-mon[92425]: pgmap v3241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:27:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:27:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:27:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:27:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:27:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:27:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:27:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:27:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:27:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:27:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:27:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:27:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:27:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:27:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:27:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:27:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:27:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:27:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:27:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:27:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:27:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:27:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:27:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:27:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:27:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:27:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:27:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:27:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:27:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:27:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:27:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:27:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:27:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:27:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:27:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:27:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:27:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:27:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:27:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:27:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:27:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:27:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:27:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:27:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:27:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:27:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:27:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:27:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:27:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:27:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=157] Local address: 172.21.15.161, port 52688 2022-07-03T12:27:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=157] Peer address: 172.21.15.101, port 22 2022-07-03T12:27:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=157] Connection lost 2022-07-03T12:27:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=157] Aborting connection 2022-07-03T12:27:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.154 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:27:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:27:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:27:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:27:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:27:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:27:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:27:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:27:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:27:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:27:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:27:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:27:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:27:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:27:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:27:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:27:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:27:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:27:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:27:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:27:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:27:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:27:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:27:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:27:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:27:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:27:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:27:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:27:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:27:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:27:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:27:07.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:27:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:27:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:27:07.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:27:07.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:27:07.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:27:07.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:27:07.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:27:07.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:27:07.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:27:07.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:27:07.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:27:07.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:27:07.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:27:07.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=157] Local address: 172.21.15.161, port 52688 2022-07-03T12:27:07.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=157] Peer address: 172.21.15.101, port 22 2022-07-03T12:27:07.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=157] Connection lost 2022-07-03T12:27:07.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=157] Aborting connection 2022-07-03T12:27:07.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.196 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:07.197 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:07 smithi161 ceph-mon[91579]: pgmap v3242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:07 smithi188 ceph-mon[92425]: pgmap v3242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: pgmap v3243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.890 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.891 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.891 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:27:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: pgmap v3243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.925 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:09.926 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:27:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:27:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:27:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:27:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:27:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:27:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:27:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:27:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:27:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:11 smithi161 ceph-mon[91579]: pgmap v3244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:11 smithi188 ceph-mon[92425]: pgmap v3244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:13 smithi161 ceph-mon[91579]: pgmap v3245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:13 smithi188 ceph-mon[92425]: pgmap v3245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:15 smithi161 ceph-mon[91579]: pgmap v3246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:15 smithi188 ceph-mon[92425]: pgmap v3246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:17 smithi161 ceph-mon[91579]: pgmap v3247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:17 smithi188 ceph-mon[92425]: pgmap v3247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:19 smithi161 ceph-mon[91579]: pgmap v3248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:19 smithi188 ceph-mon[92425]: pgmap v3248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:21 smithi161 ceph-mon[91579]: pgmap v3249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:21 smithi188 ceph-mon[92425]: pgmap v3249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:23 smithi161 ceph-mon[91579]: pgmap v3250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:23 smithi188 ceph-mon[92425]: pgmap v3250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:25 smithi161 ceph-mon[91579]: pgmap v3251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:25 smithi188 ceph-mon[92425]: pgmap v3251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:27 smithi161 ceph-mon[91579]: pgmap v3252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:27 smithi188 ceph-mon[92425]: pgmap v3252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:29 smithi161 ceph-mon[91579]: pgmap v3253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:29 smithi188 ceph-mon[92425]: pgmap v3253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:31 smithi161 ceph-mon[91579]: pgmap v3254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:31 smithi188 ceph-mon[92425]: pgmap v3254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:33 smithi161 ceph-mon[91579]: pgmap v3255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:33 smithi188 ceph-mon[92425]: pgmap v3255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:35.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:35 smithi161 ceph-mon[91579]: pgmap v3256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:35 smithi188 ceph-mon[92425]: pgmap v3256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:37 smithi161 ceph-mon[91579]: pgmap v3257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:37 smithi188 ceph-mon[92425]: pgmap v3257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:39 smithi161 ceph-mon[91579]: pgmap v3258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:39 smithi188 ceph-mon[92425]: pgmap v3258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:41 smithi161 ceph-mon[91579]: pgmap v3259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:41 smithi188 ceph-mon[92425]: pgmap v3259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:43 smithi161 ceph-mon[91579]: pgmap v3260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:43 smithi188 ceph-mon[92425]: pgmap v3260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:45 smithi161 ceph-mon[91579]: pgmap v3261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:45 smithi188 ceph-mon[92425]: pgmap v3261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:47 smithi161 ceph-mon[91579]: pgmap v3262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:47 smithi188 ceph-mon[92425]: pgmap v3262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:49 smithi161 ceph-mon[91579]: pgmap v3263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:49 smithi188 ceph-mon[92425]: pgmap v3263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:51 smithi161 ceph-mon[91579]: pgmap v3264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:51 smithi188 ceph-mon[92425]: pgmap v3264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:53 smithi161 ceph-mon[91579]: pgmap v3265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:53 smithi188 ceph-mon[92425]: pgmap v3265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:55 smithi161 ceph-mon[91579]: pgmap v3266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:55 smithi188 ceph-mon[92425]: pgmap v3266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:27:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:57 smithi161 ceph-mon[91579]: pgmap v3267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:27:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:27:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:27:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:27:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:57 smithi188 ceph-mon[92425]: pgmap v3267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:27:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:27:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:27:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:27:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:59 smithi161 ceph-mon[91579]: pgmap v3268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:27:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:59 smithi188 ceph-mon[92425]: pgmap v3268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:27:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:01 smithi188 ceph-mon[92425]: pgmap v3269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:01 smithi161 ceph-mon[91579]: pgmap v3269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:03 smithi161 ceph-mon[91579]: pgmap v3270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:03 smithi188 ceph-mon[92425]: pgmap v3270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:05 smithi161 ceph-mon[91579]: pgmap v3271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:05 smithi188 ceph-mon[92425]: pgmap v3271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:07 smithi161 ceph-mon[91579]: pgmap v3272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:07 smithi188 ceph-mon[92425]: pgmap v3272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-mon[91579]: pgmap v3273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:09.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:28:09.108+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:28:09.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:28:09.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:28:09.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:28:09.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:28:09.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:28:09.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:28:09.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:28:09.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:28:09.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:28:09.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:28:09.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:28:09.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:28:09.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:28:09.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:28:09.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:28:09.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:28:09.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:28:09.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:28:09.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:28:09.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:28:09.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:28:09.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:28:09.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:28:09.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:28:09.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:28:09.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:28:09.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:28:09.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:28:09.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:28:09.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:28:09.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:28:09.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:28:09.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:28:09.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:28:09.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:28:09.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:28:09.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:28:09.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:28:09.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:28:09.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:28:09.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:28:09.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:28:09.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:28:09.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:28:09.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:28:09.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=159] Local address: 172.21.15.161, port 52718 2022-07-03T12:28:09.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=159] Peer address: 172.21.15.101, port 22 2022-07-03T12:28:09.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=159] Connection lost 2022-07-03T12:28:09.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=159] Aborting connection 2022-07-03T12:28:09.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:28:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:28:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:09 smithi188 ceph-mon[92425]: pgmap v3273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:28:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:28:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:28:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:28:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:28:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:28:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:28:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:28:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:28:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:28:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:28:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:28:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:28:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:28:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:28:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:28:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:28:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:28:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:28:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:28:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:28:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:28:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:28:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:28:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:28:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:28:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:28:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:28:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:28:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:28:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:28:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:28:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:28:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:28:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:28:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:28:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:28:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:28:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:28:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:28:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:28:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:28:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:28:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:28:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:28:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:28:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:28:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:28:10.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:28:10.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:28:10.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=159] Local address: 172.21.15.161, port 52718 2022-07-03T12:28:10.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=159] Peer address: 172.21.15.101, port 22 2022-07-03T12:28:10.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=159] Connection lost 2022-07-03T12:28:10.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=159] Aborting connection 2022-07-03T12:28:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:28:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:28:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:28:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:28:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:28:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:28:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:28:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:28:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:28:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:28:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:28:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:28:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:28:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:28:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:28:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:28:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:28:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:28:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:28:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:28:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:28:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:28:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:28:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:28:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:28:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:28:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:28:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:28:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:28:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:28:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:28:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:28:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:28:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:28:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:28:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:28:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:28:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:28:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:28:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:28:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:28:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:28:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:28:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:28:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:28:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:28:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:28:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:28:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:28:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:28:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=159] Local address: 172.21.15.161, port 52718 2022-07-03T12:28:10.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=159] Peer address: 172.21.15.101, port 22 2022-07-03T12:28:10.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=159] Connection lost 2022-07-03T12:28:10.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=159] Aborting connection 2022-07-03T12:28:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:11 smithi161 ceph-mon[91579]: pgmap v3274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:11 smithi188 ceph-mon[92425]: pgmap v3274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:13 smithi161 ceph-mon[91579]: pgmap v3275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:13 smithi188 ceph-mon[92425]: pgmap v3275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:15 smithi161 ceph-mon[91579]: pgmap v3276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:15 smithi188 ceph-mon[92425]: pgmap v3276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:17 smithi161 ceph-mon[91579]: pgmap v3277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:17 smithi188 ceph-mon[92425]: pgmap v3277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:19 smithi161 ceph-mon[91579]: pgmap v3278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:19 smithi188 ceph-mon[92425]: pgmap v3278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:20.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:21 smithi188 ceph-mon[92425]: pgmap v3279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:21 smithi161 ceph-mon[91579]: pgmap v3279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:23 smithi188 ceph-mon[92425]: pgmap v3280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:23 smithi161 ceph-mon[91579]: pgmap v3280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:25 smithi188 ceph-mon[92425]: pgmap v3281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:25 smithi161 ceph-mon[91579]: pgmap v3281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:27 smithi188 ceph-mon[92425]: pgmap v3282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:27 smithi161 ceph-mon[91579]: pgmap v3282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:29 smithi161 ceph-mon[91579]: pgmap v3283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:29 smithi188 ceph-mon[92425]: pgmap v3283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:31 smithi161 ceph-mon[91579]: pgmap v3284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:31 smithi188 ceph-mon[92425]: pgmap v3284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:33 smithi161 ceph-mon[91579]: pgmap v3285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:33 smithi188 ceph-mon[92425]: pgmap v3285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:35 smithi161 ceph-mon[91579]: pgmap v3286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:35 smithi188 ceph-mon[92425]: pgmap v3286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:37 smithi161 ceph-mon[91579]: pgmap v3287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:37 smithi188 ceph-mon[92425]: pgmap v3287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:39 smithi161 ceph-mon[91579]: pgmap v3288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:39 smithi188 ceph-mon[92425]: pgmap v3288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:41 smithi161 ceph-mon[91579]: pgmap v3289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:41 smithi188 ceph-mon[92425]: pgmap v3289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:43 smithi161 ceph-mon[91579]: pgmap v3290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:43 smithi188 ceph-mon[92425]: pgmap v3290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:45 smithi161 ceph-mon[91579]: pgmap v3291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:45 smithi188 ceph-mon[92425]: pgmap v3291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:47 smithi161 ceph-mon[91579]: pgmap v3292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:47 smithi188 ceph-mon[92425]: pgmap v3292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:49 smithi161 ceph-mon[91579]: pgmap v3293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:49 smithi188 ceph-mon[92425]: pgmap v3293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:52.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:51 smithi161 ceph-mon[91579]: pgmap v3294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:51 smithi188 ceph-mon[92425]: pgmap v3294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:53 smithi161 ceph-mon[91579]: pgmap v3295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:53 smithi188 ceph-mon[92425]: pgmap v3295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:55 smithi161 ceph-mon[91579]: pgmap v3296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:55 smithi188 ceph-mon[92425]: pgmap v3296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:28:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:57 smithi161 ceph-mon[91579]: pgmap v3297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:28:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:28:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:28:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:28:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:57 smithi188 ceph-mon[92425]: pgmap v3297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:28:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:28:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:28:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:28:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:28:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:59 smithi161 ceph-mon[91579]: pgmap v3298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:28:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:59 smithi188 ceph-mon[92425]: pgmap v3298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:28:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:01 smithi161 ceph-mon[91579]: pgmap v3299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:01 smithi188 ceph-mon[92425]: pgmap v3299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:03 smithi161 ceph-mon[91579]: pgmap v3300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:03 smithi188 ceph-mon[92425]: pgmap v3300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:05 smithi161 ceph-mon[91579]: pgmap v3301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:05 smithi188 ceph-mon[92425]: pgmap v3301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:07 smithi161 ceph-mon[91579]: pgmap v3302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:07 smithi188 ceph-mon[92425]: pgmap v3302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:09 smithi161 ceph-mon[91579]: pgmap v3303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:09 smithi188 ceph-mon[92425]: pgmap v3303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:29:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:29:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:29:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:29:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:29:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:29:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:29:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:29:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:11 smithi161 ceph-mon[91579]: pgmap v3304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:11 smithi188 ceph-mon[92425]: pgmap v3304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:13 smithi161 ceph-mon[91579]: pgmap v3305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:13 smithi188 ceph-mon[92425]: pgmap v3305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:15 smithi161 ceph-mon[91579]: pgmap v3306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:15 smithi188 ceph-mon[92425]: pgmap v3306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:17 smithi161 ceph-mon[91579]: pgmap v3307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:17 smithi188 ceph-mon[92425]: pgmap v3307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:19 smithi161 ceph-mon[91579]: pgmap v3308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:19 smithi188 ceph-mon[92425]: pgmap v3308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:21 smithi161 ceph-mon[91579]: pgmap v3309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:21 smithi188 ceph-mon[92425]: pgmap v3309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:24.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:23 smithi161 ceph-mon[91579]: pgmap v3310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:23 smithi188 ceph-mon[92425]: pgmap v3310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:25 smithi161 ceph-mon[91579]: pgmap v3311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:25 smithi188 ceph-mon[92425]: pgmap v3311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:27 smithi161 ceph-mon[91579]: pgmap v3312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:27 smithi188 ceph-mon[92425]: pgmap v3312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:29 smithi161 ceph-mon[91579]: pgmap v3313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:29 smithi188 ceph-mon[92425]: pgmap v3313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:31 smithi161 ceph-mon[91579]: pgmap v3314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:32.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:31 smithi188 ceph-mon[92425]: pgmap v3314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:33 smithi161 ceph-mon[91579]: pgmap v3315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:33 smithi188 ceph-mon[92425]: pgmap v3315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:35 smithi161 ceph-mon[91579]: pgmap v3316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:35 smithi188 ceph-mon[92425]: pgmap v3316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:37 smithi161 ceph-mon[91579]: pgmap v3317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:37 smithi188 ceph-mon[92425]: pgmap v3317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-mon[91579]: pgmap v3318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:39 smithi188 ceph-mon[92425]: pgmap v3318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:39.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:29:39.612+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:29:39.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:29:39.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:29:39.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:29:39.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:29:39.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:29:39.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:29:39.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:29:39.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:29:39.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:29:39.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:29:39.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:29:39.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:29:39.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:29:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:29:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:29:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:29:39.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:29:39.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:29:39.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:29:39.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:29:39.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:29:39.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:29:39.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:29:39.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:29:39.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:29:39.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:29:39.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:29:39.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:29:39.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:29:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:29:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:29:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:29:39.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:29:39.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:29:39.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:29:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:29:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:29:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:29:39.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:29:39.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:29:39.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:29:39.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:29:39.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:29:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:29:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:29:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=161] Local address: 172.21.15.161, port 52746 2022-07-03T12:29:39.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=161] Peer address: 172.21.15.101, port 22 2022-07-03T12:29:39.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=480] Received exit status 0 2022-07-03T12:29:39.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=480] Received channel close 2022-07-03T12:29:39.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=480] Channel closed 2022-07-03T12:29:39.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=481] Requesting new SSH session 2022-07-03T12:29:39.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=481] Command: which python3 2022-07-03T12:29:39.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=480] Received exit status 0 2022-07-03T12:29:39.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=480] Received channel close 2022-07-03T12:29:39.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=480] Channel closed 2022-07-03T12:29:39.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=481] Requesting new SSH session 2022-07-03T12:29:39.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=481] Command: which python3 2022-07-03T12:29:39.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=481] Received exit status 0 2022-07-03T12:29:39.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=481] Received channel close 2022-07-03T12:29:39.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=481] Channel closed 2022-07-03T12:29:39.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=482] Requesting new SSH session 2022-07-03T12:29:39.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=482] Command: sudo true 2022-07-03T12:29:39.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=481] Received exit status 0 2022-07-03T12:29:39.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=481] Received channel close 2022-07-03T12:29:39.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=481] Channel closed 2022-07-03T12:29:39.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=482] Requesting new SSH session 2022-07-03T12:29:39.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=482] Command: sudo true 2022-07-03T12:29:39.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=482] Received exit status 0 2022-07-03T12:29:39.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=482] Received channel close 2022-07-03T12:29:39.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=482] Channel closed 2022-07-03T12:29:39.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=483] Requesting new SSH session 2022-07-03T12:29:39.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:39.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=482] Received exit status 0 2022-07-03T12:29:39.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=482] Received channel close 2022-07-03T12:29:39.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=482] Channel closed 2022-07-03T12:29:39.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=483] Requesting new SSH session 2022-07-03T12:29:39.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:39.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=483] Received exit status 0 2022-07-03T12:29:39.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=483] Received channel close 2022-07-03T12:29:39.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=483] Channel closed 2022-07-03T12:29:39.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=483] Received exit status 0 2022-07-03T12:29:39.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=483] Received channel close 2022-07-03T12:29:39.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=483] Channel closed 2022-07-03T12:29:39.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=161] Login timeout expired 2022-07-03T12:29:39.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=161] Aborting connection 2022-07-03T12:29:39.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:29:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:29:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:29:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:29:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:29:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:29:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:29:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:29:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:29:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:29:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:29:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:29:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:29:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:29:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:29:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:29:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:29:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:29:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:29:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:29:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:29:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:29:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:29:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:29:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:29:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:29:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:29:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:29:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:29:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:29:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:29:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:29:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:29:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:29:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:29:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:29:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:29:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:29:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:29:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:29:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:29:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:29:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:29:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:29:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:29:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:29:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:29:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:29:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=161] Local address: 172.21.15.161, port 52746 2022-07-03T12:29:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=161] Peer address: 172.21.15.101, port 22 2022-07-03T12:29:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=480] Received exit status 0 2022-07-03T12:29:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=480] Received channel close 2022-07-03T12:29:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=480] Channel closed 2022-07-03T12:29:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=481] Requesting new SSH session 2022-07-03T12:29:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=481] Command: which python3 2022-07-03T12:29:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=480] Received exit status 0 2022-07-03T12:29:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=480] Received channel close 2022-07-03T12:29:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=480] Channel closed 2022-07-03T12:29:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=481] Requesting new SSH session 2022-07-03T12:29:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=481] Command: which python3 2022-07-03T12:29:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=481] Received exit status 0 2022-07-03T12:29:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=481] Received channel close 2022-07-03T12:29:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=481] Channel closed 2022-07-03T12:29:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=482] Requesting new SSH session 2022-07-03T12:29:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=482] Command: sudo true 2022-07-03T12:29:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=481] Received exit status 0 2022-07-03T12:29:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=481] Received channel close 2022-07-03T12:29:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=481] Channel closed 2022-07-03T12:29:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=482] Requesting new SSH session 2022-07-03T12:29:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=482] Command: sudo true 2022-07-03T12:29:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=482] Received exit status 0 2022-07-03T12:29:40.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=482] Received channel close 2022-07-03T12:29:40.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=482] Channel closed 2022-07-03T12:29:40.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=483] Requesting new SSH session 2022-07-03T12:29:40.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:40.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=482] Received exit status 0 2022-07-03T12:29:40.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=482] Received channel close 2022-07-03T12:29:40.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=482] Channel closed 2022-07-03T12:29:40.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=483] Requesting new SSH session 2022-07-03T12:29:40.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:40.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=483] Received exit status 0 2022-07-03T12:29:40.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=483] Received channel close 2022-07-03T12:29:40.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=483] Channel closed 2022-07-03T12:29:40.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=483] Received exit status 0 2022-07-03T12:29:40.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=483] Received channel close 2022-07-03T12:29:40.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=483] Channel closed 2022-07-03T12:29:40.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=161] Login timeout expired 2022-07-03T12:29:40.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=161] Aborting connection 2022-07-03T12:29:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:29:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:29:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:29:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:29:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:29:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:29:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:29:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:29:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:29:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:29:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:29:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:29:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:29:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:29:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:29:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:29:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:29:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:29:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:29:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:29:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:29:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:29:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:29:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:29:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:29:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:29:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:29:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:29:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:29:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:29:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:29:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:29:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:29:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:29:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:29:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:29:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:29:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:29:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:29:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:29:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:29:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:29:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:29:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:29:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:29:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:29:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=161] Local address: 172.21.15.161, port 52746 2022-07-03T12:29:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=161] Peer address: 172.21.15.101, port 22 2022-07-03T12:29:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=480] Received exit status 0 2022-07-03T12:29:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=480] Received channel close 2022-07-03T12:29:40.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=480] Channel closed 2022-07-03T12:29:40.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=481] Requesting new SSH session 2022-07-03T12:29:40.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=481] Command: which python3 2022-07-03T12:29:40.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=480] Received exit status 0 2022-07-03T12:29:40.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=480] Received channel close 2022-07-03T12:29:40.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=480] Channel closed 2022-07-03T12:29:40.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=481] Requesting new SSH session 2022-07-03T12:29:40.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=481] Command: which python3 2022-07-03T12:29:40.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=481] Received exit status 0 2022-07-03T12:29:40.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=481] Received channel close 2022-07-03T12:29:40.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=481] Channel closed 2022-07-03T12:29:40.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=482] Requesting new SSH session 2022-07-03T12:29:40.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=482] Command: sudo true 2022-07-03T12:29:40.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=481] Received exit status 0 2022-07-03T12:29:40.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=481] Received channel close 2022-07-03T12:29:40.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=481] Channel closed 2022-07-03T12:29:40.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=482] Requesting new SSH session 2022-07-03T12:29:40.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=482] Command: sudo true 2022-07-03T12:29:40.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=482] Received exit status 0 2022-07-03T12:29:40.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=482] Received channel close 2022-07-03T12:29:40.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=482] Channel closed 2022-07-03T12:29:40.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=483] Requesting new SSH session 2022-07-03T12:29:40.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:40.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=482] Received exit status 0 2022-07-03T12:29:40.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=482] Received channel close 2022-07-03T12:29:40.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=482] Channel closed 2022-07-03T12:29:40.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=483] Requesting new SSH session 2022-07-03T12:29:40.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=483] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:29:40.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=483] Received exit status 0 2022-07-03T12:29:40.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=483] Received channel close 2022-07-03T12:29:40.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=483] Channel closed 2022-07-03T12:29:40.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=483] Received exit status 0 2022-07-03T12:29:40.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=483] Received channel close 2022-07-03T12:29:40.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=483] Channel closed 2022-07-03T12:29:40.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=161] Login timeout expired 2022-07-03T12:29:40.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=161] Aborting connection 2022-07-03T12:29:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:41 smithi161 ceph-mon[91579]: pgmap v3319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:41 smithi188 ceph-mon[92425]: pgmap v3319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:43 smithi161 ceph-mon[91579]: pgmap v3320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:43 smithi188 ceph-mon[92425]: pgmap v3320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:45 smithi161 ceph-mon[91579]: pgmap v3321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:45 smithi188 ceph-mon[92425]: pgmap v3321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:47 smithi161 ceph-mon[91579]: pgmap v3322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:47 smithi188 ceph-mon[92425]: pgmap v3322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:49 smithi161 ceph-mon[91579]: pgmap v3323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:49 smithi188 ceph-mon[92425]: pgmap v3323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:51 smithi161 ceph-mon[91579]: pgmap v3324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:51 smithi188 ceph-mon[92425]: pgmap v3324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:53 smithi161 ceph-mon[91579]: pgmap v3325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:53 smithi188 ceph-mon[92425]: pgmap v3325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:55 smithi161 ceph-mon[91579]: pgmap v3326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:55 smithi188 ceph-mon[92425]: pgmap v3326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:29:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:57 smithi161 ceph-mon[91579]: pgmap v3327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:29:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:29:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:29:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:29:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:57 smithi188 ceph-mon[92425]: pgmap v3327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:29:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:29:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:29:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:29:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:29:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:59 smithi161 ceph-mon[91579]: pgmap v3328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:29:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:59 smithi188 ceph-mon[92425]: pgmap v3328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:29:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:30:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:30:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:01 smithi161 ceph-mon[91579]: pgmap v3329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:01 smithi188 ceph-mon[92425]: pgmap v3329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:03 smithi161 ceph-mon[91579]: pgmap v3330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:03 smithi188 ceph-mon[92425]: pgmap v3330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:05 smithi161 ceph-mon[91579]: pgmap v3331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:05 smithi188 ceph-mon[92425]: pgmap v3331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:07 smithi161 ceph-mon[91579]: pgmap v3332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:07 smithi188 ceph-mon[92425]: pgmap v3332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:09 smithi161 ceph-mon[91579]: pgmap v3333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:09 smithi188 ceph-mon[92425]: pgmap v3333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:11 smithi161 ceph-mon[91579]: pgmap v3334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:11 smithi188 ceph-mon[92425]: pgmap v3334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:13 smithi161 ceph-mon[91579]: pgmap v3335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:13 smithi188 ceph-mon[92425]: pgmap v3335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:15 smithi161 ceph-mon[91579]: pgmap v3336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:15 smithi188 ceph-mon[92425]: pgmap v3336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:17 smithi161 ceph-mon[91579]: pgmap v3337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:18.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:17 smithi188 ceph-mon[92425]: pgmap v3337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:19 smithi161 ceph-mon[91579]: pgmap v3338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:19 smithi188 ceph-mon[92425]: pgmap v3338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:21 smithi161 ceph-mon[91579]: pgmap v3339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:21 smithi188 ceph-mon[92425]: pgmap v3339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:23 smithi161 ceph-mon[91579]: pgmap v3340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:23 smithi188 ceph-mon[92425]: pgmap v3340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:25 smithi161 ceph-mon[91579]: pgmap v3341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:25 smithi188 ceph-mon[92425]: pgmap v3341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:26 smithi161 ceph-mon[91579]: pgmap v3342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:26 smithi188 ceph-mon[92425]: pgmap v3342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:29 smithi161 ceph-mon[91579]: pgmap v3343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:29 smithi188 ceph-mon[92425]: pgmap v3343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:31 smithi161 ceph-mon[91579]: pgmap v3344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:31 smithi188 ceph-mon[92425]: pgmap v3344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:33 smithi161 ceph-mon[91579]: pgmap v3345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:33 smithi188 ceph-mon[92425]: pgmap v3345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:35 smithi161 ceph-mon[91579]: pgmap v3346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:35 smithi188 ceph-mon[92425]: pgmap v3346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:36 smithi161 ceph-mon[91579]: pgmap v3347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:36 smithi188 ceph-mon[92425]: pgmap v3347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-mon[91579]: pgmap v3348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:39 smithi188 ceph-mon[92425]: pgmap v3348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:40.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:30:39.647+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:30:40.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:30:40.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:30:40.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:30:40.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:30:40.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:30:40.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:30:40.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:30:40.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:30:40.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:30:40.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:30:40.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:30:40.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:30:40.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:30:40.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:30:40.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:30:40.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:30:40.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:30:40.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:30:40.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:30:40.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:30:40.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:30:40.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:30:40.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:30:40.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:30:40.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:30:40.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:30:40.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:30:40.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:30:40.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:30:40.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:30:40.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:30:40.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:30:40.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:30:40.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:30:40.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:30:40.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:30:40.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:30:40.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:30:40.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:30:40.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:30:40.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:30:40.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:30:40.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:30:40.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:30:40.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:30:40.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=163] Local address: 172.21.15.161, port 52786 2022-07-03T12:30:40.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=163] Peer address: 172.21.15.101, port 22 2022-07-03T12:30:40.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=163] Connection lost 2022-07-03T12:30:40.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=163] Aborting connection 2022-07-03T12:30:40.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:30:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:30:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:30:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:30:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:30:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:30:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:30:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:30:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:30:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:30:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:30:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:30:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:30:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:30:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:30:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:30:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:30:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:30:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:30:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:30:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:30:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:30:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:30:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:30:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:30:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:30:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:30:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:30:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:30:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:30:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:30:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:30:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:30:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:30:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:30:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:30:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:30:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:30:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:30:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:30:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:30:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:30:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:30:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:30:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:30:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:30:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:30:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:30:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:30:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:30:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:30:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:30:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=163] Local address: 172.21.15.161, port 52786 2022-07-03T12:30:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=163] Peer address: 172.21.15.101, port 22 2022-07-03T12:30:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=163] Connection lost 2022-07-03T12:30:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=163] Aborting connection 2022-07-03T12:30:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:30:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:30:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:30:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:30:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:30:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:30:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:30:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:30:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:30:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:30:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:30:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:30:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:30:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:30:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:30:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:30:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:30:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:30:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:30:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:30:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:30:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:30:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:30:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:30:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:30:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:30:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:30:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:30:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:30:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:30:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:30:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:30:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:30:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:30:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:30:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:30:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:30:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:30:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:30:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:30:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:30:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:30:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:30:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:30:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:30:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:30:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:30:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:30:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:30:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=163] Local address: 172.21.15.161, port 52786 2022-07-03T12:30:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=163] Peer address: 172.21.15.101, port 22 2022-07-03T12:30:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=163] Connection lost 2022-07-03T12:30:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=163] Aborting connection 2022-07-03T12:30:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:41 smithi161 ceph-mon[91579]: pgmap v3349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:41 smithi188 ceph-mon[92425]: pgmap v3349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:43 smithi161 ceph-mon[91579]: pgmap v3350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:43 smithi188 ceph-mon[92425]: pgmap v3350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:45 smithi161 ceph-mon[91579]: pgmap v3351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:45 smithi188 ceph-mon[92425]: pgmap v3351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:46 smithi161 ceph-mon[91579]: pgmap v3352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:46 smithi188 ceph-mon[92425]: pgmap v3352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:49 smithi161 ceph-mon[91579]: pgmap v3353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:49 smithi188 ceph-mon[92425]: pgmap v3353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:51 smithi161 ceph-mon[91579]: pgmap v3354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:52.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:51 smithi188 ceph-mon[92425]: pgmap v3354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:53 smithi161 ceph-mon[91579]: pgmap v3355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:53 smithi188 ceph-mon[92425]: pgmap v3355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:55 smithi161 ceph-mon[91579]: pgmap v3356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:55 smithi188 ceph-mon[92425]: pgmap v3356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:30:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:56 smithi161 ceph-mon[91579]: pgmap v3357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:56 smithi188 ceph-mon[92425]: pgmap v3357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:30:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:30:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:30:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:30:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:30:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:30:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:30:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:30:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:30:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:59 smithi161 ceph-mon[91579]: pgmap v3358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:30:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:59 smithi188 ceph-mon[92425]: pgmap v3358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:30:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:01 smithi161 ceph-mon[91579]: pgmap v3359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:01 smithi188 ceph-mon[92425]: pgmap v3359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:03 smithi161 ceph-mon[91579]: pgmap v3360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:03 smithi188 ceph-mon[92425]: pgmap v3360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:05 smithi161 ceph-mon[91579]: pgmap v3361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:05 smithi188 ceph-mon[92425]: pgmap v3361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:06 smithi161 ceph-mon[91579]: pgmap v3362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:06 smithi188 ceph-mon[92425]: pgmap v3362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:09 smithi161 ceph-mon[91579]: pgmap v3363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:09 smithi188 ceph-mon[92425]: pgmap v3363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:11 smithi161 ceph-mon[91579]: pgmap v3364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:11 smithi188 ceph-mon[92425]: pgmap v3364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:13 smithi161 ceph-mon[91579]: pgmap v3365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:13 smithi188 ceph-mon[92425]: pgmap v3365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:14 smithi161 ceph-mon[91579]: pgmap v3366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:14 smithi188 ceph-mon[92425]: pgmap v3366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:16 smithi161 ceph-mon[91579]: pgmap v3367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:16 smithi188 ceph-mon[92425]: pgmap v3367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:19 smithi161 ceph-mon[91579]: pgmap v3368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:19 smithi188 ceph-mon[92425]: pgmap v3368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:21 smithi161 ceph-mon[91579]: pgmap v3369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:21 smithi188 ceph-mon[92425]: pgmap v3369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:31:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:23 smithi161 ceph-mon[91579]: pgmap v3370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:23 smithi188 ceph-mon[92425]: pgmap v3370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:24 smithi161 ceph-mon[91579]: pgmap v3371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:24 smithi188 ceph-mon[92425]: pgmap v3371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:26 smithi161 ceph-mon[91579]: pgmap v3372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:26 smithi188 ceph-mon[92425]: pgmap v3372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:29 smithi161 ceph-mon[91579]: pgmap v3373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:29 smithi188 ceph-mon[92425]: pgmap v3373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:31 smithi161 ceph-mon[91579]: pgmap v3374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:31 smithi188 ceph-mon[92425]: pgmap v3374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:33 smithi161 ceph-mon[91579]: pgmap v3375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:33 smithi188 ceph-mon[92425]: pgmap v3375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:34 smithi161 ceph-mon[91579]: pgmap v3376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:34 smithi188 ceph-mon[92425]: pgmap v3376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:36 smithi161 ceph-mon[91579]: pgmap v3377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:36 smithi188 ceph-mon[92425]: pgmap v3377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:39 smithi161 ceph-mon[91579]: pgmap v3378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:39 smithi188 ceph-mon[92425]: pgmap v3378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:31:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:31:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:31:40.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:31:40.155+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:31:40.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:31:40.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:31:40.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:31:40.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:31:40.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:31:40.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:31:40.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:31:40.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:31:40.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:31:40.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:31:40.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:31:40.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:31:40.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:31:40.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:31:40.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:31:40.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:31:40.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:31:40.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:31:40.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:31:40.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:31:40.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:31:40.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:31:40.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:31:40.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:31:40.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:31:40.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:31:40.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:31:40.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:31:40.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:31:40.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:31:40.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:31:40.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:31:40.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:31:40.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:31:40.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:31:40.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:31:40.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:31:40.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:31:40.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:31:40.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:31:40.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:31:40.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:31:40.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:31:40.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:31:40.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:31:40.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=165] Local address: 172.21.15.161, port 52814 2022-07-03T12:31:40.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=165] Peer address: 172.21.15.101, port 22 2022-07-03T12:31:40.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=165] Connection lost 2022-07-03T12:31:40.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=165] Aborting connection 2022-07-03T12:31:40.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:31:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:31:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:31:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:31:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:31:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:31:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:31:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:31:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:31:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:31:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:31:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:31:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:31:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:31:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:31:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:31:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:31:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:31:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:31:41.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:31:41.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:31:41.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:31:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:31:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:31:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:31:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:31:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:31:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:31:41.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:31:41.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:31:41.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:31:41.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:31:41.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:31:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:31:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:31:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:31:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:31:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:31:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:31:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:31:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:31:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:31:41.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:31:41.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:31:41.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:31:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:31:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:31:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:31:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:31:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:31:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:31:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:31:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=165] Local address: 172.21.15.161, port 52814 2022-07-03T12:31:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=165] Peer address: 172.21.15.101, port 22 2022-07-03T12:31:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=165] Connection lost 2022-07-03T12:31:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=165] Aborting connection 2022-07-03T12:31:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:41.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:41 smithi161 ceph-mon[91579]: pgmap v3379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:31:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:31:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:31:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:31:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:31:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:31:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:31:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:31:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:31:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:31:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:31:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:31:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:31:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:31:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:31:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:31:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:31:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:31:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:31:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:31:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:31:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:31:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:31:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:31:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:31:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:31:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:31:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:31:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:31:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:31:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:31:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:31:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:31:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:31:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:31:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:31:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:31:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:31:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:31:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:31:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:31:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:31:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:31:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:31:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=165] Local address: 172.21.15.161, port 52814 2022-07-03T12:31:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=165] Peer address: 172.21.15.101, port 22 2022-07-03T12:31:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=165] Connection lost 2022-07-03T12:31:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=165] Aborting connection 2022-07-03T12:31:41.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:41.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:41 smithi188 ceph-mon[92425]: pgmap v3379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:43 smithi161 ceph-mon[91579]: pgmap v3380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:43 smithi188 ceph-mon[92425]: pgmap v3380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:44 smithi161 ceph-mon[91579]: pgmap v3381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:44 smithi188 ceph-mon[92425]: pgmap v3381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:46 smithi161 ceph-mon[91579]: pgmap v3382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:46 smithi188 ceph-mon[92425]: pgmap v3382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:49 smithi161 ceph-mon[91579]: pgmap v3383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:49 smithi188 ceph-mon[92425]: pgmap v3383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:51 smithi161 ceph-mon[91579]: pgmap v3384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:51 smithi188 ceph-mon[92425]: pgmap v3384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:53 smithi161 ceph-mon[91579]: pgmap v3385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:53 smithi188 ceph-mon[92425]: pgmap v3385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:55 smithi161 ceph-mon[91579]: pgmap v3386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:55 smithi188 ceph-mon[92425]: pgmap v3386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:31:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:57 smithi161 ceph-mon[91579]: pgmap v3387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:31:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:31:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:31:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:57 smithi188 ceph-mon[92425]: pgmap v3387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:31:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:31:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:31:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:31:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:31:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:59 smithi161 ceph-mon[91579]: pgmap v3388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:31:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:59 smithi188 ceph-mon[92425]: pgmap v3388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:31:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:01 smithi161 ceph-mon[91579]: pgmap v3389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:01 smithi188 ceph-mon[92425]: pgmap v3389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:03 smithi161 ceph-mon[91579]: pgmap v3390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:03 smithi188 ceph-mon[92425]: pgmap v3390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:05 smithi161 ceph-mon[91579]: pgmap v3391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:05 smithi188 ceph-mon[92425]: pgmap v3391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:07 smithi161 ceph-mon[91579]: pgmap v3392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:07 smithi188 ceph-mon[92425]: pgmap v3392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:09 smithi161 ceph-mon[91579]: pgmap v3393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:09 smithi188 ceph-mon[92425]: pgmap v3393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:11 smithi161 ceph-mon[91579]: pgmap v3394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:11 smithi188 ceph-mon[92425]: pgmap v3394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:13 smithi161 ceph-mon[91579]: pgmap v3395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:13 smithi188 ceph-mon[92425]: pgmap v3395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:15 smithi161 ceph-mon[91579]: pgmap v3396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:15 smithi188 ceph-mon[92425]: pgmap v3396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:16 smithi161 ceph-mon[91579]: pgmap v3397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:16 smithi188 ceph-mon[92425]: pgmap v3397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:19 smithi161 ceph-mon[91579]: pgmap v3398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:19 smithi188 ceph-mon[92425]: pgmap v3398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:21 smithi161 ceph-mon[91579]: pgmap v3399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:21 smithi188 ceph-mon[92425]: pgmap v3399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:23 smithi161 ceph-mon[91579]: pgmap v3400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:23 smithi188 ceph-mon[92425]: pgmap v3400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:25 smithi161 ceph-mon[91579]: pgmap v3401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:25 smithi188 ceph-mon[92425]: pgmap v3401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:26 smithi161 ceph-mon[91579]: pgmap v3402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:26 smithi188 ceph-mon[92425]: pgmap v3402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:29 smithi161 ceph-mon[91579]: pgmap v3403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:29 smithi188 ceph-mon[92425]: pgmap v3403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:31 smithi161 ceph-mon[91579]: pgmap v3404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:31 smithi188 ceph-mon[92425]: pgmap v3404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:33 smithi161 ceph-mon[91579]: pgmap v3405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:33 smithi188 ceph-mon[92425]: pgmap v3405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:35.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:35 smithi161 ceph-mon[91579]: pgmap v3406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:35 smithi188 ceph-mon[92425]: pgmap v3406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:36 smithi161 ceph-mon[91579]: pgmap v3407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:36 smithi188 ceph-mon[92425]: pgmap v3407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:39 smithi161 ceph-mon[91579]: pgmap v3408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:39 smithi188 ceph-mon[92425]: pgmap v3408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:32:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:32:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:32:41.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:32:40.662+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:32:41.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:32:41.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:32:41.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:32:41.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:32:41.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:32:41.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:32:41.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:32:41.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:32:41.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:32:41.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:32:41.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:32:41.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:32:41.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:32:41.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:32:41.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:32:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:32:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:32:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:32:41.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:32:41.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:32:41.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:32:41.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:32:41.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:32:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:32:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:32:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:32:41.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:32:41.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:32:41.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:32:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:32:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:32:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:32:41.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:32:41.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:32:41.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:32:41.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:32:41.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:32:41.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:32:41.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:32:41.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:32:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:32:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:32:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:32:41.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:32:41.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:32:41.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=167] Local address: 172.21.15.161, port 52842 2022-07-03T12:32:41.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=167] Peer address: 172.21.15.101, port 22 2022-07-03T12:32:41.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=167] Connection lost 2022-07-03T12:32:41.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=167] Aborting connection 2022-07-03T12:32:41.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:32:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:32:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:32:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:32:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:32:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:32:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:32:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:32:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:32:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:32:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:32:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:32:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:32:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:32:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:32:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:32:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:32:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:32:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:32:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:32:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:32:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:32:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:32:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:32:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:32:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:32:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:32:42.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:32:42.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:32:42.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:32:42.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:32:42.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:32:42.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:32:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:32:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:32:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:32:42.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:32:42.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:32:42.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:32:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:32:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:32:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:32:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:32:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:32:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:32:42.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:32:42.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:32:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:32:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:32:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:32:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:32:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:32:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=167] Local address: 172.21.15.161, port 52842 2022-07-03T12:32:42.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=167] Peer address: 172.21.15.101, port 22 2022-07-03T12:32:42.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=167] Connection lost 2022-07-03T12:32:42.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=167] Aborting connection 2022-07-03T12:32:42.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:42.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:41 smithi161 ceph-mon[91579]: pgmap v3409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:32:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:32:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:32:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:32:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:32:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:32:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:32:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:32:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:32:42.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:32:42.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:32:42.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:32:42.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:32:42.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:32:42.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:32:42.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:32:42.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:32:42.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:32:42.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:32:42.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:32:42.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:32:42.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:32:42.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:32:42.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:32:42.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:32:42.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:32:42.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:32:42.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:32:42.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:32:42.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:32:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:32:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:32:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:32:42.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:32:42.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:32:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:32:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:32:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:32:42.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:32:42.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:32:42.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:32:42.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:32:42.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:32:42.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:32:42.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:32:42.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:32:42.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:32:42.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=167] Local address: 172.21.15.161, port 52842 2022-07-03T12:32:42.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=167] Peer address: 172.21.15.101, port 22 2022-07-03T12:32:42.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=167] Connection lost 2022-07-03T12:32:42.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=167] Aborting connection 2022-07-03T12:32:42.189 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:42.190 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:41 smithi188 ceph-mon[92425]: pgmap v3409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:42 smithi161 ceph-mon[91579]: pgmap v3410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:42 smithi188 ceph-mon[92425]: pgmap v3410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:45 smithi161 ceph-mon[91579]: pgmap v3411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:45 smithi188 ceph-mon[92425]: pgmap v3411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:47 smithi161 ceph-mon[91579]: pgmap v3412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:47 smithi188 ceph-mon[92425]: pgmap v3412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:49 smithi161 ceph-mon[91579]: pgmap v3413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:49 smithi188 ceph-mon[92425]: pgmap v3413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:51 smithi161 ceph-mon[91579]: pgmap v3414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:51 smithi188 ceph-mon[92425]: pgmap v3414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:52 smithi161 ceph-mon[91579]: pgmap v3415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:52 smithi188 ceph-mon[92425]: pgmap v3415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:55 smithi161 ceph-mon[91579]: pgmap v3416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:55 smithi188 ceph-mon[92425]: pgmap v3416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:32:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:57 smithi161 ceph-mon[91579]: pgmap v3417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:57 smithi188 ceph-mon[92425]: pgmap v3417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:32:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:32:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:32:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:32:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:32:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:32:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:59 smithi161 ceph-mon[91579]: pgmap v3418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:32:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:59 smithi188 ceph-mon[92425]: pgmap v3418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:32:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:00 smithi161 ceph-mon[91579]: pgmap v3419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:00 smithi188 ceph-mon[92425]: pgmap v3419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:02 smithi161 ceph-mon[91579]: pgmap v3420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:02 smithi188 ceph-mon[92425]: pgmap v3420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:05 smithi161 ceph-mon[91579]: pgmap v3421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:05 smithi188 ceph-mon[92425]: pgmap v3421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:07 smithi161 ceph-mon[91579]: pgmap v3422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:07 smithi188 ceph-mon[92425]: pgmap v3422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:09 smithi161 ceph-mon[91579]: pgmap v3423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:09 smithi188 ceph-mon[92425]: pgmap v3423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:10 smithi161 ceph-mon[91579]: pgmap v3424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:10 smithi188 ceph-mon[92425]: pgmap v3424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:12 smithi161 ceph-mon[91579]: pgmap v3425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:12 smithi188 ceph-mon[92425]: pgmap v3425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:15 smithi161 ceph-mon[91579]: pgmap v3426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:15 smithi188 ceph-mon[92425]: pgmap v3426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:17 smithi161 ceph-mon[91579]: pgmap v3427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:17 smithi188 ceph-mon[92425]: pgmap v3427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:19 smithi161 ceph-mon[91579]: pgmap v3428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:19 smithi188 ceph-mon[92425]: pgmap v3428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:20 smithi161 ceph-mon[91579]: pgmap v3429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:20 smithi188 ceph-mon[92425]: pgmap v3429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:22 smithi161 ceph-mon[91579]: pgmap v3430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:22 smithi188 ceph-mon[92425]: pgmap v3430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:25 smithi161 ceph-mon[91579]: pgmap v3431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:25 smithi188 ceph-mon[92425]: pgmap v3431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:27 smithi161 ceph-mon[91579]: pgmap v3432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:27 smithi188 ceph-mon[92425]: pgmap v3432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:29 smithi161 ceph-mon[91579]: pgmap v3433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:29 smithi188 ceph-mon[92425]: pgmap v3433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:30 smithi161 ceph-mon[91579]: pgmap v3434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:30 smithi188 ceph-mon[92425]: pgmap v3434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:32 smithi161 ceph-mon[91579]: pgmap v3435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:32 smithi188 ceph-mon[92425]: pgmap v3435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:35 smithi161 ceph-mon[91579]: pgmap v3436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:35 smithi188 ceph-mon[92425]: pgmap v3436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:37 smithi161 ceph-mon[91579]: pgmap v3437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:37 smithi188 ceph-mon[92425]: pgmap v3437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:39 smithi161 ceph-mon[91579]: pgmap v3438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:39 smithi188 ceph-mon[92425]: pgmap v3438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:40 smithi161 ceph-mon[91579]: pgmap v3439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:40 smithi188 ceph-mon[92425]: pgmap v3439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:41.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:33:41.173+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:33:41.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:33:41.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:33:41.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:33:41.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:33:41.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:33:41.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:33:41.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:33:41.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:33:41.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:33:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:33:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:33:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:33:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:33:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:33:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:33:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:33:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:33:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:33:41.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:33:41.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:33:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:33:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:33:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:33:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:33:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:33:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:33:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:33:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:33:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:33:41.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:33:41.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:33:41.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:33:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:33:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:33:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:33:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:33:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:33:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:33:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:33:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:33:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:33:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:33:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:33:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:33:41.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:33:41.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=169] Local address: 172.21.15.161, port 52870 2022-07-03T12:33:41.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=169] Peer address: 172.21.15.101, port 22 2022-07-03T12:33:41.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=169] Connection lost 2022-07-03T12:33:41.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=169] Aborting connection 2022-07-03T12:33:41.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:33:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:33:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:33:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:33:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:33:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:33:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:33:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:33:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:33:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:33:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:33:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:33:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:33:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:33:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:33:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:33:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:33:42.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:33:42.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:33:42.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:33:42.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:33:42.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:33:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:33:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:33:42.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:33:42.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:33:42.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:33:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:33:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:33:42.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:33:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:33:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:33:42.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:33:42.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:33:42.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:33:42.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:33:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:33:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:33:42.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:33:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:33:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:33:42.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:33:42.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:33:42.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:33:42.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:33:42.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:33:42.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:33:42.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:33:42.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:33:42.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:33:42.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:33:42.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=169] Local address: 172.21.15.161, port 52870 2022-07-03T12:33:42.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=169] Peer address: 172.21.15.101, port 22 2022-07-03T12:33:42.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=169] Connection lost 2022-07-03T12:33:42.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=169] Aborting connection 2022-07-03T12:33:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:33:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:33:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:33:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:33:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:33:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:33:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:33:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:33:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:33:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:33:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:33:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:33:42.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:33:42.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:33:42.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:33:42.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:33:42.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:33:42.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:33:42.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:33:42.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:33:42.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:33:42.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:33:42.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:33:42.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:33:42.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:33:42.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:33:42.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:33:42.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:33:42.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:33:42.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:33:42.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:33:42.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:33:42.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:33:42.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:33:42.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:33:42.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:33:42.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:33:42.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:33:42.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:33:42.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:33:42.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:33:42.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:33:42.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:33:42.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:33:42.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:33:42.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:33:42.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:33:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:33:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:33:42.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:33:42.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=169] Local address: 172.21.15.161, port 52870 2022-07-03T12:33:42.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=169] Peer address: 172.21.15.101, port 22 2022-07-03T12:33:42.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=169] Connection lost 2022-07-03T12:33:42.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=169] Aborting connection 2022-07-03T12:33:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:42 smithi161 ceph-mon[91579]: pgmap v3440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:42 smithi188 ceph-mon[92425]: pgmap v3440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:45 smithi161 ceph-mon[91579]: pgmap v3441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:45 smithi188 ceph-mon[92425]: pgmap v3441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:47 smithi161 ceph-mon[91579]: pgmap v3442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:47 smithi188 ceph-mon[92425]: pgmap v3442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:49 smithi161 ceph-mon[91579]: pgmap v3443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:49 smithi188 ceph-mon[92425]: pgmap v3443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:50 smithi161 ceph-mon[91579]: pgmap v3444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:50 smithi188 ceph-mon[92425]: pgmap v3444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:52 smithi161 ceph-mon[91579]: pgmap v3445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:52 smithi188 ceph-mon[92425]: pgmap v3445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:55 smithi161 ceph-mon[91579]: pgmap v3446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:55 smithi188 ceph-mon[92425]: pgmap v3446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:33:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:57 smithi161 ceph-mon[91579]: pgmap v3447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:57 smithi188 ceph-mon[92425]: pgmap v3447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:33:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:33:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:33:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:59 smithi161 ceph-mon[91579]: pgmap v3448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:33:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:59 smithi188 ceph-mon[92425]: pgmap v3448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:33:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:00 smithi161 ceph-mon[91579]: pgmap v3449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:00 smithi188 ceph-mon[92425]: pgmap v3449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:02 smithi161 ceph-mon[91579]: pgmap v3450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:02 smithi188 ceph-mon[92425]: pgmap v3450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:05 smithi161 ceph-mon[91579]: pgmap v3451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:05 smithi188 ceph-mon[92425]: pgmap v3451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:07 smithi161 ceph-mon[91579]: pgmap v3452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:07 smithi188 ceph-mon[92425]: pgmap v3452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:09 smithi161 ceph-mon[91579]: pgmap v3453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:09 smithi188 ceph-mon[92425]: pgmap v3453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:10 smithi161 ceph-mon[91579]: pgmap v3454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:10 smithi188 ceph-mon[92425]: pgmap v3454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:12 smithi161 ceph-mon[91579]: pgmap v3455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:12 smithi188 ceph-mon[92425]: pgmap v3455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:15 smithi161 ceph-mon[91579]: pgmap v3456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:15 smithi188 ceph-mon[92425]: pgmap v3456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:17 smithi161 ceph-mon[91579]: pgmap v3457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:17 smithi188 ceph-mon[92425]: pgmap v3457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:19 smithi161 ceph-mon[91579]: pgmap v3458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:19 smithi188 ceph-mon[92425]: pgmap v3458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:20 smithi161 ceph-mon[91579]: pgmap v3459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:20 smithi188 ceph-mon[92425]: pgmap v3459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:22 smithi161 ceph-mon[91579]: pgmap v3460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:22 smithi188 ceph-mon[92425]: pgmap v3460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:25 smithi161 ceph-mon[91579]: pgmap v3461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:25 smithi188 ceph-mon[92425]: pgmap v3461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:27 smithi161 ceph-mon[91579]: pgmap v3462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:27 smithi188 ceph-mon[92425]: pgmap v3462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:29 smithi161 ceph-mon[91579]: pgmap v3463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:29 smithi188 ceph-mon[92425]: pgmap v3463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:30 smithi161 ceph-mon[91579]: pgmap v3464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:30 smithi188 ceph-mon[92425]: pgmap v3464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:32 smithi161 ceph-mon[91579]: pgmap v3465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:32 smithi188 ceph-mon[92425]: pgmap v3465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:35 smithi161 ceph-mon[91579]: pgmap v3466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:35 smithi188 ceph-mon[92425]: pgmap v3466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:37 smithi161 ceph-mon[91579]: pgmap v3467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:37 smithi188 ceph-mon[92425]: pgmap v3467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:39 smithi161 ceph-mon[91579]: pgmap v3468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:39 smithi188 ceph-mon[92425]: pgmap v3468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:40 smithi161 ceph-mon[91579]: pgmap v3469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:40 smithi188 ceph-mon[92425]: pgmap v3469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:34:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:34:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:34:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:34:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:34:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:34:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:34:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:34:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:42 smithi161 ceph-mon[91579]: pgmap v3470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:42 smithi188 ceph-mon[92425]: pgmap v3470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:45 smithi161 ceph-mon[91579]: pgmap v3471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:45 smithi188 ceph-mon[92425]: pgmap v3471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:47 smithi161 ceph-mon[91579]: pgmap v3472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:47 smithi188 ceph-mon[92425]: pgmap v3472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:49 smithi161 ceph-mon[91579]: pgmap v3473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:49 smithi188 ceph-mon[92425]: pgmap v3473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:50 smithi161 ceph-mon[91579]: pgmap v3474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:50 smithi188 ceph-mon[92425]: pgmap v3474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:52 smithi161 ceph-mon[91579]: pgmap v3475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:52 smithi188 ceph-mon[92425]: pgmap v3475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:55 smithi161 ceph-mon[91579]: pgmap v3476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:55 smithi188 ceph-mon[92425]: pgmap v3476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:34:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:57 smithi161 ceph-mon[91579]: pgmap v3477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:57 smithi188 ceph-mon[92425]: pgmap v3477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:34:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:34:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:34:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:34:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:34:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:34:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:59 smithi188 ceph-mon[92425]: pgmap v3478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:34:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:59 smithi161 ceph-mon[91579]: pgmap v3478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:34:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:00 smithi161 ceph-mon[91579]: pgmap v3479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:00 smithi188 ceph-mon[92425]: pgmap v3479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:02 smithi161 ceph-mon[91579]: pgmap v3480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:02 smithi188 ceph-mon[92425]: pgmap v3480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:05 smithi161 ceph-mon[91579]: pgmap v3481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:05 smithi188 ceph-mon[92425]: pgmap v3481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:07 smithi161 ceph-mon[91579]: pgmap v3482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:07 smithi188 ceph-mon[92425]: pgmap v3482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:09 smithi161 ceph-mon[91579]: pgmap v3483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:09 smithi188 ceph-mon[92425]: pgmap v3483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:10 smithi161 ceph-mon[91579]: pgmap v3484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:10 smithi188 ceph-mon[92425]: pgmap v3484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:35:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:35:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:35:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:35:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:35:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:35:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:35:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:35:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:35:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:35:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:35:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:35:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:35:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:35:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:35:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:35:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:35:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:35:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:35:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:35:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:35:12.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:35:12.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:35:12.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:35:12.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:35:12.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:35:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:35:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:35:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:35:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:35:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:35:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:35:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:35:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:35:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:35:12.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:35:12.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:35:12.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:35:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:35:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:35:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:35:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:35:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:35:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:35:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:35:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:35:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=171] Local address: 172.21.15.161, port 52898 2022-07-03T12:35:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=171] Peer address: 172.21.15.101, port 22 2022-07-03T12:35:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=500] Received exit status 0 2022-07-03T12:35:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=500] Received channel close 2022-07-03T12:35:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=500] Channel closed 2022-07-03T12:35:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=500] Received exit status 0 2022-07-03T12:35:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=500] Received channel close 2022-07-03T12:35:12.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=500] Channel closed 2022-07-03T12:35:12.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=501] Requesting new SSH session 2022-07-03T12:35:12.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=501] Requesting new SSH session 2022-07-03T12:35:12.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=501] Command: which python3 2022-07-03T12:35:12.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=501] Command: which python3 2022-07-03T12:35:12.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=501] Received exit status 0 2022-07-03T12:35:12.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=501] Received channel close 2022-07-03T12:35:12.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=501] Channel closed 2022-07-03T12:35:12.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=501] Received exit status 0 2022-07-03T12:35:12.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=501] Received channel close 2022-07-03T12:35:12.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=502] Requesting new SSH session 2022-07-03T12:35:12.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=501] Channel closed 2022-07-03T12:35:12.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=502] Requesting new SSH session 2022-07-03T12:35:12.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=502] Command: sudo true 2022-07-03T12:35:12.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=502] Command: sudo true 2022-07-03T12:35:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=502] Received exit status 0 2022-07-03T12:35:12.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=502] Received channel close 2022-07-03T12:35:12.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=502] Channel closed 2022-07-03T12:35:12.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=503] Requesting new SSH session 2022-07-03T12:35:12.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=502] Received exit status 0 2022-07-03T12:35:12.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=502] Received channel close 2022-07-03T12:35:12.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=502] Channel closed 2022-07-03T12:35:12.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=503] Requesting new SSH session 2022-07-03T12:35:12.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=503] Received exit status 0 2022-07-03T12:35:12.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=503] Received channel close 2022-07-03T12:35:12.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=503] Channel closed 2022-07-03T12:35:12.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=503] Received exit status 0 2022-07-03T12:35:12.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=503] Received channel close 2022-07-03T12:35:12.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=503] Channel closed 2022-07-03T12:35:12.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=171] Login timeout expired 2022-07-03T12:35:12.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=171] Aborting connection 2022-07-03T12:35:12.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:35:11.672+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:35:12.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:35:12.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:35:12.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:35:12.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:35:12.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:35:12.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:35:12.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:35:12.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:35:12.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:35:12.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:35:12.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:35:12.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:35:12.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:35:12.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:35:12.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:35:12.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:35:12.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:35:12.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:35:12.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:35:12.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:35:12.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:35:12.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:35:12.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:35:12.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:35:12.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:35:12.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:35:12.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:35:12.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:35:12.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:35:12.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:35:12.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:35:12.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:35:12.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:35:12.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:35:12.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:35:12.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:35:12.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:35:12.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:35:12.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:35:12.193 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:35:12.193 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:35:12.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:35:12.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:35:12.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:35:12.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:35:12.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=171] Local address: 172.21.15.161, port 52898 2022-07-03T12:35:12.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=171] Peer address: 172.21.15.101, port 22 2022-07-03T12:35:12.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=500] Received exit status 0 2022-07-03T12:35:12.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=500] Received channel close 2022-07-03T12:35:12.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=500] Channel closed 2022-07-03T12:35:12.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=500] Received exit status 0 2022-07-03T12:35:12.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=500] Received channel close 2022-07-03T12:35:12.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=500] Channel closed 2022-07-03T12:35:12.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=501] Requesting new SSH session 2022-07-03T12:35:12.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=501] Requesting new SSH session 2022-07-03T12:35:12.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=501] Command: which python3 2022-07-03T12:35:12.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=501] Command: which python3 2022-07-03T12:35:12.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=501] Received exit status 0 2022-07-03T12:35:12.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=501] Received channel close 2022-07-03T12:35:12.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=501] Channel closed 2022-07-03T12:35:12.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=501] Received exit status 0 2022-07-03T12:35:12.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=501] Received channel close 2022-07-03T12:35:12.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=502] Requesting new SSH session 2022-07-03T12:35:12.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=501] Channel closed 2022-07-03T12:35:12.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=502] Requesting new SSH session 2022-07-03T12:35:12.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=502] Command: sudo true 2022-07-03T12:35:12.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=502] Command: sudo true 2022-07-03T12:35:12.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=502] Received exit status 0 2022-07-03T12:35:12.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=502] Received channel close 2022-07-03T12:35:12.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=502] Channel closed 2022-07-03T12:35:12.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=503] Requesting new SSH session 2022-07-03T12:35:12.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=502] Received exit status 0 2022-07-03T12:35:12.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=502] Received channel close 2022-07-03T12:35:12.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=502] Channel closed 2022-07-03T12:35:12.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=503] Requesting new SSH session 2022-07-03T12:35:12.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=503] Received exit status 0 2022-07-03T12:35:12.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=503] Received channel close 2022-07-03T12:35:12.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=503] Channel closed 2022-07-03T12:35:12.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=503] Received exit status 0 2022-07-03T12:35:12.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=503] Received channel close 2022-07-03T12:35:12.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=503] Channel closed 2022-07-03T12:35:12.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=171] Login timeout expired 2022-07-03T12:35:12.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=171] Aborting connection 2022-07-03T12:35:12.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:35:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:35:12.209 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:35:12.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:35:12.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:35:12.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:35:12.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:35:12.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:35:12.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:35:12.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:35:12.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:35:12.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:35:12.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:35:12.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:35:12.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:35:12.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:35:12.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:35:12.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:35:12.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:35:12.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:35:12.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:35:12.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:35:12.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:35:12.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:35:12.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:35:12.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:35:12.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:35:12.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:35:12.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:35:12.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:35:12.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:35:12.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:35:12.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:35:12.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:35:12.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:35:12.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:35:12.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:35:12.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:35:12.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:35:12.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:35:12.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:35:12.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:35:12.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:35:12.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:35:12.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:35:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:35:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:35:12.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=171] Local address: 172.21.15.161, port 52898 2022-07-03T12:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=171] Peer address: 172.21.15.101, port 22 2022-07-03T12:35:12.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=500] Received exit status 0 2022-07-03T12:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=500] Received channel close 2022-07-03T12:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=500] Channel closed 2022-07-03T12:35:12.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=500] Received exit status 0 2022-07-03T12:35:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=500] Received channel close 2022-07-03T12:35:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=500] Channel closed 2022-07-03T12:35:12.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=501] Requesting new SSH session 2022-07-03T12:35:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=501] Requesting new SSH session 2022-07-03T12:35:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=501] Command: which python3 2022-07-03T12:35:12.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=501] Command: which python3 2022-07-03T12:35:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=501] Received exit status 0 2022-07-03T12:35:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=501] Received channel close 2022-07-03T12:35:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=501] Channel closed 2022-07-03T12:35:12.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=501] Received exit status 0 2022-07-03T12:35:12.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=501] Received channel close 2022-07-03T12:35:12.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=502] Requesting new SSH session 2022-07-03T12:35:12.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=501] Channel closed 2022-07-03T12:35:12.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=502] Requesting new SSH session 2022-07-03T12:35:12.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=502] Command: sudo true 2022-07-03T12:35:12.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=502] Command: sudo true 2022-07-03T12:35:12.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=502] Received exit status 0 2022-07-03T12:35:12.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=502] Received channel close 2022-07-03T12:35:12.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=502] Channel closed 2022-07-03T12:35:12.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=503] Requesting new SSH session 2022-07-03T12:35:12.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=502] Received exit status 0 2022-07-03T12:35:12.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=502] Received channel close 2022-07-03T12:35:12.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=502] Channel closed 2022-07-03T12:35:12.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=503] Requesting new SSH session 2022-07-03T12:35:12.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=503] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:35:12.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=503] Received exit status 0 2022-07-03T12:35:12.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=503] Received channel close 2022-07-03T12:35:12.235 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=503] Channel closed 2022-07-03T12:35:12.235 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=503] Received exit status 0 2022-07-03T12:35:12.235 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=503] Received channel close 2022-07-03T12:35:12.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=503] Channel closed 2022-07-03T12:35:12.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=171] Login timeout expired 2022-07-03T12:35:12.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=171] Aborting connection 2022-07-03T12:35:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:12 smithi161 ceph-mon[91579]: pgmap v3485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:12 smithi188 ceph-mon[92425]: pgmap v3485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:15 smithi161 ceph-mon[91579]: pgmap v3486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:15 smithi188 ceph-mon[92425]: pgmap v3486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:17 smithi188 ceph-mon[92425]: pgmap v3487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:17 smithi161 ceph-mon[91579]: pgmap v3487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:19 smithi161 ceph-mon[91579]: pgmap v3488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:19 smithi188 ceph-mon[92425]: pgmap v3488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:20 smithi161 ceph-mon[91579]: pgmap v3489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:20 smithi188 ceph-mon[92425]: pgmap v3489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:22 smithi161 ceph-mon[91579]: pgmap v3490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:22 smithi188 ceph-mon[92425]: pgmap v3490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:25 smithi161 ceph-mon[91579]: pgmap v3491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:25 smithi188 ceph-mon[92425]: pgmap v3491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:27 smithi161 ceph-mon[91579]: pgmap v3492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:27 smithi188 ceph-mon[92425]: pgmap v3492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:29 smithi161 ceph-mon[91579]: pgmap v3493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:29 smithi188 ceph-mon[92425]: pgmap v3493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:30 smithi161 ceph-mon[91579]: pgmap v3494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:30 smithi188 ceph-mon[92425]: pgmap v3494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:32 smithi161 ceph-mon[91579]: pgmap v3495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:32 smithi188 ceph-mon[92425]: pgmap v3495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:35 smithi188 ceph-mon[92425]: pgmap v3496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:35 smithi161 ceph-mon[91579]: pgmap v3496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:37 smithi188 ceph-mon[92425]: pgmap v3497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:37 smithi161 ceph-mon[91579]: pgmap v3497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:39 smithi161 ceph-mon[91579]: pgmap v3498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:39 smithi188 ceph-mon[92425]: pgmap v3498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:40 smithi161 ceph-mon[91579]: pgmap v3499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:40 smithi188 ceph-mon[92425]: pgmap v3499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:42 smithi161 ceph-mon[91579]: pgmap v3500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:42 smithi188 ceph-mon[92425]: pgmap v3500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:45 smithi188 ceph-mon[92425]: pgmap v3501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:45 smithi161 ceph-mon[91579]: pgmap v3501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:47 smithi188 ceph-mon[92425]: pgmap v3502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:47 smithi161 ceph-mon[91579]: pgmap v3502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:49 smithi188 ceph-mon[92425]: pgmap v3503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:49 smithi161 ceph-mon[91579]: pgmap v3503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:50 smithi161 ceph-mon[91579]: pgmap v3504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:50 smithi188 ceph-mon[92425]: pgmap v3504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:52 smithi161 ceph-mon[91579]: pgmap v3505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:52 smithi188 ceph-mon[92425]: pgmap v3505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:55 smithi188 ceph-mon[92425]: pgmap v3506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:55 smithi161 ceph-mon[91579]: pgmap v3506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:35:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:57 smithi188 ceph-mon[92425]: pgmap v3507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:57 smithi161 ceph-mon[91579]: pgmap v3507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:35:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:35:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:35:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:35:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:35:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:35:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:59 smithi188 ceph-mon[92425]: pgmap v3508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:35:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:59 smithi161 ceph-mon[91579]: pgmap v3508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:35:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:00 smithi161 ceph-mon[91579]: pgmap v3509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:00 smithi188 ceph-mon[92425]: pgmap v3509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:02 smithi161 ceph-mon[91579]: pgmap v3510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:02 smithi188 ceph-mon[92425]: pgmap v3510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:05 smithi188 ceph-mon[92425]: pgmap v3511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:05 smithi161 ceph-mon[91579]: pgmap v3511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:07 smithi188 ceph-mon[92425]: pgmap v3512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:07 smithi161 ceph-mon[91579]: pgmap v3512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:09 smithi188 ceph-mon[92425]: pgmap v3513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:09 smithi161 ceph-mon[91579]: pgmap v3513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:10 smithi161 ceph-mon[91579]: pgmap v3514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:10 smithi188 ceph-mon[92425]: pgmap v3514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:36:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:36:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:36:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:36:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:36:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:36:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:36:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:36:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:36:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:36:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:36:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:36:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:36:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:36:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:36:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:36:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:36:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:36:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:36:12.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:36:12.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:36:12.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:36:12.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:36:12.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:36:12.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:36:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:36:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:36:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:36:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:36:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:36:12.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:36:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:36:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:36:12.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:36:12.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:36:12.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:36:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:36:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:36:12.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:36:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:36:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:36:12.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:36:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:36:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:36:12.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:36:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:36:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:36:12.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=173] Local address: 172.21.15.161, port 52938 2022-07-03T12:36:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=173] Peer address: 172.21.15.101, port 22 2022-07-03T12:36:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=173] Connection lost 2022-07-03T12:36:12.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=173] Aborting connection 2022-07-03T12:36:12.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:36:11.710+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:36:12.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:36:12.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:36:12.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:36:12.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:36:12.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:36:12.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:36:12.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:36:12.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:36:12.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:36:12.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:36:12.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:36:12.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:36:12.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:36:12.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:36:12.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:36:12.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:36:12.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:36:12.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:36:12.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:36:12.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:36:12.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:36:12.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:36:12.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:36:12.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:36:12.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:36:12.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:36:12.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:36:12.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:36:12.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:36:12.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:36:12.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:36:12.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:36:12.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:36:12.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:36:12.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:36:12.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:36:12.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:36:12.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:36:12.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:36:12.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:36:12.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:36:12.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:36:12.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:36:12.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:36:12.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:36:12.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=173] Local address: 172.21.15.161, port 52938 2022-07-03T12:36:12.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=173] Peer address: 172.21.15.101, port 22 2022-07-03T12:36:12.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=173] Connection lost 2022-07-03T12:36:12.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=173] Aborting connection 2022-07-03T12:36:12.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:36:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:36:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:36:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:36:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:36:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:36:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:36:12.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:36:12.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:36:12.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:36:12.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:36:12.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:36:12.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:36:12.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:36:12.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:36:12.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:36:12.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:36:12.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:36:12.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:36:12.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:36:12.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:36:12.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:36:12.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:36:12.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:36:12.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:36:12.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:36:12.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:36:12.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:36:12.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:36:12.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:36:12.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:36:12.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:36:12.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:36:12.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:36:12.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:36:12.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:36:12.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:36:12.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:36:12.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:36:12.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:36:12.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:36:12.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:36:12.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:36:12.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:36:12.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:36:12.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:36:12.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:36:12.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:36:12.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:36:12.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=173] Local address: 172.21.15.161, port 52938 2022-07-03T12:36:12.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=173] Peer address: 172.21.15.101, port 22 2022-07-03T12:36:12.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=173] Connection lost 2022-07-03T12:36:12.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=173] Aborting connection 2022-07-03T12:36:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:12 smithi161 ceph-mon[91579]: pgmap v3515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:12 smithi188 ceph-mon[92425]: pgmap v3515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:14.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:15 smithi188 ceph-mon[92425]: pgmap v3516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:36:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:36:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:15 smithi161 ceph-mon[91579]: pgmap v3516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:36:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:36:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:16 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T12:36:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:16 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T12:36:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:17 smithi188 ceph-mon[92425]: pgmap v3517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:17 smithi161 ceph-mon[91579]: pgmap v3517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:19 smithi188 ceph-mon[92425]: pgmap v3518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:19 smithi161 ceph-mon[91579]: pgmap v3518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:20 smithi161 ceph-mon[91579]: pgmap v3519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:20 smithi188 ceph-mon[92425]: pgmap v3519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:22 smithi161 ceph-mon[91579]: pgmap v3520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:22 smithi188 ceph-mon[92425]: pgmap v3520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:25 smithi188 ceph-mon[92425]: pgmap v3521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:25 smithi161 ceph-mon[91579]: pgmap v3521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:27 smithi188 ceph-mon[92425]: pgmap v3522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:27 smithi161 ceph-mon[91579]: pgmap v3522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:29 smithi188 ceph-mon[92425]: pgmap v3523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:29 smithi161 ceph-mon[91579]: pgmap v3523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:30 smithi161 ceph-mon[91579]: pgmap v3524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:30 smithi188 ceph-mon[92425]: pgmap v3524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:32 smithi161 ceph-mon[91579]: pgmap v3525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:32 smithi188 ceph-mon[92425]: pgmap v3525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:35 smithi188 ceph-mon[92425]: pgmap v3526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:35 smithi161 ceph-mon[91579]: pgmap v3526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:37 smithi188 ceph-mon[92425]: pgmap v3527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:37 smithi161 ceph-mon[91579]: pgmap v3527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:39 smithi188 ceph-mon[92425]: pgmap v3528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:39 smithi161 ceph-mon[91579]: pgmap v3528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:40 smithi161 ceph-mon[91579]: pgmap v3529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:40 smithi188 ceph-mon[92425]: pgmap v3529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:42 smithi161 ceph-mon[91579]: pgmap v3530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:42 smithi188 ceph-mon[92425]: pgmap v3530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:45 smithi188 ceph-mon[92425]: pgmap v3531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:45 smithi161 ceph-mon[91579]: pgmap v3531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:47 smithi188 ceph-mon[92425]: pgmap v3532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:47 smithi161 ceph-mon[91579]: pgmap v3532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:49 smithi188 ceph-mon[92425]: pgmap v3533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:49 smithi161 ceph-mon[91579]: pgmap v3533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:50 smithi161 ceph-mon[91579]: pgmap v3534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:50 smithi188 ceph-mon[92425]: pgmap v3534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:52 smithi161 ceph-mon[91579]: pgmap v3535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:52 smithi188 ceph-mon[92425]: pgmap v3535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:55 smithi188 ceph-mon[92425]: pgmap v3536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:55 smithi161 ceph-mon[91579]: pgmap v3536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:36:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:57 smithi188 ceph-mon[92425]: pgmap v3537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:57 smithi161 ceph-mon[91579]: pgmap v3537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:36:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:36:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:36:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:36:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:36:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:36:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:36:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:36:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:36:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:59 smithi188 ceph-mon[92425]: pgmap v3538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:36:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:59 smithi161 ceph-mon[91579]: pgmap v3538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:36:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:00 smithi161 ceph-mon[91579]: pgmap v3539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:00 smithi188 ceph-mon[92425]: pgmap v3539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:02 smithi161 ceph-mon[91579]: pgmap v3540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:02 smithi188 ceph-mon[92425]: pgmap v3540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:05 smithi188 ceph-mon[92425]: pgmap v3541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:05.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:05 smithi161 ceph-mon[91579]: pgmap v3541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:07 smithi188 ceph-mon[92425]: pgmap v3542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:07 smithi161 ceph-mon[91579]: pgmap v3542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:09 smithi188 ceph-mon[92425]: pgmap v3543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:09 smithi161 ceph-mon[91579]: pgmap v3543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:10 smithi161 ceph-mon[91579]: pgmap v3544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:10 smithi188 ceph-mon[92425]: pgmap v3544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:12 smithi161 ceph-mon[91579]: pgmap v3545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:12 smithi188 ceph-mon[92425]: pgmap v3545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:15.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:15 smithi161 ceph-mon[91579]: pgmap v3546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:15.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:37:15.150 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:37:15.150 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:37:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:15 smithi188 ceph-mon[92425]: pgmap v3546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:37:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:37:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:37:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:37:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:37:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:17 smithi161 ceph-mon[91579]: pgmap v3547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:17 smithi188 ceph-mon[92425]: pgmap v3547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:37:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:37:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:37:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:37:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:18 smithi161 ceph-mon[91579]: pgmap v3548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:37:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:37:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:37:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:37:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:37:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:37:19.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:18 smithi188 ceph-mon[92425]: pgmap v3548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:20 smithi161 ceph-mon[91579]: pgmap v3549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:20 smithi188 ceph-mon[92425]: pgmap v3549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:22 smithi161 ceph-mon[91579]: pgmap v3550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:22 smithi188 ceph-mon[92425]: pgmap v3550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:25 smithi161 ceph-mon[91579]: pgmap v3551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:25 smithi188 ceph-mon[92425]: pgmap v3551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:27 smithi161 ceph-mon[91579]: pgmap v3552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:27 smithi188 ceph-mon[92425]: pgmap v3552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:29 smithi188 ceph-mon[92425]: pgmap v3553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:29 smithi161 ceph-mon[91579]: pgmap v3553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:30 smithi161 ceph-mon[91579]: pgmap v3554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:30 smithi188 ceph-mon[92425]: pgmap v3554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:32 smithi161 ceph-mon[91579]: pgmap v3555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:32 smithi188 ceph-mon[92425]: pgmap v3555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:35 smithi188 ceph-mon[92425]: pgmap v3556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:35 smithi161 ceph-mon[91579]: pgmap v3556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:37 smithi161 ceph-mon[91579]: pgmap v3557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:37 smithi188 ceph-mon[92425]: pgmap v3557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:39 smithi188 ceph-mon[92425]: pgmap v3558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:39 smithi161 ceph-mon[91579]: pgmap v3558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:40 smithi161 ceph-mon[91579]: pgmap v3559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:40 smithi188 ceph-mon[92425]: pgmap v3559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:42 smithi161 ceph-mon[91579]: pgmap v3560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:42 smithi188 ceph-mon[92425]: pgmap v3560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:45.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:37:44.884+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:37:45.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:37:45.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:37:45.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:37:45.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:37:45.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:37:45.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:37:45.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:37:45.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:37:45.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:37:45.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:37:45.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:37:45.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:37:45.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:37:45.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:37:45.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:37:45.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:37:45.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:37:45.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:37:45.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:37:45.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:37:45.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:37:45.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:37:45.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:37:45.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:37:45.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:37:45.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:37:45.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:37:45.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:37:45.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:37:45.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:37:45.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:37:45.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:37:45.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:37:45.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:37:45.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:37:45.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:37:45.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:37:45.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:37:45.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:37:45.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:37:45.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:37:45.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:37:45.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:37:45.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:37:45.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:37:45.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=175] Local address: 172.21.15.161, port 52968 2022-07-03T12:37:45.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=175] Peer address: 172.21.15.101, port 22 2022-07-03T12:37:45.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=508] Received exit status 0 2022-07-03T12:37:45.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=508] Received channel close 2022-07-03T12:37:45.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=508] Channel closed 2022-07-03T12:37:45.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=509] Requesting new SSH session 2022-07-03T12:37:45.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=509] Command: which python3 2022-07-03T12:37:45.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=512] Received exit status 0 2022-07-03T12:37:45.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=512] Received channel close 2022-07-03T12:37:45.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=512] Channel closed 2022-07-03T12:37:45.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=513] Requesting new SSH session 2022-07-03T12:37:45.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=513] Command: which python3 2022-07-03T12:37:45.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=509] Received exit status 0 2022-07-03T12:37:45.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=509] Received channel close 2022-07-03T12:37:45.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=509] Channel closed 2022-07-03T12:37:45.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=510] Requesting new SSH session 2022-07-03T12:37:45.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=510] Command: sudo true 2022-07-03T12:37:45.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=513] Received exit status 0 2022-07-03T12:37:45.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=513] Received channel close 2022-07-03T12:37:45.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=513] Channel closed 2022-07-03T12:37:45.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=514] Requesting new SSH session 2022-07-03T12:37:45.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=514] Command: sudo true 2022-07-03T12:37:45.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=510] Received exit status 0 2022-07-03T12:37:45.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=510] Received channel close 2022-07-03T12:37:45.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=510] Channel closed 2022-07-03T12:37:45.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=511] Requesting new SSH session 2022-07-03T12:37:45.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=511] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:45.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=514] Received exit status 0 2022-07-03T12:37:45.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=514] Received channel close 2022-07-03T12:37:45.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=514] Channel closed 2022-07-03T12:37:45.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=515] Requesting new SSH session 2022-07-03T12:37:45.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:45.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=511] Received exit status 0 2022-07-03T12:37:45.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=511] Received channel close 2022-07-03T12:37:45.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=511] Channel closed 2022-07-03T12:37:45.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=512] Requesting new SSH session 2022-07-03T12:37:45.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=512] Command: sudo true 2022-07-03T12:37:45.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=512] Received exit status 0 2022-07-03T12:37:45.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=512] Received channel close 2022-07-03T12:37:45.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=512] Channel closed 2022-07-03T12:37:45.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=513] Requesting new SSH session 2022-07-03T12:37:45.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=513] Command: which python3 2022-07-03T12:37:45.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=513] Received exit status 0 2022-07-03T12:37:45.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=513] Received channel close 2022-07-03T12:37:45.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=513] Channel closed 2022-07-03T12:37:45.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=514] Requesting new SSH session 2022-07-03T12:37:45.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=514] Command: sudo true 2022-07-03T12:37:45.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=514] Received exit status 0 2022-07-03T12:37:45.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=514] Received channel close 2022-07-03T12:37:45.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=514] Channel closed 2022-07-03T12:37:45.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=515] Requesting new SSH session 2022-07-03T12:37:45.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:45.191 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=515] Received exit status 0 2022-07-03T12:37:45.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=515] Received channel close 2022-07-03T12:37:45.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=515] Channel closed 2022-07-03T12:37:45.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=516] Requesting new SSH session 2022-07-03T12:37:45.192 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=516] Command: sudo true 2022-07-03T12:37:45.193 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=516] Received exit status 0 2022-07-03T12:37:45.193 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=516] Received channel close 2022-07-03T12:37:45.193 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=516] Channel closed 2022-07-03T12:37:45.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=517] Requesting new SSH session 2022-07-03T12:37:45.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=517] Command: which python3 2022-07-03T12:37:45.194 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=517] Received exit status 0 2022-07-03T12:37:45.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=517] Received channel close 2022-07-03T12:37:45.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=517] Channel closed 2022-07-03T12:37:45.195 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=518] Requesting new SSH session 2022-07-03T12:37:45.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=518] Command: sudo true 2022-07-03T12:37:45.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=518] Received exit status 0 2022-07-03T12:37:45.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=518] Received channel close 2022-07-03T12:37:45.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=518] Channel closed 2022-07-03T12:37:45.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=519] Requesting new SSH session 2022-07-03T12:37:45.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:45.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=515] Received exit status 0 2022-07-03T12:37:45.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=515] Received channel close 2022-07-03T12:37:45.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=515] Channel closed 2022-07-03T12:37:45.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=519] Received exit status 0 2022-07-03T12:37:45.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=519] Received channel close 2022-07-03T12:37:45.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=519] Channel closed 2022-07-03T12:37:45.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=516] Requesting new SSH session 2022-07-03T12:37:45.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=516] Command: sudo true 2022-07-03T12:37:45.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=520] Requesting new SSH session 2022-07-03T12:37:45.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=520] Command: sudo true 2022-07-03T12:37:45.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=516] Received exit status 0 2022-07-03T12:37:45.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=516] Received channel close 2022-07-03T12:37:45.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=516] Channel closed 2022-07-03T12:37:45.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=517] Requesting new SSH session 2022-07-03T12:37:45.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=517] Command: which python3 2022-07-03T12:37:45.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=520] Received exit status 0 2022-07-03T12:37:45.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=520] Received channel close 2022-07-03T12:37:45.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=520] Channel closed 2022-07-03T12:37:45.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=521] Requesting new SSH session 2022-07-03T12:37:45.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=521] Command: which python3 2022-07-03T12:37:45.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=517] Received exit status 0 2022-07-03T12:37:45.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=517] Received channel close 2022-07-03T12:37:45.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=517] Channel closed 2022-07-03T12:37:45.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=518] Requesting new SSH session 2022-07-03T12:37:45.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=518] Command: sudo true 2022-07-03T12:37:45.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=521] Received exit status 0 2022-07-03T12:37:45.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=521] Received channel close 2022-07-03T12:37:45.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=521] Channel closed 2022-07-03T12:37:45.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=522] Requesting new SSH session 2022-07-03T12:37:45.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=522] Command: sudo true 2022-07-03T12:37:45.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=518] Received exit status 0 2022-07-03T12:37:45.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=518] Received channel close 2022-07-03T12:37:45.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=518] Channel closed 2022-07-03T12:37:45.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=519] Requesting new SSH session 2022-07-03T12:37:45.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:45.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=522] Received exit status 0 2022-07-03T12:37:45.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=522] Received channel close 2022-07-03T12:37:45.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=522] Channel closed 2022-07-03T12:37:45.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=523] Requesting new SSH session 2022-07-03T12:37:45.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=523] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:45.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=519] Received exit status 0 2022-07-03T12:37:45.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=519] Received channel close 2022-07-03T12:37:45.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=519] Channel closed 2022-07-03T12:37:45.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=523] Received exit status 0 2022-07-03T12:37:45.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=523] Received channel close 2022-07-03T12:37:45.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=523] Channel closed 2022-07-03T12:37:45.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=175] Login timeout expired 2022-07-03T12:37:45.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=175] Aborting connection 2022-07-03T12:37:45.214 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:37:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:37:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:45 smithi188 ceph-mon[92425]: pgmap v3561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:45 smithi161 ceph-mon[91579]: pgmap v3561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:37:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:37:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:37:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:37:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:37:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:37:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:37:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:37:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:37:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:37:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:37:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:37:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:37:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:37:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:37:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:37:46.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:37:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:37:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:37:46.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:37:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:37:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:37:46.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:37:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:37:46.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:37:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:37:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:37:46.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:37:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:37:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:37:46.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:37:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:37:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:37:46.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:37:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:37:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:37:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:37:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:37:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:37:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:37:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:37:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:37:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:37:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:37:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:37:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:37:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=175] Local address: 172.21.15.161, port 52968 2022-07-03T12:37:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=175] Peer address: 172.21.15.101, port 22 2022-07-03T12:37:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=508] Received exit status 0 2022-07-03T12:37:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=508] Received channel close 2022-07-03T12:37:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=508] Channel closed 2022-07-03T12:37:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=509] Requesting new SSH session 2022-07-03T12:37:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=509] Command: which python3 2022-07-03T12:37:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=512] Received exit status 0 2022-07-03T12:37:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=512] Received channel close 2022-07-03T12:37:46.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=512] Channel closed 2022-07-03T12:37:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=513] Requesting new SSH session 2022-07-03T12:37:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=513] Command: which python3 2022-07-03T12:37:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=509] Received exit status 0 2022-07-03T12:37:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=509] Received channel close 2022-07-03T12:37:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=509] Channel closed 2022-07-03T12:37:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=510] Requesting new SSH session 2022-07-03T12:37:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=510] Command: sudo true 2022-07-03T12:37:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=513] Received exit status 0 2022-07-03T12:37:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=513] Received channel close 2022-07-03T12:37:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=513] Channel closed 2022-07-03T12:37:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=514] Requesting new SSH session 2022-07-03T12:37:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=514] Command: sudo true 2022-07-03T12:37:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=510] Received exit status 0 2022-07-03T12:37:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=510] Received channel close 2022-07-03T12:37:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=510] Channel closed 2022-07-03T12:37:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=511] Requesting new SSH session 2022-07-03T12:37:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=511] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=514] Received exit status 0 2022-07-03T12:37:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=514] Received channel close 2022-07-03T12:37:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=514] Channel closed 2022-07-03T12:37:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=515] Requesting new SSH session 2022-07-03T12:37:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=511] Received exit status 0 2022-07-03T12:37:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=511] Received channel close 2022-07-03T12:37:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=511] Channel closed 2022-07-03T12:37:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=512] Requesting new SSH session 2022-07-03T12:37:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=512] Command: sudo true 2022-07-03T12:37:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=512] Received exit status 0 2022-07-03T12:37:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=512] Received channel close 2022-07-03T12:37:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=512] Channel closed 2022-07-03T12:37:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=513] Requesting new SSH session 2022-07-03T12:37:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=513] Command: which python3 2022-07-03T12:37:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=513] Received exit status 0 2022-07-03T12:37:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=513] Received channel close 2022-07-03T12:37:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=513] Channel closed 2022-07-03T12:37:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=514] Requesting new SSH session 2022-07-03T12:37:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=514] Command: sudo true 2022-07-03T12:37:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=514] Received exit status 0 2022-07-03T12:37:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=514] Received channel close 2022-07-03T12:37:46.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=514] Channel closed 2022-07-03T12:37:46.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=515] Requesting new SSH session 2022-07-03T12:37:46.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:46.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=515] Received exit status 0 2022-07-03T12:37:46.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=515] Received channel close 2022-07-03T12:37:46.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=515] Channel closed 2022-07-03T12:37:46.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=516] Requesting new SSH session 2022-07-03T12:37:46.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=516] Command: sudo true 2022-07-03T12:37:46.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=516] Received exit status 0 2022-07-03T12:37:46.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=516] Received channel close 2022-07-03T12:37:46.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=516] Channel closed 2022-07-03T12:37:46.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=517] Requesting new SSH session 2022-07-03T12:37:46.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=517] Command: which python3 2022-07-03T12:37:46.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=517] Received exit status 0 2022-07-03T12:37:46.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=517] Received channel close 2022-07-03T12:37:46.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=517] Channel closed 2022-07-03T12:37:46.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=518] Requesting new SSH session 2022-07-03T12:37:46.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=518] Command: sudo true 2022-07-03T12:37:46.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=518] Received exit status 0 2022-07-03T12:37:46.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=518] Received channel close 2022-07-03T12:37:46.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=518] Channel closed 2022-07-03T12:37:46.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=519] Requesting new SSH session 2022-07-03T12:37:46.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:46.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=515] Received exit status 0 2022-07-03T12:37:46.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=515] Received channel close 2022-07-03T12:37:46.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=515] Channel closed 2022-07-03T12:37:46.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=519] Received exit status 0 2022-07-03T12:37:46.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=519] Received channel close 2022-07-03T12:37:46.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=519] Channel closed 2022-07-03T12:37:46.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=516] Requesting new SSH session 2022-07-03T12:37:46.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=516] Command: sudo true 2022-07-03T12:37:46.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=520] Requesting new SSH session 2022-07-03T12:37:46.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=520] Command: sudo true 2022-07-03T12:37:46.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=516] Received exit status 0 2022-07-03T12:37:46.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=516] Received channel close 2022-07-03T12:37:46.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=516] Channel closed 2022-07-03T12:37:46.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=517] Requesting new SSH session 2022-07-03T12:37:46.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=517] Command: which python3 2022-07-03T12:37:46.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=520] Received exit status 0 2022-07-03T12:37:46.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=520] Received channel close 2022-07-03T12:37:46.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=520] Channel closed 2022-07-03T12:37:46.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=521] Requesting new SSH session 2022-07-03T12:37:46.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=521] Command: which python3 2022-07-03T12:37:46.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=517] Received exit status 0 2022-07-03T12:37:46.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=517] Received channel close 2022-07-03T12:37:46.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=517] Channel closed 2022-07-03T12:37:46.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=518] Requesting new SSH session 2022-07-03T12:37:46.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=518] Command: sudo true 2022-07-03T12:37:46.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=521] Received exit status 0 2022-07-03T12:37:46.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=521] Received channel close 2022-07-03T12:37:46.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=521] Channel closed 2022-07-03T12:37:46.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=522] Requesting new SSH session 2022-07-03T12:37:46.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=522] Command: sudo true 2022-07-03T12:37:46.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=518] Received exit status 0 2022-07-03T12:37:46.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=518] Received channel close 2022-07-03T12:37:46.466 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=518] Channel closed 2022-07-03T12:37:46.466 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=519] Requesting new SSH session 2022-07-03T12:37:46.466 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:46.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=522] Received exit status 0 2022-07-03T12:37:46.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=522] Received channel close 2022-07-03T12:37:46.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=522] Channel closed 2022-07-03T12:37:46.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=523] Requesting new SSH session 2022-07-03T12:37:46.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=523] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:46.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=519] Received exit status 0 2022-07-03T12:37:46.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=519] Received channel close 2022-07-03T12:37:46.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=519] Channel closed 2022-07-03T12:37:46.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=523] Received exit status 0 2022-07-03T12:37:46.470 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=523] Received channel close 2022-07-03T12:37:46.470 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=523] Channel closed 2022-07-03T12:37:46.471 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=175] Login timeout expired 2022-07-03T12:37:46.471 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=175] Aborting connection 2022-07-03T12:37:46.472 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:37:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:37:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:37:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:37:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:37:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:37:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:37:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:37:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:37:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:37:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:37:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:37:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:37:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:37:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:37:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:37:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:37:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:37:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:37:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:37:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:37:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:37:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:37:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:37:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:37:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:37:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:37:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:37:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:37:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:37:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:37:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:37:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:37:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:37:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:37:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:37:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:37:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:37:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:37:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:37:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:37:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:37:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:37:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:37:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:37:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:37:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=175] Local address: 172.21.15.161, port 52968 2022-07-03T12:37:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=175] Peer address: 172.21.15.101, port 22 2022-07-03T12:37:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=508] Received exit status 0 2022-07-03T12:37:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=508] Received channel close 2022-07-03T12:37:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=508] Channel closed 2022-07-03T12:37:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=509] Requesting new SSH session 2022-07-03T12:37:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=509] Command: which python3 2022-07-03T12:37:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=512] Received exit status 0 2022-07-03T12:37:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=512] Received channel close 2022-07-03T12:37:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=512] Channel closed 2022-07-03T12:37:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=513] Requesting new SSH session 2022-07-03T12:37:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=513] Command: which python3 2022-07-03T12:37:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=509] Received exit status 0 2022-07-03T12:37:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=509] Received channel close 2022-07-03T12:37:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=509] Channel closed 2022-07-03T12:37:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=510] Requesting new SSH session 2022-07-03T12:37:46.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=510] Command: sudo true 2022-07-03T12:37:46.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=513] Received exit status 0 2022-07-03T12:37:46.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=513] Received channel close 2022-07-03T12:37:46.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=513] Channel closed 2022-07-03T12:37:46.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=514] Requesting new SSH session 2022-07-03T12:37:46.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=514] Command: sudo true 2022-07-03T12:37:46.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=510] Received exit status 0 2022-07-03T12:37:46.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=510] Received channel close 2022-07-03T12:37:46.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=510] Channel closed 2022-07-03T12:37:46.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=511] Requesting new SSH session 2022-07-03T12:37:46.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=511] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:46.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=514] Received exit status 0 2022-07-03T12:37:46.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=514] Received channel close 2022-07-03T12:37:46.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=514] Channel closed 2022-07-03T12:37:46.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=515] Requesting new SSH session 2022-07-03T12:37:46.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T12:37:46.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=511] Received exit status 0 2022-07-03T12:37:46.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=511] Received channel close 2022-07-03T12:37:46.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=511] Channel closed 2022-07-03T12:37:46.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=512] Requesting new SSH session 2022-07-03T12:37:46.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=512] Command: sudo true 2022-07-03T12:37:46.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=512] Received exit status 0 2022-07-03T12:37:46.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=512] Received channel close 2022-07-03T12:37:46.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=512] Channel closed 2022-07-03T12:37:46.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=513] Requesting new SSH session 2022-07-03T12:37:46.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=513] Command: which python3 2022-07-03T12:37:46.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=513] Received exit status 0 2022-07-03T12:37:46.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=513] Received channel close 2022-07-03T12:37:46.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=513] Channel closed 2022-07-03T12:37:46.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=514] Requesting new SSH session 2022-07-03T12:37:46.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=514] Command: sudo true 2022-07-03T12:37:46.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=514] Received exit status 0 2022-07-03T12:37:46.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=514] Received channel close 2022-07-03T12:37:46.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=514] Channel closed 2022-07-03T12:37:46.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=515] Requesting new SSH session 2022-07-03T12:37:46.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=515] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:46.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=515] Received exit status 0 2022-07-03T12:37:46.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=515] Received channel close 2022-07-03T12:37:46.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=515] Channel closed 2022-07-03T12:37:46.666 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=516] Requesting new SSH session 2022-07-03T12:37:46.666 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=516] Command: sudo true 2022-07-03T12:37:46.666 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=516] Received exit status 0 2022-07-03T12:37:46.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=516] Received channel close 2022-07-03T12:37:46.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=516] Channel closed 2022-07-03T12:37:46.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=517] Requesting new SSH session 2022-07-03T12:37:46.668 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=517] Command: which python3 2022-07-03T12:37:46.668 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=517] Received exit status 0 2022-07-03T12:37:46.669 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=517] Received channel close 2022-07-03T12:37:46.669 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=517] Channel closed 2022-07-03T12:37:46.669 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=518] Requesting new SSH session 2022-07-03T12:37:46.670 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=518] Command: sudo true 2022-07-03T12:37:46.670 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=518] Received exit status 0 2022-07-03T12:37:46.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=518] Received channel close 2022-07-03T12:37:46.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=518] Channel closed 2022-07-03T12:37:46.671 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=519] Requesting new SSH session 2022-07-03T12:37:46.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ls 2022-07-03T12:37:46.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=515] Received exit status 0 2022-07-03T12:37:46.672 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=515] Received channel close 2022-07-03T12:37:46.673 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=515] Channel closed 2022-07-03T12:37:46.673 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=519] Received exit status 0 2022-07-03T12:37:46.673 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=519] Received channel close 2022-07-03T12:37:46.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=519] Channel closed 2022-07-03T12:37:46.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=516] Requesting new SSH session 2022-07-03T12:37:46.674 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=516] Command: sudo true 2022-07-03T12:37:46.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=520] Requesting new SSH session 2022-07-03T12:37:46.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=520] Command: sudo true 2022-07-03T12:37:46.675 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=516] Received exit status 0 2022-07-03T12:37:46.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=516] Received channel close 2022-07-03T12:37:46.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=516] Channel closed 2022-07-03T12:37:46.676 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=517] Requesting new SSH session 2022-07-03T12:37:46.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=517] Command: which python3 2022-07-03T12:37:46.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=520] Received exit status 0 2022-07-03T12:37:46.677 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=520] Received channel close 2022-07-03T12:37:46.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=520] Channel closed 2022-07-03T12:37:46.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=521] Requesting new SSH session 2022-07-03T12:37:46.678 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=521] Command: which python3 2022-07-03T12:37:46.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=517] Received exit status 0 2022-07-03T12:37:46.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=517] Received channel close 2022-07-03T12:37:46.679 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=517] Channel closed 2022-07-03T12:37:46.680 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=518] Requesting new SSH session 2022-07-03T12:37:46.680 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=518] Command: sudo true 2022-07-03T12:37:46.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=521] Received exit status 0 2022-07-03T12:37:46.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=521] Received channel close 2022-07-03T12:37:46.681 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=521] Channel closed 2022-07-03T12:37:46.682 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=522] Requesting new SSH session 2022-07-03T12:37:46.682 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=522] Command: sudo true 2022-07-03T12:37:46.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=518] Received exit status 0 2022-07-03T12:37:46.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=518] Received channel close 2022-07-03T12:37:46.683 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=518] Channel closed 2022-07-03T12:37:46.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=519] Requesting new SSH session 2022-07-03T12:37:46.684 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=519] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:46.685 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=522] Received exit status 0 2022-07-03T12:37:46.685 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=522] Received channel close 2022-07-03T12:37:46.686 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=522] Channel closed 2022-07-03T12:37:46.686 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=523] Requesting new SSH session 2022-07-03T12:37:46.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=523] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:37:46.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=519] Received exit status 0 2022-07-03T12:37:46.687 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=519] Received channel close 2022-07-03T12:37:46.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=519] Channel closed 2022-07-03T12:37:46.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=523] Received exit status 0 2022-07-03T12:37:46.688 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=523] Received channel close 2022-07-03T12:37:46.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=523] Channel closed 2022-07-03T12:37:46.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=175] Login timeout expired 2022-07-03T12:37:46.689 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=175] Aborting connection 2022-07-03T12:37:46.690 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:47 smithi188 ceph-mon[92425]: pgmap v3562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:47 smithi161 ceph-mon[91579]: pgmap v3562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:49 smithi188 ceph-mon[92425]: pgmap v3563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:49 smithi161 ceph-mon[91579]: pgmap v3563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:51 smithi188 ceph-mon[92425]: pgmap v3564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:51 smithi161 ceph-mon[91579]: pgmap v3564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:54 smithi188 ceph-mon[92425]: pgmap v3565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:54 smithi161 ceph-mon[91579]: pgmap v3565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:37:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:56 smithi188 ceph-mon[92425]: pgmap v3566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:56 smithi161 ceph-mon[91579]: pgmap v3566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:37:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:37:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:37:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:37:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:37:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:37:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:37:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:37:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:37:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:58 smithi161 ceph-mon[91579]: pgmap v3567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:58 smithi188 ceph-mon[92425]: pgmap v3567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:37:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:37:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:00 smithi161 ceph-mon[91579]: pgmap v3568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:00 smithi188 ceph-mon[92425]: pgmap v3568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:02 smithi161 ceph-mon[91579]: pgmap v3569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:02 smithi188 ceph-mon[92425]: pgmap v3569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:04 smithi161 ceph-mon[91579]: pgmap v3570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:04 smithi188 ceph-mon[92425]: pgmap v3570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:06 smithi161 ceph-mon[91579]: pgmap v3571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:06 smithi188 ceph-mon[92425]: pgmap v3571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:08 smithi161 ceph-mon[91579]: pgmap v3572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:08 smithi188 ceph-mon[92425]: pgmap v3572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:10 smithi161 ceph-mon[91579]: pgmap v3573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:10 smithi188 ceph-mon[92425]: pgmap v3573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:12 smithi161 ceph-mon[91579]: pgmap v3574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:12 smithi188 ceph-mon[92425]: pgmap v3574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:14 smithi161 ceph-mon[91579]: pgmap v3575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:14 smithi188 ceph-mon[92425]: pgmap v3575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:16 smithi161 ceph-mon[91579]: pgmap v3576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:16 smithi188 ceph-mon[92425]: pgmap v3576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:18 smithi161 ceph-mon[91579]: pgmap v3577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:18 smithi188 ceph-mon[92425]: pgmap v3577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:20 smithi161 ceph-mon[91579]: pgmap v3578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:20 smithi188 ceph-mon[92425]: pgmap v3578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:22 smithi161 ceph-mon[91579]: pgmap v3579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:22 smithi188 ceph-mon[92425]: pgmap v3579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:24 smithi161 ceph-mon[91579]: pgmap v3580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:24 smithi188 ceph-mon[92425]: pgmap v3580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:26 smithi161 ceph-mon[91579]: pgmap v3581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:26 smithi188 ceph-mon[92425]: pgmap v3581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:28 smithi161 ceph-mon[91579]: pgmap v3582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:28 smithi188 ceph-mon[92425]: pgmap v3582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:30 smithi161 ceph-mon[91579]: pgmap v3583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:30 smithi188 ceph-mon[92425]: pgmap v3583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:32 smithi161 ceph-mon[91579]: pgmap v3584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:32 smithi188 ceph-mon[92425]: pgmap v3584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:34 smithi161 ceph-mon[91579]: pgmap v3585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:34 smithi188 ceph-mon[92425]: pgmap v3585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:36 smithi161 ceph-mon[91579]: pgmap v3586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:36 smithi188 ceph-mon[92425]: pgmap v3586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:38 smithi161 ceph-mon[91579]: pgmap v3587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:38 smithi188 ceph-mon[92425]: pgmap v3587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:40 smithi161 ceph-mon[91579]: pgmap v3588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:40 smithi188 ceph-mon[92425]: pgmap v3588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:42 smithi161 ceph-mon[91579]: pgmap v3589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:42 smithi188 ceph-mon[92425]: pgmap v3589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-mon[91579]: pgmap v3590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:44 smithi188 ceph-mon[92425]: pgmap v3590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:45.281 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:38:44.918+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:38:45.281 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:38:45.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:38:45.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:38:45.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:38:45.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:38:45.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:38:45.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:38:45.284 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:38:45.284 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:38:45.285 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:38:45.285 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:38:45.285 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:38:45.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:38:45.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:38:45.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:38:45.287 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:38:45.287 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:38:45.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:38:45.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:38:45.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:38:45.289 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:38:45.289 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:38:45.289 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:38:45.290 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:38:45.290 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:38:45.291 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:38:45.291 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:38:45.292 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:38:45.292 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:38:45.292 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:38:45.293 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:38:45.293 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:38:45.293 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:38:45.294 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:38:45.294 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:38:45.295 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:38:45.295 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:38:45.295 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:38:45.296 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:38:45.296 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:38:45.296 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:38:45.297 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:38:45.297 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:38:45.297 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:38:45.298 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:38:45.298 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=177] Local address: 172.21.15.161, port 53008 2022-07-03T12:38:45.299 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=177] Peer address: 172.21.15.101, port 22 2022-07-03T12:38:45.299 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=177] Connection lost 2022-07-03T12:38:45.299 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=177] Aborting connection 2022-07-03T12:38:45.300 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:38:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:38:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:38:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:38:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:38:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:38:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:38:46.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:46 smithi188 ceph-mon[92425]: pgmap v3591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:46.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:38:46.516 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:38:46.517 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:38:46.517 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:38:46.517 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:38:46.518 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:38:46.518 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:38:46.518 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:38:46.519 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:38:46.519 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:38:46.519 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:38:46.520 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:38:46.520 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:38:46.521 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:38:46.521 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:38:46.521 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:38:46.522 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:38:46.522 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:38:46.522 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:38:46.523 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:38:46.523 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:38:46.523 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:38:46.524 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:38:46.524 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:38:46.524 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:38:46.525 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:38:46.525 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:38:46.525 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:38:46.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:38:46.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:38:46.526 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:38:46.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:38:46.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:38:46.527 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:38:46.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:38:46.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:38:46.528 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:38:46.529 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:38:46.529 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:38:46.529 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:38:46.530 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:38:46.530 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:38:46.530 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:38:46.531 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:38:46.531 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:38:46.531 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:38:46.532 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:38:46.532 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=177] Local address: 172.21.15.161, port 53008 2022-07-03T12:38:46.532 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=177] Peer address: 172.21.15.101, port 22 2022-07-03T12:38:46.533 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=177] Connection lost 2022-07-03T12:38:46.533 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=177] Aborting connection 2022-07-03T12:38:46.533 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:46 smithi161 ceph-mon[91579]: pgmap v3591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:38:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:38:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:38:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:38:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:38:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:38:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:38:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:38:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:38:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:38:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:38:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:38:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:38:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:38:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:38:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:38:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:38:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:38:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:38:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:38:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:38:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:38:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:38:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:38:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:38:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:38:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:38:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:38:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:38:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:38:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:38:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:38:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:38:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:38:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:38:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:38:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:38:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:38:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:38:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:38:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:38:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:38:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:38:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:38:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:38:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:38:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:38:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=177] Local address: 172.21.15.161, port 53008 2022-07-03T12:38:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=177] Peer address: 172.21.15.101, port 22 2022-07-03T12:38:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=177] Connection lost 2022-07-03T12:38:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=177] Aborting connection 2022-07-03T12:38:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:48 smithi161 ceph-mon[91579]: pgmap v3592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:38:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:38:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:48 smithi188 ceph-mon[92425]: pgmap v3592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:38:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:38:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:49 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T12:38:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:49 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T12:38:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:50 smithi161 ceph-mon[91579]: pgmap v3593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:50 smithi188 ceph-mon[92425]: pgmap v3593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:52 smithi161 ceph-mon[91579]: pgmap v3594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:52 smithi188 ceph-mon[92425]: pgmap v3594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:54 smithi161 ceph-mon[91579]: pgmap v3595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:54 smithi188 ceph-mon[92425]: pgmap v3595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:38:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:56 smithi161 ceph-mon[91579]: pgmap v3596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:56 smithi188 ceph-mon[92425]: pgmap v3596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:38:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:38:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:38:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:38:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:38:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:38:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:38:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:38:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:38:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:58 smithi161 ceph-mon[91579]: pgmap v3597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:58 smithi188 ceph-mon[92425]: pgmap v3597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:38:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:38:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:00 smithi161 ceph-mon[91579]: pgmap v3598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:00 smithi188 ceph-mon[92425]: pgmap v3598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:02 smithi161 ceph-mon[91579]: pgmap v3599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:02 smithi188 ceph-mon[92425]: pgmap v3599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:04 smithi161 ceph-mon[91579]: pgmap v3600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:04 smithi188 ceph-mon[92425]: pgmap v3600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:06 smithi161 ceph-mon[91579]: pgmap v3601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:06 smithi188 ceph-mon[92425]: pgmap v3601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:08 smithi161 ceph-mon[91579]: pgmap v3602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:08 smithi188 ceph-mon[92425]: pgmap v3602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:10 smithi161 ceph-mon[91579]: pgmap v3603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:10 smithi188 ceph-mon[92425]: pgmap v3603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:12 smithi161 ceph-mon[91579]: pgmap v3604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:12 smithi188 ceph-mon[92425]: pgmap v3604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:14 smithi161 ceph-mon[91579]: pgmap v3605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:14 smithi188 ceph-mon[92425]: pgmap v3605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:16 smithi161 ceph-mon[91579]: pgmap v3606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:16 smithi188 ceph-mon[92425]: pgmap v3606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:18 smithi161 ceph-mon[91579]: pgmap v3607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:18 smithi188 ceph-mon[92425]: pgmap v3607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:20 smithi161 ceph-mon[91579]: pgmap v3608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:20.696 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:20 smithi188 ceph-mon[92425]: pgmap v3608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:22 smithi161 ceph-mon[91579]: pgmap v3609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:22 smithi188 ceph-mon[92425]: pgmap v3609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:24 smithi161 ceph-mon[91579]: pgmap v3610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:24 smithi188 ceph-mon[92425]: pgmap v3610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:26 smithi188 ceph-mon[92425]: pgmap v3611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:26 smithi161 ceph-mon[91579]: pgmap v3611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:28 smithi188 ceph-mon[92425]: pgmap v3612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:28 smithi161 ceph-mon[91579]: pgmap v3612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:30 smithi188 ceph-mon[92425]: pgmap v3613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:30 smithi161 ceph-mon[91579]: pgmap v3613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:32 smithi188 ceph-mon[92425]: pgmap v3614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:32 smithi161 ceph-mon[91579]: pgmap v3614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:34 smithi188 ceph-mon[92425]: pgmap v3615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:34 smithi161 ceph-mon[91579]: pgmap v3615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:36 smithi188 ceph-mon[92425]: pgmap v3616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:36 smithi161 ceph-mon[91579]: pgmap v3616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:38 smithi188 ceph-mon[92425]: pgmap v3617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:38 smithi161 ceph-mon[91579]: pgmap v3617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:40 smithi161 ceph-mon[91579]: pgmap v3618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:40 smithi188 ceph-mon[92425]: pgmap v3618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:42 smithi161 ceph-mon[91579]: pgmap v3619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:42 smithi188 ceph-mon[92425]: pgmap v3619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:44 smithi161 ceph-mon[91579]: pgmap v3620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:44 smithi188 ceph-mon[92425]: pgmap v3620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:46 smithi161 ceph-mon[91579]: pgmap v3621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:46 smithi188 ceph-mon[92425]: pgmap v3621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:48 smithi161 ceph-mon[91579]: pgmap v3622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:39:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:39:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:39:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:48 smithi188 ceph-mon[92425]: pgmap v3622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:39:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:39:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:39:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:50 smithi161 ceph-mon[91579]: pgmap v3623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:50 smithi188 ceph-mon[92425]: pgmap v3623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:52 smithi161 ceph-mon[91579]: pgmap v3624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:52 smithi188 ceph-mon[92425]: pgmap v3624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:54 smithi161 ceph-mon[91579]: pgmap v3625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:54 smithi188 ceph-mon[92425]: pgmap v3625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:39:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:56 smithi161 ceph-mon[91579]: pgmap v3626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:56 smithi188 ceph-mon[92425]: pgmap v3626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:39:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:39:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:39:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:39:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:39:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:39:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:39:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:39:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:39:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:58 smithi161 ceph-mon[91579]: pgmap v3627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:58 smithi188 ceph-mon[92425]: pgmap v3627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:39:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:39:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:00 smithi161 ceph-mon[91579]: pgmap v3628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:40:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:00 smithi188 ceph-mon[92425]: pgmap v3628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:40:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:02 smithi161 ceph-mon[91579]: pgmap v3629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:02 smithi188 ceph-mon[92425]: pgmap v3629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:04 smithi161 ceph-mon[91579]: pgmap v3630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:04 smithi188 ceph-mon[92425]: pgmap v3630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:06 smithi161 ceph-mon[91579]: pgmap v3631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:06 smithi188 ceph-mon[92425]: pgmap v3631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:08 smithi161 ceph-mon[91579]: pgmap v3632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:08 smithi188 ceph-mon[92425]: pgmap v3632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:10 smithi161 ceph-mon[91579]: pgmap v3633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:10 smithi188 ceph-mon[92425]: pgmap v3633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:12 smithi161 ceph-mon[91579]: pgmap v3634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:12 smithi188 ceph-mon[92425]: pgmap v3634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:14.839 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:14 smithi188 ceph-mon[92425]: pgmap v3635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:14.840 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:14 smithi161 ceph-mon[91579]: pgmap v3635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:16 smithi161 ceph-mon[91579]: pgmap v3636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:16 smithi188 ceph-mon[92425]: pgmap v3636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-mon[91579]: pgmap v3637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:18.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:40:18.317+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:40:18.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:40:18.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:40:18.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:40:18.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:40:18.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:40:18.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:40:18.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:40:18.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:40:18.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:40:18.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:40:18.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:40:18.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:40:18.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:40:18.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:40:18.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:40:18.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:40:18.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:40:18.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:40:18.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:40:18.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:40:18.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:40:18.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:40:18.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:40:18.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:40:18.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:40:18.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:40:18.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:40:18.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:40:18.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:40:18.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:40:18.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:40:18.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:40:18.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:40:18.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:40:18.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:40:18.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:40:18.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:40:18.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:40:18.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:40:18.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:40:18.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:40:18.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:40:18.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:40:18.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:40:18.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:40:18.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=179] Local address: 172.21.15.161, port 53072 2022-07-03T12:40:18.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=179] Peer address: 172.21.15.101, port 22 2022-07-03T12:40:18.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=179] Connection lost 2022-07-03T12:40:18.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=179] Aborting connection 2022-07-03T12:40:18.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:40:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:40:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:18 smithi188 ceph-mon[92425]: pgmap v3637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:19 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:40:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:19 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:40:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:40:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:40:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:40:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:40:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:40:19.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:40:19.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:40:19.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:40:19.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:40:19.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:40:19.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:40:19.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:40:19.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:40:19.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:40:19.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:40:19.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:40:19.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:40:19.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:40:19.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:40:19.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:40:19.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:40:19.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:40:19.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:40:19.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:40:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:40:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:40:19.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:40:19.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:40:19.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:40:19.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:40:19.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:40:19.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:40:19.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:40:19.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:40:19.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:40:19.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:40:19.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:40:19.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:40:19.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:40:19.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:40:19.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:40:19.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:40:19.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:40:19.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:40:19.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:40:19.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=179] Local address: 172.21.15.161, port 53072 2022-07-03T12:40:19.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=179] Peer address: 172.21.15.101, port 22 2022-07-03T12:40:19.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=179] Connection lost 2022-07-03T12:40:19.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=179] Aborting connection 2022-07-03T12:40:19.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:19 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:40:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:40:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:40:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:40:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:40:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:40:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:40:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:40:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:40:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:40:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:40:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:40:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:40:19.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:40:19.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:40:19.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:40:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:40:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:40:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:40:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:40:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:40:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:40:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:40:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:40:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:40:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:40:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:40:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:40:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:40:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:40:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:40:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:40:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:40:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:40:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:40:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:40:19.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:40:19.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:40:19.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:40:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:40:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:40:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:40:19.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:40:19.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:40:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:40:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:40:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:40:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=179] Local address: 172.21.15.161, port 53072 2022-07-03T12:40:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=179] Peer address: 172.21.15.101, port 22 2022-07-03T12:40:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=179] Connection lost 2022-07-03T12:40:19.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=179] Aborting connection 2022-07-03T12:40:19.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:20 smithi161 ceph-mon[91579]: pgmap v3638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:20 smithi188 ceph-mon[92425]: pgmap v3638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:22 smithi161 ceph-mon[91579]: pgmap v3639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:22 smithi188 ceph-mon[92425]: pgmap v3639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:24 smithi161 ceph-mon[91579]: pgmap v3640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:24 smithi188 ceph-mon[92425]: pgmap v3640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:26 smithi161 ceph-mon[91579]: pgmap v3641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:26 smithi188 ceph-mon[92425]: pgmap v3641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:28 smithi161 ceph-mon[91579]: pgmap v3642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:28 smithi188 ceph-mon[92425]: pgmap v3642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:30 smithi161 ceph-mon[91579]: pgmap v3643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:30 smithi188 ceph-mon[92425]: pgmap v3643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:32 smithi161 ceph-mon[91579]: pgmap v3644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:32 smithi188 ceph-mon[92425]: pgmap v3644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:34 smithi161 ceph-mon[91579]: pgmap v3645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:34 smithi188 ceph-mon[92425]: pgmap v3645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:36 smithi161 ceph-mon[91579]: pgmap v3646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:36 smithi188 ceph-mon[92425]: pgmap v3646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:38 smithi161 ceph-mon[91579]: pgmap v3647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:38 smithi188 ceph-mon[92425]: pgmap v3647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:40 smithi161 ceph-mon[91579]: pgmap v3648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:40 smithi188 ceph-mon[92425]: pgmap v3648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:42 smithi161 ceph-mon[91579]: pgmap v3649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:42 smithi188 ceph-mon[92425]: pgmap v3649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:44 smithi161 ceph-mon[91579]: pgmap v3650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:44 smithi188 ceph-mon[92425]: pgmap v3650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:46 smithi161 ceph-mon[91579]: pgmap v3651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:46 smithi188 ceph-mon[92425]: pgmap v3651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:48 smithi161 ceph-mon[91579]: pgmap v3652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:48 smithi188 ceph-mon[92425]: pgmap v3652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:50 smithi161 ceph-mon[91579]: pgmap v3653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:50 smithi188 ceph-mon[92425]: pgmap v3653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:52 smithi161 ceph-mon[91579]: pgmap v3654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:52 smithi188 ceph-mon[92425]: pgmap v3654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:54 smithi161 ceph-mon[91579]: pgmap v3655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:54 smithi188 ceph-mon[92425]: pgmap v3655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:40:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:56 smithi161 ceph-mon[91579]: pgmap v3656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:56 smithi188 ceph-mon[92425]: pgmap v3656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:40:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:40:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:40:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:40:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:40:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:40:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:40:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:40:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:40:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:58 smithi161 ceph-mon[91579]: pgmap v3657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:58 smithi188 ceph-mon[92425]: pgmap v3657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:40:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:40:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:00 smithi161 ceph-mon[91579]: pgmap v3658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:00 smithi188 ceph-mon[92425]: pgmap v3658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:02 smithi161 ceph-mon[91579]: pgmap v3659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:02 smithi188 ceph-mon[92425]: pgmap v3659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:04 smithi161 ceph-mon[91579]: pgmap v3660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:04 smithi188 ceph-mon[92425]: pgmap v3660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:06 smithi188 ceph-mon[92425]: pgmap v3661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:06 smithi161 ceph-mon[91579]: pgmap v3661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:08 smithi188 ceph-mon[92425]: pgmap v3662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:08 smithi161 ceph-mon[91579]: pgmap v3662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:10 smithi188 ceph-mon[92425]: pgmap v3663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:10 smithi161 ceph-mon[91579]: pgmap v3663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:12 smithi188 ceph-mon[92425]: pgmap v3664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:12 smithi161 ceph-mon[91579]: pgmap v3664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:14 smithi188 ceph-mon[92425]: pgmap v3665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:14 smithi161 ceph-mon[91579]: pgmap v3665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:16 smithi188 ceph-mon[92425]: pgmap v3666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:16 smithi161 ceph-mon[91579]: pgmap v3666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:18 smithi188 ceph-mon[92425]: pgmap v3667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:41:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:41:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:41:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:18 smithi161 ceph-mon[91579]: pgmap v3667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:41:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:41:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:41:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:41:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:20 smithi188 ceph-mon[92425]: pgmap v3668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:20 smithi161 ceph-mon[91579]: pgmap v3668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:22 smithi161 ceph-mon[91579]: pgmap v3669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:22 smithi188 ceph-mon[92425]: pgmap v3669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:24 smithi161 ceph-mon[91579]: pgmap v3670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:24 smithi188 ceph-mon[92425]: pgmap v3670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:26 smithi161 ceph-mon[91579]: pgmap v3671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:26 smithi188 ceph-mon[92425]: pgmap v3671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:28 smithi161 ceph-mon[91579]: pgmap v3672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:28 smithi188 ceph-mon[92425]: pgmap v3672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:30 smithi161 ceph-mon[91579]: pgmap v3673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:30 smithi188 ceph-mon[92425]: pgmap v3673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:32 smithi161 ceph-mon[91579]: pgmap v3674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:32 smithi188 ceph-mon[92425]: pgmap v3674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:34 smithi161 ceph-mon[91579]: pgmap v3675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:34 smithi188 ceph-mon[92425]: pgmap v3675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:36 smithi161 ceph-mon[91579]: pgmap v3676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:36 smithi188 ceph-mon[92425]: pgmap v3676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:38 smithi161 ceph-mon[91579]: pgmap v3677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:38 smithi188 ceph-mon[92425]: pgmap v3677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:40 smithi161 ceph-mon[91579]: pgmap v3678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:40 smithi188 ceph-mon[92425]: pgmap v3678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:42 smithi161 ceph-mon[91579]: pgmap v3679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:42 smithi188 ceph-mon[92425]: pgmap v3679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:44 smithi161 ceph-mon[91579]: pgmap v3680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:44 smithi188 ceph-mon[92425]: pgmap v3680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:46 smithi161 ceph-mon[91579]: pgmap v3681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:46 smithi188 ceph-mon[92425]: pgmap v3681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:48.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:41:48.352+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:41:48.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:41:48.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:41:48.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:41:48.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:41:48.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:41:48.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:41:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:41:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:41:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:41:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:41:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:41:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:41:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:41:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:41:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:41:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:41:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:41:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:41:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:41:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:41:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:41:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:41:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:41:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:41:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:41:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:41:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:41:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:41:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:41:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:41:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:41:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:41:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:41:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:41:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:41:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:41:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:41:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:41:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:41:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:41:48.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:41:48.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:41:48.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:41:48.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:41:48.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:41:48.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=181] Local address: 172.21.15.161, port 53112 2022-07-03T12:41:48.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=181] Peer address: 172.21.15.101, port 22 2022-07-03T12:41:48.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=181] Connection lost 2022-07-03T12:41:48.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=181] Aborting connection 2022-07-03T12:41:48.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:41:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:48 smithi161 ceph-mon[91579]: pgmap v3682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:48 smithi188 ceph-mon[92425]: pgmap v3682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:41:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:41:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:41:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:41:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:41:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:41:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:41:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:41:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:41:50.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:41:50.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:41:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:41:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:41:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:41:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:41:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:41:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:41:50.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:41:50.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:41:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:41:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:41:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:41:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:41:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:41:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:41:50.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:41:50.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:41:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:41:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:41:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:41:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:41:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:41:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:41:50.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:41:50.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:41:50.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:41:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:41:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:41:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:41:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:41:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:41:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:41:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:41:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:41:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:41:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:41:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:41:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=181] Local address: 172.21.15.161, port 53112 2022-07-03T12:41:50.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=181] Peer address: 172.21.15.101, port 22 2022-07-03T12:41:50.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=181] Connection lost 2022-07-03T12:41:50.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=181] Aborting connection 2022-07-03T12:41:50.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:41:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:41:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:41:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:41:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:41:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:41:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:41:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:41:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:41:50.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:41:50.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:41:50.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:41:50.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:41:50.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:41:50.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:41:50.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:41:50.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:41:50.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:41:50.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:41:50.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:41:50.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:41:50.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:41:50.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:41:50.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:41:50.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:41:50.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:41:50.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:41:50.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:41:50.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:41:50.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:41:50.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:41:50.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:41:50.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:41:50.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:41:50.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:41:50.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:41:50.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:41:50.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:41:50.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:41:50.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:41:50.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:41:50.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:41:50.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:41:50.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:41:50.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:41:50.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:41:50.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:41:50.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=181] Local address: 172.21.15.161, port 53112 2022-07-03T12:41:50.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=181] Peer address: 172.21.15.101, port 22 2022-07-03T12:41:50.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=181] Connection lost 2022-07-03T12:41:50.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=181] Aborting connection 2022-07-03T12:41:50.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:50 smithi161 ceph-mon[91579]: pgmap v3683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:50 smithi188 ceph-mon[92425]: pgmap v3683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:52 smithi161 ceph-mon[91579]: pgmap v3684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:52 smithi188 ceph-mon[92425]: pgmap v3684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:41:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:54 smithi161 ceph-mon[91579]: pgmap v3685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:54 smithi188 ceph-mon[92425]: pgmap v3685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:41:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:41:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:56 smithi161 ceph-mon[91579]: pgmap v3686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:41:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:56 smithi188 ceph-mon[92425]: pgmap v3686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:41:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:41:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:41:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:41:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:41:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:41:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:41:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:41:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:41:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:41:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:58 smithi161 ceph-mon[91579]: pgmap v3687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:41:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:58 smithi188 ceph-mon[92425]: pgmap v3687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:42:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:00 smithi161 ceph-mon[91579]: pgmap v3688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:00 smithi188 ceph-mon[92425]: pgmap v3688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:02 smithi161 ceph-mon[91579]: pgmap v3689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:42:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:02 smithi188 ceph-mon[92425]: pgmap v3689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:42:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:04 smithi161 ceph-mon[91579]: pgmap v3690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:04 smithi188 ceph-mon[92425]: pgmap v3690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:06 smithi161 ceph-mon[91579]: pgmap v3691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:06 smithi188 ceph-mon[92425]: pgmap v3691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:08 smithi161 ceph-mon[91579]: pgmap v3692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:08 smithi188 ceph-mon[92425]: pgmap v3692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:10 smithi161 ceph-mon[91579]: pgmap v3693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:10 smithi188 ceph-mon[92425]: pgmap v3693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:12 smithi161 ceph-mon[91579]: pgmap v3694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:12 smithi188 ceph-mon[92425]: pgmap v3694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:14 smithi161 ceph-mon[91579]: pgmap v3695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:14 smithi188 ceph-mon[92425]: pgmap v3695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:16 smithi161 ceph-mon[91579]: pgmap v3696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:16 smithi188 ceph-mon[92425]: pgmap v3696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:18 smithi161 ceph-mon[91579]: pgmap v3697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:18 smithi188 ceph-mon[92425]: pgmap v3697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:20 smithi161 ceph-mon[91579]: pgmap v3698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:20 smithi188 ceph-mon[92425]: pgmap v3698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:22 smithi161 ceph-mon[91579]: pgmap v3699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:22 smithi188 ceph-mon[92425]: pgmap v3699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:24 smithi161 ceph-mon[91579]: pgmap v3700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:24 smithi188 ceph-mon[92425]: pgmap v3700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:26 smithi161 ceph-mon[91579]: pgmap v3701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:26 smithi188 ceph-mon[92425]: pgmap v3701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:28 smithi161 ceph-mon[91579]: pgmap v3702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:28 smithi188 ceph-mon[92425]: pgmap v3702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:30 smithi161 ceph-mon[91579]: pgmap v3703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:30 smithi188 ceph-mon[92425]: pgmap v3703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:32 smithi161 ceph-mon[91579]: pgmap v3704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:32 smithi188 ceph-mon[92425]: pgmap v3704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:34 smithi161 ceph-mon[91579]: pgmap v3705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:34 smithi188 ceph-mon[92425]: pgmap v3705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:36 smithi161 ceph-mon[91579]: pgmap v3706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:36 smithi188 ceph-mon[92425]: pgmap v3706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:38 smithi161 ceph-mon[91579]: pgmap v3707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:38 smithi188 ceph-mon[92425]: pgmap v3707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:39 smithi161 ceph-mon[91579]: pgmap v3708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:39 smithi188 ceph-mon[92425]: pgmap v3708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:41 smithi161 ceph-mon[91579]: pgmap v3709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:42.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:41 smithi188 ceph-mon[92425]: pgmap v3709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:44.580 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:44 smithi161 ceph-mon[91579]: pgmap v3710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:44.580 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:44 smithi188 ceph-mon[92425]: pgmap v3710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:46 smithi161 ceph-mon[91579]: pgmap v3711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:46 smithi188 ceph-mon[92425]: pgmap v3711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-mon[91579]: pgmap v3712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:48.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:42:48.388+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:42:48.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:42:48.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:42:48.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:42:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:42:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:42:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:42:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:42:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:42:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:42:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:42:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:42:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:42:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:42:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:42:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:42:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:42:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:42:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:42:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:42:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:42:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:42:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:42:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:42:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:42:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:42:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:42:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:42:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:42:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:42:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:42:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:42:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:42:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:42:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:42:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:42:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:42:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:42:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:42:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:42:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:42:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:42:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:42:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:42:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:42:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:42:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=183] Local address: 172.21.15.161, port 53140 2022-07-03T12:42:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=183] Peer address: 172.21.15.101, port 22 2022-07-03T12:42:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=183] Connection lost 2022-07-03T12:42:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=183] Aborting connection 2022-07-03T12:42:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:42:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:42:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:48 smithi188 ceph-mon[92425]: pgmap v3712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:42:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:42:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:42:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:42:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:42:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:42:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:42:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:42:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:42:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:42:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:42:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:42:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:42:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:42:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:42:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:42:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:42:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:42:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:42:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:42:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:42:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:42:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:42:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:42:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:42:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:42:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:42:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:42:49.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:42:49.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:42:49.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:42:49.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:42:49.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:42:49.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:42:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:42:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:42:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:42:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:42:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:42:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:42:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:42:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:42:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:42:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:42:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:42:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:42:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:42:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:42:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:42:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:42:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=183] Local address: 172.21.15.161, port 53140 2022-07-03T12:42:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=183] Peer address: 172.21.15.101, port 22 2022-07-03T12:42:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=183] Connection lost 2022-07-03T12:42:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=183] Aborting connection 2022-07-03T12:42:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:42:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:42:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:42:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:42:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:42:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:42:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:42:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:42:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:42:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:42:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:42:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:42:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:42:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:42:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:42:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:42:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:42:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:42:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:42:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:42:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:42:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:42:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:42:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:42:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:42:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:42:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:42:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:42:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:42:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:42:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:42:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:42:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:42:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:42:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:42:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:42:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:42:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:42:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:42:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:42:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:42:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:42:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:42:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:42:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:42:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:42:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:42:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:42:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:42:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:42:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=183] Local address: 172.21.15.161, port 53140 2022-07-03T12:42:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=183] Peer address: 172.21.15.101, port 22 2022-07-03T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=183] Connection lost 2022-07-03T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=183] Aborting connection 2022-07-03T12:42:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:50 smithi161 ceph-mon[91579]: pgmap v3713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:50 smithi188 ceph-mon[92425]: pgmap v3713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:52 smithi161 ceph-mon[91579]: pgmap v3714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:52 smithi188 ceph-mon[92425]: pgmap v3714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:54 smithi161 ceph-mon[91579]: pgmap v3715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:54 smithi188 ceph-mon[92425]: pgmap v3715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:42:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:56 smithi161 ceph-mon[91579]: pgmap v3716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:56.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:56 smithi188 ceph-mon[92425]: pgmap v3716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:42:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:42:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:42:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:42:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:42:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:42:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:42:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:42:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:42:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:58 smithi161 ceph-mon[91579]: pgmap v3717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:58 smithi188 ceph-mon[92425]: pgmap v3717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:42:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:42:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:00 smithi161 ceph-mon[91579]: pgmap v3718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:00 smithi188 ceph-mon[92425]: pgmap v3718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:02 smithi161 ceph-mon[91579]: pgmap v3719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:02 smithi188 ceph-mon[92425]: pgmap v3719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:04 smithi161 ceph-mon[91579]: pgmap v3720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:04 smithi188 ceph-mon[92425]: pgmap v3720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:06 smithi161 ceph-mon[91579]: pgmap v3721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:06 smithi188 ceph-mon[92425]: pgmap v3721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:08 smithi161 ceph-mon[91579]: pgmap v3722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:08 smithi188 ceph-mon[92425]: pgmap v3722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:10 smithi161 ceph-mon[91579]: pgmap v3723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:10 smithi188 ceph-mon[92425]: pgmap v3723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:12 smithi161 ceph-mon[91579]: pgmap v3724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:12 smithi188 ceph-mon[92425]: pgmap v3724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:14 smithi161 ceph-mon[91579]: pgmap v3725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:14 smithi188 ceph-mon[92425]: pgmap v3725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:16 smithi161 ceph-mon[91579]: pgmap v3726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:16 smithi188 ceph-mon[92425]: pgmap v3726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:18.610 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:18 smithi188 ceph-mon[92425]: pgmap v3727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:18 smithi161 ceph-mon[91579]: pgmap v3727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:20.576 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:20 smithi188 ceph-mon[92425]: pgmap v3728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:20 smithi161 ceph-mon[91579]: pgmap v3728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:22 smithi161 ceph-mon[91579]: pgmap v3729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:22 smithi188 ceph-mon[92425]: pgmap v3729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:24 smithi161 ceph-mon[91579]: pgmap v3730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:24 smithi188 ceph-mon[92425]: pgmap v3730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:26 smithi161 ceph-mon[91579]: pgmap v3731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:26 smithi188 ceph-mon[92425]: pgmap v3731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:28.600 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:28 smithi188 ceph-mon[92425]: pgmap v3732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:28 smithi161 ceph-mon[91579]: pgmap v3732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:30 smithi161 ceph-mon[91579]: pgmap v3733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:30 smithi188 ceph-mon[92425]: pgmap v3733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:31.603 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:32 smithi161 ceph-mon[91579]: pgmap v3734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:32 smithi188 ceph-mon[92425]: pgmap v3734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:34 smithi161 ceph-mon[91579]: pgmap v3735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:34 smithi188 ceph-mon[92425]: pgmap v3735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:36 smithi161 ceph-mon[91579]: pgmap v3736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:36 smithi188 ceph-mon[92425]: pgmap v3736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:38 smithi161 ceph-mon[91579]: pgmap v3737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:38 smithi188 ceph-mon[92425]: pgmap v3737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:40 smithi161 ceph-mon[91579]: pgmap v3738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:40 smithi188 ceph-mon[92425]: pgmap v3738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:42 smithi161 ceph-mon[91579]: pgmap v3739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:42 smithi188 ceph-mon[92425]: pgmap v3739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:44 smithi161 ceph-mon[91579]: pgmap v3740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:44 smithi188 ceph-mon[92425]: pgmap v3740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:46 smithi161 ceph-mon[91579]: pgmap v3741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:46 smithi188 ceph-mon[92425]: pgmap v3741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:48 smithi188 ceph-mon[92425]: pgmap v3742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-mon[91579]: pgmap v3742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:49.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:43:48.900+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:43:49.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:43:49.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:43:49.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:43:49.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:43:49.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:43:49.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:43:49.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:43:49.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:43:49.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:43:49.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:43:49.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:43:49.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:43:49.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:43:49.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:43:49.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:43:49.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:43:49.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:43:49.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:43:49.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:43:49.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:43:49.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:43:49.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:43:49.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:43:49.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:43:49.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:43:49.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:43:49.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:43:49.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:43:49.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:43:49.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:43:49.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:43:49.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:43:49.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:43:49.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:43:49.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:43:49.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:43:49.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:43:49.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:43:49.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:43:49.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:43:49.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:43:49.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:43:49.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:43:49.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:43:49.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:43:49.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=185] Local address: 172.21.15.161, port 53168 2022-07-03T12:43:49.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=185] Peer address: 172.21.15.101, port 22 2022-07-03T12:43:49.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=185] Connection lost 2022-07-03T12:43:49.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=185] Aborting connection 2022-07-03T12:43:49.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:43:49.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:43:49.158+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T12:43:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:43:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:43:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:43:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:43:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:43:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:43:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:43:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:43:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:43:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:43:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:43:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:43:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:43:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:43:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:43:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:43:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:43:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:43:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:43:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:43:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:43:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:43:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:43:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:43:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:43:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:43:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:43:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:43:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:43:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:43:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:43:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:43:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:43:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:43:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:43:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:43:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:43:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:43:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:43:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:43:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:43:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:43:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:43:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:43:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:43:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:43:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:43:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:43:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:43:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:43:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:43:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:43:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:43:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:43:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=185] Local address: 172.21.15.161, port 53168 2022-07-03T12:43:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=185] Peer address: 172.21.15.101, port 22 2022-07-03T12:43:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=185] Connection lost 2022-07-03T12:43:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=185] Aborting connection 2022-07-03T12:43:50.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:50 smithi188 ceph-mon[92425]: pgmap v3743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:43:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:43:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:43:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:43:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:43:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:43:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:43:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:43:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:43:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:43:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:43:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:43:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:43:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:43:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:43:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:43:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:43:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:43:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:43:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:43:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:43:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:43:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:43:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:43:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:43:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:43:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:43:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:43:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:43:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:43:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:43:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:43:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:43:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:43:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:43:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:43:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:43:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:43:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:43:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:43:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:43:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:43:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:43:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:43:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:43:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:43:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:43:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=185] Local address: 172.21.15.161, port 53168 2022-07-03T12:43:50.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=185] Peer address: 172.21.15.101, port 22 2022-07-03T12:43:50.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=185] Connection lost 2022-07-03T12:43:50.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=185] Aborting connection 2022-07-03T12:43:50.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:50 smithi161 ceph-mon[91579]: pgmap v3743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:52 smithi188 ceph-mon[92425]: pgmap v3744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:52 smithi161 ceph-mon[91579]: pgmap v3744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:54 smithi188 ceph-mon[92425]: pgmap v3745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:54 smithi161 ceph-mon[91579]: pgmap v3745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:43:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:56 smithi188 ceph-mon[92425]: pgmap v3746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:56 smithi161 ceph-mon[91579]: pgmap v3746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:43:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:43:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:43:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:43:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:43:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:43:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:43:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:43:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:43:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:43:58 smithi188 ceph-mon[92425]: pgmap v3747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:43:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:43:58 smithi161 ceph-mon[91579]: pgmap v3747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:00 smithi188 ceph-mon[92425]: pgmap v3748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:00 smithi161 ceph-mon[91579]: pgmap v3748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:02 smithi161 ceph-mon[91579]: pgmap v3749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:02 smithi188 ceph-mon[92425]: pgmap v3749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:04 smithi161 ceph-mon[91579]: pgmap v3750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:04 smithi188 ceph-mon[92425]: pgmap v3750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:06 smithi161 ceph-mon[91579]: pgmap v3751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:06 smithi188 ceph-mon[92425]: pgmap v3751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:08 smithi161 ceph-mon[91579]: pgmap v3752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:08 smithi188 ceph-mon[92425]: pgmap v3752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:10 smithi161 ceph-mon[91579]: pgmap v3753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:10 smithi188 ceph-mon[92425]: pgmap v3753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:12 smithi161 ceph-mon[91579]: pgmap v3754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:12 smithi188 ceph-mon[92425]: pgmap v3754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:14 smithi161 ceph-mon[91579]: pgmap v3755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:14 smithi188 ceph-mon[92425]: pgmap v3755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:16 smithi161 ceph-mon[91579]: pgmap v3756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:16 smithi188 ceph-mon[92425]: pgmap v3756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:18 smithi161 ceph-mon[91579]: pgmap v3757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:18 smithi188 ceph-mon[92425]: pgmap v3757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:20 smithi161 ceph-mon[91579]: pgmap v3758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:20 smithi188 ceph-mon[92425]: pgmap v3758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:22 smithi161 ceph-mon[91579]: pgmap v3759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:22 smithi188 ceph-mon[92425]: pgmap v3759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:24 smithi161 ceph-mon[91579]: pgmap v3760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:24 smithi188 ceph-mon[92425]: pgmap v3760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:26 smithi161 ceph-mon[91579]: pgmap v3761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:26 smithi188 ceph-mon[92425]: pgmap v3761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:28 smithi161 ceph-mon[91579]: pgmap v3762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:28 smithi188 ceph-mon[92425]: pgmap v3762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:30 smithi161 ceph-mon[91579]: pgmap v3763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:30 smithi188 ceph-mon[92425]: pgmap v3763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:32 smithi161 ceph-mon[91579]: pgmap v3764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:32 smithi188 ceph-mon[92425]: pgmap v3764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:34 smithi161 ceph-mon[91579]: pgmap v3765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:34 smithi188 ceph-mon[92425]: pgmap v3765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:36 smithi161 ceph-mon[91579]: pgmap v3766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:36 smithi188 ceph-mon[92425]: pgmap v3766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:38 smithi161 ceph-mon[91579]: pgmap v3767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:38 smithi188 ceph-mon[92425]: pgmap v3767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:40 smithi161 ceph-mon[91579]: pgmap v3768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:40 smithi188 ceph-mon[92425]: pgmap v3768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:42 smithi161 ceph-mon[91579]: pgmap v3769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:42 smithi188 ceph-mon[92425]: pgmap v3769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:44 smithi161 ceph-mon[91579]: pgmap v3770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:44 smithi188 ceph-mon[92425]: pgmap v3770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:46 smithi161 ceph-mon[91579]: pgmap v3771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:46 smithi188 ceph-mon[92425]: pgmap v3771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:48 smithi161 ceph-mon[91579]: pgmap v3772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:48 smithi188 ceph-mon[92425]: pgmap v3772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:44:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:44:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:44:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:44:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:44:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:44:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:50 smithi161 ceph-mon[91579]: pgmap v3773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:44:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:50 smithi188 ceph-mon[92425]: pgmap v3773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:44:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:52 smithi161 ceph-mon[91579]: pgmap v3774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:52 smithi188 ceph-mon[92425]: pgmap v3774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:54 smithi161 ceph-mon[91579]: pgmap v3775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:54 smithi188 ceph-mon[92425]: pgmap v3775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:44:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:56 smithi161 ceph-mon[91579]: pgmap v3776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:56 smithi188 ceph-mon[92425]: pgmap v3776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:44:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:44:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:44:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:44:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:44:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:44:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:44:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:44:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:44:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:44:58 smithi161 ceph-mon[91579]: pgmap v3777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:44:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:44:58 smithi188 ceph-mon[92425]: pgmap v3777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:00 smithi161 ceph-mon[91579]: pgmap v3778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:00 smithi188 ceph-mon[92425]: pgmap v3778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:02 smithi161 ceph-mon[91579]: pgmap v3779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:02 smithi188 ceph-mon[92425]: pgmap v3779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:04 smithi161 ceph-mon[91579]: pgmap v3780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:04 smithi188 ceph-mon[92425]: pgmap v3780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:06 smithi161 ceph-mon[91579]: pgmap v3781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:06 smithi188 ceph-mon[92425]: pgmap v3781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:08 smithi161 ceph-mon[91579]: pgmap v3782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:08 smithi188 ceph-mon[92425]: pgmap v3782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:10 smithi161 ceph-mon[91579]: pgmap v3783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:10 smithi188 ceph-mon[92425]: pgmap v3783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:12 smithi161 ceph-mon[91579]: pgmap v3784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:12 smithi188 ceph-mon[92425]: pgmap v3784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:14 smithi161 ceph-mon[91579]: pgmap v3785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:14 smithi188 ceph-mon[92425]: pgmap v3785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:16 smithi161 ceph-mon[91579]: pgmap v3786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:16 smithi188 ceph-mon[92425]: pgmap v3786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:18 smithi161 ceph-mon[91579]: pgmap v3787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:18 smithi188 ceph-mon[92425]: pgmap v3787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:45:19.420+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:45:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:45:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:45:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:45:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:45:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:45:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:45:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:45:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:45:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:45:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:45:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:45:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:45:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:45:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:45:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:45:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:45:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:45:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:45:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:45:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:45:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:45:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:45:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:45:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:45:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:45:19.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:45:19.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:45:19.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:45:19.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:45:19.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:45:19.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:45:19.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:45:19.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:45:19.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:45:19.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:45:19.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:45:19.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:45:19.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:45:19.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:45:19.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:45:19.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:45:19.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:45:19.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:45:19.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:45:19.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:45:19.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=187] Local address: 172.21.15.161, port 53174 2022-07-03T12:45:19.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=187] Peer address: 172.21.15.101, port 22 2022-07-03T12:45:19.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=548] Received exit status 0 2022-07-03T12:45:19.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=548] Received channel close 2022-07-03T12:45:19.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=548] Channel closed 2022-07-03T12:45:19.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=549] Requesting new SSH session 2022-07-03T12:45:19.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=549] Command: which python3 2022-07-03T12:45:19.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=548] Received exit status 0 2022-07-03T12:45:19.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=548] Received channel close 2022-07-03T12:45:19.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=548] Channel closed 2022-07-03T12:45:19.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=549] Requesting new SSH session 2022-07-03T12:45:19.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=549] Command: which python3 2022-07-03T12:45:19.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=549] Received exit status 0 2022-07-03T12:45:19.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=549] Received channel close 2022-07-03T12:45:19.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=549] Channel closed 2022-07-03T12:45:19.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=550] Requesting new SSH session 2022-07-03T12:45:19.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=550] Command: sudo true 2022-07-03T12:45:19.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=549] Received exit status 0 2022-07-03T12:45:19.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=549] Received channel close 2022-07-03T12:45:19.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=549] Channel closed 2022-07-03T12:45:19.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=550] Requesting new SSH session 2022-07-03T12:45:19.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=550] Command: sudo true 2022-07-03T12:45:19.928 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=550] Received exit status 0 2022-07-03T12:45:19.929 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=550] Received channel close 2022-07-03T12:45:19.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=550] Channel closed 2022-07-03T12:45:19.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=551] Requesting new SSH session 2022-07-03T12:45:19.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:19.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=550] Received exit status 0 2022-07-03T12:45:19.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=550] Received channel close 2022-07-03T12:45:19.933 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=550] Channel closed 2022-07-03T12:45:19.934 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=551] Requesting new SSH session 2022-07-03T12:45:19.934 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:19.935 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=551] Received exit status 0 2022-07-03T12:45:19.936 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=551] Received channel close 2022-07-03T12:45:19.936 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=551] Channel closed 2022-07-03T12:45:19.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=551] Received exit status 0 2022-07-03T12:45:19.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=551] Received channel close 2022-07-03T12:45:19.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=551] Channel closed 2022-07-03T12:45:19.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=187] Login timeout expired 2022-07-03T12:45:19.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=187] Aborting connection 2022-07-03T12:45:19.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:45:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:45:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:20 smithi161 ceph-mon[91579]: pgmap v3788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:45:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:45:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:45:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:45:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:45:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:45:20.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:45:20.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:45:20.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:45:20.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:45:20.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:45:20.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:45:20.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:45:20.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:45:20.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:45:20.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:45:20.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:45:20.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:45:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:45:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:45:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:45:20.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:45:20.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:45:20.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:45:20.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:45:20.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:45:20.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:45:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:45:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:45:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:45:20.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:45:20.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:45:20.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:45:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:45:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:45:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:45:20.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:45:20.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:45:20.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:45:20.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:45:20.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:45:20.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:45:20.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:45:20.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:45:20.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:45:20.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:45:20.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=187] Local address: 172.21.15.161, port 53174 2022-07-03T12:45:20.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=187] Peer address: 172.21.15.101, port 22 2022-07-03T12:45:20.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=548] Received exit status 0 2022-07-03T12:45:20.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=548] Received channel close 2022-07-03T12:45:20.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=548] Channel closed 2022-07-03T12:45:20.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=549] Requesting new SSH session 2022-07-03T12:45:20.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=549] Command: which python3 2022-07-03T12:45:20.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=548] Received exit status 0 2022-07-03T12:45:20.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=548] Received channel close 2022-07-03T12:45:20.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=548] Channel closed 2022-07-03T12:45:20.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=549] Requesting new SSH session 2022-07-03T12:45:20.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=549] Command: which python3 2022-07-03T12:45:20.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=549] Received exit status 0 2022-07-03T12:45:20.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=549] Received channel close 2022-07-03T12:45:20.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=549] Channel closed 2022-07-03T12:45:20.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=550] Requesting new SSH session 2022-07-03T12:45:20.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=550] Command: sudo true 2022-07-03T12:45:20.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=549] Received exit status 0 2022-07-03T12:45:20.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=549] Received channel close 2022-07-03T12:45:20.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=549] Channel closed 2022-07-03T12:45:20.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=550] Requesting new SSH session 2022-07-03T12:45:20.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=550] Command: sudo true 2022-07-03T12:45:20.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=550] Received exit status 0 2022-07-03T12:45:20.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=550] Received channel close 2022-07-03T12:45:20.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=550] Channel closed 2022-07-03T12:45:20.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=551] Requesting new SSH session 2022-07-03T12:45:20.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:20.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=550] Received exit status 0 2022-07-03T12:45:20.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=550] Received channel close 2022-07-03T12:45:20.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=550] Channel closed 2022-07-03T12:45:20.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=551] Requesting new SSH session 2022-07-03T12:45:20.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:20.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=551] Received exit status 0 2022-07-03T12:45:20.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=551] Received channel close 2022-07-03T12:45:20.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=551] Channel closed 2022-07-03T12:45:20.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=551] Received exit status 0 2022-07-03T12:45:20.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=551] Received channel close 2022-07-03T12:45:20.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=551] Channel closed 2022-07-03T12:45:20.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=187] Login timeout expired 2022-07-03T12:45:20.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=187] Aborting connection 2022-07-03T12:45:20.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:20 smithi188 ceph-mon[92425]: pgmap v3788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:20.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:45:20.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:45:20.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:45:20.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:45:20.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:45:20.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:45:20.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:45:20.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:45:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:45:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:45:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:45:20.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:45:20.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:45:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:45:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:45:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:45:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:45:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:45:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:45:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:45:20.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:45:20.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:45:20.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:45:20.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:45:20.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:45:20.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:45:20.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:45:20.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:45:20.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:45:20.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:45:20.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:45:20.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:45:20.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:45:20.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:45:20.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:45:20.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:45:20.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:45:20.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:45:20.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:45:20.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:45:20.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:45:20.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:45:20.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:45:20.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:45:20.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:45:20.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:45:20.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=187] Local address: 172.21.15.161, port 53174 2022-07-03T12:45:20.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=187] Peer address: 172.21.15.101, port 22 2022-07-03T12:45:20.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=548] Received exit status 0 2022-07-03T12:45:20.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=548] Received channel close 2022-07-03T12:45:20.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=548] Channel closed 2022-07-03T12:45:20.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=549] Requesting new SSH session 2022-07-03T12:45:20.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=549] Command: which python3 2022-07-03T12:45:20.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=548] Received exit status 0 2022-07-03T12:45:20.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=548] Received channel close 2022-07-03T12:45:20.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=548] Channel closed 2022-07-03T12:45:20.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=549] Requesting new SSH session 2022-07-03T12:45:20.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=549] Command: which python3 2022-07-03T12:45:20.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=549] Received exit status 0 2022-07-03T12:45:20.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=549] Received channel close 2022-07-03T12:45:20.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=549] Channel closed 2022-07-03T12:45:20.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=550] Requesting new SSH session 2022-07-03T12:45:20.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=550] Command: sudo true 2022-07-03T12:45:20.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=549] Received exit status 0 2022-07-03T12:45:20.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=549] Received channel close 2022-07-03T12:45:20.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=549] Channel closed 2022-07-03T12:45:20.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=550] Requesting new SSH session 2022-07-03T12:45:20.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=550] Command: sudo true 2022-07-03T12:45:20.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=550] Received exit status 0 2022-07-03T12:45:20.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=550] Received channel close 2022-07-03T12:45:20.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=550] Channel closed 2022-07-03T12:45:20.958 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=551] Requesting new SSH session 2022-07-03T12:45:20.958 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:20.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=550] Received exit status 0 2022-07-03T12:45:20.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=550] Received channel close 2022-07-03T12:45:20.959 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=550] Channel closed 2022-07-03T12:45:20.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=551] Requesting new SSH session 2022-07-03T12:45:20.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=551] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:45:20.960 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=551] Received exit status 0 2022-07-03T12:45:20.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=551] Received channel close 2022-07-03T12:45:20.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=551] Channel closed 2022-07-03T12:45:20.961 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=551] Received exit status 0 2022-07-03T12:45:20.962 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=551] Received channel close 2022-07-03T12:45:20.962 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=551] Channel closed 2022-07-03T12:45:20.963 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=187] Login timeout expired 2022-07-03T12:45:20.963 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=187] Aborting connection 2022-07-03T12:45:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:22 smithi161 ceph-mon[91579]: pgmap v3789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:22 smithi188 ceph-mon[92425]: pgmap v3789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:24 smithi161 ceph-mon[91579]: pgmap v3790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:24 smithi188 ceph-mon[92425]: pgmap v3790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:26 smithi161 ceph-mon[91579]: pgmap v3791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:26 smithi188 ceph-mon[92425]: pgmap v3791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:28 smithi161 ceph-mon[91579]: pgmap v3792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:28 smithi188 ceph-mon[92425]: pgmap v3792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:30 smithi161 ceph-mon[91579]: pgmap v3793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:30 smithi188 ceph-mon[92425]: pgmap v3793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:32 smithi161 ceph-mon[91579]: pgmap v3794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:32 smithi188 ceph-mon[92425]: pgmap v3794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:34 smithi161 ceph-mon[91579]: pgmap v3795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:34 smithi188 ceph-mon[92425]: pgmap v3795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:36 smithi161 ceph-mon[91579]: pgmap v3796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:36 smithi188 ceph-mon[92425]: pgmap v3796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:38 smithi161 ceph-mon[91579]: pgmap v3797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:38 smithi188 ceph-mon[92425]: pgmap v3797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:40 smithi161 ceph-mon[91579]: pgmap v3798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:40 smithi188 ceph-mon[92425]: pgmap v3798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:42 smithi161 ceph-mon[91579]: pgmap v3799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:42 smithi188 ceph-mon[92425]: pgmap v3799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:44 smithi161 ceph-mon[91579]: pgmap v3800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:44 smithi188 ceph-mon[92425]: pgmap v3800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:46 smithi161 ceph-mon[91579]: pgmap v3801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:46 smithi188 ceph-mon[92425]: pgmap v3801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:48 smithi161 ceph-mon[91579]: pgmap v3802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:48 smithi188 ceph-mon[92425]: pgmap v3802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:50 smithi161 ceph-mon[91579]: pgmap v3803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:50 smithi188 ceph-mon[92425]: pgmap v3803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:52 smithi188 ceph-mon[92425]: pgmap v3804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:52 smithi161 ceph-mon[91579]: pgmap v3804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:54 smithi188 ceph-mon[92425]: pgmap v3805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:54 smithi161 ceph-mon[91579]: pgmap v3805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:45:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:56 smithi188 ceph-mon[92425]: pgmap v3806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:56 smithi161 ceph-mon[91579]: pgmap v3806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:45:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:45:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:45:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:45:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:45:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:45:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:45:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:45:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:45:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:45:58 smithi188 ceph-mon[92425]: pgmap v3807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:45:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:45:58 smithi161 ceph-mon[91579]: pgmap v3807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:00 smithi188 ceph-mon[92425]: pgmap v3808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:00 smithi161 ceph-mon[91579]: pgmap v3808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:02 smithi188 ceph-mon[92425]: pgmap v3809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:02 smithi161 ceph-mon[91579]: pgmap v3809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:04 smithi188 ceph-mon[92425]: pgmap v3810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:04 smithi161 ceph-mon[91579]: pgmap v3810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:06 smithi188 ceph-mon[92425]: pgmap v3811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:06 smithi161 ceph-mon[91579]: pgmap v3811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:08 smithi161 ceph-mon[91579]: pgmap v3812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:08 smithi188 ceph-mon[92425]: pgmap v3812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:10 smithi161 ceph-mon[91579]: pgmap v3813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:10 smithi188 ceph-mon[92425]: pgmap v3813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:12 smithi161 ceph-mon[91579]: pgmap v3814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:12 smithi188 ceph-mon[92425]: pgmap v3814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:14 smithi161 ceph-mon[91579]: pgmap v3815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:14 smithi188 ceph-mon[92425]: pgmap v3815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:16 smithi161 ceph-mon[91579]: pgmap v3816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:16 smithi188 ceph-mon[92425]: pgmap v3816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:18 smithi161 ceph-mon[91579]: pgmap v3817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:18 smithi188 ceph-mon[92425]: pgmap v3817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:46:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:46:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:46:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:46:19.456+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:46:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:46:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:46:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:46:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:46:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:46:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:46:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:46:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:46:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:46:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:46:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:46:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:46:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:46:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:46:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:46:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:46:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:46:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:46:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:46:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:46:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:46:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:46:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:46:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:46:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:46:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:46:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:46:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:46:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:46:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:46:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:46:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:46:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:46:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:46:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:46:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:46:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:46:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:46:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:46:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:46:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:46:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:46:19.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:46:19.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:46:19.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:46:19.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=189] Local address: 172.21.15.161, port 53178 2022-07-03T12:46:19.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=189] Peer address: 172.21.15.101, port 22 2022-07-03T12:46:19.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=189] Connection lost 2022-07-03T12:46:19.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=189] Aborting connection 2022-07-03T12:46:19.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:46:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:46:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:46:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:46:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:46:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:20 smithi161 ceph-mon[91579]: pgmap v3818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:46:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:46:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:46:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:46:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:46:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:46:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:46:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:46:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:46:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:46:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:46:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:46:21.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:46:21.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:46:21.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:46:21.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:46:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:46:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:46:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:46:21.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:46:21.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:46:21.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:46:21.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:46:21.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:46:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:46:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:46:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:46:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:46:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:46:21.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:46:21.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:46:21.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:46:21.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:46:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:46:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:46:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:46:21.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:46:21.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:46:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:46:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:46:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:46:21.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:46:21.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:46:21.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:46:21.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:46:21.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:46:21.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:46:21.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=189] Local address: 172.21.15.161, port 53178 2022-07-03T12:46:21.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=189] Peer address: 172.21.15.101, port 22 2022-07-03T12:46:21.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=189] Connection lost 2022-07-03T12:46:21.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=189] Aborting connection 2022-07-03T12:46:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:20 smithi188 ceph-mon[92425]: pgmap v3818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:46:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:46:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:46:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:46:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:46:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:46:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:46:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:46:21.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:46:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:46:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:46:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:46:21.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:46:21.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:46:21.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:46:21.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:46:21.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:46:21.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:46:21.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:46:21.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:46:21.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:46:21.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:46:21.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:46:21.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:46:21.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:46:21.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:46:21.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:46:21.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:46:21.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:46:21.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:46:21.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:46:21.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:46:21.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:46:21.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:46:21.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:46:21.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:46:21.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:46:21.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:46:21.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:46:21.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:46:21.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:46:21.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:46:21.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:46:21.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:46:21.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:46:21.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:46:21.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:46:21.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=189] Local address: 172.21.15.161, port 53178 2022-07-03T12:46:21.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=189] Peer address: 172.21.15.101, port 22 2022-07-03T12:46:21.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=189] Connection lost 2022-07-03T12:46:21.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=189] Aborting connection 2022-07-03T12:46:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:22 smithi161 ceph-mon[91579]: pgmap v3819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:22 smithi188 ceph-mon[92425]: pgmap v3819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:24 smithi161 ceph-mon[91579]: pgmap v3820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:24 smithi188 ceph-mon[92425]: pgmap v3820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:26 smithi161 ceph-mon[91579]: pgmap v3821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:26 smithi188 ceph-mon[92425]: pgmap v3821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:28 smithi161 ceph-mon[91579]: pgmap v3822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:28 smithi188 ceph-mon[92425]: pgmap v3822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:30 smithi161 ceph-mon[91579]: pgmap v3823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:30 smithi188 ceph-mon[92425]: pgmap v3823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:32 smithi161 ceph-mon[91579]: pgmap v3824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:32 smithi188 ceph-mon[92425]: pgmap v3824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:34 smithi161 ceph-mon[91579]: pgmap v3825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:34 smithi188 ceph-mon[92425]: pgmap v3825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:36 smithi161 ceph-mon[91579]: pgmap v3826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:36 smithi188 ceph-mon[92425]: pgmap v3826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:38 smithi161 ceph-mon[91579]: pgmap v3827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:38 smithi188 ceph-mon[92425]: pgmap v3827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:40 smithi161 ceph-mon[91579]: pgmap v3828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:40 smithi188 ceph-mon[92425]: pgmap v3828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:42 smithi161 ceph-mon[91579]: pgmap v3829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:42 smithi188 ceph-mon[92425]: pgmap v3829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:44 smithi161 ceph-mon[91579]: pgmap v3830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:44 smithi188 ceph-mon[92425]: pgmap v3830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:46 smithi161 ceph-mon[91579]: pgmap v3831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:46 smithi188 ceph-mon[92425]: pgmap v3831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:48 smithi161 ceph-mon[91579]: pgmap v3832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:48 smithi188 ceph-mon[92425]: pgmap v3832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:50 smithi161 ceph-mon[91579]: pgmap v3833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:50 smithi188 ceph-mon[92425]: pgmap v3833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:52 smithi161 ceph-mon[91579]: pgmap v3834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:52 smithi188 ceph-mon[92425]: pgmap v3834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:54 smithi161 ceph-mon[91579]: pgmap v3835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:54 smithi188 ceph-mon[92425]: pgmap v3835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:46:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:56 smithi161 ceph-mon[91579]: pgmap v3836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:56 smithi188 ceph-mon[92425]: pgmap v3836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:46:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:46:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:46:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:46:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:46:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:46:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:46:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:46:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:46:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:46:58 smithi161 ceph-mon[91579]: pgmap v3837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:46:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:46:58 smithi188 ceph-mon[92425]: pgmap v3837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:00 smithi161 ceph-mon[91579]: pgmap v3838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:00 smithi188 ceph-mon[92425]: pgmap v3838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:02 smithi161 ceph-mon[91579]: pgmap v3839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:02 smithi188 ceph-mon[92425]: pgmap v3839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:04 smithi161 ceph-mon[91579]: pgmap v3840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:04 smithi188 ceph-mon[92425]: pgmap v3840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:06 smithi161 ceph-mon[91579]: pgmap v3841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:06 smithi188 ceph-mon[92425]: pgmap v3841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:08 smithi161 ceph-mon[91579]: pgmap v3842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:08 smithi188 ceph-mon[92425]: pgmap v3842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:10 smithi161 ceph-mon[91579]: pgmap v3843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:10 smithi188 ceph-mon[92425]: pgmap v3843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:12 smithi161 ceph-mon[91579]: pgmap v3844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:12 smithi188 ceph-mon[92425]: pgmap v3844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:14 smithi161 ceph-mon[91579]: pgmap v3845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:14 smithi188 ceph-mon[92425]: pgmap v3845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:16 smithi161 ceph-mon[91579]: pgmap v3846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:16 smithi188 ceph-mon[92425]: pgmap v3846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:18 smithi161 ceph-mon[91579]: pgmap v3847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:18 smithi188 ceph-mon[92425]: pgmap v3847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:47:19.941+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:47:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:47:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:47:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:47:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:47:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:47:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:47:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:47:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:47:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:47:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:47:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:47:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:47:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:47:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:47:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:47:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:47:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:47:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:47:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:47:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:47:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:47:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:47:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:47:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:47:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:47:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:47:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:47:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:47:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:47:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:47:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:47:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:47:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:47:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:47:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:47:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:47:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:47:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:47:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:47:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:47:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:47:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:47:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:47:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:47:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:47:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=191] Local address: 172.21.15.161, port 53182 2022-07-03T12:47:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=191] Peer address: 172.21.15.101, port 22 2022-07-03T12:47:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=191] Connection lost 2022-07-03T12:47:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=191] Aborting connection 2022-07-03T12:47:20.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:47:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:47:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: pgmap v3848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:47:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:47:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:47:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: pgmap v3848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:47:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:47:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:47:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:47:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:47:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:47:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:47:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:47:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:47:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:47:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:47:22.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:47:22.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:47:22.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:47:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:47:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:47:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:47:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:47:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:47:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:47:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:47:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:47:22.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:47:22.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:47:22.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:47:22.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:47:22.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:47:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:47:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:47:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:47:22.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:47:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:47:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:47:22.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:47:22.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:47:22.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:47:22.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:47:22.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:47:22.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:47:22.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:47:22.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:47:22.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:47:22.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:47:22.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:47:22.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:47:22.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:47:22.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:47:22.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:47:22.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:47:22.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:47:22.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=191] Local address: 172.21.15.161, port 53182 2022-07-03T12:47:22.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=191] Peer address: 172.21.15.101, port 22 2022-07-03T12:47:22.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=191] Connection lost 2022-07-03T12:47:22.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=191] Aborting connection 2022-07-03T12:47:22.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:47:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:47:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:47:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:47:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:47:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:47:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:47:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:47:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:47:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:47:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:47:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:47:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:47:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:47:22.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:47:22.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:47:22.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:47:22.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:47:22.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:47:22.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:47:22.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:47:22.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:47:22.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:47:22.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:47:22.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:47:22.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:47:22.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:47:22.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:47:22.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:47:22.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:47:22.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:47:22.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:47:22.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:47:22.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:47:22.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:47:22.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:47:22.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:47:22.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:47:22.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:47:22.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:47:22.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:47:22.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:47:22.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:47:22.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:47:22.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:47:22.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:47:22.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:47:22.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=191] Local address: 172.21.15.161, port 53182 2022-07-03T12:47:22.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=191] Peer address: 172.21.15.101, port 22 2022-07-03T12:47:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=191] Connection lost 2022-07-03T12:47:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=191] Aborting connection 2022-07-03T12:47:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:23.081 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:22 smithi161 ceph-mon[91579]: pgmap v3849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:23.088 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:22 smithi188 ceph-mon[92425]: pgmap v3849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: pgmap v3850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:47:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:47:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:47:24.392 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:24 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:47:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: pgmap v3850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:47:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:47:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:47:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:47:24.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:47:24.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:24 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:47:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:26 smithi161 ceph-mon[91579]: pgmap v3851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:26 smithi188 ceph-mon[92425]: pgmap v3851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:28 smithi161 ceph-mon[91579]: pgmap v3852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:28 smithi188 ceph-mon[92425]: pgmap v3852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:30 smithi161 ceph-mon[91579]: pgmap v3853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:30 smithi188 ceph-mon[92425]: pgmap v3853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:32 smithi161 ceph-mon[91579]: pgmap v3854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:32 smithi188 ceph-mon[92425]: pgmap v3854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:34 smithi161 ceph-mon[91579]: pgmap v3855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:34 smithi188 ceph-mon[92425]: pgmap v3855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:36 smithi161 ceph-mon[91579]: pgmap v3856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:36 smithi188 ceph-mon[92425]: pgmap v3856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:38 smithi161 ceph-mon[91579]: pgmap v3857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:38 smithi188 ceph-mon[92425]: pgmap v3857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:40 smithi161 ceph-mon[91579]: pgmap v3858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:40 smithi188 ceph-mon[92425]: pgmap v3858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:42 smithi161 ceph-mon[91579]: pgmap v3859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:42 smithi188 ceph-mon[92425]: pgmap v3859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:44 smithi161 ceph-mon[91579]: pgmap v3860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:44 smithi188 ceph-mon[92425]: pgmap v3860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:46 smithi161 ceph-mon[91579]: pgmap v3861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:46 smithi188 ceph-mon[92425]: pgmap v3861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:48 smithi161 ceph-mon[91579]: pgmap v3862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:48 smithi188 ceph-mon[92425]: pgmap v3862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:50 smithi161 ceph-mon[91579]: pgmap v3863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:50 smithi188 ceph-mon[92425]: pgmap v3863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:52 smithi161 ceph-mon[91579]: pgmap v3864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:52 smithi188 ceph-mon[92425]: pgmap v3864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:54 smithi161 ceph-mon[91579]: pgmap v3865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:54 smithi188 ceph-mon[92425]: pgmap v3865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:47:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:56 smithi161 ceph-mon[91579]: pgmap v3866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:56 smithi188 ceph-mon[92425]: pgmap v3866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:47:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:47:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:47:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:47:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:47:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:47:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:47:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:47:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:47:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:47:58 smithi161 ceph-mon[91579]: pgmap v3867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:47:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:47:58 smithi188 ceph-mon[92425]: pgmap v3867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:00 smithi161 ceph-mon[91579]: pgmap v3868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:00 smithi188 ceph-mon[92425]: pgmap v3868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:02 smithi161 ceph-mon[91579]: pgmap v3869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:02 smithi188 ceph-mon[92425]: pgmap v3869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:04 smithi161 ceph-mon[91579]: pgmap v3870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:04 smithi188 ceph-mon[92425]: pgmap v3870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:06 smithi161 ceph-mon[91579]: pgmap v3871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:06 smithi188 ceph-mon[92425]: pgmap v3871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:08 smithi161 ceph-mon[91579]: pgmap v3872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:08 smithi188 ceph-mon[92425]: pgmap v3872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:10 smithi161 ceph-mon[91579]: pgmap v3873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:10 smithi188 ceph-mon[92425]: pgmap v3873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:12 smithi188 ceph-mon[92425]: pgmap v3874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:12 smithi161 ceph-mon[91579]: pgmap v3874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:14 smithi188 ceph-mon[92425]: pgmap v3875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:14 smithi161 ceph-mon[91579]: pgmap v3875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:16 smithi188 ceph-mon[92425]: pgmap v3876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:16 smithi161 ceph-mon[91579]: pgmap v3876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:18 smithi188 ceph-mon[92425]: pgmap v3877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:18.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:18 smithi161 ceph-mon[91579]: pgmap v3877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:20 smithi188 ceph-mon[92425]: pgmap v3878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:20 smithi161 ceph-mon[91579]: pgmap v3878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:22 smithi188 ceph-mon[92425]: pgmap v3879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:22 smithi161 ceph-mon[91579]: pgmap v3879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:24 smithi188 ceph-mon[92425]: pgmap v3880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:48:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:48:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:48:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:48:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:24 smithi161 ceph-mon[91579]: pgmap v3880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:48:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:48:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:48:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:48:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:26 smithi188 ceph-mon[92425]: pgmap v3881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:26 smithi161 ceph-mon[91579]: pgmap v3881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:28 smithi188 ceph-mon[92425]: pgmap v3882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:28 smithi161 ceph-mon[91579]: pgmap v3882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:30 smithi161 ceph-mon[91579]: pgmap v3883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:30 smithi188 ceph-mon[92425]: pgmap v3883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:32 smithi161 ceph-mon[91579]: pgmap v3884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:32 smithi188 ceph-mon[92425]: pgmap v3884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:34 smithi161 ceph-mon[91579]: pgmap v3885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:34 smithi188 ceph-mon[92425]: pgmap v3885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:36 smithi161 ceph-mon[91579]: pgmap v3886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:36 smithi188 ceph-mon[92425]: pgmap v3886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:38 smithi161 ceph-mon[91579]: pgmap v3887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:38 smithi188 ceph-mon[92425]: pgmap v3887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:40 smithi161 ceph-mon[91579]: pgmap v3888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:40 smithi188 ceph-mon[92425]: pgmap v3888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:42 smithi161 ceph-mon[91579]: pgmap v3889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:42 smithi188 ceph-mon[92425]: pgmap v3889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:44 smithi161 ceph-mon[91579]: pgmap v3890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:44 smithi188 ceph-mon[92425]: pgmap v3890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:46 smithi161 ceph-mon[91579]: pgmap v3891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:46 smithi188 ceph-mon[92425]: pgmap v3891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:48 smithi161 ceph-mon[91579]: pgmap v3892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:48 smithi188 ceph-mon[92425]: pgmap v3892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:50 smithi161 ceph-mon[91579]: pgmap v3893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:50 smithi188 ceph-mon[92425]: pgmap v3893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:52 smithi161 ceph-mon[91579]: pgmap v3894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:52 smithi188 ceph-mon[92425]: pgmap v3894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:53.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:48:53.527+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:48:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:48:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:48:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:48:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:48:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:48:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:48:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:48:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:48:53.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:48:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:48:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:48:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:48:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:48:53.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:48:53.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:48:53.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:48:53.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:48:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:48:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:48:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:48:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:48:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:48:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:48:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:48:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:48:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:48:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:48:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:48:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:48:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:48:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:48:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:48:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:48:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:48:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:48:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:48:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:48:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:48:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:48:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:48:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:48:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:48:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:48:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:48:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:48:53.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=193] Local address: 172.21.15.161, port 53186 2022-07-03T12:48:53.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=193] Peer address: 172.21.15.101, port 22 2022-07-03T12:48:53.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=568] Received exit status 0 2022-07-03T12:48:53.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=568] Received channel close 2022-07-03T12:48:53.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=568] Channel closed 2022-07-03T12:48:53.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=569] Requesting new SSH session 2022-07-03T12:48:53.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=569] Command: which python3 2022-07-03T12:48:53.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=568] Received exit status 0 2022-07-03T12:48:53.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=568] Received channel close 2022-07-03T12:48:53.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=568] Channel closed 2022-07-03T12:48:53.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=569] Requesting new SSH session 2022-07-03T12:48:53.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=569] Command: which python3 2022-07-03T12:48:53.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=569] Received exit status 0 2022-07-03T12:48:53.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=569] Received channel close 2022-07-03T12:48:53.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=569] Channel closed 2022-07-03T12:48:53.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=570] Requesting new SSH session 2022-07-03T12:48:53.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=570] Command: sudo true 2022-07-03T12:48:53.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=569] Received exit status 0 2022-07-03T12:48:53.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=569] Received channel close 2022-07-03T12:48:53.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=569] Channel closed 2022-07-03T12:48:53.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=570] Requesting new SSH session 2022-07-03T12:48:53.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=570] Command: sudo true 2022-07-03T12:48:53.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=570] Received exit status 0 2022-07-03T12:48:53.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=570] Received channel close 2022-07-03T12:48:53.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=570] Channel closed 2022-07-03T12:48:53.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=571] Requesting new SSH session 2022-07-03T12:48:53.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:53.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=570] Received exit status 0 2022-07-03T12:48:53.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=570] Received channel close 2022-07-03T12:48:53.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=570] Channel closed 2022-07-03T12:48:53.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=571] Requesting new SSH session 2022-07-03T12:48:53.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:53.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=571] Received exit status 0 2022-07-03T12:48:53.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=571] Received channel close 2022-07-03T12:48:53.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=571] Channel closed 2022-07-03T12:48:53.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=571] Received exit status 0 2022-07-03T12:48:53.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=571] Received channel close 2022-07-03T12:48:53.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=571] Channel closed 2022-07-03T12:48:53.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=193] Login timeout expired 2022-07-03T12:48:53.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=193] Aborting connection 2022-07-03T12:48:53.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:48:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:54 smithi161 ceph-mon[91579]: pgmap v3895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:48:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:48:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:48:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:48:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:48:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:48:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:48:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:48:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:48:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:48:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:48:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:48:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:48:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:48:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:48:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:48:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:48:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:48:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:48:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:48:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:48:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:48:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:48:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:48:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:48:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:48:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:48:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:48:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:48:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:48:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:48:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:48:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:48:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:48:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:48:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:48:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:48:54.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:48:54.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:48:54.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:48:54.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=193] Local address: 172.21.15.161, port 53186 2022-07-03T12:48:54.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=193] Peer address: 172.21.15.101, port 22 2022-07-03T12:48:54.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=568] Received exit status 0 2022-07-03T12:48:54.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=568] Received channel close 2022-07-03T12:48:54.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=568] Channel closed 2022-07-03T12:48:54.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=569] Requesting new SSH session 2022-07-03T12:48:54.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=569] Command: which python3 2022-07-03T12:48:54.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=568] Received exit status 0 2022-07-03T12:48:54.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=568] Received channel close 2022-07-03T12:48:54.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=568] Channel closed 2022-07-03T12:48:54.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=569] Requesting new SSH session 2022-07-03T12:48:54.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=569] Command: which python3 2022-07-03T12:48:54.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=569] Received exit status 0 2022-07-03T12:48:54.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=569] Received channel close 2022-07-03T12:48:54.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=569] Channel closed 2022-07-03T12:48:54.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=570] Requesting new SSH session 2022-07-03T12:48:54.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=570] Command: sudo true 2022-07-03T12:48:54.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=569] Received exit status 0 2022-07-03T12:48:54.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=569] Received channel close 2022-07-03T12:48:54.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=569] Channel closed 2022-07-03T12:48:54.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=570] Requesting new SSH session 2022-07-03T12:48:54.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=570] Command: sudo true 2022-07-03T12:48:54.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=570] Received exit status 0 2022-07-03T12:48:54.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=570] Received channel close 2022-07-03T12:48:54.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=570] Channel closed 2022-07-03T12:48:54.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=571] Requesting new SSH session 2022-07-03T12:48:54.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:54.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=570] Received exit status 0 2022-07-03T12:48:54.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=570] Received channel close 2022-07-03T12:48:54.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=570] Channel closed 2022-07-03T12:48:54.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=571] Requesting new SSH session 2022-07-03T12:48:54.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:54.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=571] Received exit status 0 2022-07-03T12:48:54.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=571] Received channel close 2022-07-03T12:48:54.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=571] Channel closed 2022-07-03T12:48:54.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=571] Received exit status 0 2022-07-03T12:48:54.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=571] Received channel close 2022-07-03T12:48:54.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=571] Channel closed 2022-07-03T12:48:54.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=193] Login timeout expired 2022-07-03T12:48:54.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=193] Aborting connection 2022-07-03T12:48:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:54 smithi188 ceph-mon[92425]: pgmap v3895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:48:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:48:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:48:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:48:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:48:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:48:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:48:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:48:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:48:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:48:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:48:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:48:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:48:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:48:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:48:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:48:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:48:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:48:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:48:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:48:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:48:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:48:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:48:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:48:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:48:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:48:54.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:48:54.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:48:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:48:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:48:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:48:54.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:48:54.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:48:54.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:48:54.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:48:54.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:48:54.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:48:54.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:48:54.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:48:54.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:48:54.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:48:54.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:48:54.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:48:54.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:48:54.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:48:54.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:48:54.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:48:54.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=193] Local address: 172.21.15.161, port 53186 2022-07-03T12:48:54.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=193] Peer address: 172.21.15.101, port 22 2022-07-03T12:48:54.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=568] Received exit status 0 2022-07-03T12:48:54.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=568] Received channel close 2022-07-03T12:48:54.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=568] Channel closed 2022-07-03T12:48:54.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=569] Requesting new SSH session 2022-07-03T12:48:54.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=569] Command: which python3 2022-07-03T12:48:54.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=568] Received exit status 0 2022-07-03T12:48:54.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=568] Received channel close 2022-07-03T12:48:54.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=568] Channel closed 2022-07-03T12:48:54.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=569] Requesting new SSH session 2022-07-03T12:48:54.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=569] Command: which python3 2022-07-03T12:48:54.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=569] Received exit status 0 2022-07-03T12:48:54.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=569] Received channel close 2022-07-03T12:48:54.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=569] Channel closed 2022-07-03T12:48:54.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=570] Requesting new SSH session 2022-07-03T12:48:54.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=570] Command: sudo true 2022-07-03T12:48:54.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=569] Received exit status 0 2022-07-03T12:48:54.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=569] Received channel close 2022-07-03T12:48:54.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=569] Channel closed 2022-07-03T12:48:54.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=570] Requesting new SSH session 2022-07-03T12:48:54.951 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=570] Command: sudo true 2022-07-03T12:48:54.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=570] Received exit status 0 2022-07-03T12:48:54.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=570] Received channel close 2022-07-03T12:48:54.952 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=570] Channel closed 2022-07-03T12:48:54.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=571] Requesting new SSH session 2022-07-03T12:48:54.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:54.953 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=570] Received exit status 0 2022-07-03T12:48:54.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=570] Received channel close 2022-07-03T12:48:54.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=570] Channel closed 2022-07-03T12:48:54.954 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=571] Requesting new SSH session 2022-07-03T12:48:54.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=571] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T12:48:54.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=571] Received exit status 0 2022-07-03T12:48:54.955 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=571] Received channel close 2022-07-03T12:48:54.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=571] Channel closed 2022-07-03T12:48:54.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=571] Received exit status 0 2022-07-03T12:48:54.956 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=571] Received channel close 2022-07-03T12:48:54.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=571] Channel closed 2022-07-03T12:48:54.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=193] Login timeout expired 2022-07-03T12:48:54.957 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=193] Aborting connection 2022-07-03T12:48:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:56 smithi161 ceph-mon[91579]: pgmap v3896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:56 smithi188 ceph-mon[92425]: pgmap v3896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:48:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:48:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:48:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:48:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:48:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:48:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:48:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:48:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:48:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:48:58 smithi161 ceph-mon[91579]: pgmap v3897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:48:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:48:58 smithi188 ceph-mon[92425]: pgmap v3897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:00 smithi161 ceph-mon[91579]: pgmap v3898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:00 smithi188 ceph-mon[92425]: pgmap v3898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:02 smithi161 ceph-mon[91579]: pgmap v3899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:02 smithi188 ceph-mon[92425]: pgmap v3899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:04 smithi161 ceph-mon[91579]: pgmap v3900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:04 smithi188 ceph-mon[92425]: pgmap v3900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:06 smithi161 ceph-mon[91579]: pgmap v3901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:06 smithi188 ceph-mon[92425]: pgmap v3901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:08 smithi161 ceph-mon[91579]: pgmap v3902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:08 smithi188 ceph-mon[92425]: pgmap v3902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:10 smithi161 ceph-mon[91579]: pgmap v3903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:10 smithi188 ceph-mon[92425]: pgmap v3903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:12 smithi161 ceph-mon[91579]: pgmap v3904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:12 smithi188 ceph-mon[92425]: pgmap v3904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:14 smithi161 ceph-mon[91579]: pgmap v3905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:14 smithi188 ceph-mon[92425]: pgmap v3905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:16 smithi161 ceph-mon[91579]: pgmap v3906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:16 smithi188 ceph-mon[92425]: pgmap v3906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:18 smithi161 ceph-mon[91579]: pgmap v3907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:18 smithi188 ceph-mon[92425]: pgmap v3907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:20 smithi161 ceph-mon[91579]: pgmap v3908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:20 smithi188 ceph-mon[92425]: pgmap v3908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:22 smithi161 ceph-mon[91579]: pgmap v3909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:22 smithi188 ceph-mon[92425]: pgmap v3909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:24 smithi161 ceph-mon[91579]: pgmap v3910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:24 smithi188 ceph-mon[92425]: pgmap v3910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:26 smithi161 ceph-mon[91579]: pgmap v3911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:26 smithi188 ceph-mon[92425]: pgmap v3911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:28 smithi161 ceph-mon[91579]: pgmap v3912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:28 smithi188 ceph-mon[92425]: pgmap v3912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:30 smithi161 ceph-mon[91579]: pgmap v3913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:30 smithi188 ceph-mon[92425]: pgmap v3913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:32 smithi161 ceph-mon[91579]: pgmap v3914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:32 smithi188 ceph-mon[92425]: pgmap v3914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:34 smithi161 ceph-mon[91579]: pgmap v3915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:34 smithi188 ceph-mon[92425]: pgmap v3915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:36 smithi161 ceph-mon[91579]: pgmap v3916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:36 smithi188 ceph-mon[92425]: pgmap v3916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:38 smithi161 ceph-mon[91579]: pgmap v3917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:38 smithi188 ceph-mon[92425]: pgmap v3917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:40 smithi161 ceph-mon[91579]: pgmap v3918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:40 smithi188 ceph-mon[92425]: pgmap v3918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:42 smithi161 ceph-mon[91579]: pgmap v3919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:42 smithi188 ceph-mon[92425]: pgmap v3919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:44 smithi161 ceph-mon[91579]: pgmap v3920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:44 smithi188 ceph-mon[92425]: pgmap v3920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:46 smithi161 ceph-mon[91579]: pgmap v3921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:46 smithi188 ceph-mon[92425]: pgmap v3921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:48 smithi161 ceph-mon[91579]: pgmap v3922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:48 smithi188 ceph-mon[92425]: pgmap v3922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:50 smithi161 ceph-mon[91579]: pgmap v3923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:50 smithi188 ceph-mon[92425]: pgmap v3923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:52 smithi161 ceph-mon[91579]: pgmap v3924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:52 smithi188 ceph-mon[92425]: pgmap v3924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:53.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:49:53.564+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:49:53.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:49:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:49:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:49:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:49:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:49:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:49:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:49:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:49:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:49:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:49:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:49:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:49:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:49:53.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:49:53.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:49:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:49:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:49:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:49:53.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:49:53.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:49:53.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:49:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:49:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:49:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:49:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:49:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:49:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:49:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:49:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:49:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:49:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:49:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:49:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:49:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:49:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:49:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:49:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:49:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:49:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:49:53.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:49:53.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:49:53.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:49:53.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:49:53.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:49:53.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:49:53.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=195] Local address: 172.21.15.161, port 53190 2022-07-03T12:49:53.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=195] Peer address: 172.21.15.101, port 22 2022-07-03T12:49:53.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=195] Connection lost 2022-07-03T12:49:53.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=195] Aborting connection 2022-07-03T12:49:53.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:49:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: pgmap v3925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:49:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:49:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:49:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:49:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:49:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:49:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:49:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:49:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:49:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:49:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:49:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:49:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:49:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:49:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:49:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:49:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:49:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:49:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:49:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:49:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:49:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:49:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:49:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:49:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:49:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:49:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:49:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:49:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:49:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:49:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:49:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:49:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:49:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:49:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:49:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:49:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:49:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:49:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:49:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:49:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:49:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:49:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:49:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:49:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:49:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:49:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:49:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:49:54.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=195] Local address: 172.21.15.161, port 53190 2022-07-03T12:49:54.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=195] Peer address: 172.21.15.101, port 22 2022-07-03T12:49:54.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=195] Connection lost 2022-07-03T12:49:54.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=195] Aborting connection 2022-07-03T12:49:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: pgmap v3925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:49:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:49:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:49:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:49:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:49:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:49:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:49:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:49:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:49:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:49:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:49:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:49:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:49:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:49:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:49:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:49:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:49:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:49:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:49:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:49:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:49:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:49:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:49:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:49:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:49:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:49:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:49:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:49:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:49:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:49:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:49:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:49:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:49:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:49:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:49:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:49:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:49:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:49:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:49:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:49:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:49:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:49:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:49:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:49:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:49:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:49:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:49:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:49:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=195] Local address: 172.21.15.161, port 53190 2022-07-03T12:49:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=195] Peer address: 172.21.15.101, port 22 2022-07-03T12:49:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=195] Connection lost 2022-07-03T12:49:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=195] Aborting connection 2022-07-03T12:49:56.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:56 smithi161 ceph-mon[91579]: pgmap v3926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:56.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:56 smithi188 ceph-mon[92425]: pgmap v3926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:49:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:49:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:49:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:49:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:49:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:49:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:49:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:49:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:49:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:49:58 smithi161 ceph-mon[91579]: pgmap v3927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:49:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:49:58 smithi188 ceph-mon[92425]: pgmap v3927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:00 smithi161 ceph-mon[91579]: pgmap v3928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:50:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:00 smithi188 ceph-mon[92425]: pgmap v3928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T12:50:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:02 smithi161 ceph-mon[91579]: pgmap v3929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:02 smithi188 ceph-mon[92425]: pgmap v3929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:04 smithi161 ceph-mon[91579]: pgmap v3930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:04 smithi188 ceph-mon[92425]: pgmap v3930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:06 smithi161 ceph-mon[91579]: pgmap v3931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:06 smithi188 ceph-mon[92425]: pgmap v3931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:08 smithi161 ceph-mon[91579]: pgmap v3932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:08 smithi188 ceph-mon[92425]: pgmap v3932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:10 smithi161 ceph-mon[91579]: pgmap v3933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:10 smithi188 ceph-mon[92425]: pgmap v3933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:12 smithi161 ceph-mon[91579]: pgmap v3934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:12 smithi188 ceph-mon[92425]: pgmap v3934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:14 smithi188 ceph-mon[92425]: pgmap v3935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:14 smithi161 ceph-mon[91579]: pgmap v3935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:16 smithi188 ceph-mon[92425]: pgmap v3936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:16 smithi161 ceph-mon[91579]: pgmap v3936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:18 smithi188 ceph-mon[92425]: pgmap v3937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:18 smithi161 ceph-mon[91579]: pgmap v3937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:20 smithi188 ceph-mon[92425]: pgmap v3938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:20 smithi161 ceph-mon[91579]: pgmap v3938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:22 smithi188 ceph-mon[92425]: pgmap v3939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:22 smithi161 ceph-mon[91579]: pgmap v3939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:24 smithi188 ceph-mon[92425]: pgmap v3940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:24 smithi161 ceph-mon[91579]: pgmap v3940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:26 smithi188 ceph-mon[92425]: pgmap v3941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:26 smithi161 ceph-mon[91579]: pgmap v3941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:28 smithi188 ceph-mon[92425]: pgmap v3942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:28 smithi161 ceph-mon[91579]: pgmap v3942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:30 smithi188 ceph-mon[92425]: pgmap v3943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:30 smithi161 ceph-mon[91579]: pgmap v3943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:32 smithi161 ceph-mon[91579]: pgmap v3944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:32 smithi188 ceph-mon[92425]: pgmap v3944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:34 smithi161 ceph-mon[91579]: pgmap v3945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:34 smithi188 ceph-mon[92425]: pgmap v3945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:36 smithi161 ceph-mon[91579]: pgmap v3946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:36 smithi188 ceph-mon[92425]: pgmap v3946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:38 smithi161 ceph-mon[91579]: pgmap v3947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:38 smithi188 ceph-mon[92425]: pgmap v3947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:40 smithi161 ceph-mon[91579]: pgmap v3948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:40 smithi188 ceph-mon[92425]: pgmap v3948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:42 smithi161 ceph-mon[91579]: pgmap v3949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:42 smithi188 ceph-mon[92425]: pgmap v3949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:44 smithi161 ceph-mon[91579]: pgmap v3950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:44 smithi188 ceph-mon[92425]: pgmap v3950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:46 smithi161 ceph-mon[91579]: pgmap v3951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:46 smithi188 ceph-mon[92425]: pgmap v3951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:48 smithi161 ceph-mon[91579]: pgmap v3952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:48 smithi188 ceph-mon[92425]: pgmap v3952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:50 smithi161 ceph-mon[91579]: pgmap v3953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:50 smithi188 ceph-mon[92425]: pgmap v3953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:52 smithi161 ceph-mon[91579]: pgmap v3954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:52 smithi188 ceph-mon[92425]: pgmap v3954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:54 smithi161 ceph-mon[91579]: pgmap v3955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:50:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:50:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:50:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:54 smithi188 ceph-mon[92425]: pgmap v3955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:50:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:50:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:50:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:50:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:56 smithi161 ceph-mon[91579]: pgmap v3956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:56 smithi188 ceph-mon[92425]: pgmap v3956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:50:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:50:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:50:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:50:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:50:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:50:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:50:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:50:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:50:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:50:58 smithi161 ceph-mon[91579]: pgmap v3957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:50:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:50:58 smithi188 ceph-mon[92425]: pgmap v3957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:00 smithi161 ceph-mon[91579]: pgmap v3958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:00 smithi188 ceph-mon[92425]: pgmap v3958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:02 smithi161 ceph-mon[91579]: pgmap v3959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:02 smithi188 ceph-mon[92425]: pgmap v3959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:04 smithi161 ceph-mon[91579]: pgmap v3960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:04 smithi188 ceph-mon[92425]: pgmap v3960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:06 smithi161 ceph-mon[91579]: pgmap v3961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:06 smithi188 ceph-mon[92425]: pgmap v3961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:08 smithi161 ceph-mon[91579]: pgmap v3962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:08 smithi188 ceph-mon[92425]: pgmap v3962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:10 smithi161 ceph-mon[91579]: pgmap v3963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:10 smithi188 ceph-mon[92425]: pgmap v3963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:12 smithi161 ceph-mon[91579]: pgmap v3964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:12 smithi188 ceph-mon[92425]: pgmap v3964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:14 smithi161 ceph-mon[91579]: pgmap v3965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:14 smithi188 ceph-mon[92425]: pgmap v3965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:16 smithi161 ceph-mon[91579]: pgmap v3966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:16 smithi188 ceph-mon[92425]: pgmap v3966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:18 smithi161 ceph-mon[91579]: pgmap v3967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:18 smithi188 ceph-mon[92425]: pgmap v3967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:20 smithi161 ceph-mon[91579]: pgmap v3968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:20 smithi188 ceph-mon[92425]: pgmap v3968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:22 smithi161 ceph-mon[91579]: pgmap v3969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:22 smithi188 ceph-mon[92425]: pgmap v3969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:24.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:51:24.074+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:51:24.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:51:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:51:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:51:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:51:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:51:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:51:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:51:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:51:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:51:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:51:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:51:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:51:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:51:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:51:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:51:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:51:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:51:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:51:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:51:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:51:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:51:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:51:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:51:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:51:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:51:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:51:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:51:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:51:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:51:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:51:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:51:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:51:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:51:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:51:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:51:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:51:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:51:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:51:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:51:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:51:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:51:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=197] Local address: 172.21.15.161, port 53194 2022-07-03T12:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=197] Peer address: 172.21.15.101, port 22 2022-07-03T12:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=197] Connection lost 2022-07-03T12:51:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=197] Aborting connection 2022-07-03T12:51:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:51:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:24 smithi161 ceph-mon[91579]: pgmap v3970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:24 smithi188 ceph-mon[92425]: pgmap v3970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:51:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:51:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:51:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:51:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:51:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:51:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:51:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:51:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:51:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:51:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:51:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:51:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:51:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:51:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:51:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:51:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:51:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:51:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:51:25.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:51:25.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:51:25.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:51:25.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:51:25.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:51:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:51:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:51:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:51:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:51:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:51:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:51:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:51:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:51:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:51:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:51:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:51:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:51:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:51:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:51:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:51:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:51:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:51:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:51:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:51:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:51:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:51:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:51:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:51:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=197] Local address: 172.21.15.161, port 53194 2022-07-03T12:51:25.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=197] Peer address: 172.21.15.101, port 22 2022-07-03T12:51:25.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=197] Connection lost 2022-07-03T12:51:25.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=197] Aborting connection 2022-07-03T12:51:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:51:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:51:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:51:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:51:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:51:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:51:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:51:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:51:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:51:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:51:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:51:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:51:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:51:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:51:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:51:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:51:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:51:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:51:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:51:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:51:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:51:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:51:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:51:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:51:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:51:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:51:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:51:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:51:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:51:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:51:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:51:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:51:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:51:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:51:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:51:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:51:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:51:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:51:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:51:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:51:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:51:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:51:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:51:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:51:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:51:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:51:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:51:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=197] Local address: 172.21.15.161, port 53194 2022-07-03T12:51:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=197] Peer address: 172.21.15.101, port 22 2022-07-03T12:51:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=197] Connection lost 2022-07-03T12:51:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=197] Aborting connection 2022-07-03T12:51:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:26 smithi161 ceph-mon[91579]: pgmap v3971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:26 smithi188 ceph-mon[92425]: pgmap v3971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:28 smithi161 ceph-mon[91579]: pgmap v3972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:28 smithi188 ceph-mon[92425]: pgmap v3972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:30 smithi161 ceph-mon[91579]: pgmap v3973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:30 smithi188 ceph-mon[92425]: pgmap v3973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:32 smithi161 ceph-mon[91579]: pgmap v3974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:32 smithi188 ceph-mon[92425]: pgmap v3974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:34 smithi161 ceph-mon[91579]: pgmap v3975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:34 smithi188 ceph-mon[92425]: pgmap v3975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:36 smithi161 ceph-mon[91579]: pgmap v3976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:36 smithi188 ceph-mon[92425]: pgmap v3976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:38 smithi161 ceph-mon[91579]: pgmap v3977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:38 smithi188 ceph-mon[92425]: pgmap v3977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:40 smithi161 ceph-mon[91579]: pgmap v3978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:40 smithi188 ceph-mon[92425]: pgmap v3978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:42 smithi161 ceph-mon[91579]: pgmap v3979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:42 smithi188 ceph-mon[92425]: pgmap v3979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:44 smithi161 ceph-mon[91579]: pgmap v3980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:44 smithi188 ceph-mon[92425]: pgmap v3980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:46 smithi161 ceph-mon[91579]: pgmap v3981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:46 smithi188 ceph-mon[92425]: pgmap v3981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:48 smithi161 ceph-mon[91579]: pgmap v3982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:48 smithi188 ceph-mon[92425]: pgmap v3982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:50 smithi161 ceph-mon[91579]: pgmap v3983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:50 smithi188 ceph-mon[92425]: pgmap v3983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:52 smithi161 ceph-mon[91579]: pgmap v3984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:52 smithi188 ceph-mon[92425]: pgmap v3984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:54 smithi161 ceph-mon[91579]: pgmap v3985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:54 smithi188 ceph-mon[92425]: pgmap v3985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:51:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:56 smithi161 ceph-mon[91579]: pgmap v3986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:56 smithi188 ceph-mon[92425]: pgmap v3986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:51:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:51:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:51:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:51:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:51:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:51:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:51:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:51:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:51:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:51:58 smithi161 ceph-mon[91579]: pgmap v3987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:51:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:51:58 smithi188 ceph-mon[92425]: pgmap v3987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:00 smithi161 ceph-mon[91579]: pgmap v3988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:00 smithi188 ceph-mon[92425]: pgmap v3988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:02 smithi161 ceph-mon[91579]: pgmap v3989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:02 smithi188 ceph-mon[92425]: pgmap v3989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:04 smithi161 ceph-mon[91579]: pgmap v3990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:04 smithi188 ceph-mon[92425]: pgmap v3990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:06 smithi161 ceph-mon[91579]: pgmap v3991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:06 smithi188 ceph-mon[92425]: pgmap v3991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:08 smithi161 ceph-mon[91579]: pgmap v3992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:08 smithi188 ceph-mon[92425]: pgmap v3992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:10 smithi161 ceph-mon[91579]: pgmap v3993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:10 smithi188 ceph-mon[92425]: pgmap v3993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:12 smithi188 ceph-mon[92425]: pgmap v3994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:12 smithi161 ceph-mon[91579]: pgmap v3994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:14 smithi188 ceph-mon[92425]: pgmap v3995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:14 smithi161 ceph-mon[91579]: pgmap v3995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:16 smithi188 ceph-mon[92425]: pgmap v3996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:16 smithi161 ceph-mon[91579]: pgmap v3996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:18 smithi188 ceph-mon[92425]: pgmap v3997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:18 smithi161 ceph-mon[91579]: pgmap v3997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:20 smithi188 ceph-mon[92425]: pgmap v3998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:20 smithi161 ceph-mon[91579]: pgmap v3998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:22 smithi188 ceph-mon[92425]: pgmap v3999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:22 smithi161 ceph-mon[91579]: pgmap v3999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:24.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:52:24.110+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:52:24.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:52:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:52:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:52:24.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:52:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:52:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:52:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:52:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:52:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:52:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:52:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:52:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:52:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:52:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:52:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:52:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:52:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:52:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:52:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:52:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:52:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:52:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:52:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:52:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:52:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:52:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:52:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:52:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:52:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:52:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:52:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:52:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:52:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:52:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:52:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:52:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:52:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:52:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:52:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:52:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:52:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:52:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:52:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:52:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:52:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:52:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=199] Local address: 172.21.15.161, port 53198 2022-07-03T12:52:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=199] Peer address: 172.21.15.101, port 22 2022-07-03T12:52:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=199] Connection lost 2022-07-03T12:52:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=199] Aborting connection 2022-07-03T12:52:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:52:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:24 smithi188 ceph-mon[92425]: pgmap v4000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:52:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:52:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:52:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-mon[91579]: pgmap v4000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:52:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:52:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:52:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:52:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:52:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:52:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:52:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:52:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:52:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:52:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:52:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:52:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:52:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:52:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:52:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:52:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:52:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:52:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:52:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:52:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:52:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:52:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:52:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:52:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:52:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:52:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:52:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:52:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:52:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:52:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:52:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:52:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:52:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:52:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:52:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:52:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:52:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:52:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:52:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:52:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:52:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:52:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:52:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:52:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:52:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:52:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:52:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=199] Local address: 172.21.15.161, port 53198 2022-07-03T12:52:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=199] Peer address: 172.21.15.101, port 22 2022-07-03T12:52:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=199] Connection lost 2022-07-03T12:52:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=199] Aborting connection 2022-07-03T12:52:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:52:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:52:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:52:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:52:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:52:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:52:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:52:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:52:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:52:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:52:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:52:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:52:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:52:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:52:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=199] Local address: 172.21.15.161, port 53198 2022-07-03T12:52:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=199] Peer address: 172.21.15.101, port 22 2022-07-03T12:52:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=199] Connection lost 2022-07-03T12:52:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=199] Aborting connection 2022-07-03T12:52:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:26 smithi188 ceph-mon[92425]: pgmap v4001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:26 smithi161 ceph-mon[91579]: pgmap v4001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:28 smithi161 ceph-mon[91579]: pgmap v4002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:28 smithi188 ceph-mon[92425]: pgmap v4002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:30 smithi161 ceph-mon[91579]: pgmap v4003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:30 smithi188 ceph-mon[92425]: pgmap v4003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:32 smithi161 ceph-mon[91579]: pgmap v4004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:32 smithi188 ceph-mon[92425]: pgmap v4004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:34 smithi161 ceph-mon[91579]: pgmap v4005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:34 smithi188 ceph-mon[92425]: pgmap v4005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:36 smithi161 ceph-mon[91579]: pgmap v4006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:36 smithi188 ceph-mon[92425]: pgmap v4006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:38 smithi161 ceph-mon[91579]: pgmap v4007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:38 smithi188 ceph-mon[92425]: pgmap v4007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T12:52:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:40 smithi161 ceph-mon[91579]: pgmap v4008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:40 smithi188 ceph-mon[92425]: pgmap v4008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:42 smithi161 ceph-mon[91579]: pgmap v4009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:42 smithi188 ceph-mon[92425]: pgmap v4009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:44 smithi161 ceph-mon[91579]: pgmap v4010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:44 smithi188 ceph-mon[92425]: pgmap v4010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:46 smithi161 ceph-mon[91579]: pgmap v4011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:46 smithi188 ceph-mon[92425]: pgmap v4011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:48 smithi161 ceph-mon[91579]: pgmap v4012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:48 smithi188 ceph-mon[92425]: pgmap v4012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:50 smithi161 ceph-mon[91579]: pgmap v4013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:50 smithi188 ceph-mon[92425]: pgmap v4013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:52 smithi161 ceph-mon[91579]: pgmap v4014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:52 smithi188 ceph-mon[92425]: pgmap v4014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:54 smithi161 ceph-mon[91579]: pgmap v4015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:54 smithi188 ceph-mon[92425]: pgmap v4015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:52:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:56 smithi161 ceph-mon[91579]: pgmap v4016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:56 smithi188 ceph-mon[92425]: pgmap v4016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:52:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:52:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:52:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:52:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:52:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:52:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:52:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:52:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:52:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:52:58 smithi161 ceph-mon[91579]: pgmap v4017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:52:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:52:58 smithi188 ceph-mon[92425]: pgmap v4017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:00 smithi161 ceph-mon[91579]: pgmap v4018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:00 smithi188 ceph-mon[92425]: pgmap v4018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:02 smithi161 ceph-mon[91579]: pgmap v4019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:02 smithi188 ceph-mon[92425]: pgmap v4019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:04 smithi161 ceph-mon[91579]: pgmap v4020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:04 smithi188 ceph-mon[92425]: pgmap v4020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:06 smithi161 ceph-mon[91579]: pgmap v4021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:06 smithi188 ceph-mon[92425]: pgmap v4021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:08 smithi161 ceph-mon[91579]: pgmap v4022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:08 smithi188 ceph-mon[92425]: pgmap v4022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:10 smithi161 ceph-mon[91579]: pgmap v4023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:10 smithi188 ceph-mon[92425]: pgmap v4023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:12 smithi161 ceph-mon[91579]: pgmap v4024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:12 smithi188 ceph-mon[92425]: pgmap v4024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:14 smithi161 ceph-mon[91579]: pgmap v4025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:14 smithi188 ceph-mon[92425]: pgmap v4025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:16 smithi161 ceph-mon[91579]: pgmap v4026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:16 smithi188 ceph-mon[92425]: pgmap v4026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:18 smithi161 ceph-mon[91579]: pgmap v4027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:18 smithi188 ceph-mon[92425]: pgmap v4027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:20 smithi161 ceph-mon[91579]: pgmap v4028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:20 smithi188 ceph-mon[92425]: pgmap v4028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:22 smithi161 ceph-mon[91579]: pgmap v4029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:22 smithi188 ceph-mon[92425]: pgmap v4029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:24.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:53:24.608+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:53:24.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:53:24.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:53:24.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:53:24.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:53:24.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:53:24.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:53:24.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:53:24.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:53:24.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:53:24.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:53:24.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:53:24.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:53:24.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:53:24.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:53:24.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:53:24.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:53:24.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:53:24.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:53:24.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:53:24.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:53:24.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:53:24.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:53:24.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:53:24.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:53:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:53:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:53:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:53:24.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:53:24.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:53:24.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:53:24.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:53:24.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:53:24.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:53:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:53:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:53:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:53:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:53:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:53:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:53:24.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:53:24.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:53:24.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:53:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:53:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:53:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:53:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=201] Local address: 172.21.15.161, port 53202 2022-07-03T12:53:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=201] Peer address: 172.21.15.101, port 22 2022-07-03T12:53:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=201] Connection lost 2022-07-03T12:53:24.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=201] Aborting connection 2022-07-03T12:53:24.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:53:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-mon[91579]: pgmap v4030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:53:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:53:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:53:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:24 smithi188 ceph-mon[92425]: pgmap v4030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:53:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:53:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:53:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:53:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:53:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:53:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:53:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:53:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:53:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:53:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:53:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:53:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:53:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:53:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:53:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:53:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:53:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:53:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:53:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:53:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:53:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:53:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:53:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:53:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:53:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:53:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:53:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:53:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:53:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:53:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:53:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:53:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:53:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:53:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:53:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:53:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:53:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:53:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:53:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:53:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:53:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:53:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:53:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:53:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:53:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:53:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:53:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:53:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:53:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:53:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=201] Local address: 172.21.15.161, port 53202 2022-07-03T12:53:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=201] Peer address: 172.21.15.101, port 22 2022-07-03T12:53:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=201] Connection lost 2022-07-03T12:53:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=201] Aborting connection 2022-07-03T12:53:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:53:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:53:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:53:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:53:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:53:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:53:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:53:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:53:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:53:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:53:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:53:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:53:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:53:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:53:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:53:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:53:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:53:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:53:26.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:53:26.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:53:26.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:53:26.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:53:26.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:53:26.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:53:26.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:53:26.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:53:26.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:53:26.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:53:26.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:53:26.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:53:26.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:53:26.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:53:26.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:53:26.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:53:26.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:53:26.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:53:26.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:53:26.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:53:26.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:53:26.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:53:26.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:53:26.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:53:26.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:53:26.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:53:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:53:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:53:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:53:26.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=201] Local address: 172.21.15.161, port 53202 2022-07-03T12:53:26.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=201] Peer address: 172.21.15.101, port 22 2022-07-03T12:53:26.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=201] Connection lost 2022-07-03T12:53:26.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=201] Aborting connection 2022-07-03T12:53:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:26 smithi161 ceph-mon[91579]: pgmap v4031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:26 smithi188 ceph-mon[92425]: pgmap v4031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:28 smithi161 ceph-mon[91579]: pgmap v4032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:28 smithi188 ceph-mon[92425]: pgmap v4032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:30 smithi161 ceph-mon[91579]: pgmap v4033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:30 smithi188 ceph-mon[92425]: pgmap v4033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:32 smithi161 ceph-mon[91579]: pgmap v4034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:32 smithi188 ceph-mon[92425]: pgmap v4034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:34 smithi161 ceph-mon[91579]: pgmap v4035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:34 smithi188 ceph-mon[92425]: pgmap v4035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:36 smithi161 ceph-mon[91579]: pgmap v4036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:36 smithi188 ceph-mon[92425]: pgmap v4036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:38 smithi161 ceph-mon[91579]: pgmap v4037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:38 smithi188 ceph-mon[92425]: pgmap v4037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:40 smithi161 ceph-mon[91579]: pgmap v4038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:40 smithi188 ceph-mon[92425]: pgmap v4038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:42 smithi161 ceph-mon[91579]: pgmap v4039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:42 smithi188 ceph-mon[92425]: pgmap v4039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:44 smithi161 ceph-mon[91579]: pgmap v4040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:44 smithi188 ceph-mon[92425]: pgmap v4040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:46 smithi161 ceph-mon[91579]: pgmap v4041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:46 smithi188 ceph-mon[92425]: pgmap v4041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:48 smithi161 ceph-mon[91579]: pgmap v4042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:48 smithi188 ceph-mon[92425]: pgmap v4042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:50 smithi161 ceph-mon[91579]: pgmap v4043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:50 smithi188 ceph-mon[92425]: pgmap v4043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:52 smithi161 ceph-mon[91579]: pgmap v4044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:52 smithi188 ceph-mon[92425]: pgmap v4044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:54 smithi161 ceph-mon[91579]: pgmap v4045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:54 smithi188 ceph-mon[92425]: pgmap v4045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:53:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:56 smithi161 ceph-mon[91579]: pgmap v4046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:56 smithi188 ceph-mon[92425]: pgmap v4046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:53:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:53:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:53:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:53:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:53:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:53:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:53:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:53:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:53:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:53:58 smithi161 ceph-mon[91579]: pgmap v4047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:53:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:53:58 smithi188 ceph-mon[92425]: pgmap v4047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:00 smithi161 ceph-mon[91579]: pgmap v4048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:00 smithi188 ceph-mon[92425]: pgmap v4048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:02 smithi161 ceph-mon[91579]: pgmap v4049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:02 smithi188 ceph-mon[92425]: pgmap v4049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:04 smithi161 ceph-mon[91579]: pgmap v4050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:04 smithi188 ceph-mon[92425]: pgmap v4050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:06 smithi161 ceph-mon[91579]: pgmap v4051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:06 smithi188 ceph-mon[92425]: pgmap v4051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:08 smithi188 ceph-mon[92425]: pgmap v4052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:08 smithi161 ceph-mon[91579]: pgmap v4052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:10 smithi188 ceph-mon[92425]: pgmap v4053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:10 smithi161 ceph-mon[91579]: pgmap v4053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:12 smithi188 ceph-mon[92425]: pgmap v4054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:12 smithi161 ceph-mon[91579]: pgmap v4054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:14 smithi188 ceph-mon[92425]: pgmap v4055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:14 smithi161 ceph-mon[91579]: pgmap v4055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:16 smithi188 ceph-mon[92425]: pgmap v4056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:16 smithi161 ceph-mon[91579]: pgmap v4056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:18 smithi188 ceph-mon[92425]: pgmap v4057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:18 smithi161 ceph-mon[91579]: pgmap v4057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:20 smithi188 ceph-mon[92425]: pgmap v4058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:20 smithi161 ceph-mon[91579]: pgmap v4058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:22 smithi188 ceph-mon[92425]: pgmap v4059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:22 smithi161 ceph-mon[91579]: pgmap v4059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:25.172 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:24 smithi161 ceph-mon[91579]: pgmap v4060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:25.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:54:25.113+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:54:25.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:54:25.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:54:25.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:54:25.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:54:25.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:54:25.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:54:25.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:54:25.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:54:25.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:54:25.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:54:25.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:54:25.177 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:54:25.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:54:25.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:54:25.178 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:54:25.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:54:25.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:54:25.179 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:54:25.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:54:25.180 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:54:25.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:54:25.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:54:25.181 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:54:25.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:54:25.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:54:25.182 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:54:25.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:54:25.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:54:25.183 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:54:25.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:54:25.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:54:25.184 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:54:25.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:54:25.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:54:25.185 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:54:25.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:54:25.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:54:25.186 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:54:25.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:54:25.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:54:25.187 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:54:25.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:54:25.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:54:25.188 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:54:25.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:54:25.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=203] Local address: 172.21.15.161, port 53238 2022-07-03T12:54:25.189 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=203] Peer address: 172.21.15.101, port 22 2022-07-03T12:54:25.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=203] Connection lost 2022-07-03T12:54:25.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=203] Aborting connection 2022-07-03T12:54:25.190 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:54:25.191 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:24 smithi188 ceph-mon[92425]: pgmap v4060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:54:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:54:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:54:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:54:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:54:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:54:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:54:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:54:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:54:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:54:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:54:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:54:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:54:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:54:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:54:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:54:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:54:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:54:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:54:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:54:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:54:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:54:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:54:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:54:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:54:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:54:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:54:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:54:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:54:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:54:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:54:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:54:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:54:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:54:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:54:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:54:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:54:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:54:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:54:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:54:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:54:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:54:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:54:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:54:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:54:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:54:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:54:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:54:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:54:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:54:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:54:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:54:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:54:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=203] Local address: 172.21.15.161, port 53238 2022-07-03T12:54:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=203] Peer address: 172.21.15.101, port 22 2022-07-03T12:54:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=203] Connection lost 2022-07-03T12:54:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=203] Aborting connection 2022-07-03T12:54:27.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:26 smithi161 ceph-mon[91579]: pgmap v4061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:27.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:54:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:54:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:54:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:54:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:54:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:54:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:54:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:54:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:54:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:54:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:54:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:54:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:54:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:54:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:54:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:54:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:54:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:54:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:54:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:54:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:54:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:54:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:54:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:54:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:54:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:54:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:54:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:54:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:54:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:54:27.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:54:27.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:54:27.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:54:27.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:54:27.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:54:27.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:54:27.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:54:27.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:54:27.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:54:27.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:54:27.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:54:27.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:54:27.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:54:27.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:54:27.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:54:27.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:54:27.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:54:27.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=203] Local address: 172.21.15.161, port 53238 2022-07-03T12:54:27.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=203] Peer address: 172.21.15.101, port 22 2022-07-03T12:54:27.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=203] Connection lost 2022-07-03T12:54:27.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=203] Aborting connection 2022-07-03T12:54:27.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:26 smithi188 ceph-mon[92425]: pgmap v4061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:27.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:28 smithi161 ceph-mon[91579]: pgmap v4062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:28 smithi188 ceph-mon[92425]: pgmap v4062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:30 smithi161 ceph-mon[91579]: pgmap v4063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:30 smithi188 ceph-mon[92425]: pgmap v4063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:32 smithi161 ceph-mon[91579]: pgmap v4064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:32 smithi188 ceph-mon[92425]: pgmap v4064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:34 smithi161 ceph-mon[91579]: pgmap v4065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:34 smithi188 ceph-mon[92425]: pgmap v4065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:36 smithi161 ceph-mon[91579]: pgmap v4066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:36 smithi188 ceph-mon[92425]: pgmap v4066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:38 smithi161 ceph-mon[91579]: pgmap v4067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:38 smithi188 ceph-mon[92425]: pgmap v4067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:40 smithi161 ceph-mon[91579]: pgmap v4068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:40 smithi188 ceph-mon[92425]: pgmap v4068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:42 smithi161 ceph-mon[91579]: pgmap v4069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:42 smithi188 ceph-mon[92425]: pgmap v4069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:44 smithi161 ceph-mon[91579]: pgmap v4070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:44 smithi188 ceph-mon[92425]: pgmap v4070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:46 smithi161 ceph-mon[91579]: pgmap v4071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:46 smithi188 ceph-mon[92425]: pgmap v4071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:48 smithi161 ceph-mon[91579]: pgmap v4072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:48 smithi188 ceph-mon[92425]: pgmap v4072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:50 smithi161 ceph-mon[91579]: pgmap v4073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:50 smithi188 ceph-mon[92425]: pgmap v4073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:52 smithi161 ceph-mon[91579]: pgmap v4074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:52 smithi188 ceph-mon[92425]: pgmap v4074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:54 smithi161 ceph-mon[91579]: pgmap v4075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:54 smithi188 ceph-mon[92425]: pgmap v4075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:54:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:56 smithi161 ceph-mon[91579]: pgmap v4076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:56 smithi188 ceph-mon[92425]: pgmap v4076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:54:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:54:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:54:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:54:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:54:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:54:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:54:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:54:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:54:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:54:58 smithi161 ceph-mon[91579]: pgmap v4077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:54:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:54:58 smithi188 ceph-mon[92425]: pgmap v4077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:00 smithi161 ceph-mon[91579]: pgmap v4078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:00 smithi188 ceph-mon[92425]: pgmap v4078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:02 smithi161 ceph-mon[91579]: pgmap v4079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:02 smithi188 ceph-mon[92425]: pgmap v4079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:04 smithi188 ceph-mon[92425]: pgmap v4080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:04 smithi161 ceph-mon[91579]: pgmap v4080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:06 smithi188 ceph-mon[92425]: pgmap v4081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:06 smithi161 ceph-mon[91579]: pgmap v4081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:08 smithi188 ceph-mon[92425]: pgmap v4082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:08 smithi161 ceph-mon[91579]: pgmap v4082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:10 smithi188 ceph-mon[92425]: pgmap v4083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:10 smithi161 ceph-mon[91579]: pgmap v4083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:12 smithi161 ceph-mon[91579]: pgmap v4084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:12 smithi188 ceph-mon[92425]: pgmap v4084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:14 smithi161 ceph-mon[91579]: pgmap v4085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:14 smithi188 ceph-mon[92425]: pgmap v4085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:16 smithi161 ceph-mon[91579]: pgmap v4086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:16 smithi188 ceph-mon[92425]: pgmap v4086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:18 smithi161 ceph-mon[91579]: pgmap v4087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:18 smithi188 ceph-mon[92425]: pgmap v4087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:20 smithi161 ceph-mon[91579]: pgmap v4088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:20 smithi188 ceph-mon[92425]: pgmap v4088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:22 smithi161 ceph-mon[91579]: pgmap v4089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:22 smithi188 ceph-mon[92425]: pgmap v4089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:24 smithi161 ceph-mon[91579]: pgmap v4090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:24 smithi188 ceph-mon[92425]: pgmap v4090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:25.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:55:25.613+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:55:25.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:55:25.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:55:25.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:55:25.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:55:25.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:55:25.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:55:25.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:55:25.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:55:25.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:55:25.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:55:25.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:55:25.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:55:25.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:55:25.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:55:25.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:55:25.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:55:25.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:55:25.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:55:25.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:55:25.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:55:25.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:55:25.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:55:25.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:55:25.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:55:25.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:55:25.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:55:25.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:55:25.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:55:25.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:55:25.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:55:25.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:55:25.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:55:25.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:55:25.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:55:25.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:55:25.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:55:25.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:55:25.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:55:25.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:55:25.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:55:25.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:55:25.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:55:25.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:55:25.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:55:25.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:55:25.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=205] Local address: 172.21.15.161, port 53242 2022-07-03T12:55:25.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=205] Peer address: 172.21.15.101, port 22 2022-07-03T12:55:25.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=205] Connection lost 2022-07-03T12:55:25.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=205] Aborting connection 2022-07-03T12:55:25.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:55:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:55:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: pgmap v4091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:55:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:55:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:55:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:55:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:55:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:55:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:55:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:55:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:55:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:55:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:55:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:55:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:55:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:55:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:55:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:55:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:55:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:55:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:55:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:55:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:55:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:55:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:55:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:55:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:55:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:55:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:55:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:55:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:55:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:55:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:55:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:55:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:55:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:55:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:55:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:55:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:55:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:55:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:55:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:55:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:55:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:55:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:55:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:55:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:55:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:55:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:55:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:55:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:55:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:55:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=205] Local address: 172.21.15.161, port 53242 2022-07-03T12:55:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=205] Peer address: 172.21.15.101, port 22 2022-07-03T12:55:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=205] Connection lost 2022-07-03T12:55:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=205] Aborting connection 2022-07-03T12:55:26.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: pgmap v4091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:55:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:55:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:55:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:55:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:55:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:55:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:55:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:55:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:55:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:55:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:55:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:55:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:55:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:55:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:55:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:55:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:55:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:55:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:55:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:55:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:55:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:55:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:55:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:55:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:55:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:55:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:55:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:55:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:55:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:55:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:55:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:55:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:55:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:55:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:55:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:55:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:55:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:55:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:55:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:55:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:55:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:55:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:55:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:55:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:55:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:55:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:55:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:55:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:55:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:55:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=205] Local address: 172.21.15.161, port 53242 2022-07-03T12:55:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=205] Peer address: 172.21.15.101, port 22 2022-07-03T12:55:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=205] Connection lost 2022-07-03T12:55:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=205] Aborting connection 2022-07-03T12:55:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:28 smithi161 ceph-mon[91579]: pgmap v4092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:28 smithi188 ceph-mon[92425]: pgmap v4092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:30 smithi161 ceph-mon[91579]: pgmap v4093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:30 smithi188 ceph-mon[92425]: pgmap v4093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:32 smithi161 ceph-mon[91579]: pgmap v4094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:32 smithi188 ceph-mon[92425]: pgmap v4094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:34 smithi161 ceph-mon[91579]: pgmap v4095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:34 smithi188 ceph-mon[92425]: pgmap v4095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:36 smithi161 ceph-mon[91579]: pgmap v4096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:36 smithi188 ceph-mon[92425]: pgmap v4096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:38 smithi161 ceph-mon[91579]: pgmap v4097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:38 smithi188 ceph-mon[92425]: pgmap v4097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:40 smithi161 ceph-mon[91579]: pgmap v4098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:40 smithi188 ceph-mon[92425]: pgmap v4098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:42 smithi161 ceph-mon[91579]: pgmap v4099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:42 smithi188 ceph-mon[92425]: pgmap v4099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:44 smithi161 ceph-mon[91579]: pgmap v4100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:44 smithi188 ceph-mon[92425]: pgmap v4100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:46 smithi161 ceph-mon[91579]: pgmap v4101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:46 smithi188 ceph-mon[92425]: pgmap v4101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:48 smithi161 ceph-mon[91579]: pgmap v4102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:48 smithi188 ceph-mon[92425]: pgmap v4102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:50 smithi161 ceph-mon[91579]: pgmap v4103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:50 smithi188 ceph-mon[92425]: pgmap v4103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:52 smithi161 ceph-mon[91579]: pgmap v4104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:52 smithi188 ceph-mon[92425]: pgmap v4104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:54 smithi161 ceph-mon[91579]: pgmap v4105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:54 smithi188 ceph-mon[92425]: pgmap v4105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:55:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:56 smithi161 ceph-mon[91579]: pgmap v4106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:56 smithi188 ceph-mon[92425]: pgmap v4106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:55:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:55:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:55:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:55:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:55:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:55:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:55:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:55:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:55:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:55:58 smithi161 ceph-mon[91579]: pgmap v4107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:55:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:55:58 smithi188 ceph-mon[92425]: pgmap v4107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:00 smithi161 ceph-mon[91579]: pgmap v4108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:00 smithi188 ceph-mon[92425]: pgmap v4108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:02 smithi161 ceph-mon[91579]: pgmap v4109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:02 smithi188 ceph-mon[92425]: pgmap v4109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:04 smithi161 ceph-mon[91579]: pgmap v4110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:04 smithi188 ceph-mon[92425]: pgmap v4110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:06 smithi161 ceph-mon[91579]: pgmap v4111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:06 smithi188 ceph-mon[92425]: pgmap v4111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:08 smithi161 ceph-mon[91579]: pgmap v4112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:08 smithi188 ceph-mon[92425]: pgmap v4112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:10 smithi161 ceph-mon[91579]: pgmap v4113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:10 smithi188 ceph-mon[92425]: pgmap v4113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:12 smithi161 ceph-mon[91579]: pgmap v4114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:12 smithi188 ceph-mon[92425]: pgmap v4114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:14 smithi161 ceph-mon[91579]: pgmap v4115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:14 smithi188 ceph-mon[92425]: pgmap v4115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:16 smithi161 ceph-mon[91579]: pgmap v4116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:16 smithi188 ceph-mon[92425]: pgmap v4116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:18 smithi161 ceph-mon[91579]: pgmap v4117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:18 smithi188 ceph-mon[92425]: pgmap v4117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:20 smithi161 ceph-mon[91579]: pgmap v4118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:20 smithi188 ceph-mon[92425]: pgmap v4118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:22 smithi161 ceph-mon[91579]: pgmap v4119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:22 smithi188 ceph-mon[92425]: pgmap v4119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:24 smithi161 ceph-mon[91579]: pgmap v4120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:24 smithi188 ceph-mon[92425]: pgmap v4120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:26 smithi161 ceph-mon[91579]: pgmap v4121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:56:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:56:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:56:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:26 smithi188 ceph-mon[92425]: pgmap v4121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:56:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:56:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:56:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:28 smithi161 ceph-mon[91579]: pgmap v4122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:28 smithi188 ceph-mon[92425]: pgmap v4122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:30 smithi161 ceph-mon[91579]: pgmap v4123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:30 smithi188 ceph-mon[92425]: pgmap v4123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:32 smithi161 ceph-mon[91579]: pgmap v4124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:32 smithi188 ceph-mon[92425]: pgmap v4124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:34 smithi161 ceph-mon[91579]: pgmap v4125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:34 smithi188 ceph-mon[92425]: pgmap v4125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:36 smithi161 ceph-mon[91579]: pgmap v4126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:36 smithi188 ceph-mon[92425]: pgmap v4126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:38 smithi161 ceph-mon[91579]: pgmap v4127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:38 smithi188 ceph-mon[92425]: pgmap v4127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:40 smithi161 ceph-mon[91579]: pgmap v4128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:40 smithi188 ceph-mon[92425]: pgmap v4128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:42 smithi161 ceph-mon[91579]: pgmap v4129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:42 smithi188 ceph-mon[92425]: pgmap v4129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:44 smithi161 ceph-mon[91579]: pgmap v4130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:44 smithi188 ceph-mon[92425]: pgmap v4130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:46 smithi161 ceph-mon[91579]: pgmap v4131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:46 smithi188 ceph-mon[92425]: pgmap v4131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:48 smithi161 ceph-mon[91579]: pgmap v4132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:48 smithi188 ceph-mon[92425]: pgmap v4132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:50 smithi161 ceph-mon[91579]: pgmap v4133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:50 smithi188 ceph-mon[92425]: pgmap v4133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:52 smithi161 ceph-mon[91579]: pgmap v4134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:52 smithi188 ceph-mon[92425]: pgmap v4134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:54 smithi161 ceph-mon[91579]: pgmap v4135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:54 smithi188 ceph-mon[92425]: pgmap v4135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:56:56.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:56:56.119+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:56:56.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:56:56.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:56:56.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:56:56.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:56:56.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:56:56.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:56:56.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:56:56.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:56:56.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:56:56.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:56:56.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:56:56.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:56:56.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:56:56.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:56:56.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:56:56.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:56:56.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:56:56.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:56:56.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:56:56.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:56:56.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:56:56.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:56:56.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:56:56.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:56:56.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:56:56.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:56:56.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:56:56.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:56:56.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:56:56.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:56:56.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:56:56.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:56:56.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:56:56.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:56:56.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:56:56.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:56:56.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:56:56.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:56:56.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:56:56.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:56:56.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:56:56.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:56:56.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:56:56.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:56:56.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:56:56.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=207] Local address: 172.21.15.161, port 53246 2022-07-03T12:56:56.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=207] Peer address: 172.21.15.101, port 22 2022-07-03T12:56:56.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=207] Connection lost 2022-07-03T12:56:56.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=207] Aborting connection 2022-07-03T12:56:56.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:56:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-mon[91579]: pgmap v4136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:56 smithi188 ceph-mon[92425]: pgmap v4136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:56:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:56:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:56:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:56:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:56:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:56:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:56:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:56:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:56:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:56:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:56:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:56:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:56:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:56:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:56:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:56:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:56:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:56:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:56:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:56:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:56:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:56:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:56:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:56:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:56:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:56:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:56:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:56:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:56:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:56:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:56:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:56:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:56:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:56:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:56:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:56:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:56:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:56:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:56:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:56:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:56:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:56:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:56:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:56:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:56:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:56:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:56:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:56:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=207] Local address: 172.21.15.161, port 53246 2022-07-03T12:56:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=207] Peer address: 172.21.15.101, port 22 2022-07-03T12:56:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=207] Connection lost 2022-07-03T12:56:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=207] Aborting connection 2022-07-03T12:56:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:56:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:56:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:56:57.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:56:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:56:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:56:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:56:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:56:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:56:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:56:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:56:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:56:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:56:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:56:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:56:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:56:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:56:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:56:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:56:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:56:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:56:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:56:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:56:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:56:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:56:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:56:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:56:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:56:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:56:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:56:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:56:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:56:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:56:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:56:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:56:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:56:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:56:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:56:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:56:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:56:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:56:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:56:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:56:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:56:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:56:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:56:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:56:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:56:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:56:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:56:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:56:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=207] Local address: 172.21.15.161, port 53246 2022-07-03T12:56:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=207] Peer address: 172.21.15.101, port 22 2022-07-03T12:56:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=207] Connection lost 2022-07-03T12:56:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=207] Aborting connection 2022-07-03T12:56:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:56:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:56:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:56:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:56:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:56:58 smithi161 ceph-mon[91579]: pgmap v4137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:56:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:56:58 smithi188 ceph-mon[92425]: pgmap v4137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:00 smithi161 ceph-mon[91579]: pgmap v4138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:00 smithi188 ceph-mon[92425]: pgmap v4138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:02 smithi161 ceph-mon[91579]: pgmap v4139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:02 smithi188 ceph-mon[92425]: pgmap v4139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:04 smithi188 ceph-mon[92425]: pgmap v4140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:04 smithi161 ceph-mon[91579]: pgmap v4140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:06 smithi188 ceph-mon[92425]: pgmap v4141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:06 smithi161 ceph-mon[91579]: pgmap v4141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:08 smithi188 ceph-mon[92425]: pgmap v4142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:08 smithi161 ceph-mon[91579]: pgmap v4142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:10 smithi188 ceph-mon[92425]: pgmap v4143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:10 smithi161 ceph-mon[91579]: pgmap v4143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:12 smithi188 ceph-mon[92425]: pgmap v4144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:12 smithi161 ceph-mon[91579]: pgmap v4144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:14 smithi188 ceph-mon[92425]: pgmap v4145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:14 smithi161 ceph-mon[91579]: pgmap v4145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:16 smithi188 ceph-mon[92425]: pgmap v4146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:16 smithi161 ceph-mon[91579]: pgmap v4146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:18 smithi188 ceph-mon[92425]: pgmap v4147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:20 smithi188 ceph-mon[92425]: pgmap v4148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:18 smithi161 ceph-mon[91579]: pgmap v4147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:20 smithi161 ceph-mon[91579]: pgmap v4148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:22 smithi161 ceph-mon[91579]: pgmap v4149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:22 smithi188 ceph-mon[92425]: pgmap v4149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:24 smithi161 ceph-mon[91579]: pgmap v4150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:24 smithi188 ceph-mon[92425]: pgmap v4150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:26 smithi161 ceph-mon[91579]: pgmap v4151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:26 smithi188 ceph-mon[92425]: pgmap v4151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:28 smithi161 ceph-mon[91579]: pgmap v4152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:28 smithi188 ceph-mon[92425]: pgmap v4152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:30 smithi161 ceph-mon[91579]: pgmap v4153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:30 smithi188 ceph-mon[92425]: pgmap v4153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:32 smithi161 ceph-mon[91579]: pgmap v4154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:32 smithi188 ceph-mon[92425]: pgmap v4154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:34 smithi161 ceph-mon[91579]: pgmap v4155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:34 smithi188 ceph-mon[92425]: pgmap v4155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:36 smithi161 ceph-mon[91579]: pgmap v4156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:36 smithi188 ceph-mon[92425]: pgmap v4156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:38 smithi161 ceph-mon[91579]: pgmap v4157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:38 smithi188 ceph-mon[92425]: pgmap v4157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:40 smithi161 ceph-mon[91579]: pgmap v4158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:40 smithi188 ceph-mon[92425]: pgmap v4158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:42 smithi161 ceph-mon[91579]: pgmap v4159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:42 smithi188 ceph-mon[92425]: pgmap v4159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:44 smithi161 ceph-mon[91579]: pgmap v4160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:44 smithi188 ceph-mon[92425]: pgmap v4160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:46 smithi161 ceph-mon[91579]: pgmap v4161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:46 smithi188 ceph-mon[92425]: pgmap v4161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:48 smithi161 ceph-mon[91579]: pgmap v4162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:48 smithi188 ceph-mon[92425]: pgmap v4162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:50 smithi161 ceph-mon[91579]: pgmap v4163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:50 smithi188 ceph-mon[92425]: pgmap v4163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:52 smithi161 ceph-mon[91579]: pgmap v4164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:52 smithi188 ceph-mon[92425]: pgmap v4164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:54 smithi161 ceph-mon[91579]: pgmap v4165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:54 smithi188 ceph-mon[92425]: pgmap v4165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:57:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: pgmap v4166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:57:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:57:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:57:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: pgmap v4166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:57:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:57:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:57:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:57:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:57:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:57:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:57:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:57:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:57:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:57:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:57:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:57:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:57:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:57:58 smithi161 ceph-mon[91579]: pgmap v4167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:57:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:57:58 smithi188 ceph-mon[92425]: pgmap v4167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: pgmap v4168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:58:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:58:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:58:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:00 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:58:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: pgmap v4168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T12:58:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T12:58:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T12:58:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T12:58:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T12:58:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:00 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T12:58:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:02 smithi161 ceph-mon[91579]: pgmap v4169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:02 smithi188 ceph-mon[92425]: pgmap v4169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:04 smithi161 ceph-mon[91579]: pgmap v4170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:04 smithi188 ceph-mon[92425]: pgmap v4170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:06 smithi161 ceph-mon[91579]: pgmap v4171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:06 smithi188 ceph-mon[92425]: pgmap v4171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:08 smithi161 ceph-mon[91579]: pgmap v4172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:08 smithi188 ceph-mon[92425]: pgmap v4172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:10 smithi161 ceph-mon[91579]: pgmap v4173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:10 smithi188 ceph-mon[92425]: pgmap v4173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:12 smithi161 ceph-mon[91579]: pgmap v4174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:12 smithi188 ceph-mon[92425]: pgmap v4174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:14 smithi161 ceph-mon[91579]: pgmap v4175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:14 smithi188 ceph-mon[92425]: pgmap v4175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:16 smithi161 ceph-mon[91579]: pgmap v4176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:16 smithi188 ceph-mon[92425]: pgmap v4176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:18 smithi161 ceph-mon[91579]: pgmap v4177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:18 smithi188 ceph-mon[92425]: pgmap v4177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:20 smithi161 ceph-mon[91579]: pgmap v4178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:20 smithi188 ceph-mon[92425]: pgmap v4178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:22 smithi161 ceph-mon[91579]: pgmap v4179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:22 smithi188 ceph-mon[92425]: pgmap v4179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:24 smithi161 ceph-mon[91579]: pgmap v4180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:24 smithi188 ceph-mon[92425]: pgmap v4180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:26 smithi161 ceph-mon[91579]: pgmap v4181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:26 smithi188 ceph-mon[92425]: pgmap v4181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:58:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:58:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:28 smithi161 ceph-mon[91579]: pgmap v4182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:28 smithi188 ceph-mon[92425]: pgmap v4182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:30 smithi161 ceph-mon[91579]: pgmap v4183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:30 smithi188 ceph-mon[92425]: pgmap v4183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:32 smithi161 ceph-mon[91579]: pgmap v4184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:32 smithi188 ceph-mon[92425]: pgmap v4184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:34 smithi161 ceph-mon[91579]: pgmap v4185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:34 smithi188 ceph-mon[92425]: pgmap v4185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:36 smithi161 ceph-mon[91579]: pgmap v4186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:36 smithi188 ceph-mon[92425]: pgmap v4186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:38 smithi161 ceph-mon[91579]: pgmap v4187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:38 smithi188 ceph-mon[92425]: pgmap v4187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:40 smithi161 ceph-mon[91579]: pgmap v4188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:40 smithi188 ceph-mon[92425]: pgmap v4188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:42 smithi161 ceph-mon[91579]: pgmap v4189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:42 smithi188 ceph-mon[92425]: pgmap v4189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:44 smithi161 ceph-mon[91579]: pgmap v4190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:44 smithi188 ceph-mon[92425]: pgmap v4190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:46 smithi161 ceph-mon[91579]: pgmap v4191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:46 smithi188 ceph-mon[92425]: pgmap v4191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:48 smithi161 ceph-mon[91579]: pgmap v4192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:48 smithi188 ceph-mon[92425]: pgmap v4192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:50 smithi161 ceph-mon[91579]: pgmap v4193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:50 smithi188 ceph-mon[92425]: pgmap v4193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:52 smithi161 ceph-mon[91579]: pgmap v4194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:52 smithi188 ceph-mon[92425]: pgmap v4194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:54 smithi161 ceph-mon[91579]: pgmap v4195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:58:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:54 smithi188 ceph-mon[92425]: pgmap v4195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:58:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:58:56.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:58:56.159+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:58:56.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:58:56.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:58:56.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:58:56.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:58:56.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:58:56.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:58:56.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:58:56.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:58:56.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:58:56.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:58:56.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:58:56.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:58:56.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:58:56.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:58:56.550 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:58:56.550 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:58:56.551 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:58:56.551 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:58:56.551 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:58:56.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:58:56.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:58:56.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:58:56.553 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:58:56.553 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:58:56.553 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:58:56.554 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:58:56.554 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:58:56.554 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:58:56.555 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:58:56.555 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:58:56.555 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:58:56.556 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:58:56.556 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:58:56.556 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:58:56.557 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:58:56.557 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:58:56.557 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:58:56.558 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:58:56.558 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:58:56.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:58:56.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:58:56.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:58:56.560 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:58:56.560 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:58:56.560 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:58:56.561 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=209] Local address: 172.21.15.161, port 53250 2022-07-03T12:58:56.561 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=209] Peer address: 172.21.15.101, port 22 2022-07-03T12:58:56.561 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=209] Login timeout expired 2022-07-03T12:58:56.562 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=209] Aborting connection 2022-07-03T12:58:56.562 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:58:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-mon[91579]: pgmap v4196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:56 smithi188 ceph-mon[92425]: pgmap v4196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:58:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:58:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:58:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:58:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:58:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:58:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:58:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:58:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:58:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:58:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:58:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:58:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:58:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:58:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:58:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:58:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:58:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:58:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:58:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:58:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:58:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:58:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:58:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:58:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:58:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:58:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:58:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:58:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:58:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:58:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:58:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:58:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:58:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:58:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:58:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:58:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:58:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:58:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:58:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:58:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:58:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:58:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:58:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:58:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:58:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:58:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:58:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=209] Local address: 172.21.15.161, port 53250 2022-07-03T12:58:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=209] Peer address: 172.21.15.101, port 22 2022-07-03T12:58:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=209] Login timeout expired 2022-07-03T12:58:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=209] Aborting connection 2022-07-03T12:58:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:58:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:58:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:58:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:58:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:58:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:58:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:58:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:58:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:58:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:58:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:58:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:58:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:58:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:58:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:58:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:58:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T12:58:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:58:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:58:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:58:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:58:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:58:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:58:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:58:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:58:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:58:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:58:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:58:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:58:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:58:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:58:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:58:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:58:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:58:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:58:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:58:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:58:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:58:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:58:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:58:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:58:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:58:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:58:57.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:58:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:58:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:58:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T12:58:57.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:58:57.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:58:57.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=209] Local address: 172.21.15.161, port 53250 2022-07-03T12:58:57.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=209] Peer address: 172.21.15.101, port 22 2022-07-03T12:58:57.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=209] Login timeout expired 2022-07-03T12:58:57.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=209] Aborting connection 2022-07-03T12:58:57.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:58:57.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:58:57.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:58:57.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:58:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:58 smithi161 ceph-mon[91579]: pgmap v4197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:58 smithi188 ceph-mon[92425]: pgmap v4197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:58:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:58:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:00 smithi161 ceph-mon[91579]: pgmap v4198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:00 smithi188 ceph-mon[92425]: pgmap v4198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:02 smithi161 ceph-mon[91579]: pgmap v4199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:02 smithi188 ceph-mon[92425]: pgmap v4199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:04 smithi161 ceph-mon[91579]: pgmap v4200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:04 smithi188 ceph-mon[92425]: pgmap v4200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:06 smithi161 ceph-mon[91579]: pgmap v4201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:06 smithi188 ceph-mon[92425]: pgmap v4201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:08 smithi161 ceph-mon[91579]: pgmap v4202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:08 smithi188 ceph-mon[92425]: pgmap v4202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:10 smithi161 ceph-mon[91579]: pgmap v4203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:10 smithi188 ceph-mon[92425]: pgmap v4203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:12 smithi161 ceph-mon[91579]: pgmap v4204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:12 smithi188 ceph-mon[92425]: pgmap v4204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:14 smithi161 ceph-mon[91579]: pgmap v4205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:14 smithi188 ceph-mon[92425]: pgmap v4205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:16 smithi161 ceph-mon[91579]: pgmap v4206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:16 smithi188 ceph-mon[92425]: pgmap v4206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:18 smithi161 ceph-mon[91579]: pgmap v4207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:18 smithi188 ceph-mon[92425]: pgmap v4207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:20 smithi161 ceph-mon[91579]: pgmap v4208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:20 smithi188 ceph-mon[92425]: pgmap v4208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:22 smithi161 ceph-mon[91579]: pgmap v4209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:22 smithi188 ceph-mon[92425]: pgmap v4209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:24 smithi161 ceph-mon[91579]: pgmap v4210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:24 smithi188 ceph-mon[92425]: pgmap v4210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:26 smithi161 ceph-mon[91579]: pgmap v4211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:26 smithi188 ceph-mon[92425]: pgmap v4211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:28 smithi161 ceph-mon[91579]: pgmap v4212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:28 smithi188 ceph-mon[92425]: pgmap v4212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:30 smithi161 ceph-mon[91579]: pgmap v4213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:30 smithi188 ceph-mon[92425]: pgmap v4213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:32 smithi188 ceph-mon[92425]: pgmap v4214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:32 smithi161 ceph-mon[91579]: pgmap v4214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:34 smithi188 ceph-mon[92425]: pgmap v4215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:34 smithi161 ceph-mon[91579]: pgmap v4215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:36 smithi188 ceph-mon[92425]: pgmap v4216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:36 smithi161 ceph-mon[91579]: pgmap v4216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:38 smithi188 ceph-mon[92425]: pgmap v4217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:38 smithi161 ceph-mon[91579]: pgmap v4217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:40 smithi188 ceph-mon[92425]: pgmap v4218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:40 smithi161 ceph-mon[91579]: pgmap v4218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:42 smithi188 ceph-mon[92425]: pgmap v4219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:42 smithi161 ceph-mon[91579]: pgmap v4219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:44 smithi188 ceph-mon[92425]: pgmap v4220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:44 smithi161 ceph-mon[91579]: pgmap v4220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:46 smithi188 ceph-mon[92425]: pgmap v4221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:46 smithi161 ceph-mon[91579]: pgmap v4221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:48 smithi161 ceph-mon[91579]: pgmap v4222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:48 smithi188 ceph-mon[92425]: pgmap v4222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:50 smithi161 ceph-mon[91579]: pgmap v4223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:50 smithi188 ceph-mon[92425]: pgmap v4223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:52 smithi161 ceph-mon[91579]: pgmap v4224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:52 smithi188 ceph-mon[92425]: pgmap v4224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:54 smithi161 ceph-mon[91579]: pgmap v4225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:54 smithi188 ceph-mon[92425]: pgmap v4225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T12:59:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T12:59:56.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:59:56.194+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:59:56.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:59:56.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:59:56.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T12:59:56.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:59:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:59:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:59:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T12:59:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:59:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T12:59:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:59:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T12:59:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:59:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:59:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T12:59:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:59:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T12:59:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:59:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T12:59:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:59:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T12:59:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:59:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:59:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:59:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:59:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:59:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T12:59:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:59:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:59:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:59:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T12:59:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:59:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T12:59:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:59:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:59:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:59:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T12:59:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:59:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T12:59:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:59:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T12:59:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:59:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T12:59:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:59:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:59:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:59:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=211] Local address: 172.21.15.161, port 53282 2022-07-03T12:59:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=211] Peer address: 172.21.15.101, port 22 2022-07-03T12:59:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=211] Connection lost 2022-07-03T12:59:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=211] Aborting connection 2022-07-03T12:59:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T12:59:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-mon[91579]: pgmap v4226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:59:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:59:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:59:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:56 smithi188 ceph-mon[92425]: pgmap v4226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T12:59:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T12:59:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T12:59:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T12:59:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:59:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:59:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:59:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:59:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T12:59:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:59:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:59:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:59:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T12:59:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:59:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T12:59:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:59:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T12:59:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:59:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:59:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:59:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T12:59:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T12:59:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:59:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T12:59:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:59:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T12:59:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:59:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:59:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:59:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:59:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:59:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:59:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:59:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:59:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:59:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T12:59:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:59:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T12:59:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:59:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:59:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:59:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:59:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:59:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T12:59:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:59:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:59:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:59:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T12:59:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:59:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:59:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:59:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=211] Local address: 172.21.15.161, port 53282 2022-07-03T12:59:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=211] Peer address: 172.21.15.101, port 22 2022-07-03T12:59:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=211] Connection lost 2022-07-03T12:59:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=211] Aborting connection 2022-07-03T12:59:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:59:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:59:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:59:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:59:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T12:59:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T12:59:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:59:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T12:59:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T12:59:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T12:59:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T12:59:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T12:59:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T12:59:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T12:59:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T12:59:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T12:59:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T12:59:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T12:59:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:59:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T12:59:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T12:59:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T12:59:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T12:59:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T12:59:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T12:59:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T12:59:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T12:59:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T12:59:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T12:59:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T12:59:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T12:59:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T12:59:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T12:59:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T12:59:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T12:59:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T12:59:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T12:59:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T12:59:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T12:59:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T12:59:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T12:59:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T12:59:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T12:59:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T12:59:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T12:59:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T12:59:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T12:59:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T12:59:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T12:59:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T12:59:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T12:59:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=211] Local address: 172.21.15.161, port 53282 2022-07-03T12:59:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=211] Peer address: 172.21.15.101, port 22 2022-07-03T12:59:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=211] Connection lost 2022-07-03T12:59:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=211] Aborting connection 2022-07-03T12:59:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:59:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:59:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T12:59:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T12:59:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:58 smithi161 ceph-mon[91579]: pgmap v4227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T12:59:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:58 smithi188 ceph-mon[92425]: pgmap v4227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 12:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 12:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:00 smithi161 ceph-mon[91579]: pgmap v4228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:00:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:00 smithi188 ceph-mon[92425]: pgmap v4228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:00:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:02 smithi161 ceph-mon[91579]: pgmap v4229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:02 smithi188 ceph-mon[92425]: pgmap v4229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:04 smithi161 ceph-mon[91579]: pgmap v4230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:04 smithi188 ceph-mon[92425]: pgmap v4230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:06 smithi161 ceph-mon[91579]: pgmap v4231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:06 smithi188 ceph-mon[92425]: pgmap v4231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:08 smithi161 ceph-mon[91579]: pgmap v4232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:08 smithi188 ceph-mon[92425]: pgmap v4232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:10 smithi161 ceph-mon[91579]: pgmap v4233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:10 smithi188 ceph-mon[92425]: pgmap v4233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:12 smithi161 ceph-mon[91579]: pgmap v4234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:12 smithi188 ceph-mon[92425]: pgmap v4234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:14 smithi161 ceph-mon[91579]: pgmap v4235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:14 smithi188 ceph-mon[92425]: pgmap v4235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:16 smithi161 ceph-mon[91579]: pgmap v4236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:16 smithi188 ceph-mon[92425]: pgmap v4236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:18 smithi161 ceph-mon[91579]: pgmap v4237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:18 smithi188 ceph-mon[92425]: pgmap v4237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:20 smithi161 ceph-mon[91579]: pgmap v4238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:20 smithi188 ceph-mon[92425]: pgmap v4238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:22 smithi161 ceph-mon[91579]: pgmap v4239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:22 smithi188 ceph-mon[92425]: pgmap v4239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:24 smithi161 ceph-mon[91579]: pgmap v4240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:24 smithi188 ceph-mon[92425]: pgmap v4240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:26 smithi161 ceph-mon[91579]: pgmap v4241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:26 smithi188 ceph-mon[92425]: pgmap v4241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:28 smithi161 ceph-mon[91579]: pgmap v4242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:28 smithi188 ceph-mon[92425]: pgmap v4242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:30 smithi161 ceph-mon[91579]: pgmap v4243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:30 smithi188 ceph-mon[92425]: pgmap v4243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:32 smithi161 ceph-mon[91579]: pgmap v4244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:32 smithi188 ceph-mon[92425]: pgmap v4244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:34 smithi161 ceph-mon[91579]: pgmap v4245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:34 smithi188 ceph-mon[92425]: pgmap v4245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:36 smithi161 ceph-mon[91579]: pgmap v4246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:36 smithi188 ceph-mon[92425]: pgmap v4246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:38 smithi161 ceph-mon[91579]: pgmap v4247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:38 smithi188 ceph-mon[92425]: pgmap v4247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:40 smithi161 ceph-mon[91579]: pgmap v4248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:40 smithi188 ceph-mon[92425]: pgmap v4248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:42 smithi161 ceph-mon[91579]: pgmap v4249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:42 smithi188 ceph-mon[92425]: pgmap v4249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:44 smithi161 ceph-mon[91579]: pgmap v4250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:44 smithi188 ceph-mon[92425]: pgmap v4250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:46 smithi161 ceph-mon[91579]: pgmap v4251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:46 smithi188 ceph-mon[92425]: pgmap v4251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:48 smithi161 ceph-mon[91579]: pgmap v4252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:48 smithi188 ceph-mon[92425]: pgmap v4252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:50 smithi161 ceph-mon[91579]: pgmap v4253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:50 smithi188 ceph-mon[92425]: pgmap v4253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:52 smithi161 ceph-mon[91579]: pgmap v4254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:52 smithi188 ceph-mon[92425]: pgmap v4254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:54 smithi161 ceph-mon[91579]: pgmap v4255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:54 smithi188 ceph-mon[92425]: pgmap v4255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:00:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:00:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-mon[91579]: pgmap v4256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:00:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:00:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:00:57.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:00:56.683+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:00:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:00:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:00:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:00:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:00:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:00:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:00:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:00:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:00:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:00:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:00:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:00:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:00:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:00:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:00:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:00:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:00:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:00:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:00:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:00:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:00:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:00:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:00:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:00:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:00:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:00:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:00:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:00:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:00:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:00:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:00:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:00:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:00:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:00:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:00:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:00:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:00:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:00:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:00:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:00:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:00:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:00:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:00:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:00:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:00:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:00:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=213] Local address: 172.21.15.161, port 53310 2022-07-03T13:00:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=213] Peer address: 172.21.15.101, port 22 2022-07-03T13:00:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=213] Connection lost 2022-07-03T13:00:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=213] Aborting connection 2022-07-03T13:00:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:00:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:00:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:56 smithi188 ceph-mon[92425]: pgmap v4256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:00:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:00:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:00:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:00:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:00:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:00:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:00:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:00:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:00:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:00:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:00:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:00:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:00:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:00:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:00:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:00:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:00:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:00:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:00:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:00:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:00:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:00:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:00:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:00:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:00:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:00:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:00:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:00:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:00:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:00:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:00:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:00:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:00:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:00:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:00:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:00:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:00:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:00:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:00:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:00:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:00:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:00:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:00:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:00:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:00:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:00:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:00:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:00:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:00:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:00:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:00:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=213] Local address: 172.21.15.161, port 53310 2022-07-03T13:00:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=213] Peer address: 172.21.15.101, port 22 2022-07-03T13:00:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=213] Connection lost 2022-07-03T13:00:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=213] Aborting connection 2022-07-03T13:00:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:00:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:00:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:00:58.150 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:00:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:00:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:00:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:00:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:00:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:00:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:00:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:00:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:00:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:00:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:00:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:00:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:00:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:00:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:00:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:00:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:00:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:00:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:00:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:00:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:00:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:00:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:00:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:00:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:00:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:00:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:00:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:00:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:00:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:00:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:00:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:00:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:00:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:00:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:00:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:00:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:00:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:00:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:00:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:00:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:00:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:00:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:00:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:00:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:00:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:00:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:00:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:00:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=213] Local address: 172.21.15.161, port 53310 2022-07-03T13:00:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=213] Peer address: 172.21.15.101, port 22 2022-07-03T13:00:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=213] Connection lost 2022-07-03T13:00:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=213] Aborting connection 2022-07-03T13:00:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:00:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:00:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:00:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:00:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:58 smithi161 ceph-mon[91579]: pgmap v4257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:00:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:58 smithi188 ceph-mon[92425]: pgmap v4257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:00 smithi161 ceph-mon[91579]: pgmap v4258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:00 smithi188 ceph-mon[92425]: pgmap v4258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:02 smithi161 ceph-mon[91579]: pgmap v4259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:02 smithi188 ceph-mon[92425]: pgmap v4259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:04 smithi161 ceph-mon[91579]: pgmap v4260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:04 smithi188 ceph-mon[92425]: pgmap v4260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:06 smithi161 ceph-mon[91579]: pgmap v4261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:06 smithi188 ceph-mon[92425]: pgmap v4261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:08 smithi161 ceph-mon[91579]: pgmap v4262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:08 smithi188 ceph-mon[92425]: pgmap v4262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:10 smithi161 ceph-mon[91579]: pgmap v4263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:10 smithi188 ceph-mon[92425]: pgmap v4263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:12 smithi188 ceph-mon[92425]: pgmap v4264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:12 smithi161 ceph-mon[91579]: pgmap v4264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:14 smithi188 ceph-mon[92425]: pgmap v4265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:14 smithi161 ceph-mon[91579]: pgmap v4265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:16 smithi188 ceph-mon[92425]: pgmap v4266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:16 smithi161 ceph-mon[91579]: pgmap v4266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:18 smithi188 ceph-mon[92425]: pgmap v4267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:18 smithi161 ceph-mon[91579]: pgmap v4267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:20 smithi188 ceph-mon[92425]: pgmap v4268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:20 smithi161 ceph-mon[91579]: pgmap v4268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:22 smithi188 ceph-mon[92425]: pgmap v4269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:22 smithi161 ceph-mon[91579]: pgmap v4269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:24 smithi188 ceph-mon[92425]: pgmap v4270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:24 smithi161 ceph-mon[91579]: pgmap v4270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:26 smithi188 ceph-mon[92425]: pgmap v4271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:26 smithi161 ceph-mon[91579]: pgmap v4271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:28 smithi161 ceph-mon[91579]: pgmap v4272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:28 smithi188 ceph-mon[92425]: pgmap v4272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:30 smithi161 ceph-mon[91579]: pgmap v4273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:30 smithi188 ceph-mon[92425]: pgmap v4273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:32 smithi161 ceph-mon[91579]: pgmap v4274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:32 smithi188 ceph-mon[92425]: pgmap v4274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:01:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:34 smithi161 ceph-mon[91579]: pgmap v4275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:34 smithi188 ceph-mon[92425]: pgmap v4275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:36 smithi161 ceph-mon[91579]: pgmap v4276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:36 smithi188 ceph-mon[92425]: pgmap v4276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:38 smithi161 ceph-mon[91579]: pgmap v4277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:38 smithi188 ceph-mon[92425]: pgmap v4277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:40 smithi161 ceph-mon[91579]: pgmap v4278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:40 smithi188 ceph-mon[92425]: pgmap v4278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:42 smithi161 ceph-mon[91579]: pgmap v4279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:42 smithi188 ceph-mon[92425]: pgmap v4279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:44 smithi161 ceph-mon[91579]: pgmap v4280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:44 smithi188 ceph-mon[92425]: pgmap v4280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:46 smithi161 ceph-mon[91579]: pgmap v4281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:46 smithi188 ceph-mon[92425]: pgmap v4281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:48 smithi161 ceph-mon[91579]: pgmap v4282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:48 smithi188 ceph-mon[92425]: pgmap v4282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:50 smithi161 ceph-mon[91579]: pgmap v4283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:50 smithi188 ceph-mon[92425]: pgmap v4283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:52 smithi161 ceph-mon[91579]: pgmap v4284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:52 smithi188 ceph-mon[92425]: pgmap v4284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:54 smithi161 ceph-mon[91579]: pgmap v4285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:54 smithi188 ceph-mon[92425]: pgmap v4285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:01:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:01:57.258 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:56 smithi161 ceph-mon[91579]: pgmap v4286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:57.259 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:57.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:01:57.200+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:01:57.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:01:57.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:01:57.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:01:57.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:01:57.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:01:57.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:01:57.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:01:57.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:01:57.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:01:57.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:01:57.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:01:57.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:01:57.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:01:57.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:01:57.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:01:57.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:01:57.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:01:57.266 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:01:57.266 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:01:57.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:01:57.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:01:57.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:01:57.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:01:57.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:01:57.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:01:57.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:01:57.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:01:57.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:01:57.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:01:57.270 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:01:57.270 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:01:57.270 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:01:57.271 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:01:57.271 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:01:57.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:01:57.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:01:57.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:01:57.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:01:57.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:01:57.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:01:57.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:01:57.274 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:01:57.274 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:01:57.275 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:01:57.275 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:01:57.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=215] Local address: 172.21.15.161, port 53338 2022-07-03T13:01:57.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=215] Peer address: 172.21.15.101, port 22 2022-07-03T13:01:57.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=215] Connection lost 2022-07-03T13:01:57.277 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=215] Aborting connection 2022-07-03T13:01:57.277 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:01:57.278 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:56 smithi188 ceph-mon[92425]: pgmap v4286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:57.278 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:01:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:01:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:01:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:01:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:01:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:01:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:01:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:01:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:01:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:01:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:01:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:01:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:01:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:01:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:01:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:01:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:01:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:01:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:01:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:01:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:01:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:01:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:01:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:01:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:01:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:01:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:01:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:01:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:01:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:01:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:01:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:01:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:01:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:01:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:01:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:01:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:01:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:01:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:01:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:01:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:01:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:01:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:01:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:01:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:01:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:01:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:01:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:01:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:01:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:01:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:01:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:01:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:01:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:01:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:01:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:01:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:01:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:01:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:01:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:01:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:01:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:01:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=215] Local address: 172.21.15.161, port 53338 2022-07-03T13:01:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=215] Peer address: 172.21.15.101, port 22 2022-07-03T13:01:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=215] Connection lost 2022-07-03T13:01:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=215] Aborting connection 2022-07-03T13:01:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:58 smithi161 ceph-mon[91579]: pgmap v4287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:01:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:01:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:01:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:01:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:01:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:01:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:01:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:01:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:01:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:01:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:01:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:01:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:01:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:01:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:01:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:01:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:01:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:01:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:01:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:01:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:01:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:01:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:01:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:01:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:01:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:01:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:01:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:01:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:01:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:01:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:01:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:01:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:01:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:01:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:01:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:01:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:01:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:01:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:01:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:01:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:01:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:01:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:01:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:01:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:01:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:01:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:01:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:01:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=215] Local address: 172.21.15.161, port 53338 2022-07-03T13:01:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=215] Peer address: 172.21.15.101, port 22 2022-07-03T13:01:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=215] Connection lost 2022-07-03T13:01:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=215] Aborting connection 2022-07-03T13:01:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:58 smithi188 ceph-mon[92425]: pgmap v4287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:00 smithi161 ceph-mon[91579]: pgmap v4288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:00 smithi188 ceph-mon[92425]: pgmap v4288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:02 smithi161 ceph-mon[91579]: pgmap v4289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:03 smithi188 ceph-mon[92425]: pgmap v4289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:05 smithi161 ceph-mon[91579]: pgmap v4290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:05 smithi188 ceph-mon[92425]: pgmap v4290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:07 smithi161 ceph-mon[91579]: pgmap v4291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:07 smithi188 ceph-mon[92425]: pgmap v4291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:09 smithi161 ceph-mon[91579]: pgmap v4292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:09 smithi188 ceph-mon[92425]: pgmap v4292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:11 smithi161 ceph-mon[91579]: pgmap v4293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:11 smithi188 ceph-mon[92425]: pgmap v4293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:13 smithi161 ceph-mon[91579]: pgmap v4294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:13 smithi188 ceph-mon[92425]: pgmap v4294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:15 smithi161 ceph-mon[91579]: pgmap v4295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:15 smithi188 ceph-mon[92425]: pgmap v4295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:17 smithi161 ceph-mon[91579]: pgmap v4296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:17 smithi188 ceph-mon[92425]: pgmap v4296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:19 smithi161 ceph-mon[91579]: pgmap v4297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:19 smithi188 ceph-mon[92425]: pgmap v4297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:21 smithi161 ceph-mon[91579]: pgmap v4298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:21 smithi188 ceph-mon[92425]: pgmap v4298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:23 smithi161 ceph-mon[91579]: pgmap v4299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:23 smithi188 ceph-mon[92425]: pgmap v4299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:25 smithi161 ceph-mon[91579]: pgmap v4300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:25 smithi188 ceph-mon[92425]: pgmap v4300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:27 smithi161 ceph-mon[91579]: pgmap v4301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:27 smithi188 ceph-mon[92425]: pgmap v4301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:29 smithi161 ceph-mon[91579]: pgmap v4302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:29 smithi188 ceph-mon[92425]: pgmap v4302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:31 smithi161 ceph-mon[91579]: pgmap v4303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:31 smithi188 ceph-mon[92425]: pgmap v4303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:33 smithi161 ceph-mon[91579]: pgmap v4304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:33 smithi188 ceph-mon[92425]: pgmap v4304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:34 smithi161 ceph-mon[91579]: pgmap v4305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:34 smithi188 ceph-mon[92425]: pgmap v4305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:36 smithi161 ceph-mon[91579]: pgmap v4306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:36 smithi188 ceph-mon[92425]: pgmap v4306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:38 smithi161 ceph-mon[91579]: pgmap v4307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:38 smithi188 ceph-mon[92425]: pgmap v4307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:40 smithi161 ceph-mon[91579]: pgmap v4308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:40 smithi188 ceph-mon[92425]: pgmap v4308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:42 smithi161 ceph-mon[91579]: pgmap v4309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:42 smithi188 ceph-mon[92425]: pgmap v4309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:44 smithi161 ceph-mon[91579]: pgmap v4310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:44 smithi188 ceph-mon[92425]: pgmap v4310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:46 smithi161 ceph-mon[91579]: pgmap v4311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:46 smithi188 ceph-mon[92425]: pgmap v4311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:48 smithi161 ceph-mon[91579]: pgmap v4312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:48 smithi188 ceph-mon[92425]: pgmap v4312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:50 smithi161 ceph-mon[91579]: pgmap v4313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:50 smithi188 ceph-mon[92425]: pgmap v4313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:52 smithi161 ceph-mon[91579]: pgmap v4314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:52 smithi188 ceph-mon[92425]: pgmap v4314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:54 smithi161 ceph-mon[91579]: pgmap v4315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:54 smithi188 ceph-mon[92425]: pgmap v4315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:02:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:56 smithi161 ceph-mon[91579]: pgmap v4316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:56 smithi188 ceph-mon[92425]: pgmap v4316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:02:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:02:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:02:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:02:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:02:57.699+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:02:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:02:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:02:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:02:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:02:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:02:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:02:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:02:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:02:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:02:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:02:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:02:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:02:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:02:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:02:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:02:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:02:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:02:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:02:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:02:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:02:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:02:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:02:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:02:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:02:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:02:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:02:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:02:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:02:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:02:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:02:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:02:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:02:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:02:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:02:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:02:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:02:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:02:57.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:02:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:02:57.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:02:57.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:02:57.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:02:57.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:02:57.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:02:57.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:02:57.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=217] Local address: 172.21.15.161, port 53366 2022-07-03T13:02:57.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=217] Peer address: 172.21.15.101, port 22 2022-07-03T13:02:57.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=217] Connection lost 2022-07-03T13:02:57.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=217] Aborting connection 2022-07-03T13:02:57.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:02:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:02:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:02:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:02:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:02:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:02:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:02:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: pgmap v4317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:02:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:02:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:02:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:02:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:02:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:02:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:02:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:02:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:02:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:02:58.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:02:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:02:58.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:02:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:02:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:02:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:02:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:02:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:02:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:02:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:02:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:02:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:02:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:02:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:02:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:02:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:02:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:02:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:02:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:02:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:02:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:02:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:02:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:02:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:02:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:02:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:02:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:02:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:02:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:02:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:02:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:02:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:02:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:02:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:02:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:02:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:02:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:02:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:02:58.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:02:58.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:02:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=217] Local address: 172.21.15.161, port 53366 2022-07-03T13:02:58.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=217] Peer address: 172.21.15.101, port 22 2022-07-03T13:02:58.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=217] Connection lost 2022-07-03T13:02:58.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=217] Aborting connection 2022-07-03T13:02:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: pgmap v4317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:02:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:02:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:02:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:02:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:02:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:02:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:02:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:02:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:02:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:02:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:02:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:02:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:02:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:02:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:02:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:02:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:02:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:02:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:02:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:02:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:02:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:02:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:02:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:02:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:02:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:02:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:02:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:02:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:02:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:02:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:02:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:02:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:02:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:02:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:02:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:02:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:02:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:02:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:02:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:02:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:02:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:02:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:02:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:02:58.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:02:58.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:02:58.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:02:58.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:02:58.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:02:58.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:02:58.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:02:58.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=217] Local address: 172.21.15.161, port 53366 2022-07-03T13:02:58.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=217] Peer address: 172.21.15.101, port 22 2022-07-03T13:02:58.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=217] Connection lost 2022-07-03T13:02:58.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=217] Aborting connection 2022-07-03T13:02:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:02:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:00 smithi161 ceph-mon[91579]: pgmap v4318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:00 smithi188 ceph-mon[92425]: pgmap v4318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:02 smithi161 ceph-mon[91579]: pgmap v4319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:02 smithi188 ceph-mon[92425]: pgmap v4319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:04 smithi161 ceph-mon[91579]: pgmap v4320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:04 smithi188 ceph-mon[92425]: pgmap v4320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:06 smithi161 ceph-mon[91579]: pgmap v4321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:06 smithi188 ceph-mon[92425]: pgmap v4321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:08 smithi161 ceph-mon[91579]: pgmap v4322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:08 smithi188 ceph-mon[92425]: pgmap v4322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:10 smithi161 ceph-mon[91579]: pgmap v4323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:10 smithi188 ceph-mon[92425]: pgmap v4323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:12 smithi161 ceph-mon[91579]: pgmap v4324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:12 smithi188 ceph-mon[92425]: pgmap v4324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:14 smithi161 ceph-mon[91579]: pgmap v4325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:14 smithi188 ceph-mon[92425]: pgmap v4325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:16 smithi161 ceph-mon[91579]: pgmap v4326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:16 smithi188 ceph-mon[92425]: pgmap v4326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:18 smithi161 ceph-mon[91579]: pgmap v4327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:18 smithi188 ceph-mon[92425]: pgmap v4327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:20 smithi161 ceph-mon[91579]: pgmap v4328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:20 smithi188 ceph-mon[92425]: pgmap v4328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:22 smithi161 ceph-mon[91579]: pgmap v4329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:22 smithi188 ceph-mon[92425]: pgmap v4329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:24 smithi161 ceph-mon[91579]: pgmap v4330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:24 smithi188 ceph-mon[92425]: pgmap v4330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:26 smithi188 ceph-mon[92425]: pgmap v4331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:26 smithi161 ceph-mon[91579]: pgmap v4331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:28 smithi188 ceph-mon[92425]: pgmap v4332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:28 smithi161 ceph-mon[91579]: pgmap v4332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:30 smithi188 ceph-mon[92425]: pgmap v4333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:30 smithi161 ceph-mon[91579]: pgmap v4333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:32 smithi188 ceph-mon[92425]: pgmap v4334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:33.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:32 smithi161 ceph-mon[91579]: pgmap v4334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:34 smithi188 ceph-mon[92425]: pgmap v4335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:34 smithi161 ceph-mon[91579]: pgmap v4335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:36 smithi188 ceph-mon[92425]: pgmap v4336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:36 smithi161 ceph-mon[91579]: pgmap v4336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:38 smithi188 ceph-mon[92425]: pgmap v4337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:38 smithi161 ceph-mon[91579]: pgmap v4337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:40 smithi161 ceph-mon[91579]: pgmap v4338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:40 smithi188 ceph-mon[92425]: pgmap v4338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:42 smithi161 ceph-mon[91579]: pgmap v4339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:42 smithi188 ceph-mon[92425]: pgmap v4339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:44 smithi161 ceph-mon[91579]: pgmap v4340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:44 smithi188 ceph-mon[92425]: pgmap v4340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:46 smithi161 ceph-mon[91579]: pgmap v4341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:46 smithi188 ceph-mon[92425]: pgmap v4341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:48 smithi161 ceph-mon[91579]: pgmap v4342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:48 smithi188 ceph-mon[92425]: pgmap v4342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:50 smithi161 ceph-mon[91579]: pgmap v4343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:50 smithi188 ceph-mon[92425]: pgmap v4343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:52 smithi161 ceph-mon[91579]: pgmap v4344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:52 smithi188 ceph-mon[92425]: pgmap v4344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:54 smithi161 ceph-mon[91579]: pgmap v4345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:54 smithi188 ceph-mon[92425]: pgmap v4345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:03:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:03:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:56 smithi161 ceph-mon[91579]: pgmap v4346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:56 smithi188 ceph-mon[92425]: pgmap v4346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:03:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:03:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:03:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:03:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:03:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:03:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:03:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:03:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:03:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:58 smithi161 ceph-mon[91579]: pgmap v4347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:03:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:03:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:03:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:58 smithi188 ceph-mon[92425]: pgmap v4347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:03:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:03:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:03:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:04:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:04:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:04:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:00 smithi161 ceph-mon[91579]: pgmap v4348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:00 smithi188 ceph-mon[92425]: pgmap v4348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:02 smithi161 ceph-mon[91579]: pgmap v4349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:02 smithi188 ceph-mon[92425]: pgmap v4349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:04 smithi161 ceph-mon[91579]: pgmap v4350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:04 smithi188 ceph-mon[92425]: pgmap v4350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:06 smithi161 ceph-mon[91579]: pgmap v4351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:06 smithi188 ceph-mon[92425]: pgmap v4351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:08 smithi161 ceph-mon[91579]: pgmap v4352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:08 smithi188 ceph-mon[92425]: pgmap v4352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:10 smithi161 ceph-mon[91579]: pgmap v4353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:10 smithi188 ceph-mon[92425]: pgmap v4353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:12 smithi161 ceph-mon[91579]: pgmap v4354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:12 smithi188 ceph-mon[92425]: pgmap v4354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:14 smithi161 ceph-mon[91579]: pgmap v4355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:14 smithi188 ceph-mon[92425]: pgmap v4355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:16 smithi161 ceph-mon[91579]: pgmap v4356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:16 smithi188 ceph-mon[92425]: pgmap v4356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:18 smithi161 ceph-mon[91579]: pgmap v4357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:18 smithi188 ceph-mon[92425]: pgmap v4357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:20 smithi161 ceph-mon[91579]: pgmap v4358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:20 smithi188 ceph-mon[92425]: pgmap v4358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:22 smithi161 ceph-mon[91579]: pgmap v4359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:22 smithi188 ceph-mon[92425]: pgmap v4359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:24 smithi161 ceph-mon[91579]: pgmap v4360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:24 smithi188 ceph-mon[92425]: pgmap v4360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:26 smithi161 ceph-mon[91579]: pgmap v4361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:26 smithi188 ceph-mon[92425]: pgmap v4361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:04:28.203+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:04:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:04:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:04:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:04:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:04:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:04:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:04:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:04:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:04:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:04:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:04:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:04:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:04:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:04:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:04:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:04:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:04:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:04:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:04:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:04:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:04:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:04:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:04:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:04:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:04:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:04:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:04:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:04:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:04:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:04:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:04:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:04:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:04:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:04:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:04:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:04:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:04:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:04:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:04:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:04:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:04:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:04:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:04:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:04:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:04:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:04:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=219] Local address: 172.21.15.161, port 53394 2022-07-03T13:04:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=219] Peer address: 172.21.15.101, port 22 2022-07-03T13:04:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=628] Received exit status 0 2022-07-03T13:04:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=628] Received channel close 2022-07-03T13:04:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=628] Channel closed 2022-07-03T13:04:28.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=629] Requesting new SSH session 2022-07-03T13:04:28.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=628] Received exit status 0 2022-07-03T13:04:28.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=628] Received channel close 2022-07-03T13:04:28.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=628] Channel closed 2022-07-03T13:04:28.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=629] Command: which python3 2022-07-03T13:04:28.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=629] Requesting new SSH session 2022-07-03T13:04:28.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=629] Command: which python3 2022-07-03T13:04:28.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=629] Received exit status 0 2022-07-03T13:04:28.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=629] Received channel close 2022-07-03T13:04:28.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=629] Channel closed 2022-07-03T13:04:28.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=630] Requesting new SSH session 2022-07-03T13:04:28.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=629] Received exit status 0 2022-07-03T13:04:28.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=629] Received channel close 2022-07-03T13:04:28.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=629] Channel closed 2022-07-03T13:04:28.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=630] Command: sudo true 2022-07-03T13:04:28.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=630] Requesting new SSH session 2022-07-03T13:04:28.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=630] Command: sudo true 2022-07-03T13:04:28.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=630] Received exit status 0 2022-07-03T13:04:28.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=630] Received channel close 2022-07-03T13:04:28.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=630] Channel closed 2022-07-03T13:04:28.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=631] Requesting new SSH session 2022-07-03T13:04:28.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:28.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=630] Received exit status 0 2022-07-03T13:04:28.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=630] Received channel close 2022-07-03T13:04:28.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=630] Channel closed 2022-07-03T13:04:28.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=631] Requesting new SSH session 2022-07-03T13:04:28.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:28.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=631] Received exit status 0 2022-07-03T13:04:28.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=631] Received channel close 2022-07-03T13:04:28.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=631] Channel closed 2022-07-03T13:04:28.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=631] Received exit status 0 2022-07-03T13:04:28.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=631] Received channel close 2022-07-03T13:04:28.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=631] Channel closed 2022-07-03T13:04:28.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=219] Login timeout expired 2022-07-03T13:04:28.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=219] Aborting connection 2022-07-03T13:04:28.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:04:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:28 smithi161 ceph-mon[91579]: pgmap v4362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:28 smithi188 ceph-mon[92425]: pgmap v4362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:04:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:04:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:04:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:04:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:04:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:04:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:04:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:04:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:04:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:04:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:04:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:04:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:04:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:04:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:04:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:04:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:04:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:04:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:04:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:04:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:04:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:04:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:04:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:04:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:04:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:04:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:04:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:04:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:04:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:04:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:04:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:04:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:04:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:04:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:04:30.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:04:30.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:04:30.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:04:30.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:04:30.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:04:30.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:04:30.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:04:30.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:04:30.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:04:30.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:04:30.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:04:30.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=219] Local address: 172.21.15.161, port 53394 2022-07-03T13:04:30.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=219] Peer address: 172.21.15.101, port 22 2022-07-03T13:04:30.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=628] Received exit status 0 2022-07-03T13:04:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=628] Received channel close 2022-07-03T13:04:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=628] Channel closed 2022-07-03T13:04:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=629] Requesting new SSH session 2022-07-03T13:04:30.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=628] Received exit status 0 2022-07-03T13:04:30.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=628] Received channel close 2022-07-03T13:04:30.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=628] Channel closed 2022-07-03T13:04:30.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=629] Command: which python3 2022-07-03T13:04:30.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=629] Requesting new SSH session 2022-07-03T13:04:30.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=629] Command: which python3 2022-07-03T13:04:30.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=629] Received exit status 0 2022-07-03T13:04:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=629] Received channel close 2022-07-03T13:04:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=629] Channel closed 2022-07-03T13:04:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=630] Requesting new SSH session 2022-07-03T13:04:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=629] Received exit status 0 2022-07-03T13:04:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=629] Received channel close 2022-07-03T13:04:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=629] Channel closed 2022-07-03T13:04:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=630] Command: sudo true 2022-07-03T13:04:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=630] Requesting new SSH session 2022-07-03T13:04:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=630] Command: sudo true 2022-07-03T13:04:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=630] Received exit status 0 2022-07-03T13:04:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=630] Received channel close 2022-07-03T13:04:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=630] Channel closed 2022-07-03T13:04:30.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=631] Requesting new SSH session 2022-07-03T13:04:30.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:30.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=630] Received exit status 0 2022-07-03T13:04:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=630] Received channel close 2022-07-03T13:04:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=630] Channel closed 2022-07-03T13:04:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=631] Requesting new SSH session 2022-07-03T13:04:30.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:30.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=631] Received exit status 0 2022-07-03T13:04:30.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=631] Received channel close 2022-07-03T13:04:30.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=631] Channel closed 2022-07-03T13:04:30.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=631] Received exit status 0 2022-07-03T13:04:30.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=631] Received channel close 2022-07-03T13:04:30.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=631] Channel closed 2022-07-03T13:04:30.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=219] Login timeout expired 2022-07-03T13:04:30.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=219] Aborting connection 2022-07-03T13:04:30.175 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:04:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:04:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:04:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:04:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:04:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:04:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:04:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:04:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:04:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:04:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:04:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:04:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:04:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:04:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:04:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:04:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:04:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:04:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:04:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:04:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:04:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:04:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:04:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:04:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:04:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:04:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:04:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:04:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:04:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:04:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:04:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:04:30.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:04:30.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:04:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:04:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:04:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:04:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:04:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:04:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:04:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:04:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:04:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:04:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:04:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:04:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:04:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=219] Local address: 172.21.15.161, port 53394 2022-07-03T13:04:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=219] Peer address: 172.21.15.101, port 22 2022-07-03T13:04:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=628] Received exit status 0 2022-07-03T13:04:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=628] Received channel close 2022-07-03T13:04:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=628] Channel closed 2022-07-03T13:04:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=629] Requesting new SSH session 2022-07-03T13:04:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=628] Received exit status 0 2022-07-03T13:04:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=628] Received channel close 2022-07-03T13:04:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=628] Channel closed 2022-07-03T13:04:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=629] Command: which python3 2022-07-03T13:04:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=629] Requesting new SSH session 2022-07-03T13:04:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=629] Command: which python3 2022-07-03T13:04:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=629] Received exit status 0 2022-07-03T13:04:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=629] Received channel close 2022-07-03T13:04:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=629] Channel closed 2022-07-03T13:04:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=630] Requesting new SSH session 2022-07-03T13:04:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=629] Received exit status 0 2022-07-03T13:04:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=629] Received channel close 2022-07-03T13:04:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=629] Channel closed 2022-07-03T13:04:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=630] Command: sudo true 2022-07-03T13:04:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=630] Requesting new SSH session 2022-07-03T13:04:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=630] Command: sudo true 2022-07-03T13:04:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=630] Received exit status 0 2022-07-03T13:04:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=630] Received channel close 2022-07-03T13:04:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=630] Channel closed 2022-07-03T13:04:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=631] Requesting new SSH session 2022-07-03T13:04:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=630] Received exit status 0 2022-07-03T13:04:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=630] Received channel close 2022-07-03T13:04:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=630] Channel closed 2022-07-03T13:04:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=631] Requesting new SSH session 2022-07-03T13:04:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=631] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:04:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=631] Received exit status 0 2022-07-03T13:04:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=631] Received channel close 2022-07-03T13:04:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=631] Channel closed 2022-07-03T13:04:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=631] Received exit status 0 2022-07-03T13:04:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=631] Received channel close 2022-07-03T13:04:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=631] Channel closed 2022-07-03T13:04:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=219] Login timeout expired 2022-07-03T13:04:30.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=219] Aborting connection 2022-07-03T13:04:30.208 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:30 smithi161 ceph-mon[91579]: pgmap v4363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:30 smithi188 ceph-mon[92425]: pgmap v4363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:32 smithi161 ceph-mon[91579]: pgmap v4364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:32 smithi188 ceph-mon[92425]: pgmap v4364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:34 smithi161 ceph-mon[91579]: pgmap v4365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:34 smithi188 ceph-mon[92425]: pgmap v4365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:36 smithi161 ceph-mon[91579]: pgmap v4366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:36 smithi188 ceph-mon[92425]: pgmap v4366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:38 smithi161 ceph-mon[91579]: pgmap v4367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:38 smithi188 ceph-mon[92425]: pgmap v4367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:40 smithi161 ceph-mon[91579]: pgmap v4368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:40 smithi188 ceph-mon[92425]: pgmap v4368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:42 smithi161 ceph-mon[91579]: pgmap v4369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:42 smithi188 ceph-mon[92425]: pgmap v4369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:44 smithi161 ceph-mon[91579]: pgmap v4370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:44 smithi188 ceph-mon[92425]: pgmap v4370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:46 smithi161 ceph-mon[91579]: pgmap v4371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:46 smithi188 ceph-mon[92425]: pgmap v4371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:48 smithi161 ceph-mon[91579]: pgmap v4372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:48 smithi188 ceph-mon[92425]: pgmap v4372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:50 smithi161 ceph-mon[91579]: pgmap v4373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:50 smithi188 ceph-mon[92425]: pgmap v4373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:52 smithi161 ceph-mon[91579]: pgmap v4374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:52 smithi188 ceph-mon[92425]: pgmap v4374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:54 smithi161 ceph-mon[91579]: pgmap v4375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:54 smithi188 ceph-mon[92425]: pgmap v4375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:04:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:04:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:56 smithi161 ceph-mon[91579]: pgmap v4376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:56 smithi188 ceph-mon[92425]: pgmap v4376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:04:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:04:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:04:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:04:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:04:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:04:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:04:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:04:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:04:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:58 smithi161 ceph-mon[91579]: pgmap v4377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:04:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:58 smithi188 ceph-mon[92425]: pgmap v4377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:00 smithi161 ceph-mon[91579]: pgmap v4378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:00 smithi188 ceph-mon[92425]: pgmap v4378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:02 smithi161 ceph-mon[91579]: pgmap v4379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:02 smithi188 ceph-mon[92425]: pgmap v4379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:04 smithi188 ceph-mon[92425]: pgmap v4380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:04 smithi161 ceph-mon[91579]: pgmap v4380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:06 smithi188 ceph-mon[92425]: pgmap v4381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:06 smithi161 ceph-mon[91579]: pgmap v4381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:08 smithi188 ceph-mon[92425]: pgmap v4382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:08 smithi161 ceph-mon[91579]: pgmap v4382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:10 smithi188 ceph-mon[92425]: pgmap v4383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:10 smithi161 ceph-mon[91579]: pgmap v4383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:12 smithi188 ceph-mon[92425]: pgmap v4384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:12 smithi161 ceph-mon[91579]: pgmap v4384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:14 smithi188 ceph-mon[92425]: pgmap v4385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:14 smithi161 ceph-mon[91579]: pgmap v4385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:16 smithi188 ceph-mon[92425]: pgmap v4386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:16 smithi161 ceph-mon[91579]: pgmap v4386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:18 smithi188 ceph-mon[92425]: pgmap v4387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:18 smithi161 ceph-mon[91579]: pgmap v4387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:20 smithi161 ceph-mon[91579]: pgmap v4388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:20 smithi188 ceph-mon[92425]: pgmap v4388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:22 smithi161 ceph-mon[91579]: pgmap v4389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:22 smithi188 ceph-mon[92425]: pgmap v4389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:24 smithi161 ceph-mon[91579]: pgmap v4390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:24 smithi188 ceph-mon[92425]: pgmap v4390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:26 smithi161 ceph-mon[91579]: pgmap v4391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:26 smithi188 ceph-mon[92425]: pgmap v4391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:28.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:05:28.242+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:05:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:05:28.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:05:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:05:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:05:28.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:05:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:05:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:05:28.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:05:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:05:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:05:28.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:05:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:05:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:05:28.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:05:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:05:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:05:28.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:05:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:05:28.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:05:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:05:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:05:28.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:05:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:05:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:05:28.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:05:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:05:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:05:28.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:05:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:05:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:05:28.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:05:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:05:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:05:28.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:05:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:05:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:05:28.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:05:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:05:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:05:28.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:05:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:05:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:05:28.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:05:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:05:28.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:05:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=221] Local address: 172.21.15.161, port 53434 2022-07-03T13:05:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=221] Peer address: 172.21.15.101, port 22 2022-07-03T13:05:28.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=221] Connection lost 2022-07-03T13:05:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=221] Aborting connection 2022-07-03T13:05:28.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:05:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-mon[91579]: pgmap v4392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:05:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:05:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:05:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:28 smithi188 ceph-mon[92425]: pgmap v4392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:05:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:05:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:05:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:05:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:05:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:05:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:05:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:05:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:05:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:05:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:05:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:05:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:05:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:05:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:05:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:05:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:05:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:05:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:05:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:05:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:05:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:05:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:05:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:05:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:05:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:05:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:05:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:05:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:05:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:05:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:05:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:05:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:05:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:05:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:05:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:05:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:05:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:05:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:05:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:05:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:05:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:05:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:05:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:05:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:05:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:05:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:05:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:05:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:05:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:05:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=221] Local address: 172.21.15.161, port 53434 2022-07-03T13:05:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=221] Peer address: 172.21.15.101, port 22 2022-07-03T13:05:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=221] Connection lost 2022-07-03T13:05:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=221] Aborting connection 2022-07-03T13:05:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:05:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:05:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:05:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:05:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:05:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:05:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:05:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:05:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:05:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:05:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:05:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:05:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:05:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:05:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:05:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:05:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:05:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:05:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:05:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:05:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:05:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:05:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:05:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:05:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:05:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:05:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:05:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:05:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:05:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:05:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:05:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:05:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:05:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:05:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:05:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:05:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:05:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:05:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:05:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:05:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:05:30.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:05:30.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:05:30.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:05:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:05:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:05:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:05:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=221] Local address: 172.21.15.161, port 53434 2022-07-03T13:05:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=221] Peer address: 172.21.15.101, port 22 2022-07-03T13:05:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=221] Connection lost 2022-07-03T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=221] Aborting connection 2022-07-03T13:05:30.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:30 smithi161 ceph-mon[91579]: pgmap v4393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:30 smithi188 ceph-mon[92425]: pgmap v4393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:32 smithi161 ceph-mon[91579]: pgmap v4394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:32 smithi188 ceph-mon[92425]: pgmap v4394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:34 smithi161 ceph-mon[91579]: pgmap v4395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:34 smithi188 ceph-mon[92425]: pgmap v4395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:36 smithi161 ceph-mon[91579]: pgmap v4396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:36 smithi188 ceph-mon[92425]: pgmap v4396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:38 smithi161 ceph-mon[91579]: pgmap v4397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:38 smithi188 ceph-mon[92425]: pgmap v4397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:40 smithi161 ceph-mon[91579]: pgmap v4398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:40 smithi188 ceph-mon[92425]: pgmap v4398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:42 smithi161 ceph-mon[91579]: pgmap v4399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:42 smithi188 ceph-mon[92425]: pgmap v4399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:44 smithi161 ceph-mon[91579]: pgmap v4400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:44 smithi188 ceph-mon[92425]: pgmap v4400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:46 smithi161 ceph-mon[91579]: pgmap v4401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:46 smithi188 ceph-mon[92425]: pgmap v4401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:48 smithi161 ceph-mon[91579]: pgmap v4402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:48 smithi188 ceph-mon[92425]: pgmap v4402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:50 smithi161 ceph-mon[91579]: pgmap v4403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:50 smithi188 ceph-mon[92425]: pgmap v4403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:53 smithi161 ceph-mon[91579]: pgmap v4404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:53 smithi188 ceph-mon[92425]: pgmap v4404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:55 smithi161 ceph-mon[91579]: pgmap v4405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:55 smithi188 ceph-mon[92425]: pgmap v4405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:05:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:05:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:57 smithi161 ceph-mon[91579]: pgmap v4406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:57 smithi188 ceph-mon[92425]: pgmap v4406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:05:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:05:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:05:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:05:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:05:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:05:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:05:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:05:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:05:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:05:59 smithi161 ceph-mon[91579]: pgmap v4407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:05:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:05:59 smithi188 ceph-mon[92425]: pgmap v4407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:01 smithi161 ceph-mon[91579]: pgmap v4408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:01 smithi188 ceph-mon[92425]: pgmap v4408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:03 smithi161 ceph-mon[91579]: pgmap v4409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:03 smithi188 ceph-mon[92425]: pgmap v4409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:05 smithi161 ceph-mon[91579]: pgmap v4410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:05 smithi188 ceph-mon[92425]: pgmap v4410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:07 smithi161 ceph-mon[91579]: pgmap v4411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:07 smithi188 ceph-mon[92425]: pgmap v4411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:09 smithi161 ceph-mon[91579]: pgmap v4412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:09 smithi188 ceph-mon[92425]: pgmap v4412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:11 smithi161 ceph-mon[91579]: pgmap v4413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:11 smithi188 ceph-mon[92425]: pgmap v4413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:13 smithi161 ceph-mon[91579]: pgmap v4414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:13 smithi188 ceph-mon[92425]: pgmap v4414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:15 smithi161 ceph-mon[91579]: pgmap v4415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:15 smithi188 ceph-mon[92425]: pgmap v4415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:17 smithi161 ceph-mon[91579]: pgmap v4416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:17 smithi188 ceph-mon[92425]: pgmap v4416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:19 smithi161 ceph-mon[91579]: pgmap v4417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:19 smithi188 ceph-mon[92425]: pgmap v4417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:21 smithi161 ceph-mon[91579]: pgmap v4418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:21 smithi188 ceph-mon[92425]: pgmap v4418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:23 smithi161 ceph-mon[91579]: pgmap v4419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:23 smithi188 ceph-mon[92425]: pgmap v4419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:25 smithi161 ceph-mon[91579]: pgmap v4420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:25 smithi188 ceph-mon[92425]: pgmap v4420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:27 smithi161 ceph-mon[91579]: pgmap v4421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:27 smithi188 ceph-mon[92425]: pgmap v4421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:29 smithi161 ceph-mon[91579]: pgmap v4422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:06:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:06:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:06:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:29 smithi188 ceph-mon[92425]: pgmap v4422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:06:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:06:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:06:30.230 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:06:30.230 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:06:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:31 smithi161 ceph-mon[91579]: pgmap v4423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:31 smithi188 ceph-mon[92425]: pgmap v4423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:06:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:06:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:06:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:06:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:33 smithi161 ceph-mon[91579]: pgmap v4424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:33 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T13:06:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:33 smithi188 ceph-mon[92425]: pgmap v4424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:33 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T13:06:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:34 smithi161 ceph-mon[91579]: pgmap v4425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:34 smithi188 ceph-mon[92425]: pgmap v4425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:36 smithi161 ceph-mon[91579]: pgmap v4426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:36 smithi188 ceph-mon[92425]: pgmap v4426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:38 smithi161 ceph-mon[91579]: pgmap v4427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:38 smithi188 ceph-mon[92425]: pgmap v4427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:40 smithi161 ceph-mon[91579]: pgmap v4428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:40 smithi188 ceph-mon[92425]: pgmap v4428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:42 smithi161 ceph-mon[91579]: pgmap v4429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:42 smithi188 ceph-mon[92425]: pgmap v4429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:44.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:06:44.322+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T13:06:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:44 smithi161 ceph-mon[91579]: pgmap v4430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:44 smithi188 ceph-mon[92425]: pgmap v4430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:06:46.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:46 smithi161 ceph-mon[91579]: pgmap v4431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:46 smithi188 ceph-mon[92425]: pgmap v4431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:48 smithi161 ceph-mon[91579]: pgmap v4432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:48 smithi188 ceph-mon[92425]: pgmap v4432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:50 smithi161 ceph-mon[91579]: pgmap v4433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:50 smithi188 ceph-mon[92425]: pgmap v4433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:52 smithi161 ceph-mon[91579]: pgmap v4434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:52 smithi188 ceph-mon[92425]: pgmap v4434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:54 smithi161 ceph-mon[91579]: pgmap v4435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:54 smithi188 ceph-mon[92425]: pgmap v4435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:06:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:56 smithi161 ceph-mon[91579]: pgmap v4436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:56 smithi188 ceph-mon[92425]: pgmap v4436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:06:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:06:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:06:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:06:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:06:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:06:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:06:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:06:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:06:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-mon[91579]: pgmap v4437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:58.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:06:58.740+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:06:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:06:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:06:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:06:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:06:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:06:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:06:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:06:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:06:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:06:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:06:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:06:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:06:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:06:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:06:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:06:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:06:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:06:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:06:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:06:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:06:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:06:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:06:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:06:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:06:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:06:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:06:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:06:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:06:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:06:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:06:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:06:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:06:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:06:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:06:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:06:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:06:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:06:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:06:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:06:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:06:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:06:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:06:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:06:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:06:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:06:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=223] Local address: 172.21.15.161, port 53462 2022-07-03T13:06:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=223] Peer address: 172.21.15.101, port 22 2022-07-03T13:06:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=636] Received exit status 0 2022-07-03T13:06:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=636] Received channel close 2022-07-03T13:06:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=636] Channel closed 2022-07-03T13:06:58.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=637] Requesting new SSH session 2022-07-03T13:06:58.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=636] Received exit status 0 2022-07-03T13:06:58.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=636] Received channel close 2022-07-03T13:06:58.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=636] Channel closed 2022-07-03T13:06:58.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=637] Command: which python3 2022-07-03T13:06:58.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=637] Requesting new SSH session 2022-07-03T13:06:58.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=637] Command: which python3 2022-07-03T13:06:58.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=637] Received exit status 0 2022-07-03T13:06:58.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=637] Received channel close 2022-07-03T13:06:58.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=637] Channel closed 2022-07-03T13:06:58.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=638] Requesting new SSH session 2022-07-03T13:06:58.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=637] Received exit status 0 2022-07-03T13:06:58.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=637] Received channel close 2022-07-03T13:06:58.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=637] Channel closed 2022-07-03T13:06:58.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=638] Command: sudo true 2022-07-03T13:06:58.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=638] Requesting new SSH session 2022-07-03T13:06:58.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=638] Command: sudo true 2022-07-03T13:06:58.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=638] Received exit status 0 2022-07-03T13:06:58.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=638] Received channel close 2022-07-03T13:06:58.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=638] Channel closed 2022-07-03T13:06:58.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=639] Requesting new SSH session 2022-07-03T13:06:58.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:58.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=638] Received exit status 0 2022-07-03T13:06:58.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=638] Received channel close 2022-07-03T13:06:58.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=638] Channel closed 2022-07-03T13:06:58.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=639] Requesting new SSH session 2022-07-03T13:06:58.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:58.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=639] Received exit status 0 2022-07-03T13:06:58.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=639] Received channel close 2022-07-03T13:06:58.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=639] Channel closed 2022-07-03T13:06:58.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=640] Requesting new SSH session 2022-07-03T13:06:58.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=640] Command: sudo true 2022-07-03T13:06:58.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=639] Received exit status 0 2022-07-03T13:06:58.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=639] Received channel close 2022-07-03T13:06:58.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=639] Channel closed 2022-07-03T13:06:58.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=640] Received exit status 0 2022-07-03T13:06:58.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=640] Received channel close 2022-07-03T13:06:58.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=640] Channel closed 2022-07-03T13:06:58.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=641] Requesting new SSH session 2022-07-03T13:06:58.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=641] Command: which python3 2022-07-03T13:06:58.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=641] Received exit status 0 2022-07-03T13:06:58.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=641] Received channel close 2022-07-03T13:06:58.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=641] Channel closed 2022-07-03T13:06:58.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=642] Requesting new SSH session 2022-07-03T13:06:58.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=642] Command: sudo true 2022-07-03T13:06:58.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=642] Received exit status 0 2022-07-03T13:06:58.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=642] Received channel close 2022-07-03T13:06:58.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=642] Channel closed 2022-07-03T13:06:58.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=643] Requesting new SSH session 2022-07-03T13:06:58.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ceph-volume --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- inventory --format=json-pretty --filter-for-batch 2022-07-03T13:06:58.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=643] Received exit status 0 2022-07-03T13:06:58.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=643] Received channel close 2022-07-03T13:06:58.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=643] Channel closed 2022-07-03T13:06:58.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=223] Login timeout expired 2022-07-03T13:06:58.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=223] Aborting connection 2022-07-03T13:06:58.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:06:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:06:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:58 smithi188 ceph-mon[92425]: pgmap v4437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:06:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:06:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:06:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:06:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:06:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:06:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:06:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:06:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:06:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:06:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:06:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:06:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:06:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:06:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:06:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:06:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:06:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:06:59.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:06:59.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:06:59.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:06:59.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:06:59.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:06:59.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:06:59.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:06:59.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:06:59.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:06:59.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:06:59.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:06:59.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:06:59.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:06:59.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:06:59.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:06:59.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:06:59.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:06:59.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:06:59.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:06:59.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:06:59.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:06:59.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:06:59.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:06:59.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:06:59.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:06:59.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:06:59.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:06:59.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:06:59.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:06:59.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:06:59.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=223] Local address: 172.21.15.161, port 53462 2022-07-03T13:06:59.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=223] Peer address: 172.21.15.101, port 22 2022-07-03T13:06:59.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=636] Received exit status 0 2022-07-03T13:06:59.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=636] Received channel close 2022-07-03T13:06:59.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=636] Channel closed 2022-07-03T13:06:59.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=637] Requesting new SSH session 2022-07-03T13:06:59.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=636] Received exit status 0 2022-07-03T13:06:59.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=636] Received channel close 2022-07-03T13:06:59.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=636] Channel closed 2022-07-03T13:06:59.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=637] Command: which python3 2022-07-03T13:06:59.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=637] Requesting new SSH session 2022-07-03T13:06:59.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=637] Command: which python3 2022-07-03T13:06:59.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=637] Received exit status 0 2022-07-03T13:06:59.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=637] Received channel close 2022-07-03T13:06:59.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=637] Channel closed 2022-07-03T13:06:59.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=638] Requesting new SSH session 2022-07-03T13:06:59.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=637] Received exit status 0 2022-07-03T13:06:59.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=637] Received channel close 2022-07-03T13:06:59.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=637] Channel closed 2022-07-03T13:06:59.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=638] Command: sudo true 2022-07-03T13:06:59.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=638] Requesting new SSH session 2022-07-03T13:06:59.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=638] Command: sudo true 2022-07-03T13:06:59.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=638] Received exit status 0 2022-07-03T13:06:59.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=638] Received channel close 2022-07-03T13:06:59.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=638] Channel closed 2022-07-03T13:06:59.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=639] Requesting new SSH session 2022-07-03T13:06:59.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:59.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=638] Received exit status 0 2022-07-03T13:06:59.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=638] Received channel close 2022-07-03T13:06:59.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=638] Channel closed 2022-07-03T13:06:59.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=639] Requesting new SSH session 2022-07-03T13:06:59.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:59.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=639] Received exit status 0 2022-07-03T13:06:59.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=639] Received channel close 2022-07-03T13:06:59.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=639] Channel closed 2022-07-03T13:06:59.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=640] Requesting new SSH session 2022-07-03T13:06:59.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=640] Command: sudo true 2022-07-03T13:06:59.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=639] Received exit status 0 2022-07-03T13:06:59.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=639] Received channel close 2022-07-03T13:06:59.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=639] Channel closed 2022-07-03T13:06:59.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=640] Received exit status 0 2022-07-03T13:06:59.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=640] Received channel close 2022-07-03T13:06:59.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=640] Channel closed 2022-07-03T13:06:59.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=641] Requesting new SSH session 2022-07-03T13:06:59.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=641] Command: which python3 2022-07-03T13:06:59.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=641] Received exit status 0 2022-07-03T13:06:59.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=641] Received channel close 2022-07-03T13:06:59.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=641] Channel closed 2022-07-03T13:06:59.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=642] Requesting new SSH session 2022-07-03T13:06:59.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=642] Command: sudo true 2022-07-03T13:06:59.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=642] Received exit status 0 2022-07-03T13:06:59.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=642] Received channel close 2022-07-03T13:06:59.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=642] Channel closed 2022-07-03T13:06:59.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=643] Requesting new SSH session 2022-07-03T13:06:59.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ceph-volume --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- inventory --format=json-pretty --filter-for-batch 2022-07-03T13:06:59.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=643] Received exit status 0 2022-07-03T13:06:59.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=643] Received channel close 2022-07-03T13:06:59.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=643] Channel closed 2022-07-03T13:06:59.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=223] Login timeout expired 2022-07-03T13:06:59.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=223] Aborting connection 2022-07-03T13:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:06:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:06:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:06:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:06:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:06:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:06:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:06:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:06:59.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:06:59.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:06:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:06:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:06:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:06:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:06:59.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:06:59.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:06:59.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:06:59.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:06:59.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:06:59.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:06:59.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:06:59.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:06:59.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:06:59.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:06:59.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:06:59.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:06:59.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:06:59.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:06:59.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:06:59.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:06:59.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:06:59.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:06:59.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:06:59.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:06:59.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:06:59.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:06:59.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:06:59.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:06:59.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:06:59.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:06:59.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:06:59.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:06:59.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:06:59.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=223] Local address: 172.21.15.161, port 53462 2022-07-03T13:06:59.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=223] Peer address: 172.21.15.101, port 22 2022-07-03T13:06:59.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=636] Received exit status 0 2022-07-03T13:06:59.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=636] Received channel close 2022-07-03T13:06:59.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=636] Channel closed 2022-07-03T13:06:59.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=637] Requesting new SSH session 2022-07-03T13:06:59.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=636] Received exit status 0 2022-07-03T13:06:59.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=636] Received channel close 2022-07-03T13:06:59.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=636] Channel closed 2022-07-03T13:06:59.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=637] Command: which python3 2022-07-03T13:06:59.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=637] Requesting new SSH session 2022-07-03T13:06:59.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=637] Command: which python3 2022-07-03T13:06:59.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=637] Received exit status 0 2022-07-03T13:06:59.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=637] Received channel close 2022-07-03T13:06:59.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=637] Channel closed 2022-07-03T13:06:59.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=638] Requesting new SSH session 2022-07-03T13:06:59.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=637] Received exit status 0 2022-07-03T13:06:59.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=637] Received channel close 2022-07-03T13:06:59.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=637] Channel closed 2022-07-03T13:06:59.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=638] Command: sudo true 2022-07-03T13:06:59.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=638] Requesting new SSH session 2022-07-03T13:06:59.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=638] Command: sudo true 2022-07-03T13:06:59.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=638] Received exit status 0 2022-07-03T13:06:59.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=638] Received channel close 2022-07-03T13:06:59.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=638] Channel closed 2022-07-03T13:06:59.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=639] Requesting new SSH session 2022-07-03T13:06:59.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:59.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=638] Received exit status 0 2022-07-03T13:06:59.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=638] Received channel close 2022-07-03T13:06:59.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=638] Channel closed 2022-07-03T13:06:59.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=639] Requesting new SSH session 2022-07-03T13:06:59.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=639] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:06:59.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=639] Received exit status 0 2022-07-03T13:06:59.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=639] Received channel close 2022-07-03T13:06:59.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=639] Channel closed 2022-07-03T13:06:59.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=640] Requesting new SSH session 2022-07-03T13:06:59.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=640] Command: sudo true 2022-07-03T13:06:59.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=639] Received exit status 0 2022-07-03T13:06:59.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=639] Received channel close 2022-07-03T13:06:59.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=639] Channel closed 2022-07-03T13:06:59.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=640] Received exit status 0 2022-07-03T13:06:59.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=640] Received channel close 2022-07-03T13:06:59.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=640] Channel closed 2022-07-03T13:06:59.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=641] Requesting new SSH session 2022-07-03T13:06:59.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=641] Command: which python3 2022-07-03T13:06:59.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=641] Received exit status 0 2022-07-03T13:06:59.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=641] Received channel close 2022-07-03T13:06:59.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=641] Channel closed 2022-07-03T13:06:59.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=642] Requesting new SSH session 2022-07-03T13:06:59.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=642] Command: sudo true 2022-07-03T13:06:59.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=642] Received exit status 0 2022-07-03T13:06:59.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=642] Received channel close 2022-07-03T13:06:59.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=642] Channel closed 2022-07-03T13:06:59.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=643] Requesting new SSH session 2022-07-03T13:06:59.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d ceph-volume --fsid 23a851ae-fab9-11ec-842c-001a4aab830c -- inventory --format=json-pretty --filter-for-batch 2022-07-03T13:06:59.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=643] Received exit status 0 2022-07-03T13:06:59.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=643] Received channel close 2022-07-03T13:06:59.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=643] Channel closed 2022-07-03T13:06:59.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=223] Login timeout expired 2022-07-03T13:06:59.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=223] Aborting connection 2022-07-03T13:07:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:00 smithi161 ceph-mon[91579]: pgmap v4438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:00 smithi188 ceph-mon[92425]: pgmap v4438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:02 smithi161 ceph-mon[91579]: pgmap v4439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:02 smithi188 ceph-mon[92425]: pgmap v4439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:04 smithi161 ceph-mon[91579]: pgmap v4440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:04 smithi188 ceph-mon[92425]: pgmap v4440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:06 smithi161 ceph-mon[91579]: pgmap v4441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:06 smithi188 ceph-mon[92425]: pgmap v4441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:08 smithi188 ceph-mon[92425]: pgmap v4442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:08 smithi161 ceph-mon[91579]: pgmap v4442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:10 smithi188 ceph-mon[92425]: pgmap v4443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:10 smithi161 ceph-mon[91579]: pgmap v4443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:12 smithi188 ceph-mon[92425]: pgmap v4444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:12 smithi161 ceph-mon[91579]: pgmap v4444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:14 smithi188 ceph-mon[92425]: pgmap v4445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:14 smithi161 ceph-mon[91579]: pgmap v4445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:16 smithi188 ceph-mon[92425]: pgmap v4446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:16 smithi161 ceph-mon[91579]: pgmap v4446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:18 smithi188 ceph-mon[92425]: pgmap v4447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:18 smithi161 ceph-mon[91579]: pgmap v4447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:20 smithi188 ceph-mon[92425]: pgmap v4448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:20 smithi161 ceph-mon[91579]: pgmap v4448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:22 smithi188 ceph-mon[92425]: pgmap v4449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:22 smithi161 ceph-mon[91579]: pgmap v4449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:24 smithi188 ceph-mon[92425]: pgmap v4450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:24 smithi161 ceph-mon[91579]: pgmap v4450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:26 smithi188 ceph-mon[92425]: pgmap v4451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:26 smithi161 ceph-mon[91579]: pgmap v4451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:28 smithi161 ceph-mon[91579]: pgmap v4452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:28 smithi188 ceph-mon[92425]: pgmap v4452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:30 smithi161 ceph-mon[91579]: pgmap v4453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:30 smithi188 ceph-mon[92425]: pgmap v4453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:32 smithi161 ceph-mon[91579]: pgmap v4454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:32 smithi188 ceph-mon[92425]: pgmap v4454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:34 smithi161 ceph-mon[91579]: pgmap v4455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:34 smithi188 ceph-mon[92425]: pgmap v4455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:36 smithi161 ceph-mon[91579]: pgmap v4456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:36 smithi188 ceph-mon[92425]: pgmap v4456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:38 smithi161 ceph-mon[91579]: pgmap v4457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:38 smithi188 ceph-mon[92425]: pgmap v4457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:40 smithi161 ceph-mon[91579]: pgmap v4458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:40 smithi188 ceph-mon[92425]: pgmap v4458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:42 smithi161 ceph-mon[91579]: pgmap v4459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:42 smithi188 ceph-mon[92425]: pgmap v4459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:44 smithi161 ceph-mon[91579]: pgmap v4460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:44 smithi188 ceph-mon[92425]: pgmap v4460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:46 smithi161 ceph-mon[91579]: pgmap v4461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:46 smithi188 ceph-mon[92425]: pgmap v4461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:48 smithi161 ceph-mon[91579]: pgmap v4462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:48 smithi188 ceph-mon[92425]: pgmap v4462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:50 smithi161 ceph-mon[91579]: pgmap v4463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:50 smithi188 ceph-mon[92425]: pgmap v4463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:52 smithi161 ceph-mon[91579]: pgmap v4464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:52 smithi188 ceph-mon[92425]: pgmap v4464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:54 smithi161 ceph-mon[91579]: pgmap v4465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:54 smithi188 ceph-mon[92425]: pgmap v4465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:07:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:56 smithi161 ceph-mon[91579]: pgmap v4466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:56 smithi188 ceph-mon[92425]: pgmap v4466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:07:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:07:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:07:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:07:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:07:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:07:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:07:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:07:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:07:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:58 smithi161 ceph-mon[91579]: pgmap v4467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:07:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:58 smithi188 ceph-mon[92425]: pgmap v4467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:08:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:08:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:08:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:08:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:08:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:08:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:08:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:08:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:08:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:00 smithi161 ceph-mon[91579]: pgmap v4468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:00 smithi188 ceph-mon[92425]: pgmap v4468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:02 smithi161 ceph-mon[91579]: pgmap v4469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:02 smithi188 ceph-mon[92425]: pgmap v4469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:04 smithi161 ceph-mon[91579]: pgmap v4470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:04 smithi188 ceph-mon[92425]: pgmap v4470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:06 smithi161 ceph-mon[91579]: pgmap v4471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:06 smithi188 ceph-mon[92425]: pgmap v4471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:08 smithi161 ceph-mon[91579]: pgmap v4472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:08 smithi188 ceph-mon[92425]: pgmap v4472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:10 smithi161 ceph-mon[91579]: pgmap v4473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:10 smithi188 ceph-mon[92425]: pgmap v4473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:12 smithi161 ceph-mon[91579]: pgmap v4474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:12 smithi188 ceph-mon[92425]: pgmap v4474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:14 smithi161 ceph-mon[91579]: pgmap v4475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:14 smithi188 ceph-mon[92425]: pgmap v4475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:16 smithi161 ceph-mon[91579]: pgmap v4476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:16 smithi188 ceph-mon[92425]: pgmap v4476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:18 smithi161 ceph-mon[91579]: pgmap v4477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:18 smithi188 ceph-mon[92425]: pgmap v4477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:20 smithi161 ceph-mon[91579]: pgmap v4478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:20 smithi188 ceph-mon[92425]: pgmap v4478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:22 smithi161 ceph-mon[91579]: pgmap v4479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:22 smithi188 ceph-mon[92425]: pgmap v4479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:24 smithi161 ceph-mon[91579]: pgmap v4480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:24 smithi188 ceph-mon[92425]: pgmap v4480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:26 smithi161 ceph-mon[91579]: pgmap v4481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:26 smithi188 ceph-mon[92425]: pgmap v4481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-mon[91579]: pgmap v4482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:08:28.779+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:08:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:08:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:08:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:08:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:08:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:08:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:08:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:08:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:08:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:08:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:08:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:08:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:08:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:08:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:08:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:08:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:08:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:08:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:08:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:08:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:08:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:08:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:08:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:08:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:08:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:08:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:08:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:08:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:08:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:08:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:08:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:08:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:08:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:08:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:08:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:08:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:08:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:08:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:08:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:08:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:08:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:08:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:08:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:08:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:08:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:08:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=225] Local address: 172.21.15.161, port 53474 2022-07-03T13:08:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=225] Peer address: 172.21.15.101, port 22 2022-07-03T13:08:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=640] Received exit status 0 2022-07-03T13:08:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=640] Received channel close 2022-07-03T13:08:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=640] Channel closed 2022-07-03T13:08:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=641] Requesting new SSH session 2022-07-03T13:08:29.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=641] Command: which python3 2022-07-03T13:08:29.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=644] Received exit status 0 2022-07-03T13:08:29.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=644] Received channel close 2022-07-03T13:08:29.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=644] Channel closed 2022-07-03T13:08:29.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=645] Requesting new SSH session 2022-07-03T13:08:29.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=645] Command: which python3 2022-07-03T13:08:29.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=641] Received exit status 0 2022-07-03T13:08:29.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=641] Received channel close 2022-07-03T13:08:29.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=641] Channel closed 2022-07-03T13:08:29.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=642] Requesting new SSH session 2022-07-03T13:08:29.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=642] Command: sudo true 2022-07-03T13:08:29.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=645] Received exit status 0 2022-07-03T13:08:29.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=645] Received channel close 2022-07-03T13:08:29.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=645] Channel closed 2022-07-03T13:08:29.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=646] Requesting new SSH session 2022-07-03T13:08:29.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=646] Command: sudo true 2022-07-03T13:08:29.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=642] Received exit status 0 2022-07-03T13:08:29.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=642] Received channel close 2022-07-03T13:08:29.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=642] Channel closed 2022-07-03T13:08:29.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=643] Requesting new SSH session 2022-07-03T13:08:29.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:29.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=646] Received exit status 0 2022-07-03T13:08:29.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=646] Received channel close 2022-07-03T13:08:29.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=646] Channel closed 2022-07-03T13:08:29.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=647] Requesting new SSH session 2022-07-03T13:08:29.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:29.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=643] Received exit status 0 2022-07-03T13:08:29.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=643] Received channel close 2022-07-03T13:08:29.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=643] Channel closed 2022-07-03T13:08:29.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=647] Received exit status 0 2022-07-03T13:08:29.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=647] Received channel close 2022-07-03T13:08:29.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=647] Channel closed 2022-07-03T13:08:29.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=644] Requesting new SSH session 2022-07-03T13:08:29.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=644] Command: sudo true 2022-07-03T13:08:29.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=648] Requesting new SSH session 2022-07-03T13:08:29.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=648] Command: sudo true 2022-07-03T13:08:29.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=644] Received exit status 0 2022-07-03T13:08:29.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=644] Received channel close 2022-07-03T13:08:29.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=644] Channel closed 2022-07-03T13:08:29.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=645] Requesting new SSH session 2022-07-03T13:08:29.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=645] Command: which python3 2022-07-03T13:08:29.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=645] Received exit status 0 2022-07-03T13:08:29.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=645] Received channel close 2022-07-03T13:08:29.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=645] Channel closed 2022-07-03T13:08:29.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=648] Received exit status 0 2022-07-03T13:08:29.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=648] Received channel close 2022-07-03T13:08:29.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=646] Requesting new SSH session 2022-07-03T13:08:29.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=648] Channel closed 2022-07-03T13:08:29.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=646] Command: sudo true 2022-07-03T13:08:29.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=649] Requesting new SSH session 2022-07-03T13:08:29.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=649] Command: which python3 2022-07-03T13:08:29.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=649] Received exit status 0 2022-07-03T13:08:29.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=649] Received channel close 2022-07-03T13:08:29.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=649] Channel closed 2022-07-03T13:08:29.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=650] Requesting new SSH session 2022-07-03T13:08:29.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=650] Command: sudo true 2022-07-03T13:08:29.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=646] Received exit status 0 2022-07-03T13:08:29.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=646] Received channel close 2022-07-03T13:08:29.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=646] Channel closed 2022-07-03T13:08:29.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=647] Requesting new SSH session 2022-07-03T13:08:29.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:29.171 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=650] Received exit status 0 2022-07-03T13:08:29.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=650] Received channel close 2022-07-03T13:08:29.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=650] Channel closed 2022-07-03T13:08:29.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=651] Requesting new SSH session 2022-07-03T13:08:29.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=651] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:29.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=647] Received exit status 0 2022-07-03T13:08:29.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=647] Received channel close 2022-07-03T13:08:29.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=647] Channel closed 2022-07-03T13:08:29.174 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=651] Received exit status 0 2022-07-03T13:08:29.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=651] Received channel close 2022-07-03T13:08:29.175 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=651] Channel closed 2022-07-03T13:08:29.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=225] Login timeout expired 2022-07-03T13:08:29.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=225] Aborting connection 2022-07-03T13:08:29.176 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:08:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:08:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:28 smithi188 ceph-mon[92425]: pgmap v4482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:08:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:08:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:08:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:08:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:08:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:08:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:08:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:08:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:08:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:08:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:08:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:08:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:08:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:08:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:08:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:08:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:08:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:08:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:08:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:08:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:08:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:08:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:08:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:08:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:08:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:08:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:08:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:08:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:08:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:08:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:08:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:08:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:08:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:08:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:08:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:08:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:08:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:08:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:08:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:08:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:08:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:08:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:08:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:08:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:08:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:08:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=225] Local address: 172.21.15.161, port 53474 2022-07-03T13:08:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=225] Peer address: 172.21.15.101, port 22 2022-07-03T13:08:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=640] Received exit status 0 2022-07-03T13:08:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=640] Received channel close 2022-07-03T13:08:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=640] Channel closed 2022-07-03T13:08:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=641] Requesting new SSH session 2022-07-03T13:08:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=641] Command: which python3 2022-07-03T13:08:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=644] Received exit status 0 2022-07-03T13:08:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=644] Received channel close 2022-07-03T13:08:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=644] Channel closed 2022-07-03T13:08:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=645] Requesting new SSH session 2022-07-03T13:08:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=645] Command: which python3 2022-07-03T13:08:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=641] Received exit status 0 2022-07-03T13:08:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=641] Received channel close 2022-07-03T13:08:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=641] Channel closed 2022-07-03T13:08:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=642] Requesting new SSH session 2022-07-03T13:08:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=642] Command: sudo true 2022-07-03T13:08:30.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=645] Received exit status 0 2022-07-03T13:08:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=645] Received channel close 2022-07-03T13:08:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=645] Channel closed 2022-07-03T13:08:30.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=646] Requesting new SSH session 2022-07-03T13:08:30.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=646] Command: sudo true 2022-07-03T13:08:30.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=642] Received exit status 0 2022-07-03T13:08:30.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=642] Received channel close 2022-07-03T13:08:30.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=642] Channel closed 2022-07-03T13:08:30.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=643] Requesting new SSH session 2022-07-03T13:08:30.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:30.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=646] Received exit status 0 2022-07-03T13:08:30.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=646] Received channel close 2022-07-03T13:08:30.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=646] Channel closed 2022-07-03T13:08:30.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=647] Requesting new SSH session 2022-07-03T13:08:30.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:30.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=643] Received exit status 0 2022-07-03T13:08:30.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=643] Received channel close 2022-07-03T13:08:30.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=643] Channel closed 2022-07-03T13:08:30.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=647] Received exit status 0 2022-07-03T13:08:30.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=647] Received channel close 2022-07-03T13:08:30.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=647] Channel closed 2022-07-03T13:08:30.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=644] Requesting new SSH session 2022-07-03T13:08:30.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=644] Command: sudo true 2022-07-03T13:08:30.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=648] Requesting new SSH session 2022-07-03T13:08:30.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=648] Command: sudo true 2022-07-03T13:08:30.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=644] Received exit status 0 2022-07-03T13:08:30.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=644] Received channel close 2022-07-03T13:08:30.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=644] Channel closed 2022-07-03T13:08:30.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=645] Requesting new SSH session 2022-07-03T13:08:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=645] Command: which python3 2022-07-03T13:08:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=645] Received exit status 0 2022-07-03T13:08:30.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=645] Received channel close 2022-07-03T13:08:30.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=645] Channel closed 2022-07-03T13:08:30.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=648] Received exit status 0 2022-07-03T13:08:30.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=648] Received channel close 2022-07-03T13:08:30.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=646] Requesting new SSH session 2022-07-03T13:08:30.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=648] Channel closed 2022-07-03T13:08:30.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=646] Command: sudo true 2022-07-03T13:08:30.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=649] Requesting new SSH session 2022-07-03T13:08:30.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=649] Command: which python3 2022-07-03T13:08:30.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=649] Received exit status 0 2022-07-03T13:08:30.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=649] Received channel close 2022-07-03T13:08:30.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=649] Channel closed 2022-07-03T13:08:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=650] Requesting new SSH session 2022-07-03T13:08:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=650] Command: sudo true 2022-07-03T13:08:30.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=646] Received exit status 0 2022-07-03T13:08:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=646] Received channel close 2022-07-03T13:08:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=646] Channel closed 2022-07-03T13:08:30.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=647] Requesting new SSH session 2022-07-03T13:08:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=650] Received exit status 0 2022-07-03T13:08:30.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=650] Received channel close 2022-07-03T13:08:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=650] Channel closed 2022-07-03T13:08:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=651] Requesting new SSH session 2022-07-03T13:08:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=651] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:30.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=647] Received exit status 0 2022-07-03T13:08:30.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=647] Received channel close 2022-07-03T13:08:30.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=647] Channel closed 2022-07-03T13:08:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=651] Received exit status 0 2022-07-03T13:08:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=651] Received channel close 2022-07-03T13:08:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=651] Channel closed 2022-07-03T13:08:30.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=225] Login timeout expired 2022-07-03T13:08:30.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=225] Aborting connection 2022-07-03T13:08:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:08:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:08:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:08:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:08:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:08:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:08:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:08:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:08:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:08:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:08:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:08:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:08:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:08:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:08:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:08:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:08:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:08:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:08:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:08:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:08:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:08:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:08:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:08:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:08:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:08:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:08:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:08:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:08:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:08:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:08:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:08:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:08:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:08:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:08:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:08:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:08:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:08:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:08:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:08:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:08:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:08:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:08:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:08:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:08:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:08:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:08:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=225] Local address: 172.21.15.161, port 53474 2022-07-03T13:08:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=225] Peer address: 172.21.15.101, port 22 2022-07-03T13:08:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=640] Received exit status 0 2022-07-03T13:08:30.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=640] Received channel close 2022-07-03T13:08:30.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=640] Channel closed 2022-07-03T13:08:30.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=641] Requesting new SSH session 2022-07-03T13:08:30.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=641] Command: which python3 2022-07-03T13:08:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=644] Received exit status 0 2022-07-03T13:08:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=644] Received channel close 2022-07-03T13:08:30.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=644] Channel closed 2022-07-03T13:08:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=645] Requesting new SSH session 2022-07-03T13:08:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=645] Command: which python3 2022-07-03T13:08:30.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=641] Received exit status 0 2022-07-03T13:08:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=641] Received channel close 2022-07-03T13:08:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=641] Channel closed 2022-07-03T13:08:30.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=642] Requesting new SSH session 2022-07-03T13:08:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=642] Command: sudo true 2022-07-03T13:08:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=645] Received exit status 0 2022-07-03T13:08:30.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=645] Received channel close 2022-07-03T13:08:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=645] Channel closed 2022-07-03T13:08:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=646] Requesting new SSH session 2022-07-03T13:08:30.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=646] Command: sudo true 2022-07-03T13:08:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=642] Received exit status 0 2022-07-03T13:08:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=642] Received channel close 2022-07-03T13:08:30.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=642] Channel closed 2022-07-03T13:08:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=643] Requesting new SSH session 2022-07-03T13:08:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=643] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:30.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=646] Received exit status 0 2022-07-03T13:08:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=646] Received channel close 2022-07-03T13:08:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=646] Channel closed 2022-07-03T13:08:30.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=647] Requesting new SSH session 2022-07-03T13:08:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T13:08:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=643] Received exit status 0 2022-07-03T13:08:30.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=643] Received channel close 2022-07-03T13:08:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=643] Channel closed 2022-07-03T13:08:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=647] Received exit status 0 2022-07-03T13:08:30.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=647] Received channel close 2022-07-03T13:08:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=647] Channel closed 2022-07-03T13:08:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=644] Requesting new SSH session 2022-07-03T13:08:30.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=644] Command: sudo true 2022-07-03T13:08:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=648] Requesting new SSH session 2022-07-03T13:08:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=648] Command: sudo true 2022-07-03T13:08:30.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=644] Received exit status 0 2022-07-03T13:08:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=644] Received channel close 2022-07-03T13:08:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=644] Channel closed 2022-07-03T13:08:30.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=645] Requesting new SSH session 2022-07-03T13:08:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=645] Command: which python3 2022-07-03T13:08:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=645] Received exit status 0 2022-07-03T13:08:30.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=645] Received channel close 2022-07-03T13:08:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=645] Channel closed 2022-07-03T13:08:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=648] Received exit status 0 2022-07-03T13:08:30.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=648] Received channel close 2022-07-03T13:08:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=646] Requesting new SSH session 2022-07-03T13:08:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=648] Channel closed 2022-07-03T13:08:30.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=646] Command: sudo true 2022-07-03T13:08:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=649] Requesting new SSH session 2022-07-03T13:08:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=649] Command: which python3 2022-07-03T13:08:30.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=649] Received exit status 0 2022-07-03T13:08:30.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=649] Received channel close 2022-07-03T13:08:30.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=649] Channel closed 2022-07-03T13:08:30.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=650] Requesting new SSH session 2022-07-03T13:08:30.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=650] Command: sudo true 2022-07-03T13:08:30.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=646] Received exit status 0 2022-07-03T13:08:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=646] Received channel close 2022-07-03T13:08:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=646] Channel closed 2022-07-03T13:08:30.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=647] Requesting new SSH session 2022-07-03T13:08:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=647] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=650] Received exit status 0 2022-07-03T13:08:30.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=650] Received channel close 2022-07-03T13:08:30.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=650] Channel closed 2022-07-03T13:08:30.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=651] Requesting new SSH session 2022-07-03T13:08:30.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=651] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:08:30.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=647] Received exit status 0 2022-07-03T13:08:30.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=647] Received channel close 2022-07-03T13:08:30.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=647] Channel closed 2022-07-03T13:08:30.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=651] Received exit status 0 2022-07-03T13:08:30.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=651] Received channel close 2022-07-03T13:08:30.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=651] Channel closed 2022-07-03T13:08:30.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=225] Login timeout expired 2022-07-03T13:08:30.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=225] Aborting connection 2022-07-03T13:08:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:30 smithi161 ceph-mon[91579]: pgmap v4483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:30 smithi188 ceph-mon[92425]: pgmap v4483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:32 smithi161 ceph-mon[91579]: pgmap v4484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:32 smithi188 ceph-mon[92425]: pgmap v4484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:34 smithi161 ceph-mon[91579]: pgmap v4485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:34 smithi188 ceph-mon[92425]: pgmap v4485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:36 smithi161 ceph-mon[91579]: pgmap v4486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:36 smithi188 ceph-mon[92425]: pgmap v4486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:38 smithi161 ceph-mon[91579]: pgmap v4487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:38 smithi188 ceph-mon[92425]: pgmap v4487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:40 smithi161 ceph-mon[91579]: pgmap v4488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:40 smithi188 ceph-mon[92425]: pgmap v4488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:42 smithi161 ceph-mon[91579]: pgmap v4489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:42 smithi188 ceph-mon[92425]: pgmap v4489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:44 smithi161 ceph-mon[91579]: pgmap v4490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:44 smithi188 ceph-mon[92425]: pgmap v4490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:46 smithi161 ceph-mon[91579]: pgmap v4491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:46 smithi188 ceph-mon[92425]: pgmap v4491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:48 smithi161 ceph-mon[91579]: pgmap v4492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:48 smithi188 ceph-mon[92425]: pgmap v4492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:50 smithi161 ceph-mon[91579]: pgmap v4493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:50 smithi188 ceph-mon[92425]: pgmap v4493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:52 smithi161 ceph-mon[91579]: pgmap v4494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:52 smithi188 ceph-mon[92425]: pgmap v4494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:54 smithi161 ceph-mon[91579]: pgmap v4495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:54 smithi188 ceph-mon[92425]: pgmap v4495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:08:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:56 smithi161 ceph-mon[91579]: pgmap v4496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:56 smithi188 ceph-mon[92425]: pgmap v4496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:08:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:08:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:08:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:08:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:08:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:08:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:08:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:08:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:08:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:08:58 smithi161 ceph-mon[91579]: pgmap v4497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:08:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:08:58 smithi188 ceph-mon[92425]: pgmap v4497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:00 smithi161 ceph-mon[91579]: pgmap v4498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:00 smithi188 ceph-mon[92425]: pgmap v4498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:02 smithi161 ceph-mon[91579]: pgmap v4499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:02 smithi188 ceph-mon[92425]: pgmap v4499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:04 smithi161 ceph-mon[91579]: pgmap v4500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:04 smithi188 ceph-mon[92425]: pgmap v4500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:06 smithi161 ceph-mon[91579]: pgmap v4501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:06 smithi188 ceph-mon[92425]: pgmap v4501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:08 smithi188 ceph-mon[92425]: pgmap v4502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:08 smithi161 ceph-mon[91579]: pgmap v4502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:10 smithi188 ceph-mon[92425]: pgmap v4503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:10 smithi161 ceph-mon[91579]: pgmap v4503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:12 smithi188 ceph-mon[92425]: pgmap v4504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:12 smithi161 ceph-mon[91579]: pgmap v4504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:14 smithi188 ceph-mon[92425]: pgmap v4505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:14 smithi161 ceph-mon[91579]: pgmap v4505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:16 smithi188 ceph-mon[92425]: pgmap v4506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:16 smithi161 ceph-mon[91579]: pgmap v4506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:18 smithi188 ceph-mon[92425]: pgmap v4507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:18 smithi161 ceph-mon[91579]: pgmap v4507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:20 smithi188 ceph-mon[92425]: pgmap v4508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:20 smithi161 ceph-mon[91579]: pgmap v4508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:22 smithi161 ceph-mon[91579]: pgmap v4509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:22 smithi188 ceph-mon[92425]: pgmap v4509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:24 smithi161 ceph-mon[91579]: pgmap v4510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:24 smithi188 ceph-mon[92425]: pgmap v4510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:26 smithi161 ceph-mon[91579]: pgmap v4511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:26 smithi188 ceph-mon[92425]: pgmap v4511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-mon[91579]: pgmap v4512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:09:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:09:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:09:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:09:28.816+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:09:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:09:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:09:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:09:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:09:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:09:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:09:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:09:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:09:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:09:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:09:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:09:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:09:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:09:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:09:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:09:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:09:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:09:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:09:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:09:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:09:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:09:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:09:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:09:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:09:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:09:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:09:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:09:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:09:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:09:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:09:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:09:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:09:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:09:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:09:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:09:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:09:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:09:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:09:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:09:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:09:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:09:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:09:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:09:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:09:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:09:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=227] Local address: 172.21.15.161, port 53502 2022-07-03T13:09:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=227] Peer address: 172.21.15.101, port 22 2022-07-03T13:09:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=227] Connection lost 2022-07-03T13:09:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=227] Aborting connection 2022-07-03T13:09:29.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:09:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:09:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:28 smithi188 ceph-mon[92425]: pgmap v4512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:09:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:09:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:09:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:09:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:09:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:09:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:09:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:09:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:09:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:09:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:09:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:09:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:09:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:09:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:09:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:09:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:09:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:09:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:09:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:09:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:09:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:09:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:09:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:09:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:09:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:09:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:09:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:09:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:09:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:09:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:09:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:09:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:09:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:09:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:09:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:09:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:09:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:09:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:09:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:09:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:09:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:09:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:09:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:09:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:09:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:09:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:09:30.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:09:30.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:09:30.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:09:30.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=227] Local address: 172.21.15.161, port 53502 2022-07-03T13:09:30.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=227] Peer address: 172.21.15.101, port 22 2022-07-03T13:09:30.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=227] Connection lost 2022-07-03T13:09:30.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=227] Aborting connection 2022-07-03T13:09:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:09:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:09:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:09:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:09:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:09:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:09:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:09:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:09:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:09:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:09:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:09:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:09:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:09:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:09:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:09:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:09:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:09:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:09:30.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:09:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:09:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:09:30.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:09:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:09:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:09:30.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:09:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:09:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:09:30.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:09:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:09:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:09:30.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:09:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:09:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:09:30.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:09:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:09:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:09:30.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:09:30.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:09:30.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:09:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:09:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:09:30.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:09:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:09:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:09:30.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:09:30.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:09:30.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:09:30.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=227] Local address: 172.21.15.161, port 53502 2022-07-03T13:09:30.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=227] Peer address: 172.21.15.101, port 22 2022-07-03T13:09:30.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=227] Connection lost 2022-07-03T13:09:30.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=227] Aborting connection 2022-07-03T13:09:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:30 smithi161 ceph-mon[91579]: pgmap v4513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:30 smithi188 ceph-mon[92425]: pgmap v4513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:32.860 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: pgmap v4514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:32.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.861 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.862 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:32.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.863 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.864 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:09:32.865 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:09:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: pgmap v4514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:09:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:09:34.867 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:34 smithi188 ceph-mon[92425]: pgmap v4515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:34 smithi161 ceph-mon[91579]: pgmap v4515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T13:09:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:09:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:35 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:09:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T13:09:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:09:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:09:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:09:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:35 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:09:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:36 smithi161 ceph-mon[91579]: pgmap v4516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:36 smithi188 ceph-mon[92425]: pgmap v4516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:38 smithi161 ceph-mon[91579]: pgmap v4517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:38 smithi188 ceph-mon[92425]: pgmap v4517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:40 smithi161 ceph-mon[91579]: pgmap v4518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:40 smithi188 ceph-mon[92425]: pgmap v4518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:42 smithi161 ceph-mon[91579]: pgmap v4519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:42 smithi188 ceph-mon[92425]: pgmap v4519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:44 smithi161 ceph-mon[91579]: pgmap v4520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:44 smithi188 ceph-mon[92425]: pgmap v4520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:46 smithi161 ceph-mon[91579]: pgmap v4521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:46 smithi188 ceph-mon[92425]: pgmap v4521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:48 smithi161 ceph-mon[91579]: pgmap v4522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:48 smithi188 ceph-mon[92425]: pgmap v4522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:50 smithi161 ceph-mon[91579]: pgmap v4523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:50 smithi188 ceph-mon[92425]: pgmap v4523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:52 smithi161 ceph-mon[91579]: pgmap v4524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:52 smithi188 ceph-mon[92425]: pgmap v4524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:54 smithi161 ceph-mon[91579]: pgmap v4525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:54 smithi188 ceph-mon[92425]: pgmap v4525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:09:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:56 smithi161 ceph-mon[91579]: pgmap v4526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:56 smithi188 ceph-mon[92425]: pgmap v4526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:09:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:09:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:09:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:09:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:09:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:09:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:09:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:09:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:09:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:09:58 smithi161 ceph-mon[91579]: pgmap v4527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:09:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:09:58 smithi188 ceph-mon[92425]: pgmap v4527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:00 smithi161 ceph-mon[91579]: pgmap v4528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:10:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:00 smithi188 ceph-mon[92425]: pgmap v4528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:10:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:02 smithi161 ceph-mon[91579]: pgmap v4529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:02 smithi188 ceph-mon[92425]: pgmap v4529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:04 smithi161 ceph-mon[91579]: pgmap v4530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:04 smithi188 ceph-mon[92425]: pgmap v4530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:06 smithi161 ceph-mon[91579]: pgmap v4531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:06 smithi188 ceph-mon[92425]: pgmap v4531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:08 smithi161 ceph-mon[91579]: pgmap v4532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:08 smithi188 ceph-mon[92425]: pgmap v4532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:10 smithi161 ceph-mon[91579]: pgmap v4533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:10 smithi188 ceph-mon[92425]: pgmap v4533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:12 smithi161 ceph-mon[91579]: pgmap v4534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:12 smithi188 ceph-mon[92425]: pgmap v4534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:14 smithi188 ceph-mon[92425]: pgmap v4535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:14 smithi161 ceph-mon[91579]: pgmap v4535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:17 smithi161 ceph-mon[91579]: pgmap v4536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:17 smithi188 ceph-mon[92425]: pgmap v4536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:19 smithi161 ceph-mon[91579]: pgmap v4537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:19 smithi188 ceph-mon[92425]: pgmap v4537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:21 smithi161 ceph-mon[91579]: pgmap v4538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:21 smithi188 ceph-mon[92425]: pgmap v4538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:23 smithi161 ceph-mon[91579]: pgmap v4539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:23 smithi188 ceph-mon[92425]: pgmap v4539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:25 smithi161 ceph-mon[91579]: pgmap v4540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:25 smithi188 ceph-mon[92425]: pgmap v4540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:27 smithi161 ceph-mon[91579]: pgmap v4541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:27 smithi188 ceph-mon[92425]: pgmap v4541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:29 smithi161 ceph-mon[91579]: pgmap v4542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:29 smithi188 ceph-mon[92425]: pgmap v4542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:31 smithi161 ceph-mon[91579]: pgmap v4543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:31 smithi188 ceph-mon[92425]: pgmap v4543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:33 smithi161 ceph-mon[91579]: pgmap v4544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:33 smithi188 ceph-mon[92425]: pgmap v4544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:35 smithi161 ceph-mon[91579]: pgmap v4545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:10:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:10:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:10:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:35 smithi188 ceph-mon[92425]: pgmap v4545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:10:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:10:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:10:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:37 smithi161 ceph-mon[91579]: pgmap v4546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:37 smithi188 ceph-mon[92425]: pgmap v4546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:39 smithi161 ceph-mon[91579]: pgmap v4547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:39 smithi188 ceph-mon[92425]: pgmap v4547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:41 smithi161 ceph-mon[91579]: pgmap v4548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:41 smithi188 ceph-mon[92425]: pgmap v4548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:43 smithi161 ceph-mon[91579]: pgmap v4549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:43 smithi188 ceph-mon[92425]: pgmap v4549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:45 smithi161 ceph-mon[91579]: pgmap v4550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:45 smithi188 ceph-mon[92425]: pgmap v4550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:47 smithi161 ceph-mon[91579]: pgmap v4551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:47 smithi188 ceph-mon[92425]: pgmap v4551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:49 smithi161 ceph-mon[91579]: pgmap v4552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:49 smithi188 ceph-mon[92425]: pgmap v4552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:51 smithi161 ceph-mon[91579]: pgmap v4553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:51 smithi188 ceph-mon[92425]: pgmap v4553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:53 smithi161 ceph-mon[91579]: pgmap v4554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:53 smithi188 ceph-mon[92425]: pgmap v4554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:55 smithi161 ceph-mon[91579]: pgmap v4555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:55 smithi188 ceph-mon[92425]: pgmap v4555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:10:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:57 smithi161 ceph-mon[91579]: pgmap v4556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:57 smithi188 ceph-mon[92425]: pgmap v4556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:10:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:10:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:10:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:10:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:10:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:10:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:10:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:10:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:10:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:10:59 smithi161 ceph-mon[91579]: pgmap v4557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:10:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:10:59 smithi188 ceph-mon[92425]: pgmap v4557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:01 smithi161 ceph-mon[91579]: pgmap v4558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:01 smithi188 ceph-mon[92425]: pgmap v4558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:03 smithi161 ceph-mon[91579]: pgmap v4559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:03 smithi188 ceph-mon[92425]: pgmap v4559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:05 smithi161 ceph-mon[91579]: pgmap v4560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:11:04.958+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:11:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:11:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:11:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:11:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:11:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:11:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:11:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:11:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:11:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:11:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:11:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:11:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:11:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:11:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:11:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:11:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:11:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:11:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:11:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:11:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:11:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:11:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:11:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:11:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:11:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:11:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:11:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:11:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:11:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:11:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:11:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:11:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:11:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:11:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:11:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:11:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:11:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:11:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:11:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:11:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:11:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:11:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:11:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:11:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:11:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:11:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=229] Local address: 172.21.15.161, port 53506 2022-07-03T13:11:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=229] Peer address: 172.21.15.101, port 22 2022-07-03T13:11:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=229] Connection lost 2022-07-03T13:11:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=229] Aborting connection 2022-07-03T13:11:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:11:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:11:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:05 smithi188 ceph-mon[92425]: pgmap v4560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:11:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:11:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:11:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:11:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:11:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:11:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:11:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:11:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:11:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:11:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:11:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:11:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:11:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:11:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:11:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:11:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:11:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:11:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:11:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:11:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:11:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:11:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:11:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:11:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:11:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:11:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:11:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:11:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:11:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:11:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:11:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:11:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:11:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:11:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:11:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:11:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:11:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:11:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:11:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:11:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:11:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:11:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:11:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:11:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:11:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:11:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:11:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=229] Local address: 172.21.15.161, port 53506 2022-07-03T13:11:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=229] Peer address: 172.21.15.101, port 22 2022-07-03T13:11:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=229] Connection lost 2022-07-03T13:11:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=229] Aborting connection 2022-07-03T13:11:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:11:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:11:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:11:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:11:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:11:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:11:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:11:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:11:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:11:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:11:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:11:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:11:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:11:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:11:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:11:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:11:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:11:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:11:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:11:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:11:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:11:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:11:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:11:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:11:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:11:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:11:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:11:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:11:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:11:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:11:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:11:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:11:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:11:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:11:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:11:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:11:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:11:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:11:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:11:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:11:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:11:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:11:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:11:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:11:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:11:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:11:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:11:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=229] Local address: 172.21.15.161, port 53506 2022-07-03T13:11:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=229] Peer address: 172.21.15.101, port 22 2022-07-03T13:11:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=229] Connection lost 2022-07-03T13:11:06.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=229] Aborting connection 2022-07-03T13:11:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:07 smithi161 ceph-mon[91579]: pgmap v4561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:07 smithi188 ceph-mon[92425]: pgmap v4561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:09 smithi161 ceph-mon[91579]: pgmap v4562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:09 smithi188 ceph-mon[92425]: pgmap v4562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:11 smithi161 ceph-mon[91579]: pgmap v4563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:11 smithi188 ceph-mon[92425]: pgmap v4563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:13 smithi161 ceph-mon[91579]: pgmap v4564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:13 smithi188 ceph-mon[92425]: pgmap v4564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:15 smithi161 ceph-mon[91579]: pgmap v4565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:15 smithi188 ceph-mon[92425]: pgmap v4565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:17 smithi161 ceph-mon[91579]: pgmap v4566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:17 smithi188 ceph-mon[92425]: pgmap v4566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:19 smithi161 ceph-mon[91579]: pgmap v4567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:19 smithi188 ceph-mon[92425]: pgmap v4567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:21 smithi188 ceph-mon[92425]: pgmap v4568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:21 smithi161 ceph-mon[91579]: pgmap v4568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:23 smithi188 ceph-mon[92425]: pgmap v4569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:23 smithi161 ceph-mon[91579]: pgmap v4569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:25 smithi188 ceph-mon[92425]: pgmap v4570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:25 smithi161 ceph-mon[91579]: pgmap v4570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:27 smithi188 ceph-mon[92425]: pgmap v4571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:27 smithi161 ceph-mon[91579]: pgmap v4571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:29 smithi188 ceph-mon[92425]: pgmap v4572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:29 smithi161 ceph-mon[91579]: pgmap v4572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:31 smithi188 ceph-mon[92425]: pgmap v4573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:31 smithi161 ceph-mon[91579]: pgmap v4573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:33 smithi188 ceph-mon[92425]: pgmap v4574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:33 smithi161 ceph-mon[91579]: pgmap v4574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:34 smithi161 ceph-mon[91579]: pgmap v4575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:34 smithi188 ceph-mon[92425]: pgmap v4575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:36 smithi188 ceph-mon[92425]: pgmap v4576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:36 smithi161 ceph-mon[91579]: pgmap v4576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:38 smithi188 ceph-mon[92425]: pgmap v4577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:38 smithi161 ceph-mon[91579]: pgmap v4577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:40 smithi188 ceph-mon[92425]: pgmap v4578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:40 smithi161 ceph-mon[91579]: pgmap v4578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:42 smithi188 ceph-mon[92425]: pgmap v4579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:42 smithi161 ceph-mon[91579]: pgmap v4579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:44 smithi188 ceph-mon[92425]: pgmap v4580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:44 smithi161 ceph-mon[91579]: pgmap v4580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:46 smithi188 ceph-mon[92425]: pgmap v4581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:46 smithi161 ceph-mon[91579]: pgmap v4581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:48 smithi188 ceph-mon[92425]: pgmap v4582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:48 smithi161 ceph-mon[91579]: pgmap v4582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:50 smithi188 ceph-mon[92425]: pgmap v4583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:50 smithi161 ceph-mon[91579]: pgmap v4583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:52 smithi188 ceph-mon[92425]: pgmap v4584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:52 smithi161 ceph-mon[91579]: pgmap v4584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:54 smithi161 ceph-mon[91579]: pgmap v4585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:54 smithi188 ceph-mon[92425]: pgmap v4585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:11:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:56 smithi161 ceph-mon[91579]: pgmap v4586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:56 smithi188 ceph-mon[92425]: pgmap v4586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:11:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:11:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:11:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:11:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:11:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:11:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:11:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:11:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:11:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:11:58 smithi161 ceph-mon[91579]: pgmap v4587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:11:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:11:58 smithi188 ceph-mon[92425]: pgmap v4587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:00 smithi161 ceph-mon[91579]: pgmap v4588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:00 smithi188 ceph-mon[92425]: pgmap v4588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:02 smithi161 ceph-mon[91579]: pgmap v4589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:02 smithi188 ceph-mon[92425]: pgmap v4589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:04.994 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-mon[91579]: pgmap v4590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:05.015 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:04 smithi188 ceph-mon[92425]: pgmap v4590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:05.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:12:04.993+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:12:05.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:12:05.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:12:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:12:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:12:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:12:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:12:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:12:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:12:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:12:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:12:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:12:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:12:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:12:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:12:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:12:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:12:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:12:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:12:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:12:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:12:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:12:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:12:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:12:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:12:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:12:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:12:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:12:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:12:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:12:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:12:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:12:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:12:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:12:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:12:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:12:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:12:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:12:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:12:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:12:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:12:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:12:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:12:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:12:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:12:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:12:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=231] Local address: 172.21.15.161, port 53510 2022-07-03T13:12:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=231] Peer address: 172.21.15.101, port 22 2022-07-03T13:12:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=231] Connection lost 2022-07-03T13:12:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=231] Aborting connection 2022-07-03T13:12:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:12:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:12:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:12:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:12:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:12:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:12:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:12:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:12:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:12:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:12:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:12:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:12:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:12:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:12:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:12:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:12:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:12:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:12:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:12:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:12:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:12:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:12:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:12:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:12:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:12:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:12:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:12:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:12:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:12:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:12:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:12:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:12:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:12:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:12:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:12:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:12:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:12:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:12:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:12:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:12:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:12:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:12:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:12:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:12:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:12:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:12:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:12:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:12:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:12:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:12:05.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:12:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:12:05.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=231] Local address: 172.21.15.161, port 53510 2022-07-03T13:12:05.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=231] Peer address: 172.21.15.101, port 22 2022-07-03T13:12:05.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=231] Connection lost 2022-07-03T13:12:05.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=231] Aborting connection 2022-07-03T13:12:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:12:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:12:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:12:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:12:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:12:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:12:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:12:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:12:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:12:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:12:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:12:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:12:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:12:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:12:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:12:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:12:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:12:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:12:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:12:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:12:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:12:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:12:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:12:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:12:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:12:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:12:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:12:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:12:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:12:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:12:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:12:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:12:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:12:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:12:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:12:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:12:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:12:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:12:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:12:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:12:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:12:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:12:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:12:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:12:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:12:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:12:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:12:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:12:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:12:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:12:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=231] Local address: 172.21.15.161, port 53510 2022-07-03T13:12:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=231] Peer address: 172.21.15.101, port 22 2022-07-03T13:12:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=231] Connection lost 2022-07-03T13:12:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=231] Aborting connection 2022-07-03T13:12:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: pgmap v4591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: pgmap v4591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:12:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:08 smithi161 ceph-mon[91579]: pgmap v4592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:08 smithi188 ceph-mon[92425]: pgmap v4592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:12:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:10 smithi161 ceph-mon[91579]: pgmap v4593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:10 smithi188 ceph-mon[92425]: pgmap v4593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:12 smithi188 ceph-mon[92425]: pgmap v4594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:12 smithi161 ceph-mon[91579]: pgmap v4594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:14 smithi188 ceph-mon[92425]: pgmap v4595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:14 smithi161 ceph-mon[91579]: pgmap v4595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:16 smithi188 ceph-mon[92425]: pgmap v4596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:16 smithi161 ceph-mon[91579]: pgmap v4596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:18 smithi188 ceph-mon[92425]: pgmap v4597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:18 smithi161 ceph-mon[91579]: pgmap v4597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:20 smithi188 ceph-mon[92425]: pgmap v4598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:20 smithi161 ceph-mon[91579]: pgmap v4598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:22 smithi188 ceph-mon[92425]: pgmap v4599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:22 smithi161 ceph-mon[91579]: pgmap v4599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:24 smithi188 ceph-mon[92425]: pgmap v4600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:24 smithi161 ceph-mon[91579]: pgmap v4600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:26 smithi161 ceph-mon[91579]: pgmap v4601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:26 smithi188 ceph-mon[92425]: pgmap v4601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:28 smithi161 ceph-mon[91579]: pgmap v4602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:28 smithi188 ceph-mon[92425]: pgmap v4602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:30 smithi161 ceph-mon[91579]: pgmap v4603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:30 smithi188 ceph-mon[92425]: pgmap v4603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:32 smithi161 ceph-mon[91579]: pgmap v4604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:32 smithi188 ceph-mon[92425]: pgmap v4604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:34 smithi161 ceph-mon[91579]: pgmap v4605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:34 smithi188 ceph-mon[92425]: pgmap v4605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:36 smithi161 ceph-mon[91579]: pgmap v4606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:36 smithi188 ceph-mon[92425]: pgmap v4606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:38 smithi161 ceph-mon[91579]: pgmap v4607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:38 smithi188 ceph-mon[92425]: pgmap v4607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:40 smithi161 ceph-mon[91579]: pgmap v4608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:40 smithi188 ceph-mon[92425]: pgmap v4608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:42 smithi161 ceph-mon[91579]: pgmap v4609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:42 smithi188 ceph-mon[92425]: pgmap v4609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:44 smithi161 ceph-mon[91579]: pgmap v4610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:44 smithi188 ceph-mon[92425]: pgmap v4610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:46 smithi161 ceph-mon[91579]: pgmap v4611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:46 smithi188 ceph-mon[92425]: pgmap v4611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:48 smithi161 ceph-mon[91579]: pgmap v4612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:48 smithi188 ceph-mon[92425]: pgmap v4612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:50 smithi161 ceph-mon[91579]: pgmap v4613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:50 smithi188 ceph-mon[92425]: pgmap v4613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:52 smithi161 ceph-mon[91579]: pgmap v4614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:52 smithi188 ceph-mon[92425]: pgmap v4614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:54 smithi161 ceph-mon[91579]: pgmap v4615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:54 smithi188 ceph-mon[92425]: pgmap v4615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:12:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:56 smithi161 ceph-mon[91579]: pgmap v4616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:56 smithi188 ceph-mon[92425]: pgmap v4616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:12:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:12:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:12:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:12:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:12:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:12:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:12:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:12:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:12:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:12:58 smithi161 ceph-mon[91579]: pgmap v4617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:12:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:12:58 smithi188 ceph-mon[92425]: pgmap v4617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:00 smithi161 ceph-mon[91579]: pgmap v4618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:00 smithi188 ceph-mon[92425]: pgmap v4618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:02 smithi161 ceph-mon[91579]: pgmap v4619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:02 smithi188 ceph-mon[92425]: pgmap v4619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:04 smithi161 ceph-mon[91579]: pgmap v4620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:04 smithi188 ceph-mon[92425]: pgmap v4620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:13:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:13:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:13:06.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:13:05.917+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:13:06.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:13:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:13:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:13:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:13:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:13:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:13:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:13:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:13:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:13:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:13:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:13:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:13:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:13:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:13:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:13:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:13:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:13:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:13:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:13:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:13:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:13:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:13:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:13:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:13:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:13:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:13:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:13:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:13:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:13:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:13:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:13:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:13:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:13:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:13:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:13:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:13:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:13:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:13:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:13:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:13:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:13:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:13:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:13:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:13:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:13:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=233] Local address: 172.21.15.161, port 53514 2022-07-03T13:13:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=233] Peer address: 172.21.15.101, port 22 2022-07-03T13:13:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=233] Connection lost 2022-07-03T13:13:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=233] Aborting connection 2022-07-03T13:13:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:13:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:13:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:13:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:13:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:13:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:07 smithi161 ceph-mon[91579]: pgmap v4621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:13:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:13:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:13:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:13:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:13:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:13:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:13:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:13:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:13:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:13:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:13:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:13:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:13:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:13:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:13:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:13:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:13:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:13:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:13:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:13:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:13:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:13:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:13:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:13:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:13:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:13:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:13:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:13:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:13:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:13:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:13:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:13:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:13:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:13:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:13:07.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:13:07.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:13:07.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:13:07.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:13:07.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:13:07.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:13:07.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:13:07.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:13:07.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:13:07.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:13:07.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:13:07.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:13:07.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=233] Local address: 172.21.15.161, port 53514 2022-07-03T13:13:07.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=233] Peer address: 172.21.15.101, port 22 2022-07-03T13:13:07.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=233] Connection lost 2022-07-03T13:13:07.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=233] Aborting connection 2022-07-03T13:13:07.412 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:07 smithi188 ceph-mon[92425]: pgmap v4621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:13:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:13:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:13:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:13:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:13:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:13:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:13:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:13:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:13:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:13:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:13:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:13:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:13:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:13:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:13:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:13:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:13:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:13:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:13:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:13:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:13:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:13:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:13:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:13:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:13:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:13:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:13:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:13:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:13:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:13:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:13:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:13:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:13:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:13:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:13:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:13:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:13:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:13:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:13:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:13:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:13:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:13:07.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:13:07.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:13:07.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:13:07.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:13:07.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:13:07.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=233] Local address: 172.21.15.161, port 53514 2022-07-03T13:13:07.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=233] Peer address: 172.21.15.101, port 22 2022-07-03T13:13:07.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=233] Connection lost 2022-07-03T13:13:07.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=233] Aborting connection 2022-07-03T13:13:07.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:09 smithi161 ceph-mon[91579]: pgmap v4622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:09 smithi188 ceph-mon[92425]: pgmap v4622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:11 smithi161 ceph-mon[91579]: pgmap v4623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:11 smithi188 ceph-mon[92425]: pgmap v4623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:13 smithi161 ceph-mon[91579]: pgmap v4624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:13 smithi188 ceph-mon[92425]: pgmap v4624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:15 smithi161 ceph-mon[91579]: pgmap v4625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:15 smithi188 ceph-mon[92425]: pgmap v4625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:17 smithi161 ceph-mon[91579]: pgmap v4626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:17 smithi188 ceph-mon[92425]: pgmap v4626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:19 smithi161 ceph-mon[91579]: pgmap v4627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:19 smithi188 ceph-mon[92425]: pgmap v4627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:21 smithi161 ceph-mon[91579]: pgmap v4628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:21 smithi188 ceph-mon[92425]: pgmap v4628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:23 smithi161 ceph-mon[91579]: pgmap v4629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:23 smithi188 ceph-mon[92425]: pgmap v4629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:25 smithi161 ceph-mon[91579]: pgmap v4630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:25 smithi188 ceph-mon[92425]: pgmap v4630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:27 smithi161 ceph-mon[91579]: pgmap v4631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:27 smithi188 ceph-mon[92425]: pgmap v4631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:29 smithi161 ceph-mon[91579]: pgmap v4632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:29 smithi188 ceph-mon[92425]: pgmap v4632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:31 smithi161 ceph-mon[91579]: pgmap v4633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:31 smithi188 ceph-mon[92425]: pgmap v4633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:33 smithi161 ceph-mon[91579]: pgmap v4634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:33 smithi188 ceph-mon[92425]: pgmap v4634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:35 smithi161 ceph-mon[91579]: pgmap v4635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:35 smithi188 ceph-mon[92425]: pgmap v4635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:37 smithi161 ceph-mon[91579]: pgmap v4636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:37 smithi188 ceph-mon[92425]: pgmap v4636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:39 smithi161 ceph-mon[91579]: pgmap v4637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:39 smithi188 ceph-mon[92425]: pgmap v4637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:41 smithi161 ceph-mon[91579]: pgmap v4638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:41 smithi188 ceph-mon[92425]: pgmap v4638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:43 smithi161 ceph-mon[91579]: pgmap v4639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:43 smithi188 ceph-mon[92425]: pgmap v4639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:45 smithi161 ceph-mon[91579]: pgmap v4640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:45 smithi188 ceph-mon[92425]: pgmap v4640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:47 smithi161 ceph-mon[91579]: pgmap v4641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:47 smithi188 ceph-mon[92425]: pgmap v4641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:49 smithi161 ceph-mon[91579]: pgmap v4642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:49 smithi188 ceph-mon[92425]: pgmap v4642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:51 smithi161 ceph-mon[91579]: pgmap v4643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:51 smithi188 ceph-mon[92425]: pgmap v4643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:53 smithi161 ceph-mon[91579]: pgmap v4644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:53 smithi188 ceph-mon[92425]: pgmap v4644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:55 smithi161 ceph-mon[91579]: pgmap v4645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:55 smithi188 ceph-mon[92425]: pgmap v4645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:13:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:57 smithi161 ceph-mon[91579]: pgmap v4646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:57 smithi188 ceph-mon[92425]: pgmap v4646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:13:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:13:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:13:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:13:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:13:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:13:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:13:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:13:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:13:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:13:59 smithi161 ceph-mon[91579]: pgmap v4647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:13:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:13:59 smithi188 ceph-mon[92425]: pgmap v4647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:01 smithi188 ceph-mon[92425]: pgmap v4648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:01 smithi161 ceph-mon[91579]: pgmap v4648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:03 smithi188 ceph-mon[92425]: pgmap v4649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:03 smithi161 ceph-mon[91579]: pgmap v4649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:05 smithi188 ceph-mon[92425]: pgmap v4650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:05 smithi161 ceph-mon[91579]: pgmap v4650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:07 smithi188 ceph-mon[92425]: pgmap v4651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:14:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:14:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:14:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:07 smithi161 ceph-mon[91579]: pgmap v4651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:14:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:14:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:14:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:14:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:14:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:09 smithi188 ceph-mon[92425]: pgmap v4652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:09 smithi161 ceph-mon[91579]: pgmap v4652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:11 smithi188 ceph-mon[92425]: pgmap v4653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:11 smithi161 ceph-mon[91579]: pgmap v4653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:13 smithi161 ceph-mon[91579]: pgmap v4654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:13 smithi188 ceph-mon[92425]: pgmap v4654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:15 smithi161 ceph-mon[91579]: pgmap v4655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:15 smithi188 ceph-mon[92425]: pgmap v4655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:17 smithi161 ceph-mon[91579]: pgmap v4656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:17 smithi188 ceph-mon[92425]: pgmap v4656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:19 smithi161 ceph-mon[91579]: pgmap v4657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:19 smithi188 ceph-mon[92425]: pgmap v4657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:21 smithi161 ceph-mon[91579]: pgmap v4658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:21 smithi188 ceph-mon[92425]: pgmap v4658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:23 smithi161 ceph-mon[91579]: pgmap v4659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:23 smithi188 ceph-mon[92425]: pgmap v4659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:25 smithi161 ceph-mon[91579]: pgmap v4660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:25 smithi188 ceph-mon[92425]: pgmap v4660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:26 smithi161 ceph-mon[91579]: pgmap v4661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:26 smithi188 ceph-mon[92425]: pgmap v4661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:28 smithi161 ceph-mon[91579]: pgmap v4662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:28 smithi188 ceph-mon[92425]: pgmap v4662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:30 smithi161 ceph-mon[91579]: pgmap v4663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:30 smithi188 ceph-mon[92425]: pgmap v4663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:32 smithi161 ceph-mon[91579]: pgmap v4664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:32 smithi188 ceph-mon[92425]: pgmap v4664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:34 smithi161 ceph-mon[91579]: pgmap v4665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:34 smithi188 ceph-mon[92425]: pgmap v4665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:36.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:14:36.417+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:14:36.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:14:36.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:14:36.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:14:36.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:14:36.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:14:36.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:14:36.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:14:36.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:14:36.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:14:36.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:14:36.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:14:36.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:14:36.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:14:36.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:14:36.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:14:36.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:14:36.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:14:36.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:14:36.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:14:36.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:14:36.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:14:36.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:14:36.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:14:36.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:14:36.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:14:36.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:14:36.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:14:36.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:14:36.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:14:36.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:14:36.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:14:36.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:14:36.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:14:36.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:14:36.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:14:36.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:14:36.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:14:36.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:14:36.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:14:36.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:14:36.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:14:36.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:14:36.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:14:36.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:14:36.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:14:36.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=235] Local address: 172.21.15.161, port 53518 2022-07-03T13:14:36.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=235] Peer address: 172.21.15.101, port 22 2022-07-03T13:14:36.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=676] Received exit status 0 2022-07-03T13:14:36.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=676] Received channel close 2022-07-03T13:14:36.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=676] Channel closed 2022-07-03T13:14:36.697 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=677] Requesting new SSH session 2022-07-03T13:14:36.697 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=677] Command: which python3 2022-07-03T13:14:36.697 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=676] Received exit status 0 2022-07-03T13:14:36.698 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=676] Received channel close 2022-07-03T13:14:36.698 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=676] Channel closed 2022-07-03T13:14:36.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=677] Requesting new SSH session 2022-07-03T13:14:36.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=677] Command: which python3 2022-07-03T13:14:36.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=677] Received exit status 0 2022-07-03T13:14:36.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=677] Received channel close 2022-07-03T13:14:36.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=677] Channel closed 2022-07-03T13:14:36.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=678] Requesting new SSH session 2022-07-03T13:14:36.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=678] Command: sudo true 2022-07-03T13:14:36.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=677] Received exit status 0 2022-07-03T13:14:36.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=677] Received channel close 2022-07-03T13:14:36.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=677] Channel closed 2022-07-03T13:14:36.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=678] Requesting new SSH session 2022-07-03T13:14:36.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=678] Command: sudo true 2022-07-03T13:14:36.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=678] Received exit status 0 2022-07-03T13:14:36.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=678] Received channel close 2022-07-03T13:14:36.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=678] Channel closed 2022-07-03T13:14:36.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=679] Requesting new SSH session 2022-07-03T13:14:36.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:36.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=678] Received exit status 0 2022-07-03T13:14:36.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=678] Received channel close 2022-07-03T13:14:36.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=678] Channel closed 2022-07-03T13:14:36.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=679] Requesting new SSH session 2022-07-03T13:14:36.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:36.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=679] Received exit status 0 2022-07-03T13:14:36.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=679] Received channel close 2022-07-03T13:14:36.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=679] Channel closed 2022-07-03T13:14:36.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=679] Received exit status 0 2022-07-03T13:14:36.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=679] Received channel close 2022-07-03T13:14:36.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=679] Channel closed 2022-07-03T13:14:36.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=235] Login timeout expired 2022-07-03T13:14:36.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=235] Aborting connection 2022-07-03T13:14:36.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:14:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:36 smithi161 ceph-mon[91579]: pgmap v4666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:36 smithi188 ceph-mon[92425]: pgmap v4666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:14:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:14:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:14:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:14:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:14:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:14:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:14:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:14:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:14:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:14:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:14:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:14:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:14:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:14:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:14:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:14:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:14:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:14:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:14:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:14:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:14:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:14:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:14:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:14:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:14:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:14:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:14:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:14:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:14:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:14:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:14:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:14:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:14:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:14:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:14:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:14:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:14:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:14:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:14:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:14:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:14:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:14:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:14:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:14:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:14:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:14:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=235] Local address: 172.21.15.161, port 53518 2022-07-03T13:14:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=235] Peer address: 172.21.15.101, port 22 2022-07-03T13:14:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=676] Received exit status 0 2022-07-03T13:14:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=676] Received channel close 2022-07-03T13:14:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=676] Channel closed 2022-07-03T13:14:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=677] Requesting new SSH session 2022-07-03T13:14:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=677] Command: which python3 2022-07-03T13:14:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=676] Received exit status 0 2022-07-03T13:14:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=676] Received channel close 2022-07-03T13:14:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=676] Channel closed 2022-07-03T13:14:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=677] Requesting new SSH session 2022-07-03T13:14:38.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=677] Command: which python3 2022-07-03T13:14:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=677] Received exit status 0 2022-07-03T13:14:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=677] Received channel close 2022-07-03T13:14:38.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=677] Channel closed 2022-07-03T13:14:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=678] Requesting new SSH session 2022-07-03T13:14:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=678] Command: sudo true 2022-07-03T13:14:38.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=677] Received exit status 0 2022-07-03T13:14:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=677] Received channel close 2022-07-03T13:14:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=677] Channel closed 2022-07-03T13:14:38.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=678] Requesting new SSH session 2022-07-03T13:14:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=678] Command: sudo true 2022-07-03T13:14:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=678] Received exit status 0 2022-07-03T13:14:38.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=678] Received channel close 2022-07-03T13:14:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=678] Channel closed 2022-07-03T13:14:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=679] Requesting new SSH session 2022-07-03T13:14:38.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=678] Received exit status 0 2022-07-03T13:14:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=678] Received channel close 2022-07-03T13:14:38.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=678] Channel closed 2022-07-03T13:14:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=679] Requesting new SSH session 2022-07-03T13:14:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:38.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=679] Received exit status 0 2022-07-03T13:14:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=679] Received channel close 2022-07-03T13:14:38.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=679] Channel closed 2022-07-03T13:14:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=679] Received exit status 0 2022-07-03T13:14:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=679] Received channel close 2022-07-03T13:14:38.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=679] Channel closed 2022-07-03T13:14:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=235] Login timeout expired 2022-07-03T13:14:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=235] Aborting connection 2022-07-03T13:14:38.158 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:14:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:14:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:14:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:14:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:14:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:14:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:14:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:14:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:14:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:14:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:14:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:14:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:14:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:14:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:14:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:14:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:14:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:14:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:14:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:14:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:14:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:14:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:14:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:14:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:14:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:14:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:14:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:14:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:14:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:14:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:14:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:14:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:14:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:14:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:14:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:14:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:14:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:14:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:14:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:14:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:14:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:14:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:14:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:14:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:14:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:14:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=235] Local address: 172.21.15.161, port 53518 2022-07-03T13:14:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=235] Peer address: 172.21.15.101, port 22 2022-07-03T13:14:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=676] Received exit status 0 2022-07-03T13:14:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=676] Received channel close 2022-07-03T13:14:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=676] Channel closed 2022-07-03T13:14:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=677] Requesting new SSH session 2022-07-03T13:14:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=677] Command: which python3 2022-07-03T13:14:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=676] Received exit status 0 2022-07-03T13:14:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=676] Received channel close 2022-07-03T13:14:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=676] Channel closed 2022-07-03T13:14:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=677] Requesting new SSH session 2022-07-03T13:14:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=677] Command: which python3 2022-07-03T13:14:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=677] Received exit status 0 2022-07-03T13:14:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=677] Received channel close 2022-07-03T13:14:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=677] Channel closed 2022-07-03T13:14:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=678] Requesting new SSH session 2022-07-03T13:14:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=678] Command: sudo true 2022-07-03T13:14:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=677] Received exit status 0 2022-07-03T13:14:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=677] Received channel close 2022-07-03T13:14:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=677] Channel closed 2022-07-03T13:14:38.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=678] Requesting new SSH session 2022-07-03T13:14:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=678] Command: sudo true 2022-07-03T13:14:38.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=678] Received exit status 0 2022-07-03T13:14:38.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=678] Received channel close 2022-07-03T13:14:38.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=678] Channel closed 2022-07-03T13:14:38.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=679] Requesting new SSH session 2022-07-03T13:14:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=678] Received exit status 0 2022-07-03T13:14:38.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=678] Received channel close 2022-07-03T13:14:38.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=678] Channel closed 2022-07-03T13:14:38.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=679] Requesting new SSH session 2022-07-03T13:14:38.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=679] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:14:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=679] Received exit status 0 2022-07-03T13:14:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=679] Received channel close 2022-07-03T13:14:38.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=679] Channel closed 2022-07-03T13:14:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=679] Received exit status 0 2022-07-03T13:14:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=679] Received channel close 2022-07-03T13:14:38.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=679] Channel closed 2022-07-03T13:14:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=235] Login timeout expired 2022-07-03T13:14:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=235] Aborting connection 2022-07-03T13:14:38.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:38 smithi161 ceph-mon[91579]: pgmap v4667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:38 smithi188 ceph-mon[92425]: pgmap v4667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:40 smithi161 ceph-mon[91579]: pgmap v4668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:40 smithi188 ceph-mon[92425]: pgmap v4668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:42 smithi161 ceph-mon[91579]: pgmap v4669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:42 smithi188 ceph-mon[92425]: pgmap v4669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:44 smithi161 ceph-mon[91579]: pgmap v4670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:44 smithi188 ceph-mon[92425]: pgmap v4670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:46 smithi161 ceph-mon[91579]: pgmap v4671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:46 smithi188 ceph-mon[92425]: pgmap v4671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:48 smithi161 ceph-mon[91579]: pgmap v4672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:48 smithi188 ceph-mon[92425]: pgmap v4672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:50 smithi161 ceph-mon[91579]: pgmap v4673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:50 smithi188 ceph-mon[92425]: pgmap v4673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:52 smithi161 ceph-mon[91579]: pgmap v4674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:52 smithi188 ceph-mon[92425]: pgmap v4674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:54 smithi161 ceph-mon[91579]: pgmap v4675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:54 smithi188 ceph-mon[92425]: pgmap v4675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:14:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:56 smithi161 ceph-mon[91579]: pgmap v4676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:56 smithi188 ceph-mon[92425]: pgmap v4676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:14:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:14:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:14:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:14:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:14:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:14:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:14:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:14:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:14:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:14:58 smithi161 ceph-mon[91579]: pgmap v4677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:14:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:14:58 smithi188 ceph-mon[92425]: pgmap v4677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:00 smithi161 ceph-mon[91579]: pgmap v4678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:00 smithi188 ceph-mon[92425]: pgmap v4678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:02 smithi161 ceph-mon[91579]: pgmap v4679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:02 smithi188 ceph-mon[92425]: pgmap v4679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:04 smithi161 ceph-mon[91579]: pgmap v4680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:04 smithi188 ceph-mon[92425]: pgmap v4680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:06 smithi161 ceph-mon[91579]: pgmap v4681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:06 smithi188 ceph-mon[92425]: pgmap v4681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:08 smithi161 ceph-mon[91579]: pgmap v4682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:08 smithi188 ceph-mon[92425]: pgmap v4682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:10 smithi161 ceph-mon[91579]: pgmap v4683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:10 smithi188 ceph-mon[92425]: pgmap v4683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:12 smithi161 ceph-mon[91579]: pgmap v4684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:12 smithi188 ceph-mon[92425]: pgmap v4684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:14 smithi161 ceph-mon[91579]: pgmap v4685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:14 smithi188 ceph-mon[92425]: pgmap v4685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:16 smithi161 ceph-mon[91579]: pgmap v4686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:16 smithi188 ceph-mon[92425]: pgmap v4686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:18 smithi161 ceph-mon[91579]: pgmap v4687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:18 smithi188 ceph-mon[92425]: pgmap v4687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:20 smithi161 ceph-mon[91579]: pgmap v4688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:20 smithi188 ceph-mon[92425]: pgmap v4688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:22 smithi161 ceph-mon[91579]: pgmap v4689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:22 smithi188 ceph-mon[92425]: pgmap v4689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:24 smithi161 ceph-mon[91579]: pgmap v4690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:24 smithi188 ceph-mon[92425]: pgmap v4690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:26 smithi161 ceph-mon[91579]: pgmap v4691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:26 smithi188 ceph-mon[92425]: pgmap v4691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:28 smithi161 ceph-mon[91579]: pgmap v4692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:28 smithi188 ceph-mon[92425]: pgmap v4692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:30 smithi161 ceph-mon[91579]: pgmap v4693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:30 smithi188 ceph-mon[92425]: pgmap v4693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:32 smithi161 ceph-mon[91579]: pgmap v4694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:32 smithi188 ceph-mon[92425]: pgmap v4694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:34 smithi161 ceph-mon[91579]: pgmap v4695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:34 smithi188 ceph-mon[92425]: pgmap v4695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:36 smithi161 ceph-mon[91579]: pgmap v4696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:15:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:15:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:15:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:36 smithi188 ceph-mon[92425]: pgmap v4696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:15:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:15:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:15:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:38 smithi161 ceph-mon[91579]: pgmap v4697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:38 smithi188 ceph-mon[92425]: pgmap v4697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:40 smithi161 ceph-mon[91579]: pgmap v4698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:40 smithi188 ceph-mon[92425]: pgmap v4698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:42 smithi161 ceph-mon[91579]: pgmap v4699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:42 smithi188 ceph-mon[92425]: pgmap v4699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:44 smithi161 ceph-mon[91579]: pgmap v4700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:44 smithi188 ceph-mon[92425]: pgmap v4700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:46 smithi161 ceph-mon[91579]: pgmap v4701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:46 smithi188 ceph-mon[92425]: pgmap v4701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:48 smithi161 ceph-mon[91579]: pgmap v4702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:48 smithi188 ceph-mon[92425]: pgmap v4702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:50 smithi161 ceph-mon[91579]: pgmap v4703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:50 smithi188 ceph-mon[92425]: pgmap v4703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:52 smithi161 ceph-mon[91579]: pgmap v4704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:52 smithi188 ceph-mon[92425]: pgmap v4704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:54 smithi161 ceph-mon[91579]: pgmap v4705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:54 smithi188 ceph-mon[92425]: pgmap v4705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:15:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:56 smithi161 ceph-mon[91579]: pgmap v4706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:56 smithi188 ceph-mon[92425]: pgmap v4706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:15:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:15:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:15:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:15:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:15:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:15:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:15:58 smithi161 ceph-mon[91579]: pgmap v4707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:15:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:15:58 smithi188 ceph-mon[92425]: pgmap v4707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:00 smithi161 ceph-mon[91579]: pgmap v4708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:00 smithi188 ceph-mon[92425]: pgmap v4708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:02 smithi161 ceph-mon[91579]: pgmap v4709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:02 smithi188 ceph-mon[92425]: pgmap v4709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:04 smithi161 ceph-mon[91579]: pgmap v4710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:04 smithi188 ceph-mon[92425]: pgmap v4710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:06.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:16:06.452+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:16:06.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:16:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:16:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:16:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:16:06.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:16:06.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:16:06.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:16:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:16:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:16:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:16:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:16:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:16:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:16:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:16:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:16:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:16:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:16:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:16:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:16:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:16:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:16:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:16:06.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:16:06.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:16:06.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:16:06.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:16:06.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:16:06.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:16:06.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:16:06.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:16:06.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:16:06.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:16:06.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:16:06.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:16:06.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:16:06.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:16:06.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:16:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:16:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:16:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:16:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:16:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:16:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:16:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:16:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:16:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=237] Local address: 172.21.15.161, port 53522 2022-07-03T13:16:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=237] Peer address: 172.21.15.101, port 22 2022-07-03T13:16:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=237] Connection lost 2022-07-03T13:16:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=237] Aborting connection 2022-07-03T13:16:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:16:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:06 smithi161 ceph-mon[91579]: pgmap v4711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:06 smithi188 ceph-mon[92425]: pgmap v4711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:16:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:16:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:16:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:16:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:16:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:16:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:16:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:16:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:16:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:16:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:16:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:16:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:16:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:16:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:16:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:16:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:16:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:16:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:16:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:16:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:16:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:16:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:16:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:16:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:16:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:16:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:16:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:16:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:16:08.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:16:08.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:16:08.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:16:08.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:16:08.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:16:08.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:16:08.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:16:08.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:16:08.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:16:08.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:16:08.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:16:08.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:16:08.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:16:08.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:16:08.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:16:08.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:16:08.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:16:08.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:16:08.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=237] Local address: 172.21.15.161, port 53522 2022-07-03T13:16:08.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=237] Peer address: 172.21.15.101, port 22 2022-07-03T13:16:08.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=237] Connection lost 2022-07-03T13:16:08.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=237] Aborting connection 2022-07-03T13:16:08.162 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:16:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:16:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:16:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:16:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:16:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:16:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:16:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:16:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:16:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:16:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:16:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:16:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:16:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:16:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:16:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:16:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:16:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:16:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:16:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:16:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:16:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:16:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:16:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:16:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:16:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:16:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:16:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:16:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:16:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:16:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:16:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:16:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:16:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:16:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:16:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:16:08.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:16:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:16:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:16:08.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:16:08.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:16:08.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:16:08.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:16:08.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:16:08.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:16:08.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:16:08.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:16:08.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=237] Local address: 172.21.15.161, port 53522 2022-07-03T13:16:08.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=237] Peer address: 172.21.15.101, port 22 2022-07-03T13:16:08.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=237] Connection lost 2022-07-03T13:16:08.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=237] Aborting connection 2022-07-03T13:16:08.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:08 smithi161 ceph-mon[91579]: pgmap v4712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:08 smithi188 ceph-mon[92425]: pgmap v4712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:10 smithi161 ceph-mon[91579]: pgmap v4713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:10 smithi188 ceph-mon[92425]: pgmap v4713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:12 smithi161 ceph-mon[91579]: pgmap v4714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:12 smithi188 ceph-mon[92425]: pgmap v4714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:14 smithi188 ceph-mon[92425]: pgmap v4715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:14 smithi161 ceph-mon[91579]: pgmap v4715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:16 smithi188 ceph-mon[92425]: pgmap v4716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:16 smithi161 ceph-mon[91579]: pgmap v4716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:18 smithi188 ceph-mon[92425]: pgmap v4717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:18 smithi161 ceph-mon[91579]: pgmap v4717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:20 smithi188 ceph-mon[92425]: pgmap v4718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:20 smithi161 ceph-mon[91579]: pgmap v4718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:22 smithi188 ceph-mon[92425]: pgmap v4719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:22 smithi161 ceph-mon[91579]: pgmap v4719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:24 smithi188 ceph-mon[92425]: pgmap v4720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:24 smithi161 ceph-mon[91579]: pgmap v4720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:26 smithi188 ceph-mon[92425]: pgmap v4721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:26 smithi161 ceph-mon[91579]: pgmap v4721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:28 smithi188 ceph-mon[92425]: pgmap v4722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:28 smithi161 ceph-mon[91579]: pgmap v4722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:30 smithi161 ceph-mon[91579]: pgmap v4723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:30 smithi188 ceph-mon[92425]: pgmap v4723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:32 smithi161 ceph-mon[91579]: pgmap v4724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:32 smithi188 ceph-mon[92425]: pgmap v4724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:34 smithi161 ceph-mon[91579]: pgmap v4725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:34 smithi188 ceph-mon[92425]: pgmap v4725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:36 smithi161 ceph-mon[91579]: pgmap v4726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:37.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:36 smithi188 ceph-mon[92425]: pgmap v4726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:38 smithi161 ceph-mon[91579]: pgmap v4727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:38 smithi188 ceph-mon[92425]: pgmap v4727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:40 smithi161 ceph-mon[91579]: pgmap v4728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:40 smithi188 ceph-mon[92425]: pgmap v4728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:42 smithi161 ceph-mon[91579]: pgmap v4729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:42 smithi188 ceph-mon[92425]: pgmap v4729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:44 smithi161 ceph-mon[91579]: pgmap v4730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:44 smithi188 ceph-mon[92425]: pgmap v4730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:46 smithi161 ceph-mon[91579]: pgmap v4731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:46 smithi188 ceph-mon[92425]: pgmap v4731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:48 smithi161 ceph-mon[91579]: pgmap v4732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:48 smithi188 ceph-mon[92425]: pgmap v4732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:50 smithi161 ceph-mon[91579]: pgmap v4733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:50 smithi188 ceph-mon[92425]: pgmap v4733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:53 smithi161 ceph-mon[91579]: pgmap v4734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:53 smithi188 ceph-mon[92425]: pgmap v4734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:55 smithi161 ceph-mon[91579]: pgmap v4735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:55 smithi188 ceph-mon[92425]: pgmap v4735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:16:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:57 smithi161 ceph-mon[91579]: pgmap v4736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:57 smithi188 ceph-mon[92425]: pgmap v4736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:16:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:16:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:16:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:16:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:16:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:16:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:16:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:16:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:16:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:16:59 smithi161 ceph-mon[91579]: pgmap v4737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:16:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:16:59 smithi188 ceph-mon[92425]: pgmap v4737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:01 smithi161 ceph-mon[91579]: pgmap v4738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:01 smithi188 ceph-mon[92425]: pgmap v4738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:03 smithi161 ceph-mon[91579]: pgmap v4739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:03 smithi188 ceph-mon[92425]: pgmap v4739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:05 smithi161 ceph-mon[91579]: pgmap v4740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:05 smithi188 ceph-mon[92425]: pgmap v4740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:17:06.486+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:17:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:17:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:17:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:17:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:17:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:17:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:17:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:17:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:17:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:17:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:17:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:17:06.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:17:06.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:17:06.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:17:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:17:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:17:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:17:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:17:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:17:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:17:06.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:17:06.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:17:06.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:17:06.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:17:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:17:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:17:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:17:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:17:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:17:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:17:06.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:17:06.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:17:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:17:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:17:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:17:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:17:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:17:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:17:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:17:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:17:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:17:06.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:17:06.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:17:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:17:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:17:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=239] Local address: 172.21.15.161, port 53526 2022-07-03T13:17:06.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=239] Peer address: 172.21.15.101, port 22 2022-07-03T13:17:06.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=239] Connection lost 2022-07-03T13:17:06.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=239] Aborting connection 2022-07-03T13:17:06.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:17:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:17:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:07 smithi161 ceph-mon[91579]: pgmap v4741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:17:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:17:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:17:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:07 smithi188 ceph-mon[92425]: pgmap v4741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:17:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:17:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:17:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:17:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:17:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:17:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:17:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:17:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:17:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:17:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:17:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:17:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:17:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:17:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:17:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:17:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:17:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:17:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:17:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:17:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:17:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:17:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:17:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:17:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:17:08.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:17:08.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:17:08.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:17:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:17:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:17:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:17:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:17:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:17:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:17:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:17:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:17:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:17:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:17:08.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:17:08.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:17:08.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:17:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:17:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:17:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:17:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:17:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:17:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:17:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:17:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:17:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:17:08.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=239] Local address: 172.21.15.161, port 53526 2022-07-03T13:17:08.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=239] Peer address: 172.21.15.101, port 22 2022-07-03T13:17:08.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=239] Connection lost 2022-07-03T13:17:08.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=239] Aborting connection 2022-07-03T13:17:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:17:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:17:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:17:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:17:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:17:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:17:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:17:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:17:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:17:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:17:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:17:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:17:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:17:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:17:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:17:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:17:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:17:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:17:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:17:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:17:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:17:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:17:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:17:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:17:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:17:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:17:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:17:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:17:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:17:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:17:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:17:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:17:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:17:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:17:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:17:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:17:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:17:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:17:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:17:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:17:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:17:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:17:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:17:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:17:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:17:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:17:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:17:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=239] Local address: 172.21.15.161, port 53526 2022-07-03T13:17:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=239] Peer address: 172.21.15.101, port 22 2022-07-03T13:17:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=239] Connection lost 2022-07-03T13:17:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=239] Aborting connection 2022-07-03T13:17:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:09 smithi161 ceph-mon[91579]: pgmap v4742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:09 smithi188 ceph-mon[92425]: pgmap v4742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:11 smithi161 ceph-mon[91579]: pgmap v4743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:11 smithi188 ceph-mon[92425]: pgmap v4743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:13 smithi161 ceph-mon[91579]: pgmap v4744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:13 smithi188 ceph-mon[92425]: pgmap v4744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:15 smithi161 ceph-mon[91579]: pgmap v4745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:15 smithi188 ceph-mon[92425]: pgmap v4745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:17 smithi161 ceph-mon[91579]: pgmap v4746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:17 smithi188 ceph-mon[92425]: pgmap v4746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:19 smithi161 ceph-mon[91579]: pgmap v4747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:19 smithi188 ceph-mon[92425]: pgmap v4747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:21 smithi161 ceph-mon[91579]: pgmap v4748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:21 smithi188 ceph-mon[92425]: pgmap v4748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:23 smithi161 ceph-mon[91579]: pgmap v4749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:23 smithi188 ceph-mon[92425]: pgmap v4749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:25 smithi161 ceph-mon[91579]: pgmap v4750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:25 smithi188 ceph-mon[92425]: pgmap v4750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:27 smithi161 ceph-mon[91579]: pgmap v4751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:27 smithi188 ceph-mon[92425]: pgmap v4751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:29 smithi161 ceph-mon[91579]: pgmap v4752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:29 smithi188 ceph-mon[92425]: pgmap v4752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:31 smithi161 ceph-mon[91579]: pgmap v4753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:31 smithi188 ceph-mon[92425]: pgmap v4753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:33 smithi161 ceph-mon[91579]: pgmap v4754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:33 smithi188 ceph-mon[92425]: pgmap v4754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:35 smithi161 ceph-mon[91579]: pgmap v4755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:35 smithi188 ceph-mon[92425]: pgmap v4755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:37 smithi161 ceph-mon[91579]: pgmap v4756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:37 smithi188 ceph-mon[92425]: pgmap v4756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:39 smithi161 ceph-mon[91579]: pgmap v4757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:39 smithi188 ceph-mon[92425]: pgmap v4757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:41 smithi161 ceph-mon[91579]: pgmap v4758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:41 smithi188 ceph-mon[92425]: pgmap v4758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:43 smithi161 ceph-mon[91579]: pgmap v4759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:43 smithi188 ceph-mon[92425]: pgmap v4759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:45 smithi161 ceph-mon[91579]: pgmap v4760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:45 smithi188 ceph-mon[92425]: pgmap v4760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:47 smithi161 ceph-mon[91579]: pgmap v4761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:47 smithi188 ceph-mon[92425]: pgmap v4761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:49 smithi161 ceph-mon[91579]: pgmap v4762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:49 smithi188 ceph-mon[92425]: pgmap v4762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:51 smithi161 ceph-mon[91579]: pgmap v4763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:51 smithi188 ceph-mon[92425]: pgmap v4763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:53 smithi188 ceph-mon[92425]: pgmap v4764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:53 smithi161 ceph-mon[91579]: pgmap v4764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:55 smithi188 ceph-mon[92425]: pgmap v4765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:55 smithi161 ceph-mon[91579]: pgmap v4765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:17:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:57 smithi188 ceph-mon[92425]: pgmap v4766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:57 smithi161 ceph-mon[91579]: pgmap v4766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:17:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:17:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:17:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:17:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:17:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:17:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:17:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:17:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:17:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:17:59 smithi188 ceph-mon[92425]: pgmap v4767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:17:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:17:59 smithi161 ceph-mon[91579]: pgmap v4767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:01 smithi188 ceph-mon[92425]: pgmap v4768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:01 smithi161 ceph-mon[91579]: pgmap v4768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:03 smithi188 ceph-mon[92425]: pgmap v4769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:03 smithi161 ceph-mon[91579]: pgmap v4769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:05 smithi188 ceph-mon[92425]: pgmap v4770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:05 smithi161 ceph-mon[91579]: pgmap v4770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:07.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:07 smithi188 ceph-mon[92425]: pgmap v4771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:07.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:18:07.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:18:07.348 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:18:07.349 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:07 smithi161 ceph-mon[91579]: pgmap v4771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:18:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:18:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:18:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:18:06.992+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:18:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:18:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:18:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:18:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:18:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:18:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:18:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:18:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:18:07.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:18:07.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:18:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:18:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:18:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:18:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:18:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:18:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:18:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:18:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:18:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:18:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:18:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:18:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:18:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:18:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:18:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:18:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:18:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:18:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:18:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:18:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:18:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:18:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:18:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:18:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:18:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:18:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:18:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:18:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:18:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:18:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:18:07.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:18:07.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:18:07.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:18:07.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:18:07.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:18:07.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=241] Local address: 172.21.15.161, port 53530 2022-07-03T13:18:07.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=241] Peer address: 172.21.15.101, port 22 2022-07-03T13:18:07.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=241] Connection lost 2022-07-03T13:18:07.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=241] Aborting connection 2022-07-03T13:18:07.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:18:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:18:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:18:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:18:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:18:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:18:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:18:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:18:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:18:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:18:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:18:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:18:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:18:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:18:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:18:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:18:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:18:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:18:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:18:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:18:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:18:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:18:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:18:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:18:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:18:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:18:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:18:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:18:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:18:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:18:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:18:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:18:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:18:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:18:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:18:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:18:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:18:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:18:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:18:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:18:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:18:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:18:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:18:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:18:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:18:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:18:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:18:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:18:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:18:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=241] Local address: 172.21.15.161, port 53530 2022-07-03T13:18:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=241] Peer address: 172.21.15.101, port 22 2022-07-03T13:18:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=241] Connection lost 2022-07-03T13:18:08.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=241] Aborting connection 2022-07-03T13:18:08.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:18:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:18:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:18:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:18:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:18:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:18:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:18:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:18:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:18:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:18:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:18:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:18:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:18:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:18:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:18:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:18:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:18:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:18:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:18:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:18:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:18:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:18:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:18:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:18:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:18:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:18:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:18:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:18:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:18:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:18:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:18:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:18:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:18:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:18:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:18:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:18:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:18:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:18:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:18:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:18:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:18:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:18:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:18:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:18:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:18:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:18:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:18:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=241] Local address: 172.21.15.161, port 53530 2022-07-03T13:18:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=241] Peer address: 172.21.15.101, port 22 2022-07-03T13:18:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=241] Connection lost 2022-07-03T13:18:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=241] Aborting connection 2022-07-03T13:18:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:18:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:09 smithi188 ceph-mon[92425]: pgmap v4772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:09 smithi161 ceph-mon[91579]: pgmap v4772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:11 smithi161 ceph-mon[91579]: pgmap v4773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:11 smithi188 ceph-mon[92425]: pgmap v4773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:13 smithi161 ceph-mon[91579]: pgmap v4774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:13 smithi188 ceph-mon[92425]: pgmap v4774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:15 smithi161 ceph-mon[91579]: pgmap v4775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:15 smithi188 ceph-mon[92425]: pgmap v4775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:17 smithi161 ceph-mon[91579]: pgmap v4776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:17 smithi188 ceph-mon[92425]: pgmap v4776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:19 smithi161 ceph-mon[91579]: pgmap v4777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:19 smithi188 ceph-mon[92425]: pgmap v4777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:21 smithi161 ceph-mon[91579]: pgmap v4778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:21 smithi188 ceph-mon[92425]: pgmap v4778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:23 smithi161 ceph-mon[91579]: pgmap v4779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:23 smithi188 ceph-mon[92425]: pgmap v4779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:25 smithi161 ceph-mon[91579]: pgmap v4780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:25 smithi188 ceph-mon[92425]: pgmap v4780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:27 smithi161 ceph-mon[91579]: pgmap v4781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:27 smithi188 ceph-mon[92425]: pgmap v4781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:29 smithi161 ceph-mon[91579]: pgmap v4782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:29 smithi188 ceph-mon[92425]: pgmap v4782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:31 smithi161 ceph-mon[91579]: pgmap v4783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:31 smithi188 ceph-mon[92425]: pgmap v4783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:33 smithi161 ceph-mon[91579]: pgmap v4784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:33 smithi188 ceph-mon[92425]: pgmap v4784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:35 smithi161 ceph-mon[91579]: pgmap v4785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:35 smithi188 ceph-mon[92425]: pgmap v4785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:37 smithi161 ceph-mon[91579]: pgmap v4786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:37 smithi188 ceph-mon[92425]: pgmap v4786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:39 smithi161 ceph-mon[91579]: pgmap v4787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:39 smithi188 ceph-mon[92425]: pgmap v4787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:41 smithi161 ceph-mon[91579]: pgmap v4788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:41 smithi188 ceph-mon[92425]: pgmap v4788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:42 smithi161 ceph-mon[91579]: pgmap v4789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:42 smithi188 ceph-mon[92425]: pgmap v4789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:44 smithi161 ceph-mon[91579]: pgmap v4790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:44 smithi188 ceph-mon[92425]: pgmap v4790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:46 smithi161 ceph-mon[91579]: pgmap v4791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:46 smithi188 ceph-mon[92425]: pgmap v4791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:48 smithi161 ceph-mon[91579]: pgmap v4792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:48 smithi188 ceph-mon[92425]: pgmap v4792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:50 smithi161 ceph-mon[91579]: pgmap v4793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:50 smithi188 ceph-mon[92425]: pgmap v4793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:52 smithi161 ceph-mon[91579]: pgmap v4794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:52 smithi188 ceph-mon[92425]: pgmap v4794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:54 smithi161 ceph-mon[91579]: pgmap v4795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:54 smithi188 ceph-mon[92425]: pgmap v4795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:18:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:56 smithi161 ceph-mon[91579]: pgmap v4796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:56 smithi188 ceph-mon[92425]: pgmap v4796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:18:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:18:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:18:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:18:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:18:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:18:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:18:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:18:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:18:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:18:58 smithi161 ceph-mon[91579]: pgmap v4797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:18:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:18:58 smithi188 ceph-mon[92425]: pgmap v4797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:00 smithi161 ceph-mon[91579]: pgmap v4798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:00 smithi188 ceph-mon[92425]: pgmap v4798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:02 smithi161 ceph-mon[91579]: pgmap v4799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:02 smithi188 ceph-mon[92425]: pgmap v4799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:04 smithi161 ceph-mon[91579]: pgmap v4800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:04 smithi188 ceph-mon[92425]: pgmap v4800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:06 smithi161 ceph-mon[91579]: pgmap v4801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:06 smithi188 ceph-mon[92425]: pgmap v4801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:08.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:19:08.337+0000 7f726b85f700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:08.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:19:08.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:08.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:19:08.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:08.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:08.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:08.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:19:08.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:08.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:19:08.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:08.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:19:08.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:08.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T13:19:08.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:08.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:19:08.338+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:19:08.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:19:08.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:08.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:19:08.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:08.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:08.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:08.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:19:08.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:08.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:19:08.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:08.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:19:08.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:08.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: data = self._sock.recv(self.max_size) 2022-07-03T13:19:08.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:08.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:19:08.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:19:08.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:19:08.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:19:08.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:19:08.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:19:08.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:19:08.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:19:08.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:19:08.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:19:08.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:19:08.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:19:08.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:19:08.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:19:08.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:19:08.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:19:08.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:19:08.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:19:08.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:19:08.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:19:08.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:19:08.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:19:08.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:19:08.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:19:08.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:19:08.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:19:08.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:19:08.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:19:08.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T13:19:08.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:19:08.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-mon[91579]: pgmap v4802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:19:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:19:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:19:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:19:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:08 smithi188 ceph-mon[92425]: pgmap v4802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:19:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:19:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:19:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:19:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:09 smithi161 ceph-mon[91579]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:19:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:19:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:19:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:19:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T13:19:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:19:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:19:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:19:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:19:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:19:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: data = self._sock.recv(self.max_size) 2022-07-03T13:19:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:19:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:19:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:19:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:19:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:19:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:19:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:19:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:19:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:19:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:19:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:19:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:19:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:19:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:19:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:19:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:19:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:19:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:19:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:19:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:19:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:19:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:19:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:19:10.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:19:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:19:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:19:10.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:19:10.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T13:19:10.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:19:10.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:09 smithi188 ceph-mon[92425]: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:19:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:19:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:19:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:19:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T13:19:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:19:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:19:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:19:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:19:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:19:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:19:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:19:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:19:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:19:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:19:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:19:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 714, in _read_ready 2022-07-03T13:19:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: data = self._sock.recv(self.max_size) 2022-07-03T13:19:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: ConnectionResetError: [Errno 104] Connection reset by peer 2022-07-03T13:19:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:19:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:19:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:19:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:19:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:19:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:19:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:19:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:19:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:19:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:19:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:19:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:19:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:19:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:19:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:19:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:19:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:19:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:19:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:19:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:19:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:19:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:19:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:19:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:19:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:19:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:19:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:19:10.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:19:10.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-07-03T13:19:10.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:19:10.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.101`, possibly because python3 is not installed there. [Errno 104] Connection reset by peer 2022-07-03T13:19:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:10 smithi161 ceph-mon[91579]: pgmap v4803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:10 smithi188 ceph-mon[92425]: pgmap v4803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:12 smithi161 ceph-mon[91579]: pgmap v4804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:12 smithi188 ceph-mon[92425]: pgmap v4804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:14 smithi161 ceph-mon[91579]: pgmap v4805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:14 smithi188 ceph-mon[92425]: pgmap v4805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:16 smithi161 ceph-mon[91579]: pgmap v4806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:16 smithi188 ceph-mon[92425]: pgmap v4806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:18 smithi161 ceph-mon[91579]: pgmap v4807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:18 smithi188 ceph-mon[92425]: pgmap v4807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:20 smithi161 ceph-mon[91579]: pgmap v4808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:20 smithi188 ceph-mon[92425]: pgmap v4808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:22 smithi161 ceph-mon[91579]: pgmap v4809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:22 smithi188 ceph-mon[92425]: pgmap v4809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:24 smithi161 ceph-mon[91579]: pgmap v4810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:24 smithi188 ceph-mon[92425]: pgmap v4810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:26 smithi161 ceph-mon[91579]: pgmap v4811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:26 smithi188 ceph-mon[92425]: pgmap v4811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:28 smithi161 ceph-mon[91579]: pgmap v4812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:28 smithi188 ceph-mon[92425]: pgmap v4812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:30 smithi161 ceph-mon[91579]: pgmap v4813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:30 smithi188 ceph-mon[92425]: pgmap v4813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:32 smithi161 ceph-mon[91579]: pgmap v4814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:32 smithi188 ceph-mon[92425]: pgmap v4814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:34 smithi161 ceph-mon[91579]: pgmap v4815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:34 smithi188 ceph-mon[92425]: pgmap v4815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:36 smithi161 ceph-mon[91579]: pgmap v4816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:36 smithi188 ceph-mon[92425]: pgmap v4816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:38 smithi161 ceph-mon[91579]: pgmap v4817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:38 smithi188 ceph-mon[92425]: pgmap v4817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:40 smithi161 ceph-mon[91579]: pgmap v4818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:40 smithi188 ceph-mon[92425]: pgmap v4818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:42 smithi161 ceph-mon[91579]: pgmap v4819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:42 smithi188 ceph-mon[92425]: pgmap v4819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:44 smithi161 ceph-mon[91579]: pgmap v4820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:44 smithi188 ceph-mon[92425]: pgmap v4820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:46 smithi161 ceph-mon[91579]: pgmap v4821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:46 smithi188 ceph-mon[92425]: pgmap v4821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:48 smithi161 ceph-mon[91579]: pgmap v4822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:48 smithi188 ceph-mon[92425]: pgmap v4822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:50 smithi161 ceph-mon[91579]: pgmap v4823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:50 smithi188 ceph-mon[92425]: pgmap v4823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:52 smithi161 ceph-mon[91579]: pgmap v4824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:52 smithi188 ceph-mon[92425]: pgmap v4824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:54 smithi161 ceph-mon[91579]: pgmap v4825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:54 smithi188 ceph-mon[92425]: pgmap v4825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:19:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:56 smithi161 ceph-mon[91579]: pgmap v4826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:56 smithi188 ceph-mon[92425]: pgmap v4826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:19:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:19:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:19:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:19:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:19:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:19:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:19:58 smithi161 ceph-mon[91579]: pgmap v4827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:19:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:19:58 smithi188 ceph-mon[92425]: pgmap v4827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:00 smithi188 ceph-mon[92425]: pgmap v4828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:20:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:00 smithi161 ceph-mon[91579]: pgmap v4828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:20:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:02 smithi188 ceph-mon[92425]: pgmap v4829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:02 smithi161 ceph-mon[91579]: pgmap v4829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:04 smithi188 ceph-mon[92425]: pgmap v4830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:04 smithi161 ceph-mon[91579]: pgmap v4830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:06 smithi188 ceph-mon[92425]: pgmap v4831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:06 smithi161 ceph-mon[91579]: pgmap v4831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:08.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:20:08.417+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:20:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:20:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:20:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:20:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:20:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:20:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:20:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:20:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:20:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:20:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:20:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:20:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:20:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:20:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:20:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:20:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:20:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:20:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:20:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:20:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:20:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:20:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:20:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:20:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:20:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:20:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:20:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:20:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:20:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:20:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:20:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:20:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:20:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:20:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:20:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:20:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:20:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:20:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:20:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:20:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:20:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:20:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:20:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:20:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:20:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:20:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=245] Local address: 172.21.15.161, port 53538 2022-07-03T13:20:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=245] Peer address: 172.21.15.101, port 22 2022-07-03T13:20:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=245] Connection lost 2022-07-03T13:20:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=245] Aborting connection 2022-07-03T13:20:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:20:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:08 smithi188 ceph-mon[92425]: pgmap v4832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:20:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:20:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:20:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-mon[91579]: pgmap v4832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:20:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:20:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:20:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:20:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:20:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:20:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:20:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:20:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:20:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:20:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:20:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:20:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:20:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:20:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:20:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:20:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:20:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:20:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:20:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:20:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:20:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:20:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:20:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:20:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:20:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:20:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:20:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:20:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:20:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:20:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:20:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:20:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:20:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:20:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:20:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:20:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:20:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:20:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:20:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:20:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:20:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:20:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:20:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:20:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:20:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:20:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:20:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:20:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:20:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:20:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=245] Local address: 172.21.15.161, port 53538 2022-07-03T13:20:10.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=245] Peer address: 172.21.15.101, port 22 2022-07-03T13:20:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=245] Connection lost 2022-07-03T13:20:10.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=245] Aborting connection 2022-07-03T13:20:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:20:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:20:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:20:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:20:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:20:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:20:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:20:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:20:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:20:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:20:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:20:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:20:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:20:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:20:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:20:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:20:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:20:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:20:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:20:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:20:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:20:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:20:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:20:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:20:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:20:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:20:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:20:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:20:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:20:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:20:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:20:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:20:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:20:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:20:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:20:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:20:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:20:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:20:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:20:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:20:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:20:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:20:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:20:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:20:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:20:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:20:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:20:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=245] Local address: 172.21.15.161, port 53538 2022-07-03T13:20:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=245] Peer address: 172.21.15.101, port 22 2022-07-03T13:20:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=245] Connection lost 2022-07-03T13:20:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=245] Aborting connection 2022-07-03T13:20:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:10 smithi188 ceph-mon[92425]: pgmap v4833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:11.203 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:10 smithi161 ceph-mon[91579]: pgmap v4833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:20:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:20:12.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:13 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:20:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:13 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:20:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:13 smithi161 ceph-mon[91579]: pgmap v4834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:13 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:20:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:13 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:20:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:13 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:20:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:13 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:20:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:13 smithi188 ceph-mon[92425]: pgmap v4834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:13 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:20:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:13 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:20:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:15 smithi161 ceph-mon[91579]: pgmap v4835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:15 smithi188 ceph-mon[92425]: pgmap v4835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:17 smithi161 ceph-mon[91579]: pgmap v4836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:17 smithi188 ceph-mon[92425]: pgmap v4836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:19 smithi161 ceph-mon[91579]: pgmap v4837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:19 smithi188 ceph-mon[92425]: pgmap v4837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:21 smithi161 ceph-mon[91579]: pgmap v4838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:21 smithi188 ceph-mon[92425]: pgmap v4838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:23 smithi161 ceph-mon[91579]: pgmap v4839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:23 smithi188 ceph-mon[92425]: pgmap v4839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:25 smithi161 ceph-mon[91579]: pgmap v4840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:25 smithi188 ceph-mon[92425]: pgmap v4840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:27 smithi161 ceph-mon[91579]: pgmap v4841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:27 smithi188 ceph-mon[92425]: pgmap v4841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:29 smithi161 ceph-mon[91579]: pgmap v4842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:29 smithi188 ceph-mon[92425]: pgmap v4842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:31 smithi161 ceph-mon[91579]: pgmap v4843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:31 smithi188 ceph-mon[92425]: pgmap v4843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:33 smithi161 ceph-mon[91579]: pgmap v4844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:33 smithi188 ceph-mon[92425]: pgmap v4844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:35 smithi161 ceph-mon[91579]: pgmap v4845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:35 smithi188 ceph-mon[92425]: pgmap v4845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:37 smithi161 ceph-mon[91579]: pgmap v4846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:37 smithi188 ceph-mon[92425]: pgmap v4846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:39 smithi161 ceph-mon[91579]: pgmap v4847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:39 smithi188 ceph-mon[92425]: pgmap v4847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:41 smithi161 ceph-mon[91579]: pgmap v4848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:41 smithi188 ceph-mon[92425]: pgmap v4848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:43 smithi161 ceph-mon[91579]: pgmap v4849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:43 smithi188 ceph-mon[92425]: pgmap v4849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:45 smithi161 ceph-mon[91579]: pgmap v4850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:45 smithi188 ceph-mon[92425]: pgmap v4850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:47 smithi161 ceph-mon[91579]: pgmap v4851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:47 smithi188 ceph-mon[92425]: pgmap v4851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:49 smithi161 ceph-mon[91579]: pgmap v4852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:49 smithi188 ceph-mon[92425]: pgmap v4852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:51 smithi161 ceph-mon[91579]: pgmap v4853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:51 smithi188 ceph-mon[92425]: pgmap v4853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:53 smithi161 ceph-mon[91579]: pgmap v4854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:53 smithi188 ceph-mon[92425]: pgmap v4854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:54 smithi161 ceph-mon[91579]: pgmap v4855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:54 smithi188 ceph-mon[92425]: pgmap v4855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:20:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:56 smithi161 ceph-mon[91579]: pgmap v4856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:56 smithi188 ceph-mon[92425]: pgmap v4856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:20:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:20:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:20:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:20:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:20:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:20:58 smithi161 ceph-mon[91579]: pgmap v4857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:20:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:20:58 smithi188 ceph-mon[92425]: pgmap v4857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:00 smithi161 ceph-mon[91579]: pgmap v4858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:00 smithi188 ceph-mon[92425]: pgmap v4858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:02 smithi161 ceph-mon[91579]: pgmap v4859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:02 smithi188 ceph-mon[92425]: pgmap v4859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:04 smithi161 ceph-mon[91579]: pgmap v4860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:04 smithi188 ceph-mon[92425]: pgmap v4860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:06 smithi161 ceph-mon[91579]: pgmap v4861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:06 smithi188 ceph-mon[92425]: pgmap v4861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:08 smithi161 ceph-mon[91579]: pgmap v4862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:08 smithi188 ceph-mon[92425]: pgmap v4862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:10 smithi161 ceph-mon[91579]: pgmap v4863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:10 smithi188 ceph-mon[92425]: pgmap v4863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:21:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:21:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:21:12.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:21:11.688+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:21:12.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:21:12.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:21:12.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:21:12.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:21:12.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:21:12.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:21:12.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:21:12.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:21:12.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:21:12.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:21:12.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:21:12.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:21:12.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:21:12.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:21:12.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:21:12.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:21:12.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:21:12.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:21:12.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:21:12.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:21:12.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:21:12.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:21:12.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:21:12.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:21:12.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:21:12.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:21:12.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:21:12.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:21:12.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:21:12.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:21:12.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:21:12.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:21:12.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:21:12.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:21:12.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:21:12.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:21:12.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:21:12.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:21:12.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:21:12.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:21:12.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:21:12.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:21:12.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:21:12.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:21:12.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=247] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:21:12.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=247] Local address: 172.21.15.161, port 53542 2022-07-03T13:21:12.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=247] Peer address: 172.21.15.101, port 22 2022-07-03T13:21:12.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=247] Connection lost 2022-07-03T13:21:12.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=247] Aborting connection 2022-07-03T13:21:12.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:21:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:21:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:21:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:21:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:21:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:12 smithi161 ceph-mon[91579]: pgmap v4864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:21:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:21:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:21:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:21:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:21:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:21:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:21:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:21:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:21:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:21:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:21:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:21:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:21:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:21:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:21:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:21:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:21:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:21:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:21:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:21:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:21:13.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:21:13.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:21:13.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:21:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:21:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:21:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:21:13.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:21:13.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:21:13.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:21:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:21:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:21:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:21:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:21:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:21:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:21:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:21:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:21:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:21:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:21:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:21:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:21:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:21:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:21:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:21:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:21:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=247] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:21:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=247] Local address: 172.21.15.161, port 53542 2022-07-03T13:21:13.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=247] Peer address: 172.21.15.101, port 22 2022-07-03T13:21:13.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=247] Connection lost 2022-07-03T13:21:13.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=247] Aborting connection 2022-07-03T13:21:13.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:12 smithi188 ceph-mon[92425]: pgmap v4864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:21:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:21:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:21:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:21:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:21:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:21:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:21:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:21:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:21:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:21:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:21:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:21:13.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:21:13.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:21:13.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:21:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:21:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:21:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:21:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:21:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:21:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:21:13.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:21:13.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:21:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:21:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:21:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:21:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:21:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:21:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:21:13.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:21:13.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:21:13.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:21:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:21:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:21:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:21:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:21:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:21:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:21:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:21:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:21:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:21:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:21:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:21:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:21:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:21:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:21:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=247] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:21:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=247] Local address: 172.21.15.161, port 53542 2022-07-03T13:21:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=247] Peer address: 172.21.15.101, port 22 2022-07-03T13:21:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=247] Connection lost 2022-07-03T13:21:13.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=247] Aborting connection 2022-07-03T13:21:13.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:14 smithi161 ceph-mon[91579]: pgmap v4865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:14 smithi188 ceph-mon[92425]: pgmap v4865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:16 smithi161 ceph-mon[91579]: pgmap v4866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:16 smithi188 ceph-mon[92425]: pgmap v4866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:18 smithi161 ceph-mon[91579]: pgmap v4867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:18 smithi188 ceph-mon[92425]: pgmap v4867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:20 smithi161 ceph-mon[91579]: pgmap v4868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:20 smithi188 ceph-mon[92425]: pgmap v4868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:22 smithi161 ceph-mon[91579]: pgmap v4869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:22 smithi188 ceph-mon[92425]: pgmap v4869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:24 smithi161 ceph-mon[91579]: pgmap v4870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:24 smithi188 ceph-mon[92425]: pgmap v4870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:26 smithi161 ceph-mon[91579]: pgmap v4871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:26 smithi188 ceph-mon[92425]: pgmap v4871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:28 smithi161 ceph-mon[91579]: pgmap v4872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:28 smithi188 ceph-mon[92425]: pgmap v4872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:30 smithi161 ceph-mon[91579]: pgmap v4873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:30 smithi188 ceph-mon[92425]: pgmap v4873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:32 smithi161 ceph-mon[91579]: pgmap v4874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:32 smithi188 ceph-mon[92425]: pgmap v4874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:34 smithi161 ceph-mon[91579]: pgmap v4875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:34 smithi188 ceph-mon[92425]: pgmap v4875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:36 smithi161 ceph-mon[91579]: pgmap v4876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:36 smithi188 ceph-mon[92425]: pgmap v4876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:38 smithi161 ceph-mon[91579]: pgmap v4877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:38 smithi188 ceph-mon[92425]: pgmap v4877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:40 smithi161 ceph-mon[91579]: pgmap v4878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:40 smithi188 ceph-mon[92425]: pgmap v4878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:42 smithi161 ceph-mon[91579]: pgmap v4879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:42 smithi188 ceph-mon[92425]: pgmap v4879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:44 smithi161 ceph-mon[91579]: pgmap v4880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:44 smithi188 ceph-mon[92425]: pgmap v4880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:46 smithi161 ceph-mon[91579]: pgmap v4881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:46 smithi188 ceph-mon[92425]: pgmap v4881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:48 smithi161 ceph-mon[91579]: pgmap v4882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:48 smithi188 ceph-mon[92425]: pgmap v4882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:50 smithi161 ceph-mon[91579]: pgmap v4883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:50 smithi188 ceph-mon[92425]: pgmap v4883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:52 smithi188 ceph-mon[92425]: pgmap v4884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:52 smithi161 ceph-mon[91579]: pgmap v4884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:54 smithi188 ceph-mon[92425]: pgmap v4885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:54 smithi161 ceph-mon[91579]: pgmap v4885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:21:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:56 smithi188 ceph-mon[92425]: pgmap v4886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:56 smithi161 ceph-mon[91579]: pgmap v4886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:21:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:21:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:21:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:21:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:21:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:21:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:21:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:21:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:21:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:21:58 smithi188 ceph-mon[92425]: pgmap v4887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:21:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:21:58 smithi161 ceph-mon[91579]: pgmap v4887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:00 smithi188 ceph-mon[92425]: pgmap v4888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:00 smithi161 ceph-mon[91579]: pgmap v4888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:02 smithi188 ceph-mon[92425]: pgmap v4889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:02 smithi161 ceph-mon[91579]: pgmap v4889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:04 smithi188 ceph-mon[92425]: pgmap v4890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:04 smithi161 ceph-mon[91579]: pgmap v4890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:06 smithi188 ceph-mon[92425]: pgmap v4891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:06 smithi161 ceph-mon[91579]: pgmap v4891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:08 smithi188 ceph-mon[92425]: pgmap v4892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:08 smithi161 ceph-mon[91579]: pgmap v4892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:10 smithi161 ceph-mon[91579]: pgmap v4893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:10 smithi188 ceph-mon[92425]: pgmap v4893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: pgmap v4894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:22:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:22:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:22:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:22:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: pgmap v4894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:22:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:22:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:22:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:22:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:14 smithi161 ceph-mon[91579]: pgmap v4895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:14 smithi188 ceph-mon[92425]: pgmap v4895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:16 smithi161 ceph-mon[91579]: pgmap v4896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:16 smithi188 ceph-mon[92425]: pgmap v4896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:18 smithi161 ceph-mon[91579]: pgmap v4897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:18 smithi188 ceph-mon[92425]: pgmap v4897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:20 smithi161 ceph-mon[91579]: pgmap v4898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:20 smithi188 ceph-mon[92425]: pgmap v4898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:22 smithi161 ceph-mon[91579]: pgmap v4899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:22 smithi188 ceph-mon[92425]: pgmap v4899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:24 smithi161 ceph-mon[91579]: pgmap v4900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:24 smithi188 ceph-mon[92425]: pgmap v4900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:26 smithi161 ceph-mon[91579]: pgmap v4901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:26 smithi188 ceph-mon[92425]: pgmap v4901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:28 smithi161 ceph-mon[91579]: pgmap v4902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:28 smithi188 ceph-mon[92425]: pgmap v4902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:30 smithi161 ceph-mon[91579]: pgmap v4903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:30 smithi188 ceph-mon[92425]: pgmap v4903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:32 smithi161 ceph-mon[91579]: pgmap v4904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:32 smithi188 ceph-mon[92425]: pgmap v4904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:34 smithi161 ceph-mon[91579]: pgmap v4905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:34 smithi188 ceph-mon[92425]: pgmap v4905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:36 smithi161 ceph-mon[91579]: pgmap v4906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:36 smithi188 ceph-mon[92425]: pgmap v4906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:38 smithi161 ceph-mon[91579]: pgmap v4907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:38 smithi188 ceph-mon[92425]: pgmap v4907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:40 smithi161 ceph-mon[91579]: pgmap v4908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:40 smithi188 ceph-mon[92425]: pgmap v4908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:22:42.192+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:22:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:22:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:22:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:22:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:22:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:22:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:22:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:22:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:22:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:22:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:22:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:22:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:22:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:22:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:22:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:22:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:22:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:22:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:22:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:22:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:22:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:22:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:22:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:22:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:22:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:22:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:22:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:22:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:22:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:22:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:22:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:22:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:22:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:22:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:22:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:22:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:22:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:22:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:22:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:22:42.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:22:42.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:22:42.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:22:42.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:22:42.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:22:42.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=249] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:22:42.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=249] Local address: 172.21.15.161, port 53546 2022-07-03T13:22:42.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=249] Peer address: 172.21.15.101, port 22 2022-07-03T13:22:42.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=712] Received exit status 0 2022-07-03T13:22:42.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=712] Received channel close 2022-07-03T13:22:42.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=712] Channel closed 2022-07-03T13:22:42.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=713] Requesting new SSH session 2022-07-03T13:22:42.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=712] Received exit status 0 2022-07-03T13:22:42.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=712] Received channel close 2022-07-03T13:22:42.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=712] Channel closed 2022-07-03T13:22:42.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=713] Command: which python3 2022-07-03T13:22:42.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=713] Requesting new SSH session 2022-07-03T13:22:42.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=713] Command: which python3 2022-07-03T13:22:42.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=713] Received exit status 0 2022-07-03T13:22:42.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=713] Received channel close 2022-07-03T13:22:42.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=713] Channel closed 2022-07-03T13:22:42.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=714] Requesting new SSH session 2022-07-03T13:22:42.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=714] Command: sudo true 2022-07-03T13:22:42.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=713] Received exit status 0 2022-07-03T13:22:42.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=713] Received channel close 2022-07-03T13:22:42.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=713] Channel closed 2022-07-03T13:22:42.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=714] Requesting new SSH session 2022-07-03T13:22:42.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=714] Command: sudo true 2022-07-03T13:22:42.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=714] Received exit status 0 2022-07-03T13:22:42.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=714] Received channel close 2022-07-03T13:22:42.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=714] Channel closed 2022-07-03T13:22:42.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=715] Requesting new SSH session 2022-07-03T13:22:42.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:42.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=714] Received exit status 0 2022-07-03T13:22:42.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=714] Received channel close 2022-07-03T13:22:42.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=714] Channel closed 2022-07-03T13:22:42.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=715] Requesting new SSH session 2022-07-03T13:22:42.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:42.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=715] Received exit status 0 2022-07-03T13:22:42.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=715] Received channel close 2022-07-03T13:22:42.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=715] Channel closed 2022-07-03T13:22:42.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=715] Received exit status 0 2022-07-03T13:22:42.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=715] Received channel close 2022-07-03T13:22:42.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=715] Channel closed 2022-07-03T13:22:42.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=249] Login timeout expired 2022-07-03T13:22:42.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=249] Aborting connection 2022-07-03T13:22:42.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:22:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-mon[91579]: pgmap v4909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:22:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:22:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:22:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:22:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:22:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:22:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:22:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:22:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:22:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:22:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:22:43.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:22:43.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:22:43.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:22:43.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:22:43.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:22:43.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:22:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:22:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:22:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:22:43.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:22:43.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:22:43.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:22:43.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:22:43.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:22:43.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:22:43.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:22:43.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:22:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:22:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:22:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:22:43.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:22:43.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:22:43.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:22:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:22:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:22:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:22:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:22:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:22:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:22:43.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:22:43.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:22:43.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:22:43.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:22:43.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:22:43.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=249] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:22:43.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=249] Local address: 172.21.15.161, port 53546 2022-07-03T13:22:43.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=249] Peer address: 172.21.15.101, port 22 2022-07-03T13:22:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=712] Received exit status 0 2022-07-03T13:22:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=712] Received channel close 2022-07-03T13:22:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=712] Channel closed 2022-07-03T13:22:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=713] Requesting new SSH session 2022-07-03T13:22:43.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=712] Received exit status 0 2022-07-03T13:22:43.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=712] Received channel close 2022-07-03T13:22:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=712] Channel closed 2022-07-03T13:22:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=713] Command: which python3 2022-07-03T13:22:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=713] Requesting new SSH session 2022-07-03T13:22:43.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=713] Command: which python3 2022-07-03T13:22:43.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=713] Received exit status 0 2022-07-03T13:22:43.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=713] Received channel close 2022-07-03T13:22:43.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=713] Channel closed 2022-07-03T13:22:43.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=714] Requesting new SSH session 2022-07-03T13:22:43.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=714] Command: sudo true 2022-07-03T13:22:43.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=713] Received exit status 0 2022-07-03T13:22:43.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=713] Received channel close 2022-07-03T13:22:43.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=713] Channel closed 2022-07-03T13:22:43.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=714] Requesting new SSH session 2022-07-03T13:22:43.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=714] Command: sudo true 2022-07-03T13:22:43.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=714] Received exit status 0 2022-07-03T13:22:43.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=714] Received channel close 2022-07-03T13:22:43.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=714] Channel closed 2022-07-03T13:22:43.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=715] Requesting new SSH session 2022-07-03T13:22:43.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:43.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=714] Received exit status 0 2022-07-03T13:22:43.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=714] Received channel close 2022-07-03T13:22:43.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=714] Channel closed 2022-07-03T13:22:43.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=715] Requesting new SSH session 2022-07-03T13:22:43.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:43.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=715] Received exit status 0 2022-07-03T13:22:43.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=715] Received channel close 2022-07-03T13:22:43.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=715] Channel closed 2022-07-03T13:22:43.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=715] Received exit status 0 2022-07-03T13:22:43.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=715] Received channel close 2022-07-03T13:22:43.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=715] Channel closed 2022-07-03T13:22:43.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=249] Login timeout expired 2022-07-03T13:22:43.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=249] Aborting connection 2022-07-03T13:22:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:42 smithi188 ceph-mon[92425]: pgmap v4909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:22:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:22:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:22:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:22:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:22:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:22:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:22:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:22:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:22:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:22:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:22:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:22:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:22:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T13:22:43.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:22:43.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:22:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:22:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:22:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:22:43.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:22:43.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:22:43.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:22:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:22:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:22:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:22:43.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:22:43.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:22:43.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:22:43.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:22:43.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:22:43.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:22:43.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:22:43.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:22:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:22:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:22:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:22:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:22:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:22:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:22:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:22:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:22:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:22:43.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:22:43.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:22:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T13:22:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:22:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=249] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:22:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=249] Local address: 172.21.15.161, port 53546 2022-07-03T13:22:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=249] Peer address: 172.21.15.101, port 22 2022-07-03T13:22:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=712] Received exit status 0 2022-07-03T13:22:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=712] Received channel close 2022-07-03T13:22:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=712] Channel closed 2022-07-03T13:22:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=713] Requesting new SSH session 2022-07-03T13:22:43.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=712] Received exit status 0 2022-07-03T13:22:43.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=712] Received channel close 2022-07-03T13:22:43.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=712] Channel closed 2022-07-03T13:22:43.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=713] Command: which python3 2022-07-03T13:22:43.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=713] Requesting new SSH session 2022-07-03T13:22:43.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=713] Command: which python3 2022-07-03T13:22:43.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=713] Received exit status 0 2022-07-03T13:22:43.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=713] Received channel close 2022-07-03T13:22:43.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=713] Channel closed 2022-07-03T13:22:43.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=714] Requesting new SSH session 2022-07-03T13:22:43.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=714] Command: sudo true 2022-07-03T13:22:43.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=713] Received exit status 0 2022-07-03T13:22:43.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=713] Received channel close 2022-07-03T13:22:43.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=713] Channel closed 2022-07-03T13:22:43.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=714] Requesting new SSH session 2022-07-03T13:22:43.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=714] Command: sudo true 2022-07-03T13:22:43.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=714] Received exit status 0 2022-07-03T13:22:43.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=714] Received channel close 2022-07-03T13:22:43.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=714] Channel closed 2022-07-03T13:22:43.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=715] Requesting new SSH session 2022-07-03T13:22:43.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:43.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=714] Received exit status 0 2022-07-03T13:22:43.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=714] Received channel close 2022-07-03T13:22:43.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=714] Channel closed 2022-07-03T13:22:43.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=715] Requesting new SSH session 2022-07-03T13:22:43.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=715] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T13:22:43.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=715] Received exit status 0 2022-07-03T13:22:43.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=715] Received channel close 2022-07-03T13:22:43.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=715] Channel closed 2022-07-03T13:22:43.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=715] Received exit status 0 2022-07-03T13:22:43.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=715] Received channel close 2022-07-03T13:22:43.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=715] Channel closed 2022-07-03T13:22:43.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=249] Login timeout expired 2022-07-03T13:22:43.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=249] Aborting connection 2022-07-03T13:22:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:44 smithi161 ceph-mon[91579]: pgmap v4910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:44 smithi188 ceph-mon[92425]: pgmap v4910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:46 smithi161 ceph-mon[91579]: pgmap v4911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:46 smithi188 ceph-mon[92425]: pgmap v4911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:48 smithi161 ceph-mon[91579]: pgmap v4912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:48 smithi188 ceph-mon[92425]: pgmap v4912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:50 smithi161 ceph-mon[91579]: pgmap v4913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:50 smithi188 ceph-mon[92425]: pgmap v4913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:52 smithi161 ceph-mon[91579]: pgmap v4914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:52 smithi188 ceph-mon[92425]: pgmap v4914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:55 smithi161 ceph-mon[91579]: pgmap v4915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:55 smithi188 ceph-mon[92425]: pgmap v4915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:22:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:57 smithi161 ceph-mon[91579]: pgmap v4916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:57 smithi188 ceph-mon[92425]: pgmap v4916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:22:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:22:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:22:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:22:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:22:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:22:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:22:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:22:59 smithi161 ceph-mon[91579]: pgmap v4917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:22:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:22:59 smithi188 ceph-mon[92425]: pgmap v4917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:01 smithi161 ceph-mon[91579]: pgmap v4918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:01 smithi188 ceph-mon[92425]: pgmap v4918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:03 smithi161 ceph-mon[91579]: pgmap v4919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:03 smithi188 ceph-mon[92425]: pgmap v4919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:05 smithi161 ceph-mon[91579]: pgmap v4920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:05 smithi188 ceph-mon[92425]: pgmap v4920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:07 smithi161 ceph-mon[91579]: pgmap v4921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:07 smithi188 ceph-mon[92425]: pgmap v4921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:09 smithi161 ceph-mon[91579]: pgmap v4922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:09 smithi188 ceph-mon[92425]: pgmap v4922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:11 smithi161 ceph-mon[91579]: pgmap v4923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:11 smithi188 ceph-mon[92425]: pgmap v4923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:13 smithi161 ceph-mon[91579]: pgmap v4924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:13 smithi188 ceph-mon[92425]: pgmap v4924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:15 smithi161 ceph-mon[91579]: pgmap v4925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:15 smithi188 ceph-mon[92425]: pgmap v4925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:17 smithi161 ceph-mon[91579]: pgmap v4926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:17 smithi188 ceph-mon[92425]: pgmap v4926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:19 smithi161 ceph-mon[91579]: pgmap v4927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:19 smithi188 ceph-mon[92425]: pgmap v4927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:21 smithi161 ceph-mon[91579]: pgmap v4928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:21 smithi188 ceph-mon[92425]: pgmap v4928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:23 smithi161 ceph-mon[91579]: pgmap v4929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:23 smithi188 ceph-mon[92425]: pgmap v4929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:25 smithi161 ceph-mon[91579]: pgmap v4930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:25 smithi188 ceph-mon[92425]: pgmap v4930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:27 smithi161 ceph-mon[91579]: pgmap v4931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:27 smithi188 ceph-mon[92425]: pgmap v4931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:29 smithi161 ceph-mon[91579]: pgmap v4932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:29 smithi188 ceph-mon[92425]: pgmap v4932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:31 smithi161 ceph-mon[91579]: pgmap v4933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:31 smithi188 ceph-mon[92425]: pgmap v4933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:33 smithi161 ceph-mon[91579]: pgmap v4934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:33 smithi188 ceph-mon[92425]: pgmap v4934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:35 smithi161 ceph-mon[91579]: pgmap v4935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:35 smithi188 ceph-mon[92425]: pgmap v4935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:37 smithi161 ceph-mon[91579]: pgmap v4936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:37 smithi188 ceph-mon[92425]: pgmap v4936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:39 smithi161 ceph-mon[91579]: pgmap v4937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:39 smithi188 ceph-mon[92425]: pgmap v4937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:41 smithi161 ceph-mon[91579]: pgmap v4938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:41 smithi188 ceph-mon[92425]: pgmap v4938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:23:42.228+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:23:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:23:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:23:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:23:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:23:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:23:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:23:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:23:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:23:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:23:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:23:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:23:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:23:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:23:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:23:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:23:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:23:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:23:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:23:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:23:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:23:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:23:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:23:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:23:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:23:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:23:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:23:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:23:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:23:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:23:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:23:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:23:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:23:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:23:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:23:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:23:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:23:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:23:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:23:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:23:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:23:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:23:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:23:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:23:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:23:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=251] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:23:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=251] Local address: 172.21.15.161, port 53550 2022-07-03T13:23:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=251] Peer address: 172.21.15.101, port 22 2022-07-03T13:23:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=251] Connection lost 2022-07-03T13:23:42.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=251] Aborting connection 2022-07-03T13:23:42.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:23:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:23:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: pgmap v4939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:23:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:23:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:23:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:23:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:23:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:23:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:23:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:23:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:23:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:23:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:23:43.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:23:43.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:23:43.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:23:43.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:23:43.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:23:43.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:23:43.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:23:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:23:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:23:43.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:23:43.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:23:43.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:23:43.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:23:43.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:23:43.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:23:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:23:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:23:43.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:23:43.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:23:43.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:23:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:23:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:23:43.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:23:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:23:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:23:43.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:23:43.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:23:43.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:23:43.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:23:43.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:23:43.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:23:43.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:23:43.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:23:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:23:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:23:43.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:23:43.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:23:43.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:23:43.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=251] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:23:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=251] Local address: 172.21.15.161, port 53550 2022-07-03T13:23:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=251] Peer address: 172.21.15.101, port 22 2022-07-03T13:23:43.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=251] Connection lost 2022-07-03T13:23:43.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=251] Aborting connection 2022-07-03T13:23:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: pgmap v4939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:23:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:23:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:23:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:23:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:23:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:23:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:23:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:23:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:23:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:23:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:23:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:23:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:23:43.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:23:43.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:23:43.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:23:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:23:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:23:43.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:23:43.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:23:43.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:23:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:23:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:23:43.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:23:43.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:23:43.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:23:43.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:23:43.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:23:43.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:23:43.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:23:43.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:23:43.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:23:43.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:23:43.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:23:43.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:23:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:23:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:23:43.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:23:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:23:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:23:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:23:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:23:43.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:23:43.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:23:43.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:23:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:23:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:23:43.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:23:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:23:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=251] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:23:43.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=251] Local address: 172.21.15.161, port 53550 2022-07-03T13:23:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=251] Peer address: 172.21.15.101, port 22 2022-07-03T13:23:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=251] Connection lost 2022-07-03T13:23:43.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=251] Aborting connection 2022-07-03T13:23:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:45 smithi161 ceph-mon[91579]: pgmap v4940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:45 smithi188 ceph-mon[92425]: pgmap v4940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:47 smithi161 ceph-mon[91579]: pgmap v4941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:47 smithi188 ceph-mon[92425]: pgmap v4941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:48.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:49 smithi161 ceph-mon[91579]: pgmap v4942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:49 smithi188 ceph-mon[92425]: pgmap v4942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:51 smithi161 ceph-mon[91579]: pgmap v4943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:51 smithi188 ceph-mon[92425]: pgmap v4943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:53 smithi161 ceph-mon[91579]: pgmap v4944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:53 smithi188 ceph-mon[92425]: pgmap v4944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:55 smithi161 ceph-mon[91579]: pgmap v4945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:55 smithi188 ceph-mon[92425]: pgmap v4945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:57 smithi161 ceph-mon[91579]: pgmap v4946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:57 smithi188 ceph-mon[92425]: pgmap v4946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:23:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:23:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:23:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:23:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:23:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:23:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:23:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:23:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:23:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:23:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:23:59 smithi161 ceph-mon[91579]: pgmap v4947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:23:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:23:59 smithi188 ceph-mon[92425]: pgmap v4947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:01 smithi161 ceph-mon[91579]: pgmap v4948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:01 smithi188 ceph-mon[92425]: pgmap v4948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:03 smithi188 ceph-mon[92425]: pgmap v4949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:03 smithi161 ceph-mon[91579]: pgmap v4949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:05 smithi188 ceph-mon[92425]: pgmap v4950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:05 smithi161 ceph-mon[91579]: pgmap v4950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:07 smithi188 ceph-mon[92425]: pgmap v4951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:07 smithi161 ceph-mon[91579]: pgmap v4951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:09 smithi188 ceph-mon[92425]: pgmap v4952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:09 smithi161 ceph-mon[91579]: pgmap v4952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:11 smithi188 ceph-mon[92425]: pgmap v4953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:11 smithi161 ceph-mon[91579]: pgmap v4953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:13 smithi188 ceph-mon[92425]: pgmap v4954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:13 smithi161 ceph-mon[91579]: pgmap v4954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:15 smithi188 ceph-mon[92425]: pgmap v4955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:15 smithi161 ceph-mon[91579]: pgmap v4955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:17 smithi188 ceph-mon[92425]: pgmap v4956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:17 smithi161 ceph-mon[91579]: pgmap v4956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:19 smithi188 ceph-mon[92425]: pgmap v4957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:19 smithi161 ceph-mon[91579]: pgmap v4957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:21 smithi161 ceph-mon[91579]: pgmap v4958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:21 smithi188 ceph-mon[92425]: pgmap v4958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:23 smithi161 ceph-mon[91579]: pgmap v4959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:23 smithi188 ceph-mon[92425]: pgmap v4959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:25 smithi161 ceph-mon[91579]: pgmap v4960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:25 smithi188 ceph-mon[92425]: pgmap v4960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:27 smithi161 ceph-mon[91579]: pgmap v4961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:27 smithi188 ceph-mon[92425]: pgmap v4961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:29 smithi161 ceph-mon[91579]: pgmap v4962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:29 smithi188 ceph-mon[92425]: pgmap v4962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:31 smithi161 ceph-mon[91579]: pgmap v4963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:31 smithi188 ceph-mon[92425]: pgmap v4963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:33 smithi161 ceph-mon[91579]: pgmap v4964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:33 smithi188 ceph-mon[92425]: pgmap v4964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:35 smithi161 ceph-mon[91579]: pgmap v4965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:35 smithi188 ceph-mon[92425]: pgmap v4965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:37 smithi161 ceph-mon[91579]: pgmap v4966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:37 smithi188 ceph-mon[92425]: pgmap v4966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:39 smithi161 ceph-mon[91579]: pgmap v4967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:39 smithi188 ceph-mon[92425]: pgmap v4967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:41 smithi161 ceph-mon[91579]: pgmap v4968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:41 smithi188 ceph-mon[92425]: pgmap v4968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:43.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:24:42.730+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:24:43.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:24:43.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:24:43.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:24:43.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:24:43.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:24:43.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:24:43.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:24:43.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:24:43.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:24:43.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:24:43.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:24:43.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:24:43.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:24:43.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:24:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:24:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:24:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:24:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:24:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:24:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:24:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:24:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:24:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:24:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:24:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:24:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:24:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:24:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:24:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:24:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:24:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:24:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:24:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:24:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:24:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:24:43.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:24:43.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:24:43.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:24:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:24:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:24:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:24:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:24:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:24:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:24:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=253] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:24:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=253] Local address: 172.21.15.161, port 53582 2022-07-03T13:24:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=253] Peer address: 172.21.15.101, port 22 2022-07-03T13:24:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=253] Connection lost 2022-07-03T13:24:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=253] Aborting connection 2022-07-03T13:24:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:24:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:24:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:43 smithi161 ceph-mon[91579]: pgmap v4969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:24:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:24:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:24:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:43 smithi188 ceph-mon[92425]: pgmap v4969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:24:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:24:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:24:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:24:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:24:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:24:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:24:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:24:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:24:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:24:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:24:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:24:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:24:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:24:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:24:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:24:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:24:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:24:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:24:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:24:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:24:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:24:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:24:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:24:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:24:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:24:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:24:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:24:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:24:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:24:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:24:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:24:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:24:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:24:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:24:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:24:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:24:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:24:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:24:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:24:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:24:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:24:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:24:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:24:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:24:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:24:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:24:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:24:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:24:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=253] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:24:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=253] Local address: 172.21.15.161, port 53582 2022-07-03T13:24:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=253] Peer address: 172.21.15.101, port 22 2022-07-03T13:24:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=253] Connection lost 2022-07-03T13:24:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=253] Aborting connection 2022-07-03T13:24:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:24:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:24:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:24:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:24:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:24:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:24:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:24:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:24:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:24:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:24:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:24:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:24:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:24:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:24:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:24:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:24:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:24:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:24:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:24:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:24:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:24:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:24:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:24:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:24:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:24:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:24:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:24:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:24:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:24:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:24:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:24:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:24:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:24:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:24:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:24:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:24:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:24:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:24:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:24:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:24:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:24:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:24:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:24:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:24:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:24:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:24:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=253] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:24:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=253] Local address: 172.21.15.161, port 53582 2022-07-03T13:24:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=253] Peer address: 172.21.15.101, port 22 2022-07-03T13:24:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=253] Connection lost 2022-07-03T13:24:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=253] Aborting connection 2022-07-03T13:24:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:45 smithi161 ceph-mon[91579]: pgmap v4970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:45 smithi188 ceph-mon[92425]: pgmap v4970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:47 smithi161 ceph-mon[91579]: pgmap v4971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:47 smithi188 ceph-mon[92425]: pgmap v4971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:49 smithi161 ceph-mon[91579]: pgmap v4972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:49 smithi188 ceph-mon[92425]: pgmap v4972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:51 smithi161 ceph-mon[91579]: pgmap v4973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:51 smithi188 ceph-mon[92425]: pgmap v4973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:53 smithi161 ceph-mon[91579]: pgmap v4974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:53 smithi188 ceph-mon[92425]: pgmap v4974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:55 smithi161 ceph-mon[91579]: pgmap v4975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:55 smithi188 ceph-mon[92425]: pgmap v4975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:57 smithi161 ceph-mon[91579]: pgmap v4976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:57 smithi188 ceph-mon[92425]: pgmap v4976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:24:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:24:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:24:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:24:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:24:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:24:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:24:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:24:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:24:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:24:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:24:59 smithi161 ceph-mon[91579]: pgmap v4977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:24:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:24:59 smithi188 ceph-mon[92425]: pgmap v4977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:01 smithi161 ceph-mon[91579]: pgmap v4978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:01 smithi188 ceph-mon[92425]: pgmap v4978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:03 smithi161 ceph-mon[91579]: pgmap v4979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:03 smithi188 ceph-mon[92425]: pgmap v4979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:05 smithi161 ceph-mon[91579]: pgmap v4980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:05 smithi188 ceph-mon[92425]: pgmap v4980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:07 smithi161 ceph-mon[91579]: pgmap v4981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:07 smithi188 ceph-mon[92425]: pgmap v4981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:09 smithi161 ceph-mon[91579]: pgmap v4982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:09 smithi188 ceph-mon[92425]: pgmap v4982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:11 smithi161 ceph-mon[91579]: pgmap v4983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:11 smithi188 ceph-mon[92425]: pgmap v4983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:13 smithi161 ceph-mon[91579]: pgmap v4984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:13 smithi188 ceph-mon[92425]: pgmap v4984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:15 smithi161 ceph-mon[91579]: pgmap v4985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:15 smithi188 ceph-mon[92425]: pgmap v4985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:17 smithi161 ceph-mon[91579]: pgmap v4986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:17 smithi188 ceph-mon[92425]: pgmap v4986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:18 smithi161 ceph-mon[91579]: pgmap v4987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:18 smithi188 ceph-mon[92425]: pgmap v4987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:20 smithi161 ceph-mon[91579]: pgmap v4988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:20 smithi188 ceph-mon[92425]: pgmap v4988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:22 smithi161 ceph-mon[91579]: pgmap v4989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:23.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:22 smithi188 ceph-mon[92425]: pgmap v4989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:23.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:24 smithi161 ceph-mon[91579]: pgmap v4990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:24 smithi188 ceph-mon[92425]: pgmap v4990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:26 smithi161 ceph-mon[91579]: pgmap v4991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:26 smithi188 ceph-mon[92425]: pgmap v4991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:28 smithi161 ceph-mon[91579]: pgmap v4992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:28 smithi188 ceph-mon[92425]: pgmap v4992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:30 smithi161 ceph-mon[91579]: pgmap v4993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:30 smithi188 ceph-mon[92425]: pgmap v4993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:32 smithi161 ceph-mon[91579]: pgmap v4994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:32 smithi188 ceph-mon[92425]: pgmap v4994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:34 smithi161 ceph-mon[91579]: pgmap v4995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:34 smithi188 ceph-mon[92425]: pgmap v4995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:36 smithi161 ceph-mon[91579]: pgmap v4996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:36 smithi188 ceph-mon[92425]: pgmap v4996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:38 smithi161 ceph-mon[91579]: pgmap v4997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:38 smithi188 ceph-mon[92425]: pgmap v4997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:40 smithi161 ceph-mon[91579]: pgmap v4998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:40 smithi188 ceph-mon[92425]: pgmap v4998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:42 smithi161 ceph-mon[91579]: pgmap v4999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:42 smithi188 ceph-mon[92425]: pgmap v4999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:43.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:25:43.232+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:25:43.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:25:43.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:25:43.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:25:43.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:25:43.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:25:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:25:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:25:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:25:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:25:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:25:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:25:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:25:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:25:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:25:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:25:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:25:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:25:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:25:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:25:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:25:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:25:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:25:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:25:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:25:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:25:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:25:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:25:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:25:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:25:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:25:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:25:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:25:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:25:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:25:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:25:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:25:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:25:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:25:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:25:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:25:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:25:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:25:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:25:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:25:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=255] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:25:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=255] Local address: 172.21.15.161, port 53586 2022-07-03T13:25:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=255] Peer address: 172.21.15.101, port 22 2022-07-03T13:25:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=255] Connection lost 2022-07-03T13:25:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=255] Aborting connection 2022-07-03T13:25:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:25:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:25:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:25:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:25:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:25:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:25:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:25:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:25:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:25:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:25:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:25:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:25:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:25:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:25:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:25:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:25:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:25:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:25:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:25:44.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:25:44.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:25:44.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:25:44.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:25:44.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:25:44.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:25:44.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:25:44.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:25:44.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:25:44.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:25:44.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:25:44.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:25:44.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:25:44.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:25:44.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:25:44.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:25:44.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:25:44.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:25:44.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:25:44.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:25:44.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:25:44.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:25:44.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:25:44.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:25:44.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:25:44.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:25:44.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:25:44.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:25:44.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:25:44.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:25:44.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:25:44.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=255] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:25:44.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=255] Local address: 172.21.15.161, port 53586 2022-07-03T13:25:44.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=255] Peer address: 172.21.15.101, port 22 2022-07-03T13:25:44.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=255] Connection lost 2022-07-03T13:25:44.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=255] Aborting connection 2022-07-03T13:25:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:25:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:25:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:25:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:25:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:25:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:25:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:25:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:25:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:25:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:25:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:25:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:25:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:25:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:25:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:25:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:25:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:25:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:25:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:25:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:25:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:25:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:25:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:25:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:25:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:25:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:25:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:25:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:25:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:25:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:25:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:25:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:25:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:25:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:25:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:25:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:25:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:25:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:25:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:25:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:25:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:25:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:25:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:25:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:25:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:25:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:25:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:25:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:25:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:25:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=255] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:25:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=255] Local address: 172.21.15.161, port 53586 2022-07-03T13:25:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=255] Peer address: 172.21.15.101, port 22 2022-07-03T13:25:44.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=255] Connection lost 2022-07-03T13:25:44.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=255] Aborting connection 2022-07-03T13:25:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:44 smithi188 ceph-mon[92425]: pgmap v5000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:44 smithi161 ceph-mon[91579]: pgmap v5000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:46 smithi188 ceph-mon[92425]: pgmap v5001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:46 smithi161 ceph-mon[91579]: pgmap v5001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:48 smithi188 ceph-mon[92425]: pgmap v5002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:48 smithi161 ceph-mon[91579]: pgmap v5002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:50 smithi188 ceph-mon[92425]: pgmap v5003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:50 smithi161 ceph-mon[91579]: pgmap v5003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:52 smithi188 ceph-mon[92425]: pgmap v5004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:52 smithi161 ceph-mon[91579]: pgmap v5004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:54 smithi188 ceph-mon[92425]: pgmap v5005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:54 smithi161 ceph-mon[91579]: pgmap v5005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:56 smithi188 ceph-mon[92425]: pgmap v5006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:56 smithi161 ceph-mon[91579]: pgmap v5006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:25:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:25:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:25:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:25:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:25:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:25:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:25:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:25:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:25:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:25:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:25:58 smithi161 ceph-mon[91579]: pgmap v5007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:25:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:25:58 smithi188 ceph-mon[92425]: pgmap v5007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:00 smithi161 ceph-mon[91579]: pgmap v5008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:00 smithi188 ceph-mon[92425]: pgmap v5008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:02 smithi161 ceph-mon[91579]: pgmap v5009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:02 smithi188 ceph-mon[92425]: pgmap v5009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:04 smithi161 ceph-mon[91579]: pgmap v5010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:04 smithi188 ceph-mon[92425]: pgmap v5010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:06 smithi161 ceph-mon[91579]: pgmap v5011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:06 smithi188 ceph-mon[92425]: pgmap v5011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:08 smithi161 ceph-mon[91579]: pgmap v5012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:08 smithi188 ceph-mon[92425]: pgmap v5012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:10 smithi161 ceph-mon[91579]: pgmap v5013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:10 smithi188 ceph-mon[92425]: pgmap v5013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:12 smithi161 ceph-mon[91579]: pgmap v5014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:12 smithi188 ceph-mon[92425]: pgmap v5014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:14 smithi161 ceph-mon[91579]: pgmap v5015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:14 smithi188 ceph-mon[92425]: pgmap v5015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:16 smithi161 ceph-mon[91579]: pgmap v5016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:16 smithi188 ceph-mon[92425]: pgmap v5016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:18 smithi161 ceph-mon[91579]: pgmap v5017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:18 smithi188 ceph-mon[92425]: pgmap v5017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:20 smithi161 ceph-mon[91579]: pgmap v5018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:20 smithi188 ceph-mon[92425]: pgmap v5018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:22 smithi161 ceph-mon[91579]: pgmap v5019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:22 smithi188 ceph-mon[92425]: pgmap v5019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:24 smithi161 ceph-mon[91579]: pgmap v5020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:24 smithi188 ceph-mon[92425]: pgmap v5020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:26 smithi161 ceph-mon[91579]: pgmap v5021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:26 smithi188 ceph-mon[92425]: pgmap v5021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:28 smithi161 ceph-mon[91579]: pgmap v5022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:28 smithi188 ceph-mon[92425]: pgmap v5022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:30 smithi161 ceph-mon[91579]: pgmap v5023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:30 smithi188 ceph-mon[92425]: pgmap v5023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:32 smithi161 ceph-mon[91579]: pgmap v5024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:32 smithi188 ceph-mon[92425]: pgmap v5024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:34 smithi161 ceph-mon[91579]: pgmap v5025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:34 smithi188 ceph-mon[92425]: pgmap v5025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:36 smithi161 ceph-mon[91579]: pgmap v5026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:36 smithi188 ceph-mon[92425]: pgmap v5026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:39 smithi161 ceph-mon[91579]: pgmap v5027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:39 smithi188 ceph-mon[92425]: pgmap v5027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:41 smithi161 ceph-mon[91579]: pgmap v5028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:41 smithi188 ceph-mon[92425]: pgmap v5028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-mon[91579]: pgmap v5029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:43 smithi188 ceph-mon[92425]: pgmap v5029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:44.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:26:43.728+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:26:44.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:26:44.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:26:44.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:26:44.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:26:44.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:26:44.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:26:44.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:26:44.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:26:44.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:26:44.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:26:44.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:26:44.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:26:44.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:26:44.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:26:44.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:26:44.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:26:44.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:26:44.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:26:44.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:26:44.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:26:44.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:26:44.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:26:44.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:26:44.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:26:44.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:26:44.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:26:44.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:26:44.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:26:44.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:26:44.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:26:44.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:26:44.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:26:44.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:26:44.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:26:44.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:26:44.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:26:44.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:26:44.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:26:44.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:26:44.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:26:44.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:26:44.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:26:44.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:26:44.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:26:44.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=257] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:26:44.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=257] Local address: 172.21.15.161, port 53590 2022-07-03T13:26:44.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=257] Peer address: 172.21.15.101, port 22 2022-07-03T13:26:44.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=257] Connection lost 2022-07-03T13:26:44.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=257] Aborting connection 2022-07-03T13:26:44.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:26:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:26:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:26:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:26:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:26:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:26:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:26:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:26:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:26:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:45 smithi161 ceph-mon[91579]: pgmap v5030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:26:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:26:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:26:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:26:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:26:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:26:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:26:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:26:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:26:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:26:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:26:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:26:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:26:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:26:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:26:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:26:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:26:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:26:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:26:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:26:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:26:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:26:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:26:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:26:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:26:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:26:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:26:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:26:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:26:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:26:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:26:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:26:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:26:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:26:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:26:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:26:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:26:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:26:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:26:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:26:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:26:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:26:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:26:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:26:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:26:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=257] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:26:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=257] Local address: 172.21.15.161, port 53590 2022-07-03T13:26:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=257] Peer address: 172.21.15.101, port 22 2022-07-03T13:26:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=257] Connection lost 2022-07-03T13:26:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=257] Aborting connection 2022-07-03T13:26:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:26:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:45 smithi188 ceph-mon[92425]: pgmap v5030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:26:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:26:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:26:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:26:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:26:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:26:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:26:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:26:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:26:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:26:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:26:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:26:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:26:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:26:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:26:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:26:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:26:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:26:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:26:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:26:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:26:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:26:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:26:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:26:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:26:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:26:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:26:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:26:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:26:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:26:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:26:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:26:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:26:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:26:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:26:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:26:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:26:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:26:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:26:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:26:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:26:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:26:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:26:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:26:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:26:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=257] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:26:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=257] Local address: 172.21.15.161, port 53590 2022-07-03T13:26:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=257] Peer address: 172.21.15.101, port 22 2022-07-03T13:26:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=257] Connection lost 2022-07-03T13:26:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=257] Aborting connection 2022-07-03T13:26:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:47 smithi161 ceph-mon[91579]: pgmap v5031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:47 smithi188 ceph-mon[92425]: pgmap v5031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:49 smithi161 ceph-mon[91579]: pgmap v5032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:49 smithi188 ceph-mon[92425]: pgmap v5032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:51 smithi161 ceph-mon[91579]: pgmap v5033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:51 smithi188 ceph-mon[92425]: pgmap v5033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:53 smithi161 ceph-mon[91579]: pgmap v5034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:53 smithi188 ceph-mon[92425]: pgmap v5034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:55 smithi161 ceph-mon[91579]: pgmap v5035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:55 smithi188 ceph-mon[92425]: pgmap v5035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:57 smithi161 ceph-mon[91579]: pgmap v5036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:57 smithi188 ceph-mon[92425]: pgmap v5036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:26:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:26:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:26:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:26:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:26:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:26:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:26:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:26:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:26:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:26:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:26:59 smithi161 ceph-mon[91579]: pgmap v5037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:26:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:26:59 smithi188 ceph-mon[92425]: pgmap v5037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:01 smithi161 ceph-mon[91579]: pgmap v5038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:01 smithi188 ceph-mon[92425]: pgmap v5038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:03 smithi161 ceph-mon[91579]: pgmap v5039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:03 smithi188 ceph-mon[92425]: pgmap v5039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:05 smithi161 ceph-mon[91579]: pgmap v5040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:05 smithi188 ceph-mon[92425]: pgmap v5040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:07 smithi161 ceph-mon[91579]: pgmap v5041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:07 smithi188 ceph-mon[92425]: pgmap v5041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:09 smithi161 ceph-mon[91579]: pgmap v5042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:09 smithi188 ceph-mon[92425]: pgmap v5042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:11 smithi161 ceph-mon[91579]: pgmap v5043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:11 smithi188 ceph-mon[92425]: pgmap v5043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:13 smithi161 ceph-mon[91579]: pgmap v5044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:13 smithi188 ceph-mon[92425]: pgmap v5044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:15 smithi161 ceph-mon[91579]: pgmap v5045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:15 smithi188 ceph-mon[92425]: pgmap v5045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:17 smithi161 ceph-mon[91579]: pgmap v5046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:17 smithi188 ceph-mon[92425]: pgmap v5046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:19 smithi161 ceph-mon[91579]: pgmap v5047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:19 smithi188 ceph-mon[92425]: pgmap v5047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:21 smithi161 ceph-mon[91579]: pgmap v5048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:21 smithi188 ceph-mon[92425]: pgmap v5048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:23 smithi161 ceph-mon[91579]: pgmap v5049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:23 smithi188 ceph-mon[92425]: pgmap v5049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:25 smithi161 ceph-mon[91579]: pgmap v5050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:25 smithi188 ceph-mon[92425]: pgmap v5050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:27 smithi161 ceph-mon[91579]: pgmap v5051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:27 smithi188 ceph-mon[92425]: pgmap v5051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:29 smithi161 ceph-mon[91579]: pgmap v5052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:29 smithi188 ceph-mon[92425]: pgmap v5052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:31 smithi161 ceph-mon[91579]: pgmap v5053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:31 smithi188 ceph-mon[92425]: pgmap v5053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:33 smithi161 ceph-mon[91579]: pgmap v5054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:33 smithi188 ceph-mon[92425]: pgmap v5054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:35 smithi161 ceph-mon[91579]: pgmap v5055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:35 smithi188 ceph-mon[92425]: pgmap v5055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:37 smithi161 ceph-mon[91579]: pgmap v5056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:37 smithi188 ceph-mon[92425]: pgmap v5056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:39 smithi188 ceph-mon[92425]: pgmap v5057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:39 smithi161 ceph-mon[91579]: pgmap v5057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:41 smithi188 ceph-mon[92425]: pgmap v5058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:41 smithi161 ceph-mon[91579]: pgmap v5058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:43 smithi188 ceph-mon[92425]: pgmap v5059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:43 smithi161 ceph-mon[91579]: pgmap v5059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:44.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:27:44.226+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:27:44.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:27:44.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:27:44.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:27:44.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:27:44.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:27:44.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:27:44.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:27:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:27:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:27:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:27:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:27:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:27:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:27:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:27:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:27:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:27:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:27:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:27:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:27:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:27:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:27:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:27:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:27:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:27:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:27:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:27:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:27:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:27:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:27:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:27:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:27:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:27:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:27:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:27:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:27:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:27:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:27:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:27:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:27:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:27:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:27:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:27:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:27:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:27:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=259] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:27:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=259] Local address: 172.21.15.161, port 53594 2022-07-03T13:27:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=259] Peer address: 172.21.15.101, port 22 2022-07-03T13:27:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=259] Connection lost 2022-07-03T13:27:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=259] Aborting connection 2022-07-03T13:27:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:27:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:27:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: pgmap v5060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:27:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:27:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:27:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:27:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:27:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:27:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:27:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:27:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:27:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:27:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:27:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:27:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:27:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:27:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:27:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:27:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:27:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:27:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:27:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:27:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:27:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:27:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:27:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:27:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:27:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:27:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:27:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:27:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:27:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:27:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:27:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:27:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:27:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:27:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:27:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:27:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:27:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:27:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:27:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:27:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:27:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:27:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:27:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:27:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:27:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:27:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:27:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:27:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:27:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=259] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:27:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=259] Local address: 172.21.15.161, port 53594 2022-07-03T13:27:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=259] Peer address: 172.21.15.101, port 22 2022-07-03T13:27:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=259] Connection lost 2022-07-03T13:27:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=259] Aborting connection 2022-07-03T13:27:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: pgmap v5060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:27:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:27:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:27:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:27:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:27:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:27:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:27:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:27:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:27:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:27:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:27:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:27:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:27:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:27:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:27:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:27:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:27:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:27:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:27:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:27:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:27:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:27:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:27:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:27:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:27:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:27:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:27:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:27:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:27:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:27:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:27:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:27:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:27:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:27:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:27:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:27:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:27:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:27:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:27:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:27:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:27:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:27:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:27:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:27:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:27:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:27:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:27:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:27:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:27:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=259] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:27:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=259] Local address: 172.21.15.161, port 53594 2022-07-03T13:27:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=259] Peer address: 172.21.15.101, port 22 2022-07-03T13:27:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=259] Connection lost 2022-07-03T13:27:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=259] Aborting connection 2022-07-03T13:27:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:47 smithi188 ceph-mon[92425]: pgmap v5061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:47 smithi161 ceph-mon[91579]: pgmap v5061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:49 smithi188 ceph-mon[92425]: pgmap v5062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:49 smithi161 ceph-mon[91579]: pgmap v5062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:51 smithi188 ceph-mon[92425]: pgmap v5063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:51 smithi161 ceph-mon[91579]: pgmap v5063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:53 smithi161 ceph-mon[91579]: pgmap v5064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:53 smithi188 ceph-mon[92425]: pgmap v5064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:55 smithi161 ceph-mon[91579]: pgmap v5065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:55 smithi188 ceph-mon[92425]: pgmap v5065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:57 smithi161 ceph-mon[91579]: pgmap v5066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:57 smithi188 ceph-mon[92425]: pgmap v5066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:27:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:27:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:27:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:27:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:27:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:27:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:27:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:27:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:27:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:27:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:27:59 smithi161 ceph-mon[91579]: pgmap v5067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:27:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:27:59 smithi188 ceph-mon[92425]: pgmap v5067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:01 smithi161 ceph-mon[91579]: pgmap v5068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:01 smithi188 ceph-mon[92425]: pgmap v5068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:03 smithi161 ceph-mon[91579]: pgmap v5069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:03 smithi188 ceph-mon[92425]: pgmap v5069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:05 smithi161 ceph-mon[91579]: pgmap v5070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:05 smithi188 ceph-mon[92425]: pgmap v5070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:07 smithi161 ceph-mon[91579]: pgmap v5071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:07 smithi188 ceph-mon[92425]: pgmap v5071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:09 smithi161 ceph-mon[91579]: pgmap v5072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:09 smithi188 ceph-mon[92425]: pgmap v5072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:11 smithi161 ceph-mon[91579]: pgmap v5073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:11 smithi188 ceph-mon[92425]: pgmap v5073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:13 smithi161 ceph-mon[91579]: pgmap v5074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:13 smithi188 ceph-mon[92425]: pgmap v5074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:15 smithi161 ceph-mon[91579]: pgmap v5075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:15 smithi188 ceph-mon[92425]: pgmap v5075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:17 smithi161 ceph-mon[91579]: pgmap v5076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:17 smithi188 ceph-mon[92425]: pgmap v5076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:19 smithi161 ceph-mon[91579]: pgmap v5077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:19 smithi188 ceph-mon[92425]: pgmap v5077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:21 smithi161 ceph-mon[91579]: pgmap v5078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:21 smithi188 ceph-mon[92425]: pgmap v5078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:23 smithi161 ceph-mon[91579]: pgmap v5079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:23 smithi188 ceph-mon[92425]: pgmap v5079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:25 smithi161 ceph-mon[91579]: pgmap v5080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:25 smithi188 ceph-mon[92425]: pgmap v5080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:27 smithi161 ceph-mon[91579]: pgmap v5081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:27 smithi188 ceph-mon[92425]: pgmap v5081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:29 smithi161 ceph-mon[91579]: pgmap v5082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:29 smithi188 ceph-mon[92425]: pgmap v5082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:31 smithi161 ceph-mon[91579]: pgmap v5083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:31 smithi188 ceph-mon[92425]: pgmap v5083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:33 smithi161 ceph-mon[91579]: pgmap v5084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:33 smithi188 ceph-mon[92425]: pgmap v5084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:35 smithi161 ceph-mon[91579]: pgmap v5085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:35 smithi188 ceph-mon[92425]: pgmap v5085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:37 smithi161 ceph-mon[91579]: pgmap v5086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:37 smithi188 ceph-mon[92425]: pgmap v5086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:39 smithi161 ceph-mon[91579]: pgmap v5087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:39 smithi188 ceph-mon[92425]: pgmap v5087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:41 smithi161 ceph-mon[91579]: pgmap v5088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:41 smithi188 ceph-mon[92425]: pgmap v5088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:43 smithi161 ceph-mon[91579]: pgmap v5089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:43 smithi188 ceph-mon[92425]: pgmap v5089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:45.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:28:44.737+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:28:45.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:28:45.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:28:45.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:28:45.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:28:45.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:28:45.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:28:45.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:28:45.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:28:45.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:28:45.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:28:45.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:28:45.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:28:45.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:28:45.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:28:45.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:28:45.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:28:45.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:28:45.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:28:45.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:28:45.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:28:45.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:28:45.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:28:45.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:28:45.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:28:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:28:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:28:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:28:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:28:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:28:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:28:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:28:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:28:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:28:45.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:28:45.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:28:45.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:28:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:28:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:28:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:28:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:28:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:28:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:28:45.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:28:45.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:28:45.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=261] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:28:45.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=261] Local address: 172.21.15.161, port 53598 2022-07-03T13:28:45.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=261] Peer address: 172.21.15.101, port 22 2022-07-03T13:28:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=261] Connection lost 2022-07-03T13:28:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=261] Aborting connection 2022-07-03T13:28:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:28:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:28:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: pgmap v5090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:28:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:28:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:28:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: pgmap v5090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:28:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:28:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:28:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:28:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:28:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:28:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:28:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:28:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:28:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:28:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:28:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:28:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:28:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:28:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:28:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:28:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:28:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:28:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:28:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:28:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:28:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:28:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:28:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:28:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:28:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:28:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:28:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:28:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:28:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:28:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:28:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:28:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:28:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:28:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:28:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:28:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:28:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:28:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:28:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:28:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:28:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:28:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:28:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:28:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:28:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:28:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:28:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:28:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:28:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:28:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=261] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:28:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=261] Local address: 172.21.15.161, port 53598 2022-07-03T13:28:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=261] Peer address: 172.21.15.101, port 22 2022-07-03T13:28:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=261] Connection lost 2022-07-03T13:28:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=261] Aborting connection 2022-07-03T13:28:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:28:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:28:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:28:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:28:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:28:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:28:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:28:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:28:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:28:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:28:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:28:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:28:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:28:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:28:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:28:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:28:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:28:46.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:28:46.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:28:46.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:28:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:28:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:28:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:28:46.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:28:46.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:28:46.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:28:46.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:28:46.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:28:46.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:28:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:28:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:28:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:28:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:28:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:28:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:28:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:28:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:28:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:28:46.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:28:46.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:28:46.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:28:46.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:28:46.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:28:46.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:28:46.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:28:46.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:28:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=261] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:28:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=261] Local address: 172.21.15.161, port 53598 2022-07-03T13:28:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=261] Peer address: 172.21.15.101, port 22 2022-07-03T13:28:46.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=261] Connection lost 2022-07-03T13:28:46.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=261] Aborting connection 2022-07-03T13:28:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:47 smithi161 ceph-mon[91579]: pgmap v5091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:47 smithi188 ceph-mon[92425]: pgmap v5091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:49 smithi161 ceph-mon[91579]: pgmap v5092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:49 smithi188 ceph-mon[92425]: pgmap v5092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:51 smithi161 ceph-mon[91579]: pgmap v5093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:51 smithi188 ceph-mon[92425]: pgmap v5093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:53 smithi161 ceph-mon[91579]: pgmap v5094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:53 smithi188 ceph-mon[92425]: pgmap v5094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:55 smithi161 ceph-mon[91579]: pgmap v5095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:55 smithi188 ceph-mon[92425]: pgmap v5095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:57 smithi161 ceph-mon[91579]: pgmap v5096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:57 smithi188 ceph-mon[92425]: pgmap v5096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:28:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:28:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:28:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:28:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:28:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:28:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:28:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:28:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:28:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:28:59 smithi161 ceph-mon[91579]: pgmap v5097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:28:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:28:59 smithi188 ceph-mon[92425]: pgmap v5097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:01 smithi161 ceph-mon[91579]: pgmap v5098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:01 smithi188 ceph-mon[92425]: pgmap v5098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:03 smithi161 ceph-mon[91579]: pgmap v5099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:03 smithi188 ceph-mon[92425]: pgmap v5099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:05 smithi161 ceph-mon[91579]: pgmap v5100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:05 smithi188 ceph-mon[92425]: pgmap v5100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:07 smithi161 ceph-mon[91579]: pgmap v5101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:07 smithi188 ceph-mon[92425]: pgmap v5101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:09 smithi161 ceph-mon[91579]: pgmap v5102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:09 smithi188 ceph-mon[92425]: pgmap v5102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:11 smithi161 ceph-mon[91579]: pgmap v5103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:11 smithi188 ceph-mon[92425]: pgmap v5103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:13 smithi161 ceph-mon[91579]: pgmap v5104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:13 smithi188 ceph-mon[92425]: pgmap v5104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:15 smithi161 ceph-mon[91579]: pgmap v5105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:15 smithi188 ceph-mon[92425]: pgmap v5105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:17 smithi161 ceph-mon[91579]: pgmap v5106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:17 smithi188 ceph-mon[92425]: pgmap v5106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:19 smithi161 ceph-mon[91579]: pgmap v5107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:19 smithi188 ceph-mon[92425]: pgmap v5107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:21 smithi161 ceph-mon[91579]: pgmap v5108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:21 smithi188 ceph-mon[92425]: pgmap v5108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:23 smithi161 ceph-mon[91579]: pgmap v5109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:23 smithi188 ceph-mon[92425]: pgmap v5109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:24 smithi161 ceph-mon[91579]: pgmap v5110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:24 smithi188 ceph-mon[92425]: pgmap v5110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:26 smithi161 ceph-mon[91579]: pgmap v5111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:26 smithi188 ceph-mon[92425]: pgmap v5111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:28 smithi161 ceph-mon[91579]: pgmap v5112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:28 smithi188 ceph-mon[92425]: pgmap v5112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:30 smithi188 ceph-mon[92425]: pgmap v5113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:30 smithi161 ceph-mon[91579]: pgmap v5113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:32 smithi188 ceph-mon[92425]: pgmap v5114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:32 smithi161 ceph-mon[91579]: pgmap v5114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:34 smithi188 ceph-mon[92425]: pgmap v5115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:34 smithi161 ceph-mon[91579]: pgmap v5115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:36 smithi188 ceph-mon[92425]: pgmap v5116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:36 smithi161 ceph-mon[91579]: pgmap v5116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:38 smithi188 ceph-mon[92425]: pgmap v5117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:38 smithi161 ceph-mon[91579]: pgmap v5117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:40 smithi188 ceph-mon[92425]: pgmap v5118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:40 smithi161 ceph-mon[91579]: pgmap v5118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:42 smithi188 ceph-mon[92425]: pgmap v5119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:42 smithi161 ceph-mon[91579]: pgmap v5119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:44 smithi188 ceph-mon[92425]: pgmap v5120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:44 smithi161 ceph-mon[91579]: pgmap v5120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:45.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:29:45.915 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:29:45.915 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:29:45.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:29:45.658+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:29:45.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:29:45.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:29:45.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:29:45.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:29:45.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:29:45.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:29:45.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:29:45.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:29:45.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:29:45.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:29:45.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:29:45.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:29:45.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:29:45.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:29:45.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:29:45.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:29:45.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:29:45.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:29:45.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:29:45.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:29:45.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:29:45.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:29:45.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:29:45.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:29:45.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:29:45.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:29:45.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:29:45.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:29:45.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:29:45.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:29:45.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:29:45.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:29:45.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:29:45.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:29:45.928 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:29:45.928 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:29:45.928 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:29:45.929 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:29:45.929 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:29:45.929 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:29:45.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:29:45.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:29:45.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:29:45.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:29:45.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=263] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:29:45.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=263] Local address: 172.21.15.161, port 53602 2022-07-03T13:29:45.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=263] Peer address: 172.21.15.101, port 22 2022-07-03T13:29:45.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=263] Connection lost 2022-07-03T13:29:45.933 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=263] Aborting connection 2022-07-03T13:29:45.933 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:29:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:29:46.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:29:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:29:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:29:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:29:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:29:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:29:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:29:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:29:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:29:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:29:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:29:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:29:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:29:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:29:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:29:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:29:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:29:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:29:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:29:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:29:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:29:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:29:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:29:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:29:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:29:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:29:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:29:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:29:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:29:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:29:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:29:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:29:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:29:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:29:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:29:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:29:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:29:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:29:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:29:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:29:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:29:47.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:29:47.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:29:47.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:29:47.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:29:47.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:29:47.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:29:47.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:29:47.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:29:47.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=263] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:29:47.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=263] Local address: 172.21.15.161, port 53602 2022-07-03T13:29:47.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=263] Peer address: 172.21.15.101, port 22 2022-07-03T13:29:47.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=263] Connection lost 2022-07-03T13:29:47.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=263] Aborting connection 2022-07-03T13:29:47.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:46 smithi161 ceph-mon[91579]: pgmap v5121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:29:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:29:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:29:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:29:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:29:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:29:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:29:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:29:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:29:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:29:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:29:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:29:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:29:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:29:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:29:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:29:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:29:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:29:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:29:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:29:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:29:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:29:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:29:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:29:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:29:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:29:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:29:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:29:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:29:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:29:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:29:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:29:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:29:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:29:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:29:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:29:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:29:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:29:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:29:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:29:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:29:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:29:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:29:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:29:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:29:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:29:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=263] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:29:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=263] Local address: 172.21.15.161, port 53602 2022-07-03T13:29:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=263] Peer address: 172.21.15.101, port 22 2022-07-03T13:29:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=263] Connection lost 2022-07-03T13:29:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=263] Aborting connection 2022-07-03T13:29:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:46 smithi188 ceph-mon[92425]: pgmap v5121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:48 smithi161 ceph-mon[91579]: pgmap v5122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:48 smithi188 ceph-mon[92425]: pgmap v5122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:50 smithi161 ceph-mon[91579]: pgmap v5123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:50 smithi188 ceph-mon[92425]: pgmap v5123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:52 smithi161 ceph-mon[91579]: pgmap v5124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:52 smithi188 ceph-mon[92425]: pgmap v5124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:54 smithi161 ceph-mon[91579]: pgmap v5125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:54 smithi188 ceph-mon[92425]: pgmap v5125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:56 smithi161 ceph-mon[91579]: pgmap v5126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:56 smithi188 ceph-mon[92425]: pgmap v5126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:29:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:29:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:29:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:29:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:29:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:29:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:29:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:29:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:29:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:29:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:29:58 smithi161 ceph-mon[91579]: pgmap v5127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:29:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:29:58 smithi188 ceph-mon[92425]: pgmap v5127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:00 smithi161 ceph-mon[91579]: pgmap v5128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:30:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:00 smithi188 ceph-mon[92425]: pgmap v5128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:30:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:02 smithi161 ceph-mon[91579]: pgmap v5129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:02 smithi188 ceph-mon[92425]: pgmap v5129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:04 smithi161 ceph-mon[91579]: pgmap v5130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:04 smithi188 ceph-mon[92425]: pgmap v5130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:06 smithi161 ceph-mon[91579]: pgmap v5131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:06 smithi188 ceph-mon[92425]: pgmap v5131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:08 smithi161 ceph-mon[91579]: pgmap v5132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:08 smithi188 ceph-mon[92425]: pgmap v5132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:10 smithi161 ceph-mon[91579]: pgmap v5133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:10 smithi188 ceph-mon[92425]: pgmap v5133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:12 smithi161 ceph-mon[91579]: pgmap v5134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:12 smithi188 ceph-mon[92425]: pgmap v5134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:14 smithi188 ceph-mon[92425]: pgmap v5135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:14 smithi161 ceph-mon[91579]: pgmap v5135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:16 smithi161 ceph-mon[91579]: pgmap v5136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:16 smithi188 ceph-mon[92425]: pgmap v5136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:18 smithi161 ceph-mon[91579]: pgmap v5137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:18 smithi188 ceph-mon[92425]: pgmap v5137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:20 smithi161 ceph-mon[91579]: pgmap v5138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:20 smithi188 ceph-mon[92425]: pgmap v5138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:22 smithi161 ceph-mon[91579]: pgmap v5139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:22 smithi188 ceph-mon[92425]: pgmap v5139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:24 smithi161 ceph-mon[91579]: pgmap v5140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:24 smithi188 ceph-mon[92425]: pgmap v5140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:26 smithi161 ceph-mon[91579]: pgmap v5141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:26 smithi188 ceph-mon[92425]: pgmap v5141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:29 smithi161 ceph-mon[91579]: pgmap v5142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:29 smithi188 ceph-mon[92425]: pgmap v5142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:31 smithi161 ceph-mon[91579]: pgmap v5143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:31 smithi188 ceph-mon[92425]: pgmap v5143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:33 smithi161 ceph-mon[91579]: pgmap v5144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:33 smithi188 ceph-mon[92425]: pgmap v5144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:35 smithi161 ceph-mon[91579]: pgmap v5145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:35 smithi188 ceph-mon[92425]: pgmap v5145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:37 smithi161 ceph-mon[91579]: pgmap v5146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:37 smithi188 ceph-mon[92425]: pgmap v5146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:39 smithi161 ceph-mon[91579]: pgmap v5147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:39 smithi188 ceph-mon[92425]: pgmap v5147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:41 smithi161 ceph-mon[91579]: pgmap v5148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:41 smithi188 ceph-mon[92425]: pgmap v5148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:43 smithi161 ceph-mon[91579]: pgmap v5149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:43 smithi188 ceph-mon[92425]: pgmap v5149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:45 smithi161 ceph-mon[91579]: pgmap v5150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:45 smithi188 ceph-mon[92425]: pgmap v5150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:46.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:30:46.160+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:30:46.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:30:46.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:30:46.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:30:46.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:30:46.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:30:46.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:30:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:30:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:30:46.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:30:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:30:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:30:46.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:30:46.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:30:46.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:30:46.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:30:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:30:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:30:46.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:30:46.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:30:46.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:30:46.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:30:46.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:30:46.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:30:46.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:30:46.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:30:46.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:30:46.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:30:46.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:30:46.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:30:46.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:30:46.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:30:46.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:30:46.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:30:46.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:30:46.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:30:46.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:30:46.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:30:46.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:30:46.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:30:46.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:30:46.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:30:46.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:30:46.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:30:46.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:30:46.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=265] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:30:46.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=265] Local address: 172.21.15.161, port 53606 2022-07-03T13:30:46.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=265] Peer address: 172.21.15.101, port 22 2022-07-03T13:30:46.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=265] Connection lost 2022-07-03T13:30:46.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=265] Aborting connection 2022-07-03T13:30:46.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:30:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:30:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: pgmap v5151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:30:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:30:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:30:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:30:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:30:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:30:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:30:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:30:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:30:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:30:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:30:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:30:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:30:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:30:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:30:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:30:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:30:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:30:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:30:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:30:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:30:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:30:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:30:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:30:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:30:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:30:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:30:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:30:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:30:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:30:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:30:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:30:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:30:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:30:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:30:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:30:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:30:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:30:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:30:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:30:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:30:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:30:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:30:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:30:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:30:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:30:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:30:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:30:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:30:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=265] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:30:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=265] Local address: 172.21.15.161, port 53606 2022-07-03T13:30:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=265] Peer address: 172.21.15.101, port 22 2022-07-03T13:30:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=265] Connection lost 2022-07-03T13:30:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=265] Aborting connection 2022-07-03T13:30:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: pgmap v5151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:30:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:30:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:30:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:30:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:30:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:30:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:30:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:30:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:30:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:30:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:30:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:30:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:30:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:30:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:30:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:30:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:30:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:30:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:30:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:30:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:30:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:30:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:30:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:30:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:30:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:30:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:30:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:30:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:30:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:30:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:30:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:30:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:30:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:30:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:30:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:30:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:30:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:30:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:30:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:30:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:30:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:30:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:30:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:30:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:30:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:30:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:30:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:30:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:30:47.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=265] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:30:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=265] Local address: 172.21.15.161, port 53606 2022-07-03T13:30:47.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=265] Peer address: 172.21.15.101, port 22 2022-07-03T13:30:47.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=265] Connection lost 2022-07-03T13:30:47.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=265] Aborting connection 2022-07-03T13:30:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:49 smithi188 ceph-mon[92425]: pgmap v5152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:49 smithi161 ceph-mon[91579]: pgmap v5152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:30:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:30:50.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:30:50.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:30:50.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:30:50.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:50.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:30:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:51 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:30:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:51 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:30:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:51 smithi161 ceph-mon[91579]: pgmap v5153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:51 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:30:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:51 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:30:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:51 smithi188 ceph-mon[92425]: pgmap v5153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:53 smithi161 ceph-mon[91579]: pgmap v5154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:53 smithi188 ceph-mon[92425]: pgmap v5154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:55 smithi161 ceph-mon[91579]: pgmap v5155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:55 smithi188 ceph-mon[92425]: pgmap v5155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:57 smithi161 ceph-mon[91579]: pgmap v5156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:57 smithi188 ceph-mon[92425]: pgmap v5156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:30:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:30:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:30:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:30:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:30:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:30:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:30:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:30:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:30:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:30:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:30:59 smithi161 ceph-mon[91579]: pgmap v5157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:30:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:30:59 smithi188 ceph-mon[92425]: pgmap v5157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:01 smithi161 ceph-mon[91579]: pgmap v5158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:01 smithi188 ceph-mon[92425]: pgmap v5158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:03 smithi161 ceph-mon[91579]: pgmap v5159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:03 smithi188 ceph-mon[92425]: pgmap v5159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:05 smithi161 ceph-mon[91579]: pgmap v5160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:05 smithi188 ceph-mon[92425]: pgmap v5160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:07 smithi161 ceph-mon[91579]: pgmap v5161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:07 smithi188 ceph-mon[92425]: pgmap v5161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:09 smithi161 ceph-mon[91579]: pgmap v5162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:09 smithi188 ceph-mon[92425]: pgmap v5162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:11 smithi161 ceph-mon[91579]: pgmap v5163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:11 smithi188 ceph-mon[92425]: pgmap v5163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:13 smithi161 ceph-mon[91579]: pgmap v5164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:13 smithi188 ceph-mon[92425]: pgmap v5164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:15 smithi161 ceph-mon[91579]: pgmap v5165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:15 smithi188 ceph-mon[92425]: pgmap v5165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:17 smithi161 ceph-mon[91579]: pgmap v5166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:17 smithi188 ceph-mon[92425]: pgmap v5166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:19 smithi161 ceph-mon[91579]: pgmap v5167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:19 smithi188 ceph-mon[92425]: pgmap v5167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:21 smithi161 ceph-mon[91579]: pgmap v5168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:21 smithi188 ceph-mon[92425]: pgmap v5168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:23 smithi161 ceph-mon[91579]: pgmap v5169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:23 smithi188 ceph-mon[92425]: pgmap v5169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:25 smithi161 ceph-mon[91579]: pgmap v5170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:25 smithi188 ceph-mon[92425]: pgmap v5170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:27 smithi161 ceph-mon[91579]: pgmap v5171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:27 smithi188 ceph-mon[92425]: pgmap v5171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:29 smithi161 ceph-mon[91579]: pgmap v5172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:29 smithi188 ceph-mon[92425]: pgmap v5172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:31 smithi161 ceph-mon[91579]: pgmap v5173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:31 smithi188 ceph-mon[92425]: pgmap v5173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:33 smithi188 ceph-mon[92425]: pgmap v5174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:33 smithi161 ceph-mon[91579]: pgmap v5174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:31:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:35 smithi188 ceph-mon[92425]: pgmap v5175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:35 smithi161 ceph-mon[91579]: pgmap v5175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:37 smithi188 ceph-mon[92425]: pgmap v5176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:37 smithi161 ceph-mon[91579]: pgmap v5176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:39 smithi188 ceph-mon[92425]: pgmap v5177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:39 smithi161 ceph-mon[91579]: pgmap v5177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:41 smithi188 ceph-mon[92425]: pgmap v5178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:41 smithi161 ceph-mon[91579]: pgmap v5178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:43 smithi188 ceph-mon[92425]: pgmap v5179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:43 smithi161 ceph-mon[91579]: pgmap v5179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:45 smithi188 ceph-mon[92425]: pgmap v5180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:45 smithi161 ceph-mon[91579]: pgmap v5180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:47 smithi188 ceph-mon[92425]: pgmap v5181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:47 smithi161 ceph-mon[91579]: pgmap v5181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:49.420 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-mon[91579]: pgmap v5182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:49.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:49 smithi188 ceph-mon[92425]: pgmap v5182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:49.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:31:49.420+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:31:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:31:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:31:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:31:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:31:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:31:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:31:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:31:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:31:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:31:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:31:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:31:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:31:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:31:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:31:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:31:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:31:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:31:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:31:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:31:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:31:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:31:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:31:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:31:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:31:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:31:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:31:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:31:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:31:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:31:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:31:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:31:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:31:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:31:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:31:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:31:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:31:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:31:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:31:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:31:49.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:31:49.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:31:49.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:31:49.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:31:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:31:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=267] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:31:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=267] Local address: 172.21.15.161, port 53610 2022-07-03T13:31:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=267] Peer address: 172.21.15.101, port 22 2022-07-03T13:31:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=267] Connection lost 2022-07-03T13:31:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=267] Aborting connection 2022-07-03T13:31:49.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:31:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:31:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:31:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:31:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:31:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:31:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:31:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:31:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:51 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:31:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:51 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:31:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:31:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:31:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:31:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:31:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:31:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:31:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:31:51.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:31:51.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:31:51.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:31:51.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:31:51.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:31:51.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:31:51.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:31:51.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:31:51.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:31:51.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:31:51.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:31:51.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:31:51.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:31:51.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:31:51.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:31:51.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:31:51.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:31:51.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:31:51.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:31:51.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:31:51.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:31:51.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:31:51.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:31:51.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:31:51.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:31:51.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:31:51.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:31:51.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:31:51.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:31:51.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:31:51.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:31:51.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:31:51.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:31:51.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:31:51.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:31:51.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:31:51.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:31:51.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=267] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:31:51.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=267] Local address: 172.21.15.161, port 53610 2022-07-03T13:31:51.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=267] Peer address: 172.21.15.101, port 22 2022-07-03T13:31:51.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=267] Connection lost 2022-07-03T13:31:51.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=267] Aborting connection 2022-07-03T13:31:51.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:51 smithi161 ceph-mon[91579]: pgmap v5183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:51 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:31:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:51 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:31:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:31:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:31:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:31:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:31:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:31:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:31:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:31:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:31:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:31:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:31:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:31:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:31:51.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:31:51.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:31:51.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:31:51.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:31:51.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:31:51.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:31:51.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:31:51.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:31:51.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:31:51.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:31:51.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:31:51.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:31:51.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:31:51.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:31:51.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:31:51.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:31:51.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:31:51.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:31:51.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:31:51.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:31:51.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:31:51.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:31:51.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:31:51.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:31:51.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:31:51.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:31:51.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:31:51.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:31:51.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:31:51.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:31:51.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:31:51.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:31:51.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=267] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:31:51.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=267] Local address: 172.21.15.161, port 53610 2022-07-03T13:31:51.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=267] Peer address: 172.21.15.101, port 22 2022-07-03T13:31:51.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=267] Connection lost 2022-07-03T13:31:51.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=267] Aborting connection 2022-07-03T13:31:51.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:51 smithi188 ceph-mon[92425]: pgmap v5183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:53 smithi161 ceph-mon[91579]: pgmap v5184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:53 smithi188 ceph-mon[92425]: pgmap v5184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:55 smithi161 ceph-mon[91579]: pgmap v5185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:55 smithi188 ceph-mon[92425]: pgmap v5185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:57 smithi161 ceph-mon[91579]: pgmap v5186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:57 smithi188 ceph-mon[92425]: pgmap v5186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:31:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:31:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:31:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:31:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:31:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:31:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:31:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:31:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:31:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:31:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:31:59 smithi161 ceph-mon[91579]: pgmap v5187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:31:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:31:59 smithi188 ceph-mon[92425]: pgmap v5187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:01 smithi161 ceph-mon[91579]: pgmap v5188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:01 smithi188 ceph-mon[92425]: pgmap v5188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:03 smithi161 ceph-mon[91579]: pgmap v5189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:03 smithi188 ceph-mon[92425]: pgmap v5189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:05 smithi161 ceph-mon[91579]: pgmap v5190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:05 smithi188 ceph-mon[92425]: pgmap v5190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:07 smithi161 ceph-mon[91579]: pgmap v5191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:07 smithi188 ceph-mon[92425]: pgmap v5191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:09 smithi161 ceph-mon[91579]: pgmap v5192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:09 smithi188 ceph-mon[92425]: pgmap v5192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:11 smithi161 ceph-mon[91579]: pgmap v5193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:11 smithi188 ceph-mon[92425]: pgmap v5193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:13 smithi161 ceph-mon[91579]: pgmap v5194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:13 smithi188 ceph-mon[92425]: pgmap v5194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:15 smithi161 ceph-mon[91579]: pgmap v5195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:15 smithi188 ceph-mon[92425]: pgmap v5195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:17 smithi161 ceph-mon[91579]: pgmap v5196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:17 smithi188 ceph-mon[92425]: pgmap v5196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:19 smithi161 ceph-mon[91579]: pgmap v5197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:19 smithi188 ceph-mon[92425]: pgmap v5197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:21 smithi161 ceph-mon[91579]: pgmap v5198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:21 smithi188 ceph-mon[92425]: pgmap v5198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:23 smithi161 ceph-mon[91579]: pgmap v5199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:23 smithi188 ceph-mon[92425]: pgmap v5199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:25 smithi161 ceph-mon[91579]: pgmap v5200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:25 smithi188 ceph-mon[92425]: pgmap v5200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:27 smithi161 ceph-mon[91579]: pgmap v5201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:27 smithi188 ceph-mon[92425]: pgmap v5201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:29 smithi161 ceph-mon[91579]: pgmap v5202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:29 smithi188 ceph-mon[92425]: pgmap v5202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:31 smithi161 ceph-mon[91579]: pgmap v5203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:31 smithi188 ceph-mon[92425]: pgmap v5203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:33 smithi161 ceph-mon[91579]: pgmap v5204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:33 smithi188 ceph-mon[92425]: pgmap v5204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:35 smithi161 ceph-mon[91579]: pgmap v5205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:35 smithi188 ceph-mon[92425]: pgmap v5205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:37 smithi161 ceph-mon[91579]: pgmap v5206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:37 smithi188 ceph-mon[92425]: pgmap v5206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:39 smithi161 ceph-mon[91579]: pgmap v5207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:39 smithi188 ceph-mon[92425]: pgmap v5207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:41 smithi161 ceph-mon[91579]: pgmap v5208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:41 smithi188 ceph-mon[92425]: pgmap v5208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:43 smithi161 ceph-mon[91579]: pgmap v5209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:43 smithi188 ceph-mon[92425]: pgmap v5209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:45 smithi161 ceph-mon[91579]: pgmap v5210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:45 smithi188 ceph-mon[92425]: pgmap v5210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:47 smithi161 ceph-mon[91579]: pgmap v5211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:47 smithi188 ceph-mon[92425]: pgmap v5211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:49 smithi161 ceph-mon[91579]: pgmap v5212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:49 smithi188 ceph-mon[92425]: pgmap v5212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:32:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:32:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:32:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:32:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:32:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:32:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:51 smithi161 ceph-mon[91579]: pgmap v5213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:51 smithi188 ceph-mon[92425]: pgmap v5213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:53 smithi161 ceph-mon[91579]: pgmap v5214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:53 smithi188 ceph-mon[92425]: pgmap v5214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:55 smithi161 ceph-mon[91579]: pgmap v5215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:55 smithi188 ceph-mon[92425]: pgmap v5215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:57 smithi161 ceph-mon[91579]: pgmap v5216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:57 smithi188 ceph-mon[92425]: pgmap v5216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:32:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:32:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:32:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:32:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:32:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:32:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:32:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:32:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:32:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:32:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:32:59 smithi161 ceph-mon[91579]: pgmap v5217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:32:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:32:59 smithi188 ceph-mon[92425]: pgmap v5217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:01 smithi161 ceph-mon[91579]: pgmap v5218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:01 smithi188 ceph-mon[92425]: pgmap v5218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:03 smithi161 ceph-mon[91579]: pgmap v5219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:03 smithi188 ceph-mon[92425]: pgmap v5219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:05 smithi161 ceph-mon[91579]: pgmap v5220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:05 smithi188 ceph-mon[92425]: pgmap v5220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:07 smithi161 ceph-mon[91579]: pgmap v5221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:07 smithi188 ceph-mon[92425]: pgmap v5221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:09 smithi161 ceph-mon[91579]: pgmap v5222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:09 smithi188 ceph-mon[92425]: pgmap v5222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:11 smithi161 ceph-mon[91579]: pgmap v5223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:11 smithi188 ceph-mon[92425]: pgmap v5223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:13 smithi161 ceph-mon[91579]: pgmap v5224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:13 smithi188 ceph-mon[92425]: pgmap v5224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:15 smithi161 ceph-mon[91579]: pgmap v5225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:15 smithi188 ceph-mon[92425]: pgmap v5225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:17 smithi161 ceph-mon[91579]: pgmap v5226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:17 smithi188 ceph-mon[92425]: pgmap v5226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-mon[91579]: pgmap v5227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:19 smithi188 ceph-mon[92425]: pgmap v5227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:20.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:33:19.923+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:33:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:33:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:33:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:33:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:33:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:33:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:33:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:33:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:33:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:33:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:33:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:33:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:33:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:33:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:33:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:33:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:33:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:33:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:33:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:33:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:33:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:33:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:33:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:33:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:33:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:33:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:33:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:33:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:33:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:33:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:33:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:33:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:33:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:33:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:33:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:33:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:33:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:33:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:33:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:33:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:33:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:33:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:33:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:33:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:33:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=269] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:33:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=269] Local address: 172.21.15.161, port 53614 2022-07-03T13:33:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=269] Peer address: 172.21.15.101, port 22 2022-07-03T13:33:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=269] Connection lost 2022-07-03T13:33:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=269] Aborting connection 2022-07-03T13:33:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:33:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:33:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:21 smithi161 ceph-mon[91579]: pgmap v5228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:33:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:33:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:33:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:33:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:33:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:33:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:33:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:33:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:33:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:33:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:33:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:33:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:33:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:33:21.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:33:21.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:33:21.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:33:21.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:33:21.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:33:21.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:33:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:33:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:33:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:33:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:33:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:33:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:33:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:33:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:33:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:33:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:33:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:33:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:33:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:33:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:33:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:33:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:33:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:33:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:33:21.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:33:21.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:33:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:33:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:33:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:33:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:33:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:33:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:33:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=269] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:33:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=269] Local address: 172.21.15.161, port 53614 2022-07-03T13:33:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=269] Peer address: 172.21.15.101, port 22 2022-07-03T13:33:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=269] Connection lost 2022-07-03T13:33:21.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=269] Aborting connection 2022-07-03T13:33:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:21 smithi188 ceph-mon[92425]: pgmap v5228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:33:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:33:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:33:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:33:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:33:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:33:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:33:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:33:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:33:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:33:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:33:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:33:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:33:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:33:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:33:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:33:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:33:21.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:33:21.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:33:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:33:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:33:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:33:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:33:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:33:21.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:33:21.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:33:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:33:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:33:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:33:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:33:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:33:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:33:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:33:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:33:21.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:33:21.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:33:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:33:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:33:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:33:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:33:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:33:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:33:21.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:33:21.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:33:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:33:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:33:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=269] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:33:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=269] Local address: 172.21.15.161, port 53614 2022-07-03T13:33:21.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=269] Peer address: 172.21.15.101, port 22 2022-07-03T13:33:21.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=269] Connection lost 2022-07-03T13:33:21.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=269] Aborting connection 2022-07-03T13:33:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:23 smithi161 ceph-mon[91579]: pgmap v5229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:23 smithi188 ceph-mon[92425]: pgmap v5229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:25 smithi161 ceph-mon[91579]: pgmap v5230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:25 smithi188 ceph-mon[92425]: pgmap v5230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:27 smithi161 ceph-mon[91579]: pgmap v5231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:27 smithi188 ceph-mon[92425]: pgmap v5231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:29 smithi161 ceph-mon[91579]: pgmap v5232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:29 smithi188 ceph-mon[92425]: pgmap v5232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:31 smithi161 ceph-mon[91579]: pgmap v5233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:31 smithi188 ceph-mon[92425]: pgmap v5233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:33 smithi161 ceph-mon[91579]: pgmap v5234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:33 smithi188 ceph-mon[92425]: pgmap v5234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:35 smithi161 ceph-mon[91579]: pgmap v5235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:35 smithi188 ceph-mon[92425]: pgmap v5235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:37 smithi161 ceph-mon[91579]: pgmap v5236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:37 smithi188 ceph-mon[92425]: pgmap v5236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:39 smithi188 ceph-mon[92425]: pgmap v5237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:39 smithi161 ceph-mon[91579]: pgmap v5237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:40 smithi188 ceph-mon[92425]: pgmap v5238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:40.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:40 smithi161 ceph-mon[91579]: pgmap v5238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:42 smithi188 ceph-mon[92425]: pgmap v5239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:42 smithi161 ceph-mon[91579]: pgmap v5239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:44 smithi161 ceph-mon[91579]: pgmap v5240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:44 smithi188 ceph-mon[92425]: pgmap v5240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:46 smithi161 ceph-mon[91579]: pgmap v5241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:46 smithi188 ceph-mon[92425]: pgmap v5241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:48 smithi161 ceph-mon[91579]: pgmap v5242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:48 smithi188 ceph-mon[92425]: pgmap v5242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:50 smithi161 ceph-mon[91579]: pgmap v5243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:50 smithi188 ceph-mon[92425]: pgmap v5243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:52 smithi161 ceph-mon[91579]: pgmap v5244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:52 smithi188 ceph-mon[92425]: pgmap v5244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:54 smithi161 ceph-mon[91579]: pgmap v5245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:54 smithi188 ceph-mon[92425]: pgmap v5245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:56 smithi161 ceph-mon[91579]: pgmap v5246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:56 smithi188 ceph-mon[92425]: pgmap v5246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:33:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:33:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:33:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:33:58 smithi161 ceph-mon[91579]: pgmap v5247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:33:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:33:58 smithi188 ceph-mon[92425]: pgmap v5247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:00 smithi161 ceph-mon[91579]: pgmap v5248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:00 smithi188 ceph-mon[92425]: pgmap v5248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:02 smithi161 ceph-mon[91579]: pgmap v5249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:02 smithi188 ceph-mon[92425]: pgmap v5249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:04 smithi161 ceph-mon[91579]: pgmap v5250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:04 smithi188 ceph-mon[92425]: pgmap v5250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:06 smithi161 ceph-mon[91579]: pgmap v5251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:06 smithi188 ceph-mon[92425]: pgmap v5251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:08 smithi161 ceph-mon[91579]: pgmap v5252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:08 smithi188 ceph-mon[92425]: pgmap v5252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:10 smithi161 ceph-mon[91579]: pgmap v5253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:10 smithi188 ceph-mon[92425]: pgmap v5253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:12 smithi161 ceph-mon[91579]: pgmap v5254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:12 smithi188 ceph-mon[92425]: pgmap v5254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:14 smithi161 ceph-mon[91579]: pgmap v5255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:14 smithi188 ceph-mon[92425]: pgmap v5255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:16 smithi161 ceph-mon[91579]: pgmap v5256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:16 smithi188 ceph-mon[92425]: pgmap v5256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-mon[91579]: pgmap v5257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:19 smithi188 ceph-mon[92425]: pgmap v5257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:34:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:34:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:34:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:34:19.958+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:34:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:34:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:34:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:34:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:34:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:34:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:34:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:34:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:34:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:34:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:34:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:34:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:34:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:34:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:34:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:34:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:34:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:34:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:34:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:34:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:34:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:34:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:34:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:34:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:34:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:34:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:34:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:34:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:34:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:34:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:34:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:34:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:34:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:34:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:34:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:34:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:34:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:34:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:34:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:34:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:34:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:34:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:34:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:34:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:34:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=271] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:34:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=271] Local address: 172.21.15.161, port 53618 2022-07-03T13:34:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=271] Peer address: 172.21.15.101, port 22 2022-07-03T13:34:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=271] Connection lost 2022-07-03T13:34:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=271] Aborting connection 2022-07-03T13:34:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:34:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:34:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:34:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:34:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:34:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:21 smithi161 ceph-mon[91579]: pgmap v5258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:34:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:34:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:34:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:34:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:34:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:34:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:34:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:34:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:34:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:34:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:34:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:34:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:34:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:34:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:34:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:34:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:34:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:34:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:34:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:34:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:34:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:34:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:34:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:34:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:34:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:34:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:34:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:34:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:34:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:34:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:34:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:34:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:34:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:34:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:34:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:34:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:34:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:34:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:34:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:34:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:34:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:34:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:34:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:34:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:34:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:34:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=271] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:34:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=271] Local address: 172.21.15.161, port 53618 2022-07-03T13:34:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=271] Peer address: 172.21.15.101, port 22 2022-07-03T13:34:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=271] Connection lost 2022-07-03T13:34:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=271] Aborting connection 2022-07-03T13:34:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:21 smithi188 ceph-mon[92425]: pgmap v5258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:34:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:34:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:34:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:34:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:34:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:34:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:34:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:34:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:34:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:34:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:34:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:34:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:34:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:34:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:34:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:34:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:34:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:34:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:34:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:34:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:34:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:34:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:34:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:34:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:34:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:34:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:34:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:34:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:34:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:34:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:34:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:34:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:34:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:34:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:34:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:34:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:34:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:34:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:34:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:34:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:34:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:34:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:34:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:34:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:34:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:34:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=271] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:34:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=271] Local address: 172.21.15.161, port 53618 2022-07-03T13:34:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=271] Peer address: 172.21.15.101, port 22 2022-07-03T13:34:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=271] Connection lost 2022-07-03T13:34:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=271] Aborting connection 2022-07-03T13:34:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:23 smithi161 ceph-mon[91579]: pgmap v5259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:23 smithi188 ceph-mon[92425]: pgmap v5259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:25 smithi161 ceph-mon[91579]: pgmap v5260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:25 smithi188 ceph-mon[92425]: pgmap v5260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:27 smithi161 ceph-mon[91579]: pgmap v5261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:27 smithi188 ceph-mon[92425]: pgmap v5261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:29 smithi161 ceph-mon[91579]: pgmap v5262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:29 smithi188 ceph-mon[92425]: pgmap v5262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:31 smithi161 ceph-mon[91579]: pgmap v5263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:31 smithi188 ceph-mon[92425]: pgmap v5263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:33 smithi161 ceph-mon[91579]: pgmap v5264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:33 smithi188 ceph-mon[92425]: pgmap v5264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:35 smithi161 ceph-mon[91579]: pgmap v5265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:35 smithi188 ceph-mon[92425]: pgmap v5265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:37 smithi161 ceph-mon[91579]: pgmap v5266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:37 smithi188 ceph-mon[92425]: pgmap v5266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:39 smithi161 ceph-mon[91579]: pgmap v5267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:39 smithi188 ceph-mon[92425]: pgmap v5267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:41 smithi161 ceph-mon[91579]: pgmap v5268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:41 smithi188 ceph-mon[92425]: pgmap v5268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:43 smithi161 ceph-mon[91579]: pgmap v5269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:43 smithi188 ceph-mon[92425]: pgmap v5269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:45 smithi161 ceph-mon[91579]: pgmap v5270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:45 smithi188 ceph-mon[92425]: pgmap v5270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:47 smithi161 ceph-mon[91579]: pgmap v5271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:47 smithi188 ceph-mon[92425]: pgmap v5271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:49 smithi161 ceph-mon[91579]: pgmap v5272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:49 smithi188 ceph-mon[92425]: pgmap v5272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:51 smithi161 ceph-mon[91579]: pgmap v5273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:51 smithi188 ceph-mon[92425]: pgmap v5273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:53 smithi161 ceph-mon[91579]: pgmap v5274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:53 smithi188 ceph-mon[92425]: pgmap v5274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:55 smithi161 ceph-mon[91579]: pgmap v5275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:55 smithi188 ceph-mon[92425]: pgmap v5275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:57 smithi161 ceph-mon[91579]: pgmap v5276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:57 smithi188 ceph-mon[92425]: pgmap v5276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:34:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:34:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:34:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:34:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:34:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:34:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:34:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:34:59 smithi161 ceph-mon[91579]: pgmap v5277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:34:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:34:59 smithi188 ceph-mon[92425]: pgmap v5277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:01 smithi161 ceph-mon[91579]: pgmap v5278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:01 smithi188 ceph-mon[92425]: pgmap v5278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:03 smithi161 ceph-mon[91579]: pgmap v5279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:03 smithi188 ceph-mon[92425]: pgmap v5279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:05 smithi161 ceph-mon[91579]: pgmap v5280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:05 smithi188 ceph-mon[92425]: pgmap v5280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:07 smithi161 ceph-mon[91579]: pgmap v5281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:07 smithi188 ceph-mon[92425]: pgmap v5281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:09 smithi161 ceph-mon[91579]: pgmap v5282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:09 smithi188 ceph-mon[92425]: pgmap v5282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:11 smithi161 ceph-mon[91579]: pgmap v5283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:11 smithi188 ceph-mon[92425]: pgmap v5283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:13 smithi161 ceph-mon[91579]: pgmap v5284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:13 smithi188 ceph-mon[92425]: pgmap v5284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:15 smithi161 ceph-mon[91579]: pgmap v5285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:15 smithi188 ceph-mon[92425]: pgmap v5285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:17 smithi188 ceph-mon[92425]: pgmap v5286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:17 smithi161 ceph-mon[91579]: pgmap v5286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:19 smithi188 ceph-mon[92425]: pgmap v5287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:19 smithi161 ceph-mon[91579]: pgmap v5287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:21 smithi188 ceph-mon[92425]: pgmap v5288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:35:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:35:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:35:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:21 smithi161 ceph-mon[91579]: pgmap v5288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:35:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:35:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:35:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:23 smithi188 ceph-mon[92425]: pgmap v5289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:23 smithi161 ceph-mon[91579]: pgmap v5289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:25 smithi188 ceph-mon[92425]: pgmap v5290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:25 smithi161 ceph-mon[91579]: pgmap v5290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:27 smithi188 ceph-mon[92425]: pgmap v5291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:27 smithi161 ceph-mon[91579]: pgmap v5291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:29 smithi188 ceph-mon[92425]: pgmap v5292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:29 smithi161 ceph-mon[91579]: pgmap v5292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:31 smithi188 ceph-mon[92425]: pgmap v5293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:31 smithi161 ceph-mon[91579]: pgmap v5293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:33 smithi161 ceph-mon[91579]: pgmap v5294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:33 smithi188 ceph-mon[92425]: pgmap v5294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:35 smithi161 ceph-mon[91579]: pgmap v5295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:35 smithi188 ceph-mon[92425]: pgmap v5295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:37 smithi161 ceph-mon[91579]: pgmap v5296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:37 smithi188 ceph-mon[92425]: pgmap v5296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:39 smithi161 ceph-mon[91579]: pgmap v5297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:39 smithi188 ceph-mon[92425]: pgmap v5297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:41 smithi161 ceph-mon[91579]: pgmap v5298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:41 smithi188 ceph-mon[92425]: pgmap v5298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:43 smithi161 ceph-mon[91579]: pgmap v5299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:43 smithi188 ceph-mon[92425]: pgmap v5299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:45 smithi161 ceph-mon[91579]: pgmap v5300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:45 smithi188 ceph-mon[92425]: pgmap v5300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:47 smithi161 ceph-mon[91579]: pgmap v5301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:47 smithi188 ceph-mon[92425]: pgmap v5301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:49 smithi161 ceph-mon[91579]: pgmap v5302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:49 smithi188 ceph-mon[92425]: pgmap v5302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:50.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:35:50.458+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:35:50.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:35:50.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:35:50.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:35:50.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:35:50.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:35:50.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:35:50.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:35:50.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:35:50.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:35:50.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:35:50.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:35:50.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:35:50.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:35:50.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:35:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:35:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:35:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:35:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:35:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:35:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:35:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:35:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:35:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:35:50.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:35:50.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:35:50.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:35:50.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:35:50.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:35:50.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:35:50.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:35:50.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:35:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:35:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:35:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:35:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:35:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:35:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:35:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:35:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:35:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:35:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:35:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:35:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:35:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:35:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=273] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:35:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=273] Local address: 172.21.15.161, port 53622 2022-07-03T13:35:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=273] Peer address: 172.21.15.101, port 22 2022-07-03T13:35:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=273] Connection lost 2022-07-03T13:35:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=273] Aborting connection 2022-07-03T13:35:50.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:35:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:35:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:51 smithi161 ceph-mon[91579]: pgmap v5303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:51 smithi188 ceph-mon[92425]: pgmap v5303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:52 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:35:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:52 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:35:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:35:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:35:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:35:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:35:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:35:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:35:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:35:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:35:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:35:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:35:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:35:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:35:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:35:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:35:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:35:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:35:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:35:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:35:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:35:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:35:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:35:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:35:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:35:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:35:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:35:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:35:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:35:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:35:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:35:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:35:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:35:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:35:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:35:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:35:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:35:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:35:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:35:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:35:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:35:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:35:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:35:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:35:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:35:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:35:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=273] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:35:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=273] Local address: 172.21.15.161, port 53622 2022-07-03T13:35:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=273] Peer address: 172.21.15.101, port 22 2022-07-03T13:35:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=273] Connection lost 2022-07-03T13:35:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=273] Aborting connection 2022-07-03T13:35:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:52 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:35:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:52 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:35:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:35:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:35:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:35:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:35:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:35:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:35:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:35:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:35:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:35:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:35:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:35:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:35:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:35:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:35:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:35:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:35:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:35:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:35:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:35:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:35:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:35:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:35:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:35:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:35:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:35:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:35:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:35:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:35:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:35:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:35:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:35:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:35:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:35:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:35:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:35:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:35:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:35:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:35:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:35:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:35:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:35:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:35:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:35:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:35:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=273] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:35:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=273] Local address: 172.21.15.161, port 53622 2022-07-03T13:35:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=273] Peer address: 172.21.15.101, port 22 2022-07-03T13:35:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=273] Connection lost 2022-07-03T13:35:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=273] Aborting connection 2022-07-03T13:35:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:53 smithi161 ceph-mon[91579]: pgmap v5304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:53 smithi188 ceph-mon[92425]: pgmap v5304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:55 smithi161 ceph-mon[91579]: pgmap v5305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:55 smithi188 ceph-mon[92425]: pgmap v5305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:57 smithi161 ceph-mon[91579]: pgmap v5306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:57 smithi188 ceph-mon[92425]: pgmap v5306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:35:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:35:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:35:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:35:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:35:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:35:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:35:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:35:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:35:59 smithi161 ceph-mon[91579]: pgmap v5307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:35:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:35:59 smithi188 ceph-mon[92425]: pgmap v5307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:01 smithi161 ceph-mon[91579]: pgmap v5308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:01 smithi188 ceph-mon[92425]: pgmap v5308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:03 smithi161 ceph-mon[91579]: pgmap v5309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:03 smithi188 ceph-mon[92425]: pgmap v5309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:05 smithi161 ceph-mon[91579]: pgmap v5310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:05 smithi188 ceph-mon[92425]: pgmap v5310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:07 smithi161 ceph-mon[91579]: pgmap v5311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:07 smithi188 ceph-mon[92425]: pgmap v5311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:09 smithi161 ceph-mon[91579]: pgmap v5312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:09 smithi188 ceph-mon[92425]: pgmap v5312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:11 smithi161 ceph-mon[91579]: pgmap v5313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:11 smithi188 ceph-mon[92425]: pgmap v5313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:13 smithi161 ceph-mon[91579]: pgmap v5314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:13 smithi188 ceph-mon[92425]: pgmap v5314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:15 smithi161 ceph-mon[91579]: pgmap v5315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:15 smithi188 ceph-mon[92425]: pgmap v5315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:17 smithi161 ceph-mon[91579]: pgmap v5316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:17 smithi188 ceph-mon[92425]: pgmap v5316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:19 smithi161 ceph-mon[91579]: pgmap v5317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:19 smithi188 ceph-mon[92425]: pgmap v5317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:21 smithi161 ceph-mon[91579]: pgmap v5318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:21 smithi188 ceph-mon[92425]: pgmap v5318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:23 smithi161 ceph-mon[91579]: pgmap v5319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:23 smithi188 ceph-mon[92425]: pgmap v5319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:25 smithi161 ceph-mon[91579]: pgmap v5320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:25 smithi188 ceph-mon[92425]: pgmap v5320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:27 smithi161 ceph-mon[91579]: pgmap v5321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:27 smithi188 ceph-mon[92425]: pgmap v5321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:29 smithi161 ceph-mon[91579]: pgmap v5322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:29 smithi188 ceph-mon[92425]: pgmap v5322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:31 smithi161 ceph-mon[91579]: pgmap v5323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:31 smithi188 ceph-mon[92425]: pgmap v5323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:33 smithi161 ceph-mon[91579]: pgmap v5324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:33 smithi188 ceph-mon[92425]: pgmap v5324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:35 smithi161 ceph-mon[91579]: pgmap v5325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:35 smithi188 ceph-mon[92425]: pgmap v5325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:37 smithi161 ceph-mon[91579]: pgmap v5326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:37 smithi188 ceph-mon[92425]: pgmap v5326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:39 smithi161 ceph-mon[91579]: pgmap v5327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:39 smithi188 ceph-mon[92425]: pgmap v5327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:41 smithi161 ceph-mon[91579]: pgmap v5328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:41 smithi188 ceph-mon[92425]: pgmap v5328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:43 smithi161 ceph-mon[91579]: pgmap v5329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:43 smithi188 ceph-mon[92425]: pgmap v5329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:45 smithi161 ceph-mon[91579]: pgmap v5330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:45 smithi188 ceph-mon[92425]: pgmap v5330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:47 smithi161 ceph-mon[91579]: pgmap v5331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:47 smithi188 ceph-mon[92425]: pgmap v5331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:49 smithi161 ceph-mon[91579]: pgmap v5332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:49 smithi188 ceph-mon[92425]: pgmap v5332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:36:50.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:36:50.494+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:36:50.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:36:50.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:36:50.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:36:50.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:36:50.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:36:50.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:36:50.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:36:50.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:36:50.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:36:50.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:36:50.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:36:50.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:36:50.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:36:50.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:36:50.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:36:50.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:36:50.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:36:50.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:36:50.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:36:50.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:36:50.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:36:50.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:36:50.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:36:50.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:36:50.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:36:50.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:36:50.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:36:50.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:36:50.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:36:50.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:36:50.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:36:50.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:36:50.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:36:50.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:36:50.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:36:50.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:36:50.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:36:50.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:36:50.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:36:50.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:36:50.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:36:50.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:36:50.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:36:50.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:36:50.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=275] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:36:50.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=275] Local address: 172.21.15.161, port 53630 2022-07-03T13:36:50.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=275] Peer address: 172.21.15.101, port 22 2022-07-03T13:36:50.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=275] Connection lost 2022-07-03T13:36:50.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=275] Aborting connection 2022-07-03T13:36:50.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:36:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:36:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:36:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:51 smithi161 ceph-mon[91579]: pgmap v5333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:36:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:36:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:51 smithi188 ceph-mon[92425]: pgmap v5333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:36:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:36:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:36:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:52 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:36:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:52 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:36:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:36:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:36:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:36:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:36:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:36:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:36:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:36:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:36:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:36:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:36:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:36:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:36:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:36:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:36:52.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:36:52.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:36:52.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:36:52.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:36:52.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:36:52.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:36:52.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:36:52.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:36:52.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:36:52.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:36:52.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:36:52.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:36:52.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:36:52.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:36:52.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:36:52.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:36:52.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:36:52.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:36:52.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:36:52.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:36:52.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:36:52.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:36:52.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:36:52.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:36:52.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:36:52.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:36:52.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:36:52.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:36:52.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:36:52.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:36:52.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=275] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:36:52.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=275] Local address: 172.21.15.161, port 53630 2022-07-03T13:36:52.665 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=275] Peer address: 172.21.15.101, port 22 2022-07-03T13:36:52.666 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=275] Connection lost 2022-07-03T13:36:52.667 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=275] Aborting connection 2022-07-03T13:36:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:52 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:36:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:52 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:36:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:36:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:36:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:36:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:36:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:36:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:36:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:36:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:36:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:36:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:36:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:36:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:36:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:36:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:36:52.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:36:52.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:36:52.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:36:52.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:36:52.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:36:52.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:36:52.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:36:52.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:36:52.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:36:52.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:36:52.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:36:52.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:36:52.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:36:52.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:36:52.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:36:52.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:36:52.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:36:52.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:36:52.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:36:52.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:36:52.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:36:52.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:36:52.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:36:52.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:36:52.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:36:52.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:36:52.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:36:52.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:36:52.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:36:52.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:36:52.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=275] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:36:52.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=275] Local address: 172.21.15.161, port 53630 2022-07-03T13:36:52.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=275] Peer address: 172.21.15.101, port 22 2022-07-03T13:36:52.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=275] Connection lost 2022-07-03T13:36:52.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=275] Aborting connection 2022-07-03T13:36:53.578 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:53 smithi161 ceph-mon[91579]: pgmap v5334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:53.579 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:53 smithi188 ceph-mon[92425]: pgmap v5334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:54.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:54 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T13:36:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:36:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:36:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:54 smithi161 ceph-mon[91579]: pgmap v5335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:36:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:54 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T13:36:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:36:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:36:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:54 smithi188 ceph-mon[92425]: pgmap v5335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:36:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:56 smithi161 ceph-mon[91579]: pgmap v5336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:56 smithi188 ceph-mon[92425]: pgmap v5336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:36:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:36:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:36:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:36:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:36:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:36:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:36:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:36:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:36:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:36:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:58 smithi161 ceph-mon[91579]: pgmap v5337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:36:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:58 smithi188 ceph-mon[92425]: pgmap v5337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:00 smithi161 ceph-mon[91579]: pgmap v5338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:00 smithi188 ceph-mon[92425]: pgmap v5338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:02 smithi161 ceph-mon[91579]: pgmap v5339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:02 smithi188 ceph-mon[92425]: pgmap v5339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:04 smithi161 ceph-mon[91579]: pgmap v5340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:04 smithi188 ceph-mon[92425]: pgmap v5340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:06 smithi161 ceph-mon[91579]: pgmap v5341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:06 smithi188 ceph-mon[92425]: pgmap v5341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:08 smithi161 ceph-mon[91579]: pgmap v5342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:08 smithi188 ceph-mon[92425]: pgmap v5342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:11 smithi161 ceph-mon[91579]: pgmap v5343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:11 smithi188 ceph-mon[92425]: pgmap v5343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:13 smithi161 ceph-mon[91579]: pgmap v5344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:13 smithi188 ceph-mon[92425]: pgmap v5344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:15 smithi161 ceph-mon[91579]: pgmap v5345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:15 smithi188 ceph-mon[92425]: pgmap v5345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:17 smithi161 ceph-mon[91579]: pgmap v5346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:17 smithi188 ceph-mon[92425]: pgmap v5346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:19 smithi161 ceph-mon[91579]: pgmap v5347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:19 smithi188 ceph-mon[92425]: pgmap v5347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:21 smithi161 ceph-mon[91579]: pgmap v5348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:21 smithi188 ceph-mon[92425]: pgmap v5348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:23 smithi161 ceph-mon[91579]: pgmap v5349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:23 smithi188 ceph-mon[92425]: pgmap v5349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:25 smithi161 ceph-mon[91579]: pgmap v5350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:25 smithi188 ceph-mon[92425]: pgmap v5350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:27 smithi161 ceph-mon[91579]: pgmap v5351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:27 smithi188 ceph-mon[92425]: pgmap v5351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:29 smithi161 ceph-mon[91579]: pgmap v5352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:29 smithi188 ceph-mon[92425]: pgmap v5352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:31 smithi161 ceph-mon[91579]: pgmap v5353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:31 smithi188 ceph-mon[92425]: pgmap v5353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:33 smithi161 ceph-mon[91579]: pgmap v5354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:33 smithi188 ceph-mon[92425]: pgmap v5354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:35 smithi161 ceph-mon[91579]: pgmap v5355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:35 smithi188 ceph-mon[92425]: pgmap v5355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:37 smithi161 ceph-mon[91579]: pgmap v5356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:37 smithi188 ceph-mon[92425]: pgmap v5356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:39 smithi161 ceph-mon[91579]: pgmap v5357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:39 smithi188 ceph-mon[92425]: pgmap v5357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:41 smithi161 ceph-mon[91579]: pgmap v5358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:41 smithi188 ceph-mon[92425]: pgmap v5358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:43 smithi161 ceph-mon[91579]: pgmap v5359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:43 smithi188 ceph-mon[92425]: pgmap v5359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:45 smithi161 ceph-mon[91579]: pgmap v5360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:45 smithi188 ceph-mon[92425]: pgmap v5360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:47 smithi161 ceph-mon[91579]: pgmap v5361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:47 smithi188 ceph-mon[92425]: pgmap v5361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:49 smithi161 ceph-mon[91579]: pgmap v5362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:49 smithi188 ceph-mon[92425]: pgmap v5362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:51 smithi161 ceph-mon[91579]: pgmap v5363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:51 smithi188 ceph-mon[92425]: pgmap v5363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:53 smithi161 ceph-mon[91579]: pgmap v5364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:53 smithi188 ceph-mon[92425]: pgmap v5364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:37:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:37:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:37:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:37:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:37:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:37:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:55 smithi161 ceph-mon[91579]: pgmap v5365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:55 smithi188 ceph-mon[92425]: pgmap v5365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:37:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:57 smithi161 ceph-mon[91579]: pgmap v5366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:57 smithi188 ceph-mon[92425]: pgmap v5366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:37:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:37:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:37:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:37:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:37:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:37:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:37:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:37:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:37:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:37:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:37:59 smithi161 ceph-mon[91579]: pgmap v5367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:37:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:37:59 smithi188 ceph-mon[92425]: pgmap v5367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:01 smithi161 ceph-mon[91579]: pgmap v5368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:01 smithi188 ceph-mon[92425]: pgmap v5368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:03 smithi161 ceph-mon[91579]: pgmap v5369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:03 smithi188 ceph-mon[92425]: pgmap v5369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:05 smithi161 ceph-mon[91579]: pgmap v5370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:05 smithi188 ceph-mon[92425]: pgmap v5370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:07 smithi188 ceph-mon[92425]: pgmap v5371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:07 smithi161 ceph-mon[91579]: pgmap v5371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:09 smithi188 ceph-mon[92425]: pgmap v5372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:09 smithi161 ceph-mon[91579]: pgmap v5372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:11 smithi188 ceph-mon[92425]: pgmap v5373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:11 smithi161 ceph-mon[91579]: pgmap v5373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:13 smithi188 ceph-mon[92425]: pgmap v5374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:13 smithi161 ceph-mon[91579]: pgmap v5374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:15 smithi188 ceph-mon[92425]: pgmap v5375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:15 smithi161 ceph-mon[91579]: pgmap v5375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:17 smithi188 ceph-mon[92425]: pgmap v5376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:17 smithi161 ceph-mon[91579]: pgmap v5376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:19 smithi188 ceph-mon[92425]: pgmap v5377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:19 smithi161 ceph-mon[91579]: pgmap v5377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:21 smithi161 ceph-mon[91579]: pgmap v5378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:21 smithi188 ceph-mon[92425]: pgmap v5378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-mon[91579]: pgmap v5379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:23 smithi188 ceph-mon[92425]: pgmap v5379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:24.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:38:23.670+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:38:24.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:38:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:38:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:38:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:38:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:38:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:38:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:38:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:38:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:38:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:38:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:38:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:38:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:38:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:38:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:38:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:38:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:38:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:38:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:38:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:38:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:38:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:38:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:38:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:38:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:38:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:38:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:38:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:38:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:38:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:38:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:38:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:38:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:38:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:38:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:38:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:38:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:38:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:38:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:38:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:38:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:38:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:38:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:38:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:38:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=277] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:38:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=277] Local address: 172.21.15.161, port 53670 2022-07-03T13:38:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=277] Peer address: 172.21.15.101, port 22 2022-07-03T13:38:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=277] Connection lost 2022-07-03T13:38:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=277] Aborting connection 2022-07-03T13:38:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:38:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:38:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:38:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:38:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:38:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:38:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:38:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:38:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:38:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:38:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:38:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:38:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:38:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:38:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:38:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:38:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:38:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:38:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:38:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:38:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:38:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:38:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:38:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:38:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:38:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:38:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:38:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:38:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:38:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:38:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:38:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:38:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:38:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:38:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:38:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:38:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:38:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:38:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:38:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:38:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:38:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:38:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:38:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:38:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:38:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:38:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:38:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:38:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=277] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:38:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=277] Local address: 172.21.15.161, port 53670 2022-07-03T13:38:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=277] Peer address: 172.21.15.101, port 22 2022-07-03T13:38:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=277] Connection lost 2022-07-03T13:38:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=277] Aborting connection 2022-07-03T13:38:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:25 smithi161 ceph-mon[91579]: pgmap v5380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:38:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:38:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:38:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:38:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:38:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:38:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:38:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:38:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:38:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:38:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:38:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:38:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:38:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:38:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:38:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:38:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:38:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:38:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:38:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:38:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:38:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:38:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:38:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:38:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:38:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:38:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:38:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:38:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:38:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:38:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:38:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:38:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:38:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:38:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:38:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:38:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:38:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:38:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:38:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:38:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:38:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:38:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:38:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:38:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:38:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:38:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=277] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:38:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=277] Local address: 172.21.15.161, port 53670 2022-07-03T13:38:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=277] Peer address: 172.21.15.101, port 22 2022-07-03T13:38:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=277] Connection lost 2022-07-03T13:38:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=277] Aborting connection 2022-07-03T13:38:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:25 smithi188 ceph-mon[92425]: pgmap v5380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:27 smithi161 ceph-mon[91579]: pgmap v5381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:27 smithi188 ceph-mon[92425]: pgmap v5381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:29 smithi161 ceph-mon[91579]: pgmap v5382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:29 smithi188 ceph-mon[92425]: pgmap v5382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:31 smithi161 ceph-mon[91579]: pgmap v5383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:31 smithi188 ceph-mon[92425]: pgmap v5383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:33 smithi161 ceph-mon[91579]: pgmap v5384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:33 smithi188 ceph-mon[92425]: pgmap v5384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:35 smithi161 ceph-mon[91579]: pgmap v5385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:35 smithi188 ceph-mon[92425]: pgmap v5385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:37 smithi161 ceph-mon[91579]: pgmap v5386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:37 smithi188 ceph-mon[92425]: pgmap v5386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:39 smithi161 ceph-mon[91579]: pgmap v5387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:39 smithi188 ceph-mon[92425]: pgmap v5387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:41 smithi161 ceph-mon[91579]: pgmap v5388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:41 smithi188 ceph-mon[92425]: pgmap v5388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:43 smithi161 ceph-mon[91579]: pgmap v5389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:43 smithi188 ceph-mon[92425]: pgmap v5389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:45 smithi161 ceph-mon[91579]: pgmap v5390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:45 smithi188 ceph-mon[92425]: pgmap v5390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:47 smithi161 ceph-mon[91579]: pgmap v5391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:47 smithi188 ceph-mon[92425]: pgmap v5391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:49 smithi161 ceph-mon[91579]: pgmap v5392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:49 smithi188 ceph-mon[92425]: pgmap v5392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:51 smithi161 ceph-mon[91579]: pgmap v5393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:51 smithi188 ceph-mon[92425]: pgmap v5393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:53 smithi161 ceph-mon[91579]: pgmap v5394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:53 smithi188 ceph-mon[92425]: pgmap v5394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:55 smithi161 ceph-mon[91579]: pgmap v5395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:55 smithi188 ceph-mon[92425]: pgmap v5395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:38:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:57 smithi161 ceph-mon[91579]: pgmap v5396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:57 smithi188 ceph-mon[92425]: pgmap v5396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:38:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:38:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:38:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:38:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:38:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:38:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:38:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:38:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:38:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:38:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:38:59 smithi161 ceph-mon[91579]: pgmap v5397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:38:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:38:59 smithi188 ceph-mon[92425]: pgmap v5397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:01 smithi161 ceph-mon[91579]: pgmap v5398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:01 smithi188 ceph-mon[92425]: pgmap v5398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:03 smithi161 ceph-mon[91579]: pgmap v5399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:03 smithi188 ceph-mon[92425]: pgmap v5399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:05 smithi161 ceph-mon[91579]: pgmap v5400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:05 smithi188 ceph-mon[92425]: pgmap v5400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:07 smithi161 ceph-mon[91579]: pgmap v5401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:07 smithi188 ceph-mon[92425]: pgmap v5401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:09 smithi161 ceph-mon[91579]: pgmap v5402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:09 smithi188 ceph-mon[92425]: pgmap v5402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:11 smithi161 ceph-mon[91579]: pgmap v5403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:11 smithi188 ceph-mon[92425]: pgmap v5403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:13 smithi161 ceph-mon[91579]: pgmap v5404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:13 smithi188 ceph-mon[92425]: pgmap v5404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:15 smithi161 ceph-mon[91579]: pgmap v5405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:15 smithi188 ceph-mon[92425]: pgmap v5405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:17 smithi161 ceph-mon[91579]: pgmap v5406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:17 smithi188 ceph-mon[92425]: pgmap v5406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:19 smithi161 ceph-mon[91579]: pgmap v5407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:19 smithi188 ceph-mon[92425]: pgmap v5407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:21 smithi161 ceph-mon[91579]: pgmap v5408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:21 smithi188 ceph-mon[92425]: pgmap v5408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-mon[91579]: pgmap v5409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:23 smithi188 ceph-mon[92425]: pgmap v5409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:24.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:39:23.705+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:39:24.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:39:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:39:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:39:24.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:39:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:39:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:39:24.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:39:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:39:24.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:39:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:39:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:39:24.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:39:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:39:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:39:24.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:39:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:39:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:39:24.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:39:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:39:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:39:24.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:39:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:39:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:39:24.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:39:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:39:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:39:24.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:39:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:39:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:39:24.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:39:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:39:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:39:24.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:39:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:39:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:39:24.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:39:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:39:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:39:24.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:39:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:39:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:39:24.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:39:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:39:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:39:24.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=279] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:39:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=279] Local address: 172.21.15.161, port 53724 2022-07-03T13:39:24.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=279] Peer address: 172.21.15.101, port 22 2022-07-03T13:39:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=279] Connection lost 2022-07-03T13:39:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=279] Aborting connection 2022-07-03T13:39:24.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:39:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:39:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:39:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:39:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:39:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:39:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:39:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:39:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:39:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:39:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:39:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:39:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:39:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:39:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:39:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:39:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:39:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:39:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:39:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:39:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:39:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:39:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:39:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:39:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:39:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:39:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:39:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:39:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:39:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:39:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:39:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:39:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:39:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:39:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:39:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:39:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:39:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:39:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:39:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:39:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:39:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:39:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:39:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:39:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:39:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:39:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:39:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:39:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:39:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:39:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:39:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:39:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:39:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:39:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:39:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=279] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:39:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=279] Local address: 172.21.15.161, port 53724 2022-07-03T13:39:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=279] Peer address: 172.21.15.101, port 22 2022-07-03T13:39:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=279] Connection lost 2022-07-03T13:39:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=279] Aborting connection 2022-07-03T13:39:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:25 smithi161 ceph-mon[91579]: pgmap v5410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:39:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:39:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:39:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:39:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:39:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:39:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:39:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:39:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:39:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:39:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:39:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:39:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:39:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:39:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:39:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:39:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:39:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:39:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:39:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:39:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:39:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:39:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:39:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:39:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:39:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:39:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:39:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:39:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:39:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:39:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:39:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:39:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:39:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:39:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:39:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:39:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:39:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:39:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:39:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:39:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:39:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:39:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:39:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:39:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:39:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:39:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=279] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:39:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=279] Local address: 172.21.15.161, port 53724 2022-07-03T13:39:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=279] Peer address: 172.21.15.101, port 22 2022-07-03T13:39:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=279] Connection lost 2022-07-03T13:39:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=279] Aborting connection 2022-07-03T13:39:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:25 smithi188 ceph-mon[92425]: pgmap v5410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:27 smithi161 ceph-mon[91579]: pgmap v5411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:27 smithi188 ceph-mon[92425]: pgmap v5411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:29 smithi161 ceph-mon[91579]: pgmap v5412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:29 smithi188 ceph-mon[92425]: pgmap v5412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:31 smithi161 ceph-mon[91579]: pgmap v5413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:31 smithi188 ceph-mon[92425]: pgmap v5413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:33 smithi161 ceph-mon[91579]: pgmap v5414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:33 smithi188 ceph-mon[92425]: pgmap v5414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:35 smithi161 ceph-mon[91579]: pgmap v5415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:35 smithi188 ceph-mon[92425]: pgmap v5415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:37 smithi161 ceph-mon[91579]: pgmap v5416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:37 smithi188 ceph-mon[92425]: pgmap v5416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:39 smithi161 ceph-mon[91579]: pgmap v5417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:39 smithi188 ceph-mon[92425]: pgmap v5417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:41 smithi161 ceph-mon[91579]: pgmap v5418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:41 smithi188 ceph-mon[92425]: pgmap v5418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:43 smithi161 ceph-mon[91579]: pgmap v5419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:43 smithi188 ceph-mon[92425]: pgmap v5419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:45 smithi161 ceph-mon[91579]: pgmap v5420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:45 smithi188 ceph-mon[92425]: pgmap v5420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:47 smithi161 ceph-mon[91579]: pgmap v5421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:47 smithi188 ceph-mon[92425]: pgmap v5421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:49 smithi188 ceph-mon[92425]: pgmap v5422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:49 smithi161 ceph-mon[91579]: pgmap v5422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:50.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:51 smithi188 ceph-mon[92425]: pgmap v5423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:51 smithi161 ceph-mon[91579]: pgmap v5423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:53 smithi188 ceph-mon[92425]: pgmap v5424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:53 smithi161 ceph-mon[91579]: pgmap v5424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:55 smithi161 ceph-mon[91579]: pgmap v5425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:55 smithi188 ceph-mon[92425]: pgmap v5425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:39:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:57 smithi161 ceph-mon[91579]: pgmap v5426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:57 smithi188 ceph-mon[92425]: pgmap v5426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:39:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:39:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:39:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:39:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:39:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:39:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:39:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:39:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:39:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:39:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:39:59 smithi161 ceph-mon[91579]: pgmap v5427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:39:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:39:59 smithi188 ceph-mon[92425]: pgmap v5427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:40:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:40:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:01 smithi161 ceph-mon[91579]: pgmap v5428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:01 smithi188 ceph-mon[92425]: pgmap v5428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:03 smithi161 ceph-mon[91579]: pgmap v5429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:03 smithi188 ceph-mon[92425]: pgmap v5429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:05 smithi161 ceph-mon[91579]: pgmap v5430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:05 smithi188 ceph-mon[92425]: pgmap v5430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:07 smithi161 ceph-mon[91579]: pgmap v5431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:07 smithi188 ceph-mon[92425]: pgmap v5431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:09 smithi161 ceph-mon[91579]: pgmap v5432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:09 smithi188 ceph-mon[92425]: pgmap v5432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:11 smithi161 ceph-mon[91579]: pgmap v5433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:11 smithi188 ceph-mon[92425]: pgmap v5433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:13 smithi161 ceph-mon[91579]: pgmap v5434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:13 smithi188 ceph-mon[92425]: pgmap v5434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:15 smithi188 ceph-mon[92425]: pgmap v5435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:15 smithi161 ceph-mon[91579]: pgmap v5435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:17.376 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:17 smithi161 ceph-mon[91579]: pgmap v5436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:17 smithi188 ceph-mon[92425]: pgmap v5436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:19 smithi161 ceph-mon[91579]: pgmap v5437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:19 smithi188 ceph-mon[92425]: pgmap v5437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:21 smithi161 ceph-mon[91579]: pgmap v5438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:21 smithi188 ceph-mon[92425]: pgmap v5438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:23 smithi161 ceph-mon[91579]: pgmap v5439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:23 smithi188 ceph-mon[92425]: pgmap v5439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:24.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:40:24.646+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:40:24.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:40:24.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:40:24.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:40:24.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:40:24.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:40:24.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:40:24.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:40:24.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:40:24.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:40:24.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:40:24.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:40:24.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:40:24.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:40:24.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:40:24.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:40:24.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:40:24.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:40:24.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:40:24.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:40:24.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:40:24.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:40:24.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:40:24.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:40:24.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:40:24.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:40:24.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:40:24.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:40:24.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:40:24.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:40:24.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:40:24.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:40:24.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:40:24.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:40:24.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:40:24.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:40:24.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:40:24.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:40:24.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:40:24.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:40:24.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:40:24.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:40:24.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:40:24.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:40:24.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:40:24.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=281] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:40:24.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=281] Local address: 172.21.15.161, port 53770 2022-07-03T13:40:24.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=281] Peer address: 172.21.15.101, port 22 2022-07-03T13:40:24.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=281] Connection lost 2022-07-03T13:40:24.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=281] Aborting connection 2022-07-03T13:40:24.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:40:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:40:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:25 smithi161 ceph-mon[91579]: pgmap v5440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:40:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:40:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:40:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:25 smithi188 ceph-mon[92425]: pgmap v5440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:40:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:40:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:40:26.240 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:40:26.241 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:40:26.241 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:40:26.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:40:26.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:40:26.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:40:26.243 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:40:26.243 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:40:26.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:40:26.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:40:26.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:40:26.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:40:26.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:40:26.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:40:26.246 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:40:26.246 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:40:26.246 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:40:26.247 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:40:26.247 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:40:26.248 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:40:26.248 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:40:26.248 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:40:26.249 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:40:26.249 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:40:26.249 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:40:26.250 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:40:26.250 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:40:26.250 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:40:26.251 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:40:26.251 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:40:26.251 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:40:26.252 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:40:26.252 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:40:26.252 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:40:26.253 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:40:26.253 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:40:26.253 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:40:26.254 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:40:26.254 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:40:26.254 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:40:26.255 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:40:26.256 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:40:26.256 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:40:26.257 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:40:26.257 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:40:26.257 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:40:26.258 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=281] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:40:26.258 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=281] Local address: 172.21.15.161, port 53770 2022-07-03T13:40:26.259 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=281] Peer address: 172.21.15.101, port 22 2022-07-03T13:40:26.259 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=281] Connection lost 2022-07-03T13:40:26.259 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=281] Aborting connection 2022-07-03T13:40:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:40:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:40:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:40:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:40:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:40:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:40:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:40:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:40:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:40:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:40:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:40:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:40:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:40:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:40:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:40:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:40:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:40:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:40:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:40:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:40:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:40:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:40:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:40:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:40:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:40:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:40:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:40:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:40:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:40:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:40:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:40:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:40:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:40:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:40:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:40:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:40:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:40:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:40:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:40:26.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:40:26.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:40:26.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:40:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:40:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:40:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=281] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:40:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=281] Local address: 172.21.15.161, port 53770 2022-07-03T13:40:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=281] Peer address: 172.21.15.101, port 22 2022-07-03T13:40:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=281] Connection lost 2022-07-03T13:40:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=281] Aborting connection 2022-07-03T13:40:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:27 smithi161 ceph-mon[91579]: pgmap v5441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:27 smithi188 ceph-mon[92425]: pgmap v5441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:40:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:40:28.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:28.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:40:28.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:40:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:29 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T13:40:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:29 smithi161 ceph-mon[91579]: pgmap v5442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:29 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T13:40:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:29 smithi188 ceph-mon[92425]: pgmap v5442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:31 smithi161 ceph-mon[91579]: pgmap v5443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:31 smithi188 ceph-mon[92425]: pgmap v5443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:33 smithi161 ceph-mon[91579]: pgmap v5444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:33 smithi188 ceph-mon[92425]: pgmap v5444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:35 smithi161 ceph-mon[91579]: pgmap v5445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:35 smithi188 ceph-mon[92425]: pgmap v5445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:37 smithi161 ceph-mon[91579]: pgmap v5446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:37 smithi188 ceph-mon[92425]: pgmap v5446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:39 smithi161 ceph-mon[91579]: pgmap v5447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:39 smithi188 ceph-mon[92425]: pgmap v5447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:41 smithi188 ceph-mon[92425]: pgmap v5448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:41 smithi161 ceph-mon[91579]: pgmap v5448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:43 smithi188 ceph-mon[92425]: pgmap v5449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:43 smithi161 ceph-mon[91579]: pgmap v5449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:45 smithi188 ceph-mon[92425]: pgmap v5450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:45 smithi161 ceph-mon[91579]: pgmap v5450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:47 smithi188 ceph-mon[92425]: pgmap v5451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:47 smithi161 ceph-mon[91579]: pgmap v5451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:49 smithi188 ceph-mon[92425]: pgmap v5452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:49 smithi161 ceph-mon[91579]: pgmap v5452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:51 smithi188 ceph-mon[92425]: pgmap v5453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:51 smithi161 ceph-mon[91579]: pgmap v5453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:53 smithi188 ceph-mon[92425]: pgmap v5454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:53 smithi161 ceph-mon[91579]: pgmap v5454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:55 smithi188 ceph-mon[92425]: pgmap v5455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:55 smithi161 ceph-mon[91579]: pgmap v5455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:40:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:57 smithi161 ceph-mon[91579]: pgmap v5456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:57 smithi188 ceph-mon[92425]: pgmap v5456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:40:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:40:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:40:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:40:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:40:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:40:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:40:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:40:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:40:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:40:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:40:59 smithi161 ceph-mon[91579]: pgmap v5457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:40:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:40:59 smithi188 ceph-mon[92425]: pgmap v5457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:01 smithi161 ceph-mon[91579]: pgmap v5458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:01 smithi188 ceph-mon[92425]: pgmap v5458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:03 smithi161 ceph-mon[91579]: pgmap v5459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:03 smithi188 ceph-mon[92425]: pgmap v5459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:05 smithi161 ceph-mon[91579]: pgmap v5460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:05 smithi188 ceph-mon[92425]: pgmap v5460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:07 smithi161 ceph-mon[91579]: pgmap v5461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:07 smithi188 ceph-mon[92425]: pgmap v5461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:09 smithi161 ceph-mon[91579]: pgmap v5462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:09 smithi188 ceph-mon[92425]: pgmap v5462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:11 smithi161 ceph-mon[91579]: pgmap v5463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:11 smithi188 ceph-mon[92425]: pgmap v5463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:13 smithi161 ceph-mon[91579]: pgmap v5464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:13 smithi188 ceph-mon[92425]: pgmap v5464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:15 smithi161 ceph-mon[91579]: pgmap v5465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:15 smithi188 ceph-mon[92425]: pgmap v5465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:17 smithi161 ceph-mon[91579]: pgmap v5466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:17 smithi188 ceph-mon[92425]: pgmap v5466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:19 smithi161 ceph-mon[91579]: pgmap v5467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:19 smithi188 ceph-mon[92425]: pgmap v5467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:21 smithi161 ceph-mon[91579]: pgmap v5468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:21 smithi188 ceph-mon[92425]: pgmap v5468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:23 smithi161 ceph-mon[91579]: pgmap v5469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:23 smithi188 ceph-mon[92425]: pgmap v5469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:25 smithi161 ceph-mon[91579]: pgmap v5470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:25 smithi188 ceph-mon[92425]: pgmap v5470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:27 smithi161 ceph-mon[91579]: pgmap v5471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:27 smithi188 ceph-mon[92425]: pgmap v5471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:41:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:41:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:41:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:41:28.017+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:41:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:41:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:41:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:41:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:41:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:41:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:41:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:41:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:41:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:41:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:41:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:41:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:41:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:41:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:41:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:41:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:41:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:41:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:41:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:41:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:41:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:41:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:41:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:41:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:41:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:41:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:41:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:41:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:41:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:41:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:41:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:41:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:41:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:41:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:41:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:41:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:41:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:41:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:41:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:41:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:41:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:41:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:41:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:41:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:41:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=283] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:41:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=283] Local address: 172.21.15.161, port 53798 2022-07-03T13:41:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=283] Peer address: 172.21.15.101, port 22 2022-07-03T13:41:28.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=283] Connection lost 2022-07-03T13:41:28.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=283] Aborting connection 2022-07-03T13:41:28.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:41:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:41:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:41:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:41:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:41:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:29 smithi161 ceph-mon[91579]: pgmap v5472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:41:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:41:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:41:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:41:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:41:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:41:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:41:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:41:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:41:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:41:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:41:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:41:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:41:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:41:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:41:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:41:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:41:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:41:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:41:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:41:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:41:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:41:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:41:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:41:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:41:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:41:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:41:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:41:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:41:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:41:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:41:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:41:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:41:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:41:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:41:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:41:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:41:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:41:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:41:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:41:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:41:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:41:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:41:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:41:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:41:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:41:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=283] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:41:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=283] Local address: 172.21.15.161, port 53798 2022-07-03T13:41:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=283] Peer address: 172.21.15.101, port 22 2022-07-03T13:41:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=283] Connection lost 2022-07-03T13:41:29.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=283] Aborting connection 2022-07-03T13:41:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:29 smithi188 ceph-mon[92425]: pgmap v5472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:41:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:41:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:41:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:41:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:41:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:41:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:41:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:41:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:41:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:41:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:41:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:41:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:41:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:41:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:41:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:41:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:41:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:41:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:41:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:41:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:41:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:41:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:41:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:41:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:41:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:41:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:41:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:41:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:41:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:41:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:41:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:41:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:41:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:41:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:41:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:41:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:41:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:41:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:41:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:41:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:41:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:41:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:41:29.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:41:29.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:41:29.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:41:29.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=283] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:41:29.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=283] Local address: 172.21.15.161, port 53798 2022-07-03T13:41:29.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=283] Peer address: 172.21.15.101, port 22 2022-07-03T13:41:29.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=283] Connection lost 2022-07-03T13:41:29.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=283] Aborting connection 2022-07-03T13:41:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: pgmap v5473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: pgmap v5473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:41:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:41:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:41:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:41:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:41:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:41:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:41:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:41:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:41:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:41:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:33 smithi161 ceph-mon[91579]: pgmap v5474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:33 smithi188 ceph-mon[92425]: pgmap v5474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:35 smithi161 ceph-mon[91579]: pgmap v5475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:35 smithi188 ceph-mon[92425]: pgmap v5475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:37 smithi161 ceph-mon[91579]: pgmap v5476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:37 smithi188 ceph-mon[92425]: pgmap v5476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:39 smithi161 ceph-mon[91579]: pgmap v5477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:39 smithi188 ceph-mon[92425]: pgmap v5477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:41 smithi161 ceph-mon[91579]: pgmap v5478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:41 smithi188 ceph-mon[92425]: pgmap v5478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:43 smithi161 ceph-mon[91579]: pgmap v5479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:43 smithi188 ceph-mon[92425]: pgmap v5479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:45 smithi161 ceph-mon[91579]: pgmap v5480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:45 smithi188 ceph-mon[92425]: pgmap v5480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:47 smithi161 ceph-mon[91579]: pgmap v5481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:47 smithi188 ceph-mon[92425]: pgmap v5481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:49 smithi161 ceph-mon[91579]: pgmap v5482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:49 smithi188 ceph-mon[92425]: pgmap v5482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:51 smithi161 ceph-mon[91579]: pgmap v5483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:51 smithi188 ceph-mon[92425]: pgmap v5483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:53 smithi161 ceph-mon[91579]: pgmap v5484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:53 smithi188 ceph-mon[92425]: pgmap v5484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:55 smithi161 ceph-mon[91579]: pgmap v5485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:55 smithi188 ceph-mon[92425]: pgmap v5485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:41:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:57 smithi161 ceph-mon[91579]: pgmap v5486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:57 smithi188 ceph-mon[92425]: pgmap v5486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:41:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:41:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:41:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:41:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:41:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:41:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:41:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:41:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:41:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:41:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:41:59 smithi161 ceph-mon[91579]: pgmap v5487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:41:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:41:59 smithi188 ceph-mon[92425]: pgmap v5487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:01 smithi161 ceph-mon[91579]: pgmap v5488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:01 smithi188 ceph-mon[92425]: pgmap v5488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:03 smithi161 ceph-mon[91579]: pgmap v5489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:03 smithi188 ceph-mon[92425]: pgmap v5489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:05 smithi161 ceph-mon[91579]: pgmap v5490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:05 smithi188 ceph-mon[92425]: pgmap v5490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:07 smithi161 ceph-mon[91579]: pgmap v5491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:07 smithi188 ceph-mon[92425]: pgmap v5491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:09 smithi161 ceph-mon[91579]: pgmap v5492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:09 smithi188 ceph-mon[92425]: pgmap v5492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:11 smithi161 ceph-mon[91579]: pgmap v5493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:11 smithi188 ceph-mon[92425]: pgmap v5493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:13 smithi161 ceph-mon[91579]: pgmap v5494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:13 smithi188 ceph-mon[92425]: pgmap v5494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:15 smithi161 ceph-mon[91579]: pgmap v5495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:15 smithi188 ceph-mon[92425]: pgmap v5495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:17 smithi161 ceph-mon[91579]: pgmap v5496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:17 smithi188 ceph-mon[92425]: pgmap v5496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:42:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:19 smithi161 ceph-mon[91579]: pgmap v5497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:19 smithi188 ceph-mon[92425]: pgmap v5497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:21 smithi161 ceph-mon[91579]: pgmap v5498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:21 smithi188 ceph-mon[92425]: pgmap v5498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:23 smithi188 ceph-mon[92425]: pgmap v5499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:23 smithi161 ceph-mon[91579]: pgmap v5499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:25 smithi188 ceph-mon[92425]: pgmap v5500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:25 smithi161 ceph-mon[91579]: pgmap v5500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:27 smithi188 ceph-mon[92425]: pgmap v5501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:27 smithi161 ceph-mon[91579]: pgmap v5501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:29 smithi188 ceph-mon[92425]: pgmap v5502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:29 smithi161 ceph-mon[91579]: pgmap v5502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-mon[91579]: pgmap v5503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:42:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:42:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:42:31.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:42:31.276+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:42:31.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:42:31.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:42:31.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:42:31.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:42:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:42:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:42:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:42:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:42:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:42:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:42:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:42:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:42:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:42:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:42:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:42:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:42:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:42:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:42:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:42:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:42:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:42:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:42:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:42:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:42:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:42:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:42:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:42:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:42:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:42:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:42:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:42:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:42:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:42:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:42:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:42:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:42:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:42:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:42:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:42:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:42:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:42:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:42:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:42:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:42:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=285] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:42:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=285] Local address: 172.21.15.161, port 53828 2022-07-03T13:42:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=285] Peer address: 172.21.15.101, port 22 2022-07-03T13:42:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=285] Connection lost 2022-07-03T13:42:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=285] Aborting connection 2022-07-03T13:42:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:42:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:42:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:31 smithi188 ceph-mon[92425]: pgmap v5503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:42:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:42:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:42:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:42:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:42:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:42:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:42:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:42:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:42:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:42:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:42:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:42:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:42:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:42:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:42:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:42:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:42:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:42:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:42:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:42:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:42:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:42:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:42:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:42:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:42:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:42:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:42:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:42:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:42:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:42:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:42:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:42:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:42:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:42:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:42:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:42:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:42:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:42:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:42:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:42:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:42:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:42:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:42:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:42:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:42:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:42:32.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:42:32.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:42:32.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:42:32.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=285] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:42:32.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=285] Local address: 172.21.15.161, port 53828 2022-07-03T13:42:32.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=285] Peer address: 172.21.15.101, port 22 2022-07-03T13:42:32.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=285] Connection lost 2022-07-03T13:42:32.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=285] Aborting connection 2022-07-03T13:42:32.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:42:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:42:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:42:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:42:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:42:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:42:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:42:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:42:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:42:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:42:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:42:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:42:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:42:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:42:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:42:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:42:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:42:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:42:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:42:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:42:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:42:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:42:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:42:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:42:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:42:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:42:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:42:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:42:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:42:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:42:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:42:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:42:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:42:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:42:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:42:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:42:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:42:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:42:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:42:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:42:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:42:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:42:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:42:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:42:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:42:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:42:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=285] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:42:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=285] Local address: 172.21.15.161, port 53828 2022-07-03T13:42:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=285] Peer address: 172.21.15.101, port 22 2022-07-03T13:42:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=285] Connection lost 2022-07-03T13:42:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=285] Aborting connection 2022-07-03T13:42:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:42:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:33 smithi188 ceph-mon[92425]: pgmap v5504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:33 smithi161 ceph-mon[91579]: pgmap v5504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:35 smithi188 ceph-mon[92425]: pgmap v5505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:35 smithi161 ceph-mon[91579]: pgmap v5505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:37 smithi188 ceph-mon[92425]: pgmap v5506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:37 smithi161 ceph-mon[91579]: pgmap v5506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:39 smithi161 ceph-mon[91579]: pgmap v5507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:39 smithi188 ceph-mon[92425]: pgmap v5507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:41 smithi161 ceph-mon[91579]: pgmap v5508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:41 smithi188 ceph-mon[92425]: pgmap v5508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:43 smithi161 ceph-mon[91579]: pgmap v5509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:43 smithi188 ceph-mon[92425]: pgmap v5509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:45 smithi161 ceph-mon[91579]: pgmap v5510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:45 smithi188 ceph-mon[92425]: pgmap v5510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:47 smithi161 ceph-mon[91579]: pgmap v5511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:47 smithi188 ceph-mon[92425]: pgmap v5511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:49 smithi161 ceph-mon[91579]: pgmap v5512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:49 smithi188 ceph-mon[92425]: pgmap v5512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:51 smithi161 ceph-mon[91579]: pgmap v5513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:51 smithi188 ceph-mon[92425]: pgmap v5513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:53 smithi161 ceph-mon[91579]: pgmap v5514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:53 smithi188 ceph-mon[92425]: pgmap v5514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:55 smithi161 ceph-mon[91579]: pgmap v5515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:55 smithi188 ceph-mon[92425]: pgmap v5515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:42:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:57 smithi161 ceph-mon[91579]: pgmap v5516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:57 smithi188 ceph-mon[92425]: pgmap v5516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:42:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:42:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:42:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:42:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:42:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:42:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:42:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:42:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:42:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:42:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:42:59 smithi161 ceph-mon[91579]: pgmap v5517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:42:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:42:59 smithi188 ceph-mon[92425]: pgmap v5517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:01 smithi161 ceph-mon[91579]: pgmap v5518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:01 smithi188 ceph-mon[92425]: pgmap v5518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:03 smithi161 ceph-mon[91579]: pgmap v5519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:03 smithi188 ceph-mon[92425]: pgmap v5519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:05 smithi161 ceph-mon[91579]: pgmap v5520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:05 smithi188 ceph-mon[92425]: pgmap v5520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:07 smithi161 ceph-mon[91579]: pgmap v5521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:07 smithi188 ceph-mon[92425]: pgmap v5521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:09 smithi161 ceph-mon[91579]: pgmap v5522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:09 smithi188 ceph-mon[92425]: pgmap v5522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:11 smithi161 ceph-mon[91579]: pgmap v5523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:11 smithi188 ceph-mon[92425]: pgmap v5523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:13 smithi161 ceph-mon[91579]: pgmap v5524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:13 smithi188 ceph-mon[92425]: pgmap v5524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:15 smithi161 ceph-mon[91579]: pgmap v5525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:15 smithi188 ceph-mon[92425]: pgmap v5525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:17 smithi161 ceph-mon[91579]: pgmap v5526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:17 smithi188 ceph-mon[92425]: pgmap v5526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:19 smithi161 ceph-mon[91579]: pgmap v5527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:19 smithi188 ceph-mon[92425]: pgmap v5527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:21 smithi161 ceph-mon[91579]: pgmap v5528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:21 smithi188 ceph-mon[92425]: pgmap v5528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:23 smithi161 ceph-mon[91579]: pgmap v5529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:23 smithi188 ceph-mon[92425]: pgmap v5529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:25 smithi161 ceph-mon[91579]: pgmap v5530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:25 smithi188 ceph-mon[92425]: pgmap v5530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:27 smithi161 ceph-mon[91579]: pgmap v5531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:27 smithi188 ceph-mon[92425]: pgmap v5531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:29 smithi188 ceph-mon[92425]: pgmap v5532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:29 smithi161 ceph-mon[91579]: pgmap v5532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:31 smithi188 ceph-mon[92425]: pgmap v5533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:31 smithi161 ceph-mon[91579]: pgmap v5533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:32.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:43:32.207+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:43:32.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:43:32.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:43:32.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:43:32.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:43:32.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:43:32.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:43:32.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:43:32.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:43:32.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:43:32.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:43:32.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:43:32.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:43:32.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:43:32.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:43:32.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:43:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:43:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:43:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:43:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:43:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:43:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:43:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:43:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:43:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:43:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:43:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:43:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:43:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:43:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:43:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:43:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:43:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:43:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:43:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:43:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:43:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:43:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:43:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:43:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:43:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:43:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:43:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:43:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:43:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:43:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=287] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:43:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=287] Local address: 172.21.15.161, port 53856 2022-07-03T13:43:32.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=287] Peer address: 172.21.15.101, port 22 2022-07-03T13:43:32.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=287] Connection lost 2022-07-03T13:43:32.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=287] Aborting connection 2022-07-03T13:43:32.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:43:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:43:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: pgmap v5534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:43:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:43:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:43:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:43:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:43:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:43:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:43:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:43:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:43:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:43:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:43:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:43:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:43:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:43:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:43:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:43:33.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:43:33.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:43:33.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:43:33.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:43:33.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:43:33.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:43:33.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:43:33.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:43:33.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:43:33.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:43:33.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:43:33.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:43:33.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:43:33.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:43:33.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:43:33.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:43:33.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:43:33.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:43:33.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:43:33.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:43:33.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:43:33.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:43:33.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:43:33.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:43:33.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:43:33.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:43:33.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:43:33.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:43:33.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:43:33.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:43:33.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:43:33.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:43:33.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:43:33.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=287] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:43:33.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=287] Local address: 172.21.15.161, port 53856 2022-07-03T13:43:33.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=287] Peer address: 172.21.15.101, port 22 2022-07-03T13:43:33.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=287] Connection lost 2022-07-03T13:43:33.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=287] Aborting connection 2022-07-03T13:43:33.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: pgmap v5534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:43:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:43:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:43:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:43:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:43:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:43:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:43:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:43:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:43:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:43:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:43:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:43:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:43:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:43:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:43:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:43:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:43:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:43:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:43:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:43:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:43:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:43:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:43:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:43:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:43:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:43:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:43:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:43:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:43:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:43:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:43:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:43:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:43:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:43:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:43:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:43:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:43:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:43:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:43:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:43:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:43:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:43:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:43:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:43:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:43:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:43:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=287] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:43:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=287] Local address: 172.21.15.161, port 53856 2022-07-03T13:43:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=287] Peer address: 172.21.15.101, port 22 2022-07-03T13:43:33.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=287] Connection lost 2022-07-03T13:43:33.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=287] Aborting connection 2022-07-03T13:43:33.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:35 smithi188 ceph-mon[92425]: pgmap v5535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:35 smithi161 ceph-mon[91579]: pgmap v5535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:37 smithi188 ceph-mon[92425]: pgmap v5536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:37 smithi161 ceph-mon[91579]: pgmap v5536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:39 smithi188 ceph-mon[92425]: pgmap v5537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:39 smithi161 ceph-mon[91579]: pgmap v5537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:41 smithi188 ceph-mon[92425]: pgmap v5538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:41 smithi161 ceph-mon[91579]: pgmap v5538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:43 smithi188 ceph-mon[92425]: pgmap v5539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:43 smithi161 ceph-mon[91579]: pgmap v5539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:45 smithi161 ceph-mon[91579]: pgmap v5540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:45 smithi188 ceph-mon[92425]: pgmap v5540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:47 smithi161 ceph-mon[91579]: pgmap v5541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:47 smithi188 ceph-mon[92425]: pgmap v5541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:49 smithi161 ceph-mon[91579]: pgmap v5542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:49 smithi188 ceph-mon[92425]: pgmap v5542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:51 smithi161 ceph-mon[91579]: pgmap v5543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:51 smithi188 ceph-mon[92425]: pgmap v5543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:53 smithi161 ceph-mon[91579]: pgmap v5544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:53 smithi188 ceph-mon[92425]: pgmap v5544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:55 smithi161 ceph-mon[91579]: pgmap v5545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:55 smithi188 ceph-mon[92425]: pgmap v5545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:43:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:57 smithi161 ceph-mon[91579]: pgmap v5546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:57 smithi188 ceph-mon[92425]: pgmap v5546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:43:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:43:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:43:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:43:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:43:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:43:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:43:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:43:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:43:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:43:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:43:59 smithi161 ceph-mon[91579]: pgmap v5547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:43:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:43:59 smithi188 ceph-mon[92425]: pgmap v5547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:01 smithi161 ceph-mon[91579]: pgmap v5548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:01 smithi188 ceph-mon[92425]: pgmap v5548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:03 smithi161 ceph-mon[91579]: pgmap v5549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:03 smithi188 ceph-mon[92425]: pgmap v5549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:05 smithi161 ceph-mon[91579]: pgmap v5550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:05 smithi188 ceph-mon[92425]: pgmap v5550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:07 smithi161 ceph-mon[91579]: pgmap v5551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:07 smithi188 ceph-mon[92425]: pgmap v5551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:09 smithi161 ceph-mon[91579]: pgmap v5552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:09 smithi188 ceph-mon[92425]: pgmap v5552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:11 smithi161 ceph-mon[91579]: pgmap v5553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:11 smithi188 ceph-mon[92425]: pgmap v5553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:13 smithi161 ceph-mon[91579]: pgmap v5554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:13 smithi188 ceph-mon[92425]: pgmap v5554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:15 smithi161 ceph-mon[91579]: pgmap v5555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:15 smithi188 ceph-mon[92425]: pgmap v5555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:17 smithi161 ceph-mon[91579]: pgmap v5556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:17 smithi188 ceph-mon[92425]: pgmap v5556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:19 smithi161 ceph-mon[91579]: pgmap v5557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:19 smithi188 ceph-mon[92425]: pgmap v5557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:21 smithi161 ceph-mon[91579]: pgmap v5558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:21 smithi188 ceph-mon[92425]: pgmap v5558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:23 smithi161 ceph-mon[91579]: pgmap v5559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:23 smithi188 ceph-mon[92425]: pgmap v5559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:25 smithi161 ceph-mon[91579]: pgmap v5560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:25 smithi188 ceph-mon[92425]: pgmap v5560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:27 smithi161 ceph-mon[91579]: pgmap v5561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:27 smithi188 ceph-mon[92425]: pgmap v5561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:29 smithi161 ceph-mon[91579]: pgmap v5562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:29 smithi188 ceph-mon[92425]: pgmap v5562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:31 smithi161 ceph-mon[91579]: pgmap v5563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:31 smithi188 ceph-mon[92425]: pgmap v5563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:33.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:44:32.718+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:44:33.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:44:33.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:44:33.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:44:33.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:44:33.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:44:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:44:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:44:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:44:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:44:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:44:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:44:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:44:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:44:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:44:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:44:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:44:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:44:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:44:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:44:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:44:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:44:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:44:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:44:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:44:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:44:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:44:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:44:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:44:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:44:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:44:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:44:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:44:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:44:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:44:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:44:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:44:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:44:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:44:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:44:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:44:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:44:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:44:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:44:33.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:44:33.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=289] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:44:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=289] Local address: 172.21.15.161, port 53884 2022-07-03T13:44:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=289] Peer address: 172.21.15.101, port 22 2022-07-03T13:44:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=289] Connection lost 2022-07-03T13:44:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=289] Aborting connection 2022-07-03T13:44:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:44:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:44:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:33 smithi161 ceph-mon[91579]: pgmap v5564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:44:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:44:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:44:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:33 smithi188 ceph-mon[92425]: pgmap v5564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:44:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:44:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:44:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:44:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:44:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:44:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:44:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:44:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:44:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:44:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:44:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:44:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:44:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:44:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:44:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:44:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:44:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:44:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:44:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:44:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:44:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:44:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:44:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:44:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:44:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:44:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:44:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:44:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:44:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:44:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:44:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:44:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:44:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:44:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:44:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:44:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:44:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:44:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:44:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:44:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:44:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:44:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:44:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:44:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:44:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:44:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:44:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:44:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:44:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=289] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:44:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=289] Local address: 172.21.15.161, port 53884 2022-07-03T13:44:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=289] Peer address: 172.21.15.101, port 22 2022-07-03T13:44:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=289] Connection lost 2022-07-03T13:44:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=289] Aborting connection 2022-07-03T13:44:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:44:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:44:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:44:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:44:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:44:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:44:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:44:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:44:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:44:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:44:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:44:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:44:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:44:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:44:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:44:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:44:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:44:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:44:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:44:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:44:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:44:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:44:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:44:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:44:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:44:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:44:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:44:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:44:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:44:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:44:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:44:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:44:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:44:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:44:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:44:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:44:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:44:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:44:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:44:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:44:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:44:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:44:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:44:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:44:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:44:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:44:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=289] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:44:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=289] Local address: 172.21.15.161, port 53884 2022-07-03T13:44:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=289] Peer address: 172.21.15.101, port 22 2022-07-03T13:44:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=289] Connection lost 2022-07-03T13:44:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=289] Aborting connection 2022-07-03T13:44:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:35 smithi161 ceph-mon[91579]: pgmap v5565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:35 smithi188 ceph-mon[92425]: pgmap v5565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:37 smithi161 ceph-mon[91579]: pgmap v5566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:37 smithi188 ceph-mon[92425]: pgmap v5566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:39 smithi161 ceph-mon[91579]: pgmap v5567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:39 smithi188 ceph-mon[92425]: pgmap v5567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:41 smithi161 ceph-mon[91579]: pgmap v5568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:41 smithi188 ceph-mon[92425]: pgmap v5568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:43 smithi161 ceph-mon[91579]: pgmap v5569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:43 smithi188 ceph-mon[92425]: pgmap v5569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:45 smithi161 ceph-mon[91579]: pgmap v5570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:45 smithi188 ceph-mon[92425]: pgmap v5570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:47 smithi161 ceph-mon[91579]: pgmap v5571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:47 smithi188 ceph-mon[92425]: pgmap v5571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:49 smithi161 ceph-mon[91579]: pgmap v5572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:49 smithi188 ceph-mon[92425]: pgmap v5572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:51 smithi161 ceph-mon[91579]: pgmap v5573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:51 smithi188 ceph-mon[92425]: pgmap v5573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:53 smithi161 ceph-mon[91579]: pgmap v5574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:53 smithi188 ceph-mon[92425]: pgmap v5574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:55 smithi161 ceph-mon[91579]: pgmap v5575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:55 smithi188 ceph-mon[92425]: pgmap v5575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:44:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:57 smithi161 ceph-mon[91579]: pgmap v5576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:57 smithi188 ceph-mon[92425]: pgmap v5576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:44:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:44:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:44:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:44:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:44:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:44:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:44:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:44:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:44:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:44:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:44:59 smithi161 ceph-mon[91579]: pgmap v5577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:44:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:44:59 smithi188 ceph-mon[92425]: pgmap v5577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:01 smithi161 ceph-mon[91579]: pgmap v5578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:01 smithi188 ceph-mon[92425]: pgmap v5578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:03 smithi161 ceph-mon[91579]: pgmap v5579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:03 smithi188 ceph-mon[92425]: pgmap v5579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:05 smithi161 ceph-mon[91579]: pgmap v5580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:05 smithi188 ceph-mon[92425]: pgmap v5580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:07 smithi161 ceph-mon[91579]: pgmap v5581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:07 smithi188 ceph-mon[92425]: pgmap v5581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:09 smithi161 ceph-mon[91579]: pgmap v5582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:09 smithi188 ceph-mon[92425]: pgmap v5582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:11 smithi161 ceph-mon[91579]: pgmap v5583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:11 smithi188 ceph-mon[92425]: pgmap v5583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:13 smithi161 ceph-mon[91579]: pgmap v5584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:13 smithi188 ceph-mon[92425]: pgmap v5584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:15 smithi161 ceph-mon[91579]: pgmap v5585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:15 smithi188 ceph-mon[92425]: pgmap v5585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:17 smithi161 ceph-mon[91579]: pgmap v5586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:17 smithi188 ceph-mon[92425]: pgmap v5586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:19 smithi188 ceph-mon[92425]: pgmap v5587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:19 smithi161 ceph-mon[91579]: pgmap v5587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:20.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:21 smithi188 ceph-mon[92425]: pgmap v5588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:21 smithi161 ceph-mon[91579]: pgmap v5588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:23 smithi188 ceph-mon[92425]: pgmap v5589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:23 smithi161 ceph-mon[91579]: pgmap v5589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:25 smithi188 ceph-mon[92425]: pgmap v5590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:25 smithi161 ceph-mon[91579]: pgmap v5590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:27 smithi188 ceph-mon[92425]: pgmap v5591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:27 smithi161 ceph-mon[91579]: pgmap v5591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:29 smithi188 ceph-mon[92425]: pgmap v5592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:29 smithi161 ceph-mon[91579]: pgmap v5592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:31 smithi161 ceph-mon[91579]: pgmap v5593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:31 smithi188 ceph-mon[92425]: pgmap v5593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-mon[91579]: pgmap v5594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:45:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:45:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:45:33.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:45:33.217+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:45:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:45:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:45:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:45:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:45:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:45:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:45:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:45:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:45:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:45:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:45:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:45:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:45:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:45:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:45:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:45:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:45:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:45:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:45:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:45:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:45:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:45:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:45:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:45:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:45:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:45:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:45:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:45:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:45:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:45:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:45:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:45:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:45:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:45:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:45:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:45:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:45:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:45:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:45:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:45:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:45:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:45:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:45:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:45:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:45:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=291] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:45:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=291] Local address: 172.21.15.161, port 53912 2022-07-03T13:45:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=291] Peer address: 172.21.15.101, port 22 2022-07-03T13:45:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=291] Connection lost 2022-07-03T13:45:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=291] Aborting connection 2022-07-03T13:45:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:45:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:45:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:33 smithi188 ceph-mon[92425]: pgmap v5594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:45:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:45:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:45:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:45:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:45:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:45:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:45:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:45:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:45:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:45:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:45:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:45:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:45:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:45:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:45:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:45:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:45:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:45:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:45:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:45:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:45:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:45:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:45:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:45:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:45:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:45:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:45:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:45:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:45:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:45:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:45:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:45:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:45:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:45:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:45:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:45:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:45:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:45:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:45:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:45:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:45:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:45:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:45:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:45:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:45:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:45:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:45:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:45:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:45:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=291] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:45:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=291] Local address: 172.21.15.161, port 53912 2022-07-03T13:45:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=291] Peer address: 172.21.15.101, port 22 2022-07-03T13:45:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=291] Connection lost 2022-07-03T13:45:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=291] Aborting connection 2022-07-03T13:45:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:45:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:45:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:45:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:45:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:45:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:45:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:45:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:45:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:45:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:45:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:45:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:45:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:45:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:45:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:45:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:45:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:45:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:45:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:45:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:45:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:45:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:45:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:45:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:45:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:45:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:45:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:45:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:45:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:45:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:45:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:45:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:45:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:45:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:45:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:45:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:45:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:45:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:45:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:45:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:45:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:45:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:45:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:45:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:45:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:45:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:45:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=291] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:45:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=291] Local address: 172.21.15.161, port 53912 2022-07-03T13:45:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=291] Peer address: 172.21.15.101, port 22 2022-07-03T13:45:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=291] Connection lost 2022-07-03T13:45:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=291] Aborting connection 2022-07-03T13:45:35.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:35 smithi161 ceph-mon[91579]: pgmap v5595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:35 smithi188 ceph-mon[92425]: pgmap v5595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:37 smithi161 ceph-mon[91579]: pgmap v5596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:37 smithi188 ceph-mon[92425]: pgmap v5596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:39 smithi161 ceph-mon[91579]: pgmap v5597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:39 smithi188 ceph-mon[92425]: pgmap v5597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:40.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:40.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:41 smithi161 ceph-mon[91579]: pgmap v5598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:41 smithi188 ceph-mon[92425]: pgmap v5598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:43 smithi161 ceph-mon[91579]: pgmap v5599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:43 smithi188 ceph-mon[92425]: pgmap v5599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:45 smithi161 ceph-mon[91579]: pgmap v5600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:45 smithi188 ceph-mon[92425]: pgmap v5600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:47 smithi161 ceph-mon[91579]: pgmap v5601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:47 smithi188 ceph-mon[92425]: pgmap v5601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:49 smithi161 ceph-mon[91579]: pgmap v5602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:49 smithi188 ceph-mon[92425]: pgmap v5602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:51 smithi161 ceph-mon[91579]: pgmap v5603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:51 smithi188 ceph-mon[92425]: pgmap v5603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:53 smithi161 ceph-mon[91579]: pgmap v5604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:53 smithi188 ceph-mon[92425]: pgmap v5604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:55 smithi161 ceph-mon[91579]: pgmap v5605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:55 smithi188 ceph-mon[92425]: pgmap v5605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:45:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:57 smithi161 ceph-mon[91579]: pgmap v5606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:57 smithi188 ceph-mon[92425]: pgmap v5606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:45:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:45:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:45:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:45:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:45:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:45:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:45:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:45:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:45:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:45:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:45:59 smithi161 ceph-mon[91579]: pgmap v5607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:45:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:45:59 smithi188 ceph-mon[92425]: pgmap v5607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:01 smithi161 ceph-mon[91579]: pgmap v5608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:01 smithi188 ceph-mon[92425]: pgmap v5608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:03 smithi161 ceph-mon[91579]: pgmap v5609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:03.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:03 smithi188 ceph-mon[92425]: pgmap v5609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:05 smithi161 ceph-mon[91579]: pgmap v5610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:05 smithi188 ceph-mon[92425]: pgmap v5610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:07 smithi161 ceph-mon[91579]: pgmap v5611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:07 smithi188 ceph-mon[92425]: pgmap v5611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:09 smithi161 ceph-mon[91579]: pgmap v5612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:09 smithi188 ceph-mon[92425]: pgmap v5612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:11 smithi161 ceph-mon[91579]: pgmap v5613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:11 smithi188 ceph-mon[92425]: pgmap v5613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:12.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:12 smithi161 ceph-mon[91579]: pgmap v5614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:12 smithi188 ceph-mon[92425]: pgmap v5614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:15 smithi161 ceph-mon[91579]: pgmap v5615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:15 smithi188 ceph-mon[92425]: pgmap v5615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:17 smithi161 ceph-mon[91579]: pgmap v5616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:17 smithi188 ceph-mon[92425]: pgmap v5616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:19 smithi161 ceph-mon[91579]: pgmap v5617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:19 smithi188 ceph-mon[92425]: pgmap v5617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:21 smithi161 ceph-mon[91579]: pgmap v5618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:21 smithi188 ceph-mon[92425]: pgmap v5618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:23 smithi161 ceph-mon[91579]: pgmap v5619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:23 smithi188 ceph-mon[92425]: pgmap v5619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:25 smithi161 ceph-mon[91579]: pgmap v5620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:25 smithi188 ceph-mon[92425]: pgmap v5620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:27 smithi161 ceph-mon[91579]: pgmap v5621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:27 smithi188 ceph-mon[92425]: pgmap v5621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:29 smithi188 ceph-mon[92425]: pgmap v5622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:29 smithi161 ceph-mon[91579]: pgmap v5622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:31 smithi188 ceph-mon[92425]: pgmap v5623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:31 smithi161 ceph-mon[91579]: pgmap v5623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:33 smithi188 ceph-mon[92425]: pgmap v5624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:33 smithi161 ceph-mon[91579]: pgmap v5624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:46:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:46:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:46:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:46:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:46:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:46:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:35 smithi188 ceph-mon[92425]: pgmap v5625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:35 smithi161 ceph-mon[91579]: pgmap v5625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:37 smithi188 ceph-mon[92425]: pgmap v5626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:37 smithi161 ceph-mon[91579]: pgmap v5626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:39 smithi188 ceph-mon[92425]: pgmap v5627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:39 smithi161 ceph-mon[91579]: pgmap v5627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:41 smithi161 ceph-mon[91579]: pgmap v5628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:41 smithi188 ceph-mon[92425]: pgmap v5628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:43 smithi161 ceph-mon[91579]: pgmap v5629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:43 smithi188 ceph-mon[92425]: pgmap v5629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:45 smithi161 ceph-mon[91579]: pgmap v5630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:45 smithi188 ceph-mon[92425]: pgmap v5630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:47 smithi161 ceph-mon[91579]: pgmap v5631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:47 smithi188 ceph-mon[92425]: pgmap v5631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:49 smithi161 ceph-mon[91579]: pgmap v5632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:49 smithi188 ceph-mon[92425]: pgmap v5632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:51 smithi161 ceph-mon[91579]: pgmap v5633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:51 smithi188 ceph-mon[92425]: pgmap v5633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:53 smithi161 ceph-mon[91579]: pgmap v5634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:53 smithi188 ceph-mon[92425]: pgmap v5634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:55 smithi161 ceph-mon[91579]: pgmap v5635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:55 smithi188 ceph-mon[92425]: pgmap v5635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:46:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:57 smithi161 ceph-mon[91579]: pgmap v5636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:57 smithi188 ceph-mon[92425]: pgmap v5636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:46:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:46:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:46:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:46:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:46:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:46:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:46:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:46:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:46:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:46:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:46:59 smithi161 ceph-mon[91579]: pgmap v5637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:46:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:46:59 smithi188 ceph-mon[92425]: pgmap v5637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:01 smithi161 ceph-mon[91579]: pgmap v5638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:01 smithi188 ceph-mon[92425]: pgmap v5638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-mon[91579]: pgmap v5639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:03 smithi188 ceph-mon[92425]: pgmap v5639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:04.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:47:03.715+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:47:04.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:47:04.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:47:04.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:47:04.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:47:04.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:47:04.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:47:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:47:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:47:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:47:04.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:47:04.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:47:04.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:47:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:47:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:47:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:47:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:47:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:47:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:47:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:47:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:47:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:47:04.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:47:04.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:47:04.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:47:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:47:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:47:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:47:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:47:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:47:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:47:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:47:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:47:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:47:04.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:47:04.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:47:04.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:47:04.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:47:04.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:47:04.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:47:04.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:47:04.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:47:04.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:47:04.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:47:04.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:47:04.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=293] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:47:04.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=293] Local address: 172.21.15.161, port 53952 2022-07-03T13:47:04.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=293] Peer address: 172.21.15.101, port 22 2022-07-03T13:47:04.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=293] Connection lost 2022-07-03T13:47:04.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=293] Aborting connection 2022-07-03T13:47:04.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:47:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:47:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:47:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:47:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:47:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:47:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:47:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:47:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:47:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:47:05.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:47:05.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:47:05.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:47:05.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:47:05.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:47:05.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:47:05.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:47:05.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:47:05.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:47:05.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:47:05.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:47:05.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:47:05.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:47:05.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:47:05.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:47:05.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:47:05.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:47:05.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:47:05.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:47:05.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:47:05.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:47:05.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:47:05.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:47:05.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:47:05.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:47:05.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:47:05.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:47:05.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:47:05.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:47:05.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:47:05.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:47:05.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:47:05.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:47:05.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:47:05.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:47:05.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:47:05.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:47:05.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:47:05.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=293] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:47:05.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=293] Local address: 172.21.15.161, port 53952 2022-07-03T13:47:05.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=293] Peer address: 172.21.15.101, port 22 2022-07-03T13:47:05.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=293] Connection lost 2022-07-03T13:47:05.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=293] Aborting connection 2022-07-03T13:47:05.660 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:05 smithi161 ceph-mon[91579]: pgmap v5640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:05.660 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:47:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:47:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:47:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:47:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:47:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:47:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:47:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:47:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:47:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:47:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:47:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:47:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:47:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:47:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:47:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:47:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:47:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:47:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:47:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:47:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:47:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:47:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:47:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:47:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:47:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:47:05.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:47:05.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:47:05.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:47:05.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:47:05.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:47:05.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:47:05.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:47:05.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:47:05.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:47:05.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:47:05.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:47:05.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:47:05.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:47:05.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:47:05.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:47:05.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:47:05.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:47:05.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:47:05.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:47:05.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:47:05.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=293] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:47:05.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=293] Local address: 172.21.15.161, port 53952 2022-07-03T13:47:05.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=293] Peer address: 172.21.15.101, port 22 2022-07-03T13:47:05.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=293] Connection lost 2022-07-03T13:47:05.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=293] Aborting connection 2022-07-03T13:47:05.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:05 smithi188 ceph-mon[92425]: pgmap v5640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:05.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:07 smithi161 ceph-mon[91579]: pgmap v5641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:07 smithi188 ceph-mon[92425]: pgmap v5641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:09 smithi161 ceph-mon[91579]: pgmap v5642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:09 smithi188 ceph-mon[92425]: pgmap v5642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:11 smithi161 ceph-mon[91579]: pgmap v5643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:11 smithi188 ceph-mon[92425]: pgmap v5643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:13 smithi161 ceph-mon[91579]: pgmap v5644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:13 smithi188 ceph-mon[92425]: pgmap v5644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:15 smithi161 ceph-mon[91579]: pgmap v5645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:15 smithi188 ceph-mon[92425]: pgmap v5645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:17 smithi161 ceph-mon[91579]: pgmap v5646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:17 smithi188 ceph-mon[92425]: pgmap v5646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:19 smithi161 ceph-mon[91579]: pgmap v5647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:19 smithi188 ceph-mon[92425]: pgmap v5647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:21 smithi161 ceph-mon[91579]: pgmap v5648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:21 smithi188 ceph-mon[92425]: pgmap v5648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:23 smithi161 ceph-mon[91579]: pgmap v5649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:23 smithi188 ceph-mon[92425]: pgmap v5649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:25 smithi161 ceph-mon[91579]: pgmap v5650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:25 smithi188 ceph-mon[92425]: pgmap v5650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:27 smithi161 ceph-mon[91579]: pgmap v5651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:27 smithi188 ceph-mon[92425]: pgmap v5651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:29 smithi161 ceph-mon[91579]: pgmap v5652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:29 smithi188 ceph-mon[92425]: pgmap v5652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:31 smithi161 ceph-mon[91579]: pgmap v5653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:31 smithi188 ceph-mon[92425]: pgmap v5653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:33 smithi161 ceph-mon[91579]: pgmap v5654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:33 smithi188 ceph-mon[92425]: pgmap v5654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:35 smithi161 ceph-mon[91579]: pgmap v5655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:35 smithi188 ceph-mon[92425]: pgmap v5655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:37 smithi161 ceph-mon[91579]: pgmap v5656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:37 smithi188 ceph-mon[92425]: pgmap v5656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:39 smithi161 ceph-mon[91579]: pgmap v5657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:39 smithi188 ceph-mon[92425]: pgmap v5657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:41 smithi161 ceph-mon[91579]: pgmap v5658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:41 smithi188 ceph-mon[92425]: pgmap v5658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:43 smithi161 ceph-mon[91579]: pgmap v5659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:43 smithi188 ceph-mon[92425]: pgmap v5659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:45 smithi161 ceph-mon[91579]: pgmap v5660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:45 smithi188 ceph-mon[92425]: pgmap v5660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:47 smithi161 ceph-mon[91579]: pgmap v5661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:47 smithi188 ceph-mon[92425]: pgmap v5661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:49 smithi161 ceph-mon[91579]: pgmap v5662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:49 smithi188 ceph-mon[92425]: pgmap v5662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:51 smithi161 ceph-mon[91579]: pgmap v5663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:51 smithi188 ceph-mon[92425]: pgmap v5663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:53 smithi161 ceph-mon[91579]: pgmap v5664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:53 smithi188 ceph-mon[92425]: pgmap v5664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:55 smithi161 ceph-mon[91579]: pgmap v5665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:55 smithi188 ceph-mon[92425]: pgmap v5665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:47:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:57 smithi161 ceph-mon[91579]: pgmap v5666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:57 smithi188 ceph-mon[92425]: pgmap v5666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:47:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:47:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:47:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:47:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:47:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:47:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:47:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:47:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:47:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:47:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:47:59 smithi161 ceph-mon[91579]: pgmap v5667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:47:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:47:59 smithi188 ceph-mon[92425]: pgmap v5667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:01 smithi161 ceph-mon[91579]: pgmap v5668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:01 smithi188 ceph-mon[92425]: pgmap v5668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:03 smithi161 ceph-mon[91579]: pgmap v5669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:03 smithi188 ceph-mon[92425]: pgmap v5669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:48:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:48:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:48:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:48:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:48:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:48:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:05 smithi161 ceph-mon[91579]: pgmap v5670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:05 smithi188 ceph-mon[92425]: pgmap v5670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:07 smithi161 ceph-mon[91579]: pgmap v5671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:07 smithi188 ceph-mon[92425]: pgmap v5671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:09 smithi188 ceph-mon[92425]: pgmap v5672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:09 smithi161 ceph-mon[91579]: pgmap v5672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:11 smithi188 ceph-mon[92425]: pgmap v5673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:11 smithi161 ceph-mon[91579]: pgmap v5673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:13 smithi188 ceph-mon[92425]: pgmap v5674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:13 smithi161 ceph-mon[91579]: pgmap v5674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:15 smithi188 ceph-mon[92425]: pgmap v5675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:15 smithi161 ceph-mon[91579]: pgmap v5675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:17 smithi188 ceph-mon[92425]: pgmap v5676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:17 smithi161 ceph-mon[91579]: pgmap v5676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:19 smithi188 ceph-mon[92425]: pgmap v5677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:19 smithi161 ceph-mon[91579]: pgmap v5677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:21 smithi161 ceph-mon[91579]: pgmap v5678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:21 smithi188 ceph-mon[92425]: pgmap v5678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:23 smithi161 ceph-mon[91579]: pgmap v5679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:23 smithi188 ceph-mon[92425]: pgmap v5679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:25 smithi161 ceph-mon[91579]: pgmap v5680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:25 smithi188 ceph-mon[92425]: pgmap v5680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:27 smithi161 ceph-mon[91579]: pgmap v5681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:27 smithi188 ceph-mon[92425]: pgmap v5681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:29 smithi161 ceph-mon[91579]: pgmap v5682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:29 smithi188 ceph-mon[92425]: pgmap v5682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:30.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:30.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:31 smithi161 ceph-mon[91579]: pgmap v5683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:31 smithi188 ceph-mon[92425]: pgmap v5683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:33.751 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-mon[91579]: pgmap v5684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:33.752 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:33 smithi188 ceph-mon[92425]: pgmap v5684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:34.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:48:33.750+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:48:34.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:48:34.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:48:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:48:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:48:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:48:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:48:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:48:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:48:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:48:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:48:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:48:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:48:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:48:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:48:34.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:48:34.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:48:34.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:48:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:48:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:48:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:48:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:48:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:48:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:48:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:48:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:48:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:48:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:48:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:48:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:48:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:48:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:48:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:48:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:48:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:48:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:48:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:48:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:48:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:48:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:48:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:48:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:48:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:48:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:48:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:48:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=295] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:48:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=295] Local address: 172.21.15.161, port 53992 2022-07-03T13:48:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=295] Peer address: 172.21.15.101, port 22 2022-07-03T13:48:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=295] Connection lost 2022-07-03T13:48:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=295] Aborting connection 2022-07-03T13:48:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:48:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:48:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:48:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:48:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:48:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:48:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:48:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:48:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:48:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:48:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:48:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:48:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:48:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:48:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:48:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:48:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:48:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:48:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:48:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:48:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:48:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:48:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:48:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:48:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:48:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:48:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:48:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:48:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:48:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:48:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:48:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:48:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:48:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:48:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:48:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:48:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:48:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:48:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:48:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:48:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:48:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:48:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:48:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:48:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:48:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:48:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:48:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:48:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=295] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:48:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=295] Local address: 172.21.15.161, port 53992 2022-07-03T13:48:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=295] Peer address: 172.21.15.101, port 22 2022-07-03T13:48:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=295] Connection lost 2022-07-03T13:48:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=295] Aborting connection 2022-07-03T13:48:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:35 smithi161 ceph-mon[91579]: pgmap v5685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:48:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:48:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:48:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:48:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:48:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:48:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:48:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:48:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:48:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:48:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:48:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:48:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:48:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:48:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:48:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:48:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:48:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:48:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:48:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:48:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:48:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:48:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:48:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:48:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:48:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:48:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:48:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:48:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:48:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:48:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:48:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:48:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:48:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:48:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:48:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:48:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:48:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:48:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:48:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:48:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:48:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:48:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:48:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:48:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:48:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:48:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=295] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:48:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=295] Local address: 172.21.15.161, port 53992 2022-07-03T13:48:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=295] Peer address: 172.21.15.101, port 22 2022-07-03T13:48:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=295] Connection lost 2022-07-03T13:48:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=295] Aborting connection 2022-07-03T13:48:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:35 smithi188 ceph-mon[92425]: pgmap v5685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:37 smithi161 ceph-mon[91579]: pgmap v5686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:37 smithi188 ceph-mon[92425]: pgmap v5686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:39 smithi161 ceph-mon[91579]: pgmap v5687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:39 smithi188 ceph-mon[92425]: pgmap v5687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:41 smithi161 ceph-mon[91579]: pgmap v5688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:41 smithi188 ceph-mon[92425]: pgmap v5688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:43 smithi161 ceph-mon[91579]: pgmap v5689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:43 smithi188 ceph-mon[92425]: pgmap v5689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:45 smithi161 ceph-mon[91579]: pgmap v5690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:45 smithi188 ceph-mon[92425]: pgmap v5690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:47 smithi161 ceph-mon[91579]: pgmap v5691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:47 smithi188 ceph-mon[92425]: pgmap v5691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:49 smithi161 ceph-mon[91579]: pgmap v5692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:49 smithi188 ceph-mon[92425]: pgmap v5692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:51 smithi161 ceph-mon[91579]: pgmap v5693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:51 smithi188 ceph-mon[92425]: pgmap v5693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:53 smithi161 ceph-mon[91579]: pgmap v5694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:53 smithi188 ceph-mon[92425]: pgmap v5694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:55 smithi161 ceph-mon[91579]: pgmap v5695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:55 smithi188 ceph-mon[92425]: pgmap v5695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:48:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:57 smithi161 ceph-mon[91579]: pgmap v5696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:48:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:48:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:48:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:48:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:57 smithi188 ceph-mon[92425]: pgmap v5696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:48:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:48:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:48:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:48:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:48:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:48:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:48:59 smithi161 ceph-mon[91579]: pgmap v5697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:48:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:48:59 smithi188 ceph-mon[92425]: pgmap v5697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:01 smithi161 ceph-mon[91579]: pgmap v5698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:01.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:01 smithi188 ceph-mon[92425]: pgmap v5698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:03 smithi161 ceph-mon[91579]: pgmap v5699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:03 smithi188 ceph-mon[92425]: pgmap v5699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:05 smithi161 ceph-mon[91579]: pgmap v5700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:05 smithi188 ceph-mon[92425]: pgmap v5700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:07 smithi161 ceph-mon[91579]: pgmap v5701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:07 smithi188 ceph-mon[92425]: pgmap v5701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:09 smithi161 ceph-mon[91579]: pgmap v5702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:09 smithi188 ceph-mon[92425]: pgmap v5702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:10 smithi161 ceph-mon[91579]: pgmap v5703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:10 smithi188 ceph-mon[92425]: pgmap v5703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:13 smithi161 ceph-mon[91579]: pgmap v5704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:13 smithi188 ceph-mon[92425]: pgmap v5704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:15 smithi161 ceph-mon[91579]: pgmap v5705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:15 smithi188 ceph-mon[92425]: pgmap v5705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:17 smithi161 ceph-mon[91579]: pgmap v5706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:17 smithi188 ceph-mon[92425]: pgmap v5706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:19 smithi161 ceph-mon[91579]: pgmap v5707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:19 smithi188 ceph-mon[92425]: pgmap v5707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:21 smithi161 ceph-mon[91579]: pgmap v5708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:21 smithi188 ceph-mon[92425]: pgmap v5708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:23 smithi161 ceph-mon[91579]: pgmap v5709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:23 smithi188 ceph-mon[92425]: pgmap v5709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:25 smithi161 ceph-mon[91579]: pgmap v5710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:25 smithi188 ceph-mon[92425]: pgmap v5710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:27 smithi161 ceph-mon[91579]: pgmap v5711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:27 smithi188 ceph-mon[92425]: pgmap v5711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:29 smithi161 ceph-mon[91579]: pgmap v5712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:29 smithi188 ceph-mon[92425]: pgmap v5712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:31 smithi161 ceph-mon[91579]: pgmap v5713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:31 smithi188 ceph-mon[92425]: pgmap v5713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-mon[91579]: pgmap v5714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:33 smithi188 ceph-mon[92425]: pgmap v5714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:34.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:49:33.788+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:49:34.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:49:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:49:34.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:49:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:49:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:49:34.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:49:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:49:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:49:34.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:49:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:49:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:49:34.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:49:34.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:49:34.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:49:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:49:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:49:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:49:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:49:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:49:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:49:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:49:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:49:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:49:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:49:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:49:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:49:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:49:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:49:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:49:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:49:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:49:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:49:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:49:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:49:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:49:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:49:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:49:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:49:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:49:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:49:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:49:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:49:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:49:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:49:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=297] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:49:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=297] Local address: 172.21.15.161, port 54020 2022-07-03T13:49:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=297] Peer address: 172.21.15.101, port 22 2022-07-03T13:49:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=297] Connection lost 2022-07-03T13:49:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=297] Aborting connection 2022-07-03T13:49:34.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:49:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:49:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:49:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:49:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:49:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:49:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:49:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:49:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:49:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:49:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:49:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:49:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:49:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:49:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:49:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:49:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:49:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:49:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:49:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:49:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:49:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:49:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:49:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:49:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:49:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:49:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:49:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:49:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:49:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:49:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:49:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:49:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:49:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:49:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:49:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:49:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:49:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:49:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:49:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:49:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:49:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:49:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:49:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:49:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:49:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:49:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:49:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:49:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:49:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:49:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:49:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:49:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:49:35.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:49:35.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:49:35.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=297] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:49:35.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=297] Local address: 172.21.15.161, port 54020 2022-07-03T13:49:35.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=297] Peer address: 172.21.15.101, port 22 2022-07-03T13:49:35.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=297] Connection lost 2022-07-03T13:49:35.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=297] Aborting connection 2022-07-03T13:49:35.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:35 smithi161 ceph-mon[91579]: pgmap v5715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:35.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:49:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:49:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:49:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:49:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:49:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:49:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:49:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:49:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:49:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:49:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:49:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:49:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:49:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:49:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:49:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:49:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:49:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:49:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:49:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:49:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:49:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:49:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:49:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:49:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:49:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:49:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:49:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:49:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:49:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:49:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:49:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:49:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:49:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:49:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:49:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:49:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:49:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:49:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:49:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:49:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:49:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:49:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:49:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:49:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:49:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:49:35.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=297] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:49:35.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=297] Local address: 172.21.15.161, port 54020 2022-07-03T13:49:35.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=297] Peer address: 172.21.15.101, port 22 2022-07-03T13:49:35.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=297] Connection lost 2022-07-03T13:49:35.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=297] Aborting connection 2022-07-03T13:49:35.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:35 smithi188 ceph-mon[92425]: pgmap v5715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:35.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:37 smithi161 ceph-mon[91579]: pgmap v5716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:37 smithi188 ceph-mon[92425]: pgmap v5716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:39 smithi161 ceph-mon[91579]: pgmap v5717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:39 smithi188 ceph-mon[92425]: pgmap v5717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:41 smithi161 ceph-mon[91579]: pgmap v5718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:41 smithi188 ceph-mon[92425]: pgmap v5718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:43 smithi161 ceph-mon[91579]: pgmap v5719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:43 smithi188 ceph-mon[92425]: pgmap v5719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:45 smithi161 ceph-mon[91579]: pgmap v5720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:45 smithi188 ceph-mon[92425]: pgmap v5720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:47 smithi161 ceph-mon[91579]: pgmap v5721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:47 smithi188 ceph-mon[92425]: pgmap v5721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:49 smithi161 ceph-mon[91579]: pgmap v5722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:49 smithi188 ceph-mon[92425]: pgmap v5722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:51 smithi161 ceph-mon[91579]: pgmap v5723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:51 smithi188 ceph-mon[92425]: pgmap v5723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:53 smithi161 ceph-mon[91579]: pgmap v5724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:53 smithi188 ceph-mon[92425]: pgmap v5724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:55 smithi161 ceph-mon[91579]: pgmap v5725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:55 smithi188 ceph-mon[92425]: pgmap v5725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:49:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:57 smithi161 ceph-mon[91579]: pgmap v5726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:57 smithi188 ceph-mon[92425]: pgmap v5726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:49:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:49:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:49:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:49:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:49:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:49:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:49:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:49:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:49:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:49:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:49:59 smithi161 ceph-mon[91579]: pgmap v5727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:49:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:49:59 smithi188 ceph-mon[92425]: pgmap v5727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:50:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T13:50:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:01 smithi161 ceph-mon[91579]: pgmap v5728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:01 smithi188 ceph-mon[92425]: pgmap v5728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:03 smithi161 ceph-mon[91579]: pgmap v5729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:03 smithi188 ceph-mon[92425]: pgmap v5729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:05 smithi161 ceph-mon[91579]: pgmap v5730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:05 smithi188 ceph-mon[92425]: pgmap v5730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:07 smithi161 ceph-mon[91579]: pgmap v5731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:07 smithi188 ceph-mon[92425]: pgmap v5731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:09 smithi161 ceph-mon[91579]: pgmap v5732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:09 smithi188 ceph-mon[92425]: pgmap v5732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:11 smithi161 ceph-mon[91579]: pgmap v5733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:11 smithi188 ceph-mon[92425]: pgmap v5733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:13 smithi161 ceph-mon[91579]: pgmap v5734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:13 smithi188 ceph-mon[92425]: pgmap v5734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:15 smithi161 ceph-mon[91579]: pgmap v5735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:15 smithi188 ceph-mon[92425]: pgmap v5735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:17 smithi161 ceph-mon[91579]: pgmap v5736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:17 smithi188 ceph-mon[92425]: pgmap v5736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:19 smithi161 ceph-mon[91579]: pgmap v5737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:19 smithi188 ceph-mon[92425]: pgmap v5737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:21 smithi161 ceph-mon[91579]: pgmap v5738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:21 smithi188 ceph-mon[92425]: pgmap v5738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:23 smithi161 ceph-mon[91579]: pgmap v5739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:23 smithi188 ceph-mon[92425]: pgmap v5739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:25 smithi161 ceph-mon[91579]: pgmap v5740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:25 smithi188 ceph-mon[92425]: pgmap v5740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:27 smithi161 ceph-mon[91579]: pgmap v5741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:27 smithi188 ceph-mon[92425]: pgmap v5741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:29 smithi161 ceph-mon[91579]: pgmap v5742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:29 smithi188 ceph-mon[92425]: pgmap v5742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:31 smithi161 ceph-mon[91579]: pgmap v5743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:31 smithi188 ceph-mon[92425]: pgmap v5743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:33 smithi161 ceph-mon[91579]: pgmap v5744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:33 smithi188 ceph-mon[92425]: pgmap v5744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:35.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:50:34.730+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:50:35.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:50:35.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:50:35.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:50:35.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:50:35.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:50:35.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:50:35.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:50:35.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:50:35.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:50:35.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:50:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:50:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:50:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:50:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:50:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:50:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:50:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:50:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:50:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:50:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:50:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:50:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:50:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:50:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:50:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:50:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:50:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:50:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:50:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:50:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:50:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:50:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:50:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:50:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:50:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:50:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:50:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:50:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:50:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:50:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:50:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:50:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:50:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:50:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:50:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=299] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:50:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=299] Local address: 172.21.15.161, port 54050 2022-07-03T13:50:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=299] Peer address: 172.21.15.101, port 22 2022-07-03T13:50:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=299] Connection lost 2022-07-03T13:50:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=299] Aborting connection 2022-07-03T13:50:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:50:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:50:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:35 smithi188 ceph-mon[92425]: pgmap v5745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:50:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:50:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:50:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:35 smithi161 ceph-mon[91579]: pgmap v5745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:50:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:50:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:50:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:50:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:50:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:50:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:50:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:50:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:50:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:50:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:50:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:50:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:50:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:50:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:50:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:50:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:50:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:50:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:50:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:50:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:50:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:50:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:50:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:50:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:50:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:50:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:50:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:50:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:50:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:50:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:50:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:50:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:50:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:50:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:50:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:50:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:50:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:50:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:50:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:50:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:50:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:50:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:50:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:50:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:50:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:50:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:50:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:50:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:50:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=299] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:50:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=299] Local address: 172.21.15.161, port 54050 2022-07-03T13:50:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=299] Peer address: 172.21.15.101, port 22 2022-07-03T13:50:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=299] Connection lost 2022-07-03T13:50:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=299] Aborting connection 2022-07-03T13:50:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:50:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:50:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:50:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:50:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:50:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:50:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:50:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:50:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:50:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:50:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:50:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:50:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:50:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:50:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:50:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:50:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:50:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:50:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:50:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:50:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:50:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:50:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:50:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:50:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:50:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:50:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:50:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:50:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:50:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:50:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:50:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:50:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:50:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:50:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:50:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:50:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:50:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:50:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:50:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:50:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:50:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:50:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:50:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:50:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:50:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:50:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=299] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:50:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=299] Local address: 172.21.15.161, port 54050 2022-07-03T13:50:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=299] Peer address: 172.21.15.101, port 22 2022-07-03T13:50:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=299] Connection lost 2022-07-03T13:50:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=299] Aborting connection 2022-07-03T13:50:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:37 smithi188 ceph-mon[92425]: pgmap v5746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:37 smithi161 ceph-mon[91579]: pgmap v5746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:39 smithi188 ceph-mon[92425]: pgmap v5747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:39 smithi161 ceph-mon[91579]: pgmap v5747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:41 smithi188 ceph-mon[92425]: pgmap v5748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:41 smithi161 ceph-mon[91579]: pgmap v5748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:43 smithi188 ceph-mon[92425]: pgmap v5749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:43 smithi161 ceph-mon[91579]: pgmap v5749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:45 smithi161 ceph-mon[91579]: pgmap v5750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:45 smithi188 ceph-mon[92425]: pgmap v5750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:47 smithi161 ceph-mon[91579]: pgmap v5751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:47 smithi188 ceph-mon[92425]: pgmap v5751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:49 smithi161 ceph-mon[91579]: pgmap v5752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:49 smithi188 ceph-mon[92425]: pgmap v5752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:51 smithi161 ceph-mon[91579]: pgmap v5753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:51 smithi188 ceph-mon[92425]: pgmap v5753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:53 smithi161 ceph-mon[91579]: pgmap v5754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:53 smithi188 ceph-mon[92425]: pgmap v5754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:55 smithi161 ceph-mon[91579]: pgmap v5755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:55 smithi188 ceph-mon[92425]: pgmap v5755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:50:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:57 smithi161 ceph-mon[91579]: pgmap v5756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:57 smithi188 ceph-mon[92425]: pgmap v5756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:50:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:50:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:50:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:50:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:50:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:50:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:50:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:50:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:50:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:50:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:50:59 smithi161 ceph-mon[91579]: pgmap v5757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:50:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:50:59 smithi188 ceph-mon[92425]: pgmap v5757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:01 smithi161 ceph-mon[91579]: pgmap v5758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:01 smithi188 ceph-mon[92425]: pgmap v5758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:03 smithi161 ceph-mon[91579]: pgmap v5759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:03 smithi188 ceph-mon[92425]: pgmap v5759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:05 smithi161 ceph-mon[91579]: pgmap v5760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:05 smithi188 ceph-mon[92425]: pgmap v5760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:07 smithi161 ceph-mon[91579]: pgmap v5761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:07 smithi188 ceph-mon[92425]: pgmap v5761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:09 smithi161 ceph-mon[91579]: pgmap v5762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:09 smithi188 ceph-mon[92425]: pgmap v5762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:11 smithi161 ceph-mon[91579]: pgmap v5763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:11 smithi188 ceph-mon[92425]: pgmap v5763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:13 smithi161 ceph-mon[91579]: pgmap v5764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:13 smithi188 ceph-mon[92425]: pgmap v5764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:15 smithi161 ceph-mon[91579]: pgmap v5765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:15 smithi188 ceph-mon[92425]: pgmap v5765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:17 smithi161 ceph-mon[91579]: pgmap v5766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:17 smithi188 ceph-mon[92425]: pgmap v5766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:19 smithi161 ceph-mon[91579]: pgmap v5767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:19 smithi188 ceph-mon[92425]: pgmap v5767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:21 smithi161 ceph-mon[91579]: pgmap v5768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:21 smithi188 ceph-mon[92425]: pgmap v5768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:23 smithi161 ceph-mon[91579]: pgmap v5769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:23 smithi188 ceph-mon[92425]: pgmap v5769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:25 smithi161 ceph-mon[91579]: pgmap v5770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:25 smithi188 ceph-mon[92425]: pgmap v5770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:27 smithi161 ceph-mon[91579]: pgmap v5771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:27 smithi188 ceph-mon[92425]: pgmap v5771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:28.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:29 smithi161 ceph-mon[91579]: pgmap v5772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:29 smithi188 ceph-mon[92425]: pgmap v5772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:31 smithi161 ceph-mon[91579]: pgmap v5773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:31 smithi188 ceph-mon[92425]: pgmap v5773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:33 smithi161 ceph-mon[91579]: pgmap v5774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:33 smithi188 ceph-mon[92425]: pgmap v5774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-mon[91579]: pgmap v5775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:35.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:51:35.234+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:51:35.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:51:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:51:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:51:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:51:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:51:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:51:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:51:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:51:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:51:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:51:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:51:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:51:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:51:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:51:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:51:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:51:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:51:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:51:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:51:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:51:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:51:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:51:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:51:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:51:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:51:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:51:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:51:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:51:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:51:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:51:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:51:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:51:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:51:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:51:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:51:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:51:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:51:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:51:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:51:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:51:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:51:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:51:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:51:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:51:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=301] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:51:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=301] Local address: 172.21.15.161, port 54078 2022-07-03T13:51:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=301] Peer address: 172.21.15.101, port 22 2022-07-03T13:51:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=301] Connection lost 2022-07-03T13:51:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=301] Aborting connection 2022-07-03T13:51:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:51:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:51:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:35 smithi188 ceph-mon[92425]: pgmap v5775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:51:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:51:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:51:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:51:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:51:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:51:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:51:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:51:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:51:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:51:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:51:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:51:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:51:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:51:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:51:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:51:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:51:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:51:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:51:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:51:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:51:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:51:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:51:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:51:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:51:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:51:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:51:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:51:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:51:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:51:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:51:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:51:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:51:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:51:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:51:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:51:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:51:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:51:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:51:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:51:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:51:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:51:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:51:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:51:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:51:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:51:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:51:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:51:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:51:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=301] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:51:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=301] Local address: 172.21.15.161, port 54078 2022-07-03T13:51:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=301] Peer address: 172.21.15.101, port 22 2022-07-03T13:51:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=301] Connection lost 2022-07-03T13:51:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=301] Aborting connection 2022-07-03T13:51:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:51:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:51:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:51:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:51:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:51:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:51:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:51:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:51:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:51:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:51:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:51:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:51:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:51:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:51:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:51:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:51:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:51:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:51:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:51:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:51:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:51:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:51:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:51:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:51:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:51:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:51:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:51:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:51:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:51:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:51:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:51:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:51:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:51:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:51:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:51:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:51:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:51:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:51:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:51:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:51:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:51:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:51:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:51:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:51:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:51:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:51:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:51:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:51:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:51:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=301] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:51:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=301] Local address: 172.21.15.161, port 54078 2022-07-03T13:51:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=301] Peer address: 172.21.15.101, port 22 2022-07-03T13:51:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=301] Connection lost 2022-07-03T13:51:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=301] Aborting connection 2022-07-03T13:51:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:37 smithi161 ceph-mon[91579]: pgmap v5776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:37 smithi188 ceph-mon[92425]: pgmap v5776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T13:51:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: pgmap v5777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:51:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:51:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:51:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:39 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:51:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: pgmap v5777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T13:51:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T13:51:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T13:51:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T13:51:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:39 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T13:51:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:41 smithi161 ceph-mon[91579]: pgmap v5778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:41 smithi188 ceph-mon[92425]: pgmap v5778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:43 smithi161 ceph-mon[91579]: pgmap v5779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:43 smithi188 ceph-mon[92425]: pgmap v5779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:45 smithi161 ceph-mon[91579]: pgmap v5780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:45 smithi188 ceph-mon[92425]: pgmap v5780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:47 smithi161 ceph-mon[91579]: pgmap v5781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:47 smithi188 ceph-mon[92425]: pgmap v5781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:49 smithi161 ceph-mon[91579]: pgmap v5782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:49 smithi188 ceph-mon[92425]: pgmap v5782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:51 smithi161 ceph-mon[91579]: pgmap v5783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:51 smithi188 ceph-mon[92425]: pgmap v5783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:53 smithi161 ceph-mon[91579]: pgmap v5784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:53 smithi188 ceph-mon[92425]: pgmap v5784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:55 smithi161 ceph-mon[91579]: pgmap v5785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:55 smithi188 ceph-mon[92425]: pgmap v5785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:51:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:57 smithi161 ceph-mon[91579]: pgmap v5786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:57 smithi188 ceph-mon[92425]: pgmap v5786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:51:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:51:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:51:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:51:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:51:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:51:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:51:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:51:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:51:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:51:59 smithi161 ceph-mon[91579]: pgmap v5787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:51:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:51:59 smithi188 ceph-mon[92425]: pgmap v5787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:01 smithi161 ceph-mon[91579]: pgmap v5788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:01 smithi188 ceph-mon[92425]: pgmap v5788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:03 smithi161 ceph-mon[91579]: pgmap v5789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:03 smithi188 ceph-mon[92425]: pgmap v5789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:05 smithi161 ceph-mon[91579]: pgmap v5790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:05 smithi188 ceph-mon[92425]: pgmap v5790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:07 smithi161 ceph-mon[91579]: pgmap v5791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:07 smithi188 ceph-mon[92425]: pgmap v5791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:09 smithi161 ceph-mon[91579]: pgmap v5792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:09 smithi188 ceph-mon[92425]: pgmap v5792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:11 smithi161 ceph-mon[91579]: pgmap v5793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:11 smithi188 ceph-mon[92425]: pgmap v5793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:13 smithi161 ceph-mon[91579]: pgmap v5794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:13 smithi188 ceph-mon[92425]: pgmap v5794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:15 smithi161 ceph-mon[91579]: pgmap v5795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:15 smithi188 ceph-mon[92425]: pgmap v5795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:17 smithi161 ceph-mon[91579]: pgmap v5796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:17 smithi188 ceph-mon[92425]: pgmap v5796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:19 smithi161 ceph-mon[91579]: pgmap v5797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:19 smithi188 ceph-mon[92425]: pgmap v5797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:21 smithi161 ceph-mon[91579]: pgmap v5798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:21 smithi188 ceph-mon[92425]: pgmap v5798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:23 smithi161 ceph-mon[91579]: pgmap v5799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:23 smithi188 ceph-mon[92425]: pgmap v5799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:25 smithi161 ceph-mon[91579]: pgmap v5800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:25 smithi188 ceph-mon[92425]: pgmap v5800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:27 smithi161 ceph-mon[91579]: pgmap v5801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:27 smithi188 ceph-mon[92425]: pgmap v5801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:29 smithi161 ceph-mon[91579]: pgmap v5802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:29 smithi188 ceph-mon[92425]: pgmap v5802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:31 smithi161 ceph-mon[91579]: pgmap v5803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:31 smithi188 ceph-mon[92425]: pgmap v5803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:33 smithi161 ceph-mon[91579]: pgmap v5804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:33 smithi188 ceph-mon[92425]: pgmap v5804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:35 smithi161 ceph-mon[91579]: pgmap v5805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:35 smithi188 ceph-mon[92425]: pgmap v5805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:37 smithi161 ceph-mon[91579]: pgmap v5806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:37 smithi188 ceph-mon[92425]: pgmap v5806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:38.501 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:38.512 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:38.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:52:38.500+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:52:38.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:52:38.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:52:38.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:52:38.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:52:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:52:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:52:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:52:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:52:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:52:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:52:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:52:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:52:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:52:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:52:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:52:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:52:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:52:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:52:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:52:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:52:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:52:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:52:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:52:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:52:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:52:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:52:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:52:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:52:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:52:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:52:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:52:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:52:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:52:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:52:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:52:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:52:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:52:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:52:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:52:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:52:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:52:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:52:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:52:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:52:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=303] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:52:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=303] Local address: 172.21.15.161, port 54106 2022-07-03T13:52:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=303] Peer address: 172.21.15.101, port 22 2022-07-03T13:52:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=303] Connection lost 2022-07-03T13:52:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=303] Aborting connection 2022-07-03T13:52:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:52:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:52:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: pgmap v5807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:52:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:52:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:52:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:52:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:52:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:52:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:52:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:52:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:52:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:52:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:52:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:52:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:52:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:52:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:52:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:52:39.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:52:39.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:52:39.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:52:39.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:52:39.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:52:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:52:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:52:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:52:39.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:52:39.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:52:39.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:52:39.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:52:39.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:52:39.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:52:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:52:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:52:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:52:39.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:52:39.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:52:39.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:52:39.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:52:39.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:52:39.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:52:39.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:52:39.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:52:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:52:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:52:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:52:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:52:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:52:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:52:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:52:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:52:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=303] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:52:39.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=303] Local address: 172.21.15.161, port 54106 2022-07-03T13:52:39.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=303] Peer address: 172.21.15.101, port 22 2022-07-03T13:52:39.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=303] Connection lost 2022-07-03T13:52:39.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=303] Aborting connection 2022-07-03T13:52:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: pgmap v5807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:52:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:52:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:52:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:52:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:52:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:52:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:52:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:52:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:52:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:52:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:52:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:52:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:52:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:52:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:52:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:52:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:52:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:52:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:52:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:52:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:52:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:52:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:52:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:52:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:52:39.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:52:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:52:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:52:39.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:52:39.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:52:39.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:52:39.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:52:39.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:52:39.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:52:39.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:52:39.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:52:39.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:52:39.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:52:39.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:52:39.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:52:39.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:52:39.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:52:39.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:52:39.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:52:39.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:52:39.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:52:39.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:52:39.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:52:39.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:52:39.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=303] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:52:39.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=303] Local address: 172.21.15.161, port 54106 2022-07-03T13:52:39.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=303] Peer address: 172.21.15.101, port 22 2022-07-03T13:52:39.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=303] Connection lost 2022-07-03T13:52:39.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=303] Aborting connection 2022-07-03T13:52:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:41 smithi161 ceph-mon[91579]: pgmap v5808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:41 smithi188 ceph-mon[92425]: pgmap v5808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:43 smithi161 ceph-mon[91579]: pgmap v5809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:43 smithi188 ceph-mon[92425]: pgmap v5809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:45 smithi161 ceph-mon[91579]: pgmap v5810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:45 smithi188 ceph-mon[92425]: pgmap v5810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:47 smithi161 ceph-mon[91579]: pgmap v5811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:47 smithi188 ceph-mon[92425]: pgmap v5811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T13:52:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:49 smithi161 ceph-mon[91579]: pgmap v5812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:49 smithi188 ceph-mon[92425]: pgmap v5812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:51 smithi161 ceph-mon[91579]: pgmap v5813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:51 smithi188 ceph-mon[92425]: pgmap v5813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:53 smithi161 ceph-mon[91579]: pgmap v5814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:53 smithi188 ceph-mon[92425]: pgmap v5814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:55 smithi161 ceph-mon[91579]: pgmap v5815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:55 smithi188 ceph-mon[92425]: pgmap v5815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:52:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:57 smithi161 ceph-mon[91579]: pgmap v5816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:57 smithi188 ceph-mon[92425]: pgmap v5816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:52:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:52:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:52:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:52:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:52:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:52:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:52:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:52:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:52:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:52:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:52:59 smithi161 ceph-mon[91579]: pgmap v5817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:52:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:52:59 smithi188 ceph-mon[92425]: pgmap v5817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:01 smithi161 ceph-mon[91579]: pgmap v5818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:01 smithi188 ceph-mon[92425]: pgmap v5818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:03 smithi161 ceph-mon[91579]: pgmap v5819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:03 smithi188 ceph-mon[92425]: pgmap v5819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:05 smithi161 ceph-mon[91579]: pgmap v5820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:05 smithi188 ceph-mon[92425]: pgmap v5820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:07 smithi161 ceph-mon[91579]: pgmap v5821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:07 smithi188 ceph-mon[92425]: pgmap v5821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:09 smithi161 ceph-mon[91579]: pgmap v5822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:09 smithi188 ceph-mon[92425]: pgmap v5822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:11 smithi161 ceph-mon[91579]: pgmap v5823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:11 smithi188 ceph-mon[92425]: pgmap v5823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:13 smithi161 ceph-mon[91579]: pgmap v5824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:13 smithi188 ceph-mon[92425]: pgmap v5824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:15 smithi161 ceph-mon[91579]: pgmap v5825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:15 smithi188 ceph-mon[92425]: pgmap v5825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:17 smithi161 ceph-mon[91579]: pgmap v5826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:17 smithi188 ceph-mon[92425]: pgmap v5826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:19 smithi161 ceph-mon[91579]: pgmap v5827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:19 smithi188 ceph-mon[92425]: pgmap v5827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:21 smithi161 ceph-mon[91579]: pgmap v5828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:21 smithi188 ceph-mon[92425]: pgmap v5828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:23 smithi161 ceph-mon[91579]: pgmap v5829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:23 smithi188 ceph-mon[92425]: pgmap v5829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:25 smithi161 ceph-mon[91579]: pgmap v5830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:25 smithi188 ceph-mon[92425]: pgmap v5830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:27 smithi161 ceph-mon[91579]: pgmap v5831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:27 smithi188 ceph-mon[92425]: pgmap v5831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:29 smithi161 ceph-mon[91579]: pgmap v5832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:29 smithi188 ceph-mon[92425]: pgmap v5832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:31 smithi161 ceph-mon[91579]: pgmap v5833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:31 smithi188 ceph-mon[92425]: pgmap v5833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:33 smithi161 ceph-mon[91579]: pgmap v5834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:33 smithi188 ceph-mon[92425]: pgmap v5834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:35 smithi161 ceph-mon[91579]: pgmap v5835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:35 smithi188 ceph-mon[92425]: pgmap v5835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:37 smithi161 ceph-mon[91579]: pgmap v5836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:37 smithi188 ceph-mon[92425]: pgmap v5836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:39.271 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:53:39.003+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:53:39.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:53:39.272 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:53:39.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:53:39.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:53:39.273 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:53:39.274 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:53:39.274 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:53:39.274 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:53:39.275 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:53:39.275 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:53:39.275 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:53:39.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:53:39.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:53:39.276 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:53:39.277 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:53:39.277 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:53:39.277 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:53:39.278 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:53:39.278 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:53:39.278 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:53:39.279 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:53:39.279 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:53:39.279 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:53:39.280 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:53:39.280 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:53:39.280 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:53:39.281 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:53:39.281 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:53:39.281 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:53:39.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:53:39.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:53:39.282 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:53:39.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:53:39.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:53:39.283 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:53:39.284 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:53:39.284 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:53:39.284 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:53:39.285 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:53:39.285 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:53:39.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:53:39.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:53:39.286 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:53:39.287 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:53:39.287 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=305] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:53:39.287 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=305] Local address: 172.21.15.161, port 54134 2022-07-03T13:53:39.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=305] Peer address: 172.21.15.101, port 22 2022-07-03T13:53:39.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=305] Connection lost 2022-07-03T13:53:39.288 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=305] Aborting connection 2022-07-03T13:53:39.289 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:53:39.290 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:39 smithi188 ceph-mon[92425]: pgmap v5837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:39.290 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:53:39.290 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:53:39.291 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:53:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-mon[91579]: pgmap v5837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:53:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:53:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:53:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:53:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:53:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:53:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:53:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:53:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:53:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:53:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:53:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:53:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:53:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:53:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:53:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:53:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:53:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:53:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:53:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:53:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:53:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:53:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:53:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:53:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:53:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:53:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:53:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:53:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:53:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:53:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:53:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:53:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:53:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:53:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:53:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:53:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:53:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:53:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:53:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:53:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:53:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:53:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:53:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:53:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:53:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:53:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:53:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:53:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:53:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=305] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:53:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=305] Local address: 172.21.15.161, port 54134 2022-07-03T13:53:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=305] Peer address: 172.21.15.101, port 22 2022-07-03T13:53:40.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=305] Connection lost 2022-07-03T13:53:40.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=305] Aborting connection 2022-07-03T13:53:40.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:53:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:53:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:53:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:53:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:53:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:53:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:53:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:53:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:53:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:53:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:53:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:53:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:53:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:53:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:53:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:53:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:53:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:53:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:53:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:53:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:53:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:53:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:53:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:53:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:53:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:53:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:53:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:53:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:53:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:53:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:53:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:53:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:53:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:53:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:53:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:53:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:53:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:53:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:53:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:53:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:53:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:53:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:53:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:53:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:53:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:53:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=305] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:53:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=305] Local address: 172.21.15.161, port 54134 2022-07-03T13:53:40.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=305] Peer address: 172.21.15.101, port 22 2022-07-03T13:53:40.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=305] Connection lost 2022-07-03T13:53:40.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=305] Aborting connection 2022-07-03T13:53:40.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:41 smithi161 ceph-mon[91579]: pgmap v5838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:41 smithi188 ceph-mon[92425]: pgmap v5838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:43 smithi161 ceph-mon[91579]: pgmap v5839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:43 smithi188 ceph-mon[92425]: pgmap v5839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:45 smithi161 ceph-mon[91579]: pgmap v5840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:45 smithi188 ceph-mon[92425]: pgmap v5840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:47 smithi161 ceph-mon[91579]: pgmap v5841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:47 smithi188 ceph-mon[92425]: pgmap v5841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:49 smithi161 ceph-mon[91579]: pgmap v5842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:49 smithi188 ceph-mon[92425]: pgmap v5842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:51 smithi161 ceph-mon[91579]: pgmap v5843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:51 smithi188 ceph-mon[92425]: pgmap v5843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:53 smithi161 ceph-mon[91579]: pgmap v5844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:53 smithi188 ceph-mon[92425]: pgmap v5844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:55 smithi161 ceph-mon[91579]: pgmap v5845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:55 smithi188 ceph-mon[92425]: pgmap v5845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:53:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:57 smithi161 ceph-mon[91579]: pgmap v5846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:57 smithi188 ceph-mon[92425]: pgmap v5846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:53:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:53:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:53:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:53:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:53:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:53:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:53:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:53:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:53:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:53:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:53:59 smithi161 ceph-mon[91579]: pgmap v5847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:53:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:53:59 smithi188 ceph-mon[92425]: pgmap v5847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:01 smithi161 ceph-mon[91579]: pgmap v5848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:01 smithi188 ceph-mon[92425]: pgmap v5848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:03 smithi161 ceph-mon[91579]: pgmap v5849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:03 smithi188 ceph-mon[92425]: pgmap v5849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:05 smithi161 ceph-mon[91579]: pgmap v5850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:05 smithi188 ceph-mon[92425]: pgmap v5850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:07 smithi161 ceph-mon[91579]: pgmap v5851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:07 smithi188 ceph-mon[92425]: pgmap v5851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:09 smithi161 ceph-mon[91579]: pgmap v5852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:09 smithi188 ceph-mon[92425]: pgmap v5852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:11 smithi161 ceph-mon[91579]: pgmap v5853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:11 smithi188 ceph-mon[92425]: pgmap v5853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:13 smithi161 ceph-mon[91579]: pgmap v5854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:13 smithi188 ceph-mon[92425]: pgmap v5854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:15 smithi161 ceph-mon[91579]: pgmap v5855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:15 smithi188 ceph-mon[92425]: pgmap v5855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:17 smithi161 ceph-mon[91579]: pgmap v5856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:17 smithi188 ceph-mon[92425]: pgmap v5856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:19 smithi161 ceph-mon[91579]: pgmap v5857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:19 smithi188 ceph-mon[92425]: pgmap v5857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:21 smithi161 ceph-mon[91579]: pgmap v5858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:21 smithi188 ceph-mon[92425]: pgmap v5858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:23 smithi161 ceph-mon[91579]: pgmap v5859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:23 smithi188 ceph-mon[92425]: pgmap v5859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:25 smithi161 ceph-mon[91579]: pgmap v5860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:25 smithi188 ceph-mon[92425]: pgmap v5860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:27 smithi161 ceph-mon[91579]: pgmap v5861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:27 smithi188 ceph-mon[92425]: pgmap v5861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:29 smithi161 ceph-mon[91579]: pgmap v5862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:29 smithi188 ceph-mon[92425]: pgmap v5862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:31 smithi161 ceph-mon[91579]: pgmap v5863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:31 smithi188 ceph-mon[92425]: pgmap v5863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:33 smithi161 ceph-mon[91579]: pgmap v5864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:33 smithi188 ceph-mon[92425]: pgmap v5864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:35 smithi161 ceph-mon[91579]: pgmap v5865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:35 smithi188 ceph-mon[92425]: pgmap v5865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:35.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:37 smithi161 ceph-mon[91579]: pgmap v5866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:37 smithi188 ceph-mon[92425]: pgmap v5866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:39.574 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:39 smithi161 ceph-mon[91579]: pgmap v5867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:39.575 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:39 smithi188 ceph-mon[92425]: pgmap v5867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:54:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:54:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:54:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:54:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:54:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:54:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:41 smithi161 ceph-mon[91579]: pgmap v5868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:41 smithi188 ceph-mon[92425]: pgmap v5868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:43 smithi161 ceph-mon[91579]: pgmap v5869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:43 smithi188 ceph-mon[92425]: pgmap v5869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:45 smithi161 ceph-mon[91579]: pgmap v5870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:45 smithi188 ceph-mon[92425]: pgmap v5870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:47 smithi161 ceph-mon[91579]: pgmap v5871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:47 smithi188 ceph-mon[92425]: pgmap v5871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:49 smithi161 ceph-mon[91579]: pgmap v5872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:49 smithi188 ceph-mon[92425]: pgmap v5872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:51 smithi161 ceph-mon[91579]: pgmap v5873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:51 smithi188 ceph-mon[92425]: pgmap v5873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:53 smithi161 ceph-mon[91579]: pgmap v5874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:53 smithi188 ceph-mon[92425]: pgmap v5874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:55 smithi161 ceph-mon[91579]: pgmap v5875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:55 smithi188 ceph-mon[92425]: pgmap v5875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:54:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:57 smithi161 ceph-mon[91579]: pgmap v5876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:57 smithi188 ceph-mon[92425]: pgmap v5876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:54:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:54:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:54:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:54:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:54:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:54:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:54:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:54:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:54:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:54:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:54:59 smithi161 ceph-mon[91579]: pgmap v5877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:54:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:54:59 smithi188 ceph-mon[92425]: pgmap v5877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:01 smithi161 ceph-mon[91579]: pgmap v5878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:01 smithi188 ceph-mon[92425]: pgmap v5878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:03 smithi161 ceph-mon[91579]: pgmap v5879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:03 smithi188 ceph-mon[92425]: pgmap v5879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:05 smithi161 ceph-mon[91579]: pgmap v5880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:05 smithi188 ceph-mon[92425]: pgmap v5880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:07 smithi161 ceph-mon[91579]: pgmap v5881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:07 smithi188 ceph-mon[92425]: pgmap v5881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-mon[91579]: pgmap v5882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:09.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:55:09.505+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:55:09.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:55:09.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:55:09.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:55:09.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:55:09.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:55:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:55:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:55:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:55:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:55:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:55:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:55:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:55:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:55:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:55:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:55:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:55:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:55:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:55:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:55:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:55:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:55:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:55:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:55:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:55:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:55:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:55:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:55:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:55:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:55:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:55:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:55:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:55:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:55:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:55:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:55:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:55:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:55:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:55:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:55:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:55:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:55:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:55:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:55:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:55:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=307] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:55:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=307] Local address: 172.21.15.161, port 54204 2022-07-03T13:55:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=307] Peer address: 172.21.15.101, port 22 2022-07-03T13:55:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=307] Connection lost 2022-07-03T13:55:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=307] Aborting connection 2022-07-03T13:55:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:55:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:55:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:09 smithi188 ceph-mon[92425]: pgmap v5882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:55:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:55:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:55:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:55:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:55:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:55:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:55:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:55:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:55:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:55:10.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:55:10.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:55:10.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:55:10.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:55:10.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:55:10.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:55:10.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:55:10.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:55:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:55:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:55:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:55:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:55:10.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:55:10.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:55:10.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:55:10.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:55:10.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:55:10.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:55:10.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:55:10.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:55:10.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:55:10.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:55:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:55:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:55:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:55:10.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:55:10.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:55:10.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:55:10.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:55:10.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:55:10.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:55:10.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:55:10.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:55:10.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:55:10.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:55:10.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:55:10.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=307] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:55:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=307] Local address: 172.21.15.161, port 54204 2022-07-03T13:55:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=307] Peer address: 172.21.15.101, port 22 2022-07-03T13:55:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=307] Connection lost 2022-07-03T13:55:10.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=307] Aborting connection 2022-07-03T13:55:10.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:55:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:55:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:55:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:55:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:55:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:55:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:55:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:55:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:55:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:55:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:55:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:55:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:55:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:55:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:55:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:55:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:55:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:55:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:55:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:55:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:55:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:55:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:55:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:55:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:55:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:55:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:55:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:55:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:55:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:55:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:55:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:55:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:55:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:55:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:55:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:55:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:55:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:55:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:55:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:55:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:55:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:55:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:55:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:55:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:55:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=307] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:55:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=307] Local address: 172.21.15.161, port 54204 2022-07-03T13:55:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=307] Peer address: 172.21.15.101, port 22 2022-07-03T13:55:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=307] Connection lost 2022-07-03T13:55:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=307] Aborting connection 2022-07-03T13:55:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:11 smithi161 ceph-mon[91579]: pgmap v5883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:11 smithi188 ceph-mon[92425]: pgmap v5883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:13 smithi161 ceph-mon[91579]: pgmap v5884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:13 smithi188 ceph-mon[92425]: pgmap v5884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:15 smithi161 ceph-mon[91579]: pgmap v5885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:15 smithi188 ceph-mon[92425]: pgmap v5885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:17 smithi161 ceph-mon[91579]: pgmap v5886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:17 smithi188 ceph-mon[92425]: pgmap v5886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:19 smithi161 ceph-mon[91579]: pgmap v5887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:19 smithi188 ceph-mon[92425]: pgmap v5887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:21 smithi161 ceph-mon[91579]: pgmap v5888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:21 smithi188 ceph-mon[92425]: pgmap v5888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:23 smithi161 ceph-mon[91579]: pgmap v5889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:23 smithi188 ceph-mon[92425]: pgmap v5889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:25 smithi161 ceph-mon[91579]: pgmap v5890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:25 smithi188 ceph-mon[92425]: pgmap v5890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:27 smithi161 ceph-mon[91579]: pgmap v5891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:27 smithi188 ceph-mon[92425]: pgmap v5891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:29 smithi161 ceph-mon[91579]: pgmap v5892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:29 smithi188 ceph-mon[92425]: pgmap v5892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:31 smithi161 ceph-mon[91579]: pgmap v5893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:31 smithi188 ceph-mon[92425]: pgmap v5893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:33 smithi161 ceph-mon[91579]: pgmap v5894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:33 smithi188 ceph-mon[92425]: pgmap v5894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:35 smithi161 ceph-mon[91579]: pgmap v5895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:35 smithi188 ceph-mon[92425]: pgmap v5895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:37 smithi161 ceph-mon[91579]: pgmap v5896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:37 smithi188 ceph-mon[92425]: pgmap v5896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:39 smithi161 ceph-mon[91579]: pgmap v5897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:39 smithi188 ceph-mon[92425]: pgmap v5897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:41 smithi161 ceph-mon[91579]: pgmap v5898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:41 smithi188 ceph-mon[92425]: pgmap v5898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:43 smithi161 ceph-mon[91579]: pgmap v5899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:43 smithi188 ceph-mon[92425]: pgmap v5899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:45 smithi161 ceph-mon[91579]: pgmap v5900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:45 smithi188 ceph-mon[92425]: pgmap v5900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:47 smithi161 ceph-mon[91579]: pgmap v5901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:47 smithi188 ceph-mon[92425]: pgmap v5901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:49 smithi161 ceph-mon[91579]: pgmap v5902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:49 smithi188 ceph-mon[92425]: pgmap v5902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:51 smithi161 ceph-mon[91579]: pgmap v5903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:51 smithi188 ceph-mon[92425]: pgmap v5903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:53 smithi161 ceph-mon[91579]: pgmap v5904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:53 smithi188 ceph-mon[92425]: pgmap v5904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:55 smithi161 ceph-mon[91579]: pgmap v5905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:55 smithi188 ceph-mon[92425]: pgmap v5905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:55:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:57 smithi161 ceph-mon[91579]: pgmap v5906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:57 smithi188 ceph-mon[92425]: pgmap v5906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:55:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:55:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:55:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:55:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:55:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:55:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:55:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:55:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:55:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:55:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:55:59 smithi161 ceph-mon[91579]: pgmap v5907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:55:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:55:59 smithi188 ceph-mon[92425]: pgmap v5907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:01 smithi161 ceph-mon[91579]: pgmap v5908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:01 smithi188 ceph-mon[92425]: pgmap v5908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:03 smithi161 ceph-mon[91579]: pgmap v5909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:03 smithi188 ceph-mon[92425]: pgmap v5909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:05 smithi161 ceph-mon[91579]: pgmap v5910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:05 smithi188 ceph-mon[92425]: pgmap v5910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:07 smithi161 ceph-mon[91579]: pgmap v5911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:07 smithi188 ceph-mon[92425]: pgmap v5911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:09.600 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-mon[91579]: pgmap v5912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:09.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:56:09.541+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:56:09.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:56:09.602 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:56:09.602 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:56:09.602 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:56:09.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:56:09.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:56:09.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:56:09.604 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:56:09.604 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:56:09.604 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:56:09.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:56:09.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:56:09.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:56:09.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:56:09.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:56:09.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:56:09.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:56:09.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:56:09.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:56:09.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:56:09.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:56:09.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:56:09.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:56:09.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:56:09.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:56:09.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:56:09.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:56:09.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:56:09.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:56:09.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:56:09.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:56:09.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:56:09.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:56:09.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:56:09.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:56:09.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:56:09.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:56:09.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:56:09.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:56:09.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:56:09.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:56:09.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:56:09.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:56:09.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:56:09.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=309] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:56:09.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=309] Local address: 172.21.15.161, port 54232 2022-07-03T13:56:09.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=309] Peer address: 172.21.15.101, port 22 2022-07-03T13:56:09.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=309] Connection lost 2022-07-03T13:56:09.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=309] Aborting connection 2022-07-03T13:56:09.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:56:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:56:09.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:09 smithi188 ceph-mon[92425]: pgmap v5912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:56:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:56:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:56:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:56:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:56:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:56:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:56:10.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:56:10.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:56:10.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:56:10.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:56:10.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:56:10.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:56:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:56:10.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:56:10.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:56:10.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:56:10.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:56:10.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:56:10.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:56:10.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:56:10.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:56:10.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:56:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:56:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:56:10.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:56:10.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:56:10.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:56:10.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:56:10.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:56:10.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:56:10.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:56:10.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:56:10.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:56:10.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:56:10.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:56:10.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:56:10.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:56:10.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:56:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:56:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:56:10.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:56:10.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:56:10.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:56:10.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:56:10.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:56:10.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:56:10.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:56:10.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:56:10.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=309] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:56:10.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=309] Local address: 172.21.15.161, port 54232 2022-07-03T13:56:10.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=309] Peer address: 172.21.15.101, port 22 2022-07-03T13:56:10.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=309] Connection lost 2022-07-03T13:56:10.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=309] Aborting connection 2022-07-03T13:56:10.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:56:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:56:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:56:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:56:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:56:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:56:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:56:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:56:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:56:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:56:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:56:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:56:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:56:10.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:56:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:56:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:56:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:56:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:56:10.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:56:10.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:56:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:56:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:56:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:56:10.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:56:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:56:10.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:56:10.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:56:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:56:10.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:56:10.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:56:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:56:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:56:10.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:56:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:56:10.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:56:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:56:10.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:56:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:56:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:56:10.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:56:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:56:10.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:56:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:56:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:56:10.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:56:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:56:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:56:10.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:56:10.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:56:10.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=309] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:56:10.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=309] Local address: 172.21.15.161, port 54232 2022-07-03T13:56:10.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=309] Peer address: 172.21.15.101, port 22 2022-07-03T13:56:10.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=309] Connection lost 2022-07-03T13:56:10.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=309] Aborting connection 2022-07-03T13:56:10.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:11 smithi161 ceph-mon[91579]: pgmap v5913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:11 smithi188 ceph-mon[92425]: pgmap v5913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:13 smithi161 ceph-mon[91579]: pgmap v5914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:13 smithi188 ceph-mon[92425]: pgmap v5914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:15 smithi161 ceph-mon[91579]: pgmap v5915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:15 smithi188 ceph-mon[92425]: pgmap v5915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:17 smithi161 ceph-mon[91579]: pgmap v5916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:17 smithi188 ceph-mon[92425]: pgmap v5916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:19 smithi161 ceph-mon[91579]: pgmap v5917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:19 smithi188 ceph-mon[92425]: pgmap v5917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:21 smithi161 ceph-mon[91579]: pgmap v5918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:21 smithi188 ceph-mon[92425]: pgmap v5918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:23 smithi161 ceph-mon[91579]: pgmap v5919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:23 smithi188 ceph-mon[92425]: pgmap v5919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:25 smithi161 ceph-mon[91579]: pgmap v5920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:25 smithi188 ceph-mon[92425]: pgmap v5920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:27 smithi161 ceph-mon[91579]: pgmap v5921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:27 smithi188 ceph-mon[92425]: pgmap v5921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:29 smithi161 ceph-mon[91579]: pgmap v5922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:29 smithi188 ceph-mon[92425]: pgmap v5922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:31 smithi161 ceph-mon[91579]: pgmap v5923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:31 smithi188 ceph-mon[92425]: pgmap v5923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:33 smithi161 ceph-mon[91579]: pgmap v5924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:33 smithi188 ceph-mon[92425]: pgmap v5924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:35 smithi161 ceph-mon[91579]: pgmap v5925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:35 smithi188 ceph-mon[92425]: pgmap v5925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:37 smithi161 ceph-mon[91579]: pgmap v5926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:37 smithi188 ceph-mon[92425]: pgmap v5926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:39 smithi161 ceph-mon[91579]: pgmap v5927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:39 smithi188 ceph-mon[92425]: pgmap v5927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:41 smithi161 ceph-mon[91579]: pgmap v5928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:41 smithi188 ceph-mon[92425]: pgmap v5928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:43 smithi161 ceph-mon[91579]: pgmap v5929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:43 smithi188 ceph-mon[92425]: pgmap v5929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:43.696 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:45 smithi161 ceph-mon[91579]: pgmap v5930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:45 smithi188 ceph-mon[92425]: pgmap v5930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:47 smithi161 ceph-mon[91579]: pgmap v5931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:47 smithi188 ceph-mon[92425]: pgmap v5931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:49 smithi161 ceph-mon[91579]: pgmap v5932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:49 smithi188 ceph-mon[92425]: pgmap v5932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:51 smithi161 ceph-mon[91579]: pgmap v5933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:51 smithi188 ceph-mon[92425]: pgmap v5933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:53 smithi161 ceph-mon[91579]: pgmap v5934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:53 smithi188 ceph-mon[92425]: pgmap v5934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:55 smithi161 ceph-mon[91579]: pgmap v5935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:55 smithi188 ceph-mon[92425]: pgmap v5935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:56:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:57 smithi161 ceph-mon[91579]: pgmap v5936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:57 smithi188 ceph-mon[92425]: pgmap v5936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:56:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:56:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:56:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:56:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:56:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:56:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:56:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:56:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:56:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:56:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:56:59 smithi161 ceph-mon[91579]: pgmap v5937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:56:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:56:59 smithi188 ceph-mon[92425]: pgmap v5937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:01 smithi161 ceph-mon[91579]: pgmap v5938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:01 smithi188 ceph-mon[92425]: pgmap v5938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:03 smithi161 ceph-mon[91579]: pgmap v5939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:03 smithi188 ceph-mon[92425]: pgmap v5939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:05 smithi161 ceph-mon[91579]: pgmap v5940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:05 smithi188 ceph-mon[92425]: pgmap v5940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:07 smithi161 ceph-mon[91579]: pgmap v5941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:07 smithi188 ceph-mon[92425]: pgmap v5941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:09 smithi161 ceph-mon[91579]: pgmap v5942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:09 smithi188 ceph-mon[92425]: pgmap v5942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:10.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:57:10.044+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:57:10.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:57:10.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:57:10.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:57:10.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:57:10.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:57:10.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:57:10.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:57:10.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:57:10.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:57:10.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:57:10.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:57:10.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:57:10.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:57:10.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:57:10.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:57:10.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:57:10.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:57:10.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:57:10.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:57:10.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:57:10.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:57:10.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:57:10.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:57:10.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:57:10.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:57:10.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:57:10.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:57:10.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:57:10.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:57:10.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:57:10.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:57:10.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:57:10.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:57:10.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:57:10.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:57:10.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:57:10.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:57:10.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:57:10.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:57:10.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:57:10.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:57:10.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:57:10.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:57:10.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:57:10.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=311] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:57:10.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=311] Local address: 172.21.15.161, port 54262 2022-07-03T13:57:10.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=311] Peer address: 172.21.15.101, port 22 2022-07-03T13:57:10.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=311] Connection lost 2022-07-03T13:57:10.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=311] Aborting connection 2022-07-03T13:57:10.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:57:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:57:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:57:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:57:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:57:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:57:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:57:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:57:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:57:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:57:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:57:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:57:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:57:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:57:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:57:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:57:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:57:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:57:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:57:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:57:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:57:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:57:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:57:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:57:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:57:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:57:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:57:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:57:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:57:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:57:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:57:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:57:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:57:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:57:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:57:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:57:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:57:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:57:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:57:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:57:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:57:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:57:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:57:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:57:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:57:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:57:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:57:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:57:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:57:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:57:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:57:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:57:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:57:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=311] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:57:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=311] Local address: 172.21.15.161, port 54262 2022-07-03T13:57:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=311] Peer address: 172.21.15.101, port 22 2022-07-03T13:57:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=311] Connection lost 2022-07-03T13:57:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=311] Aborting connection 2022-07-03T13:57:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:11 smithi161 ceph-mon[91579]: pgmap v5943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:57:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:57:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:57:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:57:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:57:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:57:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:57:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:57:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:57:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:57:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:57:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:57:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:57:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:57:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:57:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:57:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:57:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:57:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:57:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:57:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:57:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:57:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:57:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:57:11.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:57:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:57:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:57:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:57:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:57:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:57:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:57:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:57:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:57:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:57:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:57:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:57:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:57:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:57:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:57:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:57:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:57:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:57:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:57:11.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:57:11.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:57:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:57:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=311] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:57:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=311] Local address: 172.21.15.161, port 54262 2022-07-03T13:57:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=311] Peer address: 172.21.15.101, port 22 2022-07-03T13:57:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=311] Connection lost 2022-07-03T13:57:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=311] Aborting connection 2022-07-03T13:57:11.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:11 smithi188 ceph-mon[92425]: pgmap v5943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:13 smithi161 ceph-mon[91579]: pgmap v5944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:13 smithi188 ceph-mon[92425]: pgmap v5944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:15 smithi161 ceph-mon[91579]: pgmap v5945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:15 smithi188 ceph-mon[92425]: pgmap v5945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:17 smithi161 ceph-mon[91579]: pgmap v5946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:17 smithi188 ceph-mon[92425]: pgmap v5946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:19 smithi161 ceph-mon[91579]: pgmap v5947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:19 smithi188 ceph-mon[92425]: pgmap v5947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:21 smithi161 ceph-mon[91579]: pgmap v5948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:21 smithi188 ceph-mon[92425]: pgmap v5948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:23 smithi161 ceph-mon[91579]: pgmap v5949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:23 smithi188 ceph-mon[92425]: pgmap v5949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:25 smithi161 ceph-mon[91579]: pgmap v5950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:25 smithi188 ceph-mon[92425]: pgmap v5950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:27 smithi161 ceph-mon[91579]: pgmap v5951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:27 smithi188 ceph-mon[92425]: pgmap v5951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:29 smithi161 ceph-mon[91579]: pgmap v5952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:29 smithi188 ceph-mon[92425]: pgmap v5952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:31 smithi161 ceph-mon[91579]: pgmap v5953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:31 smithi188 ceph-mon[92425]: pgmap v5953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:33 smithi161 ceph-mon[91579]: pgmap v5954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:33 smithi188 ceph-mon[92425]: pgmap v5954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:35 smithi161 ceph-mon[91579]: pgmap v5955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:35 smithi188 ceph-mon[92425]: pgmap v5955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:37 smithi161 ceph-mon[91579]: pgmap v5956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:37 smithi188 ceph-mon[92425]: pgmap v5956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:39 smithi161 ceph-mon[91579]: pgmap v5957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:39 smithi188 ceph-mon[92425]: pgmap v5957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:41 smithi161 ceph-mon[91579]: pgmap v5958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:41 smithi188 ceph-mon[92425]: pgmap v5958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:43 smithi161 ceph-mon[91579]: pgmap v5959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:43 smithi188 ceph-mon[92425]: pgmap v5959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:45 smithi161 ceph-mon[91579]: pgmap v5960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:45 smithi188 ceph-mon[92425]: pgmap v5960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:47 smithi161 ceph-mon[91579]: pgmap v5961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:47 smithi188 ceph-mon[92425]: pgmap v5961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:49 smithi161 ceph-mon[91579]: pgmap v5962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:49 smithi188 ceph-mon[92425]: pgmap v5962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:51 smithi161 ceph-mon[91579]: pgmap v5963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:51 smithi188 ceph-mon[92425]: pgmap v5963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:53 smithi161 ceph-mon[91579]: pgmap v5964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:53 smithi188 ceph-mon[92425]: pgmap v5964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:55 smithi161 ceph-mon[91579]: pgmap v5965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:55 smithi188 ceph-mon[92425]: pgmap v5965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:57:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:57 smithi161 ceph-mon[91579]: pgmap v5966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:57 smithi188 ceph-mon[92425]: pgmap v5966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:57:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:57:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:57:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:57:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:57:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:57:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:57:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:57:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:57:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:57:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:57:59 smithi161 ceph-mon[91579]: pgmap v5967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:57:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:57:59 smithi188 ceph-mon[92425]: pgmap v5967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:01 smithi161 ceph-mon[91579]: pgmap v5968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:01 smithi188 ceph-mon[92425]: pgmap v5968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:03 smithi161 ceph-mon[91579]: pgmap v5969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:03 smithi188 ceph-mon[92425]: pgmap v5969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:05 smithi161 ceph-mon[91579]: pgmap v5970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:05 smithi188 ceph-mon[92425]: pgmap v5970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:07 smithi161 ceph-mon[91579]: pgmap v5971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:07 smithi188 ceph-mon[92425]: pgmap v5971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:09 smithi161 ceph-mon[91579]: pgmap v5972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:09 smithi188 ceph-mon[92425]: pgmap v5972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:10.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:58:10.540+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:58:10.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:58:10.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:58:10.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:58:10.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:58:10.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:58:10.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:58:10.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:58:10.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:58:10.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:58:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:58:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:58:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:58:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:58:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:58:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:58:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:58:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:58:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:58:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:58:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:58:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:58:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:58:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:58:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:58:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:58:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:58:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:58:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:58:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:58:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:58:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:58:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:58:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:58:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:58:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:58:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:58:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:58:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:58:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:58:10.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:58:10.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:58:10.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:58:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:58:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:58:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=313] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:58:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=313] Local address: 172.21.15.161, port 54290 2022-07-03T13:58:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=313] Peer address: 172.21.15.101, port 22 2022-07-03T13:58:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=313] Connection lost 2022-07-03T13:58:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=313] Aborting connection 2022-07-03T13:58:10.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:58:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:58:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: pgmap v5973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:58:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:58:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:58:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:58:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:58:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:58:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:58:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:58:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:58:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:58:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:58:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:58:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:58:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:58:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:58:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:58:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:58:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:58:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:58:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:58:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:58:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:58:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:58:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:58:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:58:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:58:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:58:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:58:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:58:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:58:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:58:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:58:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:58:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:58:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:58:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:58:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:58:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:58:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:58:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:58:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:58:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:58:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:58:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:58:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:58:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:58:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:58:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:58:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:58:11.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=313] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:58:11.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=313] Local address: 172.21.15.161, port 54290 2022-07-03T13:58:11.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=313] Peer address: 172.21.15.101, port 22 2022-07-03T13:58:11.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=313] Connection lost 2022-07-03T13:58:11.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=313] Aborting connection 2022-07-03T13:58:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: pgmap v5973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:58:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:58:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:58:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:58:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:58:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:58:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:58:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:58:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:58:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:58:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:58:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:58:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:58:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:58:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:58:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:58:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:58:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:58:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:58:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:58:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:58:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:58:11.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:58:11.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:58:11.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:58:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:58:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:58:11.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:58:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:58:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:58:11.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:58:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:58:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:58:11.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:58:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:58:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:58:11.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:58:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:58:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:58:11.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:58:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:58:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:58:11.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:58:11.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:58:11.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:58:11.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:58:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:58:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:58:11.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:58:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=313] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:58:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=313] Local address: 172.21.15.161, port 54290 2022-07-03T13:58:11.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=313] Peer address: 172.21.15.101, port 22 2022-07-03T13:58:11.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=313] Connection lost 2022-07-03T13:58:11.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=313] Aborting connection 2022-07-03T13:58:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:13 smithi161 ceph-mon[91579]: pgmap v5974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:13 smithi188 ceph-mon[92425]: pgmap v5974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:15 smithi161 ceph-mon[91579]: pgmap v5975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:15 smithi188 ceph-mon[92425]: pgmap v5975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:17 smithi161 ceph-mon[91579]: pgmap v5976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:17 smithi188 ceph-mon[92425]: pgmap v5976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:19 smithi161 ceph-mon[91579]: pgmap v5977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:19 smithi188 ceph-mon[92425]: pgmap v5977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:21 smithi161 ceph-mon[91579]: pgmap v5978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:21 smithi188 ceph-mon[92425]: pgmap v5978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:23 smithi161 ceph-mon[91579]: pgmap v5979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:23 smithi188 ceph-mon[92425]: pgmap v5979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:25 smithi161 ceph-mon[91579]: pgmap v5980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:25 smithi188 ceph-mon[92425]: pgmap v5980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:27 smithi161 ceph-mon[91579]: pgmap v5981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:27 smithi188 ceph-mon[92425]: pgmap v5981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:29 smithi161 ceph-mon[91579]: pgmap v5982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:29 smithi188 ceph-mon[92425]: pgmap v5982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:31 smithi161 ceph-mon[91579]: pgmap v5983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:31 smithi188 ceph-mon[92425]: pgmap v5983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:33 smithi161 ceph-mon[91579]: pgmap v5984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:33 smithi188 ceph-mon[92425]: pgmap v5984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:35 smithi161 ceph-mon[91579]: pgmap v5985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:35 smithi188 ceph-mon[92425]: pgmap v5985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:37 smithi161 ceph-mon[91579]: pgmap v5986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:37 smithi188 ceph-mon[92425]: pgmap v5986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:39 smithi161 ceph-mon[91579]: pgmap v5987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:39 smithi188 ceph-mon[92425]: pgmap v5987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:41 smithi161 ceph-mon[91579]: pgmap v5988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:41 smithi188 ceph-mon[92425]: pgmap v5988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:43 smithi161 ceph-mon[91579]: pgmap v5989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:43 smithi188 ceph-mon[92425]: pgmap v5989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:45 smithi161 ceph-mon[91579]: pgmap v5990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:45 smithi188 ceph-mon[92425]: pgmap v5990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:47 smithi161 ceph-mon[91579]: pgmap v5991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:47 smithi188 ceph-mon[92425]: pgmap v5991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:49 smithi161 ceph-mon[91579]: pgmap v5992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:49 smithi188 ceph-mon[92425]: pgmap v5992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:51 smithi161 ceph-mon[91579]: pgmap v5993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:51 smithi188 ceph-mon[92425]: pgmap v5993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:53 smithi161 ceph-mon[91579]: pgmap v5994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:53 smithi188 ceph-mon[92425]: pgmap v5994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:55 smithi161 ceph-mon[91579]: pgmap v5995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:55 smithi188 ceph-mon[92425]: pgmap v5995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:58:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:57 smithi161 ceph-mon[91579]: pgmap v5996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:57 smithi188 ceph-mon[92425]: pgmap v5996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:58:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:58:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:58:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:58:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:58:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:58:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:58:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:58:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:58:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:58:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:58:59 smithi161 ceph-mon[91579]: pgmap v5997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:58:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:58:59 smithi188 ceph-mon[92425]: pgmap v5997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:01 smithi161 ceph-mon[91579]: pgmap v5998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:01 smithi188 ceph-mon[92425]: pgmap v5998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:03 smithi188 ceph-mon[92425]: pgmap v5999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:03 smithi161 ceph-mon[91579]: pgmap v5999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:05 smithi161 ceph-mon[91579]: pgmap v6000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:05 smithi188 ceph-mon[92425]: pgmap v6000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:07 smithi161 ceph-mon[91579]: pgmap v6001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:07 smithi188 ceph-mon[92425]: pgmap v6001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:09 smithi161 ceph-mon[91579]: pgmap v6002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:09 smithi188 ceph-mon[92425]: pgmap v6002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:11.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:59:11.040+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:59:11.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:59:11.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:59:11.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T13:59:11.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:59:11.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:59:11.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:59:11.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T13:59:11.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:59:11.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T13:59:11.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:59:11.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T13:59:11.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:59:11.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:59:11.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T13:59:11.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:59:11.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T13:59:11.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:59:11.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T13:59:11.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:59:11.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T13:59:11.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:59:11.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:59:11.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:59:11.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:59:11.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:59:11.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T13:59:11.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:59:11.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:59:11.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:59:11.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T13:59:11.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:59:11.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T13:59:11.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:59:11.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:59:11.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:59:11.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T13:59:11.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:59:11.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T13:59:11.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:59:11.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T13:59:11.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:59:11.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T13:59:11.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:59:11.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:59:11.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=315] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:59:11.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=315] Local address: 172.21.15.161, port 54318 2022-07-03T13:59:11.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=315] Peer address: 172.21.15.101, port 22 2022-07-03T13:59:11.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=315] Connection lost 2022-07-03T13:59:11.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=315] Aborting connection 2022-07-03T13:59:11.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T13:59:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:11 smithi188 ceph-mon[92425]: pgmap v6003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:59:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:59:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:59:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-mon[91579]: pgmap v6003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T13:59:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T13:59:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T13:59:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:59:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:59:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:59:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:59:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T13:59:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:59:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:59:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:59:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T13:59:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:59:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T13:59:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:59:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T13:59:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:59:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:59:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:59:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T13:59:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T13:59:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:59:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T13:59:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:59:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T13:59:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:59:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:59:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:59:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:59:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:59:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:59:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:59:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:59:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:59:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T13:59:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:59:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T13:59:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:59:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:59:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:59:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:59:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:59:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T13:59:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:59:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:59:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:59:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T13:59:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:59:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:59:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=315] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:59:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=315] Local address: 172.21.15.161, port 54318 2022-07-03T13:59:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=315] Peer address: 172.21.15.101, port 22 2022-07-03T13:59:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=315] Connection lost 2022-07-03T13:59:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=315] Aborting connection 2022-07-03T13:59:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T13:59:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T13:59:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:59:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T13:59:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T13:59:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T13:59:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T13:59:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T13:59:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T13:59:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T13:59:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T13:59:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T13:59:12.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T13:59:12.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T13:59:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:59:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T13:59:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T13:59:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T13:59:12.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T13:59:12.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T13:59:12.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T13:59:12.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T13:59:12.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T13:59:12.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T13:59:12.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T13:59:12.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T13:59:12.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T13:59:12.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T13:59:12.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T13:59:12.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T13:59:12.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T13:59:12.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T13:59:12.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T13:59:12.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T13:59:12.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T13:59:12.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T13:59:12.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T13:59:12.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T13:59:12.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T13:59:12.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T13:59:12.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T13:59:12.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T13:59:12.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T13:59:12.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T13:59:12.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T13:59:12.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T13:59:12.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=315] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T13:59:12.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=315] Local address: 172.21.15.161, port 54318 2022-07-03T13:59:12.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=315] Peer address: 172.21.15.101, port 22 2022-07-03T13:59:12.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=315] Connection lost 2022-07-03T13:59:12.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=315] Aborting connection 2022-07-03T13:59:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:13 smithi188 ceph-mon[92425]: pgmap v6004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:13 smithi161 ceph-mon[91579]: pgmap v6004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:15 smithi161 ceph-mon[91579]: pgmap v6005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:15 smithi188 ceph-mon[92425]: pgmap v6005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:17 smithi161 ceph-mon[91579]: pgmap v6006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:17 smithi188 ceph-mon[92425]: pgmap v6006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:19 smithi161 ceph-mon[91579]: pgmap v6007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:19 smithi188 ceph-mon[92425]: pgmap v6007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:20.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:21 smithi188 ceph-mon[92425]: pgmap v6008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:21 smithi161 ceph-mon[91579]: pgmap v6008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:23 smithi188 ceph-mon[92425]: pgmap v6009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:23 smithi161 ceph-mon[91579]: pgmap v6009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:25 smithi161 ceph-mon[91579]: pgmap v6010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:25 smithi188 ceph-mon[92425]: pgmap v6010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:27 smithi161 ceph-mon[91579]: pgmap v6011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:27 smithi188 ceph-mon[92425]: pgmap v6011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:29 smithi188 ceph-mon[92425]: pgmap v6012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:29 smithi161 ceph-mon[91579]: pgmap v6012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:30.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:31 smithi188 ceph-mon[92425]: pgmap v6013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:31 smithi161 ceph-mon[91579]: pgmap v6013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:33 smithi188 ceph-mon[92425]: pgmap v6014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:33 smithi161 ceph-mon[91579]: pgmap v6014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:35 smithi161 ceph-mon[91579]: pgmap v6015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:35 smithi188 ceph-mon[92425]: pgmap v6015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:37 smithi161 ceph-mon[91579]: pgmap v6016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:37 smithi188 ceph-mon[92425]: pgmap v6016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:39 smithi188 ceph-mon[92425]: pgmap v6017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:39 smithi161 ceph-mon[91579]: pgmap v6017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:41 smithi188 ceph-mon[92425]: pgmap v6018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:41 smithi161 ceph-mon[91579]: pgmap v6018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:43 smithi188 ceph-mon[92425]: pgmap v6019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:43 smithi161 ceph-mon[91579]: pgmap v6019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:45 smithi161 ceph-mon[91579]: pgmap v6020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:45 smithi188 ceph-mon[92425]: pgmap v6020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:47 smithi188 ceph-mon[92425]: pgmap v6021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:47 smithi161 ceph-mon[91579]: pgmap v6021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:49 smithi188 ceph-mon[92425]: pgmap v6022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:49 smithi161 ceph-mon[91579]: pgmap v6022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:51 smithi188 ceph-mon[92425]: pgmap v6023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:51 smithi161 ceph-mon[91579]: pgmap v6023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:53 smithi188 ceph-mon[92425]: pgmap v6024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:53 smithi161 ceph-mon[91579]: pgmap v6024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:55 smithi161 ceph-mon[91579]: pgmap v6025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:55 smithi188 ceph-mon[92425]: pgmap v6025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T13:59:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:57 smithi188 ceph-mon[92425]: pgmap v6026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:57 smithi161 ceph-mon[91579]: pgmap v6026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T13:59:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:59:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:59:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:59:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:59:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:59:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:59:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T13:59:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T13:59:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T13:59:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 13:59:59 smithi188 ceph-mon[92425]: pgmap v6027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T13:59:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 13:59:59 smithi161 ceph-mon[91579]: pgmap v6027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:00:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:00:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:01 smithi188 ceph-mon[92425]: pgmap v6028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:01 smithi161 ceph-mon[91579]: pgmap v6028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:03 smithi188 ceph-mon[92425]: pgmap v6029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:03 smithi161 ceph-mon[91579]: pgmap v6029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:05 smithi161 ceph-mon[91579]: pgmap v6030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:05 smithi188 ceph-mon[92425]: pgmap v6030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:07 smithi188 ceph-mon[92425]: pgmap v6031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:07 smithi161 ceph-mon[91579]: pgmap v6031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:09 smithi188 ceph-mon[92425]: pgmap v6032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:09 smithi161 ceph-mon[91579]: pgmap v6032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:11 smithi188 ceph-mon[92425]: pgmap v6033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-mon[91579]: pgmap v6033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:11.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:00:11.541+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:00:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:00:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:00:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:00:11.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:00:11.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:00:11.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:00:11.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:00:11.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:00:11.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:00:11.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:00:11.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:00:11.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:00:11.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:00:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:00:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:00:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:00:11.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:00:11.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:00:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:00:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:00:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:00:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:00:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:00:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:00:11.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:00:11.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:00:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:00:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:00:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:00:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:00:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:00:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:00:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:00:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:00:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:00:11.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:00:11.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:00:11.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:00:11.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:00:11.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:00:11.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:00:11.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:00:11.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:00:11.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:00:11.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=317] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:00:11.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=317] Local address: 172.21.15.161, port 54346 2022-07-03T14:00:11.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=317] Peer address: 172.21.15.101, port 22 2022-07-03T14:00:11.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=317] Connection lost 2022-07-03T14:00:11.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=317] Aborting connection 2022-07-03T14:00:11.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:00:12.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:00:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:00:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:00:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:00:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:00:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:00:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:00:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:00:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:00:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:00:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:00:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:00:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:00:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:00:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:00:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:00:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:00:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:00:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:00:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:00:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:00:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:00:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:00:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:00:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:00:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:00:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:00:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:00:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:00:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:00:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:00:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:00:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:00:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:00:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:00:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:00:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:00:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:00:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:00:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:00:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:00:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:00:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:00:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:00:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:00:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:00:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:00:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:00:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:00:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=317] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:00:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=317] Local address: 172.21.15.161, port 54346 2022-07-03T14:00:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=317] Peer address: 172.21.15.101, port 22 2022-07-03T14:00:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=317] Connection lost 2022-07-03T14:00:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=317] Aborting connection 2022-07-03T14:00:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:00:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:00:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:00:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:00:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:00:12.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:00:12.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:00:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:00:12.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:00:12.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:00:12.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:00:12.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:00:12.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:00:12.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:00:12.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:00:12.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:00:12.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:00:12.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:00:12.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:00:12.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:00:12.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:00:12.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:00:12.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:00:12.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:00:12.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:00:12.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:00:12.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:00:12.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:00:12.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:00:12.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:00:12.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:00:12.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:00:12.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:00:12.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:00:12.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:00:12.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:00:12.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:00:12.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:00:12.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:00:12.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:00:12.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:00:12.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:00:12.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:00:12.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:00:12.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:00:12.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:00:12.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:00:12.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:00:12.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:00:12.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=317] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:00:12.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=317] Local address: 172.21.15.161, port 54346 2022-07-03T14:00:12.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=317] Peer address: 172.21.15.101, port 22 2022-07-03T14:00:12.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=317] Connection lost 2022-07-03T14:00:12.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=317] Aborting connection 2022-07-03T14:00:12.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:12.914 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:00:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:13 smithi188 ceph-mon[92425]: pgmap v6034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:13 smithi161 ceph-mon[91579]: pgmap v6034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:15 smithi161 ceph-mon[91579]: pgmap v6035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:15 smithi188 ceph-mon[92425]: pgmap v6035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:17 smithi188 ceph-mon[92425]: pgmap v6036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:17 smithi161 ceph-mon[91579]: pgmap v6036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:19 smithi188 ceph-mon[92425]: pgmap v6037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:19 smithi161 ceph-mon[91579]: pgmap v6037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:21 smithi188 ceph-mon[92425]: pgmap v6038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:21 smithi161 ceph-mon[91579]: pgmap v6038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:23 smithi188 ceph-mon[92425]: pgmap v6039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:23 smithi161 ceph-mon[91579]: pgmap v6039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:25.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:00:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:00:24.807+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T14:00:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:25 smithi161 ceph-mon[91579]: pgmap v6040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:25 smithi188 ceph-mon[92425]: pgmap v6040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:00:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:27 smithi188 ceph-mon[92425]: pgmap v6041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:27 smithi161 ceph-mon[91579]: pgmap v6041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:29 smithi188 ceph-mon[92425]: pgmap v6042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:29 smithi161 ceph-mon[91579]: pgmap v6042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:31 smithi188 ceph-mon[92425]: pgmap v6043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:31 smithi161 ceph-mon[91579]: pgmap v6043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:33 smithi188 ceph-mon[92425]: pgmap v6044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:33 smithi161 ceph-mon[91579]: pgmap v6044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:35 smithi161 ceph-mon[91579]: pgmap v6045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:35 smithi188 ceph-mon[92425]: pgmap v6045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:37 smithi188 ceph-mon[92425]: pgmap v6046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:37 smithi161 ceph-mon[91579]: pgmap v6046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:39 smithi188 ceph-mon[92425]: pgmap v6047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:39 smithi161 ceph-mon[91579]: pgmap v6047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:41 smithi188 ceph-mon[92425]: pgmap v6048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:41 smithi161 ceph-mon[91579]: pgmap v6048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:43 smithi188 ceph-mon[92425]: pgmap v6049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:43 smithi161 ceph-mon[91579]: pgmap v6049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:45 smithi161 ceph-mon[91579]: pgmap v6050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:45 smithi188 ceph-mon[92425]: pgmap v6050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:47 smithi188 ceph-mon[92425]: pgmap v6051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:47 smithi161 ceph-mon[91579]: pgmap v6051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:49 smithi188 ceph-mon[92425]: pgmap v6052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:49 smithi161 ceph-mon[91579]: pgmap v6052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:51 smithi188 ceph-mon[92425]: pgmap v6053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:51 smithi161 ceph-mon[91579]: pgmap v6053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:53 smithi188 ceph-mon[92425]: pgmap v6054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:53 smithi161 ceph-mon[91579]: pgmap v6054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:55 smithi161 ceph-mon[91579]: pgmap v6055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:55 smithi188 ceph-mon[92425]: pgmap v6055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:57 smithi188 ceph-mon[92425]: pgmap v6056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:57 smithi161 ceph-mon[91579]: pgmap v6056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:00:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:00:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:00:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:00:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:00:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:00:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:00:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:00:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:00:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:00:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:00:59 smithi188 ceph-mon[92425]: pgmap v6057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:00:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:00:59 smithi161 ceph-mon[91579]: pgmap v6057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:01 smithi188 ceph-mon[92425]: pgmap v6058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:01.876 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:01 smithi161 ceph-mon[91579]: pgmap v6058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:03 smithi161 ceph-mon[91579]: pgmap v6059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:03 smithi188 ceph-mon[92425]: pgmap v6059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:05 smithi161 ceph-mon[91579]: pgmap v6060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:05 smithi188 ceph-mon[92425]: pgmap v6060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:07 smithi188 ceph-mon[92425]: pgmap v6061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:07 smithi161 ceph-mon[91579]: pgmap v6061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:09 smithi161 ceph-mon[91579]: pgmap v6062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:09 smithi188 ceph-mon[92425]: pgmap v6062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:11 smithi161 ceph-mon[91579]: pgmap v6063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:11 smithi188 ceph-mon[92425]: pgmap v6063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:13 smithi161 ceph-mon[91579]: pgmap v6064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:01:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:01:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:01:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:13 smithi188 ceph-mon[92425]: pgmap v6064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:01:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:01:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:01:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:15 smithi161 ceph-mon[91579]: pgmap v6065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:15 smithi188 ceph-mon[92425]: pgmap v6065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:17 smithi188 ceph-mon[92425]: pgmap v6066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:17 smithi161 ceph-mon[91579]: pgmap v6066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:19 smithi161 ceph-mon[91579]: pgmap v6067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:19 smithi188 ceph-mon[92425]: pgmap v6067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:21 smithi161 ceph-mon[91579]: pgmap v6068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:21 smithi188 ceph-mon[92425]: pgmap v6068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:23 smithi161 ceph-mon[91579]: pgmap v6069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:23 smithi188 ceph-mon[92425]: pgmap v6069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:25 smithi161 ceph-mon[91579]: pgmap v6070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:25 smithi188 ceph-mon[92425]: pgmap v6070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:27 smithi188 ceph-mon[92425]: pgmap v6071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:27 smithi161 ceph-mon[91579]: pgmap v6071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:29 smithi161 ceph-mon[91579]: pgmap v6072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:29 smithi188 ceph-mon[92425]: pgmap v6072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:31 smithi161 ceph-mon[91579]: pgmap v6073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:31 smithi188 ceph-mon[92425]: pgmap v6073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:33 smithi161 ceph-mon[91579]: pgmap v6074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:33 smithi188 ceph-mon[92425]: pgmap v6074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:01:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:35 smithi161 ceph-mon[91579]: pgmap v6075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:35 smithi188 ceph-mon[92425]: pgmap v6075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:37 smithi161 ceph-mon[91579]: pgmap v6076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:37 smithi188 ceph-mon[92425]: pgmap v6076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:39 smithi161 ceph-mon[91579]: pgmap v6077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:39 smithi188 ceph-mon[92425]: pgmap v6077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:41 smithi161 ceph-mon[91579]: pgmap v6078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:41 smithi188 ceph-mon[92425]: pgmap v6078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:42.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:01:42.480+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:01:42.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:01:42.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:01:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:01:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:01:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:01:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:01:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:01:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:01:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:01:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:01:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:01:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:01:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:01:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:01:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:01:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:01:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:01:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:01:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:01:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:01:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:01:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:01:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:01:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:01:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:01:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:01:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:01:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:01:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:01:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:01:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:01:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:01:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:01:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:01:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:01:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:01:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:01:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:01:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:01:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:01:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:01:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:01:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:01:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:01:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=319] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:01:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=319] Local address: 172.21.15.161, port 54356 2022-07-03T14:01:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=319] Peer address: 172.21.15.101, port 22 2022-07-03T14:01:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=319] Connection lost 2022-07-03T14:01:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=319] Aborting connection 2022-07-03T14:01:42.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:01:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:01:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:43 smithi161 ceph-mon[91579]: pgmap v6079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:01:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:01:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:01:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:01:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:01:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:01:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:01:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:01:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:01:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:01:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:01:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:01:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:01:43.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:01:43.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:01:43.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:01:43.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:01:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:01:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:01:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:01:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:01:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:01:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:01:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:01:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:01:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:01:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:01:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:01:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:01:43.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:01:43.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:01:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:01:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:01:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:01:43.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:01:43.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:01:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:01:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:01:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:01:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:01:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:01:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:01:43.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:01:43.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:01:43.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:01:43.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:01:43.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=319] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:01:43.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=319] Local address: 172.21.15.161, port 54356 2022-07-03T14:01:43.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=319] Peer address: 172.21.15.101, port 22 2022-07-03T14:01:43.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=319] Connection lost 2022-07-03T14:01:43.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=319] Aborting connection 2022-07-03T14:01:43.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:43 smithi188 ceph-mon[92425]: pgmap v6079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:01:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:01:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:01:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:01:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:01:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:01:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:01:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:01:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:01:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:01:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:01:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:01:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:01:43.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:01:43.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:01:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:01:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:01:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:01:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:01:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:01:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:01:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:01:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:01:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:01:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:01:43.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:01:43.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:01:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:01:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:01:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:01:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:01:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:01:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:01:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:01:43.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:01:43.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:01:43.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:01:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:01:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:01:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:01:43.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:01:43.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:01:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:01:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=319] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:01:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=319] Local address: 172.21.15.161, port 54356 2022-07-03T14:01:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=319] Peer address: 172.21.15.101, port 22 2022-07-03T14:01:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=319] Connection lost 2022-07-03T14:01:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=319] Aborting connection 2022-07-03T14:01:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:45 smithi161 ceph-mon[91579]: pgmap v6080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:45 smithi188 ceph-mon[92425]: pgmap v6080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:47 smithi161 ceph-mon[91579]: pgmap v6081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:47 smithi188 ceph-mon[92425]: pgmap v6081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:49 smithi161 ceph-mon[91579]: pgmap v6082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:49 smithi188 ceph-mon[92425]: pgmap v6082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:51 smithi161 ceph-mon[91579]: pgmap v6083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:51 smithi188 ceph-mon[92425]: pgmap v6083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:53 smithi161 ceph-mon[91579]: pgmap v6084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:53 smithi188 ceph-mon[92425]: pgmap v6084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:55 smithi161 ceph-mon[91579]: pgmap v6085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:55 smithi188 ceph-mon[92425]: pgmap v6085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:57 smithi161 ceph-mon[91579]: pgmap v6086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:57 smithi188 ceph-mon[92425]: pgmap v6086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:01:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:01:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:01:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:01:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:01:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:01:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:01:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:01:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:01:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:01:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:01:59 smithi161 ceph-mon[91579]: pgmap v6087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:01:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:01:59 smithi188 ceph-mon[92425]: pgmap v6087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:01 smithi161 ceph-mon[91579]: pgmap v6088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:01 smithi188 ceph-mon[92425]: pgmap v6088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:03 smithi161 ceph-mon[91579]: pgmap v6089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:03 smithi188 ceph-mon[92425]: pgmap v6089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:05 smithi161 ceph-mon[91579]: pgmap v6090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:05 smithi188 ceph-mon[92425]: pgmap v6090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:07 smithi161 ceph-mon[91579]: pgmap v6091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:07 smithi188 ceph-mon[92425]: pgmap v6091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:09 smithi161 ceph-mon[91579]: pgmap v6092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:09 smithi188 ceph-mon[92425]: pgmap v6092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:11 smithi161 ceph-mon[91579]: pgmap v6093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:11 smithi188 ceph-mon[92425]: pgmap v6093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:13 smithi161 ceph-mon[91579]: pgmap v6094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:13 smithi188 ceph-mon[92425]: pgmap v6094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:15 smithi161 ceph-mon[91579]: pgmap v6095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:15 smithi188 ceph-mon[92425]: pgmap v6095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:17 smithi161 ceph-mon[91579]: pgmap v6096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:17 smithi188 ceph-mon[92425]: pgmap v6096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:19 smithi161 ceph-mon[91579]: pgmap v6097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:19 smithi188 ceph-mon[92425]: pgmap v6097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:21 smithi161 ceph-mon[91579]: pgmap v6098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:21 smithi188 ceph-mon[92425]: pgmap v6098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:23 smithi161 ceph-mon[91579]: pgmap v6099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:23 smithi188 ceph-mon[92425]: pgmap v6099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:25 smithi161 ceph-mon[91579]: pgmap v6100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:25 smithi188 ceph-mon[92425]: pgmap v6100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:27 smithi161 ceph-mon[91579]: pgmap v6101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:27 smithi188 ceph-mon[92425]: pgmap v6101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:29 smithi161 ceph-mon[91579]: pgmap v6102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:29 smithi188 ceph-mon[92425]: pgmap v6102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:31 smithi161 ceph-mon[91579]: pgmap v6103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:31 smithi188 ceph-mon[92425]: pgmap v6103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:33 smithi161 ceph-mon[91579]: pgmap v6104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:33 smithi188 ceph-mon[92425]: pgmap v6104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:35 smithi161 ceph-mon[91579]: pgmap v6105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:35 smithi188 ceph-mon[92425]: pgmap v6105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:37 smithi161 ceph-mon[91579]: pgmap v6106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:37 smithi188 ceph-mon[92425]: pgmap v6106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:39 smithi161 ceph-mon[91579]: pgmap v6107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:39 smithi188 ceph-mon[92425]: pgmap v6107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:41 smithi161 ceph-mon[91579]: pgmap v6108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:41 smithi188 ceph-mon[92425]: pgmap v6108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:42.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:02:42.514+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:02:42.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:02:42.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:02:42.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:02:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:02:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:02:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:02:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:02:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:02:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:02:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:02:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:02:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:02:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:02:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:02:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:02:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:02:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:02:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:02:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:02:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:02:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:02:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:02:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:02:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:02:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:02:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:02:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:02:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:02:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:02:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:02:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:02:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:02:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:02:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:02:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:02:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:02:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:02:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:02:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:02:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:02:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:02:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:02:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:02:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:02:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=321] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:02:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=321] Local address: 172.21.15.161, port 54360 2022-07-03T14:02:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=321] Peer address: 172.21.15.101, port 22 2022-07-03T14:02:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=321] Connection lost 2022-07-03T14:02:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=321] Aborting connection 2022-07-03T14:02:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:02:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:02:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: pgmap v6109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:02:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:02:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:02:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:02:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:02:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:02:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:02:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:02:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:02:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:02:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:02:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:02:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:02:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:02:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:02:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:02:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:02:43.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:02:43.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:02:43.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:02:43.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:02:43.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:02:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:02:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:02:43.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:02:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:02:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:02:43.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:02:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:02:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:02:43.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:02:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:02:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:02:43.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:02:43.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:02:43.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:02:43.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:02:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:02:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:02:43.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:02:43.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:02:43.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:02:43.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:02:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:02:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:02:43.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:02:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:02:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:02:43.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:02:43.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=321] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:02:43.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=321] Local address: 172.21.15.161, port 54360 2022-07-03T14:02:43.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=321] Peer address: 172.21.15.101, port 22 2022-07-03T14:02:43.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=321] Connection lost 2022-07-03T14:02:43.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=321] Aborting connection 2022-07-03T14:02:43.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: pgmap v6109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:02:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:02:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:02:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:02:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:02:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:02:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:02:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:02:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:02:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:02:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:02:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:02:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:02:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:02:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:02:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:02:43.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:02:43.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:02:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:02:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:02:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:02:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:02:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:02:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:02:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:02:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:02:43.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:02:43.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:02:43.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:02:43.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:02:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:02:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:02:43.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:02:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:02:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:02:43.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:02:43.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:02:43.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:02:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:02:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:02:43.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:02:43.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:02:43.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:02:43.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:02:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:02:43.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:02:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:02:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:02:43.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:02:43.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=321] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:02:43.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=321] Local address: 172.21.15.161, port 54360 2022-07-03T14:02:43.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=321] Peer address: 172.21.15.101, port 22 2022-07-03T14:02:43.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=321] Connection lost 2022-07-03T14:02:43.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=321] Aborting connection 2022-07-03T14:02:43.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:45 smithi161 ceph-mon[91579]: pgmap v6110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:45 smithi188 ceph-mon[92425]: pgmap v6110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:02:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:02:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:02:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:02:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:02:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:02:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:47 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:02:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:47 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:02:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:47 smithi161 ceph-mon[91579]: pgmap v6111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:47 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:02:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:47 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:02:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:47 smithi188 ceph-mon[92425]: pgmap v6111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:49 smithi161 ceph-mon[91579]: pgmap v6112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:49 smithi188 ceph-mon[92425]: pgmap v6112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:51 smithi161 ceph-mon[91579]: pgmap v6113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:51 smithi188 ceph-mon[92425]: pgmap v6113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:53 smithi161 ceph-mon[91579]: pgmap v6114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:53 smithi188 ceph-mon[92425]: pgmap v6114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:55 smithi161 ceph-mon[91579]: pgmap v6115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:55 smithi188 ceph-mon[92425]: pgmap v6115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:57 smithi161 ceph-mon[91579]: pgmap v6116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:57 smithi188 ceph-mon[92425]: pgmap v6116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:02:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:02:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:02:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:02:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:02:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:02:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:02:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:02:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:02:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:02:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:02:59 smithi161 ceph-mon[91579]: pgmap v6117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:02:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:02:59 smithi188 ceph-mon[92425]: pgmap v6117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:01 smithi161 ceph-mon[91579]: pgmap v6118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:01 smithi188 ceph-mon[92425]: pgmap v6118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:03 smithi161 ceph-mon[91579]: pgmap v6119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:03 smithi188 ceph-mon[92425]: pgmap v6119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:05 smithi161 ceph-mon[91579]: pgmap v6120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:05 smithi188 ceph-mon[92425]: pgmap v6120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:07 smithi161 ceph-mon[91579]: pgmap v6121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:07 smithi188 ceph-mon[92425]: pgmap v6121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:09 smithi161 ceph-mon[91579]: pgmap v6122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:09 smithi188 ceph-mon[92425]: pgmap v6122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:11 smithi161 ceph-mon[91579]: pgmap v6123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:11 smithi188 ceph-mon[92425]: pgmap v6123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:13 smithi161 ceph-mon[91579]: pgmap v6124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:13 smithi188 ceph-mon[92425]: pgmap v6124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:15 smithi161 ceph-mon[91579]: pgmap v6125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:15 smithi188 ceph-mon[92425]: pgmap v6125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:17 smithi161 ceph-mon[91579]: pgmap v6126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:17 smithi188 ceph-mon[92425]: pgmap v6126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:19 smithi161 ceph-mon[91579]: pgmap v6127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:19 smithi188 ceph-mon[92425]: pgmap v6127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:21 smithi161 ceph-mon[91579]: pgmap v6128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:21 smithi188 ceph-mon[92425]: pgmap v6128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:23 smithi161 ceph-mon[91579]: pgmap v6129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:23 smithi188 ceph-mon[92425]: pgmap v6129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:25 smithi161 ceph-mon[91579]: pgmap v6130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:25 smithi188 ceph-mon[92425]: pgmap v6130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:27 smithi161 ceph-mon[91579]: pgmap v6131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:27 smithi188 ceph-mon[92425]: pgmap v6131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:29 smithi161 ceph-mon[91579]: pgmap v6132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:29 smithi188 ceph-mon[92425]: pgmap v6132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:31 smithi161 ceph-mon[91579]: pgmap v6133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:31 smithi188 ceph-mon[92425]: pgmap v6133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:33 smithi161 ceph-mon[91579]: pgmap v6134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:33 smithi188 ceph-mon[92425]: pgmap v6134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:35 smithi161 ceph-mon[91579]: pgmap v6135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:35 smithi188 ceph-mon[92425]: pgmap v6135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:37 smithi161 ceph-mon[91579]: pgmap v6136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:37 smithi188 ceph-mon[92425]: pgmap v6136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:39 smithi161 ceph-mon[91579]: pgmap v6137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:39 smithi188 ceph-mon[92425]: pgmap v6137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:41 smithi161 ceph-mon[91579]: pgmap v6138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:41 smithi188 ceph-mon[92425]: pgmap v6138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:43 smithi161 ceph-mon[91579]: pgmap v6139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:43 smithi188 ceph-mon[92425]: pgmap v6139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-mon[91579]: pgmap v6140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:45 smithi188 ceph-mon[92425]: pgmap v6140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:46.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:03:45.775+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:03:46.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:03:46.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:03:46.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:03:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:03:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:03:46.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:03:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:03:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:03:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:03:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:03:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:03:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:03:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:03:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:03:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:03:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:03:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:03:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:03:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:03:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:03:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:03:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:03:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:03:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:03:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:03:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:03:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:03:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:03:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:03:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:03:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:03:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:03:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:03:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:03:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:03:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:03:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:03:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:03:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:03:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:03:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:03:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:03:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:03:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:03:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=323] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:03:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=323] Local address: 172.21.15.161, port 54364 2022-07-03T14:03:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=323] Peer address: 172.21.15.101, port 22 2022-07-03T14:03:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=323] Connection lost 2022-07-03T14:03:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=323] Aborting connection 2022-07-03T14:03:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:03:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:03:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:03:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:03:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:03:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:03:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:03:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:03:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:03:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:03:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:03:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:03:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:03:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:03:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:03:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:03:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:03:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:03:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:03:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:03:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:03:47.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:03:47.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:03:47.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:03:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:03:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:03:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:03:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:03:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:03:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:03:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:03:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:03:47.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:03:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:03:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:03:47.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:03:47.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:03:47.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:03:47.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:03:47.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:03:47.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:03:47.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:03:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:03:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:03:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:03:47.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:03:47.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:03:47.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:03:47.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:03:47.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:03:47.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:03:47.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:03:47.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:03:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:03:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=323] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:03:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=323] Local address: 172.21.15.161, port 54364 2022-07-03T14:03:47.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=323] Peer address: 172.21.15.101, port 22 2022-07-03T14:03:47.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=323] Connection lost 2022-07-03T14:03:47.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=323] Aborting connection 2022-07-03T14:03:47.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:47 smithi161 ceph-mon[91579]: pgmap v6141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:03:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:03:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:03:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:03:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:03:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:03:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:03:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:03:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:03:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:03:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:03:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:03:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:03:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:03:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:03:47.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:03:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:03:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:03:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:03:47.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:03:47.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:03:47.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:03:47.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:03:47.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:03:47.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:03:47.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:03:47.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:03:47.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:03:47.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:03:47.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:03:47.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:03:47.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:03:47.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:03:47.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:03:47.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:03:47.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:03:47.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:03:47.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:03:47.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:03:47.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:03:47.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:03:47.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:03:47.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:03:47.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:03:47.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:03:47.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:03:47.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=323] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:03:47.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=323] Local address: 172.21.15.161, port 54364 2022-07-03T14:03:47.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=323] Peer address: 172.21.15.101, port 22 2022-07-03T14:03:47.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=323] Connection lost 2022-07-03T14:03:47.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=323] Aborting connection 2022-07-03T14:03:47.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:47 smithi188 ceph-mon[92425]: pgmap v6141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:49 smithi161 ceph-mon[91579]: pgmap v6142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:49 smithi188 ceph-mon[92425]: pgmap v6142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:51 smithi161 ceph-mon[91579]: pgmap v6143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:51 smithi188 ceph-mon[92425]: pgmap v6143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:53 smithi161 ceph-mon[91579]: pgmap v6144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:53 smithi188 ceph-mon[92425]: pgmap v6144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:55 smithi161 ceph-mon[91579]: pgmap v6145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:55 smithi188 ceph-mon[92425]: pgmap v6145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:57 smithi161 ceph-mon[91579]: pgmap v6146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:57 smithi188 ceph-mon[92425]: pgmap v6146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:03:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:03:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:03:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:03:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:03:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:03:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:03:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:03:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:03:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:03:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:03:59 smithi161 ceph-mon[91579]: pgmap v6147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:03:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:03:59 smithi188 ceph-mon[92425]: pgmap v6147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:01 smithi161 ceph-mon[91579]: pgmap v6148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:01 smithi188 ceph-mon[92425]: pgmap v6148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:03 smithi161 ceph-mon[91579]: pgmap v6149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:03 smithi188 ceph-mon[92425]: pgmap v6149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:05 smithi161 ceph-mon[91579]: pgmap v6150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:05 smithi188 ceph-mon[92425]: pgmap v6150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:07 smithi161 ceph-mon[91579]: pgmap v6151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:07 smithi188 ceph-mon[92425]: pgmap v6151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:09 smithi161 ceph-mon[91579]: pgmap v6152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:09 smithi188 ceph-mon[92425]: pgmap v6152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:11 smithi161 ceph-mon[91579]: pgmap v6153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:11 smithi188 ceph-mon[92425]: pgmap v6153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:13 smithi161 ceph-mon[91579]: pgmap v6154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:13 smithi188 ceph-mon[92425]: pgmap v6154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:15 smithi161 ceph-mon[91579]: pgmap v6155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:15 smithi188 ceph-mon[92425]: pgmap v6155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:17 smithi161 ceph-mon[91579]: pgmap v6156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:17 smithi188 ceph-mon[92425]: pgmap v6156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:19 smithi161 ceph-mon[91579]: pgmap v6157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:19 smithi188 ceph-mon[92425]: pgmap v6157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:21 smithi161 ceph-mon[91579]: pgmap v6158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:21 smithi188 ceph-mon[92425]: pgmap v6158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:23 smithi161 ceph-mon[91579]: pgmap v6159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:23 smithi188 ceph-mon[92425]: pgmap v6159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:25 smithi161 ceph-mon[91579]: pgmap v6160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:25 smithi188 ceph-mon[92425]: pgmap v6160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:27 smithi161 ceph-mon[91579]: pgmap v6161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:27 smithi188 ceph-mon[92425]: pgmap v6161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:29 smithi161 ceph-mon[91579]: pgmap v6162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:29 smithi188 ceph-mon[92425]: pgmap v6162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:31 smithi161 ceph-mon[91579]: pgmap v6163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:31 smithi188 ceph-mon[92425]: pgmap v6163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:33 smithi161 ceph-mon[91579]: pgmap v6164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:33 smithi188 ceph-mon[92425]: pgmap v6164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:35 smithi161 ceph-mon[91579]: pgmap v6165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:35 smithi188 ceph-mon[92425]: pgmap v6165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:37 smithi161 ceph-mon[91579]: pgmap v6166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:37 smithi188 ceph-mon[92425]: pgmap v6166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:38.791 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:39 smithi161 ceph-mon[91579]: pgmap v6167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:39 smithi188 ceph-mon[92425]: pgmap v6167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:41.834 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:41 smithi188 ceph-mon[92425]: pgmap v6168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:41 smithi161 ceph-mon[91579]: pgmap v6168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:43 smithi161 ceph-mon[91579]: pgmap v6169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:43 smithi188 ceph-mon[92425]: pgmap v6169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:45 smithi161 ceph-mon[91579]: pgmap v6170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:45 smithi188 ceph-mon[92425]: pgmap v6170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:04:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:04:46.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:04:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:04:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:04:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:04:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:47 smithi161 ceph-mon[91579]: pgmap v6171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:47 smithi188 ceph-mon[92425]: pgmap v6171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:49 smithi161 ceph-mon[91579]: pgmap v6172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:49 smithi188 ceph-mon[92425]: pgmap v6172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:51 smithi161 ceph-mon[91579]: pgmap v6173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:51 smithi188 ceph-mon[92425]: pgmap v6173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:53 smithi161 ceph-mon[91579]: pgmap v6174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:53 smithi188 ceph-mon[92425]: pgmap v6174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:55 smithi161 ceph-mon[91579]: pgmap v6175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:55 smithi188 ceph-mon[92425]: pgmap v6175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:57 smithi161 ceph-mon[91579]: pgmap v6176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:57 smithi188 ceph-mon[92425]: pgmap v6176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:04:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:04:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:04:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:04:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:04:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:04:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:04:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:04:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:04:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:04:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:04:59 smithi161 ceph-mon[91579]: pgmap v6177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:04:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:04:59 smithi188 ceph-mon[92425]: pgmap v6177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:01 smithi161 ceph-mon[91579]: pgmap v6178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:01 smithi188 ceph-mon[92425]: pgmap v6178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:03 smithi161 ceph-mon[91579]: pgmap v6179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:03 smithi188 ceph-mon[92425]: pgmap v6179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:05 smithi161 ceph-mon[91579]: pgmap v6180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:05 smithi188 ceph-mon[92425]: pgmap v6180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:07 smithi161 ceph-mon[91579]: pgmap v6181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:07 smithi188 ceph-mon[92425]: pgmap v6181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:09 smithi161 ceph-mon[91579]: pgmap v6182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:09 smithi188 ceph-mon[92425]: pgmap v6182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:11 smithi161 ceph-mon[91579]: pgmap v6183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:11 smithi188 ceph-mon[92425]: pgmap v6183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:13 smithi161 ceph-mon[91579]: pgmap v6184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:13 smithi188 ceph-mon[92425]: pgmap v6184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:15 smithi161 ceph-mon[91579]: pgmap v6185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:15 smithi188 ceph-mon[92425]: pgmap v6185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:16.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:05:16.281+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:05:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:05:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:05:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:05:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:05:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:05:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:05:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:05:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:05:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:05:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:05:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:05:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:05:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:05:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:05:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:05:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:05:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:05:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:05:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:05:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:05:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:05:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:05:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:05:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:05:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:05:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:05:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:05:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:05:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:05:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:05:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:05:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:05:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:05:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:05:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:05:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:05:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:05:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:05:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:05:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:05:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:05:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:05:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:05:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:05:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=325] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:05:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=325] Local address: 172.21.15.161, port 54368 2022-07-03T14:05:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=325] Peer address: 172.21.15.101, port 22 2022-07-03T14:05:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=325] Connection lost 2022-07-03T14:05:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=325] Aborting connection 2022-07-03T14:05:16.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:05:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:05:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:05:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:05:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:05:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:05:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:05:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:05:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:05:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:05:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:05:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:05:17.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:05:17.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:05:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:05:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:05:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:05:17.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:05:17.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:05:17.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:05:17.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:05:17.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:05:17.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:05:17.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:05:17.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:05:17.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:05:17.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:05:17.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:05:17.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:05:17.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:05:17.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:05:17.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:05:17.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:05:17.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:05:17.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:05:17.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:05:17.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:05:17.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:05:17.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:05:17.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:05:17.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:05:17.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:05:17.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:05:17.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:05:17.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:05:17.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:05:17.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:05:17.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:05:17.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=325] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:05:17.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=325] Local address: 172.21.15.161, port 54368 2022-07-03T14:05:17.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=325] Peer address: 172.21.15.101, port 22 2022-07-03T14:05:17.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=325] Connection lost 2022-07-03T14:05:17.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=325] Aborting connection 2022-07-03T14:05:17.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:17 smithi161 ceph-mon[91579]: pgmap v6186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:05:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:05:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:05:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:05:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:05:17.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:05:17.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:05:17.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:05:17.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:05:17.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:05:17.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:05:17.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:05:17.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:05:17.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:05:17.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:05:17.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:05:17.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:05:17.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:05:17.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:05:17.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:05:17.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:05:17.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:05:17.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:05:17.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:05:17.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:05:17.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:05:17.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:05:17.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:05:17.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:05:17.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:05:17.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:05:17.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:05:17.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:05:17.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:05:17.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:05:17.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:05:17.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:05:17.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:05:17.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:05:17.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:05:17.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:05:17.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:05:17.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:05:17.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:05:17.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:05:17.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:05:17.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=325] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:05:17.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=325] Local address: 172.21.15.161, port 54368 2022-07-03T14:05:17.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=325] Peer address: 172.21.15.101, port 22 2022-07-03T14:05:17.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=325] Connection lost 2022-07-03T14:05:17.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=325] Aborting connection 2022-07-03T14:05:17.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:17 smithi188 ceph-mon[92425]: pgmap v6186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:19 smithi161 ceph-mon[91579]: pgmap v6187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:19 smithi188 ceph-mon[92425]: pgmap v6187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:21 smithi161 ceph-mon[91579]: pgmap v6188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:21 smithi188 ceph-mon[92425]: pgmap v6188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:23 smithi161 ceph-mon[91579]: pgmap v6189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:23 smithi188 ceph-mon[92425]: pgmap v6189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:25 smithi161 ceph-mon[91579]: pgmap v6190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:25 smithi188 ceph-mon[92425]: pgmap v6190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:27 smithi161 ceph-mon[91579]: pgmap v6191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:27 smithi188 ceph-mon[92425]: pgmap v6191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:29 smithi161 ceph-mon[91579]: pgmap v6192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:29 smithi188 ceph-mon[92425]: pgmap v6192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:31 smithi161 ceph-mon[91579]: pgmap v6193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:31 smithi188 ceph-mon[92425]: pgmap v6193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:33 smithi161 ceph-mon[91579]: pgmap v6194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:33 smithi188 ceph-mon[92425]: pgmap v6194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:35 smithi161 ceph-mon[91579]: pgmap v6195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:35 smithi188 ceph-mon[92425]: pgmap v6195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:37 smithi161 ceph-mon[91579]: pgmap v6196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:37 smithi188 ceph-mon[92425]: pgmap v6196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:39 smithi161 ceph-mon[91579]: pgmap v6197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:39 smithi188 ceph-mon[92425]: pgmap v6197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:41 smithi161 ceph-mon[91579]: pgmap v6198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:41 smithi188 ceph-mon[92425]: pgmap v6198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:43 smithi161 ceph-mon[91579]: pgmap v6199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:43 smithi188 ceph-mon[92425]: pgmap v6199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:45 smithi161 ceph-mon[91579]: pgmap v6200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:45 smithi188 ceph-mon[92425]: pgmap v6200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:47 smithi161 ceph-mon[91579]: pgmap v6201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:47 smithi188 ceph-mon[92425]: pgmap v6201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:49 smithi161 ceph-mon[91579]: pgmap v6202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:49 smithi188 ceph-mon[92425]: pgmap v6202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:51 smithi161 ceph-mon[91579]: pgmap v6203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:51 smithi188 ceph-mon[92425]: pgmap v6203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:53 smithi161 ceph-mon[91579]: pgmap v6204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:53 smithi188 ceph-mon[92425]: pgmap v6204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:55 smithi161 ceph-mon[91579]: pgmap v6205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:55 smithi188 ceph-mon[92425]: pgmap v6205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:57 smithi161 ceph-mon[91579]: pgmap v6206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:57 smithi188 ceph-mon[92425]: pgmap v6206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:05:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:05:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:05:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:05:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:05:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:05:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:05:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:05:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:05:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:05:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:05:59 smithi161 ceph-mon[91579]: pgmap v6207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:05:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:05:59 smithi188 ceph-mon[92425]: pgmap v6207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:01 smithi161 ceph-mon[91579]: pgmap v6208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:01 smithi188 ceph-mon[92425]: pgmap v6208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:03 smithi161 ceph-mon[91579]: pgmap v6209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:03 smithi188 ceph-mon[92425]: pgmap v6209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:05 smithi161 ceph-mon[91579]: pgmap v6210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:05 smithi188 ceph-mon[92425]: pgmap v6210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:07 smithi161 ceph-mon[91579]: pgmap v6211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:07 smithi188 ceph-mon[92425]: pgmap v6211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:09 smithi161 ceph-mon[91579]: pgmap v6212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:09 smithi188 ceph-mon[92425]: pgmap v6212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:11 smithi161 ceph-mon[91579]: pgmap v6213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:11 smithi188 ceph-mon[92425]: pgmap v6213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:13 smithi161 ceph-mon[91579]: pgmap v6214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:13 smithi188 ceph-mon[92425]: pgmap v6214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:15 smithi161 ceph-mon[91579]: pgmap v6215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:15 smithi188 ceph-mon[92425]: pgmap v6215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:16.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:06:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:06:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:06:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:06:16.318+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:06:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:06:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:06:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:06:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:06:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:06:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:06:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:06:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:06:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:06:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:06:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:06:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:06:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:06:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:06:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:06:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:06:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:06:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:06:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:06:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:06:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:06:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:06:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:06:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:06:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:06:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:06:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:06:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:06:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:06:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:06:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:06:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:06:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:06:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:06:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:06:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:06:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:06:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:06:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:06:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:06:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:06:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:06:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:06:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:06:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=327] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:06:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=327] Local address: 172.21.15.161, port 54372 2022-07-03T14:06:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=327] Peer address: 172.21.15.101, port 22 2022-07-03T14:06:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=327] Connection lost 2022-07-03T14:06:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=327] Aborting connection 2022-07-03T14:06:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:06:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:06:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:06:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:06:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:06:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:06:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:06:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:06:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:06:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:06:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:06:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:06:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:06:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:06:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:06:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:06:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:06:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:06:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:06:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:06:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:06:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:06:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:06:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:06:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:06:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:06:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:06:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:06:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:06:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:06:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:06:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:06:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:06:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:06:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:06:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:06:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:06:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:06:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:06:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:06:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:06:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:06:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:06:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:06:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:06:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:06:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:06:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:06:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:06:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:06:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:06:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=327] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:06:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=327] Local address: 172.21.15.161, port 54372 2022-07-03T14:06:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=327] Peer address: 172.21.15.101, port 22 2022-07-03T14:06:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=327] Connection lost 2022-07-03T14:06:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=327] Aborting connection 2022-07-03T14:06:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:17 smithi188 ceph-mon[92425]: pgmap v6216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:06:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:06:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:06:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:06:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:06:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:06:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:06:17.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:06:17.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:06:17.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:06:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:06:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:06:17.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:06:17.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:06:17.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:06:17.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:06:17.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:06:17.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:06:17.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:06:17.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:06:17.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:06:17.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:06:17.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:06:17.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:06:17.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:06:17.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:06:17.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:06:17.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:06:17.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:06:17.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:06:17.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:06:17.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:06:17.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:06:17.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:06:17.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:06:17.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:06:17.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:06:17.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:06:17.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:06:17.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:06:17.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:06:17.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:06:17.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:06:17.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:06:17.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:06:17.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:06:17.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=327] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:06:17.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=327] Local address: 172.21.15.161, port 54372 2022-07-03T14:06:17.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=327] Peer address: 172.21.15.101, port 22 2022-07-03T14:06:17.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=327] Connection lost 2022-07-03T14:06:17.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=327] Aborting connection 2022-07-03T14:06:17.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:17 smithi161 ceph-mon[91579]: pgmap v6216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:19 smithi188 ceph-mon[92425]: pgmap v6217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:19 smithi161 ceph-mon[91579]: pgmap v6217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:21 smithi161 ceph-mon[91579]: pgmap v6218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:21 smithi188 ceph-mon[92425]: pgmap v6218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:23 smithi161 ceph-mon[91579]: pgmap v6219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:23 smithi188 ceph-mon[92425]: pgmap v6219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:25 smithi188 ceph-mon[92425]: pgmap v6220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:25 smithi161 ceph-mon[91579]: pgmap v6220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:27 smithi161 ceph-mon[91579]: pgmap v6221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:27 smithi188 ceph-mon[92425]: pgmap v6221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:29 smithi161 ceph-mon[91579]: pgmap v6222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:29 smithi188 ceph-mon[92425]: pgmap v6222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:31 smithi161 ceph-mon[91579]: pgmap v6223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:31 smithi188 ceph-mon[92425]: pgmap v6223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:33 smithi161 ceph-mon[91579]: pgmap v6224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:33 smithi188 ceph-mon[92425]: pgmap v6224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:35 smithi188 ceph-mon[92425]: pgmap v6225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:35 smithi161 ceph-mon[91579]: pgmap v6225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:37 smithi161 ceph-mon[91579]: pgmap v6226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:37 smithi188 ceph-mon[92425]: pgmap v6226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:39 smithi161 ceph-mon[91579]: pgmap v6227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:39 smithi188 ceph-mon[92425]: pgmap v6227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:41 smithi161 ceph-mon[91579]: pgmap v6228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:41 smithi188 ceph-mon[92425]: pgmap v6228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:43 smithi161 ceph-mon[91579]: pgmap v6229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:43 smithi188 ceph-mon[92425]: pgmap v6229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:45 smithi188 ceph-mon[92425]: pgmap v6230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:45 smithi161 ceph-mon[91579]: pgmap v6230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:47 smithi161 ceph-mon[91579]: pgmap v6231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:47 smithi188 ceph-mon[92425]: pgmap v6231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:49 smithi161 ceph-mon[91579]: pgmap v6232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:49 smithi188 ceph-mon[92425]: pgmap v6232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:51 smithi161 ceph-mon[91579]: pgmap v6233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:51 smithi188 ceph-mon[92425]: pgmap v6233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:53 smithi161 ceph-mon[91579]: pgmap v6234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:53 smithi188 ceph-mon[92425]: pgmap v6234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:55 smithi188 ceph-mon[92425]: pgmap v6235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:55 smithi161 ceph-mon[91579]: pgmap v6235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:57 smithi161 ceph-mon[91579]: pgmap v6236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:57 smithi188 ceph-mon[92425]: pgmap v6236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:06:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:06:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:06:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:06:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:06:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:06:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:06:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:06:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:06:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:06:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:06:59 smithi161 ceph-mon[91579]: pgmap v6237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:06:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:06:59 smithi188 ceph-mon[92425]: pgmap v6237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:01 smithi161 ceph-mon[91579]: pgmap v6238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:01 smithi188 ceph-mon[92425]: pgmap v6238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:03 smithi161 ceph-mon[91579]: pgmap v6239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:03 smithi188 ceph-mon[92425]: pgmap v6239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:05 smithi188 ceph-mon[92425]: pgmap v6240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:05 smithi161 ceph-mon[91579]: pgmap v6240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:07 smithi161 ceph-mon[91579]: pgmap v6241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:07 smithi188 ceph-mon[92425]: pgmap v6241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:09 smithi161 ceph-mon[91579]: pgmap v6242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:09 smithi188 ceph-mon[92425]: pgmap v6242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:11 smithi161 ceph-mon[91579]: pgmap v6243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:11 smithi188 ceph-mon[92425]: pgmap v6243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:13 smithi161 ceph-mon[91579]: pgmap v6244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:13 smithi188 ceph-mon[92425]: pgmap v6244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:15 smithi188 ceph-mon[92425]: pgmap v6245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:15 smithi161 ceph-mon[91579]: pgmap v6245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:17.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:07:16.826+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:07:17.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:07:17.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:07:17.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:07:17.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:07:17.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:07:17.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:07:17.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:07:17.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:07:17.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:07:17.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:07:17.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:07:17.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:07:17.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:07:17.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:07:17.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:07:17.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:07:17.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:07:17.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:07:17.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:07:17.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:07:17.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:07:17.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:07:17.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:07:17.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:07:17.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:07:17.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:07:17.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:07:17.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:07:17.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:07:17.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:07:17.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:07:17.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:07:17.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:07:17.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:07:17.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:07:17.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:07:17.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:07:17.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:07:17.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:07:17.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:07:17.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:07:17.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:07:17.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:07:17.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:07:17.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=329] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:07:17.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=329] Local address: 172.21.15.161, port 54376 2022-07-03T14:07:17.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=329] Peer address: 172.21.15.101, port 22 2022-07-03T14:07:17.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=329] Connection lost 2022-07-03T14:07:17.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=329] Aborting connection 2022-07-03T14:07:17.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:07:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:07:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:17 smithi161 ceph-mon[91579]: pgmap v6246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:07:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:07:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:07:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:17 smithi188 ceph-mon[92425]: pgmap v6246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:07:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:07:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:07:18.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:07:18.841 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:18 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:07:18.841 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:07:18.841 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:07:18.842 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:07:18.842 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:07:18.843 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:07:18.843 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:07:18.843 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:07:18.844 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:07:18.844 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:07:18.844 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:07:18.845 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:07:18.845 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:07:18.845 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:07:18.846 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:07:18.846 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:07:18.846 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:07:18.847 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:07:18.847 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:07:18.847 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:07:18.848 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:07:18.848 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:07:18.848 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:07:18.848 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:07:18.849 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:07:18.849 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:07:18.850 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:07:18.850 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:07:18.850 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:07:18.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:07:18.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:07:18.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:07:18.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:07:18.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:07:18.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:07:18.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:07:18.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:07:18.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:07:18.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:07:18.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:07:18.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:07:18.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:07:18.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:07:18.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:07:18.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:07:18.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=329] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:07:18.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=329] Local address: 172.21.15.161, port 54376 2022-07-03T14:07:18.857 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=329] Peer address: 172.21.15.101, port 22 2022-07-03T14:07:18.857 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=329] Connection lost 2022-07-03T14:07:18.857 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=329] Aborting connection 2022-07-03T14:07:18.858 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:07:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:18 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:07:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:07:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:07:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:07:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:07:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:07:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:07:18.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:07:18.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:07:18.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:07:18.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:07:18.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:07:18.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:07:18.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:07:18.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:07:18.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:07:18.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:07:18.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:07:18.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:07:18.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:07:18.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:07:18.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:07:18.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:07:18.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:07:18.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:07:18.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:07:18.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:07:18.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:07:18.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:07:18.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:07:18.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:07:18.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:07:18.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:07:18.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:07:18.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:07:18.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:07:18.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:07:18.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:07:18.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:07:18.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:07:18.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:07:18.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:07:18.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:07:18.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:07:18.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:07:18.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=329] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:07:18.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=329] Local address: 172.21.15.161, port 54376 2022-07-03T14:07:18.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=329] Peer address: 172.21.15.101, port 22 2022-07-03T14:07:18.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=329] Connection lost 2022-07-03T14:07:18.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=329] Aborting connection 2022-07-03T14:07:18.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:19.840 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:19 smithi161 ceph-mon[91579]: pgmap v6247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:19 smithi188 ceph-mon[92425]: pgmap v6247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:21 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T14:07:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:07:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:07:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:21 smithi161 ceph-mon[91579]: pgmap v6248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:21 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T14:07:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:07:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:07:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:21 smithi188 ceph-mon[92425]: pgmap v6248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:23 smithi161 ceph-mon[91579]: pgmap v6249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:23 smithi188 ceph-mon[92425]: pgmap v6249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:25 smithi188 ceph-mon[92425]: pgmap v6250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:25 smithi161 ceph-mon[91579]: pgmap v6250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:27 smithi161 ceph-mon[91579]: pgmap v6251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:27 smithi188 ceph-mon[92425]: pgmap v6251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:29 smithi161 ceph-mon[91579]: pgmap v6252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:29 smithi188 ceph-mon[92425]: pgmap v6252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:31 smithi161 ceph-mon[91579]: pgmap v6253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:31 smithi188 ceph-mon[92425]: pgmap v6253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:33 smithi161 ceph-mon[91579]: pgmap v6254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:33 smithi188 ceph-mon[92425]: pgmap v6254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:35 smithi188 ceph-mon[92425]: pgmap v6255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:35 smithi161 ceph-mon[91579]: pgmap v6255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:37 smithi161 ceph-mon[91579]: pgmap v6256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:37 smithi188 ceph-mon[92425]: pgmap v6256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:39 smithi161 ceph-mon[91579]: pgmap v6257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:39 smithi188 ceph-mon[92425]: pgmap v6257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:41 smithi161 ceph-mon[91579]: pgmap v6258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:41 smithi188 ceph-mon[92425]: pgmap v6258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:43 smithi161 ceph-mon[91579]: pgmap v6259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:43 smithi188 ceph-mon[92425]: pgmap v6259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:45 smithi188 ceph-mon[92425]: pgmap v6260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:45 smithi161 ceph-mon[91579]: pgmap v6260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:47 smithi161 ceph-mon[91579]: pgmap v6261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:47 smithi188 ceph-mon[92425]: pgmap v6261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:49 smithi161 ceph-mon[91579]: pgmap v6262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:49 smithi188 ceph-mon[92425]: pgmap v6262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:51 smithi161 ceph-mon[91579]: pgmap v6263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:51 smithi188 ceph-mon[92425]: pgmap v6263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:53 smithi161 ceph-mon[91579]: pgmap v6264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:53 smithi188 ceph-mon[92425]: pgmap v6264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:55 smithi188 ceph-mon[92425]: pgmap v6265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:55 smithi161 ceph-mon[91579]: pgmap v6265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:57 smithi161 ceph-mon[91579]: pgmap v6266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:57 smithi188 ceph-mon[92425]: pgmap v6266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:07:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:07:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:07:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:07:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:07:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:07:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:07:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:07:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:07:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:07:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:07:59 smithi161 ceph-mon[91579]: pgmap v6267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:07:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:07:59 smithi188 ceph-mon[92425]: pgmap v6267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:01 smithi161 ceph-mon[91579]: pgmap v6268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:01 smithi188 ceph-mon[92425]: pgmap v6268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:03 smithi161 ceph-mon[91579]: pgmap v6269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:03 smithi188 ceph-mon[92425]: pgmap v6269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:05 smithi188 ceph-mon[92425]: pgmap v6270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:05 smithi161 ceph-mon[91579]: pgmap v6270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:07 smithi161 ceph-mon[91579]: pgmap v6271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:07 smithi188 ceph-mon[92425]: pgmap v6271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:09 smithi161 ceph-mon[91579]: pgmap v6272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:09 smithi188 ceph-mon[92425]: pgmap v6272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:11 smithi161 ceph-mon[91579]: pgmap v6273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:11 smithi188 ceph-mon[92425]: pgmap v6273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:13 smithi161 ceph-mon[91579]: pgmap v6274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:13 smithi188 ceph-mon[92425]: pgmap v6274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:15 smithi188 ceph-mon[92425]: pgmap v6275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:15 smithi161 ceph-mon[91579]: pgmap v6275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:17 smithi161 ceph-mon[91579]: pgmap v6276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:17 smithi188 ceph-mon[92425]: pgmap v6276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:19 smithi161 ceph-mon[91579]: pgmap v6277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:19 smithi188 ceph-mon[92425]: pgmap v6277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:20.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:08:20.094+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:08:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:08:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:08:20.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:08:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:08:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:08:20.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:08:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:08:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:08:20.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:08:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:08:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:08:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:08:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:08:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:08:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:08:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:08:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:08:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:08:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:08:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:08:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:08:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:08:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:08:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:08:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:08:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:08:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:08:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:08:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:08:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:08:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:08:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:08:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:08:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:08:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:08:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:08:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:08:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:08:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:08:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:08:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:08:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:08:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:08:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:08:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=331] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:08:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=331] Local address: 172.21.15.161, port 54380 2022-07-03T14:08:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=331] Peer address: 172.21.15.101, port 22 2022-07-03T14:08:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=331] Connection lost 2022-07-03T14:08:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=331] Aborting connection 2022-07-03T14:08:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:08:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:08:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:08:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:08:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:08:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:08:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:08:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:08:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:08:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:08:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:08:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:08:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:08:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:08:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:08:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:08:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:08:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:08:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:08:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:08:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:08:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:08:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:08:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:08:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:08:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:08:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:08:21.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:08:21.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:08:21.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:08:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:08:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:08:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:08:21.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:08:21.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:08:21.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:08:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:08:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:08:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:08:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:08:21.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:08:21.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:08:21.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:08:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:08:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:08:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:08:21.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:08:21.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:08:21.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:08:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:08:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:08:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:08:21.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:08:21.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=331] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:08:21.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=331] Local address: 172.21.15.161, port 54380 2022-07-03T14:08:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=331] Peer address: 172.21.15.101, port 22 2022-07-03T14:08:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=331] Connection lost 2022-07-03T14:08:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=331] Aborting connection 2022-07-03T14:08:21.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:21 smithi161 ceph-mon[91579]: pgmap v6278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:08:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:08:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:08:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:08:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:08:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:08:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:08:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:08:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:08:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:08:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:08:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:08:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:08:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:08:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:08:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:08:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:08:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:08:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:08:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:08:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:08:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:08:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:08:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:08:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:08:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:08:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:08:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:08:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:08:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:08:21.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:08:21.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:08:21.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:08:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:08:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:08:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:08:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:08:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:08:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:08:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:08:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:08:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:08:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:08:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:08:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:08:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:08:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=331] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:08:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=331] Local address: 172.21.15.161, port 54380 2022-07-03T14:08:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=331] Peer address: 172.21.15.101, port 22 2022-07-03T14:08:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=331] Connection lost 2022-07-03T14:08:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=331] Aborting connection 2022-07-03T14:08:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:21 smithi188 ceph-mon[92425]: pgmap v6278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:23 smithi161 ceph-mon[91579]: pgmap v6279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:23 smithi188 ceph-mon[92425]: pgmap v6279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:25 smithi161 ceph-mon[91579]: pgmap v6280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:25 smithi188 ceph-mon[92425]: pgmap v6280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:27 smithi161 ceph-mon[91579]: pgmap v6281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:27 smithi188 ceph-mon[92425]: pgmap v6281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:29 smithi161 ceph-mon[91579]: pgmap v6282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:29 smithi188 ceph-mon[92425]: pgmap v6282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:31 smithi161 ceph-mon[91579]: pgmap v6283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:31 smithi188 ceph-mon[92425]: pgmap v6283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:33 smithi161 ceph-mon[91579]: pgmap v6284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:33 smithi188 ceph-mon[92425]: pgmap v6284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:35 smithi161 ceph-mon[91579]: pgmap v6285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:35 smithi188 ceph-mon[92425]: pgmap v6285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:37 smithi161 ceph-mon[91579]: pgmap v6286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:37 smithi188 ceph-mon[92425]: pgmap v6286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:39 smithi161 ceph-mon[91579]: pgmap v6287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:39 smithi188 ceph-mon[92425]: pgmap v6287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:41 smithi161 ceph-mon[91579]: pgmap v6288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:41 smithi188 ceph-mon[92425]: pgmap v6288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:43 smithi161 ceph-mon[91579]: pgmap v6289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:43 smithi188 ceph-mon[92425]: pgmap v6289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:45 smithi161 ceph-mon[91579]: pgmap v6290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:45 smithi188 ceph-mon[92425]: pgmap v6290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:47 smithi161 ceph-mon[91579]: pgmap v6291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:47 smithi188 ceph-mon[92425]: pgmap v6291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:49 smithi161 ceph-mon[91579]: pgmap v6292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:49 smithi188 ceph-mon[92425]: pgmap v6292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:51 smithi161 ceph-mon[91579]: pgmap v6293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:51 smithi188 ceph-mon[92425]: pgmap v6293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:53 smithi161 ceph-mon[91579]: pgmap v6294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:53 smithi188 ceph-mon[92425]: pgmap v6294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:55 smithi161 ceph-mon[91579]: pgmap v6295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:55 smithi188 ceph-mon[92425]: pgmap v6295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:57 smithi161 ceph-mon[91579]: pgmap v6296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:57 smithi188 ceph-mon[92425]: pgmap v6296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:08:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:08:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:08:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:08:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:08:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:08:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:08:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:08:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:08:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:08:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:08:59 smithi161 ceph-mon[91579]: pgmap v6297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:08:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:08:59 smithi188 ceph-mon[92425]: pgmap v6297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:01 smithi161 ceph-mon[91579]: pgmap v6298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:01 smithi188 ceph-mon[92425]: pgmap v6298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:03 smithi161 ceph-mon[91579]: pgmap v6299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:03 smithi188 ceph-mon[92425]: pgmap v6299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:05 smithi161 ceph-mon[91579]: pgmap v6300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:05 smithi188 ceph-mon[92425]: pgmap v6300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:07 smithi161 ceph-mon[91579]: pgmap v6301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:07 smithi188 ceph-mon[92425]: pgmap v6301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:09 smithi161 ceph-mon[91579]: pgmap v6302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:09 smithi188 ceph-mon[92425]: pgmap v6302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:11 smithi161 ceph-mon[91579]: pgmap v6303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:11 smithi188 ceph-mon[92425]: pgmap v6303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:13 smithi161 ceph-mon[91579]: pgmap v6304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:13 smithi188 ceph-mon[92425]: pgmap v6304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:15 smithi161 ceph-mon[91579]: pgmap v6305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:15 smithi188 ceph-mon[92425]: pgmap v6305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:17 smithi161 ceph-mon[91579]: pgmap v6306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:17 smithi188 ceph-mon[92425]: pgmap v6306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:19 smithi161 ceph-mon[91579]: pgmap v6307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:19 smithi188 ceph-mon[92425]: pgmap v6307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:20.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:09:20.602+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:09:20.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:09:20.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:09:20.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:09:20.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:09:20.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:09:20.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:09:20.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:09:20.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:09:20.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:09:20.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:09:20.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:09:20.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:09:20.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:09:20.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:09:20.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:09:20.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:09:20.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:09:20.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:09:20.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:09:20.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:09:20.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:09:20.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:09:20.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:09:20.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:09:20.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:09:20.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:09:20.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:09:20.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:09:20.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:09:20.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:09:20.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:09:20.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:09:20.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:09:20.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:09:20.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:09:20.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:09:20.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:09:20.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:09:20.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:09:20.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:09:20.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:09:20.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:09:20.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:09:20.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:09:20.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=333] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:09:20.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=333] Local address: 172.21.15.161, port 54406 2022-07-03T14:09:20.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=333] Peer address: 172.21.15.101, port 22 2022-07-03T14:09:20.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=333] Connection lost 2022-07-03T14:09:20.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=333] Aborting connection 2022-07-03T14:09:20.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:09:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:09:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: pgmap v6308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:09:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:09:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:09:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:09:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:09:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:09:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:09:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:09:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:09:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:09:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:09:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:09:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:09:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:09:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:09:21.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:09:21.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:09:21.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:09:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:09:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:09:21.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:09:21.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:09:21.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:09:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:09:21.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:09:21.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:09:21.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:09:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:09:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:09:21.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:09:21.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:09:21.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:09:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:09:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:09:21.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:09:21.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:09:21.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:09:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:09:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:09:21.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:09:21.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:09:21.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:09:21.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:09:21.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:09:21.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:09:21.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:09:21.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:09:21.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:09:21.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:09:21.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=333] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:09:21.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=333] Local address: 172.21.15.161, port 54406 2022-07-03T14:09:21.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=333] Peer address: 172.21.15.101, port 22 2022-07-03T14:09:21.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=333] Connection lost 2022-07-03T14:09:21.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=333] Aborting connection 2022-07-03T14:09:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: pgmap v6308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:09:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:09:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:09:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:09:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:09:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:09:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:09:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:09:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:09:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:09:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:09:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:09:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:09:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:09:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:09:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:09:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:09:21.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:09:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:09:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:09:21.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:09:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:09:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:09:21.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:09:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:09:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:09:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:09:21.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:09:21.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:09:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:09:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:09:21.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:09:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:09:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:09:21.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:09:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:09:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:09:21.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:09:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:09:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:09:21.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:09:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:09:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:09:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:09:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:09:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:09:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:09:21.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:09:21.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:09:21.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=333] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:09:21.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=333] Local address: 172.21.15.161, port 54406 2022-07-03T14:09:21.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=333] Peer address: 172.21.15.101, port 22 2022-07-03T14:09:21.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=333] Connection lost 2022-07-03T14:09:21.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=333] Aborting connection 2022-07-03T14:09:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:23 smithi161 ceph-mon[91579]: pgmap v6309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:23 smithi188 ceph-mon[92425]: pgmap v6309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:25 smithi161 ceph-mon[91579]: pgmap v6310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:25 smithi188 ceph-mon[92425]: pgmap v6310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:27 smithi161 ceph-mon[91579]: pgmap v6311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:27 smithi188 ceph-mon[92425]: pgmap v6311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:29 smithi161 ceph-mon[91579]: pgmap v6312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:29 smithi188 ceph-mon[92425]: pgmap v6312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:31 smithi161 ceph-mon[91579]: pgmap v6313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:31 smithi188 ceph-mon[92425]: pgmap v6313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:33 smithi161 ceph-mon[91579]: pgmap v6314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:33 smithi188 ceph-mon[92425]: pgmap v6314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:35 smithi161 ceph-mon[91579]: pgmap v6315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:35 smithi188 ceph-mon[92425]: pgmap v6315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:37 smithi161 ceph-mon[91579]: pgmap v6316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:37 smithi188 ceph-mon[92425]: pgmap v6316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:39 smithi161 ceph-mon[91579]: pgmap v6317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:39 smithi188 ceph-mon[92425]: pgmap v6317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:41 smithi161 ceph-mon[91579]: pgmap v6318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:41 smithi188 ceph-mon[92425]: pgmap v6318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:43 smithi161 ceph-mon[91579]: pgmap v6319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:43 smithi188 ceph-mon[92425]: pgmap v6319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:45 smithi161 ceph-mon[91579]: pgmap v6320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:45 smithi188 ceph-mon[92425]: pgmap v6320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:47 smithi161 ceph-mon[91579]: pgmap v6321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:47 smithi188 ceph-mon[92425]: pgmap v6321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:49 smithi161 ceph-mon[91579]: pgmap v6322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:49 smithi188 ceph-mon[92425]: pgmap v6322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:51 smithi161 ceph-mon[91579]: pgmap v6323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:51 smithi188 ceph-mon[92425]: pgmap v6323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:53 smithi161 ceph-mon[91579]: pgmap v6324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:53 smithi188 ceph-mon[92425]: pgmap v6324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:55 smithi161 ceph-mon[91579]: pgmap v6325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:55 smithi188 ceph-mon[92425]: pgmap v6325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:57 smithi161 ceph-mon[91579]: pgmap v6326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:57 smithi188 ceph-mon[92425]: pgmap v6326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:09:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:09:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:09:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:09:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:09:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:09:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:09:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:09:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:09:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:09:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:09:59 smithi161 ceph-mon[91579]: pgmap v6327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:09:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:09:59 smithi188 ceph-mon[92425]: pgmap v6327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:10:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:10:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:01 smithi161 ceph-mon[91579]: pgmap v6328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:01 smithi188 ceph-mon[92425]: pgmap v6328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:03 smithi161 ceph-mon[91579]: pgmap v6329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:03 smithi188 ceph-mon[92425]: pgmap v6329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:05 smithi161 ceph-mon[91579]: pgmap v6330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:05 smithi188 ceph-mon[92425]: pgmap v6330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:07 smithi161 ceph-mon[91579]: pgmap v6331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:07 smithi188 ceph-mon[92425]: pgmap v6331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:09 smithi161 ceph-mon[91579]: pgmap v6332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:09 smithi188 ceph-mon[92425]: pgmap v6332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:11 smithi161 ceph-mon[91579]: pgmap v6333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:11 smithi188 ceph-mon[92425]: pgmap v6333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:13 smithi161 ceph-mon[91579]: pgmap v6334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:13 smithi188 ceph-mon[92425]: pgmap v6334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:15 smithi161 ceph-mon[91579]: pgmap v6335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:15 smithi188 ceph-mon[92425]: pgmap v6335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:17 smithi161 ceph-mon[91579]: pgmap v6336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:17 smithi188 ceph-mon[92425]: pgmap v6336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:19 smithi161 ceph-mon[91579]: pgmap v6337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:19 smithi188 ceph-mon[92425]: pgmap v6337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:21.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:10:21.099+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:10:21.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:10:21.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:10:21.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:10:21.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:10:21.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:10:21.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:10:21.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:10:21.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:10:21.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:10:21.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:10:21.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:10:21.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:10:21.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:10:21.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:10:21.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:10:21.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:10:21.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:10:21.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:10:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:10:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:10:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:10:21.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:10:21.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:10:21.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:10:21.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:10:21.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:10:21.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:10:21.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:10:21.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:10:21.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:10:21.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:10:21.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:10:21.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:10:21.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:10:21.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:10:21.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:10:21.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:10:21.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:10:21.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:10:21.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:10:21.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:10:21.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:10:21.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:10:21.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:10:21.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=335] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:10:21.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=335] Local address: 172.21.15.161, port 54410 2022-07-03T14:10:21.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=335] Peer address: 172.21.15.101, port 22 2022-07-03T14:10:21.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=335] Connection lost 2022-07-03T14:10:21.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=335] Aborting connection 2022-07-03T14:10:21.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:10:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: pgmap v6338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:10:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:10:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:10:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: pgmap v6338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:10:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:10:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:10:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:10:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:22 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:10:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:22 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:10:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:10:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:10:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:10:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:10:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:10:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:10:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:10:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:10:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:10:22.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:10:22.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:10:22.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:10:22.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:10:22.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:10:22.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:10:22.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:10:22.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:10:22.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:10:22.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:10:22.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:10:22.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:10:22.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:10:22.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:10:22.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:10:22.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:10:22.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:10:22.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:10:22.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:10:22.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:10:22.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:10:22.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:10:22.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:10:22.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:10:22.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:10:22.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:10:22.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:10:22.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:10:22.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:10:22.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:10:22.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:10:22.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:10:22.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:10:22.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:10:22.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:10:22.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=335] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:10:22.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=335] Local address: 172.21.15.161, port 54410 2022-07-03T14:10:22.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=335] Peer address: 172.21.15.101, port 22 2022-07-03T14:10:22.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=335] Connection lost 2022-07-03T14:10:22.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=335] Aborting connection 2022-07-03T14:10:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:22 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:10:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:22 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:10:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:10:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:10:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:10:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:10:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:10:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:10:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:10:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:10:22.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:10:22.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:10:22.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:10:22.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:10:22.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:10:22.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:10:22.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:10:22.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:10:22.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:10:22.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:10:22.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:10:22.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:10:22.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:10:22.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:10:22.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:10:22.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:10:22.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:10:22.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:10:22.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:10:22.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:10:22.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:10:22.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:10:22.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:10:22.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:10:22.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:10:22.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:10:22.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:10:22.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:10:22.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:10:22.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:10:22.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:10:22.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:10:22.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:10:22.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:10:22.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:10:22.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:10:22.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=335] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:10:22.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=335] Local address: 172.21.15.161, port 54410 2022-07-03T14:10:22.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=335] Peer address: 172.21.15.101, port 22 2022-07-03T14:10:22.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=335] Connection lost 2022-07-03T14:10:22.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=335] Aborting connection 2022-07-03T14:10:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:23 smithi161 ceph-mon[91579]: pgmap v6339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:23 smithi188 ceph-mon[92425]: pgmap v6339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:25 smithi161 ceph-mon[91579]: pgmap v6340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:25 smithi188 ceph-mon[92425]: pgmap v6340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:27 smithi161 ceph-mon[91579]: pgmap v6341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:27 smithi188 ceph-mon[92425]: pgmap v6341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:29 smithi161 ceph-mon[91579]: pgmap v6342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:29 smithi188 ceph-mon[92425]: pgmap v6342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:31 smithi161 ceph-mon[91579]: pgmap v6343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:31 smithi188 ceph-mon[92425]: pgmap v6343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:33 smithi161 ceph-mon[91579]: pgmap v6344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:33 smithi188 ceph-mon[92425]: pgmap v6344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:35 smithi161 ceph-mon[91579]: pgmap v6345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:35 smithi188 ceph-mon[92425]: pgmap v6345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:37 smithi161 ceph-mon[91579]: pgmap v6346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:37 smithi188 ceph-mon[92425]: pgmap v6346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:39 smithi161 ceph-mon[91579]: pgmap v6347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:39 smithi188 ceph-mon[92425]: pgmap v6347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:41 smithi161 ceph-mon[91579]: pgmap v6348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:41 smithi188 ceph-mon[92425]: pgmap v6348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:43 smithi161 ceph-mon[91579]: pgmap v6349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:43 smithi188 ceph-mon[92425]: pgmap v6349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:45 smithi161 ceph-mon[91579]: pgmap v6350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:45 smithi188 ceph-mon[92425]: pgmap v6350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:47 smithi161 ceph-mon[91579]: pgmap v6351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:47 smithi188 ceph-mon[92425]: pgmap v6351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:49 smithi161 ceph-mon[91579]: pgmap v6352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:49 smithi188 ceph-mon[92425]: pgmap v6352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:51 smithi161 ceph-mon[91579]: pgmap v6353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:51 smithi188 ceph-mon[92425]: pgmap v6353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:53 smithi161 ceph-mon[91579]: pgmap v6354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:53 smithi188 ceph-mon[92425]: pgmap v6354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:55 smithi161 ceph-mon[91579]: pgmap v6355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:55 smithi188 ceph-mon[92425]: pgmap v6355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:57 smithi161 ceph-mon[91579]: pgmap v6356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:57 smithi188 ceph-mon[92425]: pgmap v6356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:10:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:10:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:10:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:10:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:10:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:10:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:10:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:10:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:10:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:10:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:10:59 smithi161 ceph-mon[91579]: pgmap v6357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:10:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:10:59 smithi188 ceph-mon[92425]: pgmap v6357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:01 smithi161 ceph-mon[91579]: pgmap v6358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:01 smithi188 ceph-mon[92425]: pgmap v6358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:03 smithi161 ceph-mon[91579]: pgmap v6359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:03 smithi188 ceph-mon[92425]: pgmap v6359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:05 smithi161 ceph-mon[91579]: pgmap v6360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:05 smithi188 ceph-mon[92425]: pgmap v6360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:07 smithi161 ceph-mon[91579]: pgmap v6361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:07 smithi188 ceph-mon[92425]: pgmap v6361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:09 smithi161 ceph-mon[91579]: pgmap v6362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:09 smithi188 ceph-mon[92425]: pgmap v6362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:11 smithi161 ceph-mon[91579]: pgmap v6363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:11 smithi188 ceph-mon[92425]: pgmap v6363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:13 smithi161 ceph-mon[91579]: pgmap v6364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:13 smithi188 ceph-mon[92425]: pgmap v6364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:15 smithi161 ceph-mon[91579]: pgmap v6365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:15 smithi188 ceph-mon[92425]: pgmap v6365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:17 smithi161 ceph-mon[91579]: pgmap v6366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:17 smithi188 ceph-mon[92425]: pgmap v6366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:19 smithi161 ceph-mon[91579]: pgmap v6367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:19 smithi188 ceph-mon[92425]: pgmap v6367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:21 smithi161 ceph-mon[91579]: pgmap v6368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:21 smithi188 ceph-mon[92425]: pgmap v6368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:11:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:11:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:11:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:11:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:11:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:11:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:23 smithi161 ceph-mon[91579]: pgmap v6369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:23.885 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:23 smithi188 ceph-mon[92425]: pgmap v6369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:23.885 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:25 smithi188 ceph-mon[92425]: pgmap v6370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:11:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:25 smithi161 ceph-mon[91579]: pgmap v6370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:11:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:11:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:26 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T14:11:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:26 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T14:11:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:27 smithi161 ceph-mon[91579]: pgmap v6371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:27 smithi188 ceph-mon[92425]: pgmap v6371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:29 smithi161 ceph-mon[91579]: pgmap v6372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:29 smithi188 ceph-mon[92425]: pgmap v6372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:31 smithi161 ceph-mon[91579]: pgmap v6373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:31 smithi188 ceph-mon[92425]: pgmap v6373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:33 smithi161 ceph-mon[91579]: pgmap v6374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:33 smithi188 ceph-mon[92425]: pgmap v6374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:35 smithi161 ceph-mon[91579]: pgmap v6375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:35 smithi188 ceph-mon[92425]: pgmap v6375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:37 smithi161 ceph-mon[91579]: pgmap v6376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:37 smithi188 ceph-mon[92425]: pgmap v6376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:39 smithi161 ceph-mon[91579]: pgmap v6377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:39 smithi188 ceph-mon[92425]: pgmap v6377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:41 smithi161 ceph-mon[91579]: pgmap v6378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:41 smithi188 ceph-mon[92425]: pgmap v6378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:43 smithi161 ceph-mon[91579]: pgmap v6379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:43 smithi188 ceph-mon[92425]: pgmap v6379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:45 smithi161 ceph-mon[91579]: pgmap v6380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:45 smithi188 ceph-mon[92425]: pgmap v6380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:47 smithi161 ceph-mon[91579]: pgmap v6381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:47 smithi188 ceph-mon[92425]: pgmap v6381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:49 smithi161 ceph-mon[91579]: pgmap v6382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:49 smithi188 ceph-mon[92425]: pgmap v6382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:51 smithi161 ceph-mon[91579]: pgmap v6383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:51 smithi188 ceph-mon[92425]: pgmap v6383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:52.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:11:52.032+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:11:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:11:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:11:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:11:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:11:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:11:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:11:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:11:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:11:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:11:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:11:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:11:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:11:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:11:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:11:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:11:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:11:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:11:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:11:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:11:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:11:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:11:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:11:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:11:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:11:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:11:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:11:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:11:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:11:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:11:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:11:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:11:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:11:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:11:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:11:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:11:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:11:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:11:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:11:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:11:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:11:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:11:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:11:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:11:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:11:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=337] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:11:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=337] Local address: 172.21.15.161, port 54414 2022-07-03T14:11:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=337] Peer address: 172.21.15.101, port 22 2022-07-03T14:11:52.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=337] Connection lost 2022-07-03T14:11:52.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=337] Aborting connection 2022-07-03T14:11:52.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:11:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:11:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:11:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:11:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:11:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:11:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:11:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:11:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:11:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:11:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:11:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:11:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:11:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:11:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:11:53.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:11:53.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:11:53.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:11:53.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:11:53.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:11:53.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:11:53.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:11:53.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:11:53.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:11:53.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:11:53.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:11:53.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:11:53.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:11:53.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:11:53.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:11:53.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:11:53.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:11:53.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:11:53.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:11:53.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:11:53.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:11:53.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:11:53.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:11:53.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:11:53.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:11:53.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:11:53.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:11:53.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:11:53.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:11:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:11:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:11:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:11:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:11:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=337] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:11:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=337] Local address: 172.21.15.161, port 54414 2022-07-03T14:11:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=337] Peer address: 172.21.15.101, port 22 2022-07-03T14:11:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=337] Connection lost 2022-07-03T14:11:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=337] Aborting connection 2022-07-03T14:11:53.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:53 smithi161 ceph-mon[91579]: pgmap v6384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:53.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:11:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:11:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:11:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:11:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:11:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:11:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:11:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:11:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:11:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:11:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:11:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:11:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:11:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:11:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:11:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:11:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:11:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:11:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:11:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:11:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:11:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:11:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:11:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:11:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:11:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:11:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:11:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:11:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:11:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:11:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:11:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:11:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:11:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:11:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:11:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:11:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:11:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:11:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:11:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:11:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:11:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:11:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:11:53.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:11:53.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:11:53.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:11:53.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=337] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:11:53.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=337] Local address: 172.21.15.161, port 54414 2022-07-03T14:11:53.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=337] Peer address: 172.21.15.101, port 22 2022-07-03T14:11:53.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=337] Connection lost 2022-07-03T14:11:53.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=337] Aborting connection 2022-07-03T14:11:53.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:53 smithi188 ceph-mon[92425]: pgmap v6384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:53.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:55 smithi161 ceph-mon[91579]: pgmap v6385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:55 smithi188 ceph-mon[92425]: pgmap v6385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:57 smithi161 ceph-mon[91579]: pgmap v6386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:57 smithi188 ceph-mon[92425]: pgmap v6386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:11:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:11:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:11:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:11:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:11:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:11:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:11:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:11:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:11:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:11:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:11:59 smithi161 ceph-mon[91579]: pgmap v6387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:11:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:11:59 smithi188 ceph-mon[92425]: pgmap v6387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:01 smithi161 ceph-mon[91579]: pgmap v6388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:01 smithi188 ceph-mon[92425]: pgmap v6388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:03 smithi161 ceph-mon[91579]: pgmap v6389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:03 smithi188 ceph-mon[92425]: pgmap v6389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:05 smithi161 ceph-mon[91579]: pgmap v6390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:05 smithi188 ceph-mon[92425]: pgmap v6390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:07 smithi161 ceph-mon[91579]: pgmap v6391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:07 smithi188 ceph-mon[92425]: pgmap v6391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:09 smithi161 ceph-mon[91579]: pgmap v6392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:09 smithi188 ceph-mon[92425]: pgmap v6392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:11 smithi161 ceph-mon[91579]: pgmap v6393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:11 smithi188 ceph-mon[92425]: pgmap v6393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:13 smithi161 ceph-mon[91579]: pgmap v6394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:13 smithi188 ceph-mon[92425]: pgmap v6394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:12:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:15 smithi161 ceph-mon[91579]: pgmap v6395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:15 smithi188 ceph-mon[92425]: pgmap v6395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:17 smithi161 ceph-mon[91579]: pgmap v6396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:17 smithi188 ceph-mon[92425]: pgmap v6396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:19.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:19 smithi161 ceph-mon[91579]: pgmap v6397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:19 smithi188 ceph-mon[92425]: pgmap v6397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:21 smithi161 ceph-mon[91579]: pgmap v6398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:21 smithi188 ceph-mon[92425]: pgmap v6398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:23 smithi161 ceph-mon[91579]: pgmap v6399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:23 smithi188 ceph-mon[92425]: pgmap v6399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:25 smithi161 ceph-mon[91579]: pgmap v6400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:25 smithi188 ceph-mon[92425]: pgmap v6400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:27 smithi161 ceph-mon[91579]: pgmap v6401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:27 smithi188 ceph-mon[92425]: pgmap v6401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:29 smithi161 ceph-mon[91579]: pgmap v6402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:29 smithi188 ceph-mon[92425]: pgmap v6402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:31 smithi161 ceph-mon[91579]: pgmap v6403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:31 smithi188 ceph-mon[92425]: pgmap v6403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:33 smithi161 ceph-mon[91579]: pgmap v6404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:33 smithi188 ceph-mon[92425]: pgmap v6404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:35 smithi161 ceph-mon[91579]: pgmap v6405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:35 smithi188 ceph-mon[92425]: pgmap v6405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:37 smithi161 ceph-mon[91579]: pgmap v6406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:37 smithi188 ceph-mon[92425]: pgmap v6406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:39 smithi188 ceph-mon[92425]: pgmap v6407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:39 smithi161 ceph-mon[91579]: pgmap v6407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:41 smithi188 ceph-mon[92425]: pgmap v6408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:41 smithi161 ceph-mon[91579]: pgmap v6408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:43 smithi188 ceph-mon[92425]: pgmap v6409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:43 smithi161 ceph-mon[91579]: pgmap v6409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:44.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:45 smithi161 ceph-mon[91579]: pgmap v6410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:45 smithi188 ceph-mon[92425]: pgmap v6410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:47 smithi161 ceph-mon[91579]: pgmap v6411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:47 smithi188 ceph-mon[92425]: pgmap v6411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:49 smithi188 ceph-mon[92425]: pgmap v6412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:49 smithi161 ceph-mon[91579]: pgmap v6412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:51 smithi188 ceph-mon[92425]: pgmap v6413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:52.070 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:51 smithi161 ceph-mon[91579]: pgmap v6413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:52.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:12:52.069+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:12:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:12:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:12:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:12:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:12:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:12:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:12:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:12:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:12:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:12:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:12:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:12:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:12:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:12:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:12:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:12:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:12:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:12:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:12:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:12:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:12:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:12:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:12:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:12:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:12:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:12:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:12:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:12:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:12:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:12:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:12:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:12:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:12:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:12:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:12:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:12:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:12:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:12:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:12:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:12:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:12:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:12:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:12:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:12:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:12:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=339] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:12:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=339] Local address: 172.21.15.161, port 54418 2022-07-03T14:12:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=339] Peer address: 172.21.15.101, port 22 2022-07-03T14:12:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=339] Connection lost 2022-07-03T14:12:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=339] Aborting connection 2022-07-03T14:12:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:12:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:12:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:12:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:12:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:12:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:12:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:12:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:12:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:12:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:12:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:12:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:12:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:12:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:12:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:12:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:12:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:12:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:12:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:12:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:12:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:12:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:12:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:12:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:12:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:12:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:12:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:12:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:12:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:12:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:12:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:12:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:12:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:12:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:12:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:12:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:12:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:12:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:12:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:12:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:12:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:12:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:12:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:12:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:12:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:12:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:12:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:12:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:12:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:12:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:12:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:12:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:12:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:12:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=339] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:12:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=339] Local address: 172.21.15.161, port 54418 2022-07-03T14:12:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=339] Peer address: 172.21.15.101, port 22 2022-07-03T14:12:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=339] Connection lost 2022-07-03T14:12:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=339] Aborting connection 2022-07-03T14:12:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:53 smithi188 ceph-mon[92425]: pgmap v6414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:12:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:12:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:12:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:12:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:12:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:12:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:12:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:12:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:12:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:12:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:12:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:12:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:12:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:12:54.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:12:54.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:12:54.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:12:54.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:12:54.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:12:54.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:12:54.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:12:54.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:12:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:12:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:12:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:12:54.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:12:54.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:12:54.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:12:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:12:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:12:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:12:54.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:12:54.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:12:54.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:12:54.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:12:54.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:12:54.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:12:54.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:12:54.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:12:54.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:12:54.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:12:54.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:12:54.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:12:54.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=339] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:12:54.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=339] Local address: 172.21.15.161, port 54418 2022-07-03T14:12:54.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=339] Peer address: 172.21.15.101, port 22 2022-07-03T14:12:54.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=339] Connection lost 2022-07-03T14:12:54.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=339] Aborting connection 2022-07-03T14:12:54.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:53 smithi161 ceph-mon[91579]: pgmap v6414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:54.152 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:55 smithi188 ceph-mon[92425]: pgmap v6415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:55 smithi161 ceph-mon[91579]: pgmap v6415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:12:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:12:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:12:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:12:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:56 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:12:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:12:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:12:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:12:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:12:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:12:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:56 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:12:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:57 smithi161 ceph-mon[91579]: pgmap v6416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:12:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:12:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:12:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:12:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:57 smithi188 ceph-mon[92425]: pgmap v6416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:12:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:12:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:12:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:12:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:12:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:12:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:12:59 smithi161 ceph-mon[91579]: pgmap v6417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:12:59 smithi188 ceph-mon[92425]: pgmap v6417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:01 smithi161 ceph-mon[91579]: pgmap v6418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:01 smithi188 ceph-mon[92425]: pgmap v6418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:03 smithi161 ceph-mon[91579]: pgmap v6419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:03 smithi188 ceph-mon[92425]: pgmap v6419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:05 smithi161 ceph-mon[91579]: pgmap v6420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:05 smithi188 ceph-mon[92425]: pgmap v6420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:07 smithi161 ceph-mon[91579]: pgmap v6421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:07 smithi188 ceph-mon[92425]: pgmap v6421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:09 smithi188 ceph-mon[92425]: pgmap v6422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:09 smithi161 ceph-mon[91579]: pgmap v6422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:11 smithi188 ceph-mon[92425]: pgmap v6423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:11 smithi161 ceph-mon[91579]: pgmap v6423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:13 smithi188 ceph-mon[92425]: pgmap v6424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:13 smithi161 ceph-mon[91579]: pgmap v6424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:15 smithi161 ceph-mon[91579]: pgmap v6425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:15 smithi188 ceph-mon[92425]: pgmap v6425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:17 smithi188 ceph-mon[92425]: pgmap v6426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:17 smithi161 ceph-mon[91579]: pgmap v6426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:19 smithi188 ceph-mon[92425]: pgmap v6427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:19 smithi161 ceph-mon[91579]: pgmap v6427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:21 smithi188 ceph-mon[92425]: pgmap v6428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:21 smithi161 ceph-mon[91579]: pgmap v6428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:23 smithi188 ceph-mon[92425]: pgmap v6429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:23 smithi161 ceph-mon[91579]: pgmap v6429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:25 smithi161 ceph-mon[91579]: pgmap v6430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:25 smithi188 ceph-mon[92425]: pgmap v6430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:27 smithi188 ceph-mon[92425]: pgmap v6431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:27 smithi161 ceph-mon[91579]: pgmap v6431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:29 smithi188 ceph-mon[92425]: pgmap v6432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:29 smithi161 ceph-mon[91579]: pgmap v6432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:31 smithi188 ceph-mon[92425]: pgmap v6433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:31 smithi161 ceph-mon[91579]: pgmap v6433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:33 smithi188 ceph-mon[92425]: pgmap v6434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:33 smithi161 ceph-mon[91579]: pgmap v6434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:35 smithi161 ceph-mon[91579]: pgmap v6435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:35 smithi188 ceph-mon[92425]: pgmap v6435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:37 smithi188 ceph-mon[92425]: pgmap v6436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:37 smithi161 ceph-mon[91579]: pgmap v6436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:39 smithi188 ceph-mon[92425]: pgmap v6437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:39 smithi161 ceph-mon[91579]: pgmap v6437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:41 smithi188 ceph-mon[92425]: pgmap v6438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:41 smithi161 ceph-mon[91579]: pgmap v6438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:43 smithi188 ceph-mon[92425]: pgmap v6439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:43 smithi161 ceph-mon[91579]: pgmap v6439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:45 smithi161 ceph-mon[91579]: pgmap v6440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:45 smithi188 ceph-mon[92425]: pgmap v6440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:47 smithi188 ceph-mon[92425]: pgmap v6441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:47 smithi161 ceph-mon[91579]: pgmap v6441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:49 smithi188 ceph-mon[92425]: pgmap v6442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:49 smithi161 ceph-mon[91579]: pgmap v6442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:51 smithi188 ceph-mon[92425]: pgmap v6443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:51 smithi161 ceph-mon[91579]: pgmap v6443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:53 smithi188 ceph-mon[92425]: pgmap v6444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:53 smithi161 ceph-mon[91579]: pgmap v6444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:55.778 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-mon[91579]: pgmap v6445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:55.779 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:13:55.722+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:13:55.779 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:13:55.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:13:55.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:13:55.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:13:55.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:13:55.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:13:55.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:13:55.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:13:55.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:13:55.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:13:55.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:13:55.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:13:55.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:13:55.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:13:55.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:13:55.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:13:55.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:13:55.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:13:55.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:13:55.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:13:55.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:13:55.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:13:55.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:13:55.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:13:55.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:13:55.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:13:55.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:13:55.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:13:55.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:13:55.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:13:55.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:13:55.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:13:55.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:13:55.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:13:55.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:13:55.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:13:55.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:13:55.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:13:55.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:13:55.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:13:55.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:13:55.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:13:55.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:13:55.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:13:55.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=341] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:13:55.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=341] Local address: 172.21.15.161, port 54422 2022-07-03T14:13:55.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=341] Peer address: 172.21.15.101, port 22 2022-07-03T14:13:55.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=341] Connection lost 2022-07-03T14:13:55.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=341] Aborting connection 2022-07-03T14:13:55.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:13:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:13:55.798 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:55 smithi188 ceph-mon[92425]: pgmap v6445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:13:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:13:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:13:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:13:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:13:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:13:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:13:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:13:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:13:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:13:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:13:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:13:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:13:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:13:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:13:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:13:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:13:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:13:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:13:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:13:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:13:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:13:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:13:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:13:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:13:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:13:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:13:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:13:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:13:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:13:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:13:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:13:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:13:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:13:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:13:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:13:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:13:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:13:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:13:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:13:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:13:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:13:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:13:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:13:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:13:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:13:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:13:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:13:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:13:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=341] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:13:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=341] Local address: 172.21.15.161, port 54422 2022-07-03T14:13:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=341] Peer address: 172.21.15.101, port 22 2022-07-03T14:13:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=341] Connection lost 2022-07-03T14:13:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=341] Aborting connection 2022-07-03T14:13:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:13:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:13:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:13:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:13:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:13:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:13:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:13:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:13:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:13:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:13:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:13:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:13:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:13:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:13:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:13:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:13:56.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:13:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:13:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:13:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:13:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:13:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:13:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:13:56.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:13:56.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:13:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:13:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:13:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:13:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:13:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:13:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:13:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:13:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:13:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:13:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:13:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:13:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:13:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:13:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:13:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:13:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:13:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:13:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:13:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:13:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:13:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:13:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:13:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:13:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:13:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=341] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:13:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=341] Local address: 172.21.15.161, port 54422 2022-07-03T14:13:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=341] Peer address: 172.21.15.101, port 22 2022-07-03T14:13:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=341] Connection lost 2022-07-03T14:13:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=341] Aborting connection 2022-07-03T14:13:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:57 smithi161 ceph-mon[91579]: pgmap v6446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:57 smithi188 ceph-mon[92425]: pgmap v6446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:13:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:13:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:13:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:13:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:13:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:13:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:13:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:13:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:13:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:13:59 smithi161 ceph-mon[91579]: pgmap v6447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:13:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:13:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:13:59 smithi188 ceph-mon[92425]: pgmap v6447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:01 smithi188 ceph-mon[92425]: pgmap v6448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:01 smithi161 ceph-mon[91579]: pgmap v6448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:03 smithi188 ceph-mon[92425]: pgmap v6449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:03 smithi161 ceph-mon[91579]: pgmap v6449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:05 smithi161 ceph-mon[91579]: pgmap v6450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:05 smithi188 ceph-mon[92425]: pgmap v6450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:07 smithi188 ceph-mon[92425]: pgmap v6451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:07 smithi161 ceph-mon[91579]: pgmap v6451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:09 smithi188 ceph-mon[92425]: pgmap v6452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:09 smithi161 ceph-mon[91579]: pgmap v6452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:11 smithi188 ceph-mon[92425]: pgmap v6453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:11 smithi161 ceph-mon[91579]: pgmap v6453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:13 smithi188 ceph-mon[92425]: pgmap v6454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:13 smithi161 ceph-mon[91579]: pgmap v6454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:15 smithi161 ceph-mon[91579]: pgmap v6455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:15 smithi188 ceph-mon[92425]: pgmap v6455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:17 smithi188 ceph-mon[92425]: pgmap v6456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:17 smithi161 ceph-mon[91579]: pgmap v6456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:19 smithi188 ceph-mon[92425]: pgmap v6457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:19 smithi161 ceph-mon[91579]: pgmap v6457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:21 smithi161 ceph-mon[91579]: pgmap v6458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:21 smithi188 ceph-mon[92425]: pgmap v6458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:23 smithi161 ceph-mon[91579]: pgmap v6459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:23 smithi188 ceph-mon[92425]: pgmap v6459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:25 smithi161 ceph-mon[91579]: pgmap v6460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:25 smithi188 ceph-mon[92425]: pgmap v6460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:27 smithi188 ceph-mon[92425]: pgmap v6461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:27 smithi161 ceph-mon[91579]: pgmap v6461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:29 smithi161 ceph-mon[91579]: pgmap v6462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:29 smithi188 ceph-mon[92425]: pgmap v6462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:31 smithi161 ceph-mon[91579]: pgmap v6463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:31 smithi188 ceph-mon[92425]: pgmap v6463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:33 smithi161 ceph-mon[91579]: pgmap v6464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:33 smithi188 ceph-mon[92425]: pgmap v6464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:35 smithi161 ceph-mon[91579]: pgmap v6465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:35 smithi188 ceph-mon[92425]: pgmap v6465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:37 smithi188 ceph-mon[92425]: pgmap v6466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:37 smithi161 ceph-mon[91579]: pgmap v6466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:39 smithi188 ceph-mon[92425]: pgmap v6467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:39 smithi161 ceph-mon[91579]: pgmap v6467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:41 smithi161 ceph-mon[91579]: pgmap v6468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:41 smithi188 ceph-mon[92425]: pgmap v6468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:43 smithi161 ceph-mon[91579]: pgmap v6469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:43 smithi188 ceph-mon[92425]: pgmap v6469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:45 smithi161 ceph-mon[91579]: pgmap v6470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:45 smithi188 ceph-mon[92425]: pgmap v6470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:47 smithi188 ceph-mon[92425]: pgmap v6471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:47 smithi161 ceph-mon[91579]: pgmap v6471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:49 smithi161 ceph-mon[91579]: pgmap v6472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:49 smithi188 ceph-mon[92425]: pgmap v6472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:51 smithi161 ceph-mon[91579]: pgmap v6473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:51 smithi188 ceph-mon[92425]: pgmap v6473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:53 smithi161 ceph-mon[91579]: pgmap v6474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:53 smithi188 ceph-mon[92425]: pgmap v6474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:55 smithi161 ceph-mon[91579]: pgmap v6475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:55 smithi188 ceph-mon[92425]: pgmap v6475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:56.505 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:14:56.221+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:14:56.505 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:14:56.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:14:56.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:14:56.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:14:56.507 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:14:56.507 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:14:56.507 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:14:56.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:14:56.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:14:56.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:14:56.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:14:56.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:14:56.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:14:56.510 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:14:56.510 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:14:56.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:14:56.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:14:56.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:14:56.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:14:56.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:14:56.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:14:56.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:14:56.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:14:56.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:14:56.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:14:56.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:14:56.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:14:56.515 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:14:56.515 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:14:56.515 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:14:56.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:14:56.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:14:56.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:14:56.517 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:14:56.517 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:14:56.517 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:14:56.518 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:14:56.518 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:14:56.519 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:14:56.519 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:14:56.519 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:14:56.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:14:56.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:14:56.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:14:56.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=343] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:14:56.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=343] Local address: 172.21.15.161, port 54426 2022-07-03T14:14:56.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=343] Peer address: 172.21.15.101, port 22 2022-07-03T14:14:56.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=343] Connection lost 2022-07-03T14:14:56.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=343] Aborting connection 2022-07-03T14:14:56.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:14:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:14:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:14:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:14:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:14:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:14:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:14:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:14:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:14:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:14:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:14:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:14:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:14:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:14:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:14:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:14:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:14:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:14:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:14:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:14:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:14:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:14:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:14:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:14:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:14:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:14:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:14:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:14:57.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:14:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:14:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:14:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:14:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:14:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:14:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:14:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:14:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:14:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:14:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:14:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:14:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:14:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:14:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:14:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:14:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:14:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:14:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:14:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:14:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:14:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:14:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:14:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:14:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:14:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=343] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:14:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=343] Local address: 172.21.15.161, port 54426 2022-07-03T14:14:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=343] Peer address: 172.21.15.101, port 22 2022-07-03T14:14:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=343] Connection lost 2022-07-03T14:14:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=343] Aborting connection 2022-07-03T14:14:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:57 smithi161 ceph-mon[91579]: pgmap v6476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:14:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:14:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:14:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:14:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:14:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:14:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:14:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:14:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:14:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:14:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:14:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:14:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:14:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:14:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:14:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:14:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:14:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:14:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:14:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:14:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:14:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:14:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:14:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:14:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:14:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:14:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:14:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:14:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:14:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:14:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:14:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:14:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:14:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:14:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:14:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:14:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:14:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:14:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:14:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:14:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:14:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:14:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:14:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:14:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:14:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:14:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=343] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:14:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=343] Local address: 172.21.15.161, port 54426 2022-07-03T14:14:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=343] Peer address: 172.21.15.101, port 22 2022-07-03T14:14:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=343] Connection lost 2022-07-03T14:14:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=343] Aborting connection 2022-07-03T14:14:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:57 smithi188 ceph-mon[92425]: pgmap v6476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:14:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:14:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:14:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:14:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:14:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:14:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:14:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:14:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:14:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:14:59 smithi161 ceph-mon[91579]: pgmap v6477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:14:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:14:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:14:59 smithi188 ceph-mon[92425]: pgmap v6477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:01 smithi161 ceph-mon[91579]: pgmap v6478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:01 smithi188 ceph-mon[92425]: pgmap v6478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:03 smithi161 ceph-mon[91579]: pgmap v6479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:03 smithi188 ceph-mon[92425]: pgmap v6479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:05 smithi161 ceph-mon[91579]: pgmap v6480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:05 smithi188 ceph-mon[92425]: pgmap v6480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:07 smithi161 ceph-mon[91579]: pgmap v6481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:07 smithi188 ceph-mon[92425]: pgmap v6481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:09 smithi161 ceph-mon[91579]: pgmap v6482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:09 smithi188 ceph-mon[92425]: pgmap v6482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:11 smithi161 ceph-mon[91579]: pgmap v6483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:11 smithi188 ceph-mon[92425]: pgmap v6483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:13 smithi161 ceph-mon[91579]: pgmap v6484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:13 smithi188 ceph-mon[92425]: pgmap v6484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:15 smithi161 ceph-mon[91579]: pgmap v6485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:15 smithi188 ceph-mon[92425]: pgmap v6485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:17 smithi161 ceph-mon[91579]: pgmap v6486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:17 smithi188 ceph-mon[92425]: pgmap v6486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:19 smithi161 ceph-mon[91579]: pgmap v6487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:19 smithi188 ceph-mon[92425]: pgmap v6487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:21 smithi161 ceph-mon[91579]: pgmap v6488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:21 smithi188 ceph-mon[92425]: pgmap v6488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:23 smithi161 ceph-mon[91579]: pgmap v6489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:23 smithi188 ceph-mon[92425]: pgmap v6489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:25 smithi161 ceph-mon[91579]: pgmap v6490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:25 smithi188 ceph-mon[92425]: pgmap v6490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:27 smithi161 ceph-mon[91579]: pgmap v6491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:27 smithi188 ceph-mon[92425]: pgmap v6491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:29 smithi161 ceph-mon[91579]: pgmap v6492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:29 smithi188 ceph-mon[92425]: pgmap v6492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:31 smithi161 ceph-mon[91579]: pgmap v6493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:31 smithi188 ceph-mon[92425]: pgmap v6493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:33 smithi161 ceph-mon[91579]: pgmap v6494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:33 smithi188 ceph-mon[92425]: pgmap v6494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:35 smithi161 ceph-mon[91579]: pgmap v6495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:35 smithi188 ceph-mon[92425]: pgmap v6495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:37 smithi161 ceph-mon[91579]: pgmap v6496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:37 smithi188 ceph-mon[92425]: pgmap v6496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:39 smithi161 ceph-mon[91579]: pgmap v6497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:39 smithi188 ceph-mon[92425]: pgmap v6497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:41 smithi161 ceph-mon[91579]: pgmap v6498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:41 smithi188 ceph-mon[92425]: pgmap v6498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:43 smithi161 ceph-mon[91579]: pgmap v6499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:44.183 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:44.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:43 smithi188 ceph-mon[92425]: pgmap v6499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:44.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:45 smithi161 ceph-mon[91579]: pgmap v6500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:45 smithi188 ceph-mon[92425]: pgmap v6500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:47 smithi161 ceph-mon[91579]: pgmap v6501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:47 smithi188 ceph-mon[92425]: pgmap v6501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:49 smithi161 ceph-mon[91579]: pgmap v6502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:49 smithi188 ceph-mon[92425]: pgmap v6502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:51 smithi161 ceph-mon[91579]: pgmap v6503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:51 smithi188 ceph-mon[92425]: pgmap v6503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:53 smithi161 ceph-mon[91579]: pgmap v6504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:53 smithi188 ceph-mon[92425]: pgmap v6504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:55 smithi161 ceph-mon[91579]: pgmap v6505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:55 smithi188 ceph-mon[92425]: pgmap v6505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:15:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:15:57.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:15:56.720+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:15:57.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:15:57.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:15:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:15:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:15:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:15:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:15:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:15:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:15:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:15:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:15:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:15:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:15:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:15:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:15:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:15:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:15:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:15:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:15:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:15:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:15:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:15:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:15:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:15:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:15:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:15:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:15:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:15:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:15:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:15:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:15:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:15:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:15:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:15:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:15:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:15:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:15:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:15:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:15:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:15:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:15:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:15:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:15:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:15:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:15:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=345] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:15:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=345] Local address: 172.21.15.161, port 54444 2022-07-03T14:15:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=345] Peer address: 172.21.15.101, port 22 2022-07-03T14:15:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=345] Connection lost 2022-07-03T14:15:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=345] Aborting connection 2022-07-03T14:15:57.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:15:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:15:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: pgmap v6506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:15:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:15:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:15:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:15:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:15:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:15:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:15:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:15:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:15:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:15:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:15:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:15:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:15:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:15:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:15:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:15:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:15:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:15:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:15:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:15:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:15:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:15:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:15:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:15:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:15:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:15:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:15:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:15:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:15:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:15:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:15:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:15:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:15:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:15:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:15:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:15:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:15:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:15:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:15:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:15:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:15:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:15:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:15:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:15:58.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:15:58.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:15:58.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:15:58.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:15:58.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:15:58.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=345] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:15:58.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=345] Local address: 172.21.15.161, port 54444 2022-07-03T14:15:58.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=345] Peer address: 172.21.15.101, port 22 2022-07-03T14:15:58.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=345] Connection lost 2022-07-03T14:15:58.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=345] Aborting connection 2022-07-03T14:15:58.154 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:15:58.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:15:58.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:15:58.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:15:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: pgmap v6506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:15:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:15:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:15:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:15:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:15:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:15:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:15:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:15:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:15:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:15:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:15:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:15:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:15:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:15:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:15:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:15:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:15:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:15:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:15:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:15:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:15:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:15:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:15:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:15:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:15:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:15:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:15:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:15:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:15:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:15:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:15:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:15:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:15:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:15:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:15:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:15:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:15:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:15:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:15:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:15:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:15:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:15:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:15:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:15:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:15:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:15:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:15:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:15:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=345] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:15:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=345] Local address: 172.21.15.161, port 54444 2022-07-03T14:15:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=345] Peer address: 172.21.15.101, port 22 2022-07-03T14:15:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=345] Connection lost 2022-07-03T14:15:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=345] Aborting connection 2022-07-03T14:15:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:15:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:15:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:15:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:15:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:15:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:15:59 smithi161 ceph-mon[91579]: pgmap v6507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:15:59 smithi188 ceph-mon[92425]: pgmap v6507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:01 smithi161 ceph-mon[91579]: pgmap v6508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:01 smithi188 ceph-mon[92425]: pgmap v6508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:03 smithi161 ceph-mon[91579]: pgmap v6509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:03 smithi188 ceph-mon[92425]: pgmap v6509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:05 smithi161 ceph-mon[91579]: pgmap v6510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:05 smithi188 ceph-mon[92425]: pgmap v6510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:07 smithi161 ceph-mon[91579]: pgmap v6511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:07 smithi188 ceph-mon[92425]: pgmap v6511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:09 smithi161 ceph-mon[91579]: pgmap v6512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:09 smithi188 ceph-mon[92425]: pgmap v6512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:11 smithi161 ceph-mon[91579]: pgmap v6513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:11 smithi188 ceph-mon[92425]: pgmap v6513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:13 smithi161 ceph-mon[91579]: pgmap v6514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:13 smithi188 ceph-mon[92425]: pgmap v6514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:15 smithi161 ceph-mon[91579]: pgmap v6515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:15 smithi188 ceph-mon[92425]: pgmap v6515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:17 smithi161 ceph-mon[91579]: pgmap v6516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:17 smithi188 ceph-mon[92425]: pgmap v6516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:19 smithi161 ceph-mon[91579]: pgmap v6517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:19 smithi188 ceph-mon[92425]: pgmap v6517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:21 smithi161 ceph-mon[91579]: pgmap v6518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:21 smithi188 ceph-mon[92425]: pgmap v6518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:23 smithi161 ceph-mon[91579]: pgmap v6519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:23 smithi188 ceph-mon[92425]: pgmap v6519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:25 smithi161 ceph-mon[91579]: pgmap v6520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:25 smithi188 ceph-mon[92425]: pgmap v6520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:27 smithi161 ceph-mon[91579]: pgmap v6521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:27 smithi188 ceph-mon[92425]: pgmap v6521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:29 smithi161 ceph-mon[91579]: pgmap v6522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:29 smithi188 ceph-mon[92425]: pgmap v6522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:31 smithi161 ceph-mon[91579]: pgmap v6523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:31 smithi188 ceph-mon[92425]: pgmap v6523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:33 smithi161 ceph-mon[91579]: pgmap v6524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:33 smithi188 ceph-mon[92425]: pgmap v6524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:35 smithi161 ceph-mon[91579]: pgmap v6525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:35 smithi188 ceph-mon[92425]: pgmap v6525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:37 smithi161 ceph-mon[91579]: pgmap v6526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:37 smithi188 ceph-mon[92425]: pgmap v6526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:39 smithi161 ceph-mon[91579]: pgmap v6527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:39 smithi188 ceph-mon[92425]: pgmap v6527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:41 smithi161 ceph-mon[91579]: pgmap v6528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:41 smithi188 ceph-mon[92425]: pgmap v6528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:43 smithi161 ceph-mon[91579]: pgmap v6529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:43 smithi188 ceph-mon[92425]: pgmap v6529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:45 smithi161 ceph-mon[91579]: pgmap v6530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:45 smithi188 ceph-mon[92425]: pgmap v6530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:47 smithi161 ceph-mon[91579]: pgmap v6531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:47 smithi188 ceph-mon[92425]: pgmap v6531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:49 smithi161 ceph-mon[91579]: pgmap v6532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:49 smithi188 ceph-mon[92425]: pgmap v6532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:51 smithi161 ceph-mon[91579]: pgmap v6533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:51 smithi188 ceph-mon[92425]: pgmap v6533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:53 smithi161 ceph-mon[91579]: pgmap v6534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:53 smithi188 ceph-mon[92425]: pgmap v6534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:55 smithi161 ceph-mon[91579]: pgmap v6535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:55 smithi188 ceph-mon[92425]: pgmap v6535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:16:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:16:57.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:16:57.222+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:16:57.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:16:57.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:16:57.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:16:57.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:16:57.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:16:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:16:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:16:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:16:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:16:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:16:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:16:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:16:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:16:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:16:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:16:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:16:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:16:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:16:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:16:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:16:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:16:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:16:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:16:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:16:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:16:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:16:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:16:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:16:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:16:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:16:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:16:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:16:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:16:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:16:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:16:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:16:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:16:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:16:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:16:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:16:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:16:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:16:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:16:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:16:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=347] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:16:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=347] Local address: 172.21.15.161, port 54472 2022-07-03T14:16:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=347] Peer address: 172.21.15.101, port 22 2022-07-03T14:16:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=347] Connection lost 2022-07-03T14:16:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=347] Aborting connection 2022-07-03T14:16:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:16:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: pgmap v6536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:16:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:16:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:16:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:16:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:16:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:16:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:16:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: pgmap v6536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:16:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:16:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:16:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:16:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:16:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:16:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:16:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:16:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:16:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:16:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:16:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:16:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:16:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:16:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:16:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:16:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:16:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:16:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:16:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:16:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:16:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:16:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:16:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:16:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:16:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:16:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:16:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:16:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:16:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:16:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:16:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:16:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:16:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:16:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:16:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:16:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:16:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:16:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:16:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:16:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:16:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:16:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:16:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:16:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:16:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:16:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:16:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:16:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:16:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:16:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:16:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:16:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:16:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:16:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=347] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:16:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=347] Local address: 172.21.15.161, port 54472 2022-07-03T14:16:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=347] Peer address: 172.21.15.101, port 22 2022-07-03T14:16:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=347] Connection lost 2022-07-03T14:16:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=347] Aborting connection 2022-07-03T14:16:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:16:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:16:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:16:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:16:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:16:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:16:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:16:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:16:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:16:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:16:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:16:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:16:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:16:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:16:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:16:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:16:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:16:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:16:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:16:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:16:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:16:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:16:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:16:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:16:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:16:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:16:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:16:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:16:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:16:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:16:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:16:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:16:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:16:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:16:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:16:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:16:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:16:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:16:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:16:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:16:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:16:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:16:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:16:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=347] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:16:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=347] Local address: 172.21.15.161, port 54472 2022-07-03T14:16:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=347] Peer address: 172.21.15.101, port 22 2022-07-03T14:16:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=347] Connection lost 2022-07-03T14:16:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=347] Aborting connection 2022-07-03T14:16:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:16:59 smithi161 ceph-mon[91579]: pgmap v6537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:16:59 smithi188 ceph-mon[92425]: pgmap v6537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:01 smithi161 ceph-mon[91579]: pgmap v6538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:01 smithi188 ceph-mon[92425]: pgmap v6538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:03 smithi161 ceph-mon[91579]: pgmap v6539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:03 smithi188 ceph-mon[92425]: pgmap v6539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:05 smithi161 ceph-mon[91579]: pgmap v6540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:05 smithi188 ceph-mon[92425]: pgmap v6540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:07 smithi161 ceph-mon[91579]: pgmap v6541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:07 smithi188 ceph-mon[92425]: pgmap v6541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:09 smithi161 ceph-mon[91579]: pgmap v6542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:09 smithi188 ceph-mon[92425]: pgmap v6542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:11 smithi161 ceph-mon[91579]: pgmap v6543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:11 smithi188 ceph-mon[92425]: pgmap v6543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:13 smithi161 ceph-mon[91579]: pgmap v6544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:13 smithi188 ceph-mon[92425]: pgmap v6544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:15 smithi161 ceph-mon[91579]: pgmap v6545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:15 smithi188 ceph-mon[92425]: pgmap v6545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:17 smithi161 ceph-mon[91579]: pgmap v6546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:17 smithi188 ceph-mon[92425]: pgmap v6546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:19 smithi161 ceph-mon[91579]: pgmap v6547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:19 smithi188 ceph-mon[92425]: pgmap v6547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:21 smithi161 ceph-mon[91579]: pgmap v6548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:21 smithi188 ceph-mon[92425]: pgmap v6548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:23 smithi161 ceph-mon[91579]: pgmap v6549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:23 smithi188 ceph-mon[92425]: pgmap v6549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:25 smithi161 ceph-mon[91579]: pgmap v6550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:25 smithi188 ceph-mon[92425]: pgmap v6550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:27 smithi161 ceph-mon[91579]: pgmap v6551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:27 smithi188 ceph-mon[92425]: pgmap v6551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:29 smithi161 ceph-mon[91579]: pgmap v6552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:29 smithi188 ceph-mon[92425]: pgmap v6552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:31 smithi161 ceph-mon[91579]: pgmap v6553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:31 smithi188 ceph-mon[92425]: pgmap v6553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:33 smithi161 ceph-mon[91579]: pgmap v6554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:33 smithi188 ceph-mon[92425]: pgmap v6554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:35 smithi161 ceph-mon[91579]: pgmap v6555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:35 smithi188 ceph-mon[92425]: pgmap v6555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:37 smithi161 ceph-mon[91579]: pgmap v6556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:37 smithi188 ceph-mon[92425]: pgmap v6556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:39 smithi161 ceph-mon[91579]: pgmap v6557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:39 smithi188 ceph-mon[92425]: pgmap v6557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:41 smithi161 ceph-mon[91579]: pgmap v6558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:41 smithi188 ceph-mon[92425]: pgmap v6558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:43 smithi161 ceph-mon[91579]: pgmap v6559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:43 smithi188 ceph-mon[92425]: pgmap v6559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:45 smithi161 ceph-mon[91579]: pgmap v6560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:45 smithi188 ceph-mon[92425]: pgmap v6560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:47 smithi161 ceph-mon[91579]: pgmap v6561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:47 smithi188 ceph-mon[92425]: pgmap v6561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:49 smithi161 ceph-mon[91579]: pgmap v6562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:49 smithi188 ceph-mon[92425]: pgmap v6562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:51 smithi161 ceph-mon[91579]: pgmap v6563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:51 smithi188 ceph-mon[92425]: pgmap v6563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:53 smithi161 ceph-mon[91579]: pgmap v6564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:53 smithi188 ceph-mon[92425]: pgmap v6564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:55 smithi161 ceph-mon[91579]: pgmap v6565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:55 smithi188 ceph-mon[92425]: pgmap v6565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:17:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:17:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: pgmap v6566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:17:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:17:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:17:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:17:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:17:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:17:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:17:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: pgmap v6566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:17:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:17:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:17:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:17:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:17:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:17:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:17:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:17:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:17:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:17:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:17:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:17:59 smithi161 ceph-mon[91579]: pgmap v6567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:17:59 smithi188 ceph-mon[92425]: pgmap v6567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:01 smithi161 ceph-mon[91579]: pgmap v6568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:01 smithi188 ceph-mon[92425]: pgmap v6568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:03 smithi161 ceph-mon[91579]: pgmap v6569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:03 smithi188 ceph-mon[92425]: pgmap v6569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:05 smithi161 ceph-mon[91579]: pgmap v6570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:05 smithi188 ceph-mon[92425]: pgmap v6570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:07 smithi161 ceph-mon[91579]: pgmap v6571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:07 smithi188 ceph-mon[92425]: pgmap v6571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:09 smithi161 ceph-mon[91579]: pgmap v6572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:09 smithi188 ceph-mon[92425]: pgmap v6572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:11 smithi161 ceph-mon[91579]: pgmap v6573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:11 smithi188 ceph-mon[92425]: pgmap v6573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:13 smithi161 ceph-mon[91579]: pgmap v6574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:13 smithi188 ceph-mon[92425]: pgmap v6574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:15 smithi161 ceph-mon[91579]: pgmap v6575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:15 smithi188 ceph-mon[92425]: pgmap v6575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:17 smithi161 ceph-mon[91579]: pgmap v6576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:17 smithi188 ceph-mon[92425]: pgmap v6576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:19 smithi161 ceph-mon[91579]: pgmap v6577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:19 smithi188 ceph-mon[92425]: pgmap v6577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:21 smithi161 ceph-mon[91579]: pgmap v6578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:21 smithi188 ceph-mon[92425]: pgmap v6578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:23 smithi161 ceph-mon[91579]: pgmap v6579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:23 smithi188 ceph-mon[92425]: pgmap v6579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:25 smithi161 ceph-mon[91579]: pgmap v6580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:25 smithi188 ceph-mon[92425]: pgmap v6580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-mon[91579]: pgmap v6581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:18:27.734+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:18:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:18:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:18:28.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:18:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:18:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:18:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:18:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:18:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:18:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:18:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:18:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:18:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:18:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:18:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:18:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:18:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:18:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:18:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:18:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:18:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:18:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:18:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:18:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:18:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:18:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:18:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:18:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:18:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:18:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:18:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:18:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:18:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:18:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:18:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:18:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:18:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:18:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:18:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:18:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:18:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:18:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:18:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:18:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:18:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:18:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=349] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:18:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=349] Local address: 172.21.15.161, port 54500 2022-07-03T14:18:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=349] Peer address: 172.21.15.101, port 22 2022-07-03T14:18:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=968] Received exit status 0 2022-07-03T14:18:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=968] Received channel close 2022-07-03T14:18:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=968] Channel closed 2022-07-03T14:18:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=968] Received exit status 0 2022-07-03T14:18:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=968] Received channel close 2022-07-03T14:18:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=968] Channel closed 2022-07-03T14:18:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=969] Requesting new SSH session 2022-07-03T14:18:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=969] Requesting new SSH session 2022-07-03T14:18:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=969] Command: which python3 2022-07-03T14:18:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=969] Command: which python3 2022-07-03T14:18:28.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=969] Received exit status 0 2022-07-03T14:18:28.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=969] Received channel close 2022-07-03T14:18:28.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=969] Channel closed 2022-07-03T14:18:28.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=969] Received exit status 0 2022-07-03T14:18:28.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=969] Received channel close 2022-07-03T14:18:28.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=970] Requesting new SSH session 2022-07-03T14:18:28.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=969] Channel closed 2022-07-03T14:18:28.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=970] Requesting new SSH session 2022-07-03T14:18:28.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=970] Command: sudo true 2022-07-03T14:18:28.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=970] Command: sudo true 2022-07-03T14:18:28.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=970] Received exit status 0 2022-07-03T14:18:28.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=970] Received channel close 2022-07-03T14:18:28.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=970] Channel closed 2022-07-03T14:18:28.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=971] Requesting new SSH session 2022-07-03T14:18:28.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=970] Received exit status 0 2022-07-03T14:18:28.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=970] Received channel close 2022-07-03T14:18:28.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=970] Channel closed 2022-07-03T14:18:28.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:28.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=971] Requesting new SSH session 2022-07-03T14:18:28.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:28.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=971] Received exit status 0 2022-07-03T14:18:28.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=971] Received channel close 2022-07-03T14:18:28.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=971] Channel closed 2022-07-03T14:18:28.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=971] Received exit status 0 2022-07-03T14:18:28.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=971] Received channel close 2022-07-03T14:18:28.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=971] Channel closed 2022-07-03T14:18:28.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=349] Login timeout expired 2022-07-03T14:18:28.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=349] Aborting connection 2022-07-03T14:18:28.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:18:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:18:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:27 smithi188 ceph-mon[92425]: pgmap v6581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:18:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:18:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:18:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:18:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:18:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:18:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:18:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:18:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:18:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:18:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:18:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:18:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:18:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:18:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:18:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:18:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:18:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:18:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:18:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:18:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:18:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:18:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:18:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:18:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:18:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:18:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:18:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:18:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:18:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:18:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:18:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:18:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:18:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:18:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:18:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:18:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:18:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:18:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:18:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:18:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:18:29.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:18:29.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:18:29.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:18:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:18:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=349] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:18:29.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=349] Local address: 172.21.15.161, port 54500 2022-07-03T14:18:29.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=349] Peer address: 172.21.15.101, port 22 2022-07-03T14:18:29.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=968] Received exit status 0 2022-07-03T14:18:29.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=968] Received channel close 2022-07-03T14:18:29.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=968] Channel closed 2022-07-03T14:18:29.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=968] Received exit status 0 2022-07-03T14:18:29.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=968] Received channel close 2022-07-03T14:18:29.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=968] Channel closed 2022-07-03T14:18:29.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=969] Requesting new SSH session 2022-07-03T14:18:29.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=969] Requesting new SSH session 2022-07-03T14:18:29.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=969] Command: which python3 2022-07-03T14:18:29.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=969] Command: which python3 2022-07-03T14:18:29.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=969] Received exit status 0 2022-07-03T14:18:29.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=969] Received channel close 2022-07-03T14:18:29.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=969] Channel closed 2022-07-03T14:18:29.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=969] Received exit status 0 2022-07-03T14:18:29.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=969] Received channel close 2022-07-03T14:18:29.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=970] Requesting new SSH session 2022-07-03T14:18:29.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=969] Channel closed 2022-07-03T14:18:29.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=970] Requesting new SSH session 2022-07-03T14:18:29.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=970] Command: sudo true 2022-07-03T14:18:29.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=970] Command: sudo true 2022-07-03T14:18:29.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=970] Received exit status 0 2022-07-03T14:18:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=970] Received channel close 2022-07-03T14:18:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=970] Channel closed 2022-07-03T14:18:29.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=971] Requesting new SSH session 2022-07-03T14:18:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=970] Received exit status 0 2022-07-03T14:18:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=970] Received channel close 2022-07-03T14:18:29.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=970] Channel closed 2022-07-03T14:18:29.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:29.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=971] Requesting new SSH session 2022-07-03T14:18:29.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=971] Received exit status 0 2022-07-03T14:18:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=971] Received channel close 2022-07-03T14:18:29.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=971] Channel closed 2022-07-03T14:18:29.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=971] Received exit status 0 2022-07-03T14:18:29.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=971] Received channel close 2022-07-03T14:18:29.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=971] Channel closed 2022-07-03T14:18:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=349] Login timeout expired 2022-07-03T14:18:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=349] Aborting connection 2022-07-03T14:18:29.165 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:18:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:18:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:18:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:18:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:18:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:18:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:18:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:18:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:18:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:18:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:18:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:18:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:18:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:18:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:18:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:18:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:18:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:18:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:18:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:18:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:18:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:18:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:18:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:18:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:18:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:18:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:18:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:18:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:18:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:18:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:18:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:18:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:18:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:18:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:18:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:18:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:18:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:18:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:18:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:18:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:18:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:18:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:18:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:18:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:18:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=349] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:18:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=349] Local address: 172.21.15.161, port 54500 2022-07-03T14:18:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=349] Peer address: 172.21.15.101, port 22 2022-07-03T14:18:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=968] Received exit status 0 2022-07-03T14:18:29.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=968] Received channel close 2022-07-03T14:18:29.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=968] Channel closed 2022-07-03T14:18:29.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=968] Received exit status 0 2022-07-03T14:18:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=968] Received channel close 2022-07-03T14:18:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=968] Channel closed 2022-07-03T14:18:29.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=969] Requesting new SSH session 2022-07-03T14:18:29.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=969] Requesting new SSH session 2022-07-03T14:18:29.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=969] Command: which python3 2022-07-03T14:18:29.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=969] Command: which python3 2022-07-03T14:18:29.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=969] Received exit status 0 2022-07-03T14:18:29.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=969] Received channel close 2022-07-03T14:18:29.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=969] Channel closed 2022-07-03T14:18:29.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=969] Received exit status 0 2022-07-03T14:18:29.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=969] Received channel close 2022-07-03T14:18:29.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=970] Requesting new SSH session 2022-07-03T14:18:29.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=969] Channel closed 2022-07-03T14:18:29.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=970] Requesting new SSH session 2022-07-03T14:18:29.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=970] Command: sudo true 2022-07-03T14:18:29.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=970] Command: sudo true 2022-07-03T14:18:29.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=970] Received exit status 0 2022-07-03T14:18:29.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=970] Received channel close 2022-07-03T14:18:29.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=970] Channel closed 2022-07-03T14:18:29.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=971] Requesting new SSH session 2022-07-03T14:18:29.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=970] Received exit status 0 2022-07-03T14:18:29.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=970] Received channel close 2022-07-03T14:18:29.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=970] Channel closed 2022-07-03T14:18:29.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=971] Requesting new SSH session 2022-07-03T14:18:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=971] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:18:29.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=971] Received exit status 0 2022-07-03T14:18:29.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=971] Received channel close 2022-07-03T14:18:29.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=971] Channel closed 2022-07-03T14:18:29.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=971] Received exit status 0 2022-07-03T14:18:29.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=971] Received channel close 2022-07-03T14:18:29.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=971] Channel closed 2022-07-03T14:18:29.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=349] Login timeout expired 2022-07-03T14:18:29.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=349] Aborting connection 2022-07-03T14:18:29.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:29 smithi161 ceph-mon[91579]: pgmap v6582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:29 smithi188 ceph-mon[92425]: pgmap v6582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:31 smithi161 ceph-mon[91579]: pgmap v6583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:31 smithi188 ceph-mon[92425]: pgmap v6583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:33 smithi161 ceph-mon[91579]: pgmap v6584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:33 smithi188 ceph-mon[92425]: pgmap v6584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:35 smithi161 ceph-mon[91579]: pgmap v6585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:35 smithi188 ceph-mon[92425]: pgmap v6585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:37 smithi161 ceph-mon[91579]: pgmap v6586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:37 smithi188 ceph-mon[92425]: pgmap v6586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:39 smithi161 ceph-mon[91579]: pgmap v6587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:39 smithi188 ceph-mon[92425]: pgmap v6587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:42.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:41 smithi161 ceph-mon[91579]: pgmap v6588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:41 smithi188 ceph-mon[92425]: pgmap v6588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:43 smithi161 ceph-mon[91579]: pgmap v6589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:43 smithi188 ceph-mon[92425]: pgmap v6589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:45 smithi161 ceph-mon[91579]: pgmap v6590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:45 smithi188 ceph-mon[92425]: pgmap v6590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:47 smithi161 ceph-mon[91579]: pgmap v6591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:47 smithi188 ceph-mon[92425]: pgmap v6591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:49 smithi161 ceph-mon[91579]: pgmap v6592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:49 smithi188 ceph-mon[92425]: pgmap v6592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:51 smithi161 ceph-mon[91579]: pgmap v6593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:51 smithi188 ceph-mon[92425]: pgmap v6593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:53 smithi161 ceph-mon[91579]: pgmap v6594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:53 smithi188 ceph-mon[92425]: pgmap v6594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:18:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:55 smithi161 ceph-mon[91579]: pgmap v6595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:55 smithi188 ceph-mon[92425]: pgmap v6595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:18:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:57 smithi161 ceph-mon[91579]: pgmap v6596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:18:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:18:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:18:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:18:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:57 smithi188 ceph-mon[92425]: pgmap v6596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:18:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:18:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:18:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:18:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:18:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:18:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:18:59 smithi161 ceph-mon[91579]: pgmap v6597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:18:59 smithi188 ceph-mon[92425]: pgmap v6597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:01 smithi161 ceph-mon[91579]: pgmap v6598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:01 smithi188 ceph-mon[92425]: pgmap v6598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:03 smithi161 ceph-mon[91579]: pgmap v6599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:03 smithi188 ceph-mon[92425]: pgmap v6599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:05 smithi161 ceph-mon[91579]: pgmap v6600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:05 smithi188 ceph-mon[92425]: pgmap v6600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:07 smithi161 ceph-mon[91579]: pgmap v6601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:07 smithi188 ceph-mon[92425]: pgmap v6601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:09 smithi161 ceph-mon[91579]: pgmap v6602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:09 smithi188 ceph-mon[92425]: pgmap v6602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:11 smithi161 ceph-mon[91579]: pgmap v6603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:11 smithi188 ceph-mon[92425]: pgmap v6603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:13 smithi161 ceph-mon[91579]: pgmap v6604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:13 smithi188 ceph-mon[92425]: pgmap v6604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:15 smithi161 ceph-mon[91579]: pgmap v6605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:15 smithi188 ceph-mon[92425]: pgmap v6605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:17 smithi161 ceph-mon[91579]: pgmap v6606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:17 smithi188 ceph-mon[92425]: pgmap v6606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:19 smithi161 ceph-mon[91579]: pgmap v6607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:19 smithi188 ceph-mon[92425]: pgmap v6607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:21 smithi161 ceph-mon[91579]: pgmap v6608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:21 smithi188 ceph-mon[92425]: pgmap v6608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:23 smithi161 ceph-mon[91579]: pgmap v6609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:23 smithi188 ceph-mon[92425]: pgmap v6609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:25 smithi161 ceph-mon[91579]: pgmap v6610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:25 smithi188 ceph-mon[92425]: pgmap v6610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-mon[91579]: pgmap v6611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:19:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:19:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:19:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:19:27.769+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:19:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:19:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:19:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:19:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:19:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:19:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:19:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:19:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:19:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:19:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:19:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:19:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:19:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:19:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:19:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:19:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:19:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:19:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:19:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:19:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:19:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:19:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:19:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:19:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:19:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:19:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:19:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:19:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:19:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:19:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:19:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:19:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:19:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:19:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:19:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:19:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:19:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:19:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:19:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:19:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:19:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:19:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:19:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:19:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:19:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=351] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:19:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=351] Local address: 172.21.15.161, port 54540 2022-07-03T14:19:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=351] Peer address: 172.21.15.101, port 22 2022-07-03T14:19:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=351] Connection lost 2022-07-03T14:19:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=351] Aborting connection 2022-07-03T14:19:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:19:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:19:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:27 smithi188 ceph-mon[92425]: pgmap v6611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:19:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:19:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:19:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:19:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:19:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:19:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:19:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:19:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:19:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:19:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:19:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:19:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:19:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:19:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:19:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:19:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:19:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:19:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:19:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:19:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:19:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:19:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:19:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:19:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:19:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:19:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:19:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:19:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:19:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:19:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:19:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:19:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:19:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:19:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:19:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:19:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:19:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:19:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:19:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:19:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:19:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:19:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:19:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:19:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:19:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:19:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:19:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:19:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:19:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=351] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:19:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=351] Local address: 172.21.15.161, port 54540 2022-07-03T14:19:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=351] Peer address: 172.21.15.101, port 22 2022-07-03T14:19:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=351] Connection lost 2022-07-03T14:19:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=351] Aborting connection 2022-07-03T14:19:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:19:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:19:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:19:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:19:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:19:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:19:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:19:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:19:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:19:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:19:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:19:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:19:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:19:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:19:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:19:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:19:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:19:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:19:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:19:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:19:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:19:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:19:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:19:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:19:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:19:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:19:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:19:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:19:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:19:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:19:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:19:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:19:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:19:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:19:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:19:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:19:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:19:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:19:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:19:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:19:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:19:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:19:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:19:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:19:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:19:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:19:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=351] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:19:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=351] Local address: 172.21.15.161, port 54540 2022-07-03T14:19:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=351] Peer address: 172.21.15.101, port 22 2022-07-03T14:19:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=351] Connection lost 2022-07-03T14:19:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=351] Aborting connection 2022-07-03T14:19:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:29 smithi161 ceph-mon[91579]: pgmap v6612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:29 smithi188 ceph-mon[92425]: pgmap v6612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:31 smithi161 ceph-mon[91579]: pgmap v6613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:31 smithi188 ceph-mon[92425]: pgmap v6613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:33 smithi161 ceph-mon[91579]: pgmap v6614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:33 smithi188 ceph-mon[92425]: pgmap v6614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:35 smithi161 ceph-mon[91579]: pgmap v6615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:35 smithi188 ceph-mon[92425]: pgmap v6615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:37 smithi161 ceph-mon[91579]: pgmap v6616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:37 smithi188 ceph-mon[92425]: pgmap v6616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:39 smithi161 ceph-mon[91579]: pgmap v6617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:39 smithi188 ceph-mon[92425]: pgmap v6617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:41 smithi161 ceph-mon[91579]: pgmap v6618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:41 smithi188 ceph-mon[92425]: pgmap v6618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:43 smithi161 ceph-mon[91579]: pgmap v6619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:43 smithi188 ceph-mon[92425]: pgmap v6619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:45 smithi161 ceph-mon[91579]: pgmap v6620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:45 smithi188 ceph-mon[92425]: pgmap v6620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:47 smithi161 ceph-mon[91579]: pgmap v6621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:47 smithi188 ceph-mon[92425]: pgmap v6621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:49 smithi161 ceph-mon[91579]: pgmap v6622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:49 smithi188 ceph-mon[92425]: pgmap v6622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:51 smithi161 ceph-mon[91579]: pgmap v6623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:51 smithi188 ceph-mon[92425]: pgmap v6623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:53 smithi161 ceph-mon[91579]: pgmap v6624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:53 smithi188 ceph-mon[92425]: pgmap v6624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:19:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:55 smithi161 ceph-mon[91579]: pgmap v6625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:55 smithi188 ceph-mon[92425]: pgmap v6625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:19:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:57 smithi161 ceph-mon[91579]: pgmap v6626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:19:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:19:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:19:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:19:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:57 smithi188 ceph-mon[92425]: pgmap v6626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:19:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:19:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:19:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:19:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:19:59 smithi161 ceph-mon[91579]: pgmap v6627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:19:59 smithi188 ceph-mon[92425]: pgmap v6627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:20:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:20:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:01 smithi161 ceph-mon[91579]: pgmap v6628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:01 smithi188 ceph-mon[92425]: pgmap v6628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:03 smithi161 ceph-mon[91579]: pgmap v6629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:03 smithi188 ceph-mon[92425]: pgmap v6629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:05 smithi161 ceph-mon[91579]: pgmap v6630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:05 smithi188 ceph-mon[92425]: pgmap v6630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:07 smithi161 ceph-mon[91579]: pgmap v6631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:07 smithi188 ceph-mon[92425]: pgmap v6631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:09 smithi161 ceph-mon[91579]: pgmap v6632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:09 smithi188 ceph-mon[92425]: pgmap v6632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:11 smithi161 ceph-mon[91579]: pgmap v6633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:11 smithi188 ceph-mon[92425]: pgmap v6633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:13 smithi161 ceph-mon[91579]: pgmap v6634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:13 smithi188 ceph-mon[92425]: pgmap v6634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:15 smithi161 ceph-mon[91579]: pgmap v6635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:15 smithi188 ceph-mon[92425]: pgmap v6635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:17 smithi161 ceph-mon[91579]: pgmap v6636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:17 smithi188 ceph-mon[92425]: pgmap v6636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:19 smithi161 ceph-mon[91579]: pgmap v6637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:19 smithi188 ceph-mon[92425]: pgmap v6637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:21 smithi161 ceph-mon[91579]: pgmap v6638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:21 smithi188 ceph-mon[92425]: pgmap v6638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:23 smithi161 ceph-mon[91579]: pgmap v6639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:23 smithi188 ceph-mon[92425]: pgmap v6639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:25 smithi161 ceph-mon[91579]: pgmap v6640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:25 smithi188 ceph-mon[92425]: pgmap v6640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:27 smithi161 ceph-mon[91579]: pgmap v6641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:27 smithi188 ceph-mon[92425]: pgmap v6641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:20:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:20:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:20:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:20:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:20:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:20:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:20:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:20:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:29 smithi161 ceph-mon[91579]: pgmap v6642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:20:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:29 smithi188 ceph-mon[92425]: pgmap v6642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:31 smithi161 ceph-mon[91579]: pgmap v6643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:31 smithi188 ceph-mon[92425]: pgmap v6643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:33 smithi161 ceph-mon[91579]: pgmap v6644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:33 smithi188 ceph-mon[92425]: pgmap v6644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:35 smithi161 ceph-mon[91579]: pgmap v6645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:35 smithi188 ceph-mon[92425]: pgmap v6645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:37 smithi161 ceph-mon[91579]: pgmap v6646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:37 smithi188 ceph-mon[92425]: pgmap v6646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:39 smithi161 ceph-mon[91579]: pgmap v6647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:39 smithi188 ceph-mon[92425]: pgmap v6647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:41 smithi161 ceph-mon[91579]: pgmap v6648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:41 smithi188 ceph-mon[92425]: pgmap v6648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:43 smithi161 ceph-mon[91579]: pgmap v6649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:43 smithi188 ceph-mon[92425]: pgmap v6649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:45 smithi161 ceph-mon[91579]: pgmap v6650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:45 smithi188 ceph-mon[92425]: pgmap v6650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:47 smithi161 ceph-mon[91579]: pgmap v6651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:47 smithi188 ceph-mon[92425]: pgmap v6651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:49 smithi161 ceph-mon[91579]: pgmap v6652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:49 smithi188 ceph-mon[92425]: pgmap v6652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:51 smithi161 ceph-mon[91579]: pgmap v6653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:51 smithi188 ceph-mon[92425]: pgmap v6653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:53 smithi161 ceph-mon[91579]: pgmap v6654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:53 smithi188 ceph-mon[92425]: pgmap v6654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:20:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:55 smithi161 ceph-mon[91579]: pgmap v6655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:55 smithi188 ceph-mon[92425]: pgmap v6655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:20:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:57 smithi161 ceph-mon[91579]: pgmap v6656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:20:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:20:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:20:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:57 smithi188 ceph-mon[92425]: pgmap v6656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:20:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:20:58.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:20:58.271+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:20:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:20:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:20:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:20:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:20:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:20:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:20:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:20:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:20:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:20:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:20:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:20:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:20:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:20:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:20:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:20:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:20:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:20:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:20:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:20:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:20:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:20:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:20:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:20:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:20:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:20:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:20:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:20:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:20:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:20:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:20:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:20:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:20:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:20:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:20:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:20:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:20:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:20:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:20:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:20:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:20:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:20:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:20:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:20:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:20:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=353] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:20:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=353] Local address: 172.21.15.161, port 54568 2022-07-03T14:20:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=353] Peer address: 172.21.15.101, port 22 2022-07-03T14:20:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=976] Received exit status 0 2022-07-03T14:20:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=976] Received channel close 2022-07-03T14:20:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=976] Channel closed 2022-07-03T14:20:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=977] Requesting new SSH session 2022-07-03T14:20:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=977] Command: which python3 2022-07-03T14:20:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=976] Received exit status 0 2022-07-03T14:20:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=976] Received channel close 2022-07-03T14:20:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=976] Channel closed 2022-07-03T14:20:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=977] Requesting new SSH session 2022-07-03T14:20:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=977] Command: which python3 2022-07-03T14:20:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=977] Received exit status 0 2022-07-03T14:20:58.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=977] Received channel close 2022-07-03T14:20:58.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=977] Channel closed 2022-07-03T14:20:58.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=978] Requesting new SSH session 2022-07-03T14:20:58.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=978] Command: sudo true 2022-07-03T14:20:58.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=977] Received exit status 0 2022-07-03T14:20:58.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=977] Received channel close 2022-07-03T14:20:58.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=977] Channel closed 2022-07-03T14:20:58.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=978] Requesting new SSH session 2022-07-03T14:20:58.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=978] Command: sudo true 2022-07-03T14:20:58.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=978] Received exit status 0 2022-07-03T14:20:58.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=978] Received channel close 2022-07-03T14:20:58.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=978] Channel closed 2022-07-03T14:20:58.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=979] Requesting new SSH session 2022-07-03T14:20:58.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:20:58.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=978] Received exit status 0 2022-07-03T14:20:58.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=978] Received channel close 2022-07-03T14:20:58.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=978] Channel closed 2022-07-03T14:20:58.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=979] Requesting new SSH session 2022-07-03T14:20:58.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:20:58.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=979] Received exit status 0 2022-07-03T14:20:58.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=979] Received channel close 2022-07-03T14:20:58.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=979] Channel closed 2022-07-03T14:20:58.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=979] Received exit status 0 2022-07-03T14:20:58.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=979] Received channel close 2022-07-03T14:20:58.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=979] Channel closed 2022-07-03T14:20:58.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=980] Requesting new SSH session 2022-07-03T14:20:58.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=980] Command: sudo true 2022-07-03T14:20:58.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=980] Requesting new SSH session 2022-07-03T14:20:58.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=980] Command: sudo true 2022-07-03T14:20:58.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=980] Received exit status 0 2022-07-03T14:20:58.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=980] Received channel close 2022-07-03T14:20:58.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=980] Channel closed 2022-07-03T14:20:58.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=981] Requesting new SSH session 2022-07-03T14:20:58.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=981] Command: which python3 2022-07-03T14:20:58.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=980] Received exit status 0 2022-07-03T14:20:58.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=980] Received channel close 2022-07-03T14:20:58.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=980] Channel closed 2022-07-03T14:20:58.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=981] Requesting new SSH session 2022-07-03T14:20:58.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=981] Command: which python3 2022-07-03T14:20:58.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=981] Received exit status 0 2022-07-03T14:20:58.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=981] Received channel close 2022-07-03T14:20:58.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=981] Channel closed 2022-07-03T14:20:58.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=982] Requesting new SSH session 2022-07-03T14:20:58.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=982] Command: sudo true 2022-07-03T14:20:58.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=981] Received exit status 0 2022-07-03T14:20:58.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=981] Received channel close 2022-07-03T14:20:58.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=981] Channel closed 2022-07-03T14:20:58.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=982] Requesting new SSH session 2022-07-03T14:20:58.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=982] Command: sudo true 2022-07-03T14:20:58.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=982] Received exit status 0 2022-07-03T14:20:58.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=982] Received channel close 2022-07-03T14:20:58.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=982] Channel closed 2022-07-03T14:20:58.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=983] Requesting new SSH session 2022-07-03T14:20:58.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:20:58.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=982] Received exit status 0 2022-07-03T14:20:58.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=982] Received channel close 2022-07-03T14:20:58.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=982] Channel closed 2022-07-03T14:20:58.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=983] Requesting new SSH session 2022-07-03T14:20:58.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:20:58.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=983] Received exit status 0 2022-07-03T14:20:58.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=983] Received channel close 2022-07-03T14:20:58.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=983] Channel closed 2022-07-03T14:20:58.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=983] Received exit status 0 2022-07-03T14:20:58.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=983] Received channel close 2022-07-03T14:20:58.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=983] Channel closed 2022-07-03T14:20:58.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=353] Login timeout expired 2022-07-03T14:20:58.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=353] Aborting connection 2022-07-03T14:20:58.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:20:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:20:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:21:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:21:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:21:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:21:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:21:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:21:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:21:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:21:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:21:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:21:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:21:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:21:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:21:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:21:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:21:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:21:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:21:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:21:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:21:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:21:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:21:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:21:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:21:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:21:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:21:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:21:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:21:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:21:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:21:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:21:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:21:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:21:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:21:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:21:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:21:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:21:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:21:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:21:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:21:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:21:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:21:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:21:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:21:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:21:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:21:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=353] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:21:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=353] Local address: 172.21.15.161, port 54568 2022-07-03T14:21:00.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=353] Peer address: 172.21.15.101, port 22 2022-07-03T14:21:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=976] Received exit status 0 2022-07-03T14:21:00.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=976] Received channel close 2022-07-03T14:21:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=976] Channel closed 2022-07-03T14:21:00.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=977] Requesting new SSH session 2022-07-03T14:21:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=977] Command: which python3 2022-07-03T14:21:00.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=976] Received exit status 0 2022-07-03T14:21:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=976] Received channel close 2022-07-03T14:21:00.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=976] Channel closed 2022-07-03T14:21:00.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=977] Requesting new SSH session 2022-07-03T14:21:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=977] Command: which python3 2022-07-03T14:21:00.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=977] Received exit status 0 2022-07-03T14:21:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=977] Received channel close 2022-07-03T14:21:00.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=977] Channel closed 2022-07-03T14:21:00.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=978] Requesting new SSH session 2022-07-03T14:21:00.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=978] Command: sudo true 2022-07-03T14:21:00.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=977] Received exit status 0 2022-07-03T14:21:00.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=977] Received channel close 2022-07-03T14:21:00.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=977] Channel closed 2022-07-03T14:21:00.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=978] Requesting new SSH session 2022-07-03T14:21:00.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=978] Command: sudo true 2022-07-03T14:21:00.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=978] Received exit status 0 2022-07-03T14:21:00.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=978] Received channel close 2022-07-03T14:21:00.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=978] Channel closed 2022-07-03T14:21:00.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=979] Requesting new SSH session 2022-07-03T14:21:00.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:21:00.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=978] Received exit status 0 2022-07-03T14:21:00.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=978] Received channel close 2022-07-03T14:21:00.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=978] Channel closed 2022-07-03T14:21:00.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=979] Requesting new SSH session 2022-07-03T14:21:00.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:21:00.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=979] Received exit status 0 2022-07-03T14:21:00.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=979] Received channel close 2022-07-03T14:21:00.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=979] Channel closed 2022-07-03T14:21:00.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=979] Received exit status 0 2022-07-03T14:21:00.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=979] Received channel close 2022-07-03T14:21:00.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=979] Channel closed 2022-07-03T14:21:00.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=980] Requesting new SSH session 2022-07-03T14:21:00.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=980] Command: sudo true 2022-07-03T14:21:00.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=980] Requesting new SSH session 2022-07-03T14:21:00.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=980] Command: sudo true 2022-07-03T14:21:00.169 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=980] Received exit status 0 2022-07-03T14:21:00.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=980] Received channel close 2022-07-03T14:21:00.170 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=980] Channel closed 2022-07-03T14:21:00.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=981] Requesting new SSH session 2022-07-03T14:21:00.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=981] Command: which python3 2022-07-03T14:21:00.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=980] Received exit status 0 2022-07-03T14:21:00.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=980] Received channel close 2022-07-03T14:21:00.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=980] Channel closed 2022-07-03T14:21:00.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=981] Requesting new SSH session 2022-07-03T14:21:00.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=981] Command: which python3 2022-07-03T14:21:00.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=981] Received exit status 0 2022-07-03T14:21:00.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=981] Received channel close 2022-07-03T14:21:00.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=981] Channel closed 2022-07-03T14:21:00.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=982] Requesting new SSH session 2022-07-03T14:21:00.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=982] Command: sudo true 2022-07-03T14:21:00.176 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=981] Received exit status 0 2022-07-03T14:21:00.176 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=981] Received channel close 2022-07-03T14:21:00.176 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=981] Channel closed 2022-07-03T14:21:00.177 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=982] Requesting new SSH session 2022-07-03T14:21:00.177 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=982] Command: sudo true 2022-07-03T14:21:00.178 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=982] Received exit status 0 2022-07-03T14:21:00.178 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=982] Received channel close 2022-07-03T14:21:00.178 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=982] Channel closed 2022-07-03T14:21:00.179 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=983] Requesting new SSH session 2022-07-03T14:21:00.179 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:21:00.180 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=982] Received exit status 0 2022-07-03T14:21:00.180 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=982] Received channel close 2022-07-03T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=982] Channel closed 2022-07-03T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=983] Requesting new SSH session 2022-07-03T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:21:00.182 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=983] Received exit status 0 2022-07-03T14:21:00.182 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=983] Received channel close 2022-07-03T14:21:00.182 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=983] Channel closed 2022-07-03T14:21:00.183 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=983] Received exit status 0 2022-07-03T14:21:00.183 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=983] Received channel close 2022-07-03T14:21:00.183 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=983] Channel closed 2022-07-03T14:21:00.184 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=353] Login timeout expired 2022-07-03T14:21:00.184 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=353] Aborting connection 2022-07-03T14:21:00.184 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:20:59 smithi161 ceph-mon[91579]: pgmap v6657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:21:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:21:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:21:00.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:21:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:21:00.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:21:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:21:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:21:00.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:21:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:21:00.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:21:00.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:21:00.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:21:00.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:21:00.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:21:00.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:21:00.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:21:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:21:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:21:00.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:21:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:21:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:21:00.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:21:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:21:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:21:00.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:21:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:21:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:21:00.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:21:00.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:21:00.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:21:00.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:21:00.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:21:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:21:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:21:00.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:21:00.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:21:00.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:21:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:21:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:21:00.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:21:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:21:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:21:00.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:21:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:21:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=353] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:21:00.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=353] Local address: 172.21.15.161, port 54568 2022-07-03T14:21:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=353] Peer address: 172.21.15.101, port 22 2022-07-03T14:21:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=976] Received exit status 0 2022-07-03T14:21:00.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=976] Received channel close 2022-07-03T14:21:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=976] Channel closed 2022-07-03T14:21:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=977] Requesting new SSH session 2022-07-03T14:21:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=977] Command: which python3 2022-07-03T14:21:00.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=976] Received exit status 0 2022-07-03T14:21:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=976] Received channel close 2022-07-03T14:21:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=976] Channel closed 2022-07-03T14:21:00.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=977] Requesting new SSH session 2022-07-03T14:21:00.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=977] Command: which python3 2022-07-03T14:21:00.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=977] Received exit status 0 2022-07-03T14:21:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=977] Received channel close 2022-07-03T14:21:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=977] Channel closed 2022-07-03T14:21:00.208 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=978] Requesting new SSH session 2022-07-03T14:21:00.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=978] Command: sudo true 2022-07-03T14:21:00.209 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=977] Received exit status 0 2022-07-03T14:21:00.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=977] Received channel close 2022-07-03T14:21:00.210 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=977] Channel closed 2022-07-03T14:21:00.211 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=978] Requesting new SSH session 2022-07-03T14:21:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=978] Command: sudo true 2022-07-03T14:21:00.212 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=978] Received exit status 0 2022-07-03T14:21:00.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=978] Received channel close 2022-07-03T14:21:00.213 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=978] Channel closed 2022-07-03T14:21:00.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=979] Requesting new SSH session 2022-07-03T14:21:00.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:21:00.215 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=978] Received exit status 0 2022-07-03T14:21:00.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=978] Received channel close 2022-07-03T14:21:00.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=978] Channel closed 2022-07-03T14:21:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=979] Requesting new SSH session 2022-07-03T14:21:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=979] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 check-host 2022-07-03T14:21:00.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=979] Received exit status 0 2022-07-03T14:21:00.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=979] Received channel close 2022-07-03T14:21:00.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=979] Channel closed 2022-07-03T14:21:00.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=979] Received exit status 0 2022-07-03T14:21:00.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=979] Received channel close 2022-07-03T14:21:00.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=979] Channel closed 2022-07-03T14:21:00.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=980] Requesting new SSH session 2022-07-03T14:21:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=980] Command: sudo true 2022-07-03T14:21:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=980] Requesting new SSH session 2022-07-03T14:21:00.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=980] Command: sudo true 2022-07-03T14:21:00.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=980] Received exit status 0 2022-07-03T14:21:00.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=980] Received channel close 2022-07-03T14:21:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=980] Channel closed 2022-07-03T14:21:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=981] Requesting new SSH session 2022-07-03T14:21:00.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=981] Command: which python3 2022-07-03T14:21:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=980] Received exit status 0 2022-07-03T14:21:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=980] Received channel close 2022-07-03T14:21:00.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=980] Channel closed 2022-07-03T14:21:00.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=981] Requesting new SSH session 2022-07-03T14:21:00.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=981] Command: which python3 2022-07-03T14:21:00.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=981] Received exit status 0 2022-07-03T14:21:00.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=981] Received channel close 2022-07-03T14:21:00.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=981] Channel closed 2022-07-03T14:21:00.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=982] Requesting new SSH session 2022-07-03T14:21:00.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=982] Command: sudo true 2022-07-03T14:21:00.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=981] Received exit status 0 2022-07-03T14:21:00.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=981] Received channel close 2022-07-03T14:21:00.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=981] Channel closed 2022-07-03T14:21:00.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=982] Requesting new SSH session 2022-07-03T14:21:00.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=982] Command: sudo true 2022-07-03T14:21:00.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=982] Received exit status 0 2022-07-03T14:21:00.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=982] Received channel close 2022-07-03T14:21:00.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=982] Channel closed 2022-07-03T14:21:00.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=983] Requesting new SSH session 2022-07-03T14:21:00.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:21:00.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=982] Received exit status 0 2022-07-03T14:21:00.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=982] Received channel close 2022-07-03T14:21:00.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=982] Channel closed 2022-07-03T14:21:00.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=983] Requesting new SSH session 2022-07-03T14:21:00.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=983] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:21:00.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=983] Received exit status 0 2022-07-03T14:21:00.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=983] Received channel close 2022-07-03T14:21:00.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=983] Channel closed 2022-07-03T14:21:00.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=983] Received exit status 0 2022-07-03T14:21:00.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=983] Received channel close 2022-07-03T14:21:00.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=983] Channel closed 2022-07-03T14:21:00.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=353] Login timeout expired 2022-07-03T14:21:00.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=353] Aborting connection 2022-07-03T14:21:00.235 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:20:59 smithi188 ceph-mon[92425]: pgmap v6657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:01 smithi161 ceph-mon[91579]: pgmap v6658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:01 smithi188 ceph-mon[92425]: pgmap v6658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:03 smithi161 ceph-mon[91579]: pgmap v6659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:03 smithi188 ceph-mon[92425]: pgmap v6659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:05 smithi161 ceph-mon[91579]: pgmap v6660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:05 smithi188 ceph-mon[92425]: pgmap v6660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:07 smithi161 ceph-mon[91579]: pgmap v6661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:07 smithi188 ceph-mon[92425]: pgmap v6661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:09 smithi161 ceph-mon[91579]: pgmap v6662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:09 smithi188 ceph-mon[92425]: pgmap v6662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:11 smithi161 ceph-mon[91579]: pgmap v6663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:11 smithi188 ceph-mon[92425]: pgmap v6663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:13 smithi161 ceph-mon[91579]: pgmap v6664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:13 smithi188 ceph-mon[92425]: pgmap v6664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:15 smithi161 ceph-mon[91579]: pgmap v6665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:15 smithi188 ceph-mon[92425]: pgmap v6665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:17 smithi161 ceph-mon[91579]: pgmap v6666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:17 smithi188 ceph-mon[92425]: pgmap v6666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:19 smithi161 ceph-mon[91579]: pgmap v6667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:19 smithi188 ceph-mon[92425]: pgmap v6667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:21 smithi161 ceph-mon[91579]: pgmap v6668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:21 smithi188 ceph-mon[92425]: pgmap v6668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:23 smithi161 ceph-mon[91579]: pgmap v6669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:24.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:23 smithi188 ceph-mon[92425]: pgmap v6669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:25 smithi161 ceph-mon[91579]: pgmap v6670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:25 smithi188 ceph-mon[92425]: pgmap v6670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:27 smithi161 ceph-mon[91579]: pgmap v6671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:27 smithi188 ceph-mon[92425]: pgmap v6671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:29 smithi161 ceph-mon[91579]: pgmap v6672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:29 smithi188 ceph-mon[92425]: pgmap v6672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:31 smithi161 ceph-mon[91579]: pgmap v6673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:31 smithi188 ceph-mon[92425]: pgmap v6673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:33 smithi161 ceph-mon[91579]: pgmap v6674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:33 smithi188 ceph-mon[92425]: pgmap v6674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:35 smithi161 ceph-mon[91579]: pgmap v6675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:35 smithi188 ceph-mon[92425]: pgmap v6675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:37 smithi161 ceph-mon[91579]: pgmap v6676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:37 smithi188 ceph-mon[92425]: pgmap v6676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:39 smithi161 ceph-mon[91579]: pgmap v6677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:39 smithi188 ceph-mon[92425]: pgmap v6677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:41 smithi161 ceph-mon[91579]: pgmap v6678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:41 smithi188 ceph-mon[92425]: pgmap v6678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:43 smithi161 ceph-mon[91579]: pgmap v6679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:43 smithi188 ceph-mon[92425]: pgmap v6679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:45 smithi161 ceph-mon[91579]: pgmap v6680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:45 smithi188 ceph-mon[92425]: pgmap v6680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:47 smithi161 ceph-mon[91579]: pgmap v6681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:47 smithi188 ceph-mon[92425]: pgmap v6681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:49 smithi161 ceph-mon[91579]: pgmap v6682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:49 smithi188 ceph-mon[92425]: pgmap v6682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:51 smithi161 ceph-mon[91579]: pgmap v6683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:51 smithi188 ceph-mon[92425]: pgmap v6683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:53 smithi161 ceph-mon[91579]: pgmap v6684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:53 smithi188 ceph-mon[92425]: pgmap v6684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:21:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:55 smithi161 ceph-mon[91579]: pgmap v6685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:55 smithi188 ceph-mon[92425]: pgmap v6685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:21:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:57 smithi161 ceph-mon[91579]: pgmap v6686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:21:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:21:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:21:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:21:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:57 smithi188 ceph-mon[92425]: pgmap v6686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:21:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:21:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:21:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:21:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:21:58.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:21:58.307+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:21:58.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:21:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:21:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:21:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:21:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:21:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:21:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:21:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:21:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:21:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:21:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:21:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:21:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:21:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:21:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:21:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:21:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:21:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:21:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:21:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:21:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:21:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:21:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:21:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:21:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:21:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:21:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:21:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:21:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:21:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:21:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:21:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:21:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:21:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:21:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:21:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:21:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:21:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:21:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:21:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:21:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:21:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:21:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:21:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:21:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=355] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:21:58.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=355] Local address: 172.21.15.161, port 54608 2022-07-03T14:21:58.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=355] Peer address: 172.21.15.101, port 22 2022-07-03T14:21:58.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=355] Connection lost 2022-07-03T14:21:58.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=355] Aborting connection 2022-07-03T14:21:58.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:21:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:21:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:21:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:21:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:21:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:21:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:21:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:21:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:22:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:22:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:22:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:22:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:22:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:22:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:22:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:22:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:22:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:22:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:22:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:22:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:22:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:22:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:22:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:22:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:22:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:22:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:22:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:22:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:22:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:22:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:22:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:22:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:22:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:22:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:22:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:22:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:22:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:22:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:22:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:22:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:22:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:22:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:22:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:22:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:22:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:22:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:22:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:22:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:22:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:22:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:22:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:22:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:22:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:22:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=355] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:22:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=355] Local address: 172.21.15.161, port 54608 2022-07-03T14:22:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=355] Peer address: 172.21.15.101, port 22 2022-07-03T14:22:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=355] Connection lost 2022-07-03T14:22:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=355] Aborting connection 2022-07-03T14:22:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:21:59 smithi161 ceph-mon[91579]: pgmap v6687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:22:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:22:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:22:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:22:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:22:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:22:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:22:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:22:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:22:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:22:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:22:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:22:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:22:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:22:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:22:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:22:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:22:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:22:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:22:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:22:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:22:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:22:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:22:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:22:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:22:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:22:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:22:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:22:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:22:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:22:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:22:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:22:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:22:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:22:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:22:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:22:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:22:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:22:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:22:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:22:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:22:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:22:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:22:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:22:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:22:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:22:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=355] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:22:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=355] Local address: 172.21.15.161, port 54608 2022-07-03T14:22:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=355] Peer address: 172.21.15.101, port 22 2022-07-03T14:22:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=355] Connection lost 2022-07-03T14:22:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=355] Aborting connection 2022-07-03T14:22:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:21:59 smithi188 ceph-mon[92425]: pgmap v6687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:01 smithi161 ceph-mon[91579]: pgmap v6688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:01 smithi188 ceph-mon[92425]: pgmap v6688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:03 smithi161 ceph-mon[91579]: pgmap v6689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:03 smithi188 ceph-mon[92425]: pgmap v6689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:05 smithi161 ceph-mon[91579]: pgmap v6690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:05 smithi188 ceph-mon[92425]: pgmap v6690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:07 smithi161 ceph-mon[91579]: pgmap v6691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:07 smithi188 ceph-mon[92425]: pgmap v6691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:09 smithi161 ceph-mon[91579]: pgmap v6692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:09 smithi188 ceph-mon[92425]: pgmap v6692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:11 smithi161 ceph-mon[91579]: pgmap v6693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:11 smithi188 ceph-mon[92425]: pgmap v6693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:13 smithi161 ceph-mon[91579]: pgmap v6694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:13 smithi188 ceph-mon[92425]: pgmap v6694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:15 smithi161 ceph-mon[91579]: pgmap v6695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:15 smithi188 ceph-mon[92425]: pgmap v6695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:17 smithi161 ceph-mon[91579]: pgmap v6696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:17 smithi188 ceph-mon[92425]: pgmap v6696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:19 smithi161 ceph-mon[91579]: pgmap v6697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:19 smithi188 ceph-mon[92425]: pgmap v6697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:21 smithi161 ceph-mon[91579]: pgmap v6698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:21 smithi188 ceph-mon[92425]: pgmap v6698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:23 smithi161 ceph-mon[91579]: pgmap v6699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:23 smithi188 ceph-mon[92425]: pgmap v6699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:25 smithi161 ceph-mon[91579]: pgmap v6700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:25 smithi188 ceph-mon[92425]: pgmap v6700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:27 smithi161 ceph-mon[91579]: pgmap v6701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:27 smithi188 ceph-mon[92425]: pgmap v6701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:29 smithi161 ceph-mon[91579]: pgmap v6702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:29 smithi188 ceph-mon[92425]: pgmap v6702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:31 smithi161 ceph-mon[91579]: pgmap v6703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:31 smithi188 ceph-mon[92425]: pgmap v6703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:33 smithi161 ceph-mon[91579]: pgmap v6704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:33 smithi188 ceph-mon[92425]: pgmap v6704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:35 smithi161 ceph-mon[91579]: pgmap v6705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:35 smithi188 ceph-mon[92425]: pgmap v6705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:37 smithi161 ceph-mon[91579]: pgmap v6706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:37 smithi188 ceph-mon[92425]: pgmap v6706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:39 smithi161 ceph-mon[91579]: pgmap v6707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:39 smithi188 ceph-mon[92425]: pgmap v6707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:41 smithi161 ceph-mon[91579]: pgmap v6708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:41 smithi188 ceph-mon[92425]: pgmap v6708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:43 smithi161 ceph-mon[91579]: pgmap v6709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:43 smithi188 ceph-mon[92425]: pgmap v6709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:45 smithi161 ceph-mon[91579]: pgmap v6710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:45 smithi188 ceph-mon[92425]: pgmap v6710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:47 smithi161 ceph-mon[91579]: pgmap v6711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:47 smithi188 ceph-mon[92425]: pgmap v6711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:49 smithi161 ceph-mon[91579]: pgmap v6712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:49 smithi188 ceph-mon[92425]: pgmap v6712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:51 smithi161 ceph-mon[91579]: pgmap v6713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:51 smithi188 ceph-mon[92425]: pgmap v6713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:22:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:53 smithi161 ceph-mon[91579]: pgmap v6714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:53 smithi188 ceph-mon[92425]: pgmap v6714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:22:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:55 smithi161 ceph-mon[91579]: pgmap v6715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:55 smithi188 ceph-mon[92425]: pgmap v6715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:22:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:57 smithi161 ceph-mon[91579]: pgmap v6716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:22:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:22:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:22:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:22:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:57 smithi188 ceph-mon[92425]: pgmap v6716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:22:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:22:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:22:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:22:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:22:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:22:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:22:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:22:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:22:59.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:22:58.820+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:22:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:22:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:22:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:22:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:22:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:22:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:22:59.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:22:59.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:22:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:22:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:22:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:22:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:22:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:22:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:22:59.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:22:59.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:22:59.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:22:59.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:22:59.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:22:59.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:22:59.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:22:59.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:22:59.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:22:59.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:22:59.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:22:59.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:22:59.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:22:59.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:22:59.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:22:59.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:22:59.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:22:59.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:22:59.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:22:59.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:22:59.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:22:59.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:22:59.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:22:59.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:22:59.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:22:59.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:22:59.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:22:59.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:22:59.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:22:59.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:22:59.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=357] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:22:59.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=357] Local address: 172.21.15.161, port 54636 2022-07-03T14:22:59.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=357] Peer address: 172.21.15.101, port 22 2022-07-03T14:22:59.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=357] Connection lost 2022-07-03T14:22:59.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=357] Aborting connection 2022-07-03T14:22:59.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:22:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:22:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:22:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:22:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:22:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:59 smithi161 ceph-mon[91579]: pgmap v6717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:23:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:22:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:23:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:23:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:23:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:23:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:23:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:23:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:23:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:23:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:23:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:23:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:23:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:23:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:23:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:23:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:23:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:23:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:23:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:23:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:23:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:23:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:23:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:23:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:23:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:23:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:23:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:23:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:23:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:23:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:23:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:23:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:23:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:23:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:23:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:23:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:23:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:23:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:23:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:23:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:23:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:23:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:23:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:23:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:23:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:23:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:23:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=357] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:23:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=357] Local address: 172.21.15.161, port 54636 2022-07-03T14:23:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=357] Peer address: 172.21.15.101, port 22 2022-07-03T14:23:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=357] Connection lost 2022-07-03T14:23:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=357] Aborting connection 2022-07-03T14:23:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:59 smithi188 ceph-mon[92425]: pgmap v6717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:23:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:22:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:23:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:23:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:23:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:23:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:23:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:23:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:23:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:23:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:23:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:23:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:23:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:23:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:23:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:23:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:23:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:23:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:23:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:23:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:23:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:23:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:23:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:23:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:23:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:23:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:23:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:23:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:23:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:23:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:23:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:23:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:23:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:23:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:23:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:23:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:23:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:23:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:23:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:23:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:23:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:23:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:23:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:23:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:23:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:23:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:23:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=357] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:23:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=357] Local address: 172.21.15.161, port 54636 2022-07-03T14:23:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=357] Peer address: 172.21.15.101, port 22 2022-07-03T14:23:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=357] Connection lost 2022-07-03T14:23:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=357] Aborting connection 2022-07-03T14:23:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:01 smithi188 ceph-mon[92425]: pgmap v6718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:01 smithi161 ceph-mon[91579]: pgmap v6718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:23:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:03.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:23:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:23:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:23:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:23:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:23:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: pgmap v6719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:23:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:23:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:23:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:23:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: pgmap v6719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:05 smithi161 ceph-mon[91579]: pgmap v6720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:05 smithi188 ceph-mon[92425]: pgmap v6720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:07 smithi161 ceph-mon[91579]: pgmap v6721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:07 smithi188 ceph-mon[92425]: pgmap v6721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:09 smithi161 ceph-mon[91579]: pgmap v6722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:09 smithi188 ceph-mon[92425]: pgmap v6722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:11 smithi161 ceph-mon[91579]: pgmap v6723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:11 smithi188 ceph-mon[92425]: pgmap v6723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:13 smithi161 ceph-mon[91579]: pgmap v6724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:13 smithi188 ceph-mon[92425]: pgmap v6724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:15 smithi161 ceph-mon[91579]: pgmap v6725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:15 smithi188 ceph-mon[92425]: pgmap v6725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:17 smithi161 ceph-mon[91579]: pgmap v6726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:17 smithi188 ceph-mon[92425]: pgmap v6726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:19 smithi161 ceph-mon[91579]: pgmap v6727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:19 smithi188 ceph-mon[92425]: pgmap v6727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:21 smithi161 ceph-mon[91579]: pgmap v6728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:21 smithi188 ceph-mon[92425]: pgmap v6728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:23 smithi188 ceph-mon[92425]: pgmap v6729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:23 smithi161 ceph-mon[91579]: pgmap v6729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:25 smithi161 ceph-mon[91579]: pgmap v6730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:25 smithi188 ceph-mon[92425]: pgmap v6730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:27 smithi161 ceph-mon[91579]: pgmap v6731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:27 smithi188 ceph-mon[92425]: pgmap v6731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:29 smithi161 ceph-mon[91579]: pgmap v6732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:29 smithi188 ceph-mon[92425]: pgmap v6732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:31 smithi161 ceph-mon[91579]: pgmap v6733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:31 smithi188 ceph-mon[92425]: pgmap v6733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:33 smithi161 ceph-mon[91579]: pgmap v6734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:33 smithi188 ceph-mon[92425]: pgmap v6734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:35 smithi161 ceph-mon[91579]: pgmap v6735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:35 smithi188 ceph-mon[92425]: pgmap v6735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:37 smithi161 ceph-mon[91579]: pgmap v6736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:37 smithi188 ceph-mon[92425]: pgmap v6736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:39 smithi161 ceph-mon[91579]: pgmap v6737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:39 smithi188 ceph-mon[92425]: pgmap v6737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:41 smithi161 ceph-mon[91579]: pgmap v6738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:41 smithi188 ceph-mon[92425]: pgmap v6738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:43 smithi161 ceph-mon[91579]: pgmap v6739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:43 smithi188 ceph-mon[92425]: pgmap v6739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:45 smithi188 ceph-mon[92425]: pgmap v6740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:45 smithi161 ceph-mon[91579]: pgmap v6740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:47 smithi161 ceph-mon[91579]: pgmap v6741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:47 smithi188 ceph-mon[92425]: pgmap v6741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:49 smithi161 ceph-mon[91579]: pgmap v6742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:49 smithi188 ceph-mon[92425]: pgmap v6742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:51 smithi161 ceph-mon[91579]: pgmap v6743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:51 smithi188 ceph-mon[92425]: pgmap v6743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:53 smithi161 ceph-mon[91579]: pgmap v6744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:53 smithi188 ceph-mon[92425]: pgmap v6744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:23:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:23:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:55 smithi188 ceph-mon[92425]: pgmap v6745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:55 smithi161 ceph-mon[91579]: pgmap v6745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:23:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:57 smithi161 ceph-mon[91579]: pgmap v6746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:23:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:23:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:57 smithi188 ceph-mon[92425]: pgmap v6746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:23:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:23:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:23:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:23:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:24:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:59 smithi161 ceph-mon[91579]: pgmap v6747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:59 smithi188 ceph-mon[92425]: pgmap v6747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:02.087 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:01 smithi161 ceph-mon[91579]: pgmap v6748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:02.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:24:02.076+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:24:02.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:24:02.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:24:02.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:24:02.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:24:02.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:24:02.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:24:02.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:24:02.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:24:02.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:24:02.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:24:02.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:24:02.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:24:02.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:24:02.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:24:02.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:24:02.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:24:02.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:24:02.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:24:02.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:24:02.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:24:02.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:24:02.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:24:02.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:24:02.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:24:02.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:24:02.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:24:02.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:24:02.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:24:02.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:24:02.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:24:02.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:24:02.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:24:02.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:24:02.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:24:02.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:24:02.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:24:02.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:24:02.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:24:02.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:24:02.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:24:02.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:24:02.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:24:02.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:24:02.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:24:02.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=359] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:24:02.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=359] Local address: 172.21.15.161, port 54696 2022-07-03T14:24:02.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=359] Peer address: 172.21.15.101, port 22 2022-07-03T14:24:02.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=359] Connection lost 2022-07-03T14:24:02.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=359] Aborting connection 2022-07-03T14:24:02.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:24:02.108 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:01 smithi188 ceph-mon[92425]: pgmap v6748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:24:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:24:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:24:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:24:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:24:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:24:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:24:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:24:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:24:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:24:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:24:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:24:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:24:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:24:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:24:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:24:04.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:24:04.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:24:04.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:24:04.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:24:04.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:24:04.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:24:04.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:24:04.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:24:04.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:24:04.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:24:04.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:24:04.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:24:04.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:24:04.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:24:04.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:24:04.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:24:04.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:24:04.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:24:04.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:24:04.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:24:04.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:24:04.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:24:04.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:24:04.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:24:04.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:24:04.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:24:04.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:24:04.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:24:04.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:24:04.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:24:04.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:24:04.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:24:04.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:24:04.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:24:04.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:24:04.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:24:04.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=359] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:24:04.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=359] Local address: 172.21.15.161, port 54696 2022-07-03T14:24:04.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=359] Peer address: 172.21.15.101, port 22 2022-07-03T14:24:04.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=359] Connection lost 2022-07-03T14:24:04.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=359] Aborting connection 2022-07-03T14:24:04.163 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:03 smithi161 ceph-mon[91579]: pgmap v6749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:04.164 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:24:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:24:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:24:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:24:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:24:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:24:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:24:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:24:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:24:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:24:04.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:24:04.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:24:04.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:24:04.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:24:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:24:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:24:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:24:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:24:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:24:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:24:04.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:24:04.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:24:04.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:24:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:24:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:24:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:24:04.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:24:04.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:24:04.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:24:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:24:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:24:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:24:04.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:24:04.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:24:04.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:24:04.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:24:04.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:24:04.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:24:04.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:24:04.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:24:04.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:24:04.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:24:04.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:24:04.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:24:04.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:24:04.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:24:04.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=359] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:24:04.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=359] Local address: 172.21.15.161, port 54696 2022-07-03T14:24:04.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=359] Peer address: 172.21.15.101, port 22 2022-07-03T14:24:04.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=359] Connection lost 2022-07-03T14:24:04.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=359] Aborting connection 2022-07-03T14:24:04.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:03 smithi188 ceph-mon[92425]: pgmap v6749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:04.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:05 smithi188 ceph-mon[92425]: pgmap v6750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:05 smithi161 ceph-mon[91579]: pgmap v6750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:07 smithi161 ceph-mon[91579]: pgmap v6751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:07 smithi188 ceph-mon[92425]: pgmap v6751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:09 smithi161 ceph-mon[91579]: pgmap v6752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:09 smithi188 ceph-mon[92425]: pgmap v6752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:11 smithi161 ceph-mon[91579]: pgmap v6753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:11 smithi188 ceph-mon[92425]: pgmap v6753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:13 smithi161 ceph-mon[91579]: pgmap v6754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:13 smithi188 ceph-mon[92425]: pgmap v6754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:15 smithi188 ceph-mon[92425]: pgmap v6755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:15 smithi161 ceph-mon[91579]: pgmap v6755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:17 smithi161 ceph-mon[91579]: pgmap v6756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:17 smithi188 ceph-mon[92425]: pgmap v6756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:19 smithi161 ceph-mon[91579]: pgmap v6757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:19 smithi188 ceph-mon[92425]: pgmap v6757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:21 smithi161 ceph-mon[91579]: pgmap v6758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:21 smithi188 ceph-mon[92425]: pgmap v6758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:23 smithi161 ceph-mon[91579]: pgmap v6759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:23 smithi188 ceph-mon[92425]: pgmap v6759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:25 smithi188 ceph-mon[92425]: pgmap v6760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:25 smithi161 ceph-mon[91579]: pgmap v6760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:27 smithi161 ceph-mon[91579]: pgmap v6761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:27 smithi188 ceph-mon[92425]: pgmap v6761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:29 smithi161 ceph-mon[91579]: pgmap v6762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:29 smithi188 ceph-mon[92425]: pgmap v6762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:31 smithi161 ceph-mon[91579]: pgmap v6763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:31 smithi188 ceph-mon[92425]: pgmap v6763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:33 smithi161 ceph-mon[91579]: pgmap v6764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:33 smithi188 ceph-mon[92425]: pgmap v6764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:35 smithi188 ceph-mon[92425]: pgmap v6765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:35 smithi161 ceph-mon[91579]: pgmap v6765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:37 smithi161 ceph-mon[91579]: pgmap v6766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:37 smithi188 ceph-mon[92425]: pgmap v6766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:39 smithi161 ceph-mon[91579]: pgmap v6767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:39 smithi188 ceph-mon[92425]: pgmap v6767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:41 smithi161 ceph-mon[91579]: pgmap v6768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:41 smithi188 ceph-mon[92425]: pgmap v6768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:43 smithi161 ceph-mon[91579]: pgmap v6769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:43 smithi188 ceph-mon[92425]: pgmap v6769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:45 smithi188 ceph-mon[92425]: pgmap v6770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:45 smithi161 ceph-mon[91579]: pgmap v6770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:47 smithi161 ceph-mon[91579]: pgmap v6771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:47 smithi188 ceph-mon[92425]: pgmap v6771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:49 smithi161 ceph-mon[91579]: pgmap v6772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:49 smithi188 ceph-mon[92425]: pgmap v6772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:51 smithi161 ceph-mon[91579]: pgmap v6773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:51 smithi188 ceph-mon[92425]: pgmap v6773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:53 smithi161 ceph-mon[91579]: pgmap v6774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:53 smithi188 ceph-mon[92425]: pgmap v6774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:24:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:55 smithi161 ceph-mon[91579]: pgmap v6775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:55 smithi188 ceph-mon[92425]: pgmap v6775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:24:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:57 smithi161 ceph-mon[91579]: pgmap v6776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:24:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:24:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:24:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:57 smithi188 ceph-mon[92425]: pgmap v6776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:24:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:24:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:24:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:24:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:24:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:24:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:24:59 smithi161 ceph-mon[91579]: pgmap v6777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:24:59 smithi188 ceph-mon[92425]: pgmap v6777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:01 smithi161 ceph-mon[91579]: pgmap v6778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:01 smithi188 ceph-mon[92425]: pgmap v6778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:02.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:25:02.582+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:25:02.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:25:02.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:25:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:25:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:25:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:25:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:25:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:25:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:25:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:25:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:25:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:25:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:25:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:25:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:25:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:25:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:25:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:25:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:25:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:25:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:25:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:25:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:25:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:25:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:25:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:25:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:25:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:25:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:25:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:25:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:25:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:25:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:25:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:25:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:25:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:25:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:25:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:25:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:25:02.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:25:02.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:25:02.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:25:02.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:25:02.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:25:02.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:25:02.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=361] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:25:02.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=361] Local address: 172.21.15.161, port 54724 2022-07-03T14:25:02.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=361] Peer address: 172.21.15.101, port 22 2022-07-03T14:25:02.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=361] Connection lost 2022-07-03T14:25:02.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=361] Aborting connection 2022-07-03T14:25:02.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:25:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:25:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:25:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:25:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:25:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:25:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:25:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:25:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:25:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:25:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:25:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:25:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:25:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:25:04.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:25:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:25:04.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:25:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:25:04.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:25:04.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:25:04.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:25:04.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:25:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:25:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:25:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:25:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:25:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:25:04.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:25:04.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:25:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:25:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:25:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:25:04.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:25:04.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:25:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:25:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:25:04.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:25:04.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:25:04.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:25:04.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:25:04.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:25:04.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:25:04.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:25:04.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:25:04.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:25:04.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:25:04.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:25:04.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:25:04.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:25:04.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:25:04.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:25:04.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:25:04.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:25:04.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=361] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:25:04.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=361] Local address: 172.21.15.161, port 54724 2022-07-03T14:25:04.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=361] Peer address: 172.21.15.101, port 22 2022-07-03T14:25:04.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=361] Connection lost 2022-07-03T14:25:04.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=361] Aborting connection 2022-07-03T14:25:04.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:03 smithi188 ceph-mon[92425]: pgmap v6779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:04.189 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:25:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:25:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:25:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:25:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:25:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:25:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:25:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:25:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:25:04.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:25:04.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:25:04.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:25:04.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:25:04.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:25:04.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:25:04.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:25:04.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:25:04.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:25:04.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:25:04.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:25:04.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:25:04.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:25:04.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:25:04.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:25:04.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:25:04.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:25:04.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:25:04.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:25:04.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:25:04.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:25:04.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:25:04.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:25:04.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:25:04.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:25:04.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:25:04.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:25:04.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:25:04.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:25:04.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:25:04.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:25:04.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:25:04.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:25:04.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:25:04.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:25:04.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:25:04.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:25:04.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=361] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:25:04.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=361] Local address: 172.21.15.161, port 54724 2022-07-03T14:25:04.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=361] Peer address: 172.21.15.101, port 22 2022-07-03T14:25:04.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=361] Connection lost 2022-07-03T14:25:04.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=361] Aborting connection 2022-07-03T14:25:04.406 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:03 smithi161 ceph-mon[91579]: pgmap v6779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:04.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:05 smithi188 ceph-mon[92425]: pgmap v6780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:05 smithi161 ceph-mon[91579]: pgmap v6780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:07 smithi161 ceph-mon[91579]: pgmap v6781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:07 smithi188 ceph-mon[92425]: pgmap v6781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:09 smithi161 ceph-mon[91579]: pgmap v6782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:09 smithi188 ceph-mon[92425]: pgmap v6782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:11 smithi161 ceph-mon[91579]: pgmap v6783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:11 smithi188 ceph-mon[92425]: pgmap v6783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:13 smithi161 ceph-mon[91579]: pgmap v6784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:13 smithi188 ceph-mon[92425]: pgmap v6784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:15 smithi161 ceph-mon[91579]: pgmap v6785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:15 smithi188 ceph-mon[92425]: pgmap v6785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:17 smithi161 ceph-mon[91579]: pgmap v6786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:17 smithi188 ceph-mon[92425]: pgmap v6786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:19 smithi161 ceph-mon[91579]: pgmap v6787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:19 smithi188 ceph-mon[92425]: pgmap v6787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:21 smithi188 ceph-mon[92425]: pgmap v6788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:21 smithi161 ceph-mon[91579]: pgmap v6788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:23 smithi188 ceph-mon[92425]: pgmap v6789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:23 smithi161 ceph-mon[91579]: pgmap v6789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:25 smithi188 ceph-mon[92425]: pgmap v6790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:25 smithi161 ceph-mon[91579]: pgmap v6790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:27 smithi161 ceph-mon[91579]: pgmap v6791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:27 smithi188 ceph-mon[92425]: pgmap v6791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:29 smithi161 ceph-mon[91579]: pgmap v6792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:29 smithi188 ceph-mon[92425]: pgmap v6792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:31 smithi161 ceph-mon[91579]: pgmap v6793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:31 smithi188 ceph-mon[92425]: pgmap v6793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:33 smithi161 ceph-mon[91579]: pgmap v6794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:33 smithi188 ceph-mon[92425]: pgmap v6794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:35 smithi161 ceph-mon[91579]: pgmap v6795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:35 smithi188 ceph-mon[92425]: pgmap v6795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:37 smithi161 ceph-mon[91579]: pgmap v6796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:37 smithi188 ceph-mon[92425]: pgmap v6796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:39 smithi188 ceph-mon[92425]: pgmap v6797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:39 smithi161 ceph-mon[91579]: pgmap v6797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:41 smithi188 ceph-mon[92425]: pgmap v6798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:41 smithi161 ceph-mon[91579]: pgmap v6798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:43 smithi188 ceph-mon[92425]: pgmap v6799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:43 smithi161 ceph-mon[91579]: pgmap v6799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:45 smithi188 ceph-mon[92425]: pgmap v6800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:45 smithi161 ceph-mon[91579]: pgmap v6800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:47 smithi161 ceph-mon[91579]: pgmap v6801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:47 smithi188 ceph-mon[92425]: pgmap v6801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:49 smithi161 ceph-mon[91579]: pgmap v6802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:49 smithi188 ceph-mon[92425]: pgmap v6802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:51 smithi161 ceph-mon[91579]: pgmap v6803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:52.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:51 smithi188 ceph-mon[92425]: pgmap v6803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:53 smithi188 ceph-mon[92425]: pgmap v6804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:53 smithi161 ceph-mon[91579]: pgmap v6804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:25:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:25:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:55 smithi188 ceph-mon[92425]: pgmap v6805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:55 smithi161 ceph-mon[91579]: pgmap v6805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:25:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:57 smithi188 ceph-mon[92425]: pgmap v6806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:25:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:25:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:25:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:25:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:57 smithi161 ceph-mon[91579]: pgmap v6806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:25:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:25:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:25:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:25:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:26:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:59 smithi188 ceph-mon[92425]: pgmap v6807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:59 smithi161 ceph-mon[91579]: pgmap v6807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:01 smithi188 ceph-mon[92425]: pgmap v6808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:01 smithi161 ceph-mon[91579]: pgmap v6808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:03 smithi188 ceph-mon[92425]: pgmap v6809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:26:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:26:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:26:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:03 smithi161 ceph-mon[91579]: pgmap v6809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:26:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:26:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:26:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:26:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:26:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:05 smithi188 ceph-mon[92425]: pgmap v6810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:05 smithi161 ceph-mon[91579]: pgmap v6810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:07 smithi188 ceph-mon[92425]: pgmap v6811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:07 smithi161 ceph-mon[91579]: pgmap v6811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:09 smithi161 ceph-mon[91579]: pgmap v6812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:09 smithi188 ceph-mon[92425]: pgmap v6812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:10 smithi161 ceph-mon[91579]: pgmap v6813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:10 smithi188 ceph-mon[92425]: pgmap v6813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:13 smithi161 ceph-mon[91579]: pgmap v6814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:13 smithi188 ceph-mon[92425]: pgmap v6814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:15 smithi161 ceph-mon[91579]: pgmap v6815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:15 smithi188 ceph-mon[92425]: pgmap v6815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:17 smithi188 ceph-mon[92425]: pgmap v6816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:17 smithi161 ceph-mon[91579]: pgmap v6816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:19 smithi188 ceph-mon[92425]: pgmap v6817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:19 smithi161 ceph-mon[91579]: pgmap v6817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:21 smithi188 ceph-mon[92425]: pgmap v6818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:21 smithi161 ceph-mon[91579]: pgmap v6818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:23 smithi188 ceph-mon[92425]: pgmap v6819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:23 smithi161 ceph-mon[91579]: pgmap v6819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:25 smithi188 ceph-mon[92425]: pgmap v6820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:25 smithi161 ceph-mon[91579]: pgmap v6820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:27 smithi188 ceph-mon[92425]: pgmap v6821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:27 smithi161 ceph-mon[91579]: pgmap v6821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:29 smithi188 ceph-mon[92425]: pgmap v6822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:29 smithi161 ceph-mon[91579]: pgmap v6822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:31 smithi161 ceph-mon[91579]: pgmap v6823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:31 smithi188 ceph-mon[92425]: pgmap v6823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:26:33.079+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:26:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:26:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:26:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:26:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:26:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:26:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:26:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:26:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:26:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:26:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:26:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:26:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:26:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:26:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:26:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:26:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:26:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:26:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:26:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:26:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:26:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:26:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:26:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:26:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:26:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:26:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:26:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:26:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:26:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:26:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:26:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:26:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:26:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:26:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:26:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:26:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:26:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:26:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:26:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:26:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:26:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:26:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:26:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:26:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:26:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=363] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:26:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=363] Local address: 172.21.15.161, port 54752 2022-07-03T14:26:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=363] Peer address: 172.21.15.101, port 22 2022-07-03T14:26:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1004] Received exit status 0 2022-07-03T14:26:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1004] Received channel close 2022-07-03T14:26:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1004] Channel closed 2022-07-03T14:26:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1004] Received exit status 0 2022-07-03T14:26:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1004] Received channel close 2022-07-03T14:26:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1005] Requesting new SSH session 2022-07-03T14:26:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1004] Channel closed 2022-07-03T14:26:33.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1005] Command: which python3 2022-07-03T14:26:33.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1005] Requesting new SSH session 2022-07-03T14:26:33.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1005] Command: which python3 2022-07-03T14:26:33.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1005] Received exit status 0 2022-07-03T14:26:33.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1005] Received channel close 2022-07-03T14:26:33.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1005] Channel closed 2022-07-03T14:26:33.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1006] Requesting new SSH session 2022-07-03T14:26:33.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1006] Command: sudo true 2022-07-03T14:26:33.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1005] Received exit status 0 2022-07-03T14:26:33.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1005] Received channel close 2022-07-03T14:26:33.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1005] Channel closed 2022-07-03T14:26:33.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1006] Requesting new SSH session 2022-07-03T14:26:33.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1006] Command: sudo true 2022-07-03T14:26:33.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1006] Received exit status 0 2022-07-03T14:26:33.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1006] Received channel close 2022-07-03T14:26:33.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1006] Channel closed 2022-07-03T14:26:33.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1007] Requesting new SSH session 2022-07-03T14:26:33.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:33.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1006] Received exit status 0 2022-07-03T14:26:33.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1006] Received channel close 2022-07-03T14:26:33.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1006] Channel closed 2022-07-03T14:26:33.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1007] Requesting new SSH session 2022-07-03T14:26:33.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:33.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1007] Received exit status 0 2022-07-03T14:26:33.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1007] Received channel close 2022-07-03T14:26:33.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1007] Channel closed 2022-07-03T14:26:33.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1007] Received exit status 0 2022-07-03T14:26:33.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1007] Received channel close 2022-07-03T14:26:33.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1007] Channel closed 2022-07-03T14:26:33.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=363] Login timeout expired 2022-07-03T14:26:33.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=363] Aborting connection 2022-07-03T14:26:33.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:26:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-mon[91579]: pgmap v6824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:33 smithi188 ceph-mon[92425]: pgmap v6824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:26:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:26:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:26:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:26:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:26:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:26:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:26:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:26:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:26:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:26:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:26:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:26:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:26:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:26:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:26:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:26:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:26:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:26:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:26:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:26:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:26:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:26:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:26:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:26:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:26:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:26:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:26:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:26:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:26:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:26:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:26:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:26:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:26:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:26:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:26:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:26:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:26:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:26:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:26:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:26:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:26:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:26:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:26:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:26:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:26:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=363] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:26:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=363] Local address: 172.21.15.161, port 54752 2022-07-03T14:26:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=363] Peer address: 172.21.15.101, port 22 2022-07-03T14:26:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1004] Received exit status 0 2022-07-03T14:26:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1004] Received channel close 2022-07-03T14:26:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1004] Channel closed 2022-07-03T14:26:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1004] Received exit status 0 2022-07-03T14:26:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1004] Received channel close 2022-07-03T14:26:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1005] Requesting new SSH session 2022-07-03T14:26:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1004] Channel closed 2022-07-03T14:26:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1005] Command: which python3 2022-07-03T14:26:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1005] Requesting new SSH session 2022-07-03T14:26:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1005] Command: which python3 2022-07-03T14:26:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1005] Received exit status 0 2022-07-03T14:26:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1005] Received channel close 2022-07-03T14:26:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1005] Channel closed 2022-07-03T14:26:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1006] Requesting new SSH session 2022-07-03T14:26:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1006] Command: sudo true 2022-07-03T14:26:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1005] Received exit status 0 2022-07-03T14:26:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1005] Received channel close 2022-07-03T14:26:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1005] Channel closed 2022-07-03T14:26:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1006] Requesting new SSH session 2022-07-03T14:26:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1006] Command: sudo true 2022-07-03T14:26:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1006] Received exit status 0 2022-07-03T14:26:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1006] Received channel close 2022-07-03T14:26:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1006] Channel closed 2022-07-03T14:26:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1007] Requesting new SSH session 2022-07-03T14:26:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1006] Received exit status 0 2022-07-03T14:26:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1006] Received channel close 2022-07-03T14:26:35.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1006] Channel closed 2022-07-03T14:26:35.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1007] Requesting new SSH session 2022-07-03T14:26:35.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:35.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1007] Received exit status 0 2022-07-03T14:26:35.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1007] Received channel close 2022-07-03T14:26:35.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1007] Channel closed 2022-07-03T14:26:35.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1007] Received exit status 0 2022-07-03T14:26:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1007] Received channel close 2022-07-03T14:26:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1007] Channel closed 2022-07-03T14:26:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=363] Login timeout expired 2022-07-03T14:26:35.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=363] Aborting connection 2022-07-03T14:26:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:26:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:26:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:26:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:26:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:26:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:26:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:26:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:26:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:26:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:26:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:26:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:26:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:26:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:26:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:26:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:26:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:26:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:26:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:26:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:26:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:26:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:26:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:26:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:26:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:26:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:26:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:26:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:26:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:26:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:26:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:26:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:26:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:26:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:26:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:26:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:26:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:26:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:26:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:26:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:26:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:26:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:26:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:26:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:26:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:26:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=363] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:26:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=363] Local address: 172.21.15.161, port 54752 2022-07-03T14:26:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=363] Peer address: 172.21.15.101, port 22 2022-07-03T14:26:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1004] Received exit status 0 2022-07-03T14:26:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1004] Received channel close 2022-07-03T14:26:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1004] Channel closed 2022-07-03T14:26:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1004] Received exit status 0 2022-07-03T14:26:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1004] Received channel close 2022-07-03T14:26:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1005] Requesting new SSH session 2022-07-03T14:26:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1004] Channel closed 2022-07-03T14:26:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1005] Command: which python3 2022-07-03T14:26:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1005] Requesting new SSH session 2022-07-03T14:26:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1005] Command: which python3 2022-07-03T14:26:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1005] Received exit status 0 2022-07-03T14:26:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1005] Received channel close 2022-07-03T14:26:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1005] Channel closed 2022-07-03T14:26:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1006] Requesting new SSH session 2022-07-03T14:26:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1006] Command: sudo true 2022-07-03T14:26:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1005] Received exit status 0 2022-07-03T14:26:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1005] Received channel close 2022-07-03T14:26:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1005] Channel closed 2022-07-03T14:26:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1006] Requesting new SSH session 2022-07-03T14:26:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1006] Command: sudo true 2022-07-03T14:26:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1006] Received exit status 0 2022-07-03T14:26:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1006] Received channel close 2022-07-03T14:26:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1006] Channel closed 2022-07-03T14:26:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1007] Requesting new SSH session 2022-07-03T14:26:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1006] Received exit status 0 2022-07-03T14:26:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1006] Received channel close 2022-07-03T14:26:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1006] Channel closed 2022-07-03T14:26:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1007] Requesting new SSH session 2022-07-03T14:26:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1007] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:26:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1007] Received exit status 0 2022-07-03T14:26:35.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1007] Received channel close 2022-07-03T14:26:35.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1007] Channel closed 2022-07-03T14:26:35.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1007] Received exit status 0 2022-07-03T14:26:35.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1007] Received channel close 2022-07-03T14:26:35.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1007] Channel closed 2022-07-03T14:26:35.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=363] Login timeout expired 2022-07-03T14:26:35.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=363] Aborting connection 2022-07-03T14:26:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:35 smithi161 ceph-mon[91579]: pgmap v6825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:35 smithi188 ceph-mon[92425]: pgmap v6825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:37 smithi161 ceph-mon[91579]: pgmap v6826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:37 smithi188 ceph-mon[92425]: pgmap v6826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:38 smithi161 ceph-mon[91579]: pgmap v6827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:38 smithi188 ceph-mon[92425]: pgmap v6827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:41 smithi161 ceph-mon[91579]: pgmap v6828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:41 smithi188 ceph-mon[92425]: pgmap v6828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:43 smithi188 ceph-mon[92425]: pgmap v6829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:43 smithi161 ceph-mon[91579]: pgmap v6829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:45 smithi188 ceph-mon[92425]: pgmap v6830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:45 smithi161 ceph-mon[91579]: pgmap v6830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:47 smithi188 ceph-mon[92425]: pgmap v6831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:47 smithi161 ceph-mon[91579]: pgmap v6831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:49 smithi188 ceph-mon[92425]: pgmap v6832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:49 smithi161 ceph-mon[91579]: pgmap v6832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:51 smithi188 ceph-mon[92425]: pgmap v6833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:51 smithi161 ceph-mon[91579]: pgmap v6833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:53 smithi188 ceph-mon[92425]: pgmap v6834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:53 smithi161 ceph-mon[91579]: pgmap v6834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:26:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:26:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:55 smithi188 ceph-mon[92425]: pgmap v6835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:55 smithi161 ceph-mon[91579]: pgmap v6835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:57 smithi188 ceph-mon[92425]: pgmap v6836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:26:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:26:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:26:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:26:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:57 smithi161 ceph-mon[91579]: pgmap v6836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:26:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:26:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:26:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:26:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:27:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:59 smithi161 ceph-mon[91579]: pgmap v6837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:59 smithi188 ceph-mon[92425]: pgmap v6837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:01 smithi161 ceph-mon[91579]: pgmap v6838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:01 smithi188 ceph-mon[92425]: pgmap v6838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:03 smithi161 ceph-mon[91579]: pgmap v6839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:03 smithi188 ceph-mon[92425]: pgmap v6839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:05 smithi161 ceph-mon[91579]: pgmap v6840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:05 smithi188 ceph-mon[92425]: pgmap v6840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:07 smithi188 ceph-mon[92425]: pgmap v6841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:07 smithi161 ceph-mon[91579]: pgmap v6841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:09 smithi188 ceph-mon[92425]: pgmap v6842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:09 smithi161 ceph-mon[91579]: pgmap v6842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:11 smithi188 ceph-mon[92425]: pgmap v6843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:11 smithi161 ceph-mon[91579]: pgmap v6843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:13 smithi188 ceph-mon[92425]: pgmap v6844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:13 smithi161 ceph-mon[91579]: pgmap v6844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:15 smithi188 ceph-mon[92425]: pgmap v6845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:15 smithi161 ceph-mon[91579]: pgmap v6845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:17 smithi188 ceph-mon[92425]: pgmap v6846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:17 smithi161 ceph-mon[91579]: pgmap v6846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:19 smithi188 ceph-mon[92425]: pgmap v6847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:19 smithi161 ceph-mon[91579]: pgmap v6847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:21 smithi161 ceph-mon[91579]: pgmap v6848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:21 smithi188 ceph-mon[92425]: pgmap v6848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:23 smithi161 ceph-mon[91579]: pgmap v6849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:23 smithi188 ceph-mon[92425]: pgmap v6849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:25 smithi161 ceph-mon[91579]: pgmap v6850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:25 smithi188 ceph-mon[92425]: pgmap v6850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:27 smithi161 ceph-mon[91579]: pgmap v6851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:27 smithi188 ceph-mon[92425]: pgmap v6851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:29 smithi161 ceph-mon[91579]: pgmap v6852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:29 smithi188 ceph-mon[92425]: pgmap v6852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:30 smithi161 ceph-mon[91579]: pgmap v6853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:30 smithi188 ceph-mon[92425]: pgmap v6853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:33.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:27:33.117+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:27:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:27:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:27:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:27:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:27:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:27:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:27:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:27:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:27:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:27:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:27:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:27:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:27:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:27:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:27:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:27:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:27:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:27:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:27:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:27:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:27:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:27:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:27:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:27:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:27:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:27:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:27:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:27:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:27:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:27:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:27:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:27:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:27:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:27:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:27:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:27:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:27:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:27:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:27:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:27:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:27:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:27:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:27:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:27:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:27:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=365] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:27:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=365] Local address: 172.21.15.161, port 54792 2022-07-03T14:27:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=365] Peer address: 172.21.15.101, port 22 2022-07-03T14:27:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=365] Connection lost 2022-07-03T14:27:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=365] Aborting connection 2022-07-03T14:27:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:27:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:33 smithi188 ceph-mon[92425]: pgmap v6854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:27:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:27:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:27:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-mon[91579]: pgmap v6854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:27:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:27:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:27:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:27:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:27:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:27:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:27:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:27:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:27:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:27:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:27:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:27:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:27:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:27:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:27:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:27:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:27:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:27:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:27:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:27:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:27:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:27:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:27:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:27:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:27:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:27:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:27:35.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:27:35.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:27:35.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:27:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:27:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:27:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:27:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:27:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:27:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:27:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:27:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:27:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:27:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:27:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:27:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:27:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:27:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:27:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:27:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:27:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:27:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:27:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:27:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=365] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:27:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=365] Local address: 172.21.15.161, port 54792 2022-07-03T14:27:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=365] Peer address: 172.21.15.101, port 22 2022-07-03T14:27:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=365] Connection lost 2022-07-03T14:27:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=365] Aborting connection 2022-07-03T14:27:35.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:27:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:27:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:27:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:27:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:27:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:27:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:27:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:27:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:27:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:27:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:27:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:27:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:27:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:27:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:27:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:27:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:27:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:27:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:27:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:27:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:27:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:27:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:27:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:27:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:27:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:27:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:27:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:27:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:27:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:27:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:27:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:27:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:27:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:27:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:27:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:27:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:27:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:27:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:27:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:27:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:27:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:27:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:27:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:27:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:27:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:27:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=365] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:27:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=365] Local address: 172.21.15.161, port 54792 2022-07-03T14:27:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=365] Peer address: 172.21.15.101, port 22 2022-07-03T14:27:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=365] Connection lost 2022-07-03T14:27:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=365] Aborting connection 2022-07-03T14:27:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:35 smithi188 ceph-mon[92425]: pgmap v6855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:35 smithi161 ceph-mon[91579]: pgmap v6855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:37 smithi188 ceph-mon[92425]: pgmap v6856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:37 smithi161 ceph-mon[91579]: pgmap v6856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:39 smithi188 ceph-mon[92425]: pgmap v6857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:39 smithi161 ceph-mon[91579]: pgmap v6857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:41 smithi161 ceph-mon[91579]: pgmap v6858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:41 smithi188 ceph-mon[92425]: pgmap v6858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:43 smithi161 ceph-mon[91579]: pgmap v6859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:43 smithi188 ceph-mon[92425]: pgmap v6859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:45 smithi161 ceph-mon[91579]: pgmap v6860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:45 smithi188 ceph-mon[92425]: pgmap v6860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:47 smithi161 ceph-mon[91579]: pgmap v6861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:47 smithi188 ceph-mon[92425]: pgmap v6861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:48 smithi161 ceph-mon[91579]: pgmap v6862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:48 smithi188 ceph-mon[92425]: pgmap v6862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:51 smithi161 ceph-mon[91579]: pgmap v6863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:51 smithi188 ceph-mon[92425]: pgmap v6863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:53 smithi188 ceph-mon[92425]: pgmap v6864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:53 smithi161 ceph-mon[91579]: pgmap v6864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:27:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:27:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:55 smithi188 ceph-mon[92425]: pgmap v6865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:55 smithi161 ceph-mon[91579]: pgmap v6865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:27:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:57 smithi188 ceph-mon[92425]: pgmap v6866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:27:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:27:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:57 smithi161 ceph-mon[91579]: pgmap v6866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:27:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:27:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:27:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:27:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:28:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:59 smithi161 ceph-mon[91579]: pgmap v6867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:59 smithi188 ceph-mon[92425]: pgmap v6867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:01 smithi161 ceph-mon[91579]: pgmap v6868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:01 smithi188 ceph-mon[92425]: pgmap v6868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:03 smithi161 ceph-mon[91579]: pgmap v6869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:03 smithi188 ceph-mon[92425]: pgmap v6869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:05 smithi161 ceph-mon[91579]: pgmap v6870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:05 smithi188 ceph-mon[92425]: pgmap v6870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:07 smithi161 ceph-mon[91579]: pgmap v6871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:07 smithi188 ceph-mon[92425]: pgmap v6871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:09 smithi161 ceph-mon[91579]: pgmap v6872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:09 smithi188 ceph-mon[92425]: pgmap v6872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:11 smithi161 ceph-mon[91579]: pgmap v6873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:11 smithi188 ceph-mon[92425]: pgmap v6873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:12 smithi161 ceph-mon[91579]: pgmap v6874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:12 smithi188 ceph-mon[92425]: pgmap v6874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:15 smithi161 ceph-mon[91579]: pgmap v6875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:15 smithi188 ceph-mon[92425]: pgmap v6875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:17 smithi188 ceph-mon[92425]: pgmap v6876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:17 smithi161 ceph-mon[91579]: pgmap v6876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:19 smithi161 ceph-mon[91579]: pgmap v6877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:19 smithi188 ceph-mon[92425]: pgmap v6877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:21 smithi161 ceph-mon[91579]: pgmap v6878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:21 smithi188 ceph-mon[92425]: pgmap v6878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:23 smithi161 ceph-mon[91579]: pgmap v6879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:23 smithi188 ceph-mon[92425]: pgmap v6879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:25 smithi161 ceph-mon[91579]: pgmap v6880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:25 smithi188 ceph-mon[92425]: pgmap v6880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:27 smithi161 ceph-mon[91579]: pgmap v6881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:27 smithi188 ceph-mon[92425]: pgmap v6881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:29 smithi161 ceph-mon[91579]: pgmap v6882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:29 smithi188 ceph-mon[92425]: pgmap v6882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:31 smithi161 ceph-mon[91579]: pgmap v6883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:31 smithi188 ceph-mon[92425]: pgmap v6883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:32 smithi161 ceph-mon[91579]: pgmap v6884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:32 smithi188 ceph-mon[92425]: pgmap v6884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:33.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:28:33.620+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:28:33.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:28:33.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:28:33.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:28:33.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:28:33.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:28:33.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:28:33.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:28:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:28:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:28:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:28:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:28:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:28:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:28:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:28:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:28:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:28:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:28:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:28:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:28:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:28:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:28:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:28:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:28:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:28:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:28:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:28:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:28:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:28:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:28:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:28:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:28:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:28:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:28:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:28:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:28:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:28:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:28:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:28:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:28:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:28:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:28:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:28:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:28:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:28:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=367] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:28:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=367] Local address: 172.21.15.161, port 54820 2022-07-03T14:28:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=367] Peer address: 172.21.15.101, port 22 2022-07-03T14:28:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=367] Connection lost 2022-07-03T14:28:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=367] Aborting connection 2022-07-03T14:28:33.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:28:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:28:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:28:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:28:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:28:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:28:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:28:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:28:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:28:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:28:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:28:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:28:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:28:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:28:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:28:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:28:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:28:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:28:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:28:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:28:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:28:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:28:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:28:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:28:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:28:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:28:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:28:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:28:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:28:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:28:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:28:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:28:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:28:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:28:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:28:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:28:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:28:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:28:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:28:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:28:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:28:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:28:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:28:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:28:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:28:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:28:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:28:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:28:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:28:34.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:28:34.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=367] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:28:34.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=367] Local address: 172.21.15.161, port 54820 2022-07-03T14:28:34.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=367] Peer address: 172.21.15.101, port 22 2022-07-03T14:28:34.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=367] Connection lost 2022-07-03T14:28:34.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=367] Aborting connection 2022-07-03T14:28:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:28:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:28:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:28:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:28:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:28:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:28:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:28:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:28:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:28:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:28:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:28:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:28:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:28:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:28:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:28:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:28:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:28:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:28:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:28:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:28:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:28:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:28:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:28:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:28:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:28:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:28:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:28:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:28:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:28:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:28:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:28:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:28:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:28:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:28:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:28:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:28:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:28:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:28:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:28:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:28:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:28:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:28:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:28:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:28:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:28:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:28:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:28:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:28:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:28:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=367] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:28:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=367] Local address: 172.21.15.161, port 54820 2022-07-03T14:28:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=367] Peer address: 172.21.15.101, port 22 2022-07-03T14:28:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=367] Connection lost 2022-07-03T14:28:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=367] Aborting connection 2022-07-03T14:28:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:34 smithi161 ceph-mon[91579]: pgmap v6885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:34 smithi188 ceph-mon[92425]: pgmap v6885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:36 smithi161 ceph-mon[91579]: pgmap v6886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:36 smithi188 ceph-mon[92425]: pgmap v6886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:39 smithi161 ceph-mon[91579]: pgmap v6887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:39 smithi188 ceph-mon[92425]: pgmap v6887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:41 smithi161 ceph-mon[91579]: pgmap v6888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:41 smithi188 ceph-mon[92425]: pgmap v6888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:43 smithi161 ceph-mon[91579]: pgmap v6889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:43 smithi188 ceph-mon[92425]: pgmap v6889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:45 smithi161 ceph-mon[91579]: pgmap v6890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:45 smithi188 ceph-mon[92425]: pgmap v6890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:47 smithi161 ceph-mon[91579]: pgmap v6891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:47 smithi188 ceph-mon[92425]: pgmap v6891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:49 smithi161 ceph-mon[91579]: pgmap v6892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:49 smithi188 ceph-mon[92425]: pgmap v6892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:51 smithi161 ceph-mon[91579]: pgmap v6893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:52.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:51 smithi188 ceph-mon[92425]: pgmap v6893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:53 smithi161 ceph-mon[91579]: pgmap v6894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:53 smithi188 ceph-mon[92425]: pgmap v6894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:54 smithi161 ceph-mon[91579]: pgmap v6895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:28:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:54 smithi188 ceph-mon[92425]: pgmap v6895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:28:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:56 smithi161 ceph-mon[91579]: pgmap v6896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:56 smithi188 ceph-mon[92425]: pgmap v6896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:28:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:28:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:28:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:28:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:28:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:28:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:28:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:28:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:28:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:58 smithi161 ceph-mon[91579]: pgmap v6897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:28:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:58 smithi188 ceph-mon[92425]: pgmap v6897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:00 smithi161 ceph-mon[91579]: pgmap v6898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:00 smithi188 ceph-mon[92425]: pgmap v6898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:03 smithi161 ceph-mon[91579]: pgmap v6899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:03 smithi188 ceph-mon[92425]: pgmap v6899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:05 smithi161 ceph-mon[91579]: pgmap v6900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:05 smithi188 ceph-mon[92425]: pgmap v6900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:07 smithi161 ceph-mon[91579]: pgmap v6901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:07 smithi188 ceph-mon[92425]: pgmap v6901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:09 smithi161 ceph-mon[91579]: pgmap v6902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:09 smithi188 ceph-mon[92425]: pgmap v6902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:11 smithi161 ceph-mon[91579]: pgmap v6903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:11 smithi188 ceph-mon[92425]: pgmap v6903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:13 smithi161 ceph-mon[91579]: pgmap v6904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:13 smithi188 ceph-mon[92425]: pgmap v6904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:14 smithi161 ceph-mon[91579]: pgmap v6905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:14 smithi188 ceph-mon[92425]: pgmap v6905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:16 smithi161 ceph-mon[91579]: pgmap v6906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:16 smithi188 ceph-mon[92425]: pgmap v6906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:19 smithi161 ceph-mon[91579]: pgmap v6907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:19 smithi188 ceph-mon[92425]: pgmap v6907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:21 smithi161 ceph-mon[91579]: pgmap v6908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:21 smithi188 ceph-mon[92425]: pgmap v6908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:23 smithi161 ceph-mon[91579]: pgmap v6909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:23 smithi188 ceph-mon[92425]: pgmap v6909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:25 smithi161 ceph-mon[91579]: pgmap v6910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:25 smithi188 ceph-mon[92425]: pgmap v6910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:27 smithi161 ceph-mon[91579]: pgmap v6911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:27 smithi188 ceph-mon[92425]: pgmap v6911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:29 smithi161 ceph-mon[91579]: pgmap v6912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:29 smithi188 ceph-mon[92425]: pgmap v6912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:31 smithi161 ceph-mon[91579]: pgmap v6913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:31 smithi188 ceph-mon[92425]: pgmap v6913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:32 smithi161 ceph-mon[91579]: pgmap v6914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:32 smithi188 ceph-mon[92425]: pgmap v6914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:34.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:29:34.119+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:29:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:29:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:29:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:29:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:29:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:29:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:29:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:29:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:29:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:29:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:29:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:29:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:29:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:29:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:29:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:29:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:29:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:29:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:29:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:29:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:29:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:29:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:29:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:29:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:29:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:29:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:29:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:29:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:29:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:29:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:29:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:29:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:29:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:29:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:29:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:29:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:29:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:29:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:29:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:29:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:29:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:29:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:29:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:29:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:29:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=369] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:29:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=369] Local address: 172.21.15.161, port 54848 2022-07-03T14:29:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=369] Peer address: 172.21.15.101, port 22 2022-07-03T14:29:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=369] Connection lost 2022-07-03T14:29:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=369] Aborting connection 2022-07-03T14:29:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:29:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:29:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:29:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:29:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:29:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:29:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:29:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:29:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:29:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:29:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:29:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:29:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:29:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:29:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:29:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:29:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:29:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:29:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:29:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:29:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:29:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:29:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:29:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:29:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:29:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:29:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:29:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:29:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:29:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:29:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:29:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:29:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:29:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:29:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:29:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:29:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:29:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:29:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:29:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:29:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:29:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:29:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:29:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:29:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:29:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:29:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:29:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:29:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:29:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:29:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:29:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=369] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:29:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=369] Local address: 172.21.15.161, port 54848 2022-07-03T14:29:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=369] Peer address: 172.21.15.101, port 22 2022-07-03T14:29:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=369] Connection lost 2022-07-03T14:29:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=369] Aborting connection 2022-07-03T14:29:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:35 smithi161 ceph-mon[91579]: pgmap v6915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:29:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:29:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:29:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:29:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:29:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:29:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:29:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:29:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:29:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:29:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:29:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:29:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:29:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:29:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:29:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:29:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:29:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:29:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:29:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:29:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:29:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:29:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:29:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:29:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:29:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:29:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:29:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:29:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:29:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:29:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:29:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:29:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:29:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:29:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:29:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:29:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:29:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:29:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:29:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:29:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:29:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:29:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:29:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:29:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:29:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:29:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:29:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:29:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:29:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=369] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:29:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=369] Local address: 172.21.15.161, port 54848 2022-07-03T14:29:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=369] Peer address: 172.21.15.101, port 22 2022-07-03T14:29:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=369] Connection lost 2022-07-03T14:29:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=369] Aborting connection 2022-07-03T14:29:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:35 smithi188 ceph-mon[92425]: pgmap v6915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:37 smithi161 ceph-mon[91579]: pgmap v6916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:37 smithi188 ceph-mon[92425]: pgmap v6916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:39 smithi161 ceph-mon[91579]: pgmap v6917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:39 smithi188 ceph-mon[92425]: pgmap v6917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:41 smithi161 ceph-mon[91579]: pgmap v6918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:41 smithi188 ceph-mon[92425]: pgmap v6918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:43 smithi161 ceph-mon[91579]: pgmap v6919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:43 smithi188 ceph-mon[92425]: pgmap v6919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:45 smithi161 ceph-mon[91579]: pgmap v6920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:45 smithi188 ceph-mon[92425]: pgmap v6920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:47 smithi161 ceph-mon[91579]: pgmap v6921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:47 smithi188 ceph-mon[92425]: pgmap v6921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:48 smithi161 ceph-mon[91579]: pgmap v6922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:48 smithi188 ceph-mon[92425]: pgmap v6922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:50.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:50 smithi161 ceph-mon[91579]: pgmap v6923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:50 smithi188 ceph-mon[92425]: pgmap v6923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:53 smithi161 ceph-mon[91579]: pgmap v6924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:53 smithi188 ceph-mon[92425]: pgmap v6924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:29:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:29:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:55 smithi161 ceph-mon[91579]: pgmap v6925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:55 smithi188 ceph-mon[92425]: pgmap v6925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:29:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:57 smithi161 ceph-mon[91579]: pgmap v6926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:29:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:29:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:29:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:29:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:57 smithi188 ceph-mon[92425]: pgmap v6926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:29:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:29:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:29:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:29:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:30:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:59 smithi161 ceph-mon[91579]: pgmap v6927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:59 smithi188 ceph-mon[92425]: pgmap v6927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:30:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:30:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:01 smithi161 ceph-mon[91579]: pgmap v6928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:01 smithi188 ceph-mon[92425]: pgmap v6928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:03 smithi161 ceph-mon[91579]: pgmap v6929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:03 smithi188 ceph-mon[92425]: pgmap v6929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:05 smithi161 ceph-mon[91579]: pgmap v6930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:05 smithi188 ceph-mon[92425]: pgmap v6930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:07 smithi161 ceph-mon[91579]: pgmap v6931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:07 smithi188 ceph-mon[92425]: pgmap v6931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:09 smithi161 ceph-mon[91579]: pgmap v6932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:09 smithi188 ceph-mon[92425]: pgmap v6932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:11 smithi161 ceph-mon[91579]: pgmap v6933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:11 smithi188 ceph-mon[92425]: pgmap v6933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:13 smithi161 ceph-mon[91579]: pgmap v6934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:13 smithi188 ceph-mon[92425]: pgmap v6934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:15 smithi161 ceph-mon[91579]: pgmap v6935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:15 smithi188 ceph-mon[92425]: pgmap v6935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:16 smithi161 ceph-mon[91579]: pgmap v6936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:16 smithi188 ceph-mon[92425]: pgmap v6936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:19 smithi161 ceph-mon[91579]: pgmap v6937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:19 smithi188 ceph-mon[92425]: pgmap v6937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:21 smithi161 ceph-mon[91579]: pgmap v6938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:21 smithi188 ceph-mon[92425]: pgmap v6938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:23 smithi161 ceph-mon[91579]: pgmap v6939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:23 smithi188 ceph-mon[92425]: pgmap v6939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:24 smithi161 ceph-mon[91579]: pgmap v6940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:24 smithi188 ceph-mon[92425]: pgmap v6940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:26 smithi161 ceph-mon[91579]: pgmap v6941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:26 smithi188 ceph-mon[92425]: pgmap v6941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:29 smithi161 ceph-mon[91579]: pgmap v6942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:29 smithi188 ceph-mon[92425]: pgmap v6942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:31 smithi161 ceph-mon[91579]: pgmap v6943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:31 smithi188 ceph-mon[92425]: pgmap v6943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:33 smithi161 ceph-mon[91579]: pgmap v6944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:33 smithi188 ceph-mon[92425]: pgmap v6944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:34.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:30:34.618+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:30:34.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:30:34.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:30:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:30:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:30:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:30:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:30:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:30:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:30:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:30:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:30:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:30:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:30:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:30:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:30:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:30:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:30:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:30:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:30:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:30:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:30:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:30:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:30:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:30:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:30:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:30:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:30:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:30:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:30:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:30:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:30:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:30:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:30:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:30:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:30:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:30:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:30:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:30:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:30:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:30:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:30:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:30:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:30:34.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:30:34.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:30:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=371] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:30:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=371] Local address: 172.21.15.161, port 54876 2022-07-03T14:30:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=371] Peer address: 172.21.15.101, port 22 2022-07-03T14:30:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=371] Connection lost 2022-07-03T14:30:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=371] Aborting connection 2022-07-03T14:30:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:30:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:30:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:30:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:30:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:30:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:30:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:30:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:30:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:30:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:30:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:30:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:30:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:30:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:30:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:30:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:30:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:30:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:30:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:30:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:30:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:30:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:30:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:30:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:30:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:30:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:30:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:30:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:30:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:30:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:30:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:30:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:30:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:30:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:30:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:30:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:30:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:30:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:30:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:30:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:30:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:30:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:30:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:30:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:30:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:30:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:30:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:30:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:30:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:30:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:30:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:30:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=371] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:30:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=371] Local address: 172.21.15.161, port 54876 2022-07-03T14:30:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=371] Peer address: 172.21.15.101, port 22 2022-07-03T14:30:36.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=371] Connection lost 2022-07-03T14:30:36.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=371] Aborting connection 2022-07-03T14:30:36.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: pgmap v6945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:36.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.406 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:30:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:30:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:30:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:30:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:30:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:30:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:30:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:30:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:30:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:30:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:30:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:30:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:30:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:30:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:30:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:30:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:30:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:30:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:30:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:30:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:30:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:30:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:30:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:30:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:30:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:30:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:30:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:30:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:30:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:30:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:30:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:30:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:30:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:30:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:30:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:30:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:30:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:30:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:30:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:30:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:30:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:30:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:30:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:30:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=371] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:30:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=371] Local address: 172.21.15.161, port 54876 2022-07-03T14:30:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=371] Peer address: 172.21.15.101, port 22 2022-07-03T14:30:36.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=371] Connection lost 2022-07-03T14:30:36.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=371] Aborting connection 2022-07-03T14:30:36.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: pgmap v6945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:36.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:30:36.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:36 smithi161 ceph-mon[91579]: pgmap v6946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:36 smithi188 ceph-mon[92425]: pgmap v6946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:39 smithi161 ceph-mon[91579]: pgmap v6947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:39 smithi188 ceph-mon[92425]: pgmap v6947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:41 smithi161 ceph-mon[91579]: pgmap v6948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:41 smithi188 ceph-mon[92425]: pgmap v6948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:43 smithi161 ceph-mon[91579]: pgmap v6949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:43 smithi188 ceph-mon[92425]: pgmap v6949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:44 smithi161 ceph-mon[91579]: pgmap v6950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:44 smithi188 ceph-mon[92425]: pgmap v6950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:46 smithi161 ceph-mon[91579]: pgmap v6951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:46 smithi188 ceph-mon[92425]: pgmap v6951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:49 smithi161 ceph-mon[91579]: pgmap v6952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:49 smithi188 ceph-mon[92425]: pgmap v6952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:51 smithi161 ceph-mon[91579]: pgmap v6953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:52.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:51 smithi188 ceph-mon[92425]: pgmap v6953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:53 smithi161 ceph-mon[91579]: pgmap v6954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:53 smithi188 ceph-mon[92425]: pgmap v6954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:30:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:30:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:55 smithi161 ceph-mon[91579]: pgmap v6955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:55 smithi188 ceph-mon[92425]: pgmap v6955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:30:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:57 smithi161 ceph-mon[91579]: pgmap v6956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:30:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:30:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:30:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:30:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:57 smithi188 ceph-mon[92425]: pgmap v6956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:30:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:30:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:30:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:30:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:31:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:59 smithi161 ceph-mon[91579]: pgmap v6957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:59 smithi188 ceph-mon[92425]: pgmap v6957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:01 smithi161 ceph-mon[91579]: pgmap v6958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:01 smithi188 ceph-mon[92425]: pgmap v6958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:02 smithi161 ceph-mon[91579]: pgmap v6959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:02 smithi188 ceph-mon[92425]: pgmap v6959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:04.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:05 smithi161 ceph-mon[91579]: pgmap v6960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:05 smithi188 ceph-mon[92425]: pgmap v6960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:07 smithi161 ceph-mon[91579]: pgmap v6961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:07 smithi188 ceph-mon[92425]: pgmap v6961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:09 smithi161 ceph-mon[91579]: pgmap v6962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:09 smithi188 ceph-mon[92425]: pgmap v6962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:10 smithi161 ceph-mon[91579]: pgmap v6963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:10 smithi188 ceph-mon[92425]: pgmap v6963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:13 smithi161 ceph-mon[91579]: pgmap v6964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:13 smithi188 ceph-mon[92425]: pgmap v6964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:15 smithi161 ceph-mon[91579]: pgmap v6965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:15 smithi188 ceph-mon[92425]: pgmap v6965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:16 smithi161 ceph-mon[91579]: pgmap v6966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:16 smithi188 ceph-mon[92425]: pgmap v6966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:19 smithi161 ceph-mon[91579]: pgmap v6967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:19 smithi188 ceph-mon[92425]: pgmap v6967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:21 smithi161 ceph-mon[91579]: pgmap v6968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:21 smithi188 ceph-mon[92425]: pgmap v6968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:22 smithi161 ceph-mon[91579]: pgmap v6969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:22 smithi188 ceph-mon[92425]: pgmap v6969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:25 smithi161 ceph-mon[91579]: pgmap v6970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:25 smithi188 ceph-mon[92425]: pgmap v6970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:27 smithi161 ceph-mon[91579]: pgmap v6971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:27 smithi188 ceph-mon[92425]: pgmap v6971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:31:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:29 smithi161 ceph-mon[91579]: pgmap v6972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:29 smithi188 ceph-mon[92425]: pgmap v6972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:31 smithi161 ceph-mon[91579]: pgmap v6973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:31 smithi188 ceph-mon[92425]: pgmap v6973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:33 smithi161 ceph-mon[91579]: pgmap v6974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:33 smithi188 ceph-mon[92425]: pgmap v6974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:34 smithi161 ceph-mon[91579]: pgmap v6975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:34 smithi188 ceph-mon[92425]: pgmap v6975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:35.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:31:35.564+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:31:35.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:31:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:31:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:31:35.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:31:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:31:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:31:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:31:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:31:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:31:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:31:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:31:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:31:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:31:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:31:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:31:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:31:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:31:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:31:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:31:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:31:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:31:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:31:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:31:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:31:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:31:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:31:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:31:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:31:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:31:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:31:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:31:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:31:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:31:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:31:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:31:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:31:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:31:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:31:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:31:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:31:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:31:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:31:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:31:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:31:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=373] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:31:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=373] Local address: 172.21.15.161, port 54906 2022-07-03T14:31:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=373] Peer address: 172.21.15.101, port 22 2022-07-03T14:31:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=373] Connection lost 2022-07-03T14:31:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=373] Aborting connection 2022-07-03T14:31:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:31:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:31:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:31:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:31:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:31:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:31:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:31:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:31:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:31:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:31:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:31:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:31:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:31:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:31:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:31:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:31:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:31:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:31:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:31:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:31:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:31:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:31:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:31:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:31:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:31:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:31:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:31:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:31:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:31:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:31:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:31:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:31:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:31:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:31:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:31:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:31:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:31:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:31:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:31:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:31:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:31:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:31:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:31:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:31:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:31:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:31:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:31:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:31:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:31:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:31:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=373] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:31:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=373] Local address: 172.21.15.161, port 54906 2022-07-03T14:31:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=373] Peer address: 172.21.15.101, port 22 2022-07-03T14:31:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=373] Connection lost 2022-07-03T14:31:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=373] Aborting connection 2022-07-03T14:31:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:31:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:31:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:31:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:31:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:31:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:31:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:31:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:31:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:31:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:31:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:31:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:31:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:31:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:31:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:31:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:31:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:31:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:31:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:31:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:31:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:31:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:31:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:31:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:31:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:31:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:31:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:31:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:31:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:31:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:31:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:31:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:31:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:31:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:31:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:31:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:31:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:31:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:31:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:31:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:31:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:31:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:31:36.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:31:36.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:31:36.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:31:36.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:31:36.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:31:36.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:31:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:31:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=373] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:31:36.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=373] Local address: 172.21.15.161, port 54906 2022-07-03T14:31:36.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=373] Peer address: 172.21.15.101, port 22 2022-07-03T14:31:36.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=373] Connection lost 2022-07-03T14:31:36.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=373] Aborting connection 2022-07-03T14:31:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:36 smithi161 ceph-mon[91579]: pgmap v6976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:36 smithi188 ceph-mon[92425]: pgmap v6976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:39 smithi161 ceph-mon[91579]: pgmap v6977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:39 smithi188 ceph-mon[92425]: pgmap v6977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:40 smithi161 ceph-mon[91579]: pgmap v6978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:40 smithi188 ceph-mon[92425]: pgmap v6978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:43 smithi161 ceph-mon[91579]: pgmap v6979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:43 smithi188 ceph-mon[92425]: pgmap v6979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:45 smithi161 ceph-mon[91579]: pgmap v6980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:45 smithi188 ceph-mon[92425]: pgmap v6980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:47 smithi161 ceph-mon[91579]: pgmap v6981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:47 smithi188 ceph-mon[92425]: pgmap v6981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:49 smithi161 ceph-mon[91579]: pgmap v6982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:49 smithi188 ceph-mon[92425]: pgmap v6982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:51 smithi161 ceph-mon[91579]: pgmap v6983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:51 smithi188 ceph-mon[92425]: pgmap v6983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:53 smithi161 ceph-mon[91579]: pgmap v6984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:53 smithi188 ceph-mon[92425]: pgmap v6984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:31:56.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:55 smithi161 ceph-mon[91579]: pgmap v6985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:56.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:55 smithi188 ceph-mon[92425]: pgmap v6985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:31:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:57 smithi161 ceph-mon[91579]: pgmap v6986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:57 smithi188 ceph-mon[92425]: pgmap v6986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:31:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:31:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:31:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:31:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:31:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:31:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:31:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:31:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:31:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:31:59 smithi161 ceph-mon[91579]: pgmap v6987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:31:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:31:59 smithi188 ceph-mon[92425]: pgmap v6987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:01 smithi161 ceph-mon[91579]: pgmap v6988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:01 smithi188 ceph-mon[92425]: pgmap v6988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:03 smithi161 ceph-mon[91579]: pgmap v6989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:03 smithi188 ceph-mon[92425]: pgmap v6989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:05 smithi161 ceph-mon[91579]: pgmap v6990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:05 smithi188 ceph-mon[92425]: pgmap v6990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:07 smithi161 ceph-mon[91579]: pgmap v6991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:07 smithi188 ceph-mon[92425]: pgmap v6991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:09 smithi161 ceph-mon[91579]: pgmap v6992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:09 smithi188 ceph-mon[92425]: pgmap v6992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:11 smithi161 ceph-mon[91579]: pgmap v6993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:11 smithi188 ceph-mon[92425]: pgmap v6993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:13 smithi161 ceph-mon[91579]: pgmap v6994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:13 smithi188 ceph-mon[92425]: pgmap v6994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:15 smithi161 ceph-mon[91579]: pgmap v6995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:15 smithi188 ceph-mon[92425]: pgmap v6995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:17 smithi161 ceph-mon[91579]: pgmap v6996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:17 smithi188 ceph-mon[92425]: pgmap v6996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:19 smithi161 ceph-mon[91579]: pgmap v6997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:19 smithi188 ceph-mon[92425]: pgmap v6997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:21 smithi161 ceph-mon[91579]: pgmap v6998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:21 smithi188 ceph-mon[92425]: pgmap v6998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:23 smithi161 ceph-mon[91579]: pgmap v6999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:23 smithi188 ceph-mon[92425]: pgmap v6999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:25 smithi161 ceph-mon[91579]: pgmap v7000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:25 smithi188 ceph-mon[92425]: pgmap v7000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:27 smithi161 ceph-mon[91579]: pgmap v7001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:27 smithi188 ceph-mon[92425]: pgmap v7001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:29 smithi161 ceph-mon[91579]: pgmap v7002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:29 smithi188 ceph-mon[92425]: pgmap v7002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:31 smithi161 ceph-mon[91579]: pgmap v7003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:31 smithi188 ceph-mon[92425]: pgmap v7003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:33 smithi161 ceph-mon[91579]: pgmap v7004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:33 smithi188 ceph-mon[92425]: pgmap v7004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:36.068 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:35 smithi161 ceph-mon[91579]: pgmap v7005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:36.069 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:36.087 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:35 smithi188 ceph-mon[92425]: pgmap v7005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:36.088 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:32:36.067+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:32:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:32:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:32:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:32:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:32:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:32:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:32:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:32:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:32:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:32:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:32:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:32:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:32:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:32:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:32:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:32:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:32:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:32:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:32:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:32:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:32:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:32:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:32:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:32:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:32:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:32:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:32:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:32:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:32:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:32:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:32:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:32:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:32:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:32:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:32:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:32:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:32:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:32:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:32:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:32:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:32:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:32:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:32:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:32:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:32:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=375] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:32:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=375] Local address: 172.21.15.161, port 54934 2022-07-03T14:32:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=375] Peer address: 172.21.15.101, port 22 2022-07-03T14:32:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=375] Connection lost 2022-07-03T14:32:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=375] Aborting connection 2022-07-03T14:32:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:32:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:32:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:32:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:32:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:32:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:32:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:32:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:32:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:32:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:32:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:32:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:32:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:32:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:32:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:32:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:32:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:32:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:32:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:32:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:32:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:32:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:32:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:32:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:32:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:32:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:32:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:32:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:32:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:32:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:32:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:32:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:32:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:32:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:32:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:32:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:32:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:32:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:32:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:32:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:32:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:32:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:32:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:32:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:32:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:32:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:32:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:32:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:32:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:32:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:32:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:32:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:32:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:32:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=375] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:32:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=375] Local address: 172.21.15.161, port 54934 2022-07-03T14:32:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=375] Peer address: 172.21.15.101, port 22 2022-07-03T14:32:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=375] Connection lost 2022-07-03T14:32:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=375] Aborting connection 2022-07-03T14:32:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:37 smithi161 ceph-mon[91579]: pgmap v7006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:32:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:32:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:32:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:32:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:32:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:32:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:32:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:32:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:32:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:32:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:32:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:32:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:32:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:32:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:32:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:32:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:32:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:32:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:32:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:32:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:32:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:32:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:32:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:32:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:32:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:32:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:32:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:32:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:32:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:32:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:32:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:32:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:32:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:32:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:32:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:32:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:32:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:32:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:32:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:32:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:32:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:32:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:32:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:32:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:32:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:32:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=375] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:32:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=375] Local address: 172.21.15.161, port 54934 2022-07-03T14:32:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=375] Peer address: 172.21.15.101, port 22 2022-07-03T14:32:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=375] Connection lost 2022-07-03T14:32:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=375] Aborting connection 2022-07-03T14:32:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:37 smithi188 ceph-mon[92425]: pgmap v7006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:39 smithi161 ceph-mon[91579]: pgmap v7007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:39 smithi188 ceph-mon[92425]: pgmap v7007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:41 smithi161 ceph-mon[91579]: pgmap v7008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:41 smithi188 ceph-mon[92425]: pgmap v7008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:43 smithi161 ceph-mon[91579]: pgmap v7009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:43 smithi188 ceph-mon[92425]: pgmap v7009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:45 smithi161 ceph-mon[91579]: pgmap v7010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:45 smithi188 ceph-mon[92425]: pgmap v7010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:47 smithi161 ceph-mon[91579]: pgmap v7011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:47 smithi188 ceph-mon[92425]: pgmap v7011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:49 smithi161 ceph-mon[91579]: pgmap v7012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:49 smithi188 ceph-mon[92425]: pgmap v7012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:51 smithi161 ceph-mon[91579]: pgmap v7013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:51 smithi188 ceph-mon[92425]: pgmap v7013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:53 smithi161 ceph-mon[91579]: pgmap v7014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:53 smithi188 ceph-mon[92425]: pgmap v7014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:32:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:55 smithi161 ceph-mon[91579]: pgmap v7015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:55 smithi188 ceph-mon[92425]: pgmap v7015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:32:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:57 smithi161 ceph-mon[91579]: pgmap v7016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:57 smithi188 ceph-mon[92425]: pgmap v7016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:32:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:32:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:32:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:32:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:32:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:32:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:32:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:32:59 smithi161 ceph-mon[91579]: pgmap v7017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:32:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:32:59 smithi188 ceph-mon[92425]: pgmap v7017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:01 smithi161 ceph-mon[91579]: pgmap v7018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:01 smithi188 ceph-mon[92425]: pgmap v7018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:03 smithi161 ceph-mon[91579]: pgmap v7019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:03 smithi188 ceph-mon[92425]: pgmap v7019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:04.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:05 smithi161 ceph-mon[91579]: pgmap v7020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:05 smithi188 ceph-mon[92425]: pgmap v7020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:07 smithi161 ceph-mon[91579]: pgmap v7021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:07 smithi188 ceph-mon[92425]: pgmap v7021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:09 smithi161 ceph-mon[91579]: pgmap v7022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:09 smithi188 ceph-mon[92425]: pgmap v7022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:11 smithi161 ceph-mon[91579]: pgmap v7023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:11 smithi188 ceph-mon[92425]: pgmap v7023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:13 smithi161 ceph-mon[91579]: pgmap v7024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:13 smithi188 ceph-mon[92425]: pgmap v7024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:15 smithi161 ceph-mon[91579]: pgmap v7025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:15 smithi188 ceph-mon[92425]: pgmap v7025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:17 smithi161 ceph-mon[91579]: pgmap v7026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:17 smithi188 ceph-mon[92425]: pgmap v7026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:19 smithi161 ceph-mon[91579]: pgmap v7027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:19 smithi188 ceph-mon[92425]: pgmap v7027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:21 smithi161 ceph-mon[91579]: pgmap v7028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:21 smithi188 ceph-mon[92425]: pgmap v7028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:23 smithi161 ceph-mon[91579]: pgmap v7029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:23 smithi188 ceph-mon[92425]: pgmap v7029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:25.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:33:25.178+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T14:33:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:25 smithi161 ceph-mon[91579]: pgmap v7030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:25 smithi188 ceph-mon[92425]: pgmap v7030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:33:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:27 smithi161 ceph-mon[91579]: pgmap v7031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:27 smithi188 ceph-mon[92425]: pgmap v7031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:29 smithi161 ceph-mon[91579]: pgmap v7032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:29 smithi188 ceph-mon[92425]: pgmap v7032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:31 smithi161 ceph-mon[91579]: pgmap v7033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:31 smithi188 ceph-mon[92425]: pgmap v7033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:33 smithi161 ceph-mon[91579]: pgmap v7034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:33 smithi188 ceph-mon[92425]: pgmap v7034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:35 smithi161 ceph-mon[91579]: pgmap v7035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:35 smithi188 ceph-mon[92425]: pgmap v7035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:33:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:33:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:33:36.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:33:36.579+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:33:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:33:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:33:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:33:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:33:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:33:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:33:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:33:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:33:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:33:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:33:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:33:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:33:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:33:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:33:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:33:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:33:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:33:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:33:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:33:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:33:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:33:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:33:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:33:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:33:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:33:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:33:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:33:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:33:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:33:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:33:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:33:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:33:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:33:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:33:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:33:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:33:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:33:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:33:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:33:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:33:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:33:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:33:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:33:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:33:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=377] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:33:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=377] Local address: 172.21.15.161, port 54958 2022-07-03T14:33:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=377] Peer address: 172.21.15.101, port 22 2022-07-03T14:33:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=377] Connection lost 2022-07-03T14:33:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=377] Aborting connection 2022-07-03T14:33:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:33:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:33:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:33:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:33:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:33:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:33:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:33:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:33:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:33:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:33:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:33:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:33:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:33:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:33:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:33:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:33:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:33:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:33:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:33:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:33:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:33:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:33:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:33:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:33:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:33:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:33:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:33:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:33:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:33:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:33:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:33:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:33:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:33:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=377] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:33:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=377] Local address: 172.21.15.161, port 54958 2022-07-03T14:33:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=377] Peer address: 172.21.15.101, port 22 2022-07-03T14:33:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=377] Connection lost 2022-07-03T14:33:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=377] Aborting connection 2022-07-03T14:33:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:37 smithi161 ceph-mon[91579]: pgmap v7036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:33:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:33:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:33:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:33:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:33:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:33:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:33:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:33:38.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:33:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:33:38.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:33:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:33:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:33:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:33:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:33:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:33:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:33:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:33:38.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:33:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:33:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:33:38.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:33:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:33:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:33:38.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:33:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:33:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:33:38.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:33:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:33:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:33:38.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:33:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:33:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:33:38.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:33:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:33:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:33:38.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:33:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:33:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:33:38.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:33:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:33:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:33:38.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:33:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:33:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:33:38.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:33:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=377] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:33:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=377] Local address: 172.21.15.161, port 54958 2022-07-03T14:33:38.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=377] Peer address: 172.21.15.101, port 22 2022-07-03T14:33:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=377] Connection lost 2022-07-03T14:33:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=377] Aborting connection 2022-07-03T14:33:38.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:37 smithi188 ceph-mon[92425]: pgmap v7036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:39.346 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:39 smithi188 ceph-mon[92425]: pgmap v7037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:39 smithi161 ceph-mon[91579]: pgmap v7037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:33:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:33:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:33:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:33:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:33:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:33:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:33:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:33:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:33:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:33:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:41 smithi161 ceph-mon[91579]: pgmap v7038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:41 smithi188 ceph-mon[92425]: pgmap v7038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:43 smithi161 ceph-mon[91579]: pgmap v7039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:43 smithi188 ceph-mon[92425]: pgmap v7039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:45 smithi161 ceph-mon[91579]: pgmap v7040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:45 smithi188 ceph-mon[92425]: pgmap v7040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:47 smithi161 ceph-mon[91579]: pgmap v7041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:47 smithi188 ceph-mon[92425]: pgmap v7041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:49 smithi161 ceph-mon[91579]: pgmap v7042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:49 smithi188 ceph-mon[92425]: pgmap v7042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:51 smithi161 ceph-mon[91579]: pgmap v7043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:51 smithi188 ceph-mon[92425]: pgmap v7043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:53 smithi161 ceph-mon[91579]: pgmap v7044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:53 smithi188 ceph-mon[92425]: pgmap v7044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:33:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:55 smithi161 ceph-mon[91579]: pgmap v7045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:55 smithi188 ceph-mon[92425]: pgmap v7045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:57 smithi161 ceph-mon[91579]: pgmap v7046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:57 smithi188 ceph-mon[92425]: pgmap v7046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:33:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:33:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:33:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:33:59 smithi161 ceph-mon[91579]: pgmap v7047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:33:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:33:59 smithi188 ceph-mon[92425]: pgmap v7047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:01 smithi161 ceph-mon[91579]: pgmap v7048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:01 smithi188 ceph-mon[92425]: pgmap v7048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:03 smithi161 ceph-mon[91579]: pgmap v7049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:03 smithi188 ceph-mon[92425]: pgmap v7049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:05 smithi161 ceph-mon[91579]: pgmap v7050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:05 smithi188 ceph-mon[92425]: pgmap v7050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:07 smithi161 ceph-mon[91579]: pgmap v7051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:07 smithi188 ceph-mon[92425]: pgmap v7051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:09 smithi161 ceph-mon[91579]: pgmap v7052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:09 smithi188 ceph-mon[92425]: pgmap v7052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:11 smithi161 ceph-mon[91579]: pgmap v7053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:11 smithi188 ceph-mon[92425]: pgmap v7053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:13 smithi161 ceph-mon[91579]: pgmap v7054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:13 smithi188 ceph-mon[92425]: pgmap v7054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:15 smithi161 ceph-mon[91579]: pgmap v7055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:15 smithi188 ceph-mon[92425]: pgmap v7055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:17 smithi161 ceph-mon[91579]: pgmap v7056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:17 smithi188 ceph-mon[92425]: pgmap v7056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:19 smithi161 ceph-mon[91579]: pgmap v7057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:19 smithi188 ceph-mon[92425]: pgmap v7057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:21 smithi161 ceph-mon[91579]: pgmap v7058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:21 smithi188 ceph-mon[92425]: pgmap v7058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:23 smithi161 ceph-mon[91579]: pgmap v7059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:23 smithi188 ceph-mon[92425]: pgmap v7059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:25 smithi161 ceph-mon[91579]: pgmap v7060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:25 smithi188 ceph-mon[92425]: pgmap v7060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:27 smithi161 ceph-mon[91579]: pgmap v7061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:27 smithi188 ceph-mon[92425]: pgmap v7061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:29 smithi161 ceph-mon[91579]: pgmap v7062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:29 smithi188 ceph-mon[92425]: pgmap v7062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:31 smithi161 ceph-mon[91579]: pgmap v7063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:31 smithi188 ceph-mon[92425]: pgmap v7063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:33 smithi161 ceph-mon[91579]: pgmap v7064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:33 smithi188 ceph-mon[92425]: pgmap v7064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:35 smithi161 ceph-mon[91579]: pgmap v7065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:35 smithi188 ceph-mon[92425]: pgmap v7065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:37 smithi161 ceph-mon[91579]: pgmap v7066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:37 smithi188 ceph-mon[92425]: pgmap v7066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-mon[91579]: pgmap v7067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:39 smithi188 ceph-mon[92425]: pgmap v7067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:40.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:34:39.736+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:34:40.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:34:40.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:34:40.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:34:40.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:34:40.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:34:40.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:34:40.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:34:40.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:34:40.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:34:40.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:34:40.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:34:40.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:34:40.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:34:40.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:34:40.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:34:40.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:34:40.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:34:40.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:34:40.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:34:40.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:34:40.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:34:40.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:34:40.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:34:40.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:34:40.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:34:40.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:34:40.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:34:40.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:34:40.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:34:40.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:34:40.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:34:40.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:34:40.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:34:40.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:34:40.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:34:40.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:34:40.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:34:40.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:34:40.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:34:40.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:34:40.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:34:40.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:34:40.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:34:40.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:34:40.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=379] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:34:40.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=379] Local address: 172.21.15.161, port 54962 2022-07-03T14:34:40.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=379] Peer address: 172.21.15.101, port 22 2022-07-03T14:34:40.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=379] Connection lost 2022-07-03T14:34:40.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=379] Aborting connection 2022-07-03T14:34:40.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:34:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:34:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:34:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:34:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:34:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:34:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:34:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:34:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:34:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:34:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:34:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:34:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:34:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:34:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:34:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:34:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:34:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:34:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:34:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:34:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:34:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:34:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:34:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:34:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:34:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:34:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:34:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:34:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:34:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:34:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:34:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:34:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:34:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:34:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:34:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:34:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:34:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:34:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:34:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:34:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:34:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:34:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:34:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:34:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:34:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:34:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:34:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:34:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:34:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:34:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:34:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=379] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:34:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=379] Local address: 172.21.15.161, port 54962 2022-07-03T14:34:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=379] Peer address: 172.21.15.101, port 22 2022-07-03T14:34:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=379] Connection lost 2022-07-03T14:34:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=379] Aborting connection 2022-07-03T14:34:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:34:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:34:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:34:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:34:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:34:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:34:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:34:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:34:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:34:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:34:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:34:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:34:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:34:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:34:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:34:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:34:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:34:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:34:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:34:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:34:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:34:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:34:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:34:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:34:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:34:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:34:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:34:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:34:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:34:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:34:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:34:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:34:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:34:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:34:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:34:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:34:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:34:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:34:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:34:40.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:34:40.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:34:40.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:34:40.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:34:40.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:34:40.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:34:40.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:34:40.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:34:40.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:34:40.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:34:40.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=379] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:34:40.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=379] Local address: 172.21.15.161, port 54962 2022-07-03T14:34:40.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=379] Peer address: 172.21.15.101, port 22 2022-07-03T14:34:40.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=379] Connection lost 2022-07-03T14:34:40.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=379] Aborting connection 2022-07-03T14:34:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:41 smithi161 ceph-mon[91579]: pgmap v7068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:41 smithi188 ceph-mon[92425]: pgmap v7068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:43 smithi161 ceph-mon[91579]: pgmap v7069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:43 smithi188 ceph-mon[92425]: pgmap v7069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:45 smithi161 ceph-mon[91579]: pgmap v7070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:45 smithi188 ceph-mon[92425]: pgmap v7070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:47 smithi161 ceph-mon[91579]: pgmap v7071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:47 smithi188 ceph-mon[92425]: pgmap v7071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:49 smithi161 ceph-mon[91579]: pgmap v7072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:49 smithi188 ceph-mon[92425]: pgmap v7072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:51 smithi161 ceph-mon[91579]: pgmap v7073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:51 smithi188 ceph-mon[92425]: pgmap v7073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:53 smithi161 ceph-mon[91579]: pgmap v7074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:53 smithi188 ceph-mon[92425]: pgmap v7074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:34:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:55 smithi161 ceph-mon[91579]: pgmap v7075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:55 smithi188 ceph-mon[92425]: pgmap v7075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:57 smithi161 ceph-mon[91579]: pgmap v7076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:57 smithi188 ceph-mon[92425]: pgmap v7076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:34:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:34:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:34:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:34:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:34:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:34:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:34:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:34:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:34:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:34:59 smithi161 ceph-mon[91579]: pgmap v7077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:34:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:34:59 smithi188 ceph-mon[92425]: pgmap v7077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:01 smithi161 ceph-mon[91579]: pgmap v7078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:01 smithi188 ceph-mon[92425]: pgmap v7078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:03 smithi161 ceph-mon[91579]: pgmap v7079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:03 smithi188 ceph-mon[92425]: pgmap v7079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:05 smithi161 ceph-mon[91579]: pgmap v7080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:05 smithi188 ceph-mon[92425]: pgmap v7080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:07 smithi161 ceph-mon[91579]: pgmap v7081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:07 smithi188 ceph-mon[92425]: pgmap v7081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:09 smithi161 ceph-mon[91579]: pgmap v7082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:09 smithi188 ceph-mon[92425]: pgmap v7082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:11 smithi161 ceph-mon[91579]: pgmap v7083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:11 smithi188 ceph-mon[92425]: pgmap v7083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:13 smithi161 ceph-mon[91579]: pgmap v7084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:13 smithi188 ceph-mon[92425]: pgmap v7084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:15 smithi161 ceph-mon[91579]: pgmap v7085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:15 smithi188 ceph-mon[92425]: pgmap v7085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:17 smithi161 ceph-mon[91579]: pgmap v7086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:17 smithi188 ceph-mon[92425]: pgmap v7086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:19 smithi161 ceph-mon[91579]: pgmap v7087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:19 smithi188 ceph-mon[92425]: pgmap v7087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:21 smithi161 ceph-mon[91579]: pgmap v7088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:21 smithi188 ceph-mon[92425]: pgmap v7088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:23 smithi161 ceph-mon[91579]: pgmap v7089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:23 smithi188 ceph-mon[92425]: pgmap v7089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:25 smithi161 ceph-mon[91579]: pgmap v7090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:25 smithi188 ceph-mon[92425]: pgmap v7090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:27 smithi161 ceph-mon[91579]: pgmap v7091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:27 smithi188 ceph-mon[92425]: pgmap v7091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:29 smithi161 ceph-mon[91579]: pgmap v7092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:29 smithi188 ceph-mon[92425]: pgmap v7092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:31 smithi161 ceph-mon[91579]: pgmap v7093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:31 smithi188 ceph-mon[92425]: pgmap v7093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:33 smithi161 ceph-mon[91579]: pgmap v7094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:33 smithi188 ceph-mon[92425]: pgmap v7094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:35 smithi161 ceph-mon[91579]: pgmap v7095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:35 smithi188 ceph-mon[92425]: pgmap v7095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:37 smithi161 ceph-mon[91579]: pgmap v7096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:37 smithi188 ceph-mon[92425]: pgmap v7096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:39 smithi161 ceph-mon[91579]: pgmap v7097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:39 smithi188 ceph-mon[92425]: pgmap v7097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:40.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:35:40.235+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:35:40.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:35:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:35:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:35:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:35:40.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:35:40.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:35:40.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:35:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:35:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:35:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:35:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:35:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:35:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:35:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:35:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:35:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:35:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:35:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:35:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:35:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:35:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:35:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:35:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:35:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:35:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:35:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:35:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:35:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:35:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:35:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:35:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:35:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:35:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:35:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:35:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:35:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:35:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:35:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:35:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:35:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:35:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:35:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:35:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:35:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:35:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=381] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:35:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=381] Local address: 172.21.15.161, port 54966 2022-07-03T14:35:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=381] Peer address: 172.21.15.101, port 22 2022-07-03T14:35:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=381] Connection lost 2022-07-03T14:35:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=381] Aborting connection 2022-07-03T14:35:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:35:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:35:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:35:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:35:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:35:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:35:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:35:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:35:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:35:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:35:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:35:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:35:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:35:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:35:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:35:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:35:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:35:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:35:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:35:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:35:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:35:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:35:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:35:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:35:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:35:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:35:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:35:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:35:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:35:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:35:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:35:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:35:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:35:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:35:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:35:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:35:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:35:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:35:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:35:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:35:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:35:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:35:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:35:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:35:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:35:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:35:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:35:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:35:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:35:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:35:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:35:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=381] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:35:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=381] Local address: 172.21.15.161, port 54966 2022-07-03T14:35:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=381] Peer address: 172.21.15.101, port 22 2022-07-03T14:35:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=381] Connection lost 2022-07-03T14:35:41.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=381] Aborting connection 2022-07-03T14:35:41.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:41 smithi161 ceph-mon[91579]: pgmap v7098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:35:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:35:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:35:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:35:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:35:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:35:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:35:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:35:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:35:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:35:41.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:35:41.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:35:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:35:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:35:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:35:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:35:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:35:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:35:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:35:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:35:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:35:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:35:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:35:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:35:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:35:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:35:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:35:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:35:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:35:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:35:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:35:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:35:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:35:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:35:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:35:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:35:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:35:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:35:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:35:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:35:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:35:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:35:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:35:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:35:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:35:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:35:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:35:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:35:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:35:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=381] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:35:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=381] Local address: 172.21.15.161, port 54966 2022-07-03T14:35:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=381] Peer address: 172.21.15.101, port 22 2022-07-03T14:35:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=381] Connection lost 2022-07-03T14:35:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=381] Aborting connection 2022-07-03T14:35:41.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:41 smithi188 ceph-mon[92425]: pgmap v7098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:43 smithi161 ceph-mon[91579]: pgmap v7099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:43 smithi188 ceph-mon[92425]: pgmap v7099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:45 smithi161 ceph-mon[91579]: pgmap v7100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:45 smithi188 ceph-mon[92425]: pgmap v7100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:47 smithi161 ceph-mon[91579]: pgmap v7101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:47 smithi188 ceph-mon[92425]: pgmap v7101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:49 smithi161 ceph-mon[91579]: pgmap v7102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:49 smithi188 ceph-mon[92425]: pgmap v7102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:51 smithi161 ceph-mon[91579]: pgmap v7103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:51 smithi188 ceph-mon[92425]: pgmap v7103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:53 smithi161 ceph-mon[91579]: pgmap v7104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:53 smithi188 ceph-mon[92425]: pgmap v7104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:35:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:55 smithi161 ceph-mon[91579]: pgmap v7105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:55 smithi188 ceph-mon[92425]: pgmap v7105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:57 smithi161 ceph-mon[91579]: pgmap v7106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:57 smithi188 ceph-mon[92425]: pgmap v7106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:35:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:35:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:35:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:35:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:35:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:35:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:35:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:35:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:35:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:35:59 smithi161 ceph-mon[91579]: pgmap v7107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:35:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:35:59 smithi188 ceph-mon[92425]: pgmap v7107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:01 smithi161 ceph-mon[91579]: pgmap v7108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:01 smithi188 ceph-mon[92425]: pgmap v7108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:03 smithi161 ceph-mon[91579]: pgmap v7109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:03 smithi188 ceph-mon[92425]: pgmap v7109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:05 smithi161 ceph-mon[91579]: pgmap v7110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:05 smithi188 ceph-mon[92425]: pgmap v7110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:07 smithi161 ceph-mon[91579]: pgmap v7111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:07 smithi188 ceph-mon[92425]: pgmap v7111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:09 smithi161 ceph-mon[91579]: pgmap v7112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:09 smithi188 ceph-mon[92425]: pgmap v7112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:11 smithi161 ceph-mon[91579]: pgmap v7113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:11 smithi188 ceph-mon[92425]: pgmap v7113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:13 smithi161 ceph-mon[91579]: pgmap v7114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:13 smithi188 ceph-mon[92425]: pgmap v7114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:15 smithi161 ceph-mon[91579]: pgmap v7115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:15 smithi188 ceph-mon[92425]: pgmap v7115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:17 smithi161 ceph-mon[91579]: pgmap v7116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:17 smithi188 ceph-mon[92425]: pgmap v7116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:19 smithi161 ceph-mon[91579]: pgmap v7117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:19 smithi188 ceph-mon[92425]: pgmap v7117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:21 smithi161 ceph-mon[91579]: pgmap v7118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:21 smithi188 ceph-mon[92425]: pgmap v7118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:23 smithi161 ceph-mon[91579]: pgmap v7119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:23 smithi188 ceph-mon[92425]: pgmap v7119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:25 smithi161 ceph-mon[91579]: pgmap v7120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:25 smithi188 ceph-mon[92425]: pgmap v7120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:27 smithi161 ceph-mon[91579]: pgmap v7121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:27 smithi188 ceph-mon[92425]: pgmap v7121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:29 smithi161 ceph-mon[91579]: pgmap v7122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:29 smithi188 ceph-mon[92425]: pgmap v7122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:31 smithi161 ceph-mon[91579]: pgmap v7123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:31 smithi188 ceph-mon[92425]: pgmap v7123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:33 smithi161 ceph-mon[91579]: pgmap v7124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:33 smithi188 ceph-mon[92425]: pgmap v7124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:34.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:35 smithi161 ceph-mon[91579]: pgmap v7125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:35 smithi188 ceph-mon[92425]: pgmap v7125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:37 smithi161 ceph-mon[91579]: pgmap v7126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:37 smithi188 ceph-mon[92425]: pgmap v7126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:39 smithi161 ceph-mon[91579]: pgmap v7127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:39 smithi188 ceph-mon[92425]: pgmap v7127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:41.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:36:40.738+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:36:41.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:36:41.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:36:41.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:36:41.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:36:41.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:36:41.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:36:41.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:36:41.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:36:41.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:36:41.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:36:41.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:36:41.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:36:41.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:36:41.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:36:41.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:36:41.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:36:41.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:36:41.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:36:41.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:36:41.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:36:41.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:36:41.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:36:41.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:36:41.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:36:41.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:36:41.088 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:36:41.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:36:41.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:36:41.089 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:36:41.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:36:41.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:36:41.090 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:36:41.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:36:41.091 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:36:41.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:36:41.092 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:36:41.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:36:41.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:36:41.093 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:36:41.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:36:41.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:36:41.094 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:36:41.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:36:41.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:36:41.095 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=383] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:36:41.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=383] Local address: 172.21.15.161, port 54970 2022-07-03T14:36:41.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=383] Peer address: 172.21.15.101, port 22 2022-07-03T14:36:41.096 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=383] Connection lost 2022-07-03T14:36:41.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=383] Aborting connection 2022-07-03T14:36:41.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:36:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:36:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:36:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:36:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:36:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:36:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:36:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:36:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:36:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:36:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:36:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:36:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:36:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:36:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:36:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:36:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:36:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:36:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:36:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:36:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:36:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:36:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:36:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:36:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:36:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:36:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:36:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:36:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:36:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:36:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:36:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:36:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:36:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:36:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:36:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:36:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:36:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:36:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:36:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:36:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:36:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:36:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:36:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:36:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:36:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:36:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:36:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:36:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:36:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:36:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:36:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=383] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:36:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=383] Local address: 172.21.15.161, port 54970 2022-07-03T14:36:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=383] Peer address: 172.21.15.101, port 22 2022-07-03T14:36:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=383] Connection lost 2022-07-03T14:36:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=383] Aborting connection 2022-07-03T14:36:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:41 smithi161 ceph-mon[91579]: pgmap v7128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:36:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:36:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:36:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:36:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:36:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:36:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:36:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:36:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:36:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:36:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:36:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:36:41.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:36:41.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:36:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:36:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:36:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:36:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:36:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:36:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:36:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:36:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:36:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:36:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:36:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:36:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:36:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:36:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:36:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:36:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:36:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:36:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:36:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:36:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:36:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:36:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:36:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:36:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:36:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:36:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:36:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:36:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:36:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:36:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:36:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:36:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:36:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:36:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:36:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:36:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=383] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:36:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=383] Local address: 172.21.15.161, port 54970 2022-07-03T14:36:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=383] Peer address: 172.21.15.101, port 22 2022-07-03T14:36:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=383] Connection lost 2022-07-03T14:36:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=383] Aborting connection 2022-07-03T14:36:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:41 smithi188 ceph-mon[92425]: pgmap v7128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:43 smithi161 ceph-mon[91579]: pgmap v7129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:43 smithi188 ceph-mon[92425]: pgmap v7129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:45 smithi161 ceph-mon[91579]: pgmap v7130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:45 smithi188 ceph-mon[92425]: pgmap v7130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:47 smithi161 ceph-mon[91579]: pgmap v7131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:47 smithi188 ceph-mon[92425]: pgmap v7131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:49 smithi161 ceph-mon[91579]: pgmap v7132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:49 smithi188 ceph-mon[92425]: pgmap v7132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:51 smithi161 ceph-mon[91579]: pgmap v7133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:51 smithi188 ceph-mon[92425]: pgmap v7133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:53 smithi161 ceph-mon[91579]: pgmap v7134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:53 smithi188 ceph-mon[92425]: pgmap v7134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:36:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:55 smithi161 ceph-mon[91579]: pgmap v7135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:55 smithi188 ceph-mon[92425]: pgmap v7135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:57 smithi161 ceph-mon[91579]: pgmap v7136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:57 smithi188 ceph-mon[92425]: pgmap v7136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:36:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:36:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:36:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:36:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:36:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:36:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:36:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:36:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:36:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:36:59 smithi161 ceph-mon[91579]: pgmap v7137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:36:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:36:59 smithi188 ceph-mon[92425]: pgmap v7137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:01 smithi161 ceph-mon[91579]: pgmap v7138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:01 smithi188 ceph-mon[92425]: pgmap v7138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:03 smithi161 ceph-mon[91579]: pgmap v7139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:03 smithi188 ceph-mon[92425]: pgmap v7139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:05 smithi161 ceph-mon[91579]: pgmap v7140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:05 smithi188 ceph-mon[92425]: pgmap v7140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:07 smithi161 ceph-mon[91579]: pgmap v7141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:07 smithi188 ceph-mon[92425]: pgmap v7141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:09 smithi161 ceph-mon[91579]: pgmap v7142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:09 smithi188 ceph-mon[92425]: pgmap v7142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:11 smithi161 ceph-mon[91579]: pgmap v7143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:11 smithi188 ceph-mon[92425]: pgmap v7143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:13.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:13 smithi161 ceph-mon[91579]: pgmap v7144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:13 smithi188 ceph-mon[92425]: pgmap v7144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:15 smithi161 ceph-mon[91579]: pgmap v7145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:15 smithi188 ceph-mon[92425]: pgmap v7145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:17 smithi161 ceph-mon[91579]: pgmap v7146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:17 smithi188 ceph-mon[92425]: pgmap v7146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:19 smithi161 ceph-mon[91579]: pgmap v7147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:19 smithi188 ceph-mon[92425]: pgmap v7147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:21 smithi161 ceph-mon[91579]: pgmap v7148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:21 smithi188 ceph-mon[92425]: pgmap v7148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:23 smithi161 ceph-mon[91579]: pgmap v7149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:23 smithi188 ceph-mon[92425]: pgmap v7149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:25 smithi161 ceph-mon[91579]: pgmap v7150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:25 smithi188 ceph-mon[92425]: pgmap v7150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:27 smithi161 ceph-mon[91579]: pgmap v7151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:27 smithi188 ceph-mon[92425]: pgmap v7151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:29 smithi161 ceph-mon[91579]: pgmap v7152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:29 smithi188 ceph-mon[92425]: pgmap v7152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:31 smithi161 ceph-mon[91579]: pgmap v7153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:31 smithi188 ceph-mon[92425]: pgmap v7153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:33.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:33 smithi161 ceph-mon[91579]: pgmap v7154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:33 smithi188 ceph-mon[92425]: pgmap v7154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:35 smithi161 ceph-mon[91579]: pgmap v7155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:35 smithi188 ceph-mon[92425]: pgmap v7155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:37 smithi161 ceph-mon[91579]: pgmap v7156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:37 smithi188 ceph-mon[92425]: pgmap v7156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:39 smithi161 ceph-mon[91579]: pgmap v7157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:39 smithi188 ceph-mon[92425]: pgmap v7157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-mon[91579]: pgmap v7158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:41.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:37:41.238+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:37:41.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:37:41.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:37:41.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:37:41.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:37:41.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:37:41.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:37:41.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:37:41.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:37:41.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:37:41.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:37:41.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:37:41.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:37:41.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:37:41.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:37:41.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:37:41.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:37:41.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:37:41.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:37:41.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:37:41.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:37:41.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:37:41.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:37:41.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:37:41.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:37:41.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:37:41.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:37:41.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:37:41.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:37:41.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:37:41.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:37:41.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:37:41.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:37:41.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:37:41.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:37:41.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:37:41.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:37:41.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:37:41.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:37:41.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:37:41.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:37:41.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:37:41.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:37:41.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:37:41.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:37:41.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=385] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:37:41.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=385] Local address: 172.21.15.161, port 54974 2022-07-03T14:37:41.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=385] Peer address: 172.21.15.101, port 22 2022-07-03T14:37:41.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=385] Connection lost 2022-07-03T14:37:41.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=385] Aborting connection 2022-07-03T14:37:41.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:37:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:37:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:41 smithi188 ceph-mon[92425]: pgmap v7158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:37:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:37:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:37:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:42 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:37:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:37:42.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:37:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:37:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:37:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:37:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:37:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:37:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:37:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:37:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:37:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:37:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:37:42.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:37:42.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:37:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:37:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:37:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:37:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:37:42.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:37:42.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:37:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:37:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:37:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:37:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:37:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:37:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:37:42.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:37:42.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:37:42.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:37:42.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:37:42.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:37:42.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:37:42.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:37:42.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:37:42.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:37:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:37:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:37:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:37:42.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:37:42.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:37:42.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:37:42.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:37:42.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:37:42.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:37:42.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:37:42.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=385] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:37:42.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=385] Local address: 172.21.15.161, port 54974 2022-07-03T14:37:42.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=385] Peer address: 172.21.15.101, port 22 2022-07-03T14:37:42.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=385] Connection lost 2022-07-03T14:37:42.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=385] Aborting connection 2022-07-03T14:37:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:37:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:37:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:37:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:42 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:37:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:37:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:37:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:37:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:37:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:37:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:37:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:37:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:37:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:37:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:37:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:37:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:37:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:37:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:37:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:37:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:37:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:37:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:37:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:37:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:37:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:37:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:37:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:37:42.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:37:42.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:37:42.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:37:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:37:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:37:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:37:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:37:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:37:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:37:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:37:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:37:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:37:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:37:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:37:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:37:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:37:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:37:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:37:42.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:37:42.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:37:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:37:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:37:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=385] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:37:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=385] Local address: 172.21.15.161, port 54974 2022-07-03T14:37:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=385] Peer address: 172.21.15.101, port 22 2022-07-03T14:37:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=385] Connection lost 2022-07-03T14:37:42.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=385] Aborting connection 2022-07-03T14:37:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:44 smithi161 ceph-mon[91579]: pgmap v7159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:44 smithi188 ceph-mon[92425]: pgmap v7159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:45 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T14:37:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:37:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:37:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:45 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T14:37:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:37:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:37:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:46 smithi161 ceph-mon[91579]: pgmap v7160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:46 smithi188 ceph-mon[92425]: pgmap v7160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:48 smithi161 ceph-mon[91579]: pgmap v7161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:48 smithi188 ceph-mon[92425]: pgmap v7161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:50 smithi161 ceph-mon[91579]: pgmap v7162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:50 smithi188 ceph-mon[92425]: pgmap v7162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:52 smithi161 ceph-mon[91579]: pgmap v7163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:52 smithi188 ceph-mon[92425]: pgmap v7163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:54 smithi161 ceph-mon[91579]: pgmap v7164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:54 smithi188 ceph-mon[92425]: pgmap v7164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:37:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:37:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:56 smithi161 ceph-mon[91579]: pgmap v7165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:56 smithi188 ceph-mon[92425]: pgmap v7165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:37:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:37:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:37:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:37:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:37:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:37:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:37:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:37:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:58 smithi161 ceph-mon[91579]: pgmap v7166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:37:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:58 smithi188 ceph-mon[92425]: pgmap v7166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:00 smithi161 ceph-mon[91579]: pgmap v7167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:00 smithi188 ceph-mon[92425]: pgmap v7167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:02 smithi161 ceph-mon[91579]: pgmap v7168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:02 smithi188 ceph-mon[92425]: pgmap v7168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:04 smithi161 ceph-mon[91579]: pgmap v7169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:04 smithi188 ceph-mon[92425]: pgmap v7169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:06 smithi161 ceph-mon[91579]: pgmap v7170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:06 smithi188 ceph-mon[92425]: pgmap v7170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:09.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:08 smithi161 ceph-mon[91579]: pgmap v7171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:08 smithi188 ceph-mon[92425]: pgmap v7171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:10 smithi161 ceph-mon[91579]: pgmap v7172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:10 smithi188 ceph-mon[92425]: pgmap v7172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:12 smithi161 ceph-mon[91579]: pgmap v7173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:12 smithi188 ceph-mon[92425]: pgmap v7173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:14 smithi161 ceph-mon[91579]: pgmap v7174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:14 smithi188 ceph-mon[92425]: pgmap v7174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:16 smithi161 ceph-mon[91579]: pgmap v7175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:16 smithi188 ceph-mon[92425]: pgmap v7175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:19.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:18 smithi188 ceph-mon[92425]: pgmap v7176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:18 smithi161 ceph-mon[91579]: pgmap v7176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:20 smithi188 ceph-mon[92425]: pgmap v7177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:20 smithi161 ceph-mon[91579]: pgmap v7177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:22 smithi188 ceph-mon[92425]: pgmap v7178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:22 smithi161 ceph-mon[91579]: pgmap v7178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:24 smithi188 ceph-mon[92425]: pgmap v7179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:24 smithi161 ceph-mon[91579]: pgmap v7179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:26 smithi188 ceph-mon[92425]: pgmap v7180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:26 smithi161 ceph-mon[91579]: pgmap v7180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:28 smithi188 ceph-mon[92425]: pgmap v7181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:28 smithi161 ceph-mon[91579]: pgmap v7181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:30 smithi188 ceph-mon[92425]: pgmap v7182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:30 smithi161 ceph-mon[91579]: pgmap v7182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:32 smithi188 ceph-mon[92425]: pgmap v7183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:32 smithi161 ceph-mon[91579]: pgmap v7183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:34 smithi161 ceph-mon[91579]: pgmap v7184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:34 smithi188 ceph-mon[92425]: pgmap v7184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:36 smithi161 ceph-mon[91579]: pgmap v7185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:36 smithi188 ceph-mon[92425]: pgmap v7185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:38 smithi161 ceph-mon[91579]: pgmap v7186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:38 smithi188 ceph-mon[92425]: pgmap v7186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:40 smithi161 ceph-mon[91579]: pgmap v7187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:40 smithi188 ceph-mon[92425]: pgmap v7187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:42 smithi161 ceph-mon[91579]: pgmap v7188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:42 smithi188 ceph-mon[92425]: pgmap v7188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:44.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:38:44.519+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:38:44.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:38:44.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:38:44.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:38:44.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:38:44.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:38:44.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:38:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:38:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:38:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:38:44.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:38:44.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:38:44.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:38:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:38:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:38:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:38:44.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:38:44.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:38:44.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:38:44.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:38:44.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:38:44.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:38:44.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:38:44.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:38:44.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:38:44.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:38:44.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:38:44.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:38:44.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:38:44.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:38:44.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:38:44.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:38:44.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:38:44.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:38:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:38:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:38:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:38:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:38:44.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:38:44.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:38:44.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:38:44.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:38:44.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:38:44.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:38:44.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:38:44.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=387] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:38:44.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=387] Local address: 172.21.15.161, port 54978 2022-07-03T14:38:44.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=387] Peer address: 172.21.15.101, port 22 2022-07-03T14:38:44.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=387] Connection lost 2022-07-03T14:38:44.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=387] Aborting connection 2022-07-03T14:38:44.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:38:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-mon[91579]: pgmap v7189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:38:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:38:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:38:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:44 smithi188 ceph-mon[92425]: pgmap v7189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:38:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:38:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:38:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:38:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:38:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:38:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:38:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:38:46.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:38:46.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:38:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:38:46.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:38:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:38:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:38:46.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:38:46.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:38:46.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:38:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:38:46.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:38:46.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:38:46.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:38:46.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:38:46.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:38:46.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:38:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:38:46.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:38:46.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:38:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:38:46.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:38:46.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:38:46.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:38:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:38:46.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:38:46.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:38:46.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:38:46.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:38:46.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:38:46.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:38:46.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:38:46.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:38:46.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:38:46.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:38:46.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:38:46.417 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:38:46.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:38:46.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:38:46.419 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:38:46.420 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:38:46.421 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:38:46.421 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=387] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:38:46.422 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=387] Local address: 172.21.15.161, port 54978 2022-07-03T14:38:46.423 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=387] Peer address: 172.21.15.101, port 22 2022-07-03T14:38:46.424 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=387] Connection lost 2022-07-03T14:38:46.424 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=387] Aborting connection 2022-07-03T14:38:46.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:38:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:38:46.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:38:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:38:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:38:46.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:38:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:38:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:38:46.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:38:46.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:38:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:38:46.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:38:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:38:46.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:38:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:38:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:38:46.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:38:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:38:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:38:46.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:38:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:38:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:38:46.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:38:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:38:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:38:46.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:38:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:38:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:38:46.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:38:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:38:46.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:38:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:38:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:38:46.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:38:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:38:46.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:38:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:38:46.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:38:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:38:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:38:46.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:38:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:38:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:38:46.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:38:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:38:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:38:46.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=387] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:38:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=387] Local address: 172.21.15.161, port 54978 2022-07-03T14:38:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=387] Peer address: 172.21.15.101, port 22 2022-07-03T14:38:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=387] Connection lost 2022-07-03T14:38:46.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=387] Aborting connection 2022-07-03T14:38:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:46 smithi161 ceph-mon[91579]: pgmap v7190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:46 smithi188 ceph-mon[92425]: pgmap v7190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:48 smithi161 ceph-mon[91579]: pgmap v7191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:48 smithi188 ceph-mon[92425]: pgmap v7191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:50 smithi161 ceph-mon[91579]: pgmap v7192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:51.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:50 smithi188 ceph-mon[92425]: pgmap v7192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:52 smithi161 ceph-mon[91579]: pgmap v7193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:52 smithi188 ceph-mon[92425]: pgmap v7193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:54 smithi161 ceph-mon[91579]: pgmap v7194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:54 smithi188 ceph-mon[92425]: pgmap v7194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:38:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:38:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:56 smithi161 ceph-mon[91579]: pgmap v7195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:56 smithi188 ceph-mon[92425]: pgmap v7195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:38:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:38:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:38:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:38:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:38:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:38:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:38:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:38:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:58 smithi161 ceph-mon[91579]: pgmap v7196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:38:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:58 smithi188 ceph-mon[92425]: pgmap v7196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:00 smithi161 ceph-mon[91579]: pgmap v7197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:01.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:00 smithi188 ceph-mon[92425]: pgmap v7197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:02 smithi161 ceph-mon[91579]: pgmap v7198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:02 smithi188 ceph-mon[92425]: pgmap v7198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:04 smithi161 ceph-mon[91579]: pgmap v7199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:04 smithi188 ceph-mon[92425]: pgmap v7199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:06 smithi161 ceph-mon[91579]: pgmap v7200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:06 smithi188 ceph-mon[92425]: pgmap v7200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:08 smithi161 ceph-mon[91579]: pgmap v7201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:08 smithi188 ceph-mon[92425]: pgmap v7201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:10 smithi161 ceph-mon[91579]: pgmap v7202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:10 smithi188 ceph-mon[92425]: pgmap v7202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:13 smithi161 ceph-mon[91579]: pgmap v7203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:13 smithi188 ceph-mon[92425]: pgmap v7203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:15 smithi161 ceph-mon[91579]: pgmap v7204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:15 smithi188 ceph-mon[92425]: pgmap v7204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:17 smithi161 ceph-mon[91579]: pgmap v7205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:17 smithi188 ceph-mon[92425]: pgmap v7205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:19 smithi161 ceph-mon[91579]: pgmap v7206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:19 smithi188 ceph-mon[92425]: pgmap v7206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:21 smithi161 ceph-mon[91579]: pgmap v7207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:21 smithi188 ceph-mon[92425]: pgmap v7207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:23 smithi161 ceph-mon[91579]: pgmap v7208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:23 smithi188 ceph-mon[92425]: pgmap v7208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:25 smithi161 ceph-mon[91579]: pgmap v7209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:25 smithi188 ceph-mon[92425]: pgmap v7209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:27 smithi161 ceph-mon[91579]: pgmap v7210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:27 smithi188 ceph-mon[92425]: pgmap v7210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:29 smithi161 ceph-mon[91579]: pgmap v7211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:29 smithi188 ceph-mon[92425]: pgmap v7211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:31 smithi161 ceph-mon[91579]: pgmap v7212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:31 smithi188 ceph-mon[92425]: pgmap v7212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:32 smithi161 ceph-mon[91579]: pgmap v7213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:32 smithi188 ceph-mon[92425]: pgmap v7213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:34 smithi161 ceph-mon[91579]: pgmap v7214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:34 smithi188 ceph-mon[92425]: pgmap v7214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:36 smithi161 ceph-mon[91579]: pgmap v7215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:36 smithi188 ceph-mon[92425]: pgmap v7215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:38 smithi161 ceph-mon[91579]: pgmap v7216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:38 smithi188 ceph-mon[92425]: pgmap v7216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:40 smithi161 ceph-mon[91579]: pgmap v7217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:41.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:40 smithi188 ceph-mon[92425]: pgmap v7217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:42 smithi161 ceph-mon[91579]: pgmap v7218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:42 smithi188 ceph-mon[92425]: pgmap v7218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:44 smithi161 ceph-mon[91579]: pgmap v7219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:44 smithi188 ceph-mon[92425]: pgmap v7219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:39:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:39:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:39:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:39:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:39:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:39:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:46 smithi161 ceph-mon[91579]: pgmap v7220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:46 smithi188 ceph-mon[92425]: pgmap v7220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:48 smithi161 ceph-mon[91579]: pgmap v7221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:48 smithi188 ceph-mon[92425]: pgmap v7221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:50 smithi188 ceph-mon[92425]: pgmap v7222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:50 smithi161 ceph-mon[91579]: pgmap v7222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:52 smithi188 ceph-mon[92425]: pgmap v7223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:52 smithi161 ceph-mon[91579]: pgmap v7223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:54 smithi188 ceph-mon[92425]: pgmap v7224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:54 smithi161 ceph-mon[91579]: pgmap v7224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:39:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:39:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:56 smithi188 ceph-mon[92425]: pgmap v7225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:56 smithi161 ceph-mon[91579]: pgmap v7225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:39:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:39:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:39:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:39:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:39:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:39:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:39:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:39:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:58 smithi188 ceph-mon[92425]: pgmap v7226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:39:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:58 smithi161 ceph-mon[91579]: pgmap v7226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:00 smithi188 ceph-mon[92425]: pgmap v7227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:40:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:00 smithi161 ceph-mon[91579]: pgmap v7227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:40:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:02 smithi188 ceph-mon[92425]: pgmap v7228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:02 smithi161 ceph-mon[91579]: pgmap v7228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:04 smithi161 ceph-mon[91579]: pgmap v7229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:04 smithi188 ceph-mon[92425]: pgmap v7229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:06 smithi161 ceph-mon[91579]: pgmap v7230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:06 smithi188 ceph-mon[92425]: pgmap v7230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:08 smithi161 ceph-mon[91579]: pgmap v7231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:08 smithi188 ceph-mon[92425]: pgmap v7231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:10 smithi161 ceph-mon[91579]: pgmap v7232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:10 smithi188 ceph-mon[92425]: pgmap v7232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:12 smithi161 ceph-mon[91579]: pgmap v7233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:12 smithi188 ceph-mon[92425]: pgmap v7233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:14 smithi188 ceph-mon[92425]: pgmap v7234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:14 smithi161 ceph-mon[91579]: pgmap v7234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:15.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:40:15.023+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:40:15.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:40:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:40:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:40:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:40:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:40:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:40:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:40:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:40:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:40:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:40:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:40:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:40:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:40:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:40:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:40:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:40:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:40:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:40:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:40:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:40:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:40:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:40:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:40:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:40:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:40:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:40:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:40:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:40:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:40:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:40:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:40:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:40:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:40:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:40:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:40:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:40:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:40:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:40:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:40:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:40:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:40:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:40:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:40:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:40:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=389] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:40:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=389] Local address: 172.21.15.161, port 55006 2022-07-03T14:40:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=389] Peer address: 172.21.15.101, port 22 2022-07-03T14:40:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=389] Connection lost 2022-07-03T14:40:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=389] Aborting connection 2022-07-03T14:40:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:40:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:40:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:40:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:40:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:40:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:40:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:40:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:40:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:40:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:40:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:40:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:40:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:40:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:40:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:40:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:40:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:40:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:40:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:40:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:40:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:40:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:40:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:40:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:40:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:40:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:40:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:40:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:40:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:40:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:40:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:40:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:40:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:40:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:40:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:40:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:40:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:40:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:40:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:40:16.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:40:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:40:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:40:16.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:40:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:40:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:40:16.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:40:16.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:40:16.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=389] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:40:16.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=389] Local address: 172.21.15.161, port 55006 2022-07-03T14:40:16.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=389] Peer address: 172.21.15.101, port 22 2022-07-03T14:40:16.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=389] Connection lost 2022-07-03T14:40:16.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=389] Aborting connection 2022-07-03T14:40:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:40:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:40:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:40:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:40:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:40:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:40:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:40:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:40:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:40:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:40:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:40:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:40:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:40:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:40:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:40:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:40:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:40:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:40:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:40:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:40:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:40:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:40:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:40:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:40:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:40:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:40:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:40:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:40:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:40:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:40:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:40:16.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:40:16.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:40:16.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:40:16.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:40:16.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:40:16.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:40:16.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:40:16.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:40:16.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:40:16.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:40:16.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:40:16.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:40:16.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:40:16.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:40:16.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:40:16.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=389] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:40:16.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=389] Local address: 172.21.15.161, port 55006 2022-07-03T14:40:16.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=389] Peer address: 172.21.15.101, port 22 2022-07-03T14:40:16.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=389] Connection lost 2022-07-03T14:40:16.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=389] Aborting connection 2022-07-03T14:40:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:16 smithi161 ceph-mon[91579]: pgmap v7235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:16 smithi188 ceph-mon[92425]: pgmap v7235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:18 smithi161 ceph-mon[91579]: pgmap v7236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:18 smithi188 ceph-mon[92425]: pgmap v7236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:20 smithi161 ceph-mon[91579]: pgmap v7237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:20 smithi188 ceph-mon[92425]: pgmap v7237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:22 smithi161 ceph-mon[91579]: pgmap v7238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:22 smithi188 ceph-mon[92425]: pgmap v7238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:24 smithi161 ceph-mon[91579]: pgmap v7239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:24 smithi188 ceph-mon[92425]: pgmap v7239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:26 smithi161 ceph-mon[91579]: pgmap v7240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:26 smithi188 ceph-mon[92425]: pgmap v7240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:28 smithi161 ceph-mon[91579]: pgmap v7241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:28 smithi188 ceph-mon[92425]: pgmap v7241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:30 smithi161 ceph-mon[91579]: pgmap v7242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:30 smithi188 ceph-mon[92425]: pgmap v7242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:32 smithi161 ceph-mon[91579]: pgmap v7243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:32 smithi188 ceph-mon[92425]: pgmap v7243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:34 smithi161 ceph-mon[91579]: pgmap v7244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:34 smithi188 ceph-mon[92425]: pgmap v7244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:36 smithi161 ceph-mon[91579]: pgmap v7245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:36 smithi188 ceph-mon[92425]: pgmap v7245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:38 smithi161 ceph-mon[91579]: pgmap v7246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:38 smithi188 ceph-mon[92425]: pgmap v7246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:40 smithi161 ceph-mon[91579]: pgmap v7247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:40 smithi188 ceph-mon[92425]: pgmap v7247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:42 smithi161 ceph-mon[91579]: pgmap v7248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:42 smithi188 ceph-mon[92425]: pgmap v7248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:44 smithi161 ceph-mon[91579]: pgmap v7249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:44 smithi188 ceph-mon[92425]: pgmap v7249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:46 smithi161 ceph-mon[91579]: pgmap v7250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:46 smithi188 ceph-mon[92425]: pgmap v7250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:49 smithi161 ceph-mon[91579]: pgmap v7251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:49 smithi188 ceph-mon[92425]: pgmap v7251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:51 smithi161 ceph-mon[91579]: pgmap v7252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:51 smithi188 ceph-mon[92425]: pgmap v7252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:53 smithi161 ceph-mon[91579]: pgmap v7253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:53 smithi188 ceph-mon[92425]: pgmap v7253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:55 smithi161 ceph-mon[91579]: pgmap v7254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:55 smithi188 ceph-mon[92425]: pgmap v7254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:40:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:40:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:57 smithi161 ceph-mon[91579]: pgmap v7255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:57 smithi188 ceph-mon[92425]: pgmap v7255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:40:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:40:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:40:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:40:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:40:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:40:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:40:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:40:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:40:59 smithi161 ceph-mon[91579]: pgmap v7256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:40:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:40:59 smithi188 ceph-mon[92425]: pgmap v7256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:01 smithi161 ceph-mon[91579]: pgmap v7257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:01 smithi188 ceph-mon[92425]: pgmap v7257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:03 smithi161 ceph-mon[91579]: pgmap v7258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:03 smithi188 ceph-mon[92425]: pgmap v7258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:05 smithi161 ceph-mon[91579]: pgmap v7259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:05 smithi188 ceph-mon[92425]: pgmap v7259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:07 smithi161 ceph-mon[91579]: pgmap v7260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:07 smithi188 ceph-mon[92425]: pgmap v7260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:09 smithi161 ceph-mon[91579]: pgmap v7261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:09 smithi188 ceph-mon[92425]: pgmap v7261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:11 smithi161 ceph-mon[91579]: pgmap v7262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:11 smithi188 ceph-mon[92425]: pgmap v7262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:13 smithi161 ceph-mon[91579]: pgmap v7263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:13 smithi188 ceph-mon[92425]: pgmap v7263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-mon[91579]: pgmap v7264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:41:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:41:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:41:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:41:15.057+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:41:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:41:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:41:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:41:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:41:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:41:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:41:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:41:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:41:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:41:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:41:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:41:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:41:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:41:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:41:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:41:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:41:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:41:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:41:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:41:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:41:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:41:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:41:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:41:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:41:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:41:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:41:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:41:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:41:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:41:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:41:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:41:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:41:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:41:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:41:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:41:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:41:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:41:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:41:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:41:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:41:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:41:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:41:15.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:41:15.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:41:15.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=391] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:41:15.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=391] Local address: 172.21.15.161, port 55010 2022-07-03T14:41:15.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=391] Peer address: 172.21.15.101, port 22 2022-07-03T14:41:15.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=391] Connection lost 2022-07-03T14:41:15.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=391] Aborting connection 2022-07-03T14:41:15.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:41:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:41:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:15 smithi188 ceph-mon[92425]: pgmap v7264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:41:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:41:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:41:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:41:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:41:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:41:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:41:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:41:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:41:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:41:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:41:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:41:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:41:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:41:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:41:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:41:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:41:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:41:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:41:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:41:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:41:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:41:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:41:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:41:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:41:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:41:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:41:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:41:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:41:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:41:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:41:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:41:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:41:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:41:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:41:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:41:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:41:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:41:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:41:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:41:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:41:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:41:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:41:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:41:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:41:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:41:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:41:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:41:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:41:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=391] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:41:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=391] Local address: 172.21.15.161, port 55010 2022-07-03T14:41:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=391] Peer address: 172.21.15.101, port 22 2022-07-03T14:41:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=391] Connection lost 2022-07-03T14:41:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=391] Aborting connection 2022-07-03T14:41:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:41:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:41:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:41:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:41:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:41:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:41:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:41:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:41:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:41:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:41:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:41:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:41:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:41:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:41:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:41:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:41:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:41:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:41:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:41:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:41:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:41:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:41:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:41:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:41:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:41:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:41:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:41:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:41:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:41:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:41:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:41:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:41:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:41:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:41:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:41:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:41:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:41:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:41:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:41:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:41:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:41:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:41:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:41:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:41:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:41:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:41:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=391] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:41:16.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=391] Local address: 172.21.15.161, port 55010 2022-07-03T14:41:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=391] Peer address: 172.21.15.101, port 22 2022-07-03T14:41:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=391] Connection lost 2022-07-03T14:41:16.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=391] Aborting connection 2022-07-03T14:41:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:16.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:41:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:17 smithi161 ceph-mon[91579]: pgmap v7265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:17 smithi188 ceph-mon[92425]: pgmap v7265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:18 smithi161 ceph-mon[91579]: pgmap v7266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:18 smithi188 ceph-mon[92425]: pgmap v7266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:20 smithi188 ceph-mon[92425]: pgmap v7267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:20.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:20 smithi161 ceph-mon[91579]: pgmap v7267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:20.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:22 smithi188 ceph-mon[92425]: pgmap v7268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:22 smithi161 ceph-mon[91579]: pgmap v7268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:24 smithi188 ceph-mon[92425]: pgmap v7269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:24 smithi161 ceph-mon[91579]: pgmap v7269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:27.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:26 smithi161 ceph-mon[91579]: pgmap v7270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:26 smithi188 ceph-mon[92425]: pgmap v7270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:28 smithi161 ceph-mon[91579]: pgmap v7271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:28 smithi188 ceph-mon[92425]: pgmap v7271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:30 smithi161 ceph-mon[91579]: pgmap v7272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:30 smithi188 ceph-mon[92425]: pgmap v7272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:32 smithi188 ceph-mon[92425]: pgmap v7273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:32 smithi161 ceph-mon[91579]: pgmap v7273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:34 smithi188 ceph-mon[92425]: pgmap v7274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:34 smithi161 ceph-mon[91579]: pgmap v7274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:36 smithi188 ceph-mon[92425]: pgmap v7275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:36 smithi161 ceph-mon[91579]: pgmap v7275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:38 smithi188 ceph-mon[92425]: pgmap v7276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:38 smithi161 ceph-mon[91579]: pgmap v7276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:40 smithi188 ceph-mon[92425]: pgmap v7277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:40 smithi161 ceph-mon[91579]: pgmap v7277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:42 smithi188 ceph-mon[92425]: pgmap v7278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:42 smithi161 ceph-mon[91579]: pgmap v7278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:44 smithi188 ceph-mon[92425]: pgmap v7279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:44 smithi161 ceph-mon[91579]: pgmap v7279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:46 smithi188 ceph-mon[92425]: pgmap v7280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:46 smithi161 ceph-mon[91579]: pgmap v7280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:48 smithi188 ceph-mon[92425]: pgmap v7281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:48 smithi161 ceph-mon[91579]: pgmap v7281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:50 smithi161 ceph-mon[91579]: pgmap v7282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:50 smithi188 ceph-mon[92425]: pgmap v7282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:52 smithi161 ceph-mon[91579]: pgmap v7283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:52 smithi188 ceph-mon[92425]: pgmap v7283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:41:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:54 smithi161 ceph-mon[91579]: pgmap v7284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:54 smithi188 ceph-mon[92425]: pgmap v7284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:41:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:41:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:56 smithi161 ceph-mon[91579]: pgmap v7285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:41:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:56 smithi188 ceph-mon[92425]: pgmap v7285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:41:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:41:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:41:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:41:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:41:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:41:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:41:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:41:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:41:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:58 smithi161 ceph-mon[91579]: pgmap v7286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:41:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:58 smithi188 ceph-mon[92425]: pgmap v7286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:42:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:00 smithi161 ceph-mon[91579]: pgmap v7287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:00 smithi188 ceph-mon[92425]: pgmap v7287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:02 smithi161 ceph-mon[91579]: pgmap v7288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:42:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:02 smithi188 ceph-mon[92425]: pgmap v7288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:42:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:04 smithi161 ceph-mon[91579]: pgmap v7289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:04 smithi188 ceph-mon[92425]: pgmap v7289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:06 smithi161 ceph-mon[91579]: pgmap v7290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:06 smithi188 ceph-mon[92425]: pgmap v7290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:08 smithi161 ceph-mon[91579]: pgmap v7291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:08 smithi188 ceph-mon[92425]: pgmap v7291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:10 smithi161 ceph-mon[91579]: pgmap v7292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:10 smithi188 ceph-mon[92425]: pgmap v7292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:12 smithi161 ceph-mon[91579]: pgmap v7293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:12 smithi188 ceph-mon[92425]: pgmap v7293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:14 smithi161 ceph-mon[91579]: pgmap v7294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:14 smithi188 ceph-mon[92425]: pgmap v7294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:16.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:42:16.003+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:42:16.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:42:16.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:42:16.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:42:16.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:42:16.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:42:16.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:42:16.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:42:16.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:42:16.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:42:16.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:42:16.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:42:16.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:42:16.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:42:16.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:42:16.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:42:16.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:42:16.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:42:16.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:42:16.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:42:16.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:42:16.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:42:16.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:42:16.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:42:16.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:42:16.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:42:16.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:42:16.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:42:16.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:42:16.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:42:16.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:42:16.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:42:16.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:42:16.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:42:16.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:42:16.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:42:16.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:42:16.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:42:16.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:42:16.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:42:16.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:42:16.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:42:16.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:42:16.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:42:16.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:42:16.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=393] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:42:16.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=393] Local address: 172.21.15.161, port 55014 2022-07-03T14:42:16.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=393] Peer address: 172.21.15.101, port 22 2022-07-03T14:42:16.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=393] Connection lost 2022-07-03T14:42:16.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=393] Aborting connection 2022-07-03T14:42:16.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:42:17.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: pgmap v7295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:17.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:42:17.228 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:42:17.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:42:17.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:42:17.229 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:42:17.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:42:17.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:42:17.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:42:17.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:42:17.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:42:17.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:42:17.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:42:17.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:42:17.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:42:17.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:42:17.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:42:17.233 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:42:17.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:42:17.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:42:17.234 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:42:17.235 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:42:17.235 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:42:17.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:42:17.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:42:17.236 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:42:17.237 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:42:17.237 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:42:17.237 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:42:17.238 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:42:17.238 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:42:17.238 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:42:17.239 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:42:17.239 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:42:17.239 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:42:17.240 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:42:17.240 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:42:17.240 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:42:17.241 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:42:17.241 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:42:17.241 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:42:17.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:42:17.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:42:17.242 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:42:17.243 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:42:17.243 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:42:17.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:42:17.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:42:17.244 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:42:17.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:42:17.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=393] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:42:17.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=393] Local address: 172.21.15.161, port 55014 2022-07-03T14:42:17.245 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=393] Peer address: 172.21.15.101, port 22 2022-07-03T14:42:17.246 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=393] Connection lost 2022-07-03T14:42:17.246 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=393] Aborting connection 2022-07-03T14:42:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: pgmap v7295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:42:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:42:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:42:17.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:42:17.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:42:17.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:42:17.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:42:17.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:42:17.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:42:17.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:42:17.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:42:17.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:42:17.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:42:17.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:42:17.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:42:17.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:42:17.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:42:17.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:42:17.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:42:17.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:42:17.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:42:17.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:42:17.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:42:17.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:42:17.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:42:17.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:42:17.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:42:17.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:42:17.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:42:17.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:42:17.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:42:17.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:42:17.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:42:17.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:42:17.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:42:17.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:42:17.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:42:17.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:42:17.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:42:17.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:42:17.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:42:17.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:42:17.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:42:17.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:42:17.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:42:17.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:42:17.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:42:17.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:42:17.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:42:17.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=393] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:42:17.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=393] Local address: 172.21.15.161, port 55014 2022-07-03T14:42:17.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=393] Peer address: 172.21.15.101, port 22 2022-07-03T14:42:17.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=393] Connection lost 2022-07-03T14:42:17.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=393] Aborting connection 2022-07-03T14:42:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:18 smithi188 ceph-mon[92425]: pgmap v7296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:18 smithi161 ceph-mon[91579]: pgmap v7296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:20 smithi161 ceph-mon[91579]: pgmap v7297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:20 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T14:42:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:42:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:42:20.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:20 smithi188 ceph-mon[92425]: pgmap v7297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:20 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T14:42:20.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:42:20.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:42:20.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:22 smithi161 ceph-mon[91579]: pgmap v7298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:22 smithi188 ceph-mon[92425]: pgmap v7298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:24 smithi161 ceph-mon[91579]: pgmap v7299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:24 smithi188 ceph-mon[92425]: pgmap v7299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:26 smithi188 ceph-mon[92425]: pgmap v7300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:26 smithi161 ceph-mon[91579]: pgmap v7300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:28 smithi188 ceph-mon[92425]: pgmap v7301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:28 smithi161 ceph-mon[91579]: pgmap v7301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:30 smithi188 ceph-mon[92425]: pgmap v7302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:30 smithi161 ceph-mon[91579]: pgmap v7302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:32 smithi188 ceph-mon[92425]: pgmap v7303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:32 smithi161 ceph-mon[91579]: pgmap v7303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:34 smithi188 ceph-mon[92425]: pgmap v7304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:34 smithi161 ceph-mon[91579]: pgmap v7304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:36 smithi188 ceph-mon[92425]: pgmap v7305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:36 smithi161 ceph-mon[91579]: pgmap v7305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:38 smithi188 ceph-mon[92425]: pgmap v7306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:38 smithi161 ceph-mon[91579]: pgmap v7306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:40 smithi161 ceph-mon[91579]: pgmap v7307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:40 smithi188 ceph-mon[92425]: pgmap v7307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:42 smithi161 ceph-mon[91579]: pgmap v7308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:42 smithi188 ceph-mon[92425]: pgmap v7308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:45.314 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:44 smithi161 ceph-mon[91579]: pgmap v7309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:45.315 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:44 smithi188 ceph-mon[92425]: pgmap v7309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:46 smithi161 ceph-mon[91579]: pgmap v7310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:46 smithi188 ceph-mon[92425]: pgmap v7310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:48 smithi161 ceph-mon[91579]: pgmap v7311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:48 smithi188 ceph-mon[92425]: pgmap v7311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:50 smithi161 ceph-mon[91579]: pgmap v7312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:50 smithi188 ceph-mon[92425]: pgmap v7312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:52 smithi161 ceph-mon[91579]: pgmap v7313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:52 smithi188 ceph-mon[92425]: pgmap v7313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:54 smithi161 ceph-mon[91579]: pgmap v7314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:54 smithi188 ceph-mon[92425]: pgmap v7314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:42:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:42:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:56 smithi161 ceph-mon[91579]: pgmap v7315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:56 smithi188 ceph-mon[92425]: pgmap v7315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:42:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:42:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:42:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:42:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:42:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:42:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:42:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:42:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:58 smithi161 ceph-mon[91579]: pgmap v7316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:42:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:58 smithi188 ceph-mon[92425]: pgmap v7316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:00 smithi161 ceph-mon[91579]: pgmap v7317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:00 smithi188 ceph-mon[92425]: pgmap v7317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:02 smithi161 ceph-mon[91579]: pgmap v7318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:02 smithi188 ceph-mon[92425]: pgmap v7318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:04 smithi161 ceph-mon[91579]: pgmap v7319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:04 smithi188 ceph-mon[92425]: pgmap v7319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:06 smithi161 ceph-mon[91579]: pgmap v7320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:06 smithi188 ceph-mon[92425]: pgmap v7320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:08 smithi161 ceph-mon[91579]: pgmap v7321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:08 smithi188 ceph-mon[92425]: pgmap v7321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:10 smithi161 ceph-mon[91579]: pgmap v7322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:10 smithi188 ceph-mon[92425]: pgmap v7322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:12 smithi161 ceph-mon[91579]: pgmap v7323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:12 smithi188 ceph-mon[92425]: pgmap v7323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:14 smithi161 ceph-mon[91579]: pgmap v7324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:14 smithi188 ceph-mon[92425]: pgmap v7324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:16 smithi161 ceph-mon[91579]: pgmap v7325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:16 smithi188 ceph-mon[92425]: pgmap v7325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:19.365 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:18 smithi161 ceph-mon[91579]: pgmap v7326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:19.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:18 smithi188 ceph-mon[92425]: pgmap v7326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:43:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:43:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:43:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:43:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:43:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:43:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: pgmap v7327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:43:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: pgmap v7327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:43:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:43:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:22 smithi161 ceph-mon[91579]: pgmap v7328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:22 smithi188 ceph-mon[92425]: pgmap v7328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:24 smithi161 ceph-mon[91579]: pgmap v7329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:24 smithi188 ceph-mon[92425]: pgmap v7329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:26 smithi161 ceph-mon[91579]: pgmap v7330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:26 smithi188 ceph-mon[92425]: pgmap v7330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:29 smithi161 ceph-mon[91579]: pgmap v7331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:29 smithi188 ceph-mon[92425]: pgmap v7331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:31 smithi161 ceph-mon[91579]: pgmap v7332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:31 smithi188 ceph-mon[92425]: pgmap v7332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:33 smithi161 ceph-mon[91579]: pgmap v7333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:33 smithi188 ceph-mon[92425]: pgmap v7333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:35 smithi161 ceph-mon[91579]: pgmap v7334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:35 smithi188 ceph-mon[92425]: pgmap v7334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:37 smithi161 ceph-mon[91579]: pgmap v7335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:37 smithi188 ceph-mon[92425]: pgmap v7335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:39 smithi161 ceph-mon[91579]: pgmap v7336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:39 smithi188 ceph-mon[92425]: pgmap v7336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:41 smithi161 ceph-mon[91579]: pgmap v7337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:41 smithi188 ceph-mon[92425]: pgmap v7337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:43 smithi161 ceph-mon[91579]: pgmap v7338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:43 smithi188 ceph-mon[92425]: pgmap v7338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:45 smithi161 ceph-mon[91579]: pgmap v7339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:45 smithi188 ceph-mon[92425]: pgmap v7339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:47 smithi161 ceph-mon[91579]: pgmap v7340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:47 smithi188 ceph-mon[92425]: pgmap v7340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-mon[91579]: pgmap v7341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:49 smithi188 ceph-mon[92425]: pgmap v7341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:43:49.379+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:43:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:43:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:43:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:43:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:43:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:43:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:43:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:43:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:43:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:43:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:43:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:43:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:43:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:43:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:43:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:43:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:43:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:43:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:43:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:43:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:43:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:43:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:43:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:43:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:43:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:43:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:43:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:43:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:43:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:43:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:43:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:43:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:43:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:43:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:43:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:43:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:43:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:43:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:43:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:43:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:43:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:43:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:43:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:43:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:43:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=395] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:43:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=395] Local address: 172.21.15.161, port 55018 2022-07-03T14:43:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=395] Peer address: 172.21.15.101, port 22 2022-07-03T14:43:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1084] Received exit status 0 2022-07-03T14:43:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1084] Received channel close 2022-07-03T14:43:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1084] Channel closed 2022-07-03T14:43:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1085] Requesting new SSH session 2022-07-03T14:43:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1085] Command: which python3 2022-07-03T14:43:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1084] Received exit status 0 2022-07-03T14:43:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1084] Received channel close 2022-07-03T14:43:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1084] Channel closed 2022-07-03T14:43:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1085] Requesting new SSH session 2022-07-03T14:43:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1085] Command: which python3 2022-07-03T14:43:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1085] Received exit status 0 2022-07-03T14:43:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1085] Received channel close 2022-07-03T14:43:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1085] Channel closed 2022-07-03T14:43:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1086] Requesting new SSH session 2022-07-03T14:43:49.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1086] Command: sudo true 2022-07-03T14:43:49.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1085] Received exit status 0 2022-07-03T14:43:49.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1085] Received channel close 2022-07-03T14:43:49.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1085] Channel closed 2022-07-03T14:43:49.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1086] Requesting new SSH session 2022-07-03T14:43:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1086] Command: sudo true 2022-07-03T14:43:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1086] Received exit status 0 2022-07-03T14:43:49.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1086] Received channel close 2022-07-03T14:43:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1086] Channel closed 2022-07-03T14:43:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1087] Requesting new SSH session 2022-07-03T14:43:49.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:49.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1086] Received exit status 0 2022-07-03T14:43:49.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1086] Received channel close 2022-07-03T14:43:49.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1086] Channel closed 2022-07-03T14:43:49.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1087] Requesting new SSH session 2022-07-03T14:43:49.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:49.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1087] Received exit status 0 2022-07-03T14:43:49.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1087] Received channel close 2022-07-03T14:43:49.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1087] Channel closed 2022-07-03T14:43:49.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1088] Requesting new SSH session 2022-07-03T14:43:49.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1088] Command: sudo true 2022-07-03T14:43:49.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1087] Received exit status 0 2022-07-03T14:43:49.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1087] Received channel close 2022-07-03T14:43:49.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1087] Channel closed 2022-07-03T14:43:49.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1088] Requesting new SSH session 2022-07-03T14:43:49.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1088] Command: sudo true 2022-07-03T14:43:49.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1088] Received exit status 0 2022-07-03T14:43:49.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1088] Received channel close 2022-07-03T14:43:49.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1088] Channel closed 2022-07-03T14:43:49.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1089] Requesting new SSH session 2022-07-03T14:43:49.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1089] Command: which python3 2022-07-03T14:43:49.914 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1088] Received exit status 0 2022-07-03T14:43:49.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1088] Received channel close 2022-07-03T14:43:49.915 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1088] Channel closed 2022-07-03T14:43:49.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1089] Requesting new SSH session 2022-07-03T14:43:49.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1089] Command: which python3 2022-07-03T14:43:49.916 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1089] Received exit status 0 2022-07-03T14:43:49.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1089] Received channel close 2022-07-03T14:43:49.917 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1089] Channel closed 2022-07-03T14:43:49.918 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1090] Requesting new SSH session 2022-07-03T14:43:49.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1090] Command: sudo true 2022-07-03T14:43:49.919 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1089] Received exit status 0 2022-07-03T14:43:49.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1089] Received channel close 2022-07-03T14:43:49.920 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1089] Channel closed 2022-07-03T14:43:49.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1090] Requesting new SSH session 2022-07-03T14:43:49.921 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1090] Command: sudo true 2022-07-03T14:43:49.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1090] Received exit status 0 2022-07-03T14:43:49.922 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1090] Received channel close 2022-07-03T14:43:49.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1090] Channel closed 2022-07-03T14:43:49.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1091] Requesting new SSH session 2022-07-03T14:43:49.923 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:49.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1090] Received exit status 0 2022-07-03T14:43:49.924 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1090] Received channel close 2022-07-03T14:43:49.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1090] Channel closed 2022-07-03T14:43:49.925 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1091] Requesting new SSH session 2022-07-03T14:43:49.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:49.926 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1091] Received exit status 0 2022-07-03T14:43:49.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1091] Received channel close 2022-07-03T14:43:49.927 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1091] Channel closed 2022-07-03T14:43:49.928 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1091] Received exit status 0 2022-07-03T14:43:49.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1091] Received channel close 2022-07-03T14:43:49.930 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1091] Channel closed 2022-07-03T14:43:49.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=395] Login timeout expired 2022-07-03T14:43:49.931 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=395] Aborting connection 2022-07-03T14:43:49.932 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:43:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:43:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:51 smithi161 ceph-mon[91579]: pgmap v7342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:51 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:43:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:43:51.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:43:51.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:43:51.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:43:51.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:43:51.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:43:51.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:43:51.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:43:51.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:43:51.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:43:51.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:43:51.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:43:51.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:43:51.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:43:51.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:43:51.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:43:51.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:43:51.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:43:51.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:43:51.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:43:51.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:43:51.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:43:51.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:43:51.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:43:51.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:43:51.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:43:51.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:43:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:43:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:43:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:43:51.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:43:51.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:43:51.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:43:51.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:43:51.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:43:51.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:43:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:43:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:43:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:43:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:43:51.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:43:51.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:43:51.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:43:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:43:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=395] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:43:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=395] Local address: 172.21.15.161, port 55018 2022-07-03T14:43:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=395] Peer address: 172.21.15.101, port 22 2022-07-03T14:43:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1084] Received exit status 0 2022-07-03T14:43:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1084] Received channel close 2022-07-03T14:43:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1084] Channel closed 2022-07-03T14:43:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1085] Requesting new SSH session 2022-07-03T14:43:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1085] Command: which python3 2022-07-03T14:43:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1084] Received exit status 0 2022-07-03T14:43:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1084] Received channel close 2022-07-03T14:43:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1084] Channel closed 2022-07-03T14:43:51.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1085] Requesting new SSH session 2022-07-03T14:43:51.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1085] Command: which python3 2022-07-03T14:43:51.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1085] Received exit status 0 2022-07-03T14:43:51.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1085] Received channel close 2022-07-03T14:43:51.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1085] Channel closed 2022-07-03T14:43:51.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1086] Requesting new SSH session 2022-07-03T14:43:51.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1086] Command: sudo true 2022-07-03T14:43:51.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1085] Received exit status 0 2022-07-03T14:43:51.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1085] Received channel close 2022-07-03T14:43:51.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1085] Channel closed 2022-07-03T14:43:51.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1086] Requesting new SSH session 2022-07-03T14:43:51.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1086] Command: sudo true 2022-07-03T14:43:51.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1086] Received exit status 0 2022-07-03T14:43:51.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1086] Received channel close 2022-07-03T14:43:51.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1086] Channel closed 2022-07-03T14:43:51.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1087] Requesting new SSH session 2022-07-03T14:43:51.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:51.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1086] Received exit status 0 2022-07-03T14:43:51.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1086] Received channel close 2022-07-03T14:43:51.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1086] Channel closed 2022-07-03T14:43:51.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1087] Requesting new SSH session 2022-07-03T14:43:51.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:51.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1087] Received exit status 0 2022-07-03T14:43:51.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1087] Received channel close 2022-07-03T14:43:51.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1087] Channel closed 2022-07-03T14:43:51.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1088] Requesting new SSH session 2022-07-03T14:43:51.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1088] Command: sudo true 2022-07-03T14:43:51.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1087] Received exit status 0 2022-07-03T14:43:51.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1087] Received channel close 2022-07-03T14:43:51.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1087] Channel closed 2022-07-03T14:43:51.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1088] Requesting new SSH session 2022-07-03T14:43:51.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1088] Command: sudo true 2022-07-03T14:43:51.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1088] Received exit status 0 2022-07-03T14:43:51.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1088] Received channel close 2022-07-03T14:43:51.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1088] Channel closed 2022-07-03T14:43:51.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1089] Requesting new SSH session 2022-07-03T14:43:51.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1089] Command: which python3 2022-07-03T14:43:51.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1088] Received exit status 0 2022-07-03T14:43:51.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1088] Received channel close 2022-07-03T14:43:51.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1088] Channel closed 2022-07-03T14:43:51.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1089] Requesting new SSH session 2022-07-03T14:43:51.417 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1089] Command: which python3 2022-07-03T14:43:51.417 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1089] Received exit status 0 2022-07-03T14:43:51.417 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1089] Received channel close 2022-07-03T14:43:51.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1089] Channel closed 2022-07-03T14:43:51.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1090] Requesting new SSH session 2022-07-03T14:43:51.419 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1090] Command: sudo true 2022-07-03T14:43:51.419 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1089] Received exit status 0 2022-07-03T14:43:51.419 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1089] Received channel close 2022-07-03T14:43:51.420 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1089] Channel closed 2022-07-03T14:43:51.420 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1090] Requesting new SSH session 2022-07-03T14:43:51.420 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1090] Command: sudo true 2022-07-03T14:43:51.421 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1090] Received exit status 0 2022-07-03T14:43:51.421 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1090] Received channel close 2022-07-03T14:43:51.421 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1090] Channel closed 2022-07-03T14:43:51.422 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1091] Requesting new SSH session 2022-07-03T14:43:51.422 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:51.422 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1090] Received exit status 0 2022-07-03T14:43:51.423 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1090] Received channel close 2022-07-03T14:43:51.423 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1090] Channel closed 2022-07-03T14:43:51.423 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1091] Requesting new SSH session 2022-07-03T14:43:51.424 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:51.424 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1091] Received exit status 0 2022-07-03T14:43:51.424 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1091] Received channel close 2022-07-03T14:43:51.425 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1091] Channel closed 2022-07-03T14:43:51.425 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1091] Received exit status 0 2022-07-03T14:43:51.425 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1091] Received channel close 2022-07-03T14:43:51.425 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1091] Channel closed 2022-07-03T14:43:51.426 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=395] Login timeout expired 2022-07-03T14:43:51.426 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=395] Aborting connection 2022-07-03T14:43:51.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:51 smithi188 ceph-mon[92425]: pgmap v7342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:51 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:43:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:43:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:43:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:43:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:43:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:43:51.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:43:51.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:43:51.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:43:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:43:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:43:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:43:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T14:43:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:43:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:43:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:43:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:43:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:43:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:43:51.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:43:51.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:43:51.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:43:51.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:43:51.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:43:51.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:43:51.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:43:51.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:43:51.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:43:51.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:43:51.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:43:51.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:43:51.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:43:51.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:43:51.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:43:51.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:43:51.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:43:51.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:43:51.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:43:51.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:43:51.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:43:51.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:43:51.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:43:51.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:43:51.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T14:43:51.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:43:51.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=395] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:43:51.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=395] Local address: 172.21.15.161, port 55018 2022-07-03T14:43:51.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=395] Peer address: 172.21.15.101, port 22 2022-07-03T14:43:51.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1084] Received exit status 0 2022-07-03T14:43:51.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1084] Received channel close 2022-07-03T14:43:51.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1084] Channel closed 2022-07-03T14:43:51.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1085] Requesting new SSH session 2022-07-03T14:43:51.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1085] Command: which python3 2022-07-03T14:43:51.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1084] Received exit status 0 2022-07-03T14:43:51.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1084] Received channel close 2022-07-03T14:43:51.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1084] Channel closed 2022-07-03T14:43:51.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1085] Requesting new SSH session 2022-07-03T14:43:51.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1085] Command: which python3 2022-07-03T14:43:51.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1085] Received exit status 0 2022-07-03T14:43:51.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1085] Received channel close 2022-07-03T14:43:51.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1085] Channel closed 2022-07-03T14:43:51.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1086] Requesting new SSH session 2022-07-03T14:43:51.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1086] Command: sudo true 2022-07-03T14:43:51.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1085] Received exit status 0 2022-07-03T14:43:51.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1085] Received channel close 2022-07-03T14:43:51.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1085] Channel closed 2022-07-03T14:43:51.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1086] Requesting new SSH session 2022-07-03T14:43:51.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1086] Command: sudo true 2022-07-03T14:43:51.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1086] Received exit status 0 2022-07-03T14:43:51.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1086] Received channel close 2022-07-03T14:43:51.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1086] Channel closed 2022-07-03T14:43:51.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1087] Requesting new SSH session 2022-07-03T14:43:51.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:51.452 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1086] Received exit status 0 2022-07-03T14:43:51.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1086] Received channel close 2022-07-03T14:43:51.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1086] Channel closed 2022-07-03T14:43:51.453 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1087] Requesting new SSH session 2022-07-03T14:43:51.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1087] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T14:43:51.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1087] Received exit status 0 2022-07-03T14:43:51.454 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1087] Received channel close 2022-07-03T14:43:51.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1087] Channel closed 2022-07-03T14:43:51.455 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1088] Requesting new SSH session 2022-07-03T14:43:51.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1088] Command: sudo true 2022-07-03T14:43:51.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1087] Received exit status 0 2022-07-03T14:43:51.456 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1087] Received channel close 2022-07-03T14:43:51.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1087] Channel closed 2022-07-03T14:43:51.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1088] Requesting new SSH session 2022-07-03T14:43:51.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1088] Command: sudo true 2022-07-03T14:43:51.457 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1088] Received exit status 0 2022-07-03T14:43:51.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1088] Received channel close 2022-07-03T14:43:51.458 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1088] Channel closed 2022-07-03T14:43:51.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1089] Requesting new SSH session 2022-07-03T14:43:51.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1089] Command: which python3 2022-07-03T14:43:51.459 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1088] Received exit status 0 2022-07-03T14:43:51.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1088] Received channel close 2022-07-03T14:43:51.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1088] Channel closed 2022-07-03T14:43:51.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1089] Requesting new SSH session 2022-07-03T14:43:51.460 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1089] Command: which python3 2022-07-03T14:43:51.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1089] Received exit status 0 2022-07-03T14:43:51.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1089] Received channel close 2022-07-03T14:43:51.461 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1089] Channel closed 2022-07-03T14:43:51.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1090] Requesting new SSH session 2022-07-03T14:43:51.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1090] Command: sudo true 2022-07-03T14:43:51.462 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1089] Received exit status 0 2022-07-03T14:43:51.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1089] Received channel close 2022-07-03T14:43:51.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1089] Channel closed 2022-07-03T14:43:51.463 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1090] Requesting new SSH session 2022-07-03T14:43:51.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1090] Command: sudo true 2022-07-03T14:43:51.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1090] Received exit status 0 2022-07-03T14:43:51.464 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1090] Received channel close 2022-07-03T14:43:51.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1090] Channel closed 2022-07-03T14:43:51.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1091] Requesting new SSH session 2022-07-03T14:43:51.465 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:51.466 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1090] Received exit status 0 2022-07-03T14:43:51.466 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1090] Received channel close 2022-07-03T14:43:51.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1090] Channel closed 2022-07-03T14:43:51.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1091] Requesting new SSH session 2022-07-03T14:43:51.467 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1091] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 --image quay.ceph.io/ceph-ci/ceph@sha256:2bea05871d2fede960bdb0dd84ef36efc745bc9e475973bd65c7675dc2475b6d list-networks 2022-07-03T14:43:51.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1091] Received exit status 0 2022-07-03T14:43:51.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1091] Received channel close 2022-07-03T14:43:51.468 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1091] Channel closed 2022-07-03T14:43:51.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1091] Received exit status 0 2022-07-03T14:43:51.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1091] Received channel close 2022-07-03T14:43:51.469 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1091] Channel closed 2022-07-03T14:43:51.470 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=395] Login timeout expired 2022-07-03T14:43:51.470 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=395] Aborting connection 2022-07-03T14:43:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:53 smithi161 ceph-mon[91579]: pgmap v7343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:53 smithi188 ceph-mon[92425]: pgmap v7343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:55 smithi161 ceph-mon[91579]: pgmap v7344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:55 smithi188 ceph-mon[92425]: pgmap v7344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:43:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:43:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:57 smithi161 ceph-mon[91579]: pgmap v7345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:57 smithi188 ceph-mon[92425]: pgmap v7345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:43:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:43:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:43:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:43:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:43:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:43:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:43:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:43:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:43:59 smithi161 ceph-mon[91579]: pgmap v7346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:43:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:43:59 smithi188 ceph-mon[92425]: pgmap v7346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:01 smithi161 ceph-mon[91579]: pgmap v7347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:01 smithi188 ceph-mon[92425]: pgmap v7347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:03 smithi161 ceph-mon[91579]: pgmap v7348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:03 smithi188 ceph-mon[92425]: pgmap v7348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:05 smithi161 ceph-mon[91579]: pgmap v7349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:05 smithi188 ceph-mon[92425]: pgmap v7349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:07 smithi161 ceph-mon[91579]: pgmap v7350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:07 smithi188 ceph-mon[92425]: pgmap v7350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:09 smithi161 ceph-mon[91579]: pgmap v7351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:09 smithi188 ceph-mon[92425]: pgmap v7351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:11 smithi161 ceph-mon[91579]: pgmap v7352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:11 smithi188 ceph-mon[92425]: pgmap v7352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:13 smithi161 ceph-mon[91579]: pgmap v7353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:13 smithi188 ceph-mon[92425]: pgmap v7353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:15 smithi161 ceph-mon[91579]: pgmap v7354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:15 smithi188 ceph-mon[92425]: pgmap v7354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:16 smithi161 ceph-mon[91579]: pgmap v7355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:16 smithi188 ceph-mon[92425]: pgmap v7355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:18 smithi161 ceph-mon[91579]: pgmap v7356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:18 smithi188 ceph-mon[92425]: pgmap v7356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:20 smithi161 ceph-mon[91579]: pgmap v7357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:20 smithi188 ceph-mon[92425]: pgmap v7357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:22 smithi161 ceph-mon[91579]: pgmap v7358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:22 smithi188 ceph-mon[92425]: pgmap v7358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:24 smithi161 ceph-mon[91579]: pgmap v7359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:24 smithi188 ceph-mon[92425]: pgmap v7359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:26 smithi188 ceph-mon[92425]: pgmap v7360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:26 smithi161 ceph-mon[91579]: pgmap v7360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:28 smithi161 ceph-mon[91579]: pgmap v7361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:28 smithi188 ceph-mon[92425]: pgmap v7361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:30 smithi161 ceph-mon[91579]: pgmap v7362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:30 smithi188 ceph-mon[92425]: pgmap v7362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:32 smithi161 ceph-mon[91579]: pgmap v7363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:32 smithi188 ceph-mon[92425]: pgmap v7363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:34 smithi161 ceph-mon[91579]: pgmap v7364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:34 smithi188 ceph-mon[92425]: pgmap v7364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:36 smithi161 ceph-mon[91579]: pgmap v7365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:36 smithi188 ceph-mon[92425]: pgmap v7365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:38 smithi161 ceph-mon[91579]: pgmap v7366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:38 smithi188 ceph-mon[92425]: pgmap v7366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:40 smithi161 ceph-mon[91579]: pgmap v7367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:40 smithi188 ceph-mon[92425]: pgmap v7367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:42 smithi161 ceph-mon[91579]: pgmap v7368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:42 smithi188 ceph-mon[92425]: pgmap v7368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:44 smithi161 ceph-mon[91579]: pgmap v7369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:44 smithi188 ceph-mon[92425]: pgmap v7369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:46 smithi161 ceph-mon[91579]: pgmap v7370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:46 smithi188 ceph-mon[92425]: pgmap v7370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:48 smithi161 ceph-mon[91579]: pgmap v7371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:48 smithi188 ceph-mon[92425]: pgmap v7371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:49.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:44:49.413+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:44:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:44:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:44:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:44:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:44:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:44:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:44:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:44:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:44:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:44:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:44:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:44:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:44:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:44:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:44:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:44:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:44:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:44:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:44:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:44:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:44:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:44:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:44:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:44:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:44:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:44:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:44:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:44:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:44:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:44:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:44:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:44:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:44:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:44:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:44:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:44:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:44:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:44:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:44:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:44:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:44:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:44:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:44:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:44:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:44:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=397] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:44:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=397] Local address: 172.21.15.161, port 55022 2022-07-03T14:44:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=397] Peer address: 172.21.15.101, port 22 2022-07-03T14:44:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=397] Connection lost 2022-07-03T14:44:49.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=397] Aborting connection 2022-07-03T14:44:49.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:44:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:44:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:44:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:44:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:44:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:44:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:44:50.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:50 smithi161 ceph-mon[91579]: pgmap v7372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:44:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:44:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:44:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:44:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:44:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:44:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:44:51.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:44:51.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:44:51.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:44:51.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:44:51.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:44:51.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:44:51.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:44:51.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:44:51.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:44:51.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:44:51.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:44:51.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:44:51.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:44:51.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:44:51.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:44:51.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:44:51.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:44:51.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:44:51.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:44:51.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:44:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:44:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:44:51.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:44:51.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:44:51.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:44:51.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:44:51.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:44:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:44:51.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:44:51.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:44:51.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:44:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:44:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:44:51.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:44:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:44:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:44:51.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:44:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:44:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:44:51.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=397] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:44:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=397] Local address: 172.21.15.161, port 55022 2022-07-03T14:44:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=397] Peer address: 172.21.15.101, port 22 2022-07-03T14:44:51.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=397] Connection lost 2022-07-03T14:44:51.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=397] Aborting connection 2022-07-03T14:44:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:50 smithi188 ceph-mon[92425]: pgmap v7372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:44:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:44:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:44:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:44:51.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:44:51.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:44:51.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:44:51.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:44:51.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:44:51.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:44:51.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:44:51.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:44:51.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:44:51.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:44:51.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:44:51.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:44:51.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:44:51.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:44:51.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:44:51.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:44:51.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:44:51.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:44:51.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:44:51.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:44:51.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:44:51.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:44:51.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:44:51.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:44:51.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:44:51.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:44:51.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:44:51.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:44:51.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:44:51.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:44:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:44:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:44:51.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:44:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:44:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:44:51.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:44:51.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:44:51.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:44:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:44:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:44:51.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:44:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:44:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=397] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:44:51.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=397] Local address: 172.21.15.161, port 55022 2022-07-03T14:44:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=397] Peer address: 172.21.15.101, port 22 2022-07-03T14:44:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=397] Connection lost 2022-07-03T14:44:51.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=397] Aborting connection 2022-07-03T14:44:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: pgmap v7373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: pgmap v7373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:44:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:44:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:44:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:44:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:44:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:44:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:44:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:44:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:44:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:44:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:55 smithi161 ceph-mon[91579]: pgmap v7374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:55 smithi188 ceph-mon[92425]: pgmap v7374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:44:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:44:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:57 smithi161 ceph-mon[91579]: pgmap v7375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:57 smithi188 ceph-mon[92425]: pgmap v7375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:44:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:44:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:44:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:44:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:44:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:44:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:44:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:44:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:44:59 smithi161 ceph-mon[91579]: pgmap v7376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:44:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:44:59 smithi188 ceph-mon[92425]: pgmap v7376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:01 smithi161 ceph-mon[91579]: pgmap v7377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:01 smithi188 ceph-mon[92425]: pgmap v7377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:03 smithi161 ceph-mon[91579]: pgmap v7378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:03 smithi188 ceph-mon[92425]: pgmap v7378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:05 smithi161 ceph-mon[91579]: pgmap v7379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:05 smithi188 ceph-mon[92425]: pgmap v7379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:07 smithi161 ceph-mon[91579]: pgmap v7380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:07 smithi188 ceph-mon[92425]: pgmap v7380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:09 smithi161 ceph-mon[91579]: pgmap v7381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:09 smithi188 ceph-mon[92425]: pgmap v7381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:11 smithi161 ceph-mon[91579]: pgmap v7382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:11 smithi188 ceph-mon[92425]: pgmap v7382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:13 smithi161 ceph-mon[91579]: pgmap v7383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:13 smithi188 ceph-mon[92425]: pgmap v7383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:14 smithi161 ceph-mon[91579]: pgmap v7384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:14 smithi188 ceph-mon[92425]: pgmap v7384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:16 smithi161 ceph-mon[91579]: pgmap v7385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:16 smithi188 ceph-mon[92425]: pgmap v7385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:18 smithi161 ceph-mon[91579]: pgmap v7386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:18 smithi188 ceph-mon[92425]: pgmap v7386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:20 smithi161 ceph-mon[91579]: pgmap v7387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:20 smithi188 ceph-mon[92425]: pgmap v7387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:22 smithi161 ceph-mon[91579]: pgmap v7388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:22 smithi188 ceph-mon[92425]: pgmap v7388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:24 smithi161 ceph-mon[91579]: pgmap v7389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:24 smithi188 ceph-mon[92425]: pgmap v7389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:26 smithi161 ceph-mon[91579]: pgmap v7390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:26 smithi188 ceph-mon[92425]: pgmap v7390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:28 smithi161 ceph-mon[91579]: pgmap v7391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:28 smithi188 ceph-mon[92425]: pgmap v7391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:30 smithi161 ceph-mon[91579]: pgmap v7392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:30 smithi188 ceph-mon[92425]: pgmap v7392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:32 smithi161 ceph-mon[91579]: pgmap v7393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:32 smithi188 ceph-mon[92425]: pgmap v7393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:34 smithi161 ceph-mon[91579]: pgmap v7394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:34 smithi188 ceph-mon[92425]: pgmap v7394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:36 smithi161 ceph-mon[91579]: pgmap v7395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:36 smithi188 ceph-mon[92425]: pgmap v7395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:38 smithi161 ceph-mon[91579]: pgmap v7396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:38 smithi188 ceph-mon[92425]: pgmap v7396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:40 smithi161 ceph-mon[91579]: pgmap v7397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:40 smithi188 ceph-mon[92425]: pgmap v7397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:42 smithi161 ceph-mon[91579]: pgmap v7398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:42 smithi188 ceph-mon[92425]: pgmap v7398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:44 smithi161 ceph-mon[91579]: pgmap v7399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:45.410 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:44 smithi188 ceph-mon[92425]: pgmap v7399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:46 smithi161 ceph-mon[91579]: pgmap v7400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:46 smithi188 ceph-mon[92425]: pgmap v7400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:48 smithi161 ceph-mon[91579]: pgmap v7401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:48 smithi188 ceph-mon[92425]: pgmap v7401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:50 smithi161 ceph-mon[91579]: pgmap v7402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:50 smithi188 ceph-mon[92425]: pgmap v7402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:52.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:45:52.578+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:45:52.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:45:52.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:45:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:45:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:45:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:45:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:45:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:45:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:45:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:45:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:45:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:45:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:45:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:45:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:45:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:45:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:45:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:45:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:45:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:45:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:45:52.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:45:52.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:45:52.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:45:52.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:45:52.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:45:52.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:45:52.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:45:52.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:45:52.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:45:52.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:45:52.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:45:52.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:45:52.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:45:52.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:45:52.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:45:52.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:45:52.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:45:52.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:45:52.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:45:52.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:45:52.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:45:52.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:45:52.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:45:52.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:45:52.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=399] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:45:52.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=399] Local address: 172.21.15.161, port 55026 2022-07-03T14:45:52.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=399] Peer address: 172.21.15.101, port 22 2022-07-03T14:45:52.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=399] Connection lost 2022-07-03T14:45:52.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=399] Aborting connection 2022-07-03T14:45:52.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:45:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-mon[91579]: pgmap v7403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:45:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:45:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:45:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:52 smithi188 ceph-mon[92425]: pgmap v7403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:45:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:45:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:45:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:45:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:45:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:45:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:45:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:45:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:45:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:45:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:45:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:45:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:45:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:45:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:45:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:45:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:45:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:45:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:45:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:45:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:45:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:45:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:45:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:45:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:45:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:45:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:45:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:45:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:45:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:45:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:45:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:45:54.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:45:54.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:45:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:45:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:45:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:45:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:45:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:45:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:45:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:45:54.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:45:54.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:45:54.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:45:54.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:45:54.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:45:54.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:45:54.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:45:54.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:45:54.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=399] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:45:54.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=399] Local address: 172.21.15.161, port 55026 2022-07-03T14:45:54.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=399] Peer address: 172.21.15.101, port 22 2022-07-03T14:45:54.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=399] Connection lost 2022-07-03T14:45:54.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=399] Aborting connection 2022-07-03T14:45:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:45:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:45:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:45:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:45:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:45:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:45:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:45:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:45:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:45:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:45:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:45:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:45:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:45:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:45:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:45:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:45:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:45:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:45:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:45:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:45:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:45:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:45:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:45:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:45:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:45:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:45:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:45:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:45:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:45:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:45:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:45:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:45:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:45:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:45:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:45:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:45:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:45:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:45:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:45:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:45:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:45:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:45:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:45:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:45:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:45:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:45:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=399] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:45:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=399] Local address: 172.21.15.161, port 55026 2022-07-03T14:45:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=399] Peer address: 172.21.15.101, port 22 2022-07-03T14:45:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=399] Connection lost 2022-07-03T14:45:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=399] Aborting connection 2022-07-03T14:45:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:54 smithi161 ceph-mon[91579]: pgmap v7404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:54 smithi188 ceph-mon[92425]: pgmap v7404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:45:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:45:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:56 smithi161 ceph-mon[91579]: pgmap v7405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:56 smithi188 ceph-mon[92425]: pgmap v7405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:45:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:45:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:45:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:45:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:45:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:45:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:45:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:45:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:45:58 smithi161 ceph-mon[91579]: pgmap v7406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:45:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:45:58 smithi188 ceph-mon[92425]: pgmap v7406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:01 smithi161 ceph-mon[91579]: pgmap v7407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:01 smithi188 ceph-mon[92425]: pgmap v7407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:03 smithi161 ceph-mon[91579]: pgmap v7408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:03 smithi188 ceph-mon[92425]: pgmap v7408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:05 smithi161 ceph-mon[91579]: pgmap v7409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:05 smithi188 ceph-mon[92425]: pgmap v7409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:07 smithi161 ceph-mon[91579]: pgmap v7410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:07 smithi188 ceph-mon[92425]: pgmap v7410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:09 smithi161 ceph-mon[91579]: pgmap v7411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:09 smithi188 ceph-mon[92425]: pgmap v7411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:11 smithi161 ceph-mon[91579]: pgmap v7412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:11 smithi188 ceph-mon[92425]: pgmap v7412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:13 smithi161 ceph-mon[91579]: pgmap v7413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:13 smithi188 ceph-mon[92425]: pgmap v7413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:15 smithi161 ceph-mon[91579]: pgmap v7414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:15 smithi188 ceph-mon[92425]: pgmap v7414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:17 smithi161 ceph-mon[91579]: pgmap v7415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:17 smithi188 ceph-mon[92425]: pgmap v7415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:19 smithi161 ceph-mon[91579]: pgmap v7416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:19 smithi188 ceph-mon[92425]: pgmap v7416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:21 smithi161 ceph-mon[91579]: pgmap v7417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:21 smithi188 ceph-mon[92425]: pgmap v7417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:23 smithi161 ceph-mon[91579]: pgmap v7418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:23 smithi188 ceph-mon[92425]: pgmap v7418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:25 smithi161 ceph-mon[91579]: pgmap v7419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:25 smithi188 ceph-mon[92425]: pgmap v7419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:27 smithi161 ceph-mon[91579]: pgmap v7420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:27 smithi188 ceph-mon[92425]: pgmap v7420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:29 smithi161 ceph-mon[91579]: pgmap v7421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:29 smithi188 ceph-mon[92425]: pgmap v7421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:31 smithi161 ceph-mon[91579]: pgmap v7422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:31 smithi188 ceph-mon[92425]: pgmap v7422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:33 smithi161 ceph-mon[91579]: pgmap v7423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:33 smithi188 ceph-mon[92425]: pgmap v7423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:35 smithi161 ceph-mon[91579]: pgmap v7424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:35 smithi188 ceph-mon[92425]: pgmap v7424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:37 smithi161 ceph-mon[91579]: pgmap v7425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:37 smithi188 ceph-mon[92425]: pgmap v7425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:39 smithi161 ceph-mon[91579]: pgmap v7426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:39 smithi188 ceph-mon[92425]: pgmap v7426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:41 smithi161 ceph-mon[91579]: pgmap v7427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:41 smithi188 ceph-mon[92425]: pgmap v7427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:43 smithi161 ceph-mon[91579]: pgmap v7428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:43 smithi188 ceph-mon[92425]: pgmap v7428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:45 smithi161 ceph-mon[91579]: pgmap v7429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:45 smithi188 ceph-mon[92425]: pgmap v7429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:47 smithi161 ceph-mon[91579]: pgmap v7430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:47 smithi188 ceph-mon[92425]: pgmap v7430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:49 smithi161 ceph-mon[91579]: pgmap v7431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:49 smithi188 ceph-mon[92425]: pgmap v7431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:51 smithi161 ceph-mon[91579]: pgmap v7432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:51 smithi188 ceph-mon[92425]: pgmap v7432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-mon[91579]: pgmap v7433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:46:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:46:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:46:53.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:46:53.080+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:46:53.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:46:53.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:46:53.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:46:53.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:46:53.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:46:53.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:46:53.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:46:53.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:46:53.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:46:53.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:46:53.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:46:53.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:46:53.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:46:53.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:46:53.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:46:53.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:46:53.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:46:53.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:46:53.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:46:53.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:46:53.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:46:53.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:46:53.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:46:53.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:46:53.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:46:53.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:46:53.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:46:53.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:46:53.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:46:53.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:46:53.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:46:53.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:46:53.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:46:53.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:46:53.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:46:53.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:46:53.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:46:53.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:46:53.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:46:53.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:46:53.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:46:53.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:46:53.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:46:53.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:46:53.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=401] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:46:53.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=401] Local address: 172.21.15.161, port 55030 2022-07-03T14:46:53.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=401] Peer address: 172.21.15.101, port 22 2022-07-03T14:46:53.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=401] Connection lost 2022-07-03T14:46:53.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=401] Aborting connection 2022-07-03T14:46:53.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:46:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:46:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:53 smithi188 ceph-mon[92425]: pgmap v7433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:46:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:46:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:46:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:46:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:46:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:46:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:46:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:46:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:46:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:46:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:46:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:46:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:46:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:46:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:46:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:46:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:46:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:46:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:46:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:46:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:46:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:46:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:46:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:46:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:46:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:46:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:46:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:46:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:46:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:46:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:46:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:46:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:46:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:46:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:46:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:46:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:46:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:46:54.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:46:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:46:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:46:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:46:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:46:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:46:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:46:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:46:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:46:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:46:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:46:54.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=401] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:46:54.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=401] Local address: 172.21.15.161, port 55030 2022-07-03T14:46:54.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=401] Peer address: 172.21.15.101, port 22 2022-07-03T14:46:54.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=401] Connection lost 2022-07-03T14:46:54.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=401] Aborting connection 2022-07-03T14:46:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:46:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:46:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:46:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:46:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:46:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:46:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:46:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:46:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:46:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:46:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:46:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:46:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:46:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:46:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:46:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:46:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:46:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:46:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:46:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:46:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:46:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:46:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:46:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:46:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:46:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:46:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:46:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:46:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:46:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:46:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:46:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:46:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:46:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:46:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:46:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:46:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:46:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:46:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:46:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:46:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:46:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:46:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:46:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:46:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:46:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:46:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=401] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:46:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=401] Local address: 172.21.15.161, port 55030 2022-07-03T14:46:54.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=401] Peer address: 172.21.15.101, port 22 2022-07-03T14:46:54.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=401] Connection lost 2022-07-03T14:46:54.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=401] Aborting connection 2022-07-03T14:46:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:55 smithi161 ceph-mon[91579]: pgmap v7434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:55 smithi188 ceph-mon[92425]: pgmap v7434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:46:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:46:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:57 smithi161 ceph-mon[91579]: pgmap v7435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:57 smithi188 ceph-mon[92425]: pgmap v7435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:46:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:46:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:46:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:46:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:46:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:46:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:46:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:46:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:46:59 smithi161 ceph-mon[91579]: pgmap v7436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:46:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:46:59 smithi188 ceph-mon[92425]: pgmap v7436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:01 smithi161 ceph-mon[91579]: pgmap v7437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:01 smithi188 ceph-mon[92425]: pgmap v7437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:03 smithi161 ceph-mon[91579]: pgmap v7438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:03 smithi188 ceph-mon[92425]: pgmap v7438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:04 smithi161 ceph-mon[91579]: pgmap v7439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:04 smithi188 ceph-mon[92425]: pgmap v7439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:06 smithi161 ceph-mon[91579]: pgmap v7440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:06 smithi188 ceph-mon[92425]: pgmap v7440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:08 smithi161 ceph-mon[91579]: pgmap v7441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:08 smithi188 ceph-mon[92425]: pgmap v7441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:10 smithi161 ceph-mon[91579]: pgmap v7442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:10 smithi188 ceph-mon[92425]: pgmap v7442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:12.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:12 smithi161 ceph-mon[91579]: pgmap v7443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:12 smithi188 ceph-mon[92425]: pgmap v7443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:14.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:14 smithi161 ceph-mon[91579]: pgmap v7444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:14 smithi188 ceph-mon[92425]: pgmap v7444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:16 smithi161 ceph-mon[91579]: pgmap v7445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:16 smithi188 ceph-mon[92425]: pgmap v7445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:18 smithi161 ceph-mon[91579]: pgmap v7446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:18 smithi188 ceph-mon[92425]: pgmap v7446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:20 smithi161 ceph-mon[91579]: pgmap v7447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:20 smithi188 ceph-mon[92425]: pgmap v7447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:22 smithi161 ceph-mon[91579]: pgmap v7448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:22 smithi188 ceph-mon[92425]: pgmap v7448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:24 smithi161 ceph-mon[91579]: pgmap v7449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:24 smithi188 ceph-mon[92425]: pgmap v7449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:26 smithi161 ceph-mon[91579]: pgmap v7450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:26 smithi188 ceph-mon[92425]: pgmap v7450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:28 smithi161 ceph-mon[91579]: pgmap v7451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:28 smithi188 ceph-mon[92425]: pgmap v7451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:30 smithi161 ceph-mon[91579]: pgmap v7452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:30 smithi188 ceph-mon[92425]: pgmap v7452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:32 smithi161 ceph-mon[91579]: pgmap v7453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:32 smithi188 ceph-mon[92425]: pgmap v7453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:34 smithi161 ceph-mon[91579]: pgmap v7454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:34 smithi188 ceph-mon[92425]: pgmap v7454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:36 smithi161 ceph-mon[91579]: pgmap v7455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:36 smithi188 ceph-mon[92425]: pgmap v7455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:38 smithi161 ceph-mon[91579]: pgmap v7456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:38 smithi188 ceph-mon[92425]: pgmap v7456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:40 smithi161 ceph-mon[91579]: pgmap v7457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:40 smithi188 ceph-mon[92425]: pgmap v7457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:43 smithi161 ceph-mon[91579]: pgmap v7458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:43 smithi188 ceph-mon[92425]: pgmap v7458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:45 smithi161 ceph-mon[91579]: pgmap v7459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:45 smithi188 ceph-mon[92425]: pgmap v7459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:47 smithi161 ceph-mon[91579]: pgmap v7460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:47 smithi188 ceph-mon[92425]: pgmap v7460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:49 smithi161 ceph-mon[91579]: pgmap v7461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:49 smithi188 ceph-mon[92425]: pgmap v7461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:51 smithi161 ceph-mon[91579]: pgmap v7462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:51 smithi188 ceph-mon[92425]: pgmap v7462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-mon[91579]: pgmap v7463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:53 smithi188 ceph-mon[92425]: pgmap v7463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:47:53.582+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:47:53.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:47:53.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:47:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:47:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:47:53.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:47:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:47:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:47:53.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:47:53.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:47:53.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:47:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:47:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:47:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:47:53.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:47:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:47:53.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:47:53.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:47:53.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:47:53.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:47:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:47:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:47:53.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:47:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:47:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:47:53.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:47:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:47:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:47:53.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:47:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:47:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:47:53.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:47:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:47:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:47:53.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:47:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:47:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:47:53.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:47:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:47:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:47:53.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:47:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:47:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:47:53.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:47:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:47:53.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=403] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:47:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=403] Local address: 172.21.15.161, port 55034 2022-07-03T14:47:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=403] Peer address: 172.21.15.101, port 22 2022-07-03T14:47:53.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=403] Connection lost 2022-07-03T14:47:53.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=403] Aborting connection 2022-07-03T14:47:53.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:47:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:47:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:47:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:47:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:47:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:47:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:47:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:47:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:55 smithi161 ceph-mon[91579]: pgmap v7464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:47:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:47:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:47:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:47:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:47:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:47:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:47:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:47:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:47:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:47:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:47:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:47:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:47:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:47:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:47:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:47:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:47:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:47:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:47:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:47:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:47:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:47:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:47:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:47:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:47:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:47:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:47:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:47:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:47:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:47:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:47:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:47:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:47:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:47:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:47:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:47:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:47:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:47:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:47:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:47:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:47:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:47:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:47:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:47:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:47:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:47:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=403] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:47:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=403] Local address: 172.21.15.161, port 55034 2022-07-03T14:47:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=403] Peer address: 172.21.15.101, port 22 2022-07-03T14:47:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=403] Connection lost 2022-07-03T14:47:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=403] Aborting connection 2022-07-03T14:47:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:55 smithi188 ceph-mon[92425]: pgmap v7464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:47:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:47:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:47:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:47:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:47:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:47:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:47:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:47:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:47:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:47:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:47:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:47:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:47:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:47:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:47:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:47:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:47:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:47:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:47:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:47:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:47:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:47:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:47:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:47:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:47:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:47:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:47:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:47:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:47:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:47:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:47:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:47:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:47:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:47:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:47:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:47:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:47:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:47:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:47:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:47:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:47:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:47:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:47:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:47:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:47:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:47:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:47:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=403] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:47:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=403] Local address: 172.21.15.161, port 55034 2022-07-03T14:47:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=403] Peer address: 172.21.15.101, port 22 2022-07-03T14:47:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=403] Connection lost 2022-07-03T14:47:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=403] Aborting connection 2022-07-03T14:47:55.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:47:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:57 smithi161 ceph-mon[91579]: pgmap v7465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:57 smithi188 ceph-mon[92425]: pgmap v7465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:47:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:47:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:47:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:47:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:47:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:47:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:47:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:47:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:47:59 smithi161 ceph-mon[91579]: pgmap v7466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:47:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:47:59 smithi188 ceph-mon[92425]: pgmap v7466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:01 smithi161 ceph-mon[91579]: pgmap v7467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:01 smithi188 ceph-mon[92425]: pgmap v7467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:03 smithi161 ceph-mon[91579]: pgmap v7468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:03 smithi188 ceph-mon[92425]: pgmap v7468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:05 smithi161 ceph-mon[91579]: pgmap v7469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:05 smithi188 ceph-mon[92425]: pgmap v7469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:07 smithi161 ceph-mon[91579]: pgmap v7470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:07 smithi188 ceph-mon[92425]: pgmap v7470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:09 smithi161 ceph-mon[91579]: pgmap v7471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:09 smithi188 ceph-mon[92425]: pgmap v7471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:11 smithi161 ceph-mon[91579]: pgmap v7472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:11 smithi188 ceph-mon[92425]: pgmap v7472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:13 smithi161 ceph-mon[91579]: pgmap v7473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:13 smithi188 ceph-mon[92425]: pgmap v7473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:15 smithi161 ceph-mon[91579]: pgmap v7474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:15 smithi188 ceph-mon[92425]: pgmap v7474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:17 smithi161 ceph-mon[91579]: pgmap v7475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:17 smithi188 ceph-mon[92425]: pgmap v7475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:19 smithi161 ceph-mon[91579]: pgmap v7476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:19 smithi188 ceph-mon[92425]: pgmap v7476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:21 smithi161 ceph-mon[91579]: pgmap v7477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:21 smithi188 ceph-mon[92425]: pgmap v7477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:23 smithi161 ceph-mon[91579]: pgmap v7478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:23 smithi188 ceph-mon[92425]: pgmap v7478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:25 smithi161 ceph-mon[91579]: pgmap v7479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:25 smithi188 ceph-mon[92425]: pgmap v7479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:27 smithi161 ceph-mon[91579]: pgmap v7480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:27 smithi188 ceph-mon[92425]: pgmap v7480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:29 smithi161 ceph-mon[91579]: pgmap v7481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:29 smithi188 ceph-mon[92425]: pgmap v7481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:31 smithi161 ceph-mon[91579]: pgmap v7482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:31 smithi188 ceph-mon[92425]: pgmap v7482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:33 smithi161 ceph-mon[91579]: pgmap v7483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:33 smithi188 ceph-mon[92425]: pgmap v7483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:35 smithi161 ceph-mon[91579]: pgmap v7484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:35 smithi188 ceph-mon[92425]: pgmap v7484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:37 smithi161 ceph-mon[91579]: pgmap v7485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:37 smithi188 ceph-mon[92425]: pgmap v7485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:39 smithi188 ceph-mon[92425]: pgmap v7486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:39 smithi161 ceph-mon[91579]: pgmap v7486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:41 smithi188 ceph-mon[92425]: pgmap v7487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:41 smithi161 ceph-mon[91579]: pgmap v7487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:43 smithi188 ceph-mon[92425]: pgmap v7488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:43 smithi161 ceph-mon[91579]: pgmap v7488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:45 smithi188 ceph-mon[92425]: pgmap v7489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:45 smithi161 ceph-mon[91579]: pgmap v7489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:47 smithi188 ceph-mon[92425]: pgmap v7490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:47 smithi161 ceph-mon[91579]: pgmap v7490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:48.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:48 smithi188 ceph-mon[92425]: pgmap v7491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:48 smithi161 ceph-mon[91579]: pgmap v7491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:50 smithi161 ceph-mon[91579]: pgmap v7492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:50 smithi188 ceph-mon[92425]: pgmap v7492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:52 smithi161 ceph-mon[91579]: pgmap v7493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:52 smithi188 ceph-mon[92425]: pgmap v7493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: pgmap v7494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:48:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:48:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:48:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:48:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:48:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:48:54.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:48:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:48:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:48:54.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:48:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:48:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:48:54.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:48:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:48:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:48:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:48:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:48:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:48:54.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:48:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:48:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:48:54.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:48:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:48:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:48:54.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:48:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:48:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:48:54.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:48:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:48:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:48:54.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:48:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:48:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:48:54.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:48:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:48:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:48:54.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:48:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:48:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:48:54.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:48:54.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:48:54.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:48:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:48:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:48:54.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:48:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:48:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:48:54.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:48:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:48:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=405] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:48:54.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=405] Local address: 172.21.15.161, port 55050 2022-07-03T14:48:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=405] Peer address: 172.21.15.101, port 22 2022-07-03T14:48:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=405] Connection lost 2022-07-03T14:48:54.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=405] Aborting connection 2022-07-03T14:48:54.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:48:54.080+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:48:54.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:48:54.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:48:54.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:48:54.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:48:54.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:48:54.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:48:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:48:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:48:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:48:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:48:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:48:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:48:54.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:48:54.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:48:54.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:48:54.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:48:54.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:48:54.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:48:54.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:48:54.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:48:54.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:48:54.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:48:54.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:48:54.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:48:54.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:48:54.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:48:54.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:48:54.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:48:54.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:48:54.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:48:54.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:48:54.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:48:54.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:48:54.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:48:54.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:48:54.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:48:54.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:48:54.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:48:54.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:48:54.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:48:54.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:48:54.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:48:54.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:48:54.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:48:54.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=405] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:48:54.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=405] Local address: 172.21.15.161, port 55050 2022-07-03T14:48:54.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=405] Peer address: 172.21.15.101, port 22 2022-07-03T14:48:54.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=405] Connection lost 2022-07-03T14:48:54.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=405] Aborting connection 2022-07-03T14:48:54.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:48:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:48:54.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: pgmap v7494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:48:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:48:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:48:54.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:48:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:48:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:48:54.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:48:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:48:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:48:54.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:48:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:48:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:48:54.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:48:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:48:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:48:54.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:48:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:48:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:48:54.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:48:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:48:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:48:54.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:48:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:48:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:48:54.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:48:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:48:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:48:54.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:48:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:48:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:48:54.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:48:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:48:54.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:48:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:48:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:48:54.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:48:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:48:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:48:54.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:48:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:48:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:48:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:48:54.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:48:54.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:48:54.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:48:54.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:48:54.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:48:54.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:48:54.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:48:54.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:48:54.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=405] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:48:54.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=405] Local address: 172.21.15.161, port 55050 2022-07-03T14:48:54.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=405] Peer address: 172.21.15.101, port 22 2022-07-03T14:48:54.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=405] Connection lost 2022-07-03T14:48:54.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=405] Aborting connection 2022-07-03T14:48:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:48:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:48:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:56 smithi161 ceph-mon[91579]: pgmap v7495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:56 smithi188 ceph-mon[92425]: pgmap v7495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:48:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:48:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:48:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:48:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:48:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:48:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:48:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:48:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:58 smithi161 ceph-mon[91579]: pgmap v7496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:48:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:58 smithi188 ceph-mon[92425]: pgmap v7496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:00 smithi161 ceph-mon[91579]: pgmap v7497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:00 smithi188 ceph-mon[92425]: pgmap v7497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:02 smithi161 ceph-mon[91579]: pgmap v7498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:02 smithi188 ceph-mon[92425]: pgmap v7498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:04 smithi161 ceph-mon[91579]: pgmap v7499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:04 smithi188 ceph-mon[92425]: pgmap v7499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:07 smithi161 ceph-mon[91579]: pgmap v7500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:07 smithi188 ceph-mon[92425]: pgmap v7500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:09 smithi161 ceph-mon[91579]: pgmap v7501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:09 smithi188 ceph-mon[92425]: pgmap v7501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:11 smithi161 ceph-mon[91579]: pgmap v7502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:11 smithi188 ceph-mon[92425]: pgmap v7502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:13 smithi161 ceph-mon[91579]: pgmap v7503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:13 smithi188 ceph-mon[92425]: pgmap v7503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:15 smithi161 ceph-mon[91579]: pgmap v7504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:15 smithi188 ceph-mon[92425]: pgmap v7504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:17 smithi161 ceph-mon[91579]: pgmap v7505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:17 smithi188 ceph-mon[92425]: pgmap v7505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:19 smithi161 ceph-mon[91579]: pgmap v7506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:19 smithi188 ceph-mon[92425]: pgmap v7506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:21 smithi161 ceph-mon[91579]: pgmap v7507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:21 smithi188 ceph-mon[92425]: pgmap v7507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:23 smithi161 ceph-mon[91579]: pgmap v7508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:23 smithi188 ceph-mon[92425]: pgmap v7508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:25 smithi161 ceph-mon[91579]: pgmap v7509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:25 smithi188 ceph-mon[92425]: pgmap v7509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:27 smithi161 ceph-mon[91579]: pgmap v7510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:27 smithi188 ceph-mon[92425]: pgmap v7510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:29 smithi161 ceph-mon[91579]: pgmap v7511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:29 smithi188 ceph-mon[92425]: pgmap v7511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:31 smithi161 ceph-mon[91579]: pgmap v7512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:31 smithi188 ceph-mon[92425]: pgmap v7512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:33 smithi161 ceph-mon[91579]: pgmap v7513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:33 smithi188 ceph-mon[92425]: pgmap v7513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:35 smithi161 ceph-mon[91579]: pgmap v7514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:35 smithi188 ceph-mon[92425]: pgmap v7514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:37 smithi161 ceph-mon[91579]: pgmap v7515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:37 smithi188 ceph-mon[92425]: pgmap v7515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:39 smithi161 ceph-mon[91579]: pgmap v7516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:39 smithi188 ceph-mon[92425]: pgmap v7516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:41 smithi161 ceph-mon[91579]: pgmap v7517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:41 smithi188 ceph-mon[92425]: pgmap v7517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:43 smithi161 ceph-mon[91579]: pgmap v7518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:43 smithi188 ceph-mon[92425]: pgmap v7518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:45 smithi161 ceph-mon[91579]: pgmap v7519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:45 smithi188 ceph-mon[92425]: pgmap v7519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:47 smithi161 ceph-mon[91579]: pgmap v7520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:47 smithi188 ceph-mon[92425]: pgmap v7520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:49 smithi161 ceph-mon[91579]: pgmap v7521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:49 smithi188 ceph-mon[92425]: pgmap v7521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:51 smithi161 ceph-mon[91579]: pgmap v7522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:51 smithi188 ceph-mon[92425]: pgmap v7522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:53 smithi161 ceph-mon[91579]: pgmap v7523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:53 smithi188 ceph-mon[92425]: pgmap v7523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:54.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:49:54.583+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:49:54.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:49:54.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:49:54.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:49:54.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:49:54.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:49:54.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:49:54.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:49:54.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:49:54.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:49:54.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:49:54.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:49:54.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:49:54.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:49:54.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:49:54.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:49:54.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:49:54.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:49:54.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:49:54.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:49:54.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:49:54.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:49:54.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:49:54.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:49:54.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:49:54.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:49:54.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:49:54.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:49:54.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:49:54.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:49:54.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:49:54.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:49:54.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:49:54.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:49:54.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:49:54.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:49:54.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:49:54.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:49:54.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:49:54.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:49:54.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:49:54.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:49:54.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:49:54.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:49:54.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:49:54.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=407] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:49:54.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=407] Local address: 172.21.15.161, port 55078 2022-07-03T14:49:54.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=407] Peer address: 172.21.15.101, port 22 2022-07-03T14:49:54.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=407] Connection lost 2022-07-03T14:49:54.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=407] Aborting connection 2022-07-03T14:49:54.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:49:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:49:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:55 smithi161 ceph-mon[91579]: pgmap v7524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:49:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:49:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:49:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:55 smithi188 ceph-mon[92425]: pgmap v7524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:49:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:49:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:49:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:49:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:49:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:49:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:49:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:49:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:49:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:49:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:49:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:49:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:49:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:49:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:49:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:49:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:49:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:49:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:49:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:49:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:49:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:49:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:49:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:49:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:49:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:49:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:49:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:49:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:49:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:49:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:49:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:49:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:49:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:49:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:49:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:49:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:49:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:49:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:49:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:49:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:49:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:49:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:49:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:49:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:49:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:49:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:49:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:49:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:49:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:49:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:49:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=407] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:49:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=407] Local address: 172.21.15.161, port 55078 2022-07-03T14:49:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=407] Peer address: 172.21.15.101, port 22 2022-07-03T14:49:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=407] Connection lost 2022-07-03T14:49:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=407] Aborting connection 2022-07-03T14:49:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:49:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:49:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:49:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:49:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:49:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:49:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:49:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:49:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:49:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:49:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:49:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:49:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:49:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:49:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:49:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:49:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:49:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:49:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:49:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:49:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:49:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:49:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:49:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:49:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:49:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:49:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:49:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:49:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:49:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:49:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:49:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:49:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:49:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:49:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:49:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:49:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:49:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:49:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:49:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:49:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:49:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:49:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:49:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:49:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:49:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:49:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=407] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:49:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=407] Local address: 172.21.15.161, port 55078 2022-07-03T14:49:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=407] Peer address: 172.21.15.101, port 22 2022-07-03T14:49:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=407] Connection lost 2022-07-03T14:49:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=407] Aborting connection 2022-07-03T14:49:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:49:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:57 smithi161 ceph-mon[91579]: pgmap v7525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:57 smithi188 ceph-mon[92425]: pgmap v7525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:49:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:49:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:49:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:49:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:49:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:49:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:49:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:49:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:49:59 smithi161 ceph-mon[91579]: pgmap v7526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:49:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:49:59 smithi188 ceph-mon[92425]: pgmap v7526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:50:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T14:50:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:01 smithi161 ceph-mon[91579]: pgmap v7527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:01 smithi188 ceph-mon[92425]: pgmap v7527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:03 smithi161 ceph-mon[91579]: pgmap v7528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:03 smithi188 ceph-mon[92425]: pgmap v7528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:05 smithi188 ceph-mon[92425]: pgmap v7529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:05 smithi161 ceph-mon[91579]: pgmap v7529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:07 smithi188 ceph-mon[92425]: pgmap v7530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:07 smithi161 ceph-mon[91579]: pgmap v7530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:09 smithi188 ceph-mon[92425]: pgmap v7531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:09 smithi161 ceph-mon[91579]: pgmap v7531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:11 smithi188 ceph-mon[92425]: pgmap v7532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:11 smithi161 ceph-mon[91579]: pgmap v7532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:13 smithi188 ceph-mon[92425]: pgmap v7533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:13 smithi161 ceph-mon[91579]: pgmap v7533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:15 smithi188 ceph-mon[92425]: pgmap v7534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:15 smithi161 ceph-mon[91579]: pgmap v7534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:17 smithi188 ceph-mon[92425]: pgmap v7535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:17 smithi161 ceph-mon[91579]: pgmap v7535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:19 smithi188 ceph-mon[92425]: pgmap v7536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:19 smithi161 ceph-mon[91579]: pgmap v7536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:21 smithi161 ceph-mon[91579]: pgmap v7537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:21 smithi188 ceph-mon[92425]: pgmap v7537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:22 smithi161 ceph-mon[91579]: pgmap v7538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:22 smithi188 ceph-mon[92425]: pgmap v7538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:24.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:24 smithi161 ceph-mon[91579]: pgmap v7539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:24 smithi188 ceph-mon[92425]: pgmap v7539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:26 smithi161 ceph-mon[91579]: pgmap v7540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:26 smithi188 ceph-mon[92425]: pgmap v7540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:29 smithi161 ceph-mon[91579]: pgmap v7541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:29 smithi188 ceph-mon[92425]: pgmap v7541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:31 smithi161 ceph-mon[91579]: pgmap v7542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:31 smithi188 ceph-mon[92425]: pgmap v7542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:33 smithi161 ceph-mon[91579]: pgmap v7543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:33 smithi188 ceph-mon[92425]: pgmap v7543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:35 smithi161 ceph-mon[91579]: pgmap v7544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:35 smithi188 ceph-mon[92425]: pgmap v7544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:36.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:37 smithi161 ceph-mon[91579]: pgmap v7545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:37 smithi188 ceph-mon[92425]: pgmap v7545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:39 smithi161 ceph-mon[91579]: pgmap v7546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:39 smithi188 ceph-mon[92425]: pgmap v7546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:41 smithi161 ceph-mon[91579]: pgmap v7547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:41 smithi188 ceph-mon[92425]: pgmap v7547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:43 smithi161 ceph-mon[91579]: pgmap v7548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:43 smithi188 ceph-mon[92425]: pgmap v7548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:45 smithi161 ceph-mon[91579]: pgmap v7549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:45 smithi188 ceph-mon[92425]: pgmap v7549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:46.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:46.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:47 smithi161 ceph-mon[91579]: pgmap v7550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:47 smithi188 ceph-mon[92425]: pgmap v7550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:49 smithi161 ceph-mon[91579]: pgmap v7551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:49 smithi188 ceph-mon[92425]: pgmap v7551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:51 smithi161 ceph-mon[91579]: pgmap v7552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:51 smithi188 ceph-mon[92425]: pgmap v7552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:53 smithi161 ceph-mon[91579]: pgmap v7553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:53 smithi188 ceph-mon[92425]: pgmap v7553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-mon[91579]: pgmap v7554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:50:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:50:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:50:55.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:50:55.084+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:50:55.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:50:55.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:50:55.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:50:55.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:50:55.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:50:55.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:50:55.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:50:55.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:50:55.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:50:55.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:50:55.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:50:55.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:50:55.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:50:55.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:50:55.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:50:55.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:50:55.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:50:55.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:50:55.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:50:55.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:50:55.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:50:55.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:50:55.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:50:55.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:50:55.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:50:55.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:50:55.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:50:55.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:50:55.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:50:55.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:50:55.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:50:55.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:50:55.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:50:55.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:50:55.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:50:55.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:50:55.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:50:55.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:50:55.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:50:55.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:50:55.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:50:55.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:50:55.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:50:55.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:50:55.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=409] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:50:55.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=409] Local address: 172.21.15.161, port 55106 2022-07-03T14:50:55.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=409] Peer address: 172.21.15.101, port 22 2022-07-03T14:50:55.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=409] Connection lost 2022-07-03T14:50:55.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=409] Aborting connection 2022-07-03T14:50:55.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:50:55.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:50:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:50:55.230+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T14:50:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:55 smithi188 ceph-mon[92425]: pgmap v7554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:50:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:50:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:50:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:50:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:50:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:50:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:50:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:50:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:50:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:50:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:50:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:50:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:50:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:50:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:50:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:50:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:50:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:50:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:50:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:50:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:50:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:50:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:50:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:50:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:50:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:50:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:50:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:50:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:50:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:50:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:50:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:50:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:50:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:50:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:50:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:50:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:50:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:50:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:50:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:50:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:50:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:50:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:50:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:50:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:50:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:50:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:50:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:50:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:50:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:50:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:50:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=409] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:50:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=409] Local address: 172.21.15.161, port 55106 2022-07-03T14:50:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=409] Peer address: 172.21.15.101, port 22 2022-07-03T14:50:56.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=409] Connection lost 2022-07-03T14:50:56.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=409] Aborting connection 2022-07-03T14:50:56.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:50:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:50:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:50:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:50:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:50:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:50:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:50:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:50:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:50:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:50:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:50:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:50:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:50:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:50:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:50:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:50:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:50:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:50:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:50:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:50:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:50:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:50:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:50:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:50:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:50:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:50:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:50:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:50:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:50:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:50:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:50:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:50:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:50:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:50:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:50:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:50:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:50:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:50:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:50:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:50:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:50:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:50:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:50:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:50:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:50:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:50:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=409] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:50:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=409] Local address: 172.21.15.161, port 55106 2022-07-03T14:50:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=409] Peer address: 172.21.15.101, port 22 2022-07-03T14:50:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=409] Connection lost 2022-07-03T14:50:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=409] Aborting connection 2022-07-03T14:50:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T14:50:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:57 smithi161 ceph-mon[91579]: pgmap v7555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:57 smithi188 ceph-mon[92425]: pgmap v7555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:50:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:50:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:50:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:50:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:50:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:50:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:50:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:50:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:50:59 smithi161 ceph-mon[91579]: pgmap v7556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:50:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:50:59 smithi188 ceph-mon[92425]: pgmap v7556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:01 smithi161 ceph-mon[91579]: pgmap v7557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:01 smithi188 ceph-mon[92425]: pgmap v7557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:03 smithi161 ceph-mon[91579]: pgmap v7558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:03 smithi188 ceph-mon[92425]: pgmap v7558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:05 smithi161 ceph-mon[91579]: pgmap v7559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:05 smithi188 ceph-mon[92425]: pgmap v7559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:07 smithi161 ceph-mon[91579]: pgmap v7560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:07 smithi188 ceph-mon[92425]: pgmap v7560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:09 smithi161 ceph-mon[91579]: pgmap v7561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:09 smithi188 ceph-mon[92425]: pgmap v7561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:11 smithi161 ceph-mon[91579]: pgmap v7562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:11 smithi188 ceph-mon[92425]: pgmap v7562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:13 smithi161 ceph-mon[91579]: pgmap v7563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:13 smithi188 ceph-mon[92425]: pgmap v7563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:15 smithi161 ceph-mon[91579]: pgmap v7564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:15 smithi188 ceph-mon[92425]: pgmap v7564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:17 smithi161 ceph-mon[91579]: pgmap v7565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:17 smithi188 ceph-mon[92425]: pgmap v7565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:19 smithi161 ceph-mon[91579]: pgmap v7566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:19 smithi188 ceph-mon[92425]: pgmap v7566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:21 smithi161 ceph-mon[91579]: pgmap v7567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:21 smithi188 ceph-mon[92425]: pgmap v7567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:23 smithi161 ceph-mon[91579]: pgmap v7568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:23 smithi188 ceph-mon[92425]: pgmap v7568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:25 smithi188 ceph-mon[92425]: pgmap v7569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:25 smithi161 ceph-mon[91579]: pgmap v7569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:27 smithi188 ceph-mon[92425]: pgmap v7570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:27 smithi161 ceph-mon[91579]: pgmap v7570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:29 smithi188 ceph-mon[92425]: pgmap v7571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:29 smithi161 ceph-mon[91579]: pgmap v7571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:31 smithi188 ceph-mon[92425]: pgmap v7572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:31 smithi161 ceph-mon[91579]: pgmap v7572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:33 smithi188 ceph-mon[92425]: pgmap v7573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:33 smithi161 ceph-mon[91579]: pgmap v7573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:35 smithi188 ceph-mon[92425]: pgmap v7574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:35 smithi161 ceph-mon[91579]: pgmap v7574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:37 smithi188 ceph-mon[92425]: pgmap v7575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:37 smithi161 ceph-mon[91579]: pgmap v7575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:39 smithi188 ceph-mon[92425]: pgmap v7576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:39 smithi161 ceph-mon[91579]: pgmap v7576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:41 smithi161 ceph-mon[91579]: pgmap v7577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:41 smithi188 ceph-mon[92425]: pgmap v7577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:43 smithi161 ceph-mon[91579]: pgmap v7578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:43 smithi188 ceph-mon[92425]: pgmap v7578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:45 smithi161 ceph-mon[91579]: pgmap v7579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:45 smithi188 ceph-mon[92425]: pgmap v7579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:47 smithi161 ceph-mon[91579]: pgmap v7580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:47 smithi188 ceph-mon[92425]: pgmap v7580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:48 smithi161 ceph-mon[91579]: pgmap v7581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:48 smithi188 ceph-mon[92425]: pgmap v7581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:50 smithi161 ceph-mon[91579]: pgmap v7582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:50 smithi188 ceph-mon[92425]: pgmap v7582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:52 smithi161 ceph-mon[91579]: pgmap v7583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:52 smithi188 ceph-mon[92425]: pgmap v7583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:54 smithi161 ceph-mon[91579]: pgmap v7584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:54 smithi188 ceph-mon[92425]: pgmap v7584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:51:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:55.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:51:55.591+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:51:55.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:51:55.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:51:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:51:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:51:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:51:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:51:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:51:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:51:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:51:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:51:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:51:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:51:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:51:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:51:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:51:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:51:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:51:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:51:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:51:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:51:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:51:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:51:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:51:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:51:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:51:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:51:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:51:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:51:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:51:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:51:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:51:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:51:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:51:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:51:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:51:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:51:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:51:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:51:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:51:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:51:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:51:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:51:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:51:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:51:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=411] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:51:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=411] Local address: 172.21.15.161, port 55112 2022-07-03T14:51:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=411] Peer address: 172.21.15.101, port 22 2022-07-03T14:51:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=411] Connection lost 2022-07-03T14:51:55.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=411] Aborting connection 2022-07-03T14:51:55.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:51:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:51:55.654 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:51:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: pgmap v7585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:51:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:51:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:51:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:51:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:51:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:51:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:51:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:51:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:51:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:51:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:51:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:51:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:51:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:51:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:51:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:51:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:51:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:51:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:51:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:51:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:51:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:51:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:51:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:51:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:51:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:51:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:51:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:51:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:51:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:51:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:51:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:51:56.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:51:56.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:51:56.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:51:56.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:51:56.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:51:56.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:51:56.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:51:56.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:51:56.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:51:56.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:51:56.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:51:56.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:51:56.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:51:56.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:51:56.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:51:56.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:51:56.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:51:56.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=411] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:51:56.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=411] Local address: 172.21.15.161, port 55112 2022-07-03T14:51:56.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=411] Peer address: 172.21.15.101, port 22 2022-07-03T14:51:56.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=411] Connection lost 2022-07-03T14:51:56.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=411] Aborting connection 2022-07-03T14:51:56.662 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.663 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.663 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.663 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: pgmap v7585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:51:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:51:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:51:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:51:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:51:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:51:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:51:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:51:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:51:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:51:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:51:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:51:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:51:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:51:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:51:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:51:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:51:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:51:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:51:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:51:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:51:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:51:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:51:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:51:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:51:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:51:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:51:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:51:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:51:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:51:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:51:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:51:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:51:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:51:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:51:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:51:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:51:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:51:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:51:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:51:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:51:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:51:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:51:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:51:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:51:56.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:51:56.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:51:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:51:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:51:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=411] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:51:56.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=411] Local address: 172.21.15.161, port 55112 2022-07-03T14:51:56.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=411] Peer address: 172.21.15.101, port 22 2022-07-03T14:51:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=411] Connection lost 2022-07-03T14:51:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=411] Aborting connection 2022-07-03T14:51:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:56.690 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:51:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:58 smithi161 ceph-mon[91579]: pgmap v7586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:51:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:51:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:51:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:58 smithi188 ceph-mon[92425]: pgmap v7586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:51:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:51:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:51:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:51:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:52:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:00 smithi161 ceph-mon[91579]: pgmap v7587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:00 smithi188 ceph-mon[92425]: pgmap v7587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:02 smithi161 ceph-mon[91579]: pgmap v7588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:02 smithi188 ceph-mon[92425]: pgmap v7588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:04 smithi161 ceph-mon[91579]: pgmap v7589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:04 smithi188 ceph-mon[92425]: pgmap v7589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:07 smithi161 ceph-mon[91579]: pgmap v7590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:07 smithi188 ceph-mon[92425]: pgmap v7590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:09 smithi161 ceph-mon[91579]: pgmap v7591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:09 smithi188 ceph-mon[92425]: pgmap v7591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:11 smithi161 ceph-mon[91579]: pgmap v7592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:11 smithi188 ceph-mon[92425]: pgmap v7592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:13 smithi161 ceph-mon[91579]: pgmap v7593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:13 smithi188 ceph-mon[92425]: pgmap v7593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:15 smithi161 ceph-mon[91579]: pgmap v7594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:15 smithi188 ceph-mon[92425]: pgmap v7594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:17 smithi161 ceph-mon[91579]: pgmap v7595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:17 smithi188 ceph-mon[92425]: pgmap v7595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:19 smithi161 ceph-mon[91579]: pgmap v7596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:19 smithi188 ceph-mon[92425]: pgmap v7596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:21 smithi161 ceph-mon[91579]: pgmap v7597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:21 smithi188 ceph-mon[92425]: pgmap v7597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:23 smithi161 ceph-mon[91579]: pgmap v7598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:23 smithi188 ceph-mon[92425]: pgmap v7598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:25 smithi161 ceph-mon[91579]: pgmap v7599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:25 smithi188 ceph-mon[92425]: pgmap v7599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:27 smithi161 ceph-mon[91579]: pgmap v7600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:27 smithi188 ceph-mon[92425]: pgmap v7600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:29 smithi161 ceph-mon[91579]: pgmap v7601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:29 smithi188 ceph-mon[92425]: pgmap v7601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:31 smithi161 ceph-mon[91579]: pgmap v7602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:31 smithi188 ceph-mon[92425]: pgmap v7602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:33 smithi161 ceph-mon[91579]: pgmap v7603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:33 smithi188 ceph-mon[92425]: pgmap v7603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:35 smithi161 ceph-mon[91579]: pgmap v7604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:35 smithi188 ceph-mon[92425]: pgmap v7604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:37 smithi161 ceph-mon[91579]: pgmap v7605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:37 smithi188 ceph-mon[92425]: pgmap v7605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:39 smithi161 ceph-mon[91579]: pgmap v7606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:39 smithi188 ceph-mon[92425]: pgmap v7606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T14:52:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:41 smithi161 ceph-mon[91579]: pgmap v7607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:41 smithi188 ceph-mon[92425]: pgmap v7607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:43 smithi161 ceph-mon[91579]: pgmap v7608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:43 smithi188 ceph-mon[92425]: pgmap v7608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:45 smithi161 ceph-mon[91579]: pgmap v7609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:45 smithi188 ceph-mon[92425]: pgmap v7609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:47 smithi161 ceph-mon[91579]: pgmap v7610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:47 smithi188 ceph-mon[92425]: pgmap v7610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:49 smithi161 ceph-mon[91579]: pgmap v7611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:49 smithi188 ceph-mon[92425]: pgmap v7611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:51 smithi161 ceph-mon[91579]: pgmap v7612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:51 smithi188 ceph-mon[92425]: pgmap v7612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:53 smithi161 ceph-mon[91579]: pgmap v7613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:53 smithi188 ceph-mon[92425]: pgmap v7613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:55 smithi161 ceph-mon[91579]: pgmap v7614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:55 smithi188 ceph-mon[92425]: pgmap v7614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:52:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:52:56.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:52:56.529+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:52:56.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:52:56.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:52:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:52:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:52:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:52:56.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:52:56.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:52:56.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:52:56.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:52:56.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:52:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:52:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:52:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:52:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:52:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:52:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:52:56.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:52:56.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:52:56.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:52:56.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:52:56.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:52:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:52:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:52:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:52:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:52:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:52:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:52:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:52:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:52:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:52:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:52:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:52:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:52:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:52:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:52:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:52:56.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:52:56.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:52:56.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:52:56.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:52:56.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:52:56.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:52:56.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:52:56.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:52:56.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=413] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:52:56.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=413] Local address: 172.21.15.161, port 55116 2022-07-03T14:52:56.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=413] Peer address: 172.21.15.101, port 22 2022-07-03T14:52:56.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=413] Connection lost 2022-07-03T14:52:56.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=413] Aborting connection 2022-07-03T14:52:56.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:52:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:52:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:57 smithi161 ceph-mon[91579]: pgmap v7615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:52:57.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:52:57.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:52:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:57 smithi188 ceph-mon[92425]: pgmap v7615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:52:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:52:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:52:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:52:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:52:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:52:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:52:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:52:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:52:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:52:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:52:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:52:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:52:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:52:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:52:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:52:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:52:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:52:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:52:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:52:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:52:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:52:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:52:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:52:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:52:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:52:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:52:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:52:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:52:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:52:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:52:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:52:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:52:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:52:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:52:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:52:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:52:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:52:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:52:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:52:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:52:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:52:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:52:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:52:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:52:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:52:58.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:52:58.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:52:58.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:52:58.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=413] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:52:58.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=413] Local address: 172.21.15.161, port 55116 2022-07-03T14:52:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=413] Peer address: 172.21.15.101, port 22 2022-07-03T14:52:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=413] Connection lost 2022-07-03T14:52:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=413] Aborting connection 2022-07-03T14:52:58.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:52:58.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:52:58.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:52:58.406 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:52:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:52:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:52:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:52:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:52:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:52:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:52:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:52:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:52:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:52:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:52:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:52:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:52:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:52:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:52:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:52:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:52:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:52:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:52:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:52:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:52:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:52:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:52:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:52:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:52:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:52:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:52:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:52:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:52:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:52:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:52:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:52:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:52:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:52:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:52:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:52:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:52:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:52:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:52:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:52:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:52:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:52:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:52:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:52:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:52:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:52:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:52:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=413] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:52:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=413] Local address: 172.21.15.161, port 55116 2022-07-03T14:52:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=413] Peer address: 172.21.15.101, port 22 2022-07-03T14:52:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=413] Connection lost 2022-07-03T14:52:58.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=413] Aborting connection 2022-07-03T14:52:58.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:52:58.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:52:58.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:52:58.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:52:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:52:59 smithi161 ceph-mon[91579]: pgmap v7616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:52:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:52:59 smithi188 ceph-mon[92425]: pgmap v7616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:01 smithi161 ceph-mon[91579]: pgmap v7617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:01 smithi188 ceph-mon[92425]: pgmap v7617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:03 smithi188 ceph-mon[92425]: pgmap v7618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:03 smithi161 ceph-mon[91579]: pgmap v7618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:05 smithi188 ceph-mon[92425]: pgmap v7619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:05 smithi161 ceph-mon[91579]: pgmap v7619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:07 smithi188 ceph-mon[92425]: pgmap v7620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:07 smithi161 ceph-mon[91579]: pgmap v7620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:09 smithi188 ceph-mon[92425]: pgmap v7621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:09 smithi161 ceph-mon[91579]: pgmap v7621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:11 smithi188 ceph-mon[92425]: pgmap v7622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:11 smithi161 ceph-mon[91579]: pgmap v7622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:13 smithi188 ceph-mon[92425]: pgmap v7623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:13 smithi161 ceph-mon[91579]: pgmap v7623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:15 smithi188 ceph-mon[92425]: pgmap v7624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:15 smithi161 ceph-mon[91579]: pgmap v7624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:17 smithi188 ceph-mon[92425]: pgmap v7625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:17 smithi161 ceph-mon[91579]: pgmap v7625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:19 smithi188 ceph-mon[92425]: pgmap v7626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:19 smithi161 ceph-mon[91579]: pgmap v7626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:21 smithi188 ceph-mon[92425]: pgmap v7627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:21 smithi161 ceph-mon[91579]: pgmap v7627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:23 smithi161 ceph-mon[91579]: pgmap v7628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:23 smithi188 ceph-mon[92425]: pgmap v7628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:25 smithi161 ceph-mon[91579]: pgmap v7629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:25 smithi188 ceph-mon[92425]: pgmap v7629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:27 smithi161 ceph-mon[91579]: pgmap v7630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:27 smithi188 ceph-mon[92425]: pgmap v7630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:29 smithi161 ceph-mon[91579]: pgmap v7631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:29 smithi188 ceph-mon[92425]: pgmap v7631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:31 smithi161 ceph-mon[91579]: pgmap v7632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:31 smithi188 ceph-mon[92425]: pgmap v7632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:33 smithi161 ceph-mon[91579]: pgmap v7633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:33 smithi188 ceph-mon[92425]: pgmap v7633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:35 smithi161 ceph-mon[91579]: pgmap v7634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:35 smithi188 ceph-mon[92425]: pgmap v7634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:37 smithi161 ceph-mon[91579]: pgmap v7635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:37 smithi188 ceph-mon[92425]: pgmap v7635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:39 smithi161 ceph-mon[91579]: pgmap v7636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:39 smithi188 ceph-mon[92425]: pgmap v7636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:41 smithi161 ceph-mon[91579]: pgmap v7637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:41 smithi188 ceph-mon[92425]: pgmap v7637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:42 smithi161 ceph-mon[91579]: pgmap v7638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:42 smithi188 ceph-mon[92425]: pgmap v7638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:44 smithi188 ceph-mon[92425]: pgmap v7639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:44 smithi161 ceph-mon[91579]: pgmap v7639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:47 smithi161 ceph-mon[91579]: pgmap v7640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:47 smithi188 ceph-mon[92425]: pgmap v7640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:49 smithi161 ceph-mon[91579]: pgmap v7641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:49 smithi188 ceph-mon[92425]: pgmap v7641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:51 smithi161 ceph-mon[91579]: pgmap v7642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:51 smithi188 ceph-mon[92425]: pgmap v7642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:53 smithi161 ceph-mon[91579]: pgmap v7643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:53 smithi188 ceph-mon[92425]: pgmap v7643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:55 smithi161 ceph-mon[91579]: pgmap v7644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:55 smithi188 ceph-mon[92425]: pgmap v7644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:53:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:53:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-mon[91579]: pgmap v7645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:53:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:53:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:53:57.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:53:57.033+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:53:57.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:53:57.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:53:57.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:53:57.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:53:57.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:53:57.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:53:57.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:53:57.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:53:57.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:53:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:53:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:53:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:53:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:53:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:53:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:53:57.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:53:57.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:53:57.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:53:57.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:53:57.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:53:57.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:53:57.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:53:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:53:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:53:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:53:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:53:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:53:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:53:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:53:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:53:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:53:57.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:53:57.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:53:57.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:53:57.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:53:57.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:53:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:53:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:53:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:53:57.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:53:57.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:53:57.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:53:57.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:53:57.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:53:57.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=415] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:53:57.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=415] Local address: 172.21.15.161, port 55124 2022-07-03T14:53:57.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=415] Peer address: 172.21.15.101, port 22 2022-07-03T14:53:57.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=415] Connection lost 2022-07-03T14:53:57.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=415] Aborting connection 2022-07-03T14:53:57.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:53:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:53:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:57 smithi188 ceph-mon[92425]: pgmap v7645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:53:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:53:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:53:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:53:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:53:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:53:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:53:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:53:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:53:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:53:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:53:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:53:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:53:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:53:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:53:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:53:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:53:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:53:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:53:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:53:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:53:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:53:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:53:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:53:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:53:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:53:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:53:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:53:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:53:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:53:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:53:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:53:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:53:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:53:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:53:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:53:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:53:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:53:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:53:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:53:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:53:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:53:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:53:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:53:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:53:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:53:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:53:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:53:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:53:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=415] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:53:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=415] Local address: 172.21.15.161, port 55124 2022-07-03T14:53:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=415] Peer address: 172.21.15.101, port 22 2022-07-03T14:53:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=415] Connection lost 2022-07-03T14:53:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=415] Aborting connection 2022-07-03T14:53:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:53:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:53:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:53:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:53:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:53:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:53:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:53:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:53:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:53:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:53:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:53:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:53:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:53:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:53:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:53:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:53:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:53:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:53:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:53:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:53:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:53:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:53:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:53:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:53:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:53:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:53:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:53:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:53:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:53:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:53:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:53:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:53:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:53:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:53:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:53:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:53:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:53:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:53:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:53:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:53:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:53:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:53:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:53:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:53:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:53:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:53:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:53:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:53:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:53:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:53:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=415] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:53:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=415] Local address: 172.21.15.161, port 55124 2022-07-03T14:53:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=415] Peer address: 172.21.15.101, port 22 2022-07-03T14:53:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=415] Connection lost 2022-07-03T14:53:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=415] Aborting connection 2022-07-03T14:53:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:53:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:53:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:53:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:53:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:53:59 smithi161 ceph-mon[91579]: pgmap v7646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:53:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:53:59 smithi188 ceph-mon[92425]: pgmap v7646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:01 smithi161 ceph-mon[91579]: pgmap v7647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:01 smithi188 ceph-mon[92425]: pgmap v7647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:03 smithi161 ceph-mon[91579]: pgmap v7648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:03 smithi188 ceph-mon[92425]: pgmap v7648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:05 smithi161 ceph-mon[91579]: pgmap v7649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:05 smithi188 ceph-mon[92425]: pgmap v7649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:07 smithi161 ceph-mon[91579]: pgmap v7650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:07 smithi188 ceph-mon[92425]: pgmap v7650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:09 smithi161 ceph-mon[91579]: pgmap v7651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:09 smithi188 ceph-mon[92425]: pgmap v7651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:11 smithi161 ceph-mon[91579]: pgmap v7652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:11 smithi188 ceph-mon[92425]: pgmap v7652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:13 smithi161 ceph-mon[91579]: pgmap v7653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:13 smithi188 ceph-mon[92425]: pgmap v7653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:15 smithi161 ceph-mon[91579]: pgmap v7654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:15 smithi188 ceph-mon[92425]: pgmap v7654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:17 smithi161 ceph-mon[91579]: pgmap v7655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:17 smithi188 ceph-mon[92425]: pgmap v7655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:19 smithi161 ceph-mon[91579]: pgmap v7656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:19 smithi188 ceph-mon[92425]: pgmap v7656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:21 smithi161 ceph-mon[91579]: pgmap v7657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:21 smithi188 ceph-mon[92425]: pgmap v7657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:23 smithi161 ceph-mon[91579]: pgmap v7658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:23 smithi188 ceph-mon[92425]: pgmap v7658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:25 smithi161 ceph-mon[91579]: pgmap v7659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:25 smithi188 ceph-mon[92425]: pgmap v7659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:27 smithi161 ceph-mon[91579]: pgmap v7660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:27 smithi188 ceph-mon[92425]: pgmap v7660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:29 smithi161 ceph-mon[91579]: pgmap v7661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:29 smithi188 ceph-mon[92425]: pgmap v7661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:31 smithi188 ceph-mon[92425]: pgmap v7662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:31 smithi161 ceph-mon[91579]: pgmap v7662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:33 smithi188 ceph-mon[92425]: pgmap v7663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:33 smithi161 ceph-mon[91579]: pgmap v7663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:35 smithi188 ceph-mon[92425]: pgmap v7664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:35 smithi161 ceph-mon[91579]: pgmap v7664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:37 smithi188 ceph-mon[92425]: pgmap v7665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:37 smithi161 ceph-mon[91579]: pgmap v7665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:39 smithi188 ceph-mon[92425]: pgmap v7666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:39 smithi161 ceph-mon[91579]: pgmap v7666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:41 smithi188 ceph-mon[92425]: pgmap v7667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:41 smithi161 ceph-mon[91579]: pgmap v7667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:43 smithi188 ceph-mon[92425]: pgmap v7668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:43 smithi161 ceph-mon[91579]: pgmap v7668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:45 smithi188 ceph-mon[92425]: pgmap v7669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:45 smithi161 ceph-mon[91579]: pgmap v7669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:47 smithi161 ceph-mon[91579]: pgmap v7670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:47 smithi188 ceph-mon[92425]: pgmap v7670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:49 smithi161 ceph-mon[91579]: pgmap v7671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:49 smithi188 ceph-mon[92425]: pgmap v7671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:51 smithi161 ceph-mon[91579]: pgmap v7672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:51 smithi188 ceph-mon[92425]: pgmap v7672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:53 smithi161 ceph-mon[91579]: pgmap v7673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:53 smithi188 ceph-mon[92425]: pgmap v7673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:55 smithi161 ceph-mon[91579]: pgmap v7674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:55 smithi188 ceph-mon[92425]: pgmap v7674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:54:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:54:57.516 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-mon[91579]: pgmap v7675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:57.524 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:57 smithi188 ceph-mon[92425]: pgmap v7675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:54:57.514+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:54:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:54:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:54:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:54:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:54:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:54:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:54:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:54:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:54:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:54:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:54:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:54:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:54:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:54:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:54:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:54:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:54:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:54:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:54:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:54:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:54:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:54:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:54:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:54:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:54:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:54:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:54:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:54:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:54:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:54:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:54:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:54:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:54:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:54:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:54:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:54:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:54:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:54:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:54:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:54:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:54:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:54:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:54:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:54:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:54:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=417] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:54:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=417] Local address: 172.21.15.161, port 55154 2022-07-03T14:54:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=417] Peer address: 172.21.15.101, port 22 2022-07-03T14:54:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=417] Connection lost 2022-07-03T14:54:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=417] Aborting connection 2022-07-03T14:54:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:54:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:54:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:54:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:54:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:54:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:54:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:54:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:54:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:54:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:54:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:54:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:54:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:54:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:59 smithi161 ceph-mon[91579]: pgmap v7676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:54:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:54:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:54:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:54:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:54:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:54:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:54:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:54:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:54:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:54:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:54:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:54:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:54:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:54:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:54:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:54:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:54:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:54:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:54:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:54:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:54:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:54:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:54:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:54:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:54:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:54:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:54:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:54:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:54:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:54:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:54:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:54:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:54:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:54:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:54:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:54:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:54:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:54:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:54:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:54:59.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:54:59.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:54:59.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:54:59.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:54:59.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:54:59.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:54:59.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:54:59.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=417] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:54:59.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=417] Local address: 172.21.15.161, port 55154 2022-07-03T14:54:59.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=417] Peer address: 172.21.15.101, port 22 2022-07-03T14:54:59.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=417] Connection lost 2022-07-03T14:54:59.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=417] Aborting connection 2022-07-03T14:54:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:59 smithi188 ceph-mon[92425]: pgmap v7676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:54:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:54:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:54:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:54:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:54:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:54:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:54:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:54:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:54:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:54:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:54:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:54:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:54:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:54:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:54:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:54:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:54:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:54:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:54:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:54:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:54:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:54:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:54:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:54:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:54:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:54:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:54:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:54:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:54:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:54:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:54:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:54:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:54:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:54:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:54:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:54:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:54:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:54:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:54:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=417] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:54:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=417] Local address: 172.21.15.161, port 55154 2022-07-03T14:54:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=417] Peer address: 172.21.15.101, port 22 2022-07-03T14:54:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=417] Connection lost 2022-07-03T14:54:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=417] Aborting connection 2022-07-03T14:55:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T14:55:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: pgmap v7677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:01.666 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.667 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.667 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.667 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.668 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.668 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.668 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.669 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.669 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.669 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.670 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.670 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.671 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.672 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: pgmap v7677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T14:55:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:55:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:55:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:55:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:55:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 196.2M 2022-07-03T14:55:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 205771776: error parsing value: Value '205771776' is below minimum 939524096 2022-07-03T14:55:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T14:55:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T14:55:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:03 smithi161 ceph-mon[91579]: pgmap v7678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:03 smithi188 ceph-mon[92425]: pgmap v7678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:05 smithi161 ceph-mon[91579]: pgmap v7679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:05 smithi188 ceph-mon[92425]: pgmap v7679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:07 smithi161 ceph-mon[91579]: pgmap v7680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:07 smithi188 ceph-mon[92425]: pgmap v7680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:09 smithi161 ceph-mon[91579]: pgmap v7681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:09 smithi188 ceph-mon[92425]: pgmap v7681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:10 smithi161 ceph-mon[91579]: pgmap v7682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:10 smithi188 ceph-mon[92425]: pgmap v7682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:12 smithi188 ceph-mon[92425]: pgmap v7683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:12 smithi161 ceph-mon[91579]: pgmap v7683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:14 smithi161 ceph-mon[91579]: pgmap v7684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:14 smithi188 ceph-mon[92425]: pgmap v7684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:17.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:17 smithi161 ceph-mon[91579]: pgmap v7685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:17 smithi188 ceph-mon[92425]: pgmap v7685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:19 smithi161 ceph-mon[91579]: pgmap v7686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:19 smithi188 ceph-mon[92425]: pgmap v7686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:21 smithi161 ceph-mon[91579]: pgmap v7687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:21 smithi188 ceph-mon[92425]: pgmap v7687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:23 smithi161 ceph-mon[91579]: pgmap v7688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:23 smithi188 ceph-mon[92425]: pgmap v7688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:25 smithi161 ceph-mon[91579]: pgmap v7689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:25 smithi188 ceph-mon[92425]: pgmap v7689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:27.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:27 smithi161 ceph-mon[91579]: pgmap v7690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:27 smithi188 ceph-mon[92425]: pgmap v7690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:29 smithi161 ceph-mon[91579]: pgmap v7691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:29 smithi188 ceph-mon[92425]: pgmap v7691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:31 smithi161 ceph-mon[91579]: pgmap v7692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:31 smithi188 ceph-mon[92425]: pgmap v7692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:33 smithi161 ceph-mon[91579]: pgmap v7693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:33 smithi188 ceph-mon[92425]: pgmap v7693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:35 smithi161 ceph-mon[91579]: pgmap v7694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:35 smithi188 ceph-mon[92425]: pgmap v7694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:37 smithi161 ceph-mon[91579]: pgmap v7695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:37 smithi188 ceph-mon[92425]: pgmap v7695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:39 smithi161 ceph-mon[91579]: pgmap v7696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:39 smithi188 ceph-mon[92425]: pgmap v7696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:41 smithi161 ceph-mon[91579]: pgmap v7697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:41 smithi188 ceph-mon[92425]: pgmap v7697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:43 smithi161 ceph-mon[91579]: pgmap v7698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:43 smithi188 ceph-mon[92425]: pgmap v7698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:45 smithi161 ceph-mon[91579]: pgmap v7699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:45 smithi188 ceph-mon[92425]: pgmap v7699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:47 smithi161 ceph-mon[91579]: pgmap v7700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:47 smithi188 ceph-mon[92425]: pgmap v7700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:49 smithi161 ceph-mon[91579]: pgmap v7701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:49 smithi188 ceph-mon[92425]: pgmap v7701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:51 smithi188 ceph-mon[92425]: pgmap v7702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:51 smithi161 ceph-mon[91579]: pgmap v7702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:53 smithi188 ceph-mon[92425]: pgmap v7703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:53 smithi161 ceph-mon[91579]: pgmap v7703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:55 smithi188 ceph-mon[92425]: pgmap v7704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:55 smithi161 ceph-mon[91579]: pgmap v7704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:55:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:55:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:57 smithi188 ceph-mon[92425]: pgmap v7705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:57 smithi161 ceph-mon[91579]: pgmap v7705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:55:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:55:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:55:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:55:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:55:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:55:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:55:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:55:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:55:59 smithi188 ceph-mon[92425]: pgmap v7706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:55:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:55:59 smithi161 ceph-mon[91579]: pgmap v7706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:01.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:56:00.675+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:56:01.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:56:01.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:56:01.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:56:01.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:56:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:56:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:56:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:56:01.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:56:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:56:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:56:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:56:01.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:56:01.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:56:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:56:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:56:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:56:01.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:56:01.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:56:01.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:56:01.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:56:01.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:56:01.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:56:01.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:56:01.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:56:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:56:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:56:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:56:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:56:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:56:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:56:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:56:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:56:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:56:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:56:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:56:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:56:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:56:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:56:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:56:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:56:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:56:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:56:01.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:56:01.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:56:01.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=419] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:56:01.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=419] Local address: 172.21.15.161, port 55158 2022-07-03T14:56:01.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=419] Peer address: 172.21.15.101, port 22 2022-07-03T14:56:01.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=419] Connection lost 2022-07-03T14:56:01.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=419] Aborting connection 2022-07-03T14:56:01.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:56:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:56:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:01 smithi161 ceph-mon[91579]: pgmap v7707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:56:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:56:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:56:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:01 smithi188 ceph-mon[92425]: pgmap v7707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:56:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:56:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:56:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:56:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:56:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:56:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:56:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:56:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:56:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:56:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:56:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:56:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:56:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:56:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:56:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:56:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:56:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:56:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:56:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:56:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:56:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:56:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:56:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:56:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:56:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:56:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:56:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:56:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:56:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:56:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:56:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:56:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:56:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:56:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:56:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:56:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:56:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:56:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:56:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:56:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:56:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:56:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:56:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:56:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:56:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:56:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:56:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:56:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:56:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=419] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:56:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=419] Local address: 172.21.15.161, port 55158 2022-07-03T14:56:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=419] Peer address: 172.21.15.101, port 22 2022-07-03T14:56:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=419] Connection lost 2022-07-03T14:56:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=419] Aborting connection 2022-07-03T14:56:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:56:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:56:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:56:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:56:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:56:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:56:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:56:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:56:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:56:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:56:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:56:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:56:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:56:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:56:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:56:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:56:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:56:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:56:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:56:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:56:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:56:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:56:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:56:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:56:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:56:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:56:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:56:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:56:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:56:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:56:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:56:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:56:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:56:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:56:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:56:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:56:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:56:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:56:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:56:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:56:02.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:56:02.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:56:02.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:56:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:56:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:56:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:56:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=419] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:56:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=419] Local address: 172.21.15.161, port 55158 2022-07-03T14:56:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=419] Peer address: 172.21.15.101, port 22 2022-07-03T14:56:02.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=419] Connection lost 2022-07-03T14:56:02.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=419] Aborting connection 2022-07-03T14:56:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:03 smithi161 ceph-mon[91579]: pgmap v7708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:03 smithi188 ceph-mon[92425]: pgmap v7708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:05 smithi161 ceph-mon[91579]: pgmap v7709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:05 smithi188 ceph-mon[92425]: pgmap v7709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:07 smithi161 ceph-mon[91579]: pgmap v7710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:07 smithi188 ceph-mon[92425]: pgmap v7710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:09 smithi161 ceph-mon[91579]: pgmap v7711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:09 smithi188 ceph-mon[92425]: pgmap v7711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:11 smithi161 ceph-mon[91579]: pgmap v7712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:11 smithi188 ceph-mon[92425]: pgmap v7712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:13 smithi161 ceph-mon[91579]: pgmap v7713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:13 smithi188 ceph-mon[92425]: pgmap v7713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:15 smithi161 ceph-mon[91579]: pgmap v7714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:15 smithi188 ceph-mon[92425]: pgmap v7714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:17 smithi161 ceph-mon[91579]: pgmap v7715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:17 smithi188 ceph-mon[92425]: pgmap v7715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:19 smithi161 ceph-mon[91579]: pgmap v7716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:19 smithi188 ceph-mon[92425]: pgmap v7716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:21 smithi161 ceph-mon[91579]: pgmap v7717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:21 smithi188 ceph-mon[92425]: pgmap v7717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:23 smithi161 ceph-mon[91579]: pgmap v7718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:23 smithi188 ceph-mon[92425]: pgmap v7718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:25 smithi161 ceph-mon[91579]: pgmap v7719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:25 smithi188 ceph-mon[92425]: pgmap v7719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:27 smithi161 ceph-mon[91579]: pgmap v7720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:27 smithi188 ceph-mon[92425]: pgmap v7720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:29 smithi161 ceph-mon[91579]: pgmap v7721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:29 smithi188 ceph-mon[92425]: pgmap v7721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:31 smithi161 ceph-mon[91579]: pgmap v7722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:31 smithi188 ceph-mon[92425]: pgmap v7722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:32 smithi161 ceph-mon[91579]: pgmap v7723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:32 smithi188 ceph-mon[92425]: pgmap v7723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:34 smithi161 ceph-mon[91579]: pgmap v7724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:34 smithi188 ceph-mon[92425]: pgmap v7724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:37.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:37 smithi161 ceph-mon[91579]: pgmap v7725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:37 smithi188 ceph-mon[92425]: pgmap v7725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:39 smithi161 ceph-mon[91579]: pgmap v7726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:39 smithi188 ceph-mon[92425]: pgmap v7726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:41 smithi161 ceph-mon[91579]: pgmap v7727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:41 smithi188 ceph-mon[92425]: pgmap v7727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:43 smithi161 ceph-mon[91579]: pgmap v7728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:43 smithi188 ceph-mon[92425]: pgmap v7728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:45 smithi161 ceph-mon[91579]: pgmap v7729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:45 smithi188 ceph-mon[92425]: pgmap v7729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:47.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:47 smithi161 ceph-mon[91579]: pgmap v7730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:47 smithi188 ceph-mon[92425]: pgmap v7730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:49 smithi161 ceph-mon[91579]: pgmap v7731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:49 smithi188 ceph-mon[92425]: pgmap v7731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:51.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:51 smithi161 ceph-mon[91579]: pgmap v7732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:51 smithi188 ceph-mon[92425]: pgmap v7732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:53 smithi161 ceph-mon[91579]: pgmap v7733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:53 smithi188 ceph-mon[92425]: pgmap v7733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:55 smithi161 ceph-mon[91579]: pgmap v7734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:55 smithi188 ceph-mon[92425]: pgmap v7734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:56:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:56:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:57 smithi161 ceph-mon[91579]: pgmap v7735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:57 smithi188 ceph-mon[92425]: pgmap v7735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:56:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:56:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:56:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:56:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:56:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:56:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:56:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:56:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:56:59 smithi188 ceph-mon[92425]: pgmap v7736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:56:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:56:59 smithi161 ceph-mon[91579]: pgmap v7736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:01 smithi188 ceph-mon[92425]: pgmap v7737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-mon[91579]: pgmap v7737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:57:01.178+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:57:01.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:57:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:57:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:57:01.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:57:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:57:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:57:01.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:57:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:57:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:57:01.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:57:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:57:01.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:57:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:57:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:57:01.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:57:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:57:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:57:01.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:57:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:57:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:57:01.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:57:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:57:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:57:01.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:57:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:57:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:57:01.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:57:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:57:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:57:01.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:57:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:57:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:57:01.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:57:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:57:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:57:01.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:57:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:57:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:57:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:57:01.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:57:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:57:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:57:01.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:57:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:57:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=421] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:57:01.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=421] Local address: 172.21.15.161, port 55162 2022-07-03T14:57:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=421] Peer address: 172.21.15.101, port 22 2022-07-03T14:57:01.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=421] Connection lost 2022-07-03T14:57:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=421] Aborting connection 2022-07-03T14:57:01.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:57:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:57:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:57:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:57:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:57:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:57:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:57:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:57:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:57:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:57:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:57:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:57:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:57:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:57:02.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:57:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:57:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:57:02.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:57:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:57:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:57:02.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:57:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:57:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:57:02.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:57:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:57:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:57:02.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:57:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:57:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:57:02.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:57:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:57:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:57:02.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:57:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:57:02.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:57:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:57:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:57:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:57:02.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:57:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:57:02.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:57:02.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:57:02.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:57:02.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:57:02.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:57:02.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:57:02.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:57:02.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:57:02.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:57:02.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:57:02.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:57:02.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=421] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:57:02.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=421] Local address: 172.21.15.161, port 55162 2022-07-03T14:57:02.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=421] Peer address: 172.21.15.101, port 22 2022-07-03T14:57:02.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=421] Connection lost 2022-07-03T14:57:02.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=421] Aborting connection 2022-07-03T14:57:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:57:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:57:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:57:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:57:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:57:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:57:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:57:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:57:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:57:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:57:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:57:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:57:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:57:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:57:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:57:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:57:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:57:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:57:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:57:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:57:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:57:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:57:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:57:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:57:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:57:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:57:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:57:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:57:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:57:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:57:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:57:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:57:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:57:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:57:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:57:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:57:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:57:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:57:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:57:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:57:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:57:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:57:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:57:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:57:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:57:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:57:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:57:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:57:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:57:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=421] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:57:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=421] Local address: 172.21.15.161, port 55162 2022-07-03T14:57:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=421] Peer address: 172.21.15.101, port 22 2022-07-03T14:57:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=421] Connection lost 2022-07-03T14:57:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=421] Aborting connection 2022-07-03T14:57:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:03 smithi188 ceph-mon[92425]: pgmap v7738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:03 smithi161 ceph-mon[91579]: pgmap v7738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:05 smithi188 ceph-mon[92425]: pgmap v7739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:05 smithi161 ceph-mon[91579]: pgmap v7739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:07 smithi188 ceph-mon[92425]: pgmap v7740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:07 smithi161 ceph-mon[91579]: pgmap v7740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:09 smithi188 ceph-mon[92425]: pgmap v7741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:09 smithi161 ceph-mon[91579]: pgmap v7741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:11 smithi188 ceph-mon[92425]: pgmap v7742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:11 smithi161 ceph-mon[91579]: pgmap v7742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:13 smithi161 ceph-mon[91579]: pgmap v7743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:13 smithi188 ceph-mon[92425]: pgmap v7743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:15 smithi161 ceph-mon[91579]: pgmap v7744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:15 smithi188 ceph-mon[92425]: pgmap v7744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:17 smithi161 ceph-mon[91579]: pgmap v7745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:17 smithi188 ceph-mon[92425]: pgmap v7745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:19 smithi161 ceph-mon[91579]: pgmap v7746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:19 smithi188 ceph-mon[92425]: pgmap v7746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:21 smithi161 ceph-mon[91579]: pgmap v7747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:21 smithi188 ceph-mon[92425]: pgmap v7747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:23 smithi161 ceph-mon[91579]: pgmap v7748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:23 smithi188 ceph-mon[92425]: pgmap v7748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:25 smithi161 ceph-mon[91579]: pgmap v7749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:25 smithi188 ceph-mon[92425]: pgmap v7749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:27 smithi161 ceph-mon[91579]: pgmap v7750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:27 smithi188 ceph-mon[92425]: pgmap v7750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:29 smithi161 ceph-mon[91579]: pgmap v7751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:29 smithi188 ceph-mon[92425]: pgmap v7751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:31 smithi161 ceph-mon[91579]: pgmap v7752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:31 smithi188 ceph-mon[92425]: pgmap v7752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:33 smithi161 ceph-mon[91579]: pgmap v7753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:33 smithi188 ceph-mon[92425]: pgmap v7753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:35 smithi161 ceph-mon[91579]: pgmap v7754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:35 smithi188 ceph-mon[92425]: pgmap v7754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:37 smithi161 ceph-mon[91579]: pgmap v7755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:37 smithi188 ceph-mon[92425]: pgmap v7755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:39 smithi161 ceph-mon[91579]: pgmap v7756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:39 smithi188 ceph-mon[92425]: pgmap v7756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:41 smithi161 ceph-mon[91579]: pgmap v7757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:41 smithi188 ceph-mon[92425]: pgmap v7757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:43 smithi161 ceph-mon[91579]: pgmap v7758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:43 smithi188 ceph-mon[92425]: pgmap v7758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:45 smithi161 ceph-mon[91579]: pgmap v7759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:45 smithi188 ceph-mon[92425]: pgmap v7759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:47 smithi161 ceph-mon[91579]: pgmap v7760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:47 smithi188 ceph-mon[92425]: pgmap v7760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:49 smithi161 ceph-mon[91579]: pgmap v7761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:49 smithi188 ceph-mon[92425]: pgmap v7761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:51 smithi161 ceph-mon[91579]: pgmap v7762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:51 smithi188 ceph-mon[92425]: pgmap v7762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:52 smithi161 ceph-mon[91579]: pgmap v7763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:52 smithi188 ceph-mon[92425]: pgmap v7763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:54.854 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:54 smithi188 ceph-mon[92425]: pgmap v7764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:54 smithi161 ceph-mon[91579]: pgmap v7764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:57:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:57:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:57 smithi161 ceph-mon[91579]: pgmap v7765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:57 smithi188 ceph-mon[92425]: pgmap v7765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:57:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:57:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:57:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:57:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:57:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:57:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:57:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:57:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:57:59 smithi161 ceph-mon[91579]: pgmap v7766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:57:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:57:59 smithi188 ceph-mon[92425]: pgmap v7766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:01.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-mon[91579]: pgmap v7767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:01 smithi188 ceph-mon[92425]: pgmap v7767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:02.119 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:58:01.679+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:58:02.119 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:58:02.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:58:02.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:58:02.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:58:02.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:58:02.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:58:02.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:58:02.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:58:02.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:58:02.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:58:02.123 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:58:02.123 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:58:02.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:58:02.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:58:02.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:58:02.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:58:02.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:58:02.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:58:02.126 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:58:02.126 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:58:02.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:58:02.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:58:02.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:58:02.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:58:02.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:58:02.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:58:02.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:58:02.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:58:02.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:58:02.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:58:02.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:58:02.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:58:02.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:58:02.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:58:02.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:58:02.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:58:02.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:58:02.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:58:02.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:58:02.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:58:02.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:58:02.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:58:02.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:58:02.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:58:02.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=423] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:58:02.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=423] Local address: 172.21.15.161, port 55166 2022-07-03T14:58:02.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=423] Peer address: 172.21.15.101, port 22 2022-07-03T14:58:02.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=423] Connection lost 2022-07-03T14:58:02.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=423] Aborting connection 2022-07-03T14:58:02.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:58:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:58:02.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:58:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:58:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:58:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:58:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:58:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:58:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:03 smithi161 ceph-mon[91579]: pgmap v7768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:58:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:58:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:58:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:58:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:58:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:58:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:58:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:58:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:58:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:58:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:58:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:58:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:58:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:58:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:58:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:58:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:58:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:58:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:58:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:58:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:58:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:58:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:58:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:58:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:58:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:58:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:58:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:58:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:58:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:58:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:58:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:58:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:58:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:58:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:58:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:58:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:58:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:58:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:58:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:58:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:58:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:58:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:58:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:58:03.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:58:03.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:58:03.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=423] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:58:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=423] Local address: 172.21.15.161, port 55166 2022-07-03T14:58:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=423] Peer address: 172.21.15.101, port 22 2022-07-03T14:58:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=423] Connection lost 2022-07-03T14:58:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=423] Aborting connection 2022-07-03T14:58:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:03 smithi188 ceph-mon[92425]: pgmap v7768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:58:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:58:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:58:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:58:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:58:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:58:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:58:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:58:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:58:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:58:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:58:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:58:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:58:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:58:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:58:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:58:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:58:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:58:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:58:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:58:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:58:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:58:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:58:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:58:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:58:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:58:03.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:58:03.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:58:03.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:58:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:58:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:58:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:58:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:58:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:58:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:58:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:58:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:58:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:58:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:58:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:58:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:58:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:58:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:58:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:58:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:58:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:58:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=423] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:58:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=423] Local address: 172.21.15.161, port 55166 2022-07-03T14:58:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=423] Peer address: 172.21.15.101, port 22 2022-07-03T14:58:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=423] Connection lost 2022-07-03T14:58:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=423] Aborting connection 2022-07-03T14:58:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:05 smithi161 ceph-mon[91579]: pgmap v7769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:05 smithi188 ceph-mon[92425]: pgmap v7769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:07 smithi161 ceph-mon[91579]: pgmap v7770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:07 smithi188 ceph-mon[92425]: pgmap v7770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:09 smithi188 ceph-mon[92425]: pgmap v7771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:09 smithi161 ceph-mon[91579]: pgmap v7771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:11 smithi188 ceph-mon[92425]: pgmap v7772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:11 smithi161 ceph-mon[91579]: pgmap v7772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:13 smithi188 ceph-mon[92425]: pgmap v7773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:13 smithi161 ceph-mon[91579]: pgmap v7773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:15 smithi188 ceph-mon[92425]: pgmap v7774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:15 smithi161 ceph-mon[91579]: pgmap v7774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:17 smithi188 ceph-mon[92425]: pgmap v7775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:17 smithi161 ceph-mon[91579]: pgmap v7775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:19 smithi188 ceph-mon[92425]: pgmap v7776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:19 smithi161 ceph-mon[91579]: pgmap v7776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:21 smithi188 ceph-mon[92425]: pgmap v7777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:21 smithi161 ceph-mon[91579]: pgmap v7777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:23 smithi188 ceph-mon[92425]: pgmap v7778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:23 smithi161 ceph-mon[91579]: pgmap v7778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:25 smithi161 ceph-mon[91579]: pgmap v7779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:25 smithi188 ceph-mon[92425]: pgmap v7779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:27 smithi161 ceph-mon[91579]: pgmap v7780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:27 smithi188 ceph-mon[92425]: pgmap v7780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:29 smithi161 ceph-mon[91579]: pgmap v7781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:29 smithi188 ceph-mon[92425]: pgmap v7781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:31 smithi161 ceph-mon[91579]: pgmap v7782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:31 smithi188 ceph-mon[92425]: pgmap v7782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:33 smithi161 ceph-mon[91579]: pgmap v7783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:33 smithi188 ceph-mon[92425]: pgmap v7783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:35 smithi161 ceph-mon[91579]: pgmap v7784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:35 smithi188 ceph-mon[92425]: pgmap v7784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:37 smithi161 ceph-mon[91579]: pgmap v7785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:37 smithi188 ceph-mon[92425]: pgmap v7785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:39 smithi161 ceph-mon[91579]: pgmap v7786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:39 smithi188 ceph-mon[92425]: pgmap v7786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:41 smithi161 ceph-mon[91579]: pgmap v7787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:41 smithi188 ceph-mon[92425]: pgmap v7787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:43 smithi161 ceph-mon[91579]: pgmap v7788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:43 smithi188 ceph-mon[92425]: pgmap v7788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:45 smithi161 ceph-mon[91579]: pgmap v7789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:45 smithi188 ceph-mon[92425]: pgmap v7789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:47 smithi161 ceph-mon[91579]: pgmap v7790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:47 smithi188 ceph-mon[92425]: pgmap v7790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:49 smithi161 ceph-mon[91579]: pgmap v7791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:49 smithi188 ceph-mon[92425]: pgmap v7791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:51 smithi161 ceph-mon[91579]: pgmap v7792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:51 smithi188 ceph-mon[92425]: pgmap v7792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:53 smithi161 ceph-mon[91579]: pgmap v7793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:53 smithi188 ceph-mon[92425]: pgmap v7793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:55 smithi161 ceph-mon[91579]: pgmap v7794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:55 smithi188 ceph-mon[92425]: pgmap v7794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:58:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:58:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:57 smithi161 ceph-mon[91579]: pgmap v7795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:57 smithi188 ceph-mon[92425]: pgmap v7795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:58:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:58:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:58:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:58:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:58:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:58:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:58:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:58:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:58:59 smithi161 ceph-mon[91579]: pgmap v7796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:58:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:58:59 smithi188 ceph-mon[92425]: pgmap v7796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:01 smithi161 ceph-mon[91579]: pgmap v7797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:01 smithi188 ceph-mon[92425]: pgmap v7797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:59:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:59:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:59:02.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:59:02.183+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:59:02.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:59:02.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:59:02.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T14:59:02.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:59:02.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:59:02.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:59:02.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T14:59:02.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:59:02.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T14:59:02.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:59:02.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T14:59:02.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:59:02.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:59:02.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T14:59:02.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:59:02.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T14:59:02.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:59:02.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T14:59:02.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:59:02.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T14:59:02.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:59:02.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:59:02.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:59:02.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:59:02.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:59:02.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T14:59:02.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:59:02.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:59:02.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:59:02.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T14:59:02.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:59:02.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T14:59:02.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:59:02.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:59:02.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:59:02.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T14:59:02.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:59:02.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T14:59:02.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:59:02.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T14:59:02.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:59:02.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T14:59:02.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:59:02.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:59:02.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=425] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:59:02.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=425] Local address: 172.21.15.161, port 55170 2022-07-03T14:59:02.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=425] Peer address: 172.21.15.101, port 22 2022-07-03T14:59:02.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=425] Connection lost 2022-07-03T14:59:02.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=425] Aborting connection 2022-07-03T14:59:02.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 14:59:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T14:59:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T14:59:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T14:59:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T14:59:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:03 smithi161 ceph-mon[91579]: pgmap v7798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:59:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:59:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:59:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:59:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T14:59:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:59:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:59:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:59:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T14:59:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:59:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T14:59:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:59:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T14:59:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:59:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:59:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:59:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T14:59:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T14:59:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:59:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T14:59:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:59:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T14:59:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:59:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:59:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:59:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:59:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:59:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:59:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:59:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:59:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:59:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T14:59:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:59:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T14:59:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:59:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:59:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:59:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:59:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:59:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T14:59:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:59:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:59:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:59:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T14:59:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:59:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:59:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=425] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:59:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=425] Local address: 172.21.15.161, port 55170 2022-07-03T14:59:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=425] Peer address: 172.21.15.101, port 22 2022-07-03T14:59:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=425] Connection lost 2022-07-03T14:59:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=425] Aborting connection 2022-07-03T14:59:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:03 smithi188 ceph-mon[92425]: pgmap v7798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T14:59:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T14:59:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:59:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T14:59:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T14:59:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T14:59:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T14:59:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T14:59:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T14:59:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T14:59:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T14:59:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T14:59:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T14:59:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T14:59:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:59:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T14:59:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T14:59:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T14:59:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T14:59:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T14:59:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T14:59:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T14:59:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T14:59:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T14:59:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T14:59:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T14:59:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T14:59:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T14:59:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T14:59:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T14:59:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T14:59:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T14:59:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T14:59:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T14:59:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T14:59:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T14:59:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T14:59:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T14:59:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T14:59:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T14:59:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T14:59:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T14:59:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T14:59:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T14:59:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T14:59:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T14:59:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=425] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T14:59:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=425] Local address: 172.21.15.161, port 55170 2022-07-03T14:59:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=425] Peer address: 172.21.15.101, port 22 2022-07-03T14:59:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=425] Connection lost 2022-07-03T14:59:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=425] Aborting connection 2022-07-03T14:59:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:05 smithi161 ceph-mon[91579]: pgmap v7799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:05 smithi188 ceph-mon[92425]: pgmap v7799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:07 smithi161 ceph-mon[91579]: pgmap v7800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:07 smithi188 ceph-mon[92425]: pgmap v7800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:09 smithi161 ceph-mon[91579]: pgmap v7801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:09 smithi188 ceph-mon[92425]: pgmap v7801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:11 smithi161 ceph-mon[91579]: pgmap v7802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:11 smithi188 ceph-mon[92425]: pgmap v7802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:12.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:12 smithi161 ceph-mon[91579]: pgmap v7803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:12.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:12 smithi188 ceph-mon[92425]: pgmap v7803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:14 smithi161 ceph-mon[91579]: pgmap v7804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:14 smithi188 ceph-mon[92425]: pgmap v7804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:17 smithi161 ceph-mon[91579]: pgmap v7805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:17 smithi188 ceph-mon[92425]: pgmap v7805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:19 smithi161 ceph-mon[91579]: pgmap v7806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:19 smithi188 ceph-mon[92425]: pgmap v7806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:21 smithi161 ceph-mon[91579]: pgmap v7807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:21 smithi188 ceph-mon[92425]: pgmap v7807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:23 smithi188 ceph-mon[92425]: pgmap v7808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:23 smithi161 ceph-mon[91579]: pgmap v7808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:25 smithi188 ceph-mon[92425]: pgmap v7809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:25 smithi161 ceph-mon[91579]: pgmap v7809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:27 smithi188 ceph-mon[92425]: pgmap v7810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:27 smithi161 ceph-mon[91579]: pgmap v7810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:29 smithi188 ceph-mon[92425]: pgmap v7811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:29 smithi161 ceph-mon[91579]: pgmap v7811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:31 smithi188 ceph-mon[92425]: pgmap v7812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:31 smithi161 ceph-mon[91579]: pgmap v7812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:33 smithi188 ceph-mon[92425]: pgmap v7813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:33 smithi161 ceph-mon[91579]: pgmap v7813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:35 smithi188 ceph-mon[92425]: pgmap v7814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:35 smithi161 ceph-mon[91579]: pgmap v7814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:37.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:37 smithi188 ceph-mon[92425]: pgmap v7815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:37 smithi161 ceph-mon[91579]: pgmap v7815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:39 smithi188 ceph-mon[92425]: pgmap v7816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:39 smithi161 ceph-mon[91579]: pgmap v7816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:41 smithi161 ceph-mon[91579]: pgmap v7817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:41 smithi188 ceph-mon[92425]: pgmap v7817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:43 smithi161 ceph-mon[91579]: pgmap v7818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:43 smithi188 ceph-mon[92425]: pgmap v7818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:45 smithi161 ceph-mon[91579]: pgmap v7819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:45 smithi188 ceph-mon[92425]: pgmap v7819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:47 smithi161 ceph-mon[91579]: pgmap v7820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:47 smithi188 ceph-mon[92425]: pgmap v7820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:49 smithi161 ceph-mon[91579]: pgmap v7821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:49 smithi188 ceph-mon[92425]: pgmap v7821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:51 smithi161 ceph-mon[91579]: pgmap v7822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:51 smithi188 ceph-mon[92425]: pgmap v7822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:53 smithi161 ceph-mon[91579]: pgmap v7823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:53 smithi188 ceph-mon[92425]: pgmap v7823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:55 smithi161 ceph-mon[91579]: pgmap v7824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:55 smithi188 ceph-mon[92425]: pgmap v7824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T14:59:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T14:59:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:57 smithi161 ceph-mon[91579]: pgmap v7825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:57 smithi188 ceph-mon[92425]: pgmap v7825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:59:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:59:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:59:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:59:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:59:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:59:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T14:59:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T14:59:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 14:59:59 smithi161 ceph-mon[91579]: pgmap v7826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T14:59:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 14:59:59 smithi188 ceph-mon[92425]: pgmap v7826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:00:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:00:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:01 smithi161 ceph-mon[91579]: pgmap v7827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:01 smithi188 ceph-mon[92425]: pgmap v7827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:03.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:00:02.683+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:00:03.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:00:03.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:00:03.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:00:03.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:00:03.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:00:03.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:00:03.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:00:03.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:00:03.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:00:03.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:00:03.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:00:03.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:00:03.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:00:03.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:00:03.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:00:03.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:00:03.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:00:03.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:00:03.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:00:03.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:00:03.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:00:03.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:00:03.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:00:03.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:00:03.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:00:03.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:00:03.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:00:03.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:00:03.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:00:03.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:00:03.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:00:03.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:00:03.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:00:03.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:00:03.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:00:03.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:00:03.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:00:03.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:00:03.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:00:03.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:00:03.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:00:03.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:00:03.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:00:03.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:00:03.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=427] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:00:03.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=427] Local address: 172.21.15.161, port 55174 2022-07-03T15:00:03.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=427] Peer address: 172.21.15.101, port 22 2022-07-03T15:00:03.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=427] Connection lost 2022-07-03T15:00:03.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=427] Aborting connection 2022-07-03T15:00:03.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:00:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:00:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:03 smithi161 ceph-mon[91579]: pgmap v7828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:00:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:00:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:00:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:03 smithi188 ceph-mon[92425]: pgmap v7828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:00:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:00:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:00:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:00:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:00:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:00:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:00:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:00:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:00:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:00:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:00:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:00:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:00:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:00:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:00:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:00:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:00:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:00:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:00:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:00:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:00:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:00:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:00:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:00:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:00:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:00:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:00:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:00:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:00:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:00:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:00:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:00:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:00:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:00:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:00:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:00:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:00:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:00:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:00:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:00:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:00:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:00:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:00:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:00:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:00:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:00:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:00:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:00:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:00:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=427] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:00:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=427] Local address: 172.21.15.161, port 55174 2022-07-03T15:00:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=427] Peer address: 172.21.15.101, port 22 2022-07-03T15:00:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=427] Connection lost 2022-07-03T15:00:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=427] Aborting connection 2022-07-03T15:00:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:00:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:00:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:00:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:00:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:00:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:00:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:00:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:00:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:00:04.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:00:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:00:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:00:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:00:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:00:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:00:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:00:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:00:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:00:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:00:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:00:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:00:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:00:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:00:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:00:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:00:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:00:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:00:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:00:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:00:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:00:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:00:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:00:04.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:00:04.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:00:04.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:00:04.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:00:04.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:00:04.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:00:04.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:00:04.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:00:04.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:00:04.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:00:04.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:00:04.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:00:04.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:00:04.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:00:04.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=427] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:00:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=427] Local address: 172.21.15.161, port 55174 2022-07-03T15:00:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=427] Peer address: 172.21.15.101, port 22 2022-07-03T15:00:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=427] Connection lost 2022-07-03T15:00:04.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=427] Aborting connection 2022-07-03T15:00:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:05 smithi161 ceph-mon[91579]: pgmap v7829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:05 smithi188 ceph-mon[92425]: pgmap v7829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:07 smithi161 ceph-mon[91579]: pgmap v7830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:07 smithi188 ceph-mon[92425]: pgmap v7830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:09 smithi161 ceph-mon[91579]: pgmap v7831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:09 smithi188 ceph-mon[92425]: pgmap v7831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:11 smithi161 ceph-mon[91579]: pgmap v7832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:11 smithi188 ceph-mon[92425]: pgmap v7832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:13 smithi161 ceph-mon[91579]: pgmap v7833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:13 smithi188 ceph-mon[92425]: pgmap v7833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:15 smithi161 ceph-mon[91579]: pgmap v7834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:15 smithi188 ceph-mon[92425]: pgmap v7834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:17 smithi161 ceph-mon[91579]: pgmap v7835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:17 smithi188 ceph-mon[92425]: pgmap v7835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:19 smithi161 ceph-mon[91579]: pgmap v7836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:19 smithi188 ceph-mon[92425]: pgmap v7836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:21 smithi161 ceph-mon[91579]: pgmap v7837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:21 smithi188 ceph-mon[92425]: pgmap v7837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:23 smithi161 ceph-mon[91579]: pgmap v7838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:23 smithi188 ceph-mon[92425]: pgmap v7838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:25 smithi161 ceph-mon[91579]: pgmap v7839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:25 smithi188 ceph-mon[92425]: pgmap v7839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:27 smithi161 ceph-mon[91579]: pgmap v7840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:27 smithi188 ceph-mon[92425]: pgmap v7840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:29 smithi161 ceph-mon[91579]: pgmap v7841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:29 smithi188 ceph-mon[92425]: pgmap v7841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:31 smithi161 ceph-mon[91579]: pgmap v7842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:31 smithi188 ceph-mon[92425]: pgmap v7842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:33 smithi161 ceph-mon[91579]: pgmap v7843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:33 smithi188 ceph-mon[92425]: pgmap v7843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:35 smithi161 ceph-mon[91579]: pgmap v7844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:35 smithi188 ceph-mon[92425]: pgmap v7844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:36 smithi161 ceph-mon[91579]: pgmap v7845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:36 smithi188 ceph-mon[92425]: pgmap v7845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:38 smithi161 ceph-mon[91579]: pgmap v7846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:38 smithi188 ceph-mon[92425]: pgmap v7846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:40 smithi161 ceph-mon[91579]: pgmap v7847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:40 smithi188 ceph-mon[92425]: pgmap v7847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:42 smithi161 ceph-mon[91579]: pgmap v7848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:42 smithi188 ceph-mon[92425]: pgmap v7848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:44 smithi161 ceph-mon[91579]: pgmap v7849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:44 smithi188 ceph-mon[92425]: pgmap v7849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:47 smithi188 ceph-mon[92425]: pgmap v7850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:47 smithi161 ceph-mon[91579]: pgmap v7850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:49 smithi188 ceph-mon[92425]: pgmap v7851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:49 smithi161 ceph-mon[91579]: pgmap v7851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:51 smithi188 ceph-mon[92425]: pgmap v7852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:51 smithi161 ceph-mon[91579]: pgmap v7852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:53 smithi188 ceph-mon[92425]: pgmap v7853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:53 smithi161 ceph-mon[91579]: pgmap v7853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:55 smithi188 ceph-mon[92425]: pgmap v7854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:55 smithi161 ceph-mon[91579]: pgmap v7854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:00:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:00:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:57 smithi188 ceph-mon[92425]: pgmap v7855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:57 smithi161 ceph-mon[91579]: pgmap v7855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:00:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:00:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:00:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:00:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:00:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:00:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:00:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:00:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:00:59 smithi161 ceph-mon[91579]: pgmap v7856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:00:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:00:59 smithi188 ceph-mon[92425]: pgmap v7856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:01.568 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:01 smithi188 ceph-mon[92425]: pgmap v7857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:01 smithi161 ceph-mon[91579]: pgmap v7857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-mon[91579]: pgmap v7858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:01:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:01:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:01:03.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:01:03.189+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:01:03.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:01:03.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:01:03.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:01:03.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:01:03.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:01:03.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:01:03.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:01:03.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:01:03.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:01:03.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:01:03.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:01:03.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:01:03.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:01:03.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:01:03.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:01:03.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:01:03.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:01:03.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:01:03.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:01:03.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:01:03.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:01:03.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:01:03.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:01:03.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:01:03.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:01:03.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:01:03.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:01:03.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:01:03.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:01:03.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:01:03.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:01:03.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:01:03.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:01:03.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:01:03.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:01:03.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:01:03.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:01:03.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:01:03.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:01:03.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:01:03.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:01:03.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:01:03.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:01:03.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:01:03.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=429] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:01:03.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=429] Local address: 172.21.15.161, port 55178 2022-07-03T15:01:03.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=429] Peer address: 172.21.15.101, port 22 2022-07-03T15:01:03.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=429] Connection lost 2022-07-03T15:01:03.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=429] Aborting connection 2022-07-03T15:01:03.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:01:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:01:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:03 smithi188 ceph-mon[92425]: pgmap v7858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:01:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:01:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:01:04.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:01:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:01:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:01:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:01:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:01:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:01:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:01:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:01:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:01:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:01:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:01:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:01:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:01:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:01:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:01:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:01:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:01:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:01:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:01:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:01:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:01:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:01:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:01:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:01:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:01:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:01:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:01:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:01:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:01:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:01:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:01:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:01:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:01:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:01:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:01:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:01:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:01:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:01:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:01:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:01:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:01:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:01:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:01:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:01:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:01:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:01:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=429] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:01:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=429] Local address: 172.21.15.161, port 55178 2022-07-03T15:01:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=429] Peer address: 172.21.15.101, port 22 2022-07-03T15:01:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=429] Connection lost 2022-07-03T15:01:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=429] Aborting connection 2022-07-03T15:01:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:01:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:01:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:01:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:01:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:01:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:01:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:01:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:01:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:01:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:01:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:01:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:01:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:01:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:01:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:01:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:01:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:01:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:01:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:01:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:01:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:01:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:01:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:01:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:01:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:01:04.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:01:04.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:01:04.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:01:04.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:01:04.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:01:04.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:01:04.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:01:04.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:01:04.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:01:04.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:01:04.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:01:04.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:01:04.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:01:04.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:01:04.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:01:04.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:01:04.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:01:04.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:01:04.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:01:04.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:01:04.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:01:04.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=429] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:01:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=429] Local address: 172.21.15.161, port 55178 2022-07-03T15:01:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=429] Peer address: 172.21.15.101, port 22 2022-07-03T15:01:04.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=429] Connection lost 2022-07-03T15:01:04.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=429] Aborting connection 2022-07-03T15:01:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:05 smithi161 ceph-mon[91579]: pgmap v7859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:05 smithi188 ceph-mon[92425]: pgmap v7859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:07 smithi161 ceph-mon[91579]: pgmap v7860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:07 smithi188 ceph-mon[92425]: pgmap v7860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:09 smithi161 ceph-mon[91579]: pgmap v7861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:09 smithi188 ceph-mon[92425]: pgmap v7861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:11 smithi161 ceph-mon[91579]: pgmap v7862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:11 smithi188 ceph-mon[92425]: pgmap v7862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:13 smithi161 ceph-mon[91579]: pgmap v7863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:13 smithi188 ceph-mon[92425]: pgmap v7863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:15 smithi161 ceph-mon[91579]: pgmap v7864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:15 smithi188 ceph-mon[92425]: pgmap v7864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:17 smithi161 ceph-mon[91579]: pgmap v7865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:17 smithi188 ceph-mon[92425]: pgmap v7865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:19 smithi161 ceph-mon[91579]: pgmap v7866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:19 smithi188 ceph-mon[92425]: pgmap v7866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:21 smithi161 ceph-mon[91579]: pgmap v7867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:21 smithi188 ceph-mon[92425]: pgmap v7867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:23 smithi161 ceph-mon[91579]: pgmap v7868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:23 smithi188 ceph-mon[92425]: pgmap v7868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:01:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:25 smithi161 ceph-mon[91579]: pgmap v7869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:25 smithi188 ceph-mon[92425]: pgmap v7869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:27 smithi161 ceph-mon[91579]: pgmap v7870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:27 smithi188 ceph-mon[92425]: pgmap v7870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:29 smithi161 ceph-mon[91579]: pgmap v7871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:29 smithi188 ceph-mon[92425]: pgmap v7871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:31 smithi161 ceph-mon[91579]: pgmap v7872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:31 smithi188 ceph-mon[92425]: pgmap v7872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:33 smithi161 ceph-mon[91579]: pgmap v7873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:33 smithi188 ceph-mon[92425]: pgmap v7873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:35 smithi161 ceph-mon[91579]: pgmap v7874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:35 smithi188 ceph-mon[92425]: pgmap v7874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:37 smithi161 ceph-mon[91579]: pgmap v7875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:37 smithi188 ceph-mon[92425]: pgmap v7875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:39 smithi161 ceph-mon[91579]: pgmap v7876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:39 smithi188 ceph-mon[92425]: pgmap v7876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:41 smithi161 ceph-mon[91579]: pgmap v7877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:41 smithi188 ceph-mon[92425]: pgmap v7877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:43 smithi161 ceph-mon[91579]: pgmap v7878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:43 smithi188 ceph-mon[92425]: pgmap v7878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:45 smithi161 ceph-mon[91579]: pgmap v7879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:45 smithi188 ceph-mon[92425]: pgmap v7879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:47 smithi161 ceph-mon[91579]: pgmap v7880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:47 smithi188 ceph-mon[92425]: pgmap v7880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:49 smithi161 ceph-mon[91579]: pgmap v7881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:49 smithi188 ceph-mon[92425]: pgmap v7881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:51 smithi161 ceph-mon[91579]: pgmap v7882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:51 smithi188 ceph-mon[92425]: pgmap v7882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:53 smithi161 ceph-mon[91579]: pgmap v7883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:53 smithi188 ceph-mon[92425]: pgmap v7883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:55 smithi161 ceph-mon[91579]: pgmap v7884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:55 smithi188 ceph-mon[92425]: pgmap v7884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:01:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:01:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:57 smithi161 ceph-mon[91579]: pgmap v7885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:57 smithi188 ceph-mon[92425]: pgmap v7885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:01:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:01:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:01:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:01:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:01:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:01:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:01:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:01:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:01:59 smithi161 ceph-mon[91579]: pgmap v7886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:01:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:01:59 smithi188 ceph-mon[92425]: pgmap v7886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:01 smithi161 ceph-mon[91579]: pgmap v7887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:01 smithi188 ceph-mon[92425]: pgmap v7887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:02 smithi161 ceph-mon[91579]: pgmap v7888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:02 smithi188 ceph-mon[92425]: pgmap v7888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:04.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:02:03.691+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:02:04.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:02:04.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:02:04.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:02:04.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:02:04.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:02:04.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:02:04.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:02:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:02:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:02:04.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:02:04.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:02:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:02:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:02:04.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:02:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:02:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:02:04.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:02:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:02:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:02:04.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:02:04.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:02:04.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:02:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:02:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:02:04.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:02:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:02:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:02:04.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:02:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:02:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:02:04.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:02:04.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:02:04.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:02:04.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:02:04.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:02:04.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:02:04.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:02:04.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:02:04.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:02:04.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:02:04.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:02:04.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:02:04.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:02:04.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:02:04.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=431] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:02:04.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=431] Local address: 172.21.15.161, port 55182 2022-07-03T15:02:04.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=431] Peer address: 172.21.15.101, port 22 2022-07-03T15:02:04.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=431] Connection lost 2022-07-03T15:02:04.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=431] Aborting connection 2022-07-03T15:02:04.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:02:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:02:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: pgmap v7889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:02:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:02:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:02:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:02:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:02:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:02:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:02:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:02:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:02:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:02:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:02:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:02:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:02:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:02:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:02:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:02:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:02:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:02:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:02:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:02:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:02:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:02:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:02:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:02:04.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:02:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:02:04.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:02:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:02:04.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:02:04.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:02:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:02:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:02:04.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:02:04.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:02:04.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:02:04.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:02:04.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:02:04.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:02:04.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:02:04.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:02:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:02:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:02:04.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:02:04.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:02:04.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:02:04.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:02:04.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:02:04.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:02:04.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:02:04.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=431] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:02:04.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=431] Local address: 172.21.15.161, port 55182 2022-07-03T15:02:04.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=431] Peer address: 172.21.15.101, port 22 2022-07-03T15:02:04.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=431] Connection lost 2022-07-03T15:02:04.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=431] Aborting connection 2022-07-03T15:02:04.908 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: pgmap v7889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:02:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:02:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:02:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:02:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:02:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:02:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:02:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:02:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:02:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:02:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:02:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:02:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:02:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:02:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:02:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:02:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:02:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:02:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:02:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:02:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:02:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:02:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:02:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:02:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:02:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:02:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:02:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:02:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:02:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:02:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:02:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:02:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:02:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:02:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:02:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:02:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:02:04.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:02:04.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:02:04.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:02:04.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:02:04.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:02:04.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:02:04.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:02:04.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:02:04.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:02:04.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:02:04.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:02:04.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:02:04.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=431] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:02:04.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=431] Local address: 172.21.15.161, port 55182 2022-07-03T15:02:04.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=431] Peer address: 172.21.15.101, port 22 2022-07-03T15:02:04.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=431] Connection lost 2022-07-03T15:02:04.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=431] Aborting connection 2022-07-03T15:02:04.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.941 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:04.943 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:02:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:07 smithi188 ceph-mon[92425]: pgmap v7890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:07 smithi161 ceph-mon[91579]: pgmap v7890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:09 smithi188 ceph-mon[92425]: pgmap v7891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:09 smithi161 ceph-mon[91579]: pgmap v7891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:11 smithi161 ceph-mon[91579]: pgmap v7892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:11 smithi188 ceph-mon[92425]: pgmap v7892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:13 smithi161 ceph-mon[91579]: pgmap v7893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:13 smithi188 ceph-mon[92425]: pgmap v7893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:15 smithi161 ceph-mon[91579]: pgmap v7894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:15 smithi188 ceph-mon[92425]: pgmap v7894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:17 smithi161 ceph-mon[91579]: pgmap v7895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:17 smithi188 ceph-mon[92425]: pgmap v7895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:19 smithi161 ceph-mon[91579]: pgmap v7896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:19 smithi188 ceph-mon[92425]: pgmap v7896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:21 smithi161 ceph-mon[91579]: pgmap v7897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:21 smithi188 ceph-mon[92425]: pgmap v7897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:23 smithi161 ceph-mon[91579]: pgmap v7898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:23 smithi188 ceph-mon[92425]: pgmap v7898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:25 smithi161 ceph-mon[91579]: pgmap v7899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:25 smithi188 ceph-mon[92425]: pgmap v7899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:27 smithi161 ceph-mon[91579]: pgmap v7900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:27 smithi188 ceph-mon[92425]: pgmap v7900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:29 smithi161 ceph-mon[91579]: pgmap v7901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:29 smithi188 ceph-mon[92425]: pgmap v7901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:31 smithi161 ceph-mon[91579]: pgmap v7902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:31 smithi188 ceph-mon[92425]: pgmap v7902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:33 smithi161 ceph-mon[91579]: pgmap v7903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:33 smithi188 ceph-mon[92425]: pgmap v7903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:35 smithi161 ceph-mon[91579]: pgmap v7904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:35 smithi188 ceph-mon[92425]: pgmap v7904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:37 smithi161 ceph-mon[91579]: pgmap v7905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:37 smithi188 ceph-mon[92425]: pgmap v7905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:39 smithi161 ceph-mon[91579]: pgmap v7906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:39 smithi188 ceph-mon[92425]: pgmap v7906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:41 smithi161 ceph-mon[91579]: pgmap v7907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:41 smithi188 ceph-mon[92425]: pgmap v7907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:43 smithi161 ceph-mon[91579]: pgmap v7908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:43 smithi188 ceph-mon[92425]: pgmap v7908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:45 smithi161 ceph-mon[91579]: pgmap v7909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:45 smithi188 ceph-mon[92425]: pgmap v7909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:47 smithi161 ceph-mon[91579]: pgmap v7910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:47 smithi188 ceph-mon[92425]: pgmap v7910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:49 smithi161 ceph-mon[91579]: pgmap v7911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:49 smithi188 ceph-mon[92425]: pgmap v7911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:51 smithi161 ceph-mon[91579]: pgmap v7912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:51 smithi188 ceph-mon[92425]: pgmap v7912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:53 smithi161 ceph-mon[91579]: pgmap v7913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:53 smithi188 ceph-mon[92425]: pgmap v7913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:55 smithi161 ceph-mon[91579]: pgmap v7914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:55 smithi188 ceph-mon[92425]: pgmap v7914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:02:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:02:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:57 smithi161 ceph-mon[91579]: pgmap v7915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:57 smithi188 ceph-mon[92425]: pgmap v7915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:02:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:02:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:02:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:02:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:02:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:02:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:02:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:02:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:02:59 smithi161 ceph-mon[91579]: pgmap v7916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:02:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:02:59 smithi188 ceph-mon[92425]: pgmap v7916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:01 smithi161 ceph-mon[91579]: pgmap v7917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:01 smithi188 ceph-mon[92425]: pgmap v7917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:03 smithi161 ceph-mon[91579]: pgmap v7918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:03 smithi188 ceph-mon[92425]: pgmap v7918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:04.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:03:04.614+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:03:04.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:03:04.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:03:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:03:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:03:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:03:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:03:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:03:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:03:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:03:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:03:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:03:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:03:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:03:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:03:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:03:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:03:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:03:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:03:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:03:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:03:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:03:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:03:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:03:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:03:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:03:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:03:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:03:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:03:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:03:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:03:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:03:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:03:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:03:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:03:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:03:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:03:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:03:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:03:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:03:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:03:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:03:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:03:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:03:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:03:04.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=433] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:03:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=433] Local address: 172.21.15.161, port 55186 2022-07-03T15:03:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=433] Peer address: 172.21.15.101, port 22 2022-07-03T15:03:04.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=433] Connection lost 2022-07-03T15:03:04.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=433] Aborting connection 2022-07-03T15:03:04.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:03:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:03:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:05 smithi161 ceph-mon[91579]: pgmap v7919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:03:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:03:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:03:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:05 smithi188 ceph-mon[92425]: pgmap v7919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:03:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:03:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:03:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:03:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:03:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:03:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:03:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:03:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:03:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:03:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:03:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:03:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:03:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:03:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:03:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:03:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:03:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:03:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:03:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:03:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:03:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:03:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:03:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:03:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:03:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:03:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:03:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:03:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:03:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:03:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:03:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:03:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:03:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:03:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:03:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:03:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:03:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:03:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:03:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:03:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:03:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:03:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:03:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:03:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:03:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:03:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:03:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:03:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:03:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=433] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:03:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=433] Local address: 172.21.15.161, port 55186 2022-07-03T15:03:06.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=433] Peer address: 172.21.15.101, port 22 2022-07-03T15:03:06.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=433] Connection lost 2022-07-03T15:03:06.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=433] Aborting connection 2022-07-03T15:03:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:03:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:03:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:03:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:03:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:03:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:03:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:03:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:03:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:03:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:03:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:03:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:03:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:03:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:03:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:03:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:03:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:03:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:03:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:03:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:03:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:03:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:03:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:03:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:03:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:03:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:03:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:03:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:03:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:03:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:03:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:03:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:03:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:03:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:03:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:03:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:03:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:03:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:03:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:03:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:03:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:03:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:03:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:03:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:03:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:03:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:03:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=433] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:03:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=433] Local address: 172.21.15.161, port 55186 2022-07-03T15:03:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=433] Peer address: 172.21.15.101, port 22 2022-07-03T15:03:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=433] Connection lost 2022-07-03T15:03:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=433] Aborting connection 2022-07-03T15:03:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:07 smithi161 ceph-mon[91579]: pgmap v7920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:07 smithi188 ceph-mon[92425]: pgmap v7920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:09 smithi161 ceph-mon[91579]: pgmap v7921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:09 smithi188 ceph-mon[92425]: pgmap v7921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:11 smithi161 ceph-mon[91579]: pgmap v7922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:11 smithi188 ceph-mon[92425]: pgmap v7922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:13 smithi161 ceph-mon[91579]: pgmap v7923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:13 smithi188 ceph-mon[92425]: pgmap v7923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:15 smithi161 ceph-mon[91579]: pgmap v7924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:15 smithi188 ceph-mon[92425]: pgmap v7924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:17 smithi161 ceph-mon[91579]: pgmap v7925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:17 smithi188 ceph-mon[92425]: pgmap v7925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:18 smithi161 ceph-mon[91579]: pgmap v7926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:18 smithi188 ceph-mon[92425]: pgmap v7926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:20 smithi161 ceph-mon[91579]: pgmap v7927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:20 smithi188 ceph-mon[92425]: pgmap v7927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:22 smithi161 ceph-mon[91579]: pgmap v7928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:22 smithi188 ceph-mon[92425]: pgmap v7928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:24 smithi161 ceph-mon[91579]: pgmap v7929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:24 smithi188 ceph-mon[92425]: pgmap v7929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:27 smithi161 ceph-mon[91579]: pgmap v7930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:27 smithi188 ceph-mon[92425]: pgmap v7930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:29 smithi161 ceph-mon[91579]: pgmap v7931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:29 smithi188 ceph-mon[92425]: pgmap v7931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:31 smithi161 ceph-mon[91579]: pgmap v7932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:31 smithi188 ceph-mon[92425]: pgmap v7932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:33 smithi161 ceph-mon[91579]: pgmap v7933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:33 smithi188 ceph-mon[92425]: pgmap v7933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:35 smithi161 ceph-mon[91579]: pgmap v7934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:35 smithi188 ceph-mon[92425]: pgmap v7934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:37 smithi161 ceph-mon[91579]: pgmap v7935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:37 smithi188 ceph-mon[92425]: pgmap v7935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:39 smithi161 ceph-mon[91579]: pgmap v7936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:39 smithi188 ceph-mon[92425]: pgmap v7936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:41 smithi161 ceph-mon[91579]: pgmap v7937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:41 smithi188 ceph-mon[92425]: pgmap v7937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:43 smithi161 ceph-mon[91579]: pgmap v7938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:43 smithi188 ceph-mon[92425]: pgmap v7938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:45 smithi161 ceph-mon[91579]: pgmap v7939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:45 smithi188 ceph-mon[92425]: pgmap v7939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:47 smithi161 ceph-mon[91579]: pgmap v7940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:47 smithi188 ceph-mon[92425]: pgmap v7940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:49 smithi161 ceph-mon[91579]: pgmap v7941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:49 smithi188 ceph-mon[92425]: pgmap v7941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:51 smithi161 ceph-mon[91579]: pgmap v7942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:51 smithi188 ceph-mon[92425]: pgmap v7942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:53 smithi161 ceph-mon[91579]: pgmap v7943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:53 smithi188 ceph-mon[92425]: pgmap v7943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:55 smithi161 ceph-mon[91579]: pgmap v7944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:55 smithi188 ceph-mon[92425]: pgmap v7944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:03:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:03:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:57 smithi161 ceph-mon[91579]: pgmap v7945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:57 smithi188 ceph-mon[92425]: pgmap v7945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:03:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:03:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:03:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:03:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:03:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:03:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:03:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:03:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:03:59 smithi161 ceph-mon[91579]: pgmap v7946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:03:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:03:59 smithi188 ceph-mon[92425]: pgmap v7946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:01 smithi161 ceph-mon[91579]: pgmap v7947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:01 smithi188 ceph-mon[92425]: pgmap v7947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:03 smithi161 ceph-mon[91579]: pgmap v7948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:03 smithi188 ceph-mon[92425]: pgmap v7948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:05 smithi161 ceph-mon[91579]: pgmap v7949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:04:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:04:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:04:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:05 smithi188 ceph-mon[92425]: pgmap v7949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:04:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:04:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:04:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:07 smithi161 ceph-mon[91579]: pgmap v7950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:07 smithi188 ceph-mon[92425]: pgmap v7950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:09.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:09 smithi161 ceph-mon[91579]: pgmap v7951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:09 smithi188 ceph-mon[92425]: pgmap v7951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:11 smithi161 ceph-mon[91579]: pgmap v7952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:11 smithi188 ceph-mon[92425]: pgmap v7952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:13 smithi161 ceph-mon[91579]: pgmap v7953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:13 smithi188 ceph-mon[92425]: pgmap v7953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:15 smithi161 ceph-mon[91579]: pgmap v7954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:15 smithi188 ceph-mon[92425]: pgmap v7954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:17 smithi161 ceph-mon[91579]: pgmap v7955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:17 smithi188 ceph-mon[92425]: pgmap v7955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:19 smithi161 ceph-mon[91579]: pgmap v7956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:19 smithi188 ceph-mon[92425]: pgmap v7956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:21 smithi161 ceph-mon[91579]: pgmap v7957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:21 smithi188 ceph-mon[92425]: pgmap v7957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:23 smithi161 ceph-mon[91579]: pgmap v7958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:23 smithi188 ceph-mon[92425]: pgmap v7958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:25 smithi161 ceph-mon[91579]: pgmap v7959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:25 smithi188 ceph-mon[92425]: pgmap v7959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:27 smithi161 ceph-mon[91579]: pgmap v7960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:27 smithi188 ceph-mon[92425]: pgmap v7960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:29 smithi161 ceph-mon[91579]: pgmap v7961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:29 smithi188 ceph-mon[92425]: pgmap v7961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:30 smithi161 ceph-mon[91579]: pgmap v7962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:30 smithi188 ceph-mon[92425]: pgmap v7962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:32 smithi161 ceph-mon[91579]: pgmap v7963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:32 smithi188 ceph-mon[92425]: pgmap v7963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:34 smithi161 ceph-mon[91579]: pgmap v7964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:34 smithi188 ceph-mon[92425]: pgmap v7964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:04:35.111+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:04:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:04:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:04:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:04:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:04:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:04:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:04:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:04:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:04:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:04:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:04:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:04:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:04:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:04:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:04:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:04:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:04:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:04:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:04:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:04:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:04:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:04:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:04:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:04:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:04:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:04:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:04:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:04:35.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:04:35.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:04:35.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:04:35.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:04:35.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:04:35.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:04:35.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:04:35.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:04:35.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:04:35.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:04:35.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:04:35.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:04:35.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:04:35.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:04:35.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:04:35.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:04:35.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:04:35.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=435] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:04:35.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=435] Local address: 172.21.15.161, port 55190 2022-07-03T15:04:35.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=435] Peer address: 172.21.15.101, port 22 2022-07-03T15:04:35.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=435] Connection lost 2022-07-03T15:04:35.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=435] Aborting connection 2022-07-03T15:04:35.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:04:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:04:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:04:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:04:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:04:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:04:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:04:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:04:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:04:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:04:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:04:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:04:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:04:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:04:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:04:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:04:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:04:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:04:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:04:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:04:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:04:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:04:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:04:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:04:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:04:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:04:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:04:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:04:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:04:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:04:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:04:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:04:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:04:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:04:35.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:04:35.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=435] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:04:35.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=435] Local address: 172.21.15.161, port 55190 2022-07-03T15:04:35.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=435] Peer address: 172.21.15.101, port 22 2022-07-03T15:04:35.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=435] Connection lost 2022-07-03T15:04:35.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=435] Aborting connection 2022-07-03T15:04:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:04:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:04:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:04:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:04:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:04:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:04:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:04:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:04:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:04:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:04:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:04:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:04:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:04:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:04:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:04:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:04:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:04:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:04:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:04:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:04:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:04:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:04:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=435] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:04:35.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=435] Local address: 172.21.15.161, port 55190 2022-07-03T15:04:35.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=435] Peer address: 172.21.15.101, port 22 2022-07-03T15:04:35.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=435] Connection lost 2022-07-03T15:04:35.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=435] Aborting connection 2022-07-03T15:04:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:37 smithi161 ceph-mon[91579]: pgmap v7965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:37 smithi188 ceph-mon[92425]: pgmap v7965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:39 smithi161 ceph-mon[91579]: pgmap v7966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:39 smithi188 ceph-mon[92425]: pgmap v7966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:41 smithi161 ceph-mon[91579]: pgmap v7967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:41 smithi188 ceph-mon[92425]: pgmap v7967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:43 smithi161 ceph-mon[91579]: pgmap v7968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:43 smithi188 ceph-mon[92425]: pgmap v7968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:45 smithi161 ceph-mon[91579]: pgmap v7969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:45 smithi188 ceph-mon[92425]: pgmap v7969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:47 smithi161 ceph-mon[91579]: pgmap v7970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:47 smithi188 ceph-mon[92425]: pgmap v7970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:49 smithi161 ceph-mon[91579]: pgmap v7971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:49 smithi188 ceph-mon[92425]: pgmap v7971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:51 smithi161 ceph-mon[91579]: pgmap v7972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:51 smithi188 ceph-mon[92425]: pgmap v7972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:53 smithi161 ceph-mon[91579]: pgmap v7973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:53 smithi188 ceph-mon[92425]: pgmap v7973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:55 smithi161 ceph-mon[91579]: pgmap v7974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:55 smithi188 ceph-mon[92425]: pgmap v7974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:04:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:04:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:57 smithi161 ceph-mon[91579]: pgmap v7975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:57 smithi188 ceph-mon[92425]: pgmap v7975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:04:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:04:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:04:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:04:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:04:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:04:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:04:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:04:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:04:59 smithi161 ceph-mon[91579]: pgmap v7976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:04:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:04:59 smithi188 ceph-mon[92425]: pgmap v7976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:01 smithi161 ceph-mon[91579]: pgmap v7977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:01 smithi188 ceph-mon[92425]: pgmap v7977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:03 smithi161 ceph-mon[91579]: pgmap v7978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:03 smithi188 ceph-mon[92425]: pgmap v7978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:05 smithi161 ceph-mon[91579]: pgmap v7979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:05 smithi188 ceph-mon[92425]: pgmap v7979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:07 smithi161 ceph-mon[91579]: pgmap v7980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:07 smithi188 ceph-mon[92425]: pgmap v7980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:09 smithi161 ceph-mon[91579]: pgmap v7981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:09 smithi188 ceph-mon[92425]: pgmap v7981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:11 smithi161 ceph-mon[91579]: pgmap v7982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:11 smithi188 ceph-mon[92425]: pgmap v7982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:13 smithi161 ceph-mon[91579]: pgmap v7983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:13 smithi188 ceph-mon[92425]: pgmap v7983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:15 smithi161 ceph-mon[91579]: pgmap v7984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:15 smithi188 ceph-mon[92425]: pgmap v7984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:17 smithi161 ceph-mon[91579]: pgmap v7985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:17 smithi188 ceph-mon[92425]: pgmap v7985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:19 smithi161 ceph-mon[91579]: pgmap v7986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:19 smithi188 ceph-mon[92425]: pgmap v7986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:21 smithi161 ceph-mon[91579]: pgmap v7987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:21 smithi188 ceph-mon[92425]: pgmap v7987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:23 smithi161 ceph-mon[91579]: pgmap v7988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:23 smithi188 ceph-mon[92425]: pgmap v7988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:25 smithi161 ceph-mon[91579]: pgmap v7989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:25 smithi188 ceph-mon[92425]: pgmap v7989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:27 smithi161 ceph-mon[91579]: pgmap v7990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:27 smithi188 ceph-mon[92425]: pgmap v7990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:29 smithi161 ceph-mon[91579]: pgmap v7991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:29 smithi188 ceph-mon[92425]: pgmap v7991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:31 smithi161 ceph-mon[91579]: pgmap v7992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:31 smithi188 ceph-mon[92425]: pgmap v7992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:33 smithi161 ceph-mon[91579]: pgmap v7993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:33 smithi188 ceph-mon[92425]: pgmap v7993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:35 smithi188 ceph-mon[92425]: pgmap v7994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:05:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:05:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:05:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-mon[91579]: pgmap v7994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:05:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:05:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:05:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:05:35.146+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:05:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:05:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:05:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:05:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:05:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:05:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:05:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:05:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:05:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:05:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:05:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:05:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:05:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:05:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:05:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:05:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:05:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:05:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:05:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:05:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:05:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=437] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:05:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=437] Local address: 172.21.15.161, port 55194 2022-07-03T15:05:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=437] Peer address: 172.21.15.101, port 22 2022-07-03T15:05:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=437] Connection lost 2022-07-03T15:05:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=437] Aborting connection 2022-07-03T15:05:35.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:05:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:05:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:05:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:05:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:05:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:05:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:05:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:05:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:05:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:05:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:05:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:05:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:05:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:05:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:05:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:05:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:05:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:05:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:05:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:05:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:05:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:05:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:05:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:05:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:05:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:05:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:05:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:05:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:05:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:05:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:05:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:05:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:05:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:05:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:05:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:05:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:05:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:05:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:05:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:05:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:05:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:05:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:05:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:05:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:05:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:05:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:05:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=437] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:05:36.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=437] Local address: 172.21.15.161, port 55194 2022-07-03T15:05:36.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=437] Peer address: 172.21.15.101, port 22 2022-07-03T15:05:36.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=437] Connection lost 2022-07-03T15:05:36.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=437] Aborting connection 2022-07-03T15:05:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:05:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:05:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:05:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:05:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:05:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:05:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:05:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:05:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:05:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:05:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:05:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:05:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:05:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:05:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:05:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:05:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:05:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:05:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:05:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:05:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:05:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:05:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:05:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:05:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:05:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:05:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:05:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:05:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:05:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:05:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:05:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:05:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:05:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:05:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:05:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:05:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:05:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:05:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:05:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:05:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:05:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:05:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:05:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=437] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:05:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=437] Local address: 172.21.15.161, port 55194 2022-07-03T15:05:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=437] Peer address: 172.21.15.101, port 22 2022-07-03T15:05:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=437] Connection lost 2022-07-03T15:05:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=437] Aborting connection 2022-07-03T15:05:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:37 smithi161 ceph-mon[91579]: pgmap v7995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:37 smithi188 ceph-mon[92425]: pgmap v7995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: pgmap v7996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 189.3M 2022-07-03T15:05:39.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 198524018: error parsing value: Value '198524018' is below minimum 939524096 2022-07-03T15:05:39.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: pgmap v7996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:05:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 189.3M 2022-07-03T15:05:39.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 198524018: error parsing value: Value '198524018' is below minimum 939524096 2022-07-03T15:05:39.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:05:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:39 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:05:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:39 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:05:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:39 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:05:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:39 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:05:41.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:40 smithi161 ceph-mon[91579]: pgmap v7997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:40 smithi188 ceph-mon[92425]: pgmap v7997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:42 smithi161 ceph-mon[91579]: pgmap v7998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:42 smithi188 ceph-mon[92425]: pgmap v7998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:44 smithi161 ceph-mon[91579]: pgmap v7999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:44 smithi188 ceph-mon[92425]: pgmap v7999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:47 smithi161 ceph-mon[91579]: pgmap v8000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:47 smithi188 ceph-mon[92425]: pgmap v8000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:49 smithi161 ceph-mon[91579]: pgmap v8001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:49 smithi188 ceph-mon[92425]: pgmap v8001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:51 smithi161 ceph-mon[91579]: pgmap v8002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:51 smithi188 ceph-mon[92425]: pgmap v8002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:53 smithi161 ceph-mon[91579]: pgmap v8003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:53 smithi188 ceph-mon[92425]: pgmap v8003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:55 smithi161 ceph-mon[91579]: pgmap v8004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:55 smithi188 ceph-mon[92425]: pgmap v8004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:05:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:05:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:57 smithi161 ceph-mon[91579]: pgmap v8005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:57 smithi188 ceph-mon[92425]: pgmap v8005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:05:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:05:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:05:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:05:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:05:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:05:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:05:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:05:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:05:59 smithi161 ceph-mon[91579]: pgmap v8006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:05:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:05:59 smithi188 ceph-mon[92425]: pgmap v8006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:01 smithi161 ceph-mon[91579]: pgmap v8007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:01 smithi188 ceph-mon[92425]: pgmap v8007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:03 smithi161 ceph-mon[91579]: pgmap v8008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:03 smithi188 ceph-mon[92425]: pgmap v8008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:05 smithi161 ceph-mon[91579]: pgmap v8009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:05 smithi188 ceph-mon[92425]: pgmap v8009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:07 smithi161 ceph-mon[91579]: pgmap v8010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:07 smithi188 ceph-mon[92425]: pgmap v8010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:09 smithi161 ceph-mon[91579]: pgmap v8011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:09 smithi188 ceph-mon[92425]: pgmap v8011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:11 smithi161 ceph-mon[91579]: pgmap v8012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:11 smithi188 ceph-mon[92425]: pgmap v8012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:13 smithi161 ceph-mon[91579]: pgmap v8013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:13 smithi188 ceph-mon[92425]: pgmap v8013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:15.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:15 smithi161 ceph-mon[91579]: pgmap v8014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:15.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:15.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:15 smithi188 ceph-mon[92425]: pgmap v8014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:17 smithi161 ceph-mon[91579]: pgmap v8015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:17 smithi188 ceph-mon[92425]: pgmap v8015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:19 smithi161 ceph-mon[91579]: pgmap v8016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:19 smithi188 ceph-mon[92425]: pgmap v8016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:21 smithi161 ceph-mon[91579]: pgmap v8017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:21 smithi188 ceph-mon[92425]: pgmap v8017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:23 smithi161 ceph-mon[91579]: pgmap v8018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:23 smithi188 ceph-mon[92425]: pgmap v8018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:25 smithi161 ceph-mon[91579]: pgmap v8019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:25 smithi188 ceph-mon[92425]: pgmap v8019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:27 smithi161 ceph-mon[91579]: pgmap v8020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:27 smithi188 ceph-mon[92425]: pgmap v8020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:29 smithi161 ceph-mon[91579]: pgmap v8021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:29 smithi188 ceph-mon[92425]: pgmap v8021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:31 smithi161 ceph-mon[91579]: pgmap v8022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:31 smithi188 ceph-mon[92425]: pgmap v8022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:33 smithi161 ceph-mon[91579]: pgmap v8023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:33 smithi188 ceph-mon[92425]: pgmap v8023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:35 smithi161 ceph-mon[91579]: pgmap v8024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:35 smithi188 ceph-mon[92425]: pgmap v8024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:37 smithi161 ceph-mon[91579]: pgmap v8025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:37 smithi188 ceph-mon[92425]: pgmap v8025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:38.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:06:38.394+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:06:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:06:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:06:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:06:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:06:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:06:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:06:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:06:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:06:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:06:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:06:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:06:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:06:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:06:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:06:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:06:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:06:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:06:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:06:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:06:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:06:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:06:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:06:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:06:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:06:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:06:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:06:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:06:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:06:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:06:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:06:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:06:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:06:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:06:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:06:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:06:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:06:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:06:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:06:38.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:06:38.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:06:38.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:06:38.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:06:38.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:06:38.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:06:38.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=439] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:06:38.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=439] Local address: 172.21.15.161, port 55216 2022-07-03T15:06:38.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=439] Peer address: 172.21.15.101, port 22 2022-07-03T15:06:38.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=439] Connection lost 2022-07-03T15:06:38.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=439] Aborting connection 2022-07-03T15:06:38.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:06:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:06:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:39 smithi161 ceph-mon[91579]: pgmap v8026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:06:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:06:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:06:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:39 smithi188 ceph-mon[92425]: pgmap v8026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:06:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:06:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:06:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:06:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:06:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:06:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:06:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:06:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:06:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:06:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:06:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:06:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:06:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:06:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:06:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:06:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:06:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:06:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:06:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:06:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:06:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:06:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:06:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:06:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:06:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:06:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:06:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:06:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:06:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:06:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:06:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:06:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:06:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:06:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:06:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:06:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:06:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:06:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:06:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:06:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:06:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:06:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:06:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:06:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:06:40.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:06:40.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:06:40.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:06:40.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:06:40.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=439] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:06:40.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=439] Local address: 172.21.15.161, port 55216 2022-07-03T15:06:40.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=439] Peer address: 172.21.15.101, port 22 2022-07-03T15:06:40.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=439] Connection lost 2022-07-03T15:06:40.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=439] Aborting connection 2022-07-03T15:06:40.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:40.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:06:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:06:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:06:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:06:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:06:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:06:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:06:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:06:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:06:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:06:40.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:06:40.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:06:40.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:06:40.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:06:40.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:06:40.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:06:40.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:06:40.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:06:40.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:06:40.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:06:40.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:06:40.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:06:40.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:06:40.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:06:40.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:06:40.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:06:40.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:06:40.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:06:40.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:06:40.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:06:40.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:06:40.702 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:06:40.703 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:06:40.703 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:06:40.704 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:06:40.704 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:06:40.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:06:40.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:06:40.705 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:06:40.706 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:06:40.706 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:06:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:06:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:06:40.707 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:06:40.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:06:40.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:06:40.708 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=439] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:06:40.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=439] Local address: 172.21.15.161, port 55216 2022-07-03T15:06:40.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=439] Peer address: 172.21.15.101, port 22 2022-07-03T15:06:40.709 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=439] Connection lost 2022-07-03T15:06:40.710 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=439] Aborting connection 2022-07-03T15:06:40.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:40.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:41 smithi161 ceph-mon[91579]: pgmap v8027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:41 smithi188 ceph-mon[92425]: pgmap v8027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:43 smithi161 ceph-mon[91579]: pgmap v8028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:43 smithi188 ceph-mon[92425]: pgmap v8028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:45 smithi161 ceph-mon[91579]: pgmap v8029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:45 smithi188 ceph-mon[92425]: pgmap v8029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:47 smithi161 ceph-mon[91579]: pgmap v8030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:47 smithi188 ceph-mon[92425]: pgmap v8030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:48 smithi161 ceph-mon[91579]: pgmap v8031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:48 smithi188 ceph-mon[92425]: pgmap v8031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:50 smithi161 ceph-mon[91579]: pgmap v8032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:50 smithi188 ceph-mon[92425]: pgmap v8032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:52 smithi188 ceph-mon[92425]: pgmap v8033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:52 smithi161 ceph-mon[91579]: pgmap v8033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:54 smithi188 ceph-mon[92425]: pgmap v8034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:54 smithi161 ceph-mon[91579]: pgmap v8034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:06:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:06:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:06:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:57 smithi161 ceph-mon[91579]: pgmap v8035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:57 smithi188 ceph-mon[92425]: pgmap v8035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:06:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:06:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:06:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:06:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:06:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:06:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:06:59 smithi161 ceph-mon[91579]: pgmap v8036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:06:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:06:59 smithi188 ceph-mon[92425]: pgmap v8036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:01 smithi161 ceph-mon[91579]: pgmap v8037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:01 smithi188 ceph-mon[92425]: pgmap v8037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:03 smithi161 ceph-mon[91579]: pgmap v8038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:03 smithi188 ceph-mon[92425]: pgmap v8038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:05 smithi161 ceph-mon[91579]: pgmap v8039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:05.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:05 smithi188 ceph-mon[92425]: pgmap v8039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:07 smithi161 ceph-mon[91579]: pgmap v8040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:07 smithi188 ceph-mon[92425]: pgmap v8040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:09 smithi161 ceph-mon[91579]: pgmap v8041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:09 smithi188 ceph-mon[92425]: pgmap v8041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:11 smithi161 ceph-mon[91579]: pgmap v8042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:11 smithi188 ceph-mon[92425]: pgmap v8042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:13 smithi161 ceph-mon[91579]: pgmap v8043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:13 smithi188 ceph-mon[92425]: pgmap v8043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:15 smithi161 ceph-mon[91579]: pgmap v8044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:15 smithi188 ceph-mon[92425]: pgmap v8044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:17 smithi161 ceph-mon[91579]: pgmap v8045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:17 smithi188 ceph-mon[92425]: pgmap v8045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:19 smithi161 ceph-mon[91579]: pgmap v8046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:19 smithi188 ceph-mon[92425]: pgmap v8046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:21 smithi161 ceph-mon[91579]: pgmap v8047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:21 smithi188 ceph-mon[92425]: pgmap v8047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:23 smithi161 ceph-mon[91579]: pgmap v8048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:23 smithi188 ceph-mon[92425]: pgmap v8048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:25 smithi161 ceph-mon[91579]: pgmap v8049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:25 smithi188 ceph-mon[92425]: pgmap v8049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:27 smithi161 ceph-mon[91579]: pgmap v8050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:27 smithi188 ceph-mon[92425]: pgmap v8050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:29 smithi161 ceph-mon[91579]: pgmap v8051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:29 smithi188 ceph-mon[92425]: pgmap v8051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:31 smithi161 ceph-mon[91579]: pgmap v8052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:31 smithi188 ceph-mon[92425]: pgmap v8052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:33 smithi161 ceph-mon[91579]: pgmap v8053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:33 smithi188 ceph-mon[92425]: pgmap v8053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:35 smithi161 ceph-mon[91579]: pgmap v8054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:35 smithi188 ceph-mon[92425]: pgmap v8054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:37 smithi161 ceph-mon[91579]: pgmap v8055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:37 smithi188 ceph-mon[92425]: pgmap v8055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:39.315 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:07:38.896+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:07:39.316 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:07:39.316 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:07:39.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:07:39.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:07:39.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:07:39.318 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:07:39.318 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:07:39.318 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:07:39.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:07:39.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:07:39.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:07:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:07:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:07:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:07:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:07:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:07:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:07:39.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:07:39.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:07:39.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:07:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:07:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:07:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:07:39.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:07:39.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:07:39.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:07:39.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:07:39.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:07:39.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:07:39.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:07:39.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:07:39.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:07:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:07:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:07:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:07:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:07:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:07:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:07:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:07:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:07:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:07:39.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:07:39.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:07:39.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:07:39.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=441] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:07:39.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=441] Local address: 172.21.15.161, port 55244 2022-07-03T15:07:39.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=441] Peer address: 172.21.15.101, port 22 2022-07-03T15:07:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=441] Connection lost 2022-07-03T15:07:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=441] Aborting connection 2022-07-03T15:07:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:07:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:07:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:39 smithi161 ceph-mon[91579]: pgmap v8056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:07:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:07:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:07:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:39 smithi188 ceph-mon[92425]: pgmap v8056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:07:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:07:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:07:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:07:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:07:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:07:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:07:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:07:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:07:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:07:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:07:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:07:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:07:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:07:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:07:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:07:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:07:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:07:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:07:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:07:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:07:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:07:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:07:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:07:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:07:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:07:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:07:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:07:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:07:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:07:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:07:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:07:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:07:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:07:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:07:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:07:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:07:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:07:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:07:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:07:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:07:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:07:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:07:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:07:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:07:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:07:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:07:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:07:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:07:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=441] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:07:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=441] Local address: 172.21.15.161, port 55244 2022-07-03T15:07:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=441] Peer address: 172.21.15.101, port 22 2022-07-03T15:07:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=441] Connection lost 2022-07-03T15:07:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=441] Aborting connection 2022-07-03T15:07:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:40.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:07:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:07:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:07:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:07:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:07:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:07:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:07:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:07:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:07:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:07:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:07:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:07:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:07:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:07:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:07:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:07:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:07:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:07:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:07:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:07:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:07:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:07:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:07:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:07:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:07:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:07:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:07:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:07:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:07:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:07:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:07:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:07:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:07:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:07:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:07:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:07:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:07:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:07:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:07:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:07:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:07:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:07:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:07:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:07:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:07:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:07:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=441] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:07:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=441] Local address: 172.21.15.161, port 55244 2022-07-03T15:07:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=441] Peer address: 172.21.15.101, port 22 2022-07-03T15:07:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=441] Connection lost 2022-07-03T15:07:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=441] Aborting connection 2022-07-03T15:07:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:41 smithi161 ceph-mon[91579]: pgmap v8057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:41 smithi188 ceph-mon[92425]: pgmap v8057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:43 smithi161 ceph-mon[91579]: pgmap v8058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:43 smithi188 ceph-mon[92425]: pgmap v8058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:45 smithi161 ceph-mon[91579]: pgmap v8059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:45 smithi188 ceph-mon[92425]: pgmap v8059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:47 smithi161 ceph-mon[91579]: pgmap v8060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:47 smithi188 ceph-mon[92425]: pgmap v8060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:49 smithi161 ceph-mon[91579]: pgmap v8061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:49 smithi188 ceph-mon[92425]: pgmap v8061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:51 smithi161 ceph-mon[91579]: pgmap v8062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:51 smithi188 ceph-mon[92425]: pgmap v8062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:53 smithi161 ceph-mon[91579]: pgmap v8063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:53 smithi188 ceph-mon[92425]: pgmap v8063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:55 smithi161 ceph-mon[91579]: pgmap v8064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:55 smithi188 ceph-mon[92425]: pgmap v8064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:07:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:07:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:07:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:57 smithi161 ceph-mon[91579]: pgmap v8065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:57 smithi188 ceph-mon[92425]: pgmap v8065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:58 smithi161 ceph-mon[91579]: pgmap v8066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:07:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:07:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:07:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:07:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:58 smithi188 ceph-mon[92425]: pgmap v8066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:07:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:07:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:07:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:07:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:08:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:00 smithi188 ceph-mon[92425]: pgmap v8067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:00 smithi161 ceph-mon[91579]: pgmap v8067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:02 smithi188 ceph-mon[92425]: pgmap v8068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:02 smithi161 ceph-mon[91579]: pgmap v8068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:04 smithi188 ceph-mon[92425]: pgmap v8069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:04 smithi161 ceph-mon[91579]: pgmap v8069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:07 smithi161 ceph-mon[91579]: pgmap v8070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:07 smithi188 ceph-mon[92425]: pgmap v8070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:09 smithi161 ceph-mon[91579]: pgmap v8071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:09 smithi188 ceph-mon[92425]: pgmap v8071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:11 smithi161 ceph-mon[91579]: pgmap v8072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:11 smithi188 ceph-mon[92425]: pgmap v8072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:13 smithi161 ceph-mon[91579]: pgmap v8073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:13 smithi188 ceph-mon[92425]: pgmap v8073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:15 smithi161 ceph-mon[91579]: pgmap v8074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:15 smithi188 ceph-mon[92425]: pgmap v8074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:17 smithi161 ceph-mon[91579]: pgmap v8075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:17 smithi188 ceph-mon[92425]: pgmap v8075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:19 smithi161 ceph-mon[91579]: pgmap v8076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:19 smithi188 ceph-mon[92425]: pgmap v8076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:21 smithi161 ceph-mon[91579]: pgmap v8077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:21 smithi188 ceph-mon[92425]: pgmap v8077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:23 smithi161 ceph-mon[91579]: pgmap v8078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:23 smithi188 ceph-mon[92425]: pgmap v8078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:25 smithi161 ceph-mon[91579]: pgmap v8079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:25 smithi188 ceph-mon[92425]: pgmap v8079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:27 smithi161 ceph-mon[91579]: pgmap v8080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:27 smithi188 ceph-mon[92425]: pgmap v8080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:29 smithi161 ceph-mon[91579]: pgmap v8081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:29 smithi188 ceph-mon[92425]: pgmap v8081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:31 smithi161 ceph-mon[91579]: pgmap v8082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:31 smithi188 ceph-mon[92425]: pgmap v8082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:33 smithi161 ceph-mon[91579]: pgmap v8083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:33 smithi188 ceph-mon[92425]: pgmap v8083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:35 smithi161 ceph-mon[91579]: pgmap v8084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:35 smithi188 ceph-mon[92425]: pgmap v8084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:37 smithi161 ceph-mon[91579]: pgmap v8085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:37 smithi188 ceph-mon[92425]: pgmap v8085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-mon[91579]: pgmap v8086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:39.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:08:39.388+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:08:39.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:08:39.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:08:39.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:08:39.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:08:39.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:08:39.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:08:39.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:08:39.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:08:39.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:08:39.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:08:39.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:08:39.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:08:39.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:08:39.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:08:39.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:08:39.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:08:39.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:08:39.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:08:39.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:08:39.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:08:39.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:08:39.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:08:39.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:08:39.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:08:39.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:08:39.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:08:39.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:08:39.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:08:39.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:08:39.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:08:39.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:08:39.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:08:39.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:08:39.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:08:39.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:08:39.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:08:39.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:08:39.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:08:39.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:08:39.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:08:39.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:08:39.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:08:39.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:08:39.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:08:39.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=443] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:08:39.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=443] Local address: 172.21.15.161, port 55272 2022-07-03T15:08:39.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=443] Peer address: 172.21.15.101, port 22 2022-07-03T15:08:39.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=443] Connection lost 2022-07-03T15:08:39.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=443] Aborting connection 2022-07-03T15:08:39.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:08:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:08:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:39 smithi188 ceph-mon[92425]: pgmap v8086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:08:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:08:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:08:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:08:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:08:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:08:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:08:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:08:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:08:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:08:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:08:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:08:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:08:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:08:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:08:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:08:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:08:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:08:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:08:41.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:08:41.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:08:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:08:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:08:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:08:41.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:08:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:08:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:08:41.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:08:41.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:08:41.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:08:41.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:08:41.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:08:41.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:08:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:08:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:08:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:08:41.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:08:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:08:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:08:41.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:08:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:08:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:08:41.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:08:41.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:08:41.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:08:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:08:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:08:41.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:08:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:08:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:08:41.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:08:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:08:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=443] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:08:41.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=443] Local address: 172.21.15.161, port 55272 2022-07-03T15:08:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=443] Peer address: 172.21.15.101, port 22 2022-07-03T15:08:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=443] Connection lost 2022-07-03T15:08:41.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=443] Aborting connection 2022-07-03T15:08:41.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:41 smithi161 ceph-mon[91579]: pgmap v8087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:08:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:08:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:08:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:08:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:08:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:08:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:08:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:08:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:08:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:08:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:08:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:08:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:08:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:08:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:08:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:08:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:08:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:08:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:08:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:08:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:08:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:08:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:08:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:08:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:08:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:08:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:08:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:08:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:08:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:08:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:08:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:08:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:08:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:08:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:08:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:08:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:08:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:08:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:08:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:08:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:08:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:08:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:08:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:08:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:08:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:08:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=443] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:08:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=443] Local address: 172.21.15.161, port 55272 2022-07-03T15:08:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=443] Peer address: 172.21.15.101, port 22 2022-07-03T15:08:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=443] Connection lost 2022-07-03T15:08:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=443] Aborting connection 2022-07-03T15:08:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:41 smithi188 ceph-mon[92425]: pgmap v8087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:43 smithi161 ceph-mon[91579]: pgmap v8088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:08:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:08:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:43 smithi188 ceph-mon[92425]: pgmap v8088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:08:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:08:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:44 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T15:08:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:44 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T15:08:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:45 smithi161 ceph-mon[91579]: pgmap v8089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:45 smithi188 ceph-mon[92425]: pgmap v8089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:47 smithi161 ceph-mon[91579]: pgmap v8090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:47 smithi188 ceph-mon[92425]: pgmap v8090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:49 smithi161 ceph-mon[91579]: pgmap v8091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:49 smithi188 ceph-mon[92425]: pgmap v8091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:51 smithi161 ceph-mon[91579]: pgmap v8092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:51 smithi188 ceph-mon[92425]: pgmap v8092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:53 smithi161 ceph-mon[91579]: pgmap v8093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:53 smithi188 ceph-mon[92425]: pgmap v8093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:55 smithi161 ceph-mon[91579]: pgmap v8094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:55 smithi188 ceph-mon[92425]: pgmap v8094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:08:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:08:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:57 smithi161 ceph-mon[91579]: pgmap v8095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:57 smithi188 ceph-mon[92425]: pgmap v8095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:08:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:08:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:08:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:08:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:08:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:08:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:08:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:08:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:08:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:08:59 smithi161 ceph-mon[91579]: pgmap v8096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:08:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:08:59 smithi188 ceph-mon[92425]: pgmap v8096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:01 smithi161 ceph-mon[91579]: pgmap v8097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:01 smithi188 ceph-mon[92425]: pgmap v8097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:02 smithi161 ceph-mon[91579]: pgmap v8098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:02 smithi188 ceph-mon[92425]: pgmap v8098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:04 smithi161 ceph-mon[91579]: pgmap v8099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:04 smithi188 ceph-mon[92425]: pgmap v8099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:07 smithi161 ceph-mon[91579]: pgmap v8100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:07 smithi188 ceph-mon[92425]: pgmap v8100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:09 smithi161 ceph-mon[91579]: pgmap v8101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:09 smithi188 ceph-mon[92425]: pgmap v8101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:11 smithi161 ceph-mon[91579]: pgmap v8102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:11 smithi188 ceph-mon[92425]: pgmap v8102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:13 smithi161 ceph-mon[91579]: pgmap v8103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:13 smithi188 ceph-mon[92425]: pgmap v8103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:15 smithi161 ceph-mon[91579]: pgmap v8104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:15 smithi188 ceph-mon[92425]: pgmap v8104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:17 smithi161 ceph-mon[91579]: pgmap v8105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:17 smithi188 ceph-mon[92425]: pgmap v8105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:19 smithi161 ceph-mon[91579]: pgmap v8106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:19 smithi188 ceph-mon[92425]: pgmap v8106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:21 smithi161 ceph-mon[91579]: pgmap v8107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:21 smithi188 ceph-mon[92425]: pgmap v8107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:23 smithi161 ceph-mon[91579]: pgmap v8108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:23 smithi188 ceph-mon[92425]: pgmap v8108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:25 smithi161 ceph-mon[91579]: pgmap v8109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:25 smithi188 ceph-mon[92425]: pgmap v8109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:27 smithi161 ceph-mon[91579]: pgmap v8110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:27 smithi188 ceph-mon[92425]: pgmap v8110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:29 smithi161 ceph-mon[91579]: pgmap v8111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:29 smithi188 ceph-mon[92425]: pgmap v8111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:31 smithi161 ceph-mon[91579]: pgmap v8112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:31 smithi188 ceph-mon[92425]: pgmap v8112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:33 smithi161 ceph-mon[91579]: pgmap v8113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:33 smithi188 ceph-mon[92425]: pgmap v8113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:35 smithi161 ceph-mon[91579]: pgmap v8114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:35 smithi188 ceph-mon[92425]: pgmap v8114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:37 smithi161 ceph-mon[91579]: pgmap v8115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:37 smithi188 ceph-mon[92425]: pgmap v8115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:39 smithi161 ceph-mon[91579]: pgmap v8116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:39 smithi188 ceph-mon[92425]: pgmap v8116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:41 smithi161 ceph-mon[91579]: pgmap v8117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:41 smithi188 ceph-mon[92425]: pgmap v8117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:09:42.576+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:09:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:09:42.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:09:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:09:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:09:42.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:09:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:09:42.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:09:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:09:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:09:42.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:09:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:09:42.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:09:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:09:42.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:09:42.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:09:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:09:42.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:09:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:09:42.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:09:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:09:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:09:42.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:09:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:09:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:09:42.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:09:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:09:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:09:42.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:09:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:09:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:09:42.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:09:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:09:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:09:42.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:09:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:09:42.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:09:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:09:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:09:42.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:09:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:09:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:09:42.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:09:42.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:09:42.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:09:42.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=445] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:09:42.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=445] Local address: 172.21.15.161, port 55332 2022-07-03T15:09:42.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=445] Peer address: 172.21.15.101, port 22 2022-07-03T15:09:42.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=445] Connection lost 2022-07-03T15:09:42.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=445] Aborting connection 2022-07-03T15:09:42.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:09:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:09:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:43 smithi161 ceph-mon[91579]: pgmap v8118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:09:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:09:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:09:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:43 smithi188 ceph-mon[92425]: pgmap v8118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:09:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:09:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:09:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:09:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:09:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:09:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:09:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:09:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:09:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:09:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:09:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:09:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:09:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:09:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:09:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:09:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:09:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:09:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:09:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:09:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:09:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:09:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:09:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:09:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:09:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:09:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:09:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:09:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:09:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:09:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:09:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:09:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:09:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:09:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:09:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:09:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:09:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:09:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:09:44.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:09:44.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:09:44.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:09:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:09:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:09:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:09:44.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:09:44.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:09:44.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:09:44.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:09:44.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=445] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:09:44.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=445] Local address: 172.21.15.161, port 55332 2022-07-03T15:09:44.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=445] Peer address: 172.21.15.101, port 22 2022-07-03T15:09:44.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=445] Connection lost 2022-07-03T15:09:44.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=445] Aborting connection 2022-07-03T15:09:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:09:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:09:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:09:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:09:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:09:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:09:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:09:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:09:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:09:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:09:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:09:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:09:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:09:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:09:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:09:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:09:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:09:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:09:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:09:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:09:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:09:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:09:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:09:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:09:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:09:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:09:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:09:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:09:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:09:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:09:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:09:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:09:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:09:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:09:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:09:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:09:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:09:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:09:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:09:44.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:09:44.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:09:44.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:09:44.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:09:44.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:09:44.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:09:44.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:09:44.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=445] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:09:44.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=445] Local address: 172.21.15.161, port 55332 2022-07-03T15:09:44.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=445] Peer address: 172.21.15.101, port 22 2022-07-03T15:09:44.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=445] Connection lost 2022-07-03T15:09:44.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=445] Aborting connection 2022-07-03T15:09:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:45 smithi161 ceph-mon[91579]: pgmap v8119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:45 smithi188 ceph-mon[92425]: pgmap v8119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:47 smithi161 ceph-mon[91579]: pgmap v8120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:47 smithi188 ceph-mon[92425]: pgmap v8120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:49 smithi161 ceph-mon[91579]: pgmap v8121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:49 smithi188 ceph-mon[92425]: pgmap v8121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:51 smithi161 ceph-mon[91579]: pgmap v8122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:51 smithi188 ceph-mon[92425]: pgmap v8122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:53 smithi161 ceph-mon[91579]: pgmap v8123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:53 smithi188 ceph-mon[92425]: pgmap v8123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:54 smithi161 ceph-mon[91579]: pgmap v8124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:54 smithi188 ceph-mon[92425]: pgmap v8124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:09:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:09:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:56 smithi161 ceph-mon[91579]: pgmap v8125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:56 smithi188 ceph-mon[92425]: pgmap v8125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:09:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:58 smithi161 ceph-mon[91579]: pgmap v8126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:09:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:09:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:09:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:09:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:58 smithi188 ceph-mon[92425]: pgmap v8126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:09:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:09:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:09:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:09:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:10:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:00 smithi161 ceph-mon[91579]: pgmap v8127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:10:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:00 smithi188 ceph-mon[92425]: pgmap v8127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:10:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:02 smithi161 ceph-mon[91579]: pgmap v8128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:02 smithi188 ceph-mon[92425]: pgmap v8128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:04 smithi161 ceph-mon[91579]: pgmap v8129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:04 smithi188 ceph-mon[92425]: pgmap v8129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:07 smithi161 ceph-mon[91579]: pgmap v8130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:07 smithi188 ceph-mon[92425]: pgmap v8130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:09 smithi161 ceph-mon[91579]: pgmap v8131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:09 smithi188 ceph-mon[92425]: pgmap v8131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:11 smithi161 ceph-mon[91579]: pgmap v8132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:11 smithi188 ceph-mon[92425]: pgmap v8132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:13 smithi161 ceph-mon[91579]: pgmap v8133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:13 smithi188 ceph-mon[92425]: pgmap v8133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:15 smithi161 ceph-mon[91579]: pgmap v8134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:15 smithi188 ceph-mon[92425]: pgmap v8134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:17 smithi161 ceph-mon[91579]: pgmap v8135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:17 smithi188 ceph-mon[92425]: pgmap v8135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:19 smithi161 ceph-mon[91579]: pgmap v8136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:19 smithi188 ceph-mon[92425]: pgmap v8136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:21 smithi161 ceph-mon[91579]: pgmap v8137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:21 smithi188 ceph-mon[92425]: pgmap v8137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:23 smithi161 ceph-mon[91579]: pgmap v8138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:23 smithi188 ceph-mon[92425]: pgmap v8138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:25 smithi161 ceph-mon[91579]: pgmap v8139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:25 smithi188 ceph-mon[92425]: pgmap v8139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:27 smithi161 ceph-mon[91579]: pgmap v8140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:27 smithi188 ceph-mon[92425]: pgmap v8140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:29 smithi161 ceph-mon[91579]: pgmap v8141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:29 smithi188 ceph-mon[92425]: pgmap v8141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:31 smithi161 ceph-mon[91579]: pgmap v8142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:31 smithi188 ceph-mon[92425]: pgmap v8142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:33 smithi161 ceph-mon[91579]: pgmap v8143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:33 smithi188 ceph-mon[92425]: pgmap v8143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:35 smithi161 ceph-mon[91579]: pgmap v8144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:35 smithi188 ceph-mon[92425]: pgmap v8144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:37 smithi161 ceph-mon[91579]: pgmap v8145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:37 smithi188 ceph-mon[92425]: pgmap v8145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:39 smithi161 ceph-mon[91579]: pgmap v8146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:39 smithi188 ceph-mon[92425]: pgmap v8146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:41 smithi161 ceph-mon[91579]: pgmap v8147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:41 smithi188 ceph-mon[92425]: pgmap v8147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:43.345 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-mon[91579]: pgmap v8148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:43.345 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:10:43.346 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:10:43.346 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:10:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:10:43.082+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:10:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:10:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:10:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:10:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:10:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:10:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:10:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:10:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:10:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:10:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:10:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:10:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:10:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:10:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:10:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:10:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:10:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:10:43.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:10:43.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:10:43.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:10:43.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:10:43.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:10:43.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:10:43.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:10:43.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:10:43.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:10:43.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:10:43.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:10:43.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:10:43.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:10:43.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:10:43.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:10:43.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:10:43.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:10:43.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:10:43.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:10:43.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:10:43.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:10:43.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:10:43.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:10:43.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:10:43.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:10:43.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:10:43.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:10:43.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=447] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:10:43.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=447] Local address: 172.21.15.161, port 55360 2022-07-03T15:10:43.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=447] Peer address: 172.21.15.101, port 22 2022-07-03T15:10:43.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=447] Connection lost 2022-07-03T15:10:43.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=447] Aborting connection 2022-07-03T15:10:43.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:10:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:10:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:43 smithi188 ceph-mon[92425]: pgmap v8148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:10:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:10:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:10:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:10:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:10:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:10:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:10:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:10:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:10:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:10:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:10:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:10:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:10:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:10:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:10:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:10:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:10:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:10:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:10:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:10:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:10:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:10:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:10:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:10:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:10:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:10:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:10:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:10:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:10:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:10:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:10:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:10:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:10:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:10:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:10:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:10:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:10:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:10:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:10:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:10:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:10:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:10:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:10:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:10:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:10:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:10:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:10:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:10:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:10:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=447] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:10:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=447] Local address: 172.21.15.161, port 55360 2022-07-03T15:10:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=447] Peer address: 172.21.15.101, port 22 2022-07-03T15:10:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=447] Connection lost 2022-07-03T15:10:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=447] Aborting connection 2022-07-03T15:10:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:10:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:10:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:10:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:10:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:10:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:10:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:10:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:10:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:10:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:10:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:10:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:10:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:10:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:10:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:10:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:10:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:10:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:10:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:10:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:10:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:10:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:10:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:10:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:10:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:10:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:10:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:10:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:10:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:10:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:10:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:10:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:10:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:10:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:10:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:10:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:10:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:10:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:10:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:10:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:10:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:10:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:10:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:10:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:10:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:10:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:10:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=447] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:10:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=447] Local address: 172.21.15.161, port 55360 2022-07-03T15:10:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=447] Peer address: 172.21.15.101, port 22 2022-07-03T15:10:44.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=447] Connection lost 2022-07-03T15:10:44.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=447] Aborting connection 2022-07-03T15:10:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:45 smithi161 ceph-mon[91579]: pgmap v8149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:45 smithi188 ceph-mon[92425]: pgmap v8149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:47 smithi161 ceph-mon[91579]: pgmap v8150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:47 smithi188 ceph-mon[92425]: pgmap v8150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:49 smithi161 ceph-mon[91579]: pgmap v8151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:49 smithi188 ceph-mon[92425]: pgmap v8151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:51 smithi161 ceph-mon[91579]: pgmap v8152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:51 smithi188 ceph-mon[92425]: pgmap v8152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:53 smithi161 ceph-mon[91579]: pgmap v8153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:53 smithi188 ceph-mon[92425]: pgmap v8153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:55 smithi161 ceph-mon[91579]: pgmap v8154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:55 smithi188 ceph-mon[92425]: pgmap v8154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:10:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:10:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:57 smithi161 ceph-mon[91579]: pgmap v8155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:57 smithi188 ceph-mon[92425]: pgmap v8155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:10:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:10:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:10:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:10:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:10:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:10:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:10:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:10:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:10:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:10:59 smithi188 ceph-mon[92425]: pgmap v8156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:10:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:10:59 smithi161 ceph-mon[91579]: pgmap v8156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:00 smithi188 ceph-mon[92425]: pgmap v8157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:00 smithi161 ceph-mon[91579]: pgmap v8157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:03.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:02 smithi161 ceph-mon[91579]: pgmap v8158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:02 smithi188 ceph-mon[92425]: pgmap v8158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:04 smithi161 ceph-mon[91579]: pgmap v8159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:04 smithi188 ceph-mon[92425]: pgmap v8159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:07 smithi161 ceph-mon[91579]: pgmap v8160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:07 smithi188 ceph-mon[92425]: pgmap v8160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:11:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:09 smithi161 ceph-mon[91579]: pgmap v8161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:09 smithi188 ceph-mon[92425]: pgmap v8161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:11 smithi161 ceph-mon[91579]: pgmap v8162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:11 smithi188 ceph-mon[92425]: pgmap v8162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:13 smithi161 ceph-mon[91579]: pgmap v8163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:13 smithi188 ceph-mon[92425]: pgmap v8163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:15 smithi161 ceph-mon[91579]: pgmap v8164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:15 smithi188 ceph-mon[92425]: pgmap v8164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:17 smithi161 ceph-mon[91579]: pgmap v8165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:17 smithi188 ceph-mon[92425]: pgmap v8165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:19 smithi161 ceph-mon[91579]: pgmap v8166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:19 smithi188 ceph-mon[92425]: pgmap v8166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:21 smithi161 ceph-mon[91579]: pgmap v8167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:21 smithi188 ceph-mon[92425]: pgmap v8167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:23 smithi161 ceph-mon[91579]: pgmap v8168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:23 smithi188 ceph-mon[92425]: pgmap v8168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:25 smithi161 ceph-mon[91579]: pgmap v8169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:25 smithi188 ceph-mon[92425]: pgmap v8169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:27 smithi161 ceph-mon[91579]: pgmap v8170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:27 smithi188 ceph-mon[92425]: pgmap v8170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:29 smithi161 ceph-mon[91579]: pgmap v8171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:29 smithi188 ceph-mon[92425]: pgmap v8171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:31 smithi161 ceph-mon[91579]: pgmap v8172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:31 smithi188 ceph-mon[92425]: pgmap v8172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:33 smithi161 ceph-mon[91579]: pgmap v8173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:33 smithi188 ceph-mon[92425]: pgmap v8173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:35 smithi161 ceph-mon[91579]: pgmap v8174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:35 smithi188 ceph-mon[92425]: pgmap v8174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:37 smithi161 ceph-mon[91579]: pgmap v8175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:37 smithi188 ceph-mon[92425]: pgmap v8175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:39 smithi161 ceph-mon[91579]: pgmap v8176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:39 smithi188 ceph-mon[92425]: pgmap v8176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:41 smithi161 ceph-mon[91579]: pgmap v8177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:41 smithi188 ceph-mon[92425]: pgmap v8177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-mon[91579]: pgmap v8178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:43.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:11:43.583+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:11:43.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:11:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:11:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:11:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:11:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:11:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:11:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:11:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:11:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:11:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:11:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:11:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:11:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:11:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:11:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:11:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:11:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:11:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:11:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:11:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:11:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:11:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:11:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:11:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:11:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:11:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:11:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:11:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:11:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:11:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:11:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:11:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:11:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:11:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:11:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:11:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:11:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:11:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:11:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:11:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:11:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:11:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:11:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:11:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:11:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=449] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:11:43.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=449] Local address: 172.21.15.161, port 55388 2022-07-03T15:11:43.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=449] Peer address: 172.21.15.101, port 22 2022-07-03T15:11:43.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=449] Connection lost 2022-07-03T15:11:43.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=449] Aborting connection 2022-07-03T15:11:43.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:11:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:11:43.650 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:43 smithi188 ceph-mon[92425]: pgmap v8178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:11:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:11:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:11:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:11:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:11:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:11:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:11:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:11:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:11:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:11:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:11:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:11:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:11:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:11:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:11:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:11:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:11:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:11:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:11:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:11:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:11:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:11:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:11:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:11:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:11:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:11:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:11:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:11:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:11:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:11:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:11:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:11:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:11:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:11:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:11:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:11:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:11:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:11:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:11:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:11:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:11:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:11:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:11:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:11:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:11:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:11:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:11:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:11:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:11:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:11:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:11:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:11:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=449] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:11:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=449] Local address: 172.21.15.161, port 55388 2022-07-03T15:11:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=449] Peer address: 172.21.15.101, port 22 2022-07-03T15:11:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=449] Connection lost 2022-07-03T15:11:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=449] Aborting connection 2022-07-03T15:11:45.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:45 smithi161 ceph-mon[91579]: pgmap v8179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:45.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:45.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:11:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:11:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:11:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:11:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:11:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:11:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:11:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:11:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:11:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:11:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:11:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:11:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:11:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:11:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:11:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:11:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:11:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:11:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:11:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:11:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:11:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:11:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:11:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:11:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:11:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:11:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:11:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:11:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:11:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:11:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:11:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:11:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:11:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:11:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:11:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:11:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:11:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:11:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:11:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:11:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:11:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:11:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:11:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:11:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:11:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:11:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=449] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:11:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=449] Local address: 172.21.15.161, port 55388 2022-07-03T15:11:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=449] Peer address: 172.21.15.101, port 22 2022-07-03T15:11:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=449] Connection lost 2022-07-03T15:11:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=449] Aborting connection 2022-07-03T15:11:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:45 smithi188 ceph-mon[92425]: pgmap v8179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:47 smithi188 ceph-mon[92425]: pgmap v8180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:47 smithi161 ceph-mon[91579]: pgmap v8180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:49 smithi188 ceph-mon[92425]: pgmap v8181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:49 smithi161 ceph-mon[91579]: pgmap v8181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:51 smithi188 ceph-mon[92425]: pgmap v8182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:51 smithi161 ceph-mon[91579]: pgmap v8182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:52 smithi188 ceph-mon[92425]: pgmap v8183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:52 smithi161 ceph-mon[91579]: pgmap v8183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:54 smithi161 ceph-mon[91579]: pgmap v8184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:54 smithi188 ceph-mon[92425]: pgmap v8184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:56.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:11:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:11:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:57 smithi161 ceph-mon[91579]: pgmap v8185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:57 smithi188 ceph-mon[92425]: pgmap v8185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:11:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:11:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:11:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:11:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:11:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:11:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:11:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:11:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:11:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:11:59 smithi161 ceph-mon[91579]: pgmap v8186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:11:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:11:59 smithi188 ceph-mon[92425]: pgmap v8186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:01 smithi161 ceph-mon[91579]: pgmap v8187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:01 smithi188 ceph-mon[92425]: pgmap v8187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:03 smithi161 ceph-mon[91579]: pgmap v8188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:03 smithi188 ceph-mon[92425]: pgmap v8188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:05 smithi161 ceph-mon[91579]: pgmap v8189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:05 smithi188 ceph-mon[92425]: pgmap v8189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:07 smithi161 ceph-mon[91579]: pgmap v8190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:07 smithi188 ceph-mon[92425]: pgmap v8190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:09 smithi161 ceph-mon[91579]: pgmap v8191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:09 smithi188 ceph-mon[92425]: pgmap v8191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:11 smithi161 ceph-mon[91579]: pgmap v8192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:11 smithi188 ceph-mon[92425]: pgmap v8192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:13 smithi161 ceph-mon[91579]: pgmap v8193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:13 smithi188 ceph-mon[92425]: pgmap v8193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:15 smithi161 ceph-mon[91579]: pgmap v8194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:15 smithi188 ceph-mon[92425]: pgmap v8194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:17 smithi161 ceph-mon[91579]: pgmap v8195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:17 smithi188 ceph-mon[92425]: pgmap v8195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:19 smithi161 ceph-mon[91579]: pgmap v8196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:19 smithi188 ceph-mon[92425]: pgmap v8196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:21 smithi161 ceph-mon[91579]: pgmap v8197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:21 smithi188 ceph-mon[92425]: pgmap v8197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:23 smithi161 ceph-mon[91579]: pgmap v8198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:23 smithi188 ceph-mon[92425]: pgmap v8198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:25 smithi161 ceph-mon[91579]: pgmap v8199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:25 smithi188 ceph-mon[92425]: pgmap v8199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:27 smithi161 ceph-mon[91579]: pgmap v8200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:27 smithi188 ceph-mon[92425]: pgmap v8200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:29 smithi161 ceph-mon[91579]: pgmap v8201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:29 smithi188 ceph-mon[92425]: pgmap v8201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:31 smithi161 ceph-mon[91579]: pgmap v8202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:31 smithi188 ceph-mon[92425]: pgmap v8202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:33 smithi188 ceph-mon[92425]: pgmap v8203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:33 smithi161 ceph-mon[91579]: pgmap v8203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:35 smithi188 ceph-mon[92425]: pgmap v8204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:35 smithi161 ceph-mon[91579]: pgmap v8204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:37 smithi188 ceph-mon[92425]: pgmap v8205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:37 smithi161 ceph-mon[91579]: pgmap v8205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:39 smithi188 ceph-mon[92425]: pgmap v8206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:39 smithi161 ceph-mon[91579]: pgmap v8206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:41 smithi188 ceph-mon[92425]: pgmap v8207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:41 smithi161 ceph-mon[91579]: pgmap v8207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:42 smithi188 ceph-mon[92425]: pgmap v8208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:42 smithi161 ceph-mon[91579]: pgmap v8208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:44.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:12:44.088+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:12:44.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:12:44.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:12:44.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:12:44.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:12:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:12:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:12:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:12:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:12:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:12:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:12:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:12:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:12:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:12:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:12:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:12:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:12:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:12:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:12:44.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:12:44.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:12:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:12:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:12:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:12:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:12:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:12:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:12:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:12:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:12:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:12:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:12:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:12:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:12:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:12:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:12:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:12:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:12:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:12:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:12:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:12:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:12:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:12:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:12:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:12:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:12:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=451] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:12:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=451] Local address: 172.21.15.161, port 55416 2022-07-03T15:12:44.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=451] Peer address: 172.21.15.101, port 22 2022-07-03T15:12:44.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=451] Connection lost 2022-07-03T15:12:44.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=451] Aborting connection 2022-07-03T15:12:44.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:12:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: pgmap v8209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:12:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:12:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:12:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:12:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:12:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:12:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:12:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:12:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:12:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:12:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:12:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:12:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:12:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:12:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:12:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:12:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:12:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:12:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:12:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:12:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:12:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:12:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:12:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:12:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:12:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:12:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:12:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:12:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:12:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:12:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:12:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:12:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:12:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:12:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:12:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:12:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:12:44.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:12:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:12:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:12:44.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:12:44.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:12:44.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:12:44.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:12:44.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:12:44.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:12:44.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:12:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:12:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:12:44.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=451] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:12:44.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=451] Local address: 172.21.15.161, port 55416 2022-07-03T15:12:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=451] Peer address: 172.21.15.101, port 22 2022-07-03T15:12:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=451] Connection lost 2022-07-03T15:12:44.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=451] Aborting connection 2022-07-03T15:12:44.910 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.912 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.912 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:44 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: pgmap v8209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:12:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:12:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:12:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:12:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:12:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:12:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:12:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:12:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:12:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:12:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:12:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:12:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:12:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:12:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:12:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:12:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:12:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:12:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:12:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:12:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:12:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:12:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:12:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:12:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:12:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:12:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:12:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:12:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:12:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:12:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:12:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:12:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:12:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:12:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:12:44.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:12:44.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:12:44.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:12:44.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:12:44.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:12:44.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:12:44.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:12:44.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:12:44.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:12:44.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:12:44.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:12:44.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:12:44.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:12:44.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:12:44.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=451] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:12:44.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=451] Local address: 172.21.15.161, port 55416 2022-07-03T15:12:44.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=451] Peer address: 172.21.15.101, port 22 2022-07-03T15:12:44.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=451] Connection lost 2022-07-03T15:12:44.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=451] Aborting connection 2022-07-03T15:12:44.945 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.946 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:44.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:44 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:46.027 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:46.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:47.588 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:47 smithi188 ceph-mon[92425]: pgmap v8210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:47 smithi161 ceph-mon[91579]: pgmap v8210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:48 smithi161 ceph-mon[91579]: pgmap v8211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:48 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T15:12:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:48 smithi188 ceph-mon[92425]: pgmap v8211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:48 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T15:12:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:49.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:12:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:50 smithi161 ceph-mon[91579]: pgmap v8212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:50 smithi188 ceph-mon[92425]: pgmap v8212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:52 smithi161 ceph-mon[91579]: pgmap v8213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:52 smithi188 ceph-mon[92425]: pgmap v8213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:54 smithi161 ceph-mon[91579]: pgmap v8214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:54 smithi188 ceph-mon[92425]: pgmap v8214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:12:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:12:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:57 smithi161 ceph-mon[91579]: pgmap v8215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:57 smithi188 ceph-mon[92425]: pgmap v8215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:12:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:12:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:12:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:12:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:12:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:12:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:12:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:12:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:12:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:12:59 smithi161 ceph-mon[91579]: pgmap v8216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:12:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:12:59 smithi188 ceph-mon[92425]: pgmap v8216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:01 smithi161 ceph-mon[91579]: pgmap v8217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:01 smithi188 ceph-mon[92425]: pgmap v8217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:03 smithi161 ceph-mon[91579]: pgmap v8218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:03 smithi188 ceph-mon[92425]: pgmap v8218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:05 smithi161 ceph-mon[91579]: pgmap v8219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:05 smithi188 ceph-mon[92425]: pgmap v8219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:07 smithi161 ceph-mon[91579]: pgmap v8220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:07 smithi188 ceph-mon[92425]: pgmap v8220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:09 smithi161 ceph-mon[91579]: pgmap v8221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:09 smithi188 ceph-mon[92425]: pgmap v8221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:11 smithi161 ceph-mon[91579]: pgmap v8222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:11 smithi188 ceph-mon[92425]: pgmap v8222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:13 smithi161 ceph-mon[91579]: pgmap v8223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:13 smithi188 ceph-mon[92425]: pgmap v8223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:15 smithi161 ceph-mon[91579]: pgmap v8224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:15 smithi188 ceph-mon[92425]: pgmap v8224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:17 smithi161 ceph-mon[91579]: pgmap v8225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:17 smithi188 ceph-mon[92425]: pgmap v8225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:19 smithi161 ceph-mon[91579]: pgmap v8226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:19 smithi188 ceph-mon[92425]: pgmap v8226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:21 smithi161 ceph-mon[91579]: pgmap v8227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:21.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:21 smithi188 ceph-mon[92425]: pgmap v8227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:23.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:23 smithi161 ceph-mon[91579]: pgmap v8228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:23 smithi188 ceph-mon[92425]: pgmap v8228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:25 smithi161 ceph-mon[91579]: pgmap v8229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:25 smithi188 ceph-mon[92425]: pgmap v8229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:27 smithi161 ceph-mon[91579]: pgmap v8230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:27 smithi188 ceph-mon[92425]: pgmap v8230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:29 smithi188 ceph-mon[92425]: pgmap v8231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:29 smithi161 ceph-mon[91579]: pgmap v8231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:31 smithi188 ceph-mon[92425]: pgmap v8232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:31 smithi161 ceph-mon[91579]: pgmap v8232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:33 smithi188 ceph-mon[92425]: pgmap v8233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:33 smithi161 ceph-mon[91579]: pgmap v8233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:35 smithi188 ceph-mon[92425]: pgmap v8234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:35 smithi161 ceph-mon[91579]: pgmap v8234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:37 smithi188 ceph-mon[92425]: pgmap v8235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:37 smithi161 ceph-mon[91579]: pgmap v8235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:39 smithi188 ceph-mon[92425]: pgmap v8236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:39 smithi161 ceph-mon[91579]: pgmap v8236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:41 smithi188 ceph-mon[92425]: pgmap v8237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:41 smithi161 ceph-mon[91579]: pgmap v8237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:43 smithi188 ceph-mon[92425]: pgmap v8238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:43 smithi161 ceph-mon[91579]: pgmap v8238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:44 smithi188 ceph-mon[92425]: pgmap v8239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:44 smithi161 ceph-mon[91579]: pgmap v8239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:46 smithi161 ceph-mon[91579]: pgmap v8240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:46 smithi188 ceph-mon[92425]: pgmap v8240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:48.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:13:47.898+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:13:48.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:13:48.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:13:48.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:13:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:13:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:13:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:13:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:13:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:13:48.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:13:48.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:13:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:13:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:13:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:13:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:13:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:13:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:13:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:13:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:13:48.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:13:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:13:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:13:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:13:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:13:48.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:13:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:13:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:13:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:13:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:13:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:13:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:13:48.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:13:48.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:13:48.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:13:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:13:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:13:48.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:13:48.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:13:48.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:13:48.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:13:48.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:13:48.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:13:48.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:13:48.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:13:48.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:13:48.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=453] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:13:48.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=453] Local address: 172.21.15.161, port 55446 2022-07-03T15:13:48.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=453] Peer address: 172.21.15.101, port 22 2022-07-03T15:13:48.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=453] Connection lost 2022-07-03T15:13:48.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=453] Aborting connection 2022-07-03T15:13:48.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:13:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:13:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: pgmap v8241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:13:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:13:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:13:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:13:49.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:13:49.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:13:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:13:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:13:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:13:49.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:13:49.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:13:49.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:13:49.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:13:49.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:13:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:13:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:13:49.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:13:49.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:13:49.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:13:49.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:13:49.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:13:49.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:13:49.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:13:49.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:13:49.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:13:49.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:13:49.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:13:49.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:13:49.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:13:49.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:13:49.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:13:49.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:13:49.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:13:49.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:13:49.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:13:49.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:13:49.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:13:49.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:13:49.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:13:49.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:13:49.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:13:49.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:13:49.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:13:49.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:13:49.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:13:49.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:13:49.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:13:49.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:13:49.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:13:49.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=453] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:13:49.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=453] Local address: 172.21.15.161, port 55446 2022-07-03T15:13:49.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=453] Peer address: 172.21.15.101, port 22 2022-07-03T15:13:49.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=453] Connection lost 2022-07-03T15:13:49.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=453] Aborting connection 2022-07-03T15:13:49.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: pgmap v8241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:49.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:13:49.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:13:49.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:13:49.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:13:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:13:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:13:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:13:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:13:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:13:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:13:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:13:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:13:49.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:13:49.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:13:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:13:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:13:49.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:13:49.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:13:49.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:13:49.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:13:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:13:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:13:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:13:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:13:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:13:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:13:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:13:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:13:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:13:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:13:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:13:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:13:49.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:13:49.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:13:49.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:13:49.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:13:49.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:13:49.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:13:49.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:13:49.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:13:49.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:13:49.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:13:49.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:13:49.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:13:49.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:13:49.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:13:49.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:13:49.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:13:49.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:13:49.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=453] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:13:49.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=453] Local address: 172.21.15.161, port 55446 2022-07-03T15:13:49.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=453] Peer address: 172.21.15.101, port 22 2022-07-03T15:13:49.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=453] Connection lost 2022-07-03T15:13:49.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=453] Aborting connection 2022-07-03T15:13:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:13:51.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:50 smithi161 ceph-mon[91579]: pgmap v8242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:51.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:50 smithi188 ceph-mon[92425]: pgmap v8242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:52 smithi161 ceph-mon[91579]: pgmap v8243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:52 smithi188 ceph-mon[92425]: pgmap v8243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:54 smithi161 ceph-mon[91579]: pgmap v8244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:54 smithi188 ceph-mon[92425]: pgmap v8244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:13:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:13:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:57 smithi161 ceph-mon[91579]: pgmap v8245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:57 smithi188 ceph-mon[92425]: pgmap v8245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:13:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:13:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:13:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:13:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:13:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:13:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:13:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:13:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:13:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:13:59 smithi161 ceph-mon[91579]: pgmap v8246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:13:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:13:59 smithi188 ceph-mon[92425]: pgmap v8246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:01 smithi161 ceph-mon[91579]: pgmap v8247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:01 smithi188 ceph-mon[92425]: pgmap v8247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:03 smithi161 ceph-mon[91579]: pgmap v8248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:03 smithi188 ceph-mon[92425]: pgmap v8248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:05 smithi161 ceph-mon[91579]: pgmap v8249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:05 smithi188 ceph-mon[92425]: pgmap v8249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:07 smithi161 ceph-mon[91579]: pgmap v8250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:07 smithi188 ceph-mon[92425]: pgmap v8250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:09 smithi161 ceph-mon[91579]: pgmap v8251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:09 smithi188 ceph-mon[92425]: pgmap v8251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:11 smithi161 ceph-mon[91579]: pgmap v8252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:11 smithi188 ceph-mon[92425]: pgmap v8252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:13 smithi161 ceph-mon[91579]: pgmap v8253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:13 smithi188 ceph-mon[92425]: pgmap v8253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:15 smithi161 ceph-mon[91579]: pgmap v8254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:15 smithi188 ceph-mon[92425]: pgmap v8254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:16.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:17 smithi161 ceph-mon[91579]: pgmap v8255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:17 smithi188 ceph-mon[92425]: pgmap v8255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:19 smithi161 ceph-mon[91579]: pgmap v8256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:19 smithi188 ceph-mon[92425]: pgmap v8256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:21 smithi161 ceph-mon[91579]: pgmap v8257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:21 smithi188 ceph-mon[92425]: pgmap v8257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:23 smithi161 ceph-mon[91579]: pgmap v8258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:23 smithi188 ceph-mon[92425]: pgmap v8258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:25 smithi188 ceph-mon[92425]: pgmap v8259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:25 smithi161 ceph-mon[91579]: pgmap v8259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:27 smithi188 ceph-mon[92425]: pgmap v8260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:27 smithi161 ceph-mon[91579]: pgmap v8260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:29 smithi188 ceph-mon[92425]: pgmap v8261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:29 smithi161 ceph-mon[91579]: pgmap v8261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:31 smithi188 ceph-mon[92425]: pgmap v8262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:31 smithi161 ceph-mon[91579]: pgmap v8262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:33 smithi188 ceph-mon[92425]: pgmap v8263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:33 smithi161 ceph-mon[91579]: pgmap v8263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:35 smithi188 ceph-mon[92425]: pgmap v8264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:35 smithi161 ceph-mon[91579]: pgmap v8264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:37 smithi188 ceph-mon[92425]: pgmap v8265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:37 smithi161 ceph-mon[91579]: pgmap v8265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:39 smithi188 ceph-mon[92425]: pgmap v8266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:39 smithi161 ceph-mon[91579]: pgmap v8266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:40.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:41 smithi161 ceph-mon[91579]: pgmap v8267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:41 smithi188 ceph-mon[92425]: pgmap v8267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:42 smithi161 ceph-mon[91579]: pgmap v8268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:42 smithi188 ceph-mon[92425]: pgmap v8268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:44 smithi161 ceph-mon[91579]: pgmap v8269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:44 smithi188 ceph-mon[92425]: pgmap v8269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:47 smithi161 ceph-mon[91579]: pgmap v8270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:47 smithi188 ceph-mon[92425]: pgmap v8270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:49.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:14:48.824+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:14:49.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:14:49.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:14:49.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:14:49.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:14:49.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:14:49.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:14:49.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:14:49.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:14:49.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:14:49.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:14:49.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:14:49.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:14:49.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:14:49.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:14:49.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:14:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:14:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:14:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:14:49.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:14:49.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:14:49.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:14:49.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:14:49.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:14:49.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:14:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:14:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:14:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:14:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:14:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:14:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:14:49.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:14:49.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:14:49.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:14:49.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:14:49.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:14:49.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:14:49.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:14:49.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:14:49.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:14:49.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:14:49.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:14:49.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:14:49.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:14:49.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:14:49.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=455] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:14:49.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=455] Local address: 172.21.15.161, port 55474 2022-07-03T15:14:49.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=455] Peer address: 172.21.15.101, port 22 2022-07-03T15:14:49.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=455] Connection lost 2022-07-03T15:14:49.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=455] Aborting connection 2022-07-03T15:14:49.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:14:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:14:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:49 smithi161 ceph-mon[91579]: pgmap v8271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:14:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:14:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:14:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:49 smithi188 ceph-mon[92425]: pgmap v8271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:14:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:14:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:14:50.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:14:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:14:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:14:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:14:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:14:50.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:14:50.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:14:50.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:14:50.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:14:50.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:14:50.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:14:50.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:14:50.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:14:50.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:14:50.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:14:50.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:14:50.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:14:50.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:14:50.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:14:50.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:14:50.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:14:50.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:14:50.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:14:50.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:14:50.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:14:50.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:14:50.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:14:50.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:14:50.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:14:50.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:14:50.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:14:50.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:14:50.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:14:50.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:14:50.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:14:50.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:14:50.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:14:50.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:14:50.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:14:50.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:14:50.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:14:50.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:14:50.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:14:50.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:14:50.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:14:50.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:14:50.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=455] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:14:50.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=455] Local address: 172.21.15.161, port 55474 2022-07-03T15:14:50.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=455] Peer address: 172.21.15.101, port 22 2022-07-03T15:14:50.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=455] Connection lost 2022-07-03T15:14:50.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=455] Aborting connection 2022-07-03T15:14:50.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:14:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:14:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:14:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:14:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:14:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:14:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:14:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:14:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:14:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:14:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:14:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:14:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:14:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:14:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:14:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:14:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:14:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:14:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:14:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:14:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:14:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:14:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:14:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:14:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:14:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:14:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:14:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:14:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:14:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:14:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:14:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:14:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:14:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:14:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:14:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:14:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:14:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:14:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:14:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:14:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:14:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:14:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:14:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:14:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:14:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:14:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=455] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:14:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=455] Local address: 172.21.15.161, port 55474 2022-07-03T15:14:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=455] Peer address: 172.21.15.101, port 22 2022-07-03T15:14:50.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=455] Connection lost 2022-07-03T15:14:50.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=455] Aborting connection 2022-07-03T15:14:50.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:51 smithi161 ceph-mon[91579]: pgmap v8272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:51 smithi188 ceph-mon[92425]: pgmap v8272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:53 smithi161 ceph-mon[91579]: pgmap v8273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:53 smithi188 ceph-mon[92425]: pgmap v8273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:55 smithi161 ceph-mon[91579]: pgmap v8274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:55 smithi188 ceph-mon[92425]: pgmap v8274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:14:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:14:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:57 smithi161 ceph-mon[91579]: pgmap v8275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:57 smithi188 ceph-mon[92425]: pgmap v8275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:14:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:14:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:14:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:14:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:14:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:14:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:14:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:14:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:14:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:14:59 smithi161 ceph-mon[91579]: pgmap v8276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:14:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:14:59 smithi188 ceph-mon[92425]: pgmap v8276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:01 smithi161 ceph-mon[91579]: pgmap v8277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:01 smithi188 ceph-mon[92425]: pgmap v8277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:03 smithi161 ceph-mon[91579]: pgmap v8278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:03 smithi188 ceph-mon[92425]: pgmap v8278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:05 smithi161 ceph-mon[91579]: pgmap v8279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:05 smithi188 ceph-mon[92425]: pgmap v8279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:07 smithi188 ceph-mon[92425]: pgmap v8280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:07 smithi161 ceph-mon[91579]: pgmap v8280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:09 smithi188 ceph-mon[92425]: pgmap v8281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:09 smithi161 ceph-mon[91579]: pgmap v8281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:11 smithi188 ceph-mon[92425]: pgmap v8282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:11 smithi161 ceph-mon[91579]: pgmap v8282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:13 smithi188 ceph-mon[92425]: pgmap v8283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:13 smithi161 ceph-mon[91579]: pgmap v8283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:15 smithi188 ceph-mon[92425]: pgmap v8284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:15 smithi161 ceph-mon[91579]: pgmap v8284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:17 smithi188 ceph-mon[92425]: pgmap v8285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:17 smithi161 ceph-mon[91579]: pgmap v8285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:19 smithi188 ceph-mon[92425]: pgmap v8286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:19 smithi161 ceph-mon[91579]: pgmap v8286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:21 smithi188 ceph-mon[92425]: pgmap v8287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:21 smithi161 ceph-mon[91579]: pgmap v8287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:23 smithi161 ceph-mon[91579]: pgmap v8288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:23 smithi188 ceph-mon[92425]: pgmap v8288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:25 smithi161 ceph-mon[91579]: pgmap v8289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:25 smithi188 ceph-mon[92425]: pgmap v8289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:27 smithi161 ceph-mon[91579]: pgmap v8290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:27 smithi188 ceph-mon[92425]: pgmap v8290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:28 smithi161 ceph-mon[91579]: pgmap v8291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:28 smithi188 ceph-mon[92425]: pgmap v8291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:30 smithi161 ceph-mon[91579]: pgmap v8292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:31.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:30 smithi188 ceph-mon[92425]: pgmap v8292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:32 smithi161 ceph-mon[91579]: pgmap v8293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:32 smithi188 ceph-mon[92425]: pgmap v8293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:34 smithi161 ceph-mon[91579]: pgmap v8294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:34 smithi188 ceph-mon[92425]: pgmap v8294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:37 smithi161 ceph-mon[91579]: pgmap v8295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:37 smithi188 ceph-mon[92425]: pgmap v8295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:39 smithi161 ceph-mon[91579]: pgmap v8296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:39 smithi188 ceph-mon[92425]: pgmap v8296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:41 smithi161 ceph-mon[91579]: pgmap v8297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:41 smithi188 ceph-mon[92425]: pgmap v8297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:43 smithi161 ceph-mon[91579]: pgmap v8298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:43 smithi188 ceph-mon[92425]: pgmap v8298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:45 smithi161 ceph-mon[91579]: pgmap v8299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:45 smithi188 ceph-mon[92425]: pgmap v8299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:46.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:47 smithi161 ceph-mon[91579]: pgmap v8300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:47 smithi188 ceph-mon[92425]: pgmap v8300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-mon[91579]: pgmap v8301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:15:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:15:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:15:49.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:15:49.329+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:15:49.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:15:49.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:15:49.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:15:49.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:15:49.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:15:49.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:15:49.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:15:49.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:15:49.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:15:49.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:15:49.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:15:49.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:15:49.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:15:49.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:15:49.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:15:49.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:15:49.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:15:49.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:15:49.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:15:49.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:15:49.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:15:49.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:15:49.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:15:49.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:15:49.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:15:49.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:15:49.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:15:49.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:15:49.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:15:49.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:15:49.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:15:49.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:15:49.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:15:49.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:15:49.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:15:49.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:15:49.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:15:49.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:15:49.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:15:49.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:15:49.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:15:49.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:15:49.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:15:49.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:15:49.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=457] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:15:49.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=457] Local address: 172.21.15.161, port 55502 2022-07-03T15:15:49.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=457] Peer address: 172.21.15.101, port 22 2022-07-03T15:15:49.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=457] Connection lost 2022-07-03T15:15:49.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=457] Aborting connection 2022-07-03T15:15:49.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:15:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:15:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:49 smithi188 ceph-mon[92425]: pgmap v8301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:15:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:15:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:15:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:15:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:15:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:15:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:15:50.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:15:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:15:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:15:50.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:15:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:15:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:15:50.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:15:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:15:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:15:50.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:15:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:15:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:15:50.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:15:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:15:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:15:50.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:15:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:15:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:15:50.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:15:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:15:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:15:50.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:15:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:15:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:15:50.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:15:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:15:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:15:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:15:50.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:15:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:15:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:15:50.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:15:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:15:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:15:50.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:15:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:15:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:15:50.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:15:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:15:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:15:50.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:15:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:15:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=457] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:15:50.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=457] Local address: 172.21.15.161, port 55502 2022-07-03T15:15:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=457] Peer address: 172.21.15.101, port 22 2022-07-03T15:15:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=457] Connection lost 2022-07-03T15:15:50.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=457] Aborting connection 2022-07-03T15:15:50.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:15:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:15:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:15:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:15:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:15:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:15:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:15:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:15:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:15:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:15:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:15:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:15:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:15:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:15:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:15:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:15:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:15:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:15:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:15:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:15:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:15:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:15:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:15:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:15:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:15:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:15:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:15:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:15:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:15:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:15:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:15:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:15:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:15:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:15:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:15:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:15:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:15:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:15:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:15:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:15:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:15:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:15:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:15:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:15:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:15:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:15:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=457] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:15:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=457] Local address: 172.21.15.161, port 55502 2022-07-03T15:15:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=457] Peer address: 172.21.15.101, port 22 2022-07-03T15:15:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=457] Connection lost 2022-07-03T15:15:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=457] Aborting connection 2022-07-03T15:15:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:51 smithi188 ceph-mon[92425]: pgmap v8302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:51 smithi161 ceph-mon[91579]: pgmap v8302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: pgmap v8303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: pgmap v8303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:15:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:15:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 180.3M 2022-07-03T15:15:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 189128777: error parsing value: Value '189128777' is below minimum 939524096 2022-07-03T15:15:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:15:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:15:54.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 180.3M 2022-07-03T15:15:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 189128777: error parsing value: Value '189128777' is below minimum 939524096 2022-07-03T15:15:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:15:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:15:55.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:55 smithi161 ceph-mon[91579]: pgmap v8304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:55 smithi188 ceph-mon[92425]: pgmap v8304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:56.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:56.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:15:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:15:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:57 smithi161 ceph-mon[91579]: pgmap v8305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:57 smithi188 ceph-mon[92425]: pgmap v8305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:15:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:15:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:15:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:15:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:15:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:15:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:15:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:15:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:15:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:15:59 smithi161 ceph-mon[91579]: pgmap v8306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:15:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:15:59 smithi188 ceph-mon[92425]: pgmap v8306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:01 smithi161 ceph-mon[91579]: pgmap v8307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:01 smithi188 ceph-mon[92425]: pgmap v8307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:03 smithi161 ceph-mon[91579]: pgmap v8308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:03 smithi188 ceph-mon[92425]: pgmap v8308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:05 smithi161 ceph-mon[91579]: pgmap v8309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:05 smithi188 ceph-mon[92425]: pgmap v8309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:07 smithi161 ceph-mon[91579]: pgmap v8310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:07 smithi188 ceph-mon[92425]: pgmap v8310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:09 smithi161 ceph-mon[91579]: pgmap v8311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:09 smithi188 ceph-mon[92425]: pgmap v8311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:11 smithi188 ceph-mon[92425]: pgmap v8312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:11 smithi161 ceph-mon[91579]: pgmap v8312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:13 smithi188 ceph-mon[92425]: pgmap v8313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:13 smithi161 ceph-mon[91579]: pgmap v8313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:15 smithi188 ceph-mon[92425]: pgmap v8314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:15 smithi161 ceph-mon[91579]: pgmap v8314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:17 smithi188 ceph-mon[92425]: pgmap v8315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:17 smithi161 ceph-mon[91579]: pgmap v8315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:19 smithi188 ceph-mon[92425]: pgmap v8316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:19 smithi161 ceph-mon[91579]: pgmap v8316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:21 smithi188 ceph-mon[92425]: pgmap v8317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:21 smithi161 ceph-mon[91579]: pgmap v8317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:23 smithi161 ceph-mon[91579]: pgmap v8318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:23 smithi188 ceph-mon[92425]: pgmap v8318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:25 smithi161 ceph-mon[91579]: pgmap v8319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:25 smithi188 ceph-mon[92425]: pgmap v8319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:27 smithi161 ceph-mon[91579]: pgmap v8320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:27 smithi188 ceph-mon[92425]: pgmap v8320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:29 smithi161 ceph-mon[91579]: pgmap v8321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:29 smithi188 ceph-mon[92425]: pgmap v8321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:31 smithi161 ceph-mon[91579]: pgmap v8322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:31 smithi188 ceph-mon[92425]: pgmap v8322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:32 smithi161 ceph-mon[91579]: pgmap v8323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:32 smithi188 ceph-mon[92425]: pgmap v8323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:34 smithi161 ceph-mon[91579]: pgmap v8324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:34 smithi188 ceph-mon[92425]: pgmap v8324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:37 smithi161 ceph-mon[91579]: pgmap v8325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:37 smithi188 ceph-mon[92425]: pgmap v8325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:39 smithi161 ceph-mon[91579]: pgmap v8326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:39 smithi188 ceph-mon[92425]: pgmap v8326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:41 smithi161 ceph-mon[91579]: pgmap v8327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:41 smithi188 ceph-mon[92425]: pgmap v8327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:43 smithi161 ceph-mon[91579]: pgmap v8328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:43 smithi188 ceph-mon[92425]: pgmap v8328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:45 smithi188 ceph-mon[92425]: pgmap v8329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:45 smithi161 ceph-mon[91579]: pgmap v8329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:46.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:47 smithi188 ceph-mon[92425]: pgmap v8330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:47 smithi161 ceph-mon[91579]: pgmap v8330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:49 smithi188 ceph-mon[92425]: pgmap v8331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:49 smithi161 ceph-mon[91579]: pgmap v8331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:51 smithi188 ceph-mon[92425]: pgmap v8332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:51 smithi161 ceph-mon[91579]: pgmap v8332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:53 smithi188 ceph-mon[92425]: pgmap v8333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:16:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:16:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:16:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:53 smithi161 ceph-mon[91579]: pgmap v8333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:16:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:16:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:16:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:16:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:16:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:55 smithi188 ceph-mon[92425]: pgmap v8334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:55 smithi161 ceph-mon[91579]: pgmap v8334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:16:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:16:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:57 smithi188 ceph-mon[92425]: pgmap v8335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:57 smithi161 ceph-mon[91579]: pgmap v8335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:16:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:16:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:16:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:16:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:16:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:16:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:16:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:16:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:16:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:16:59 smithi161 ceph-mon[91579]: pgmap v8336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:16:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:16:59 smithi188 ceph-mon[92425]: pgmap v8336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:01 smithi161 ceph-mon[91579]: pgmap v8337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:01 smithi188 ceph-mon[92425]: pgmap v8337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:03 smithi161 ceph-mon[91579]: pgmap v8338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:03 smithi188 ceph-mon[92425]: pgmap v8338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:05 smithi161 ceph-mon[91579]: pgmap v8339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:05 smithi188 ceph-mon[92425]: pgmap v8339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:07 smithi161 ceph-mon[91579]: pgmap v8340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:07 smithi188 ceph-mon[92425]: pgmap v8340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:08 smithi161 ceph-mon[91579]: pgmap v8341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:08 smithi188 ceph-mon[92425]: pgmap v8341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:10 smithi161 ceph-mon[91579]: pgmap v8342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:10 smithi188 ceph-mon[92425]: pgmap v8342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:12 smithi161 ceph-mon[91579]: pgmap v8343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:12 smithi188 ceph-mon[92425]: pgmap v8343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:14 smithi161 ceph-mon[91579]: pgmap v8344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:14 smithi188 ceph-mon[92425]: pgmap v8344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:17 smithi161 ceph-mon[91579]: pgmap v8345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:17 smithi188 ceph-mon[92425]: pgmap v8345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:19.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:19 smithi161 ceph-mon[91579]: pgmap v8346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:19 smithi188 ceph-mon[92425]: pgmap v8346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:21 smithi188 ceph-mon[92425]: pgmap v8347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:21 smithi161 ceph-mon[91579]: pgmap v8347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:17:22.587+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:17:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:17:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:17:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:17:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:17:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:17:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:17:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:17:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:17:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:17:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:17:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:17:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T15:17:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:17:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:17:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:17:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:17:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:17:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:17:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:17:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:17:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:17:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:17:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:17:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:17:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:17:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:17:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:17:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:17:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:17:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:17:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:17:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:17:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:17:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:17:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:17:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:17:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:17:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:17:22.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:17:22.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:17:22.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:17:22.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:17:22.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T15:17:22.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:17:22.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=459] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:17:22.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=459] Local address: 172.21.15.161, port 55532 2022-07-03T15:17:22.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=459] Peer address: 172.21.15.101, port 22 2022-07-03T15:17:22.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1252] Received exit status 0 2022-07-03T15:17:22.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1252] Received channel close 2022-07-03T15:17:22.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1252] Channel closed 2022-07-03T15:17:22.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1252] Received exit status 0 2022-07-03T15:17:22.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1252] Received channel close 2022-07-03T15:17:22.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1253] Requesting new SSH session 2022-07-03T15:17:22.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1252] Channel closed 2022-07-03T15:17:22.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1253] Command: which python3 2022-07-03T15:17:22.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1253] Requesting new SSH session 2022-07-03T15:17:22.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1253] Command: which python3 2022-07-03T15:17:22.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1253] Received exit status 0 2022-07-03T15:17:22.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1253] Received channel close 2022-07-03T15:17:22.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1253] Channel closed 2022-07-03T15:17:22.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1254] Requesting new SSH session 2022-07-03T15:17:22.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1254] Command: sudo true 2022-07-03T15:17:22.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1253] Received exit status 0 2022-07-03T15:17:22.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1253] Received channel close 2022-07-03T15:17:22.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1253] Channel closed 2022-07-03T15:17:22.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1254] Requesting new SSH session 2022-07-03T15:17:22.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1254] Command: sudo true 2022-07-03T15:17:22.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1254] Received exit status 0 2022-07-03T15:17:22.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1254] Received channel close 2022-07-03T15:17:22.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1254] Channel closed 2022-07-03T15:17:22.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1255] Requesting new SSH session 2022-07-03T15:17:22.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:22.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1254] Received exit status 0 2022-07-03T15:17:22.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1254] Received channel close 2022-07-03T15:17:22.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1254] Channel closed 2022-07-03T15:17:22.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1255] Requesting new SSH session 2022-07-03T15:17:22.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:22.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1255] Received exit status 0 2022-07-03T15:17:22.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1255] Received channel close 2022-07-03T15:17:22.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1255] Channel closed 2022-07-03T15:17:22.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1255] Received exit status 0 2022-07-03T15:17:22.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1255] Received channel close 2022-07-03T15:17:22.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1255] Channel closed 2022-07-03T15:17:22.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=459] Login timeout expired 2022-07-03T15:17:22.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=459] Aborting connection 2022-07-03T15:17:22.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:17:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:17:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:23 smithi188 ceph-mon[92425]: pgmap v8348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:23 smithi161 ceph-mon[91579]: pgmap v8348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:17:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:17:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:17:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:17:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:17:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:17:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:17:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:17:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:17:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:17:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:17:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:17:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T15:17:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:17:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:17:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:17:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:17:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:17:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:17:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:17:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:17:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:17:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:17:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:17:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:17:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:17:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:17:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:17:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:17:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:17:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:17:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:17:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:17:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:17:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:17:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:17:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:17:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:17:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:17:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:17:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:17:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:17:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:17:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T15:17:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:17:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=459] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:17:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=459] Local address: 172.21.15.161, port 55532 2022-07-03T15:17:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=459] Peer address: 172.21.15.101, port 22 2022-07-03T15:17:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1252] Received exit status 0 2022-07-03T15:17:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1252] Received channel close 2022-07-03T15:17:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1252] Channel closed 2022-07-03T15:17:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1252] Received exit status 0 2022-07-03T15:17:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1252] Received channel close 2022-07-03T15:17:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1253] Requesting new SSH session 2022-07-03T15:17:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1252] Channel closed 2022-07-03T15:17:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1253] Command: which python3 2022-07-03T15:17:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1253] Requesting new SSH session 2022-07-03T15:17:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1253] Command: which python3 2022-07-03T15:17:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1253] Received exit status 0 2022-07-03T15:17:24.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1253] Received channel close 2022-07-03T15:17:24.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1253] Channel closed 2022-07-03T15:17:24.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1254] Requesting new SSH session 2022-07-03T15:17:24.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1254] Command: sudo true 2022-07-03T15:17:24.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1253] Received exit status 0 2022-07-03T15:17:24.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1253] Received channel close 2022-07-03T15:17:24.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1253] Channel closed 2022-07-03T15:17:24.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1254] Requesting new SSH session 2022-07-03T15:17:24.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1254] Command: sudo true 2022-07-03T15:17:24.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1254] Received exit status 0 2022-07-03T15:17:24.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1254] Received channel close 2022-07-03T15:17:24.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1254] Channel closed 2022-07-03T15:17:24.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1255] Requesting new SSH session 2022-07-03T15:17:24.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:24.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1254] Received exit status 0 2022-07-03T15:17:24.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1254] Received channel close 2022-07-03T15:17:24.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1254] Channel closed 2022-07-03T15:17:24.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1255] Requesting new SSH session 2022-07-03T15:17:24.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:24.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1255] Received exit status 0 2022-07-03T15:17:24.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1255] Received channel close 2022-07-03T15:17:24.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1255] Channel closed 2022-07-03T15:17:24.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1255] Received exit status 0 2022-07-03T15:17:24.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1255] Received channel close 2022-07-03T15:17:24.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1255] Channel closed 2022-07-03T15:17:24.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=459] Login timeout expired 2022-07-03T15:17:24.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=459] Aborting connection 2022-07-03T15:17:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:17:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:17:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:17:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:17:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:17:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:17:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:17:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:17:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:17:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:17:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:17:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:17:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T15:17:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:17:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:17:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:17:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:17:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:17:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:17:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:17:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:17:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:17:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:17:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:17:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:17:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:17:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:17:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:17:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:17:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:17:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:17:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:17:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:17:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:17:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:17:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:17:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:17:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:17:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:17:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:17:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:17:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:17:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:17:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T15:17:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:17:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=459] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:17:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=459] Local address: 172.21.15.161, port 55532 2022-07-03T15:17:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=459] Peer address: 172.21.15.101, port 22 2022-07-03T15:17:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1252] Received exit status 0 2022-07-03T15:17:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1252] Received channel close 2022-07-03T15:17:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1252] Channel closed 2022-07-03T15:17:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1252] Received exit status 0 2022-07-03T15:17:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1252] Received channel close 2022-07-03T15:17:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1253] Requesting new SSH session 2022-07-03T15:17:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1252] Channel closed 2022-07-03T15:17:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1253] Command: which python3 2022-07-03T15:17:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1253] Requesting new SSH session 2022-07-03T15:17:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1253] Command: which python3 2022-07-03T15:17:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1253] Received exit status 0 2022-07-03T15:17:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1253] Received channel close 2022-07-03T15:17:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1253] Channel closed 2022-07-03T15:17:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1254] Requesting new SSH session 2022-07-03T15:17:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1254] Command: sudo true 2022-07-03T15:17:24.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1253] Received exit status 0 2022-07-03T15:17:24.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1253] Received channel close 2022-07-03T15:17:24.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1253] Channel closed 2022-07-03T15:17:24.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1254] Requesting new SSH session 2022-07-03T15:17:24.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1254] Command: sudo true 2022-07-03T15:17:24.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1254] Received exit status 0 2022-07-03T15:17:24.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1254] Received channel close 2022-07-03T15:17:24.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1254] Channel closed 2022-07-03T15:17:24.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1255] Requesting new SSH session 2022-07-03T15:17:24.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:24.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1254] Received exit status 0 2022-07-03T15:17:24.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1254] Received channel close 2022-07-03T15:17:24.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1254] Channel closed 2022-07-03T15:17:24.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1255] Requesting new SSH session 2022-07-03T15:17:24.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1255] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T15:17:24.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1255] Received exit status 0 2022-07-03T15:17:24.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1255] Received channel close 2022-07-03T15:17:24.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1255] Channel closed 2022-07-03T15:17:24.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1255] Received exit status 0 2022-07-03T15:17:24.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1255] Received channel close 2022-07-03T15:17:24.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1255] Channel closed 2022-07-03T15:17:24.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=459] Login timeout expired 2022-07-03T15:17:24.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=459] Aborting connection 2022-07-03T15:17:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:25 smithi188 ceph-mon[92425]: pgmap v8349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:25 smithi161 ceph-mon[91579]: pgmap v8349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:27 smithi188 ceph-mon[92425]: pgmap v8350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:27 smithi161 ceph-mon[91579]: pgmap v8350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:29 smithi188 ceph-mon[92425]: pgmap v8351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:29 smithi161 ceph-mon[91579]: pgmap v8351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:31 smithi188 ceph-mon[92425]: pgmap v8352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:31 smithi161 ceph-mon[91579]: pgmap v8352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:33 smithi188 ceph-mon[92425]: pgmap v8353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:33 smithi161 ceph-mon[91579]: pgmap v8353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:35 smithi188 ceph-mon[92425]: pgmap v8354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:35 smithi161 ceph-mon[91579]: pgmap v8354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:37 smithi161 ceph-mon[91579]: pgmap v8355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:37 smithi188 ceph-mon[92425]: pgmap v8355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:39 smithi161 ceph-mon[91579]: pgmap v8356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:39 smithi188 ceph-mon[92425]: pgmap v8356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:41 smithi161 ceph-mon[91579]: pgmap v8357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:41 smithi188 ceph-mon[92425]: pgmap v8357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:43 smithi161 ceph-mon[91579]: pgmap v8358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:43 smithi188 ceph-mon[92425]: pgmap v8358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:45 smithi161 ceph-mon[91579]: pgmap v8359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:45 smithi188 ceph-mon[92425]: pgmap v8359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:47 smithi161 ceph-mon[91579]: pgmap v8360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:47 smithi188 ceph-mon[92425]: pgmap v8360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:49 smithi161 ceph-mon[91579]: pgmap v8361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:49 smithi188 ceph-mon[92425]: pgmap v8361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:51 smithi161 ceph-mon[91579]: pgmap v8362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:51 smithi188 ceph-mon[92425]: pgmap v8362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:52 smithi161 ceph-mon[91579]: pgmap v8363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:52 smithi188 ceph-mon[92425]: pgmap v8363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:54 smithi161 ceph-mon[91579]: pgmap v8364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:54 smithi188 ceph-mon[92425]: pgmap v8364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:17:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:17:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:57 smithi161 ceph-mon[91579]: pgmap v8365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:57 smithi188 ceph-mon[92425]: pgmap v8365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:17:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:17:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:17:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:17:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:17:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:17:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:17:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:17:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:17:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:17:59 smithi188 ceph-mon[92425]: pgmap v8366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:17:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:17:59 smithi161 ceph-mon[91579]: pgmap v8366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:01 smithi188 ceph-mon[92425]: pgmap v8367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:01 smithi161 ceph-mon[91579]: pgmap v8367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:03 smithi188 ceph-mon[92425]: pgmap v8368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:03 smithi161 ceph-mon[91579]: pgmap v8368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:05 smithi188 ceph-mon[92425]: pgmap v8369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:05 smithi161 ceph-mon[91579]: pgmap v8369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:07 smithi188 ceph-mon[92425]: pgmap v8370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:07 smithi161 ceph-mon[91579]: pgmap v8370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:09 smithi188 ceph-mon[92425]: pgmap v8371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:09 smithi161 ceph-mon[91579]: pgmap v8371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:11 smithi161 ceph-mon[91579]: pgmap v8372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:11 smithi188 ceph-mon[92425]: pgmap v8372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:13 smithi188 ceph-mon[92425]: pgmap v8373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:13 smithi161 ceph-mon[91579]: pgmap v8373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:15 smithi188 ceph-mon[92425]: pgmap v8374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:15 smithi161 ceph-mon[91579]: pgmap v8374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:17 smithi188 ceph-mon[92425]: pgmap v8375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:17 smithi161 ceph-mon[91579]: pgmap v8375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:19 smithi188 ceph-mon[92425]: pgmap v8376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:19 smithi161 ceph-mon[91579]: pgmap v8376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:21 smithi188 ceph-mon[92425]: pgmap v8377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:21 smithi161 ceph-mon[91579]: pgmap v8377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:18:22.624+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:18:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:18:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:18:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:18:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:18:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:18:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:18:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:18:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:18:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:18:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:18:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:18:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:18:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:18:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:18:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:18:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:18:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:18:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:18:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:18:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:18:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:18:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:18:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:18:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:18:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:18:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:18:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:18:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:18:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:18:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:18:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:18:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:18:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:18:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:18:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:18:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:18:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:18:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:18:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:18:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:18:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:18:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:18:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:18:22.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:18:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=461] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:18:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=461] Local address: 172.21.15.161, port 55572 2022-07-03T15:18:22.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=461] Peer address: 172.21.15.101, port 22 2022-07-03T15:18:22.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=461] Connection lost 2022-07-03T15:18:22.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=461] Aborting connection 2022-07-03T15:18:22.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:18:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:18:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:23 smithi188 ceph-mon[92425]: pgmap v8378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:18:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:18:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:18:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:23 smithi161 ceph-mon[91579]: pgmap v8378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:18:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:18:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:18:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:18:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:18:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:18:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:18:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:18:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:18:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:18:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:18:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:18:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:18:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:18:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:18:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:18:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:18:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:18:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:18:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:18:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:18:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:18:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:18:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:18:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:18:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:18:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:18:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:18:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:18:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:18:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:18:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:18:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:18:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:18:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:18:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:18:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:18:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:18:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:18:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:18:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:18:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:18:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:18:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:18:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:18:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:18:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:18:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:18:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:18:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=461] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:18:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=461] Local address: 172.21.15.161, port 55572 2022-07-03T15:18:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=461] Peer address: 172.21.15.101, port 22 2022-07-03T15:18:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=461] Connection lost 2022-07-03T15:18:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=461] Aborting connection 2022-07-03T15:18:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:18:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:18:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:18:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:18:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:18:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:18:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:18:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:18:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:18:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:18:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:18:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:18:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:18:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:18:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:18:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:18:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:18:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:18:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:18:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:18:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:18:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:18:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:18:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:18:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:18:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:18:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:18:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:18:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:18:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:18:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:18:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:18:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:18:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:18:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:18:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:18:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:18:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:18:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:18:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:18:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:18:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:18:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:18:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:18:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:18:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:18:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=461] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:18:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=461] Local address: 172.21.15.161, port 55572 2022-07-03T15:18:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=461] Peer address: 172.21.15.101, port 22 2022-07-03T15:18:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=461] Connection lost 2022-07-03T15:18:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=461] Aborting connection 2022-07-03T15:18:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:25 smithi188 ceph-mon[92425]: pgmap v8379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:25 smithi161 ceph-mon[91579]: pgmap v8379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:27 smithi188 ceph-mon[92425]: pgmap v8380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:27 smithi161 ceph-mon[91579]: pgmap v8380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:29 smithi188 ceph-mon[92425]: pgmap v8381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:29 smithi161 ceph-mon[91579]: pgmap v8381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:31 smithi188 ceph-mon[92425]: pgmap v8382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:31 smithi161 ceph-mon[91579]: pgmap v8382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:33 smithi188 ceph-mon[92425]: pgmap v8383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:33 smithi161 ceph-mon[91579]: pgmap v8383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:35 smithi188 ceph-mon[92425]: pgmap v8384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:35 smithi161 ceph-mon[91579]: pgmap v8384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:37 smithi188 ceph-mon[92425]: pgmap v8385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:37 smithi161 ceph-mon[91579]: pgmap v8385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:39 smithi188 ceph-mon[92425]: pgmap v8386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:39 smithi161 ceph-mon[91579]: pgmap v8386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:41 smithi188 ceph-mon[92425]: pgmap v8387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:41 smithi161 ceph-mon[91579]: pgmap v8387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:43 smithi188 ceph-mon[92425]: pgmap v8388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:43 smithi161 ceph-mon[91579]: pgmap v8388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:45 smithi188 ceph-mon[92425]: pgmap v8389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:45 smithi161 ceph-mon[91579]: pgmap v8389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:47 smithi188 ceph-mon[92425]: pgmap v8390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:47 smithi161 ceph-mon[91579]: pgmap v8390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:49 smithi188 ceph-mon[92425]: pgmap v8391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:49 smithi161 ceph-mon[91579]: pgmap v8391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:51 smithi188 ceph-mon[92425]: pgmap v8392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:51 smithi161 ceph-mon[91579]: pgmap v8392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:53 smithi188 ceph-mon[92425]: pgmap v8393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:53 smithi161 ceph-mon[91579]: pgmap v8393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:55 smithi188 ceph-mon[92425]: pgmap v8394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:55 smithi161 ceph-mon[91579]: pgmap v8394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:56.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:18:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:18:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:57 smithi188 ceph-mon[92425]: pgmap v8395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:57 smithi161 ceph-mon[91579]: pgmap v8395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:18:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:18:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:18:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:18:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:18:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:18:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:18:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:18:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:18:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:18:59 smithi188 ceph-mon[92425]: pgmap v8396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:18:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:18:59 smithi161 ceph-mon[91579]: pgmap v8396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:01 smithi188 ceph-mon[92425]: pgmap v8397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:01 smithi161 ceph-mon[91579]: pgmap v8397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:03 smithi188 ceph-mon[92425]: pgmap v8398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:03 smithi161 ceph-mon[91579]: pgmap v8398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:05 smithi188 ceph-mon[92425]: pgmap v8399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:05 smithi161 ceph-mon[91579]: pgmap v8399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:07 smithi188 ceph-mon[92425]: pgmap v8400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:07 smithi161 ceph-mon[91579]: pgmap v8400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:09 smithi188 ceph-mon[92425]: pgmap v8401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:09 smithi161 ceph-mon[91579]: pgmap v8401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:11 smithi188 ceph-mon[92425]: pgmap v8402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:11 smithi161 ceph-mon[91579]: pgmap v8402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:13 smithi188 ceph-mon[92425]: pgmap v8403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:13 smithi161 ceph-mon[91579]: pgmap v8403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:15 smithi188 ceph-mon[92425]: pgmap v8404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:15 smithi161 ceph-mon[91579]: pgmap v8404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:17 smithi188 ceph-mon[92425]: pgmap v8405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:17 smithi161 ceph-mon[91579]: pgmap v8405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:19 smithi188 ceph-mon[92425]: pgmap v8406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:19 smithi161 ceph-mon[91579]: pgmap v8406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:21 smithi188 ceph-mon[92425]: pgmap v8407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:21 smithi161 ceph-mon[91579]: pgmap v8407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:23.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:19:23.124+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:19:23.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:19:23.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:19:23.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:19:23.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:19:23.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:19:23.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:19:23.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:19:23.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:19:23.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:19:23.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:19:23.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:19:23.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:19:23.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:19:23.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:19:23.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:19:23.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:19:23.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:19:23.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:19:23.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:19:23.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:19:23.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:19:23.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:19:23.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:19:23.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:19:23.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:19:23.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:19:23.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:19:23.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:19:23.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:19:23.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:19:23.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:19:23.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:19:23.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:19:23.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:19:23.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:19:23.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:19:23.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:19:23.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:19:23.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:19:23.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:19:23.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:19:23.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:19:23.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:19:23.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:19:23.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=463] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:19:23.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=463] Local address: 172.21.15.161, port 55600 2022-07-03T15:19:23.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=463] Peer address: 172.21.15.101, port 22 2022-07-03T15:19:23.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=463] Connection lost 2022-07-03T15:19:23.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=463] Aborting connection 2022-07-03T15:19:23.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:19:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:23 smithi188 ceph-mon[92425]: pgmap v8408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:19:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:19:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:19:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-mon[91579]: pgmap v8408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:19:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:19:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:19:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:19:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:19:24.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:19:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:19:24.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:19:24.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:19:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:19:24.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:19:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:19:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:19:24.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:19:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:19:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:19:24.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:19:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:19:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:19:24.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:19:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:19:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:19:24.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:19:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:19:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:19:24.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:19:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:19:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:19:24.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:19:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:19:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:19:24.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:19:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:19:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:19:24.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:19:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:19:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:19:24.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:19:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:19:24.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:19:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:19:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:19:24.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:19:24.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:19:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:19:24.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:19:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:19:24.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:19:24.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:19:24.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=463] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:19:24.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=463] Local address: 172.21.15.161, port 55600 2022-07-03T15:19:24.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=463] Peer address: 172.21.15.101, port 22 2022-07-03T15:19:24.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=463] Connection lost 2022-07-03T15:19:24.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=463] Aborting connection 2022-07-03T15:19:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:19:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:19:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:19:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:19:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:19:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:19:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:19:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:19:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:19:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:19:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:19:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:19:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:19:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:19:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:19:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:19:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:19:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:19:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:19:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:19:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:19:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:19:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:19:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:19:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:19:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:19:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:19:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:19:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:19:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:19:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:19:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:19:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:19:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:19:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:19:24.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:19:24.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:19:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:19:24.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:19:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:19:24.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:19:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:19:24.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:19:24.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:19:24.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:19:24.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:19:24.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=463] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:19:24.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=463] Local address: 172.21.15.161, port 55600 2022-07-03T15:19:24.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=463] Peer address: 172.21.15.101, port 22 2022-07-03T15:19:24.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=463] Connection lost 2022-07-03T15:19:24.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=463] Aborting connection 2022-07-03T15:19:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:25 smithi188 ceph-mon[92425]: pgmap v8409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:25 smithi161 ceph-mon[91579]: pgmap v8409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:27 smithi161 ceph-mon[91579]: pgmap v8410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:27 smithi188 ceph-mon[92425]: pgmap v8410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:29 smithi161 ceph-mon[91579]: pgmap v8411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:29 smithi188 ceph-mon[92425]: pgmap v8411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:31 smithi188 ceph-mon[92425]: pgmap v8412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:31 smithi161 ceph-mon[91579]: pgmap v8412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:33 smithi188 ceph-mon[92425]: pgmap v8413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:33 smithi161 ceph-mon[91579]: pgmap v8413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:35 smithi188 ceph-mon[92425]: pgmap v8414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:35 smithi161 ceph-mon[91579]: pgmap v8414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:37 smithi188 ceph-mon[92425]: pgmap v8415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:37 smithi161 ceph-mon[91579]: pgmap v8415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:39 smithi188 ceph-mon[92425]: pgmap v8416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:39 smithi161 ceph-mon[91579]: pgmap v8416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:40.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:19:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:19:40.416+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.c 2022-07-03T15:19:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:41 smithi161 ceph-mon[91579]: pgmap v8417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:41 smithi188 ceph-mon[92425]: pgmap v8417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:19:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:43 smithi161 ceph-mon[91579]: pgmap v8418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:43 smithi188 ceph-mon[92425]: pgmap v8418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:45 smithi161 ceph-mon[91579]: pgmap v8419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:45 smithi188 ceph-mon[92425]: pgmap v8419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:47 smithi161 ceph-mon[91579]: pgmap v8420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:47 smithi188 ceph-mon[92425]: pgmap v8420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:49 smithi161 ceph-mon[91579]: pgmap v8421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:49 smithi188 ceph-mon[92425]: pgmap v8421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:51 smithi161 ceph-mon[91579]: pgmap v8422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:51 smithi188 ceph-mon[92425]: pgmap v8422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:53 smithi161 ceph-mon[91579]: pgmap v8423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:53 smithi188 ceph-mon[92425]: pgmap v8423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:55 smithi161 ceph-mon[91579]: pgmap v8424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:55 smithi188 ceph-mon[92425]: pgmap v8424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:19:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:57 smithi161 ceph-mon[91579]: pgmap v8425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:57 smithi188 ceph-mon[92425]: pgmap v8425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:19:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:19:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:19:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:19:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:19:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:19:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:19:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:19:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:19:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:19:59 smithi161 ceph-mon[91579]: pgmap v8426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:19:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:19:59 smithi188 ceph-mon[92425]: pgmap v8426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:20:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:20:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:01 smithi161 ceph-mon[91579]: pgmap v8427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:01 smithi188 ceph-mon[92425]: pgmap v8427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:03 smithi161 ceph-mon[91579]: pgmap v8428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:03 smithi188 ceph-mon[92425]: pgmap v8428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:05 smithi161 ceph-mon[91579]: pgmap v8429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:05 smithi188 ceph-mon[92425]: pgmap v8429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:07 smithi161 ceph-mon[91579]: pgmap v8430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:07 smithi188 ceph-mon[92425]: pgmap v8430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:08 smithi161 ceph-mon[91579]: pgmap v8431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:08 smithi188 ceph-mon[92425]: pgmap v8431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:11 smithi188 ceph-mon[92425]: pgmap v8432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:11 smithi161 ceph-mon[91579]: pgmap v8432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:13 smithi188 ceph-mon[92425]: pgmap v8433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:13 smithi161 ceph-mon[91579]: pgmap v8433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:15 smithi161 ceph-mon[91579]: pgmap v8434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:15 smithi188 ceph-mon[92425]: pgmap v8434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:17 smithi161 ceph-mon[91579]: pgmap v8435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:17 smithi188 ceph-mon[92425]: pgmap v8435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:19 smithi161 ceph-mon[91579]: pgmap v8436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:19 smithi188 ceph-mon[92425]: pgmap v8436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:21 smithi161 ceph-mon[91579]: pgmap v8437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:21 smithi188 ceph-mon[92425]: pgmap v8437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:23.685 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-mon[91579]: pgmap v8438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:23.687 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:20:23.625+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:20:23.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:20:23.688 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:20:23.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:20:23.689 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:20:23.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:20:23.690 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:20:23.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:20:23.691 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:20:23.692 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:20:23.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:20:23.693 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:20:23.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:20:23.694 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:20:23.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:20:23.695 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:20:23.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:20:23.696 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:20:23.697 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:20:23.697 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:20:23.698 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:20:23.698 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:20:23.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:20:23.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:20:23.699 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:20:23.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:20:23.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:20:23.700 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:20:23.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:20:23.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:20:23.701 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:20:23.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:20:23.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:20:23.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:20:23.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:20:23.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:20:23.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:20:23.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:20:23.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:20:23.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:20:23.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:20:23.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:20:23.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:20:23.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:20:23.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:20:23.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=465] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:20:23.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=465] Local address: 172.21.15.161, port 55628 2022-07-03T15:20:23.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=465] Peer address: 172.21.15.101, port 22 2022-07-03T15:20:23.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=465] Connection lost 2022-07-03T15:20:23.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=465] Aborting connection 2022-07-03T15:20:23.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:20:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:20:23.710 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:23 smithi188 ceph-mon[92425]: pgmap v8438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:20:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:20:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:20:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:20:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:20:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:20:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:20:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:20:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:20:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:20:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:20:25.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:20:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:20:25.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:20:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:20:25.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:20:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:20:25.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:20:25.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:20:25.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:20:25.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:20:25.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:20:25.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:20:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:20:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:20:25.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:20:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:20:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:20:25.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:20:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:20:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:20:25.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:20:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:20:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:20:25.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:20:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:20:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:20:25.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:20:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:20:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:20:25.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:20:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:20:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:20:25.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:20:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:20:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:20:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:20:25.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:20:25.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:20:25.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:20:25.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:20:25.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:20:25.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=465] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:20:25.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=465] Local address: 172.21.15.161, port 55628 2022-07-03T15:20:25.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=465] Peer address: 172.21.15.101, port 22 2022-07-03T15:20:25.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=465] Connection lost 2022-07-03T15:20:25.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=465] Aborting connection 2022-07-03T15:20:25.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:25 smithi161 ceph-mon[91579]: pgmap v8439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:20:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:20:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:20:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:20:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:20:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:20:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:20:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:20:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:20:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:20:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:20:25.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:20:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:20:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:20:25.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:20:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:20:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:20:25.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:20:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:20:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:20:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:20:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:20:25.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:20:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:20:25.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:20:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:20:25.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:20:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:20:25.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:20:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:20:25.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:20:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:20:25.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:20:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:20:25.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:20:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:20:25.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:20:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:20:25.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:20:25.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:20:25.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:20:25.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:20:25.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:20:25.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:20:25.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:20:25.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:20:25.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=465] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:20:25.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=465] Local address: 172.21.15.161, port 55628 2022-07-03T15:20:25.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=465] Peer address: 172.21.15.101, port 22 2022-07-03T15:20:25.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=465] Connection lost 2022-07-03T15:20:25.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=465] Aborting connection 2022-07-03T15:20:25.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:25 smithi188 ceph-mon[92425]: pgmap v8439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:27 smithi161 ceph-mon[91579]: pgmap v8440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:27 smithi188 ceph-mon[92425]: pgmap v8440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:29 smithi161 ceph-mon[91579]: pgmap v8441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:29 smithi188 ceph-mon[92425]: pgmap v8441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:31 smithi161 ceph-mon[91579]: pgmap v8442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:31 smithi188 ceph-mon[92425]: pgmap v8442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:33 smithi161 ceph-mon[91579]: pgmap v8443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:33 smithi188 ceph-mon[92425]: pgmap v8443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:35 smithi161 ceph-mon[91579]: pgmap v8444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:35 smithi188 ceph-mon[92425]: pgmap v8444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:37 smithi161 ceph-mon[91579]: pgmap v8445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:37 smithi188 ceph-mon[92425]: pgmap v8445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:39 smithi161 ceph-mon[91579]: pgmap v8446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:39 smithi188 ceph-mon[92425]: pgmap v8446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:41 smithi161 ceph-mon[91579]: pgmap v8447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:41 smithi188 ceph-mon[92425]: pgmap v8447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:43 smithi161 ceph-mon[91579]: pgmap v8448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:43 smithi188 ceph-mon[92425]: pgmap v8448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:45 smithi161 ceph-mon[91579]: pgmap v8449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:45 smithi188 ceph-mon[92425]: pgmap v8449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:47 smithi161 ceph-mon[91579]: pgmap v8450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:47 smithi188 ceph-mon[92425]: pgmap v8450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:48 smithi161 ceph-mon[91579]: pgmap v8451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:48 smithi188 ceph-mon[92425]: pgmap v8451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:51 smithi161 ceph-mon[91579]: pgmap v8452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:51 smithi188 ceph-mon[92425]: pgmap v8452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:53 smithi161 ceph-mon[91579]: pgmap v8453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:53 smithi188 ceph-mon[92425]: pgmap v8453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:55 smithi161 ceph-mon[91579]: pgmap v8454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:55 smithi188 ceph-mon[92425]: pgmap v8454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:20:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:57 smithi161 ceph-mon[91579]: pgmap v8455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:57 smithi188 ceph-mon[92425]: pgmap v8455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:20:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:20:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:20:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:20:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:20:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:20:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:20:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:20:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:20:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:20:59 smithi161 ceph-mon[91579]: pgmap v8456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:20:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:20:59 smithi188 ceph-mon[92425]: pgmap v8456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:01 smithi161 ceph-mon[91579]: pgmap v8457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:01 smithi188 ceph-mon[92425]: pgmap v8457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:03 smithi161 ceph-mon[91579]: pgmap v8458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:03 smithi188 ceph-mon[92425]: pgmap v8458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:05 smithi161 ceph-mon[91579]: pgmap v8459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:05 smithi188 ceph-mon[92425]: pgmap v8459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:07 smithi161 ceph-mon[91579]: pgmap v8460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:07 smithi188 ceph-mon[92425]: pgmap v8460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:09 smithi161 ceph-mon[91579]: pgmap v8461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:09 smithi188 ceph-mon[92425]: pgmap v8461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:11 smithi161 ceph-mon[91579]: pgmap v8462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:11 smithi188 ceph-mon[92425]: pgmap v8462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:13 smithi161 ceph-mon[91579]: pgmap v8463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:13 smithi188 ceph-mon[92425]: pgmap v8463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:15 smithi161 ceph-mon[91579]: pgmap v8464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:15 smithi188 ceph-mon[92425]: pgmap v8464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:17 smithi161 ceph-mon[91579]: pgmap v8465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:17 smithi188 ceph-mon[92425]: pgmap v8465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:19 smithi161 ceph-mon[91579]: pgmap v8466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:19 smithi188 ceph-mon[92425]: pgmap v8466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:21 smithi161 ceph-mon[91579]: pgmap v8467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:21 smithi188 ceph-mon[92425]: pgmap v8467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:22 smithi161 ceph-mon[91579]: pgmap v8468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:22 smithi188 ceph-mon[92425]: pgmap v8468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:24.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:21:24.136+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:21:24.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:21:24.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:21:24.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:21:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:21:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:21:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:21:24.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:21:24.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:21:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:21:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:21:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:21:24.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:21:24.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:21:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:21:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:21:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:21:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:21:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:21:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:21:24.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:21:24.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:21:24.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:21:24.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:21:24.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:21:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:21:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:21:24.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:21:24.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:21:24.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:21:24.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:21:24.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:21:24.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:21:24.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:21:24.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:21:24.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:21:24.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:21:24.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:21:24.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:21:24.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:21:24.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:21:24.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:21:24.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:21:24.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:21:24.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:21:24.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=467] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:21:24.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=467] Local address: 172.21.15.161, port 55656 2022-07-03T15:21:24.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=467] Peer address: 172.21.15.101, port 22 2022-07-03T15:21:24.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=467] Connection lost 2022-07-03T15:21:24.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=467] Aborting connection 2022-07-03T15:21:24.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:21:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: pgmap v8469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:21:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:21:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:21:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:21:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:21:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:21:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:21:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:21:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:21:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:21:25.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:21:25.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:21:25.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:21:25.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:21:25.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:21:25.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:21:25.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:21:25.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:21:25.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:21:25.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:21:25.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:21:25.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:21:25.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:21:25.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:21:25.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:21:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:21:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:21:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:21:25.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:21:25.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:21:25.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:21:25.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:21:25.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:21:25.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:21:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:21:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:21:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:21:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:21:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:21:25.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:21:25.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:21:25.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:21:25.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:21:25.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:21:25.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:21:25.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:21:25.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:21:25.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:21:25.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:21:25.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=467] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:21:25.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=467] Local address: 172.21.15.161, port 55656 2022-07-03T15:21:25.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=467] Peer address: 172.21.15.101, port 22 2022-07-03T15:21:25.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=467] Connection lost 2022-07-03T15:21:25.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=467] Aborting connection 2022-07-03T15:21:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: pgmap v8469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:21:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:21:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:21:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:21:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:21:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:21:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:21:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:21:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:21:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:21:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:21:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:21:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:21:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:21:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:21:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:21:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:21:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:21:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:21:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:21:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:21:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:21:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:21:25.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:21:25.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:21:25.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:21:25.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:21:25.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:21:25.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:21:25.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:21:25.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:21:25.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:21:25.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:21:25.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:21:25.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:21:25.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:21:25.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:21:25.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:21:25.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:21:25.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:21:25.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:21:25.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:21:25.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:21:25.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:21:25.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:21:25.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:21:25.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:21:25.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:21:25.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:21:25.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=467] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:21:25.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=467] Local address: 172.21.15.161, port 55656 2022-07-03T15:21:25.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=467] Peer address: 172.21.15.101, port 22 2022-07-03T15:21:25.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=467] Connection lost 2022-07-03T15:21:25.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=467] Aborting connection 2022-07-03T15:21:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:27 smithi161 ceph-mon[91579]: pgmap v8470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:27 smithi188 ceph-mon[92425]: pgmap v8470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:29 smithi161 ceph-mon[91579]: pgmap v8471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:29 smithi188 ceph-mon[92425]: pgmap v8471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:31 smithi161 ceph-mon[91579]: pgmap v8472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:31 smithi188 ceph-mon[92425]: pgmap v8472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:33 smithi161 ceph-mon[91579]: pgmap v8473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:33 smithi188 ceph-mon[92425]: pgmap v8473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:35 smithi161 ceph-mon[91579]: pgmap v8474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:35 smithi188 ceph-mon[92425]: pgmap v8474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:37 smithi161 ceph-mon[91579]: pgmap v8475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:37 smithi188 ceph-mon[92425]: pgmap v8475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:39 smithi161 ceph-mon[91579]: pgmap v8476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:39 smithi188 ceph-mon[92425]: pgmap v8476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:41 smithi161 ceph-mon[91579]: pgmap v8477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:41 smithi188 ceph-mon[92425]: pgmap v8477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:43 smithi161 ceph-mon[91579]: pgmap v8478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:43 smithi188 ceph-mon[92425]: pgmap v8478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:45 smithi161 ceph-mon[91579]: pgmap v8479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:45 smithi188 ceph-mon[92425]: pgmap v8479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:47 smithi161 ceph-mon[91579]: pgmap v8480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:47 smithi188 ceph-mon[92425]: pgmap v8480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:21:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:49 smithi161 ceph-mon[91579]: pgmap v8481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:49 smithi188 ceph-mon[92425]: pgmap v8481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:51 smithi161 ceph-mon[91579]: pgmap v8482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:51 smithi188 ceph-mon[92425]: pgmap v8482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:53 smithi161 ceph-mon[91579]: pgmap v8483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:53 smithi188 ceph-mon[92425]: pgmap v8483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:55 smithi161 ceph-mon[91579]: pgmap v8484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:55 smithi188 ceph-mon[92425]: pgmap v8484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:56 smithi161 ceph-mon[91579]: pgmap v8485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:21:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:56 smithi188 ceph-mon[92425]: pgmap v8485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:21:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:58 smithi188 ceph-mon[92425]: pgmap v8486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:21:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:21:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:21:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:21:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:58 smithi161 ceph-mon[91579]: pgmap v8486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:21:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:21:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:21:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:21:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:22:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:01 smithi161 ceph-mon[91579]: pgmap v8487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:01 smithi188 ceph-mon[92425]: pgmap v8487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:03 smithi161 ceph-mon[91579]: pgmap v8488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:03 smithi188 ceph-mon[92425]: pgmap v8488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:05 smithi161 ceph-mon[91579]: pgmap v8489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:05 smithi188 ceph-mon[92425]: pgmap v8489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:07 smithi161 ceph-mon[91579]: pgmap v8490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:07 smithi188 ceph-mon[92425]: pgmap v8490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:09 smithi161 ceph-mon[91579]: pgmap v8491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:09 smithi188 ceph-mon[92425]: pgmap v8491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:11 smithi161 ceph-mon[91579]: pgmap v8492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:11 smithi188 ceph-mon[92425]: pgmap v8492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:13 smithi161 ceph-mon[91579]: pgmap v8493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:13 smithi188 ceph-mon[92425]: pgmap v8493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:15 smithi161 ceph-mon[91579]: pgmap v8494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:15 smithi188 ceph-mon[92425]: pgmap v8494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:17 smithi161 ceph-mon[91579]: pgmap v8495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:17 smithi188 ceph-mon[92425]: pgmap v8495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:19 smithi161 ceph-mon[91579]: pgmap v8496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:19 smithi188 ceph-mon[92425]: pgmap v8496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:21 smithi161 ceph-mon[91579]: pgmap v8497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:21 smithi188 ceph-mon[92425]: pgmap v8497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:23 smithi161 ceph-mon[91579]: pgmap v8498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:23 smithi188 ceph-mon[92425]: pgmap v8498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:25.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:22:24.640+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:22:25.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:22:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:22:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:22:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:22:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:22:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:22:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:22:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:22:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:22:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:22:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:22:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:22:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:22:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:22:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:22:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:22:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:22:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:22:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:22:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:22:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:22:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:22:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:22:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:22:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:22:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:22:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:22:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:22:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:22:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:22:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:22:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:22:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:22:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:22:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:22:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:22:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:22:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:22:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:22:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:22:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:22:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:22:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:22:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:22:25.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=469] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:22:25.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=469] Local address: 172.21.15.161, port 55684 2022-07-03T15:22:25.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=469] Peer address: 172.21.15.101, port 22 2022-07-03T15:22:25.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=469] Connection lost 2022-07-03T15:22:25.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=469] Aborting connection 2022-07-03T15:22:25.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:22:25.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:22:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:22:25.567+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T15:22:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:25 smithi161 ceph-mon[91579]: pgmap v8499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:22:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:22:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:22:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:25 smithi188 ceph-mon[92425]: pgmap v8499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:22:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:22:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:22:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:22:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:22:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:22:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:22:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:22:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:22:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:22:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:22:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:22:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:22:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:22:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:22:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:22:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:22:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:22:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:22:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:22:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:22:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:22:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:22:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:22:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:22:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:22:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:22:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:22:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:22:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:22:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:22:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:22:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:22:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:22:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:22:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:22:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:22:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:22:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:22:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:22:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:22:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:22:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:22:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:22:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:22:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:22:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:22:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:22:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:22:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=469] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:22:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=469] Local address: 172.21.15.161, port 55684 2022-07-03T15:22:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=469] Peer address: 172.21.15.101, port 22 2022-07-03T15:22:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=469] Connection lost 2022-07-03T15:22:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=469] Aborting connection 2022-07-03T15:22:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:22:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:22:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:22:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:22:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:22:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:22:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:22:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:22:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:22:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:22:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:22:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:22:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:22:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:22:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:22:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:22:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:22:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:22:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:22:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:22:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:22:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:22:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:22:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:22:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:22:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:22:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:22:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:22:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:22:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:22:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:22:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:22:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:22:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:22:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:22:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:22:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:22:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:22:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:22:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:22:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:22:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:22:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:22:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:22:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:22:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:22:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=469] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:22:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=469] Local address: 172.21.15.161, port 55684 2022-07-03T15:22:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=469] Peer address: 172.21.15.101, port 22 2022-07-03T15:22:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=469] Connection lost 2022-07-03T15:22:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=469] Aborting connection 2022-07-03T15:22:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:22:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:27 smithi161 ceph-mon[91579]: pgmap v8500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:27 smithi188 ceph-mon[92425]: pgmap v8500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:29 smithi161 ceph-mon[91579]: pgmap v8501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:29 smithi188 ceph-mon[92425]: pgmap v8501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:31 smithi161 ceph-mon[91579]: pgmap v8502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:31 smithi188 ceph-mon[92425]: pgmap v8502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:32 smithi161 ceph-mon[91579]: pgmap v8503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:32 smithi188 ceph-mon[92425]: pgmap v8503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:34 smithi161 ceph-mon[91579]: pgmap v8504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:34 smithi188 ceph-mon[92425]: pgmap v8504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:37 smithi161 ceph-mon[91579]: pgmap v8505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:37 smithi188 ceph-mon[92425]: pgmap v8505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:39 smithi161 ceph-mon[91579]: pgmap v8506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:39 smithi188 ceph-mon[92425]: pgmap v8506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:41 smithi161 ceph-mon[91579]: pgmap v8507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:41 smithi188 ceph-mon[92425]: pgmap v8507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:43 smithi161 ceph-mon[91579]: pgmap v8508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:43 smithi188 ceph-mon[92425]: pgmap v8508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:45 smithi161 ceph-mon[91579]: pgmap v8509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:45 smithi188 ceph-mon[92425]: pgmap v8509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:47 smithi161 ceph-mon[91579]: pgmap v8510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:47 smithi188 ceph-mon[92425]: pgmap v8510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:49 smithi161 ceph-mon[91579]: pgmap v8511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:49 smithi188 ceph-mon[92425]: pgmap v8511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:51 smithi161 ceph-mon[91579]: pgmap v8512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:51 smithi188 ceph-mon[92425]: pgmap v8512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:53 smithi161 ceph-mon[91579]: pgmap v8513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:53 smithi188 ceph-mon[92425]: pgmap v8513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:55 smithi161 ceph-mon[91579]: pgmap v8514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:55 smithi188 ceph-mon[92425]: pgmap v8514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:57 smithi161 ceph-mon[91579]: pgmap v8515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:57 smithi188 ceph-mon[92425]: pgmap v8515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:22:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:22:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:22:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:22:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:22:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:22:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:22:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:22:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:22:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:22:59 smithi161 ceph-mon[91579]: pgmap v8516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:22:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:22:59 smithi188 ceph-mon[92425]: pgmap v8516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:01 smithi161 ceph-mon[91579]: pgmap v8517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:01 smithi188 ceph-mon[92425]: pgmap v8517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:03 smithi161 ceph-mon[91579]: pgmap v8518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:03 smithi188 ceph-mon[92425]: pgmap v8518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:05 smithi161 ceph-mon[91579]: pgmap v8519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:05 smithi188 ceph-mon[92425]: pgmap v8519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:07 smithi161 ceph-mon[91579]: pgmap v8520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:07 smithi188 ceph-mon[92425]: pgmap v8520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:08 smithi161 ceph-mon[91579]: pgmap v8521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:08 smithi188 ceph-mon[92425]: pgmap v8521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:11 smithi161 ceph-mon[91579]: pgmap v8522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:11 smithi188 ceph-mon[92425]: pgmap v8522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:13 smithi161 ceph-mon[91579]: pgmap v8523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:13 smithi188 ceph-mon[92425]: pgmap v8523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:15 smithi161 ceph-mon[91579]: pgmap v8524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:15 smithi188 ceph-mon[92425]: pgmap v8524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:17 smithi161 ceph-mon[91579]: pgmap v8525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:17 smithi188 ceph-mon[92425]: pgmap v8525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:19 smithi161 ceph-mon[91579]: pgmap v8526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:19 smithi188 ceph-mon[92425]: pgmap v8526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:21 smithi161 ceph-mon[91579]: pgmap v8527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:21 smithi188 ceph-mon[92425]: pgmap v8527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:23 smithi161 ceph-mon[91579]: pgmap v8528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:23 smithi188 ceph-mon[92425]: pgmap v8528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:25.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:23:25.144+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:23:25.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:23:25.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:23:25.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:23:25.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:23:25.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:23:25.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:23:25.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:23:25.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:23:25.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:23:25.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:23:25.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:23:25.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:23:25.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:23:25.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:23:25.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:23:25.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:23:25.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:23:25.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:23:25.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:23:25.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:23:25.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:23:25.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:23:25.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:23:25.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:23:25.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:23:25.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:23:25.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:23:25.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:23:25.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:23:25.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:23:25.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:23:25.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:23:25.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:23:25.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:23:25.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:23:25.485 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:23:25.485 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:23:25.485 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:23:25.486 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:23:25.486 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:23:25.486 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:23:25.487 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:23:25.487 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:23:25.487 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:23:25.488 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=471] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:23:25.488 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=471] Local address: 172.21.15.161, port 55690 2022-07-03T15:23:25.488 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=471] Peer address: 172.21.15.101, port 22 2022-07-03T15:23:25.489 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=471] Connection lost 2022-07-03T15:23:25.489 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=471] Aborting connection 2022-07-03T15:23:25.489 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:23:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-mon[91579]: pgmap v8529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:23:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:23:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:23:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:25 smithi188 ceph-mon[92425]: pgmap v8529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:23:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:23:25.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:23:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:23:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:23:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:23:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:23:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:23:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:23:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:23:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:23:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:23:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:23:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:23:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:23:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:23:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:23:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:23:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:23:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:23:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:23:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:23:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:23:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:23:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:23:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:23:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:23:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:23:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:23:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:23:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:23:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:23:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:23:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:23:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:23:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:23:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:23:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:23:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:23:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:23:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:23:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:23:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:23:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:23:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:23:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:23:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:23:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:23:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:23:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=471] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:23:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=471] Local address: 172.21.15.161, port 55690 2022-07-03T15:23:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=471] Peer address: 172.21.15.101, port 22 2022-07-03T15:23:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=471] Connection lost 2022-07-03T15:23:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=471] Aborting connection 2022-07-03T15:23:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:23:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:23:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:23:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:23:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:23:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:23:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:23:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:23:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:23:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:23:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:23:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:23:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:23:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:23:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:23:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:23:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:23:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:23:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:23:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:23:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:23:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:23:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:23:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:23:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:23:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:23:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:23:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:23:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:23:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:23:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:23:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:23:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:23:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:23:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:23:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:23:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:23:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:23:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:23:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:23:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:23:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:23:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:23:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:23:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:23:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:23:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=471] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:23:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=471] Local address: 172.21.15.161, port 55690 2022-07-03T15:23:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=471] Peer address: 172.21.15.101, port 22 2022-07-03T15:23:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=471] Connection lost 2022-07-03T15:23:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=471] Aborting connection 2022-07-03T15:23:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:23:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:27 smithi161 ceph-mon[91579]: pgmap v8530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:27 smithi188 ceph-mon[92425]: pgmap v8530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:29 smithi161 ceph-mon[91579]: pgmap v8531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:29 smithi188 ceph-mon[92425]: pgmap v8531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:31 smithi161 ceph-mon[91579]: pgmap v8532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:31 smithi188 ceph-mon[92425]: pgmap v8532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:33 smithi161 ceph-mon[91579]: pgmap v8533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:33 smithi188 ceph-mon[92425]: pgmap v8533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:35 smithi161 ceph-mon[91579]: pgmap v8534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:35 smithi188 ceph-mon[92425]: pgmap v8534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:37 smithi161 ceph-mon[91579]: pgmap v8535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:37 smithi188 ceph-mon[92425]: pgmap v8535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:39 smithi161 ceph-mon[91579]: pgmap v8536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:39 smithi188 ceph-mon[92425]: pgmap v8536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:41 smithi161 ceph-mon[91579]: pgmap v8537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:41 smithi188 ceph-mon[92425]: pgmap v8537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:43 smithi161 ceph-mon[91579]: pgmap v8538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:43 smithi188 ceph-mon[92425]: pgmap v8538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:45 smithi161 ceph-mon[91579]: pgmap v8539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:45 smithi188 ceph-mon[92425]: pgmap v8539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:47 smithi161 ceph-mon[91579]: pgmap v8540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:47 smithi188 ceph-mon[92425]: pgmap v8540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:49 smithi161 ceph-mon[91579]: pgmap v8541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:49 smithi188 ceph-mon[92425]: pgmap v8541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:51 smithi161 ceph-mon[91579]: pgmap v8542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:51 smithi188 ceph-mon[92425]: pgmap v8542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:53 smithi161 ceph-mon[91579]: pgmap v8543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:53 smithi188 ceph-mon[92425]: pgmap v8543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:55 smithi161 ceph-mon[91579]: pgmap v8544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:55 smithi188 ceph-mon[92425]: pgmap v8544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:57 smithi161 ceph-mon[91579]: pgmap v8545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:57 smithi188 ceph-mon[92425]: pgmap v8545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:23:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:23:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:23:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:23:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:23:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:23:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:23:59 smithi161 ceph-mon[91579]: pgmap v8546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:23:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:23:59 smithi188 ceph-mon[92425]: pgmap v8546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:01 smithi161 ceph-mon[91579]: pgmap v8547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:01 smithi188 ceph-mon[92425]: pgmap v8547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:03 smithi161 ceph-mon[91579]: pgmap v8548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:03 smithi188 ceph-mon[92425]: pgmap v8548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:05 smithi161 ceph-mon[91579]: pgmap v8549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:05 smithi188 ceph-mon[92425]: pgmap v8549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:07 smithi161 ceph-mon[91579]: pgmap v8550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:07 smithi188 ceph-mon[92425]: pgmap v8550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:09 smithi161 ceph-mon[91579]: pgmap v8551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:09 smithi188 ceph-mon[92425]: pgmap v8551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:11 smithi161 ceph-mon[91579]: pgmap v8552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:11 smithi188 ceph-mon[92425]: pgmap v8552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:13 smithi161 ceph-mon[91579]: pgmap v8553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:13 smithi188 ceph-mon[92425]: pgmap v8553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:15 smithi161 ceph-mon[91579]: pgmap v8554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:15 smithi188 ceph-mon[92425]: pgmap v8554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:17 smithi161 ceph-mon[91579]: pgmap v8555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:17 smithi188 ceph-mon[92425]: pgmap v8555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:19 smithi161 ceph-mon[91579]: pgmap v8556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:19 smithi188 ceph-mon[92425]: pgmap v8556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:21 smithi161 ceph-mon[91579]: pgmap v8557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:21 smithi188 ceph-mon[92425]: pgmap v8557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:23 smithi161 ceph-mon[91579]: pgmap v8558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:23 smithi188 ceph-mon[92425]: pgmap v8558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:25 smithi161 ceph-mon[91579]: pgmap v8559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:25 smithi188 ceph-mon[92425]: pgmap v8559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:26.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:24:26.070+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:24:26.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:24:26.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:24:26.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:24:26.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:24:26.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:24:26.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:24:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:24:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:24:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:24:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:24:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:24:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:24:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:24:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:24:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:24:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:24:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:24:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:24:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:24:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:24:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:24:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:24:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:24:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:24:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:24:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:24:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:24:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:24:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:24:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:24:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:24:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:24:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:24:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:24:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:24:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:24:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:24:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:24:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:24:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:24:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:24:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:24:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:24:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:24:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=473] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:24:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=473] Local address: 172.21.15.161, port 55720 2022-07-03T15:24:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=473] Peer address: 172.21.15.101, port 22 2022-07-03T15:24:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=473] Connection lost 2022-07-03T15:24:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=473] Aborting connection 2022-07-03T15:24:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:24:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: pgmap v8560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:24:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:24:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:24:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:24:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:24:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:24:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:24:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:24:26.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:24:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:24:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:24:26.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:24:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:24:26.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:24:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:24:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:24:26.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:24:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:24:26.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:24:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:24:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:24:26.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:24:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:24:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:24:26.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:24:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:24:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:24:26.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:24:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:24:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:24:26.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:24:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:24:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:24:26.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:24:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:24:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:24:26.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:24:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:24:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:24:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:24:26.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:24:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:24:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:24:26.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:24:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:24:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:24:26.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:24:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:24:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:24:26.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=473] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:24:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=473] Local address: 172.21.15.161, port 55720 2022-07-03T15:24:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=473] Peer address: 172.21.15.101, port 22 2022-07-03T15:24:26.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=473] Connection lost 2022-07-03T15:24:26.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=473] Aborting connection 2022-07-03T15:24:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: pgmap v8560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:24:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:24:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:24:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:24:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:24:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:24:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:24:26.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:24:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:24:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:24:26.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:24:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:24:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:24:26.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:24:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:24:26.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:24:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:24:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:24:26.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:24:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:24:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:24:26.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:24:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:24:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:24:26.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:24:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:24:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:24:26.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:24:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:24:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:24:26.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:24:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:24:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:24:26.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:24:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:24:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:24:26.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:24:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:24:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:24:26.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:24:26.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:24:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:24:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:24:26.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:24:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:24:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:24:26.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:24:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:24:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:24:26.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=473] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:24:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=473] Local address: 172.21.15.161, port 55720 2022-07-03T15:24:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=473] Peer address: 172.21.15.101, port 22 2022-07-03T15:24:26.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=473] Connection lost 2022-07-03T15:24:26.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=473] Aborting connection 2022-07-03T15:24:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:28 smithi161 ceph-mon[91579]: pgmap v8561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:28 smithi188 ceph-mon[92425]: pgmap v8561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:31 smithi161 ceph-mon[91579]: pgmap v8562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:31 smithi188 ceph-mon[92425]: pgmap v8562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:33 smithi161 ceph-mon[91579]: pgmap v8563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:33 smithi188 ceph-mon[92425]: pgmap v8563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:35 smithi161 ceph-mon[91579]: pgmap v8564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:35 smithi188 ceph-mon[92425]: pgmap v8564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:37 smithi161 ceph-mon[91579]: pgmap v8565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:37 smithi188 ceph-mon[92425]: pgmap v8565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:39 smithi161 ceph-mon[91579]: pgmap v8566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:39 smithi188 ceph-mon[92425]: pgmap v8566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:41 smithi161 ceph-mon[91579]: pgmap v8567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:41 smithi188 ceph-mon[92425]: pgmap v8567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:43 smithi161 ceph-mon[91579]: pgmap v8568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:43 smithi188 ceph-mon[92425]: pgmap v8568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:45 smithi161 ceph-mon[91579]: pgmap v8569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:45 smithi188 ceph-mon[92425]: pgmap v8569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:47 smithi161 ceph-mon[91579]: pgmap v8570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:47 smithi188 ceph-mon[92425]: pgmap v8570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:49 smithi161 ceph-mon[91579]: pgmap v8571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:49 smithi188 ceph-mon[92425]: pgmap v8571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:51 smithi161 ceph-mon[91579]: pgmap v8572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:51 smithi188 ceph-mon[92425]: pgmap v8572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:53 smithi161 ceph-mon[91579]: pgmap v8573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:53 smithi188 ceph-mon[92425]: pgmap v8573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:55 smithi161 ceph-mon[91579]: pgmap v8574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:55 smithi188 ceph-mon[92425]: pgmap v8574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:57 smithi161 ceph-mon[91579]: pgmap v8575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:57 smithi188 ceph-mon[92425]: pgmap v8575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:24:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:24:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:24:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:24:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:24:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:24:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:24:59 smithi161 ceph-mon[91579]: pgmap v8576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:24:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:24:59 smithi188 ceph-mon[92425]: pgmap v8576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:01 smithi161 ceph-mon[91579]: pgmap v8577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:01 smithi188 ceph-mon[92425]: pgmap v8577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:02 smithi161 ceph-mon[91579]: pgmap v8578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:02 smithi188 ceph-mon[92425]: pgmap v8578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:04 smithi161 ceph-mon[91579]: pgmap v8579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:04 smithi188 ceph-mon[92425]: pgmap v8579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:07 smithi161 ceph-mon[91579]: pgmap v8580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:07 smithi188 ceph-mon[92425]: pgmap v8580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:09 smithi161 ceph-mon[91579]: pgmap v8581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:09 smithi188 ceph-mon[92425]: pgmap v8581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:11 smithi161 ceph-mon[91579]: pgmap v8582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:11 smithi188 ceph-mon[92425]: pgmap v8582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:13 smithi161 ceph-mon[91579]: pgmap v8583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:13 smithi188 ceph-mon[92425]: pgmap v8583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:15 smithi161 ceph-mon[91579]: pgmap v8584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:15 smithi188 ceph-mon[92425]: pgmap v8584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:17 smithi161 ceph-mon[91579]: pgmap v8585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:17 smithi188 ceph-mon[92425]: pgmap v8585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:19 smithi161 ceph-mon[91579]: pgmap v8586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:19 smithi188 ceph-mon[92425]: pgmap v8586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:21 smithi161 ceph-mon[91579]: pgmap v8587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:21 smithi188 ceph-mon[92425]: pgmap v8587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:23 smithi161 ceph-mon[91579]: pgmap v8588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:23 smithi188 ceph-mon[92425]: pgmap v8588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:25 smithi161 ceph-mon[91579]: pgmap v8589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:25 smithi188 ceph-mon[92425]: pgmap v8589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:26.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:25:26.565+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:25:26.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:25:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:25:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:25:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:25:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:25:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:25:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:25:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:25:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:25:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:25:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:25:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:25:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:25:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:25:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:25:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:25:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:25:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:25:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:25:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:25:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:25:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:25:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:25:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:25:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:25:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:25:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:25:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:25:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:25:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:25:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:25:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:25:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:25:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:25:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:25:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:25:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:25:26.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:25:26.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:25:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:25:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:25:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:25:26.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:25:26.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:25:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=475] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:25:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=475] Local address: 172.21.15.161, port 55724 2022-07-03T15:25:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=475] Peer address: 172.21.15.101, port 22 2022-07-03T15:25:26.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=475] Connection lost 2022-07-03T15:25:26.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=475] Aborting connection 2022-07-03T15:25:26.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:25:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:27 smithi161 ceph-mon[91579]: pgmap v8590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:25:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:25:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:25:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:27 smithi188 ceph-mon[92425]: pgmap v8590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:25:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:25:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:25:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:25:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:25:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:25:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:25:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:25:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:25:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:25:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:25:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:25:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:25:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:25:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:25:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:25:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:25:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:25:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:25:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:25:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:25:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:25:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:25:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:25:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:25:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:25:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:25:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:25:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:25:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:25:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:25:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:25:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:25:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:25:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:25:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:25:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:25:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:25:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:25:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:25:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:25:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:25:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:25:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:25:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:25:28.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:25:28.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:25:28.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:25:28.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:25:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=475] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:25:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=475] Local address: 172.21.15.161, port 55724 2022-07-03T15:25:28.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=475] Peer address: 172.21.15.101, port 22 2022-07-03T15:25:28.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=475] Connection lost 2022-07-03T15:25:28.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=475] Aborting connection 2022-07-03T15:25:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:25:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:25:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:25:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:25:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:25:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:25:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:25:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:25:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:25:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:25:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:25:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:25:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:25:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:25:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:25:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:25:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:25:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:25:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:25:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:25:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:25:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:25:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:25:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:25:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:25:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:25:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:25:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:25:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:25:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:25:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:25:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:25:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:25:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:25:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:25:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:25:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:25:28.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:25:28.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:25:28.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:25:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:25:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:25:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:25:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:25:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:25:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:25:28.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=475] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:25:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=475] Local address: 172.21.15.161, port 55724 2022-07-03T15:25:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=475] Peer address: 172.21.15.101, port 22 2022-07-03T15:25:28.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=475] Connection lost 2022-07-03T15:25:28.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=475] Aborting connection 2022-07-03T15:25:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:29 smithi161 ceph-mon[91579]: pgmap v8591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:29 smithi188 ceph-mon[92425]: pgmap v8591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:31 smithi161 ceph-mon[91579]: pgmap v8592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:31 smithi188 ceph-mon[92425]: pgmap v8592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:33 smithi161 ceph-mon[91579]: pgmap v8593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:33 smithi188 ceph-mon[92425]: pgmap v8593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:35 smithi161 ceph-mon[91579]: pgmap v8594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:35 smithi188 ceph-mon[92425]: pgmap v8594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:37 smithi161 ceph-mon[91579]: pgmap v8595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:37 smithi188 ceph-mon[92425]: pgmap v8595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:38 smithi161 ceph-mon[91579]: pgmap v8596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:38 smithi188 ceph-mon[92425]: pgmap v8596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:41 smithi161 ceph-mon[91579]: pgmap v8597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:41 smithi188 ceph-mon[92425]: pgmap v8597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:43 smithi161 ceph-mon[91579]: pgmap v8598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:43 smithi188 ceph-mon[92425]: pgmap v8598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:45 smithi161 ceph-mon[91579]: pgmap v8599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:45 smithi188 ceph-mon[92425]: pgmap v8599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:47 smithi161 ceph-mon[91579]: pgmap v8600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:47 smithi188 ceph-mon[92425]: pgmap v8600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:49 smithi161 ceph-mon[91579]: pgmap v8601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:49 smithi188 ceph-mon[92425]: pgmap v8601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:51 smithi161 ceph-mon[91579]: pgmap v8602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:51 smithi188 ceph-mon[92425]: pgmap v8602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:53 smithi161 ceph-mon[91579]: pgmap v8603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:53 smithi188 ceph-mon[92425]: pgmap v8603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:55 smithi161 ceph-mon[91579]: pgmap v8604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:55 smithi188 ceph-mon[92425]: pgmap v8604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:57 smithi161 ceph-mon[91579]: pgmap v8605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:57 smithi188 ceph-mon[92425]: pgmap v8605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:25:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:25:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:25:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:25:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:25:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:25:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:25:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:25:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:25:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:25:59 smithi161 ceph-mon[91579]: pgmap v8606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:25:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:25:59 smithi188 ceph-mon[92425]: pgmap v8606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:01 smithi161 ceph-mon[91579]: pgmap v8607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:01 smithi188 ceph-mon[92425]: pgmap v8607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:03 smithi161 ceph-mon[91579]: pgmap v8608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:03 smithi188 ceph-mon[92425]: pgmap v8608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:05 smithi161 ceph-mon[91579]: pgmap v8609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:05 smithi188 ceph-mon[92425]: pgmap v8609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:07 smithi161 ceph-mon[91579]: pgmap v8610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:07 smithi188 ceph-mon[92425]: pgmap v8610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:09 smithi161 ceph-mon[91579]: pgmap v8611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:09 smithi188 ceph-mon[92425]: pgmap v8611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:11 smithi161 ceph-mon[91579]: pgmap v8612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:11 smithi188 ceph-mon[92425]: pgmap v8612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:13 smithi161 ceph-mon[91579]: pgmap v8613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:13 smithi188 ceph-mon[92425]: pgmap v8613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:15 smithi161 ceph-mon[91579]: pgmap v8614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:15 smithi188 ceph-mon[92425]: pgmap v8614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:17 smithi161 ceph-mon[91579]: pgmap v8615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:17 smithi188 ceph-mon[92425]: pgmap v8615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:19 smithi161 ceph-mon[91579]: pgmap v8616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:19 smithi188 ceph-mon[92425]: pgmap v8616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:21 smithi161 ceph-mon[91579]: pgmap v8617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:21 smithi188 ceph-mon[92425]: pgmap v8617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:23 smithi161 ceph-mon[91579]: pgmap v8618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:23 smithi188 ceph-mon[92425]: pgmap v8618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:25 smithi161 ceph-mon[91579]: pgmap v8619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:25 smithi188 ceph-mon[92425]: pgmap v8619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:27.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:26:27.066+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:26:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:26:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:26:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:26:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:26:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:26:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:26:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:26:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:26:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:26:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:26:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:26:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:26:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:26:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:26:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:26:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:26:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:26:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:26:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:26:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:26:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:26:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:26:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:26:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:26:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:26:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:26:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:26:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:26:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:26:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:26:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:26:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:26:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:26:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:26:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:26:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:26:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:26:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:26:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:26:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:26:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:26:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:26:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:26:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:26:27.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=477] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:26:27.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=477] Local address: 172.21.15.161, port 55728 2022-07-03T15:26:27.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=477] Peer address: 172.21.15.101, port 22 2022-07-03T15:26:27.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=477] Connection lost 2022-07-03T15:26:27.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=477] Aborting connection 2022-07-03T15:26:27.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:26:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-mon[91579]: pgmap v8620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:26:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:26:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:26:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:27 smithi188 ceph-mon[92425]: pgmap v8620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:26:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:26:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:26:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:26:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:26:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:26:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:26:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:26:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:26:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:26:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:26:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:26:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:26:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:26:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:26:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:26:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:26:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:26:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:26:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:26:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:26:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:26:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:26:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:26:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:26:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:26:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:26:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:26:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:26:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:26:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:26:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:26:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:26:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:26:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:26:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:26:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:26:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:26:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:26:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:26:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:26:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:26:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:26:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:26:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:26:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:26:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:26:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:26:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:26:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=477] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:26:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=477] Local address: 172.21.15.161, port 55728 2022-07-03T15:26:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=477] Peer address: 172.21.15.101, port 22 2022-07-03T15:26:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=477] Connection lost 2022-07-03T15:26:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=477] Aborting connection 2022-07-03T15:26:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:26:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:26:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:26:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:26:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:26:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:26:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:26:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:26:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:26:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:26:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:26:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:26:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:26:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:26:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:26:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:26:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:26:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:26:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:26:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:26:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:26:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:26:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:26:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:26:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:26:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:26:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:26:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:26:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:26:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:26:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:26:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:26:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:26:28.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:26:28.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:26:28.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:26:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:26:28.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:26:28.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:26:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:26:28.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:26:28.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:26:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:26:28.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:26:28.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:26:28.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:26:28.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=477] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:26:28.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=477] Local address: 172.21.15.161, port 55728 2022-07-03T15:26:28.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=477] Peer address: 172.21.15.101, port 22 2022-07-03T15:26:28.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=477] Connection lost 2022-07-03T15:26:28.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=477] Aborting connection 2022-07-03T15:26:29.835 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:29 smithi188 ceph-mon[92425]: pgmap v8621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:29.844 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:29 smithi161 ceph-mon[91579]: pgmap v8621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: pgmap v8622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:26:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:26:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 171.4M 2022-07-03T15:26:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:30 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 179733537: error parsing value: Value '179733537' is below minimum 939524096 2022-07-03T15:26:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:26:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: pgmap v8622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:26:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:26:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:26:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 171.4M 2022-07-03T15:26:31.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:30 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 179733537: error parsing value: Value '179733537' is below minimum 939524096 2022-07-03T15:26:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:32 smithi161 ceph-mon[91579]: pgmap v8623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:32 smithi188 ceph-mon[92425]: pgmap v8623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:34 smithi161 ceph-mon[91579]: pgmap v8624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:34 smithi188 ceph-mon[92425]: pgmap v8624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:37 smithi161 ceph-mon[91579]: pgmap v8625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:37 smithi188 ceph-mon[92425]: pgmap v8625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:39 smithi161 ceph-mon[91579]: pgmap v8626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:39 smithi188 ceph-mon[92425]: pgmap v8626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:41 smithi161 ceph-mon[91579]: pgmap v8627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:41 smithi188 ceph-mon[92425]: pgmap v8627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:43 smithi161 ceph-mon[91579]: pgmap v8628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:43 smithi188 ceph-mon[92425]: pgmap v8628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:45 smithi161 ceph-mon[91579]: pgmap v8629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:45 smithi188 ceph-mon[92425]: pgmap v8629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:47 smithi161 ceph-mon[91579]: pgmap v8630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:47 smithi188 ceph-mon[92425]: pgmap v8630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:49 smithi161 ceph-mon[91579]: pgmap v8631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:49 smithi188 ceph-mon[92425]: pgmap v8631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:51 smithi161 ceph-mon[91579]: pgmap v8632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:51 smithi188 ceph-mon[92425]: pgmap v8632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:53 smithi161 ceph-mon[91579]: pgmap v8633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:53 smithi188 ceph-mon[92425]: pgmap v8633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:55 smithi161 ceph-mon[91579]: pgmap v8634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:55 smithi188 ceph-mon[92425]: pgmap v8634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:57 smithi161 ceph-mon[91579]: pgmap v8635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:57 smithi188 ceph-mon[92425]: pgmap v8635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:26:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:26:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:26:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:26:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:26:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:26:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:26:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:26:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:26:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:26:59 smithi161 ceph-mon[91579]: pgmap v8636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:26:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:26:59 smithi188 ceph-mon[92425]: pgmap v8636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:01 smithi161 ceph-mon[91579]: pgmap v8637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:01 smithi188 ceph-mon[92425]: pgmap v8637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:03 smithi161 ceph-mon[91579]: pgmap v8638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:03 smithi188 ceph-mon[92425]: pgmap v8638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:04 smithi161 ceph-mon[91579]: pgmap v8639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:04 smithi188 ceph-mon[92425]: pgmap v8639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:07 smithi161 ceph-mon[91579]: pgmap v8640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:07 smithi188 ceph-mon[92425]: pgmap v8640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:09 smithi161 ceph-mon[91579]: pgmap v8641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:09 smithi188 ceph-mon[92425]: pgmap v8641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:11 smithi161 ceph-mon[91579]: pgmap v8642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:11 smithi188 ceph-mon[92425]: pgmap v8642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:13 smithi161 ceph-mon[91579]: pgmap v8643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:13 smithi188 ceph-mon[92425]: pgmap v8643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:15 smithi161 ceph-mon[91579]: pgmap v8644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:15 smithi188 ceph-mon[92425]: pgmap v8644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:17 smithi161 ceph-mon[91579]: pgmap v8645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:17 smithi188 ceph-mon[92425]: pgmap v8645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:19 smithi161 ceph-mon[91579]: pgmap v8646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:19 smithi188 ceph-mon[92425]: pgmap v8646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:21 smithi161 ceph-mon[91579]: pgmap v8647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:21 smithi188 ceph-mon[92425]: pgmap v8647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:23 smithi161 ceph-mon[91579]: pgmap v8648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:23 smithi188 ceph-mon[92425]: pgmap v8648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:25 smithi161 ceph-mon[91579]: pgmap v8649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:25 smithi188 ceph-mon[92425]: pgmap v8649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:27 smithi161 ceph-mon[91579]: pgmap v8650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:27 smithi188 ceph-mon[92425]: pgmap v8650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:29 smithi161 ceph-mon[91579]: pgmap v8651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:29 smithi188 ceph-mon[92425]: pgmap v8651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:27:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:27:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:27:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:27:30.323+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:27:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:27:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:27:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:27:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:27:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:27:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:27:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:27:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:27:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:27:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:27:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:27:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:27:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:27:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:27:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:27:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:27:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:27:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:27:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:27:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:27:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:27:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:27:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:27:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:27:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:27:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:27:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:27:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:27:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:27:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:27:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:27:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:27:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:27:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:27:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:27:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:27:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:27:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:27:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:27:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:27:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:27:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:27:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:27:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:27:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=479] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:27:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=479] Local address: 172.21.15.161, port 55732 2022-07-03T15:27:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=479] Peer address: 172.21.15.101, port 22 2022-07-03T15:27:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=479] Connection lost 2022-07-03T15:27:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=479] Aborting connection 2022-07-03T15:27:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:27:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:27:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:27:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:27:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:27:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:31 smithi161 ceph-mon[91579]: pgmap v8652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:27:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:27:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:27:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:27:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:27:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:27:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:27:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:27:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:27:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:27:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:27:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:27:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:27:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:27:31.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:27:31.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:27:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:27:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:27:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:27:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:27:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:27:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:27:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:27:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:27:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:27:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:27:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:27:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:27:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:27:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:27:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:27:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:27:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:27:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:27:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:27:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:27:31.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:27:31.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:27:31.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:27:31.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:27:31.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:27:31.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:27:31.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:27:31.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:27:31.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:27:31.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:27:31.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=479] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:27:31.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=479] Local address: 172.21.15.161, port 55732 2022-07-03T15:27:31.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=479] Peer address: 172.21.15.101, port 22 2022-07-03T15:27:31.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=479] Connection lost 2022-07-03T15:27:31.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=479] Aborting connection 2022-07-03T15:27:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:31 smithi188 ceph-mon[92425]: pgmap v8652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:27:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:27:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:27:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:27:31.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:27:31.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:27:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:27:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:27:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:27:31.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:27:31.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:27:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:27:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:27:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:27:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:27:31.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:27:31.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:27:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:27:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:27:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:27:31.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:27:31.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:27:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:27:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:27:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:27:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:27:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:27:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:27:31.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:27:31.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:27:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:27:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:27:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:27:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:27:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:27:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:27:31.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:27:31.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:27:31.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:27:31.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:27:31.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:27:31.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:27:31.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:27:31.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:27:31.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:27:31.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:27:31.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=479] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:27:31.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=479] Local address: 172.21.15.161, port 55732 2022-07-03T15:27:31.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=479] Peer address: 172.21.15.101, port 22 2022-07-03T15:27:31.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=479] Connection lost 2022-07-03T15:27:31.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=479] Aborting connection 2022-07-03T15:27:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:32 smithi161 ceph-mon[91579]: pgmap v8653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:32 smithi188 ceph-mon[92425]: pgmap v8653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:35 smithi161 ceph-mon[91579]: pgmap v8654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:35 smithi188 ceph-mon[92425]: pgmap v8654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:37 smithi161 ceph-mon[91579]: pgmap v8655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:37 smithi188 ceph-mon[92425]: pgmap v8655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:39 smithi161 ceph-mon[91579]: pgmap v8656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:39 smithi188 ceph-mon[92425]: pgmap v8656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:41 smithi161 ceph-mon[91579]: pgmap v8657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:41 smithi188 ceph-mon[92425]: pgmap v8657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:43 smithi161 ceph-mon[91579]: pgmap v8658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:43 smithi188 ceph-mon[92425]: pgmap v8658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:45 smithi161 ceph-mon[91579]: pgmap v8659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:45 smithi188 ceph-mon[92425]: pgmap v8659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:47 smithi161 ceph-mon[91579]: pgmap v8660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:47 smithi188 ceph-mon[92425]: pgmap v8660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:49 smithi161 ceph-mon[91579]: pgmap v8661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:49 smithi188 ceph-mon[92425]: pgmap v8661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:51 smithi161 ceph-mon[91579]: pgmap v8662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:51 smithi188 ceph-mon[92425]: pgmap v8662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:53 smithi161 ceph-mon[91579]: pgmap v8663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:53 smithi188 ceph-mon[92425]: pgmap v8663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:55 smithi161 ceph-mon[91579]: pgmap v8664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:55 smithi188 ceph-mon[92425]: pgmap v8664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:57 smithi161 ceph-mon[91579]: pgmap v8665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:57 smithi188 ceph-mon[92425]: pgmap v8665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:27:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:27:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:27:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:27:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:27:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:27:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:27:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:27:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:27:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:27:59 smithi161 ceph-mon[91579]: pgmap v8666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:27:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:27:59 smithi188 ceph-mon[92425]: pgmap v8666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:01 smithi161 ceph-mon[91579]: pgmap v8667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:01 smithi188 ceph-mon[92425]: pgmap v8667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:03 smithi161 ceph-mon[91579]: pgmap v8668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:03 smithi188 ceph-mon[92425]: pgmap v8668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:05 smithi161 ceph-mon[91579]: pgmap v8669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:05 smithi188 ceph-mon[92425]: pgmap v8669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:07 smithi161 ceph-mon[91579]: pgmap v8670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:07 smithi188 ceph-mon[92425]: pgmap v8670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:09 smithi161 ceph-mon[91579]: pgmap v8671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:09 smithi188 ceph-mon[92425]: pgmap v8671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:11 smithi161 ceph-mon[91579]: pgmap v8672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:11 smithi188 ceph-mon[92425]: pgmap v8672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:13 smithi161 ceph-mon[91579]: pgmap v8673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:13 smithi188 ceph-mon[92425]: pgmap v8673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:15 smithi161 ceph-mon[91579]: pgmap v8674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:15 smithi188 ceph-mon[92425]: pgmap v8674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:17 smithi161 ceph-mon[91579]: pgmap v8675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:17 smithi188 ceph-mon[92425]: pgmap v8675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:19 smithi161 ceph-mon[91579]: pgmap v8676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:19 smithi188 ceph-mon[92425]: pgmap v8676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:21 smithi161 ceph-mon[91579]: pgmap v8677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:21 smithi188 ceph-mon[92425]: pgmap v8677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:22 smithi161 ceph-mon[91579]: pgmap v8678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:22 smithi188 ceph-mon[92425]: pgmap v8678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:25 smithi161 ceph-mon[91579]: pgmap v8679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:25 smithi188 ceph-mon[92425]: pgmap v8679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:27 smithi161 ceph-mon[91579]: pgmap v8680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:27 smithi188 ceph-mon[92425]: pgmap v8680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:29 smithi161 ceph-mon[91579]: pgmap v8681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:29 smithi188 ceph-mon[92425]: pgmap v8681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:31.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:28:30.825+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:28:31.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:28:31.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:28:31.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:28:31.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:28:31.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:28:31.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:28:31.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:28:31.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:28:31.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:28:31.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:28:31.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:28:31.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:28:31.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:28:31.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:28:31.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:28:31.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:28:31.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:28:31.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:28:31.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:28:31.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:28:31.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:28:31.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:28:31.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:28:31.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:28:31.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:28:31.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:28:31.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:28:31.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:28:31.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:28:31.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:28:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:28:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:28:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:28:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:28:31.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:28:31.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:28:31.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:28:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:28:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:28:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:28:31.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:28:31.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:28:31.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:28:31.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:28:31.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=481] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:28:31.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=481] Local address: 172.21.15.161, port 55736 2022-07-03T15:28:31.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=481] Peer address: 172.21.15.101, port 22 2022-07-03T15:28:31.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=481] Connection lost 2022-07-03T15:28:31.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=481] Aborting connection 2022-07-03T15:28:31.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:28:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:31 smithi161 ceph-mon[91579]: pgmap v8682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:28:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:28:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:28:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:31 smithi188 ceph-mon[92425]: pgmap v8682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:28:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:28:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:28:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:28:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:28:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:28:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:28:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:28:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:28:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:28:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:28:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:28:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:28:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:28:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:28:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:28:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:28:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:28:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:28:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:28:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:28:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:28:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:28:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:28:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:28:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:28:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:28:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:28:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:28:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:28:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:28:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:28:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:28:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:28:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:28:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:28:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:28:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:28:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:28:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:28:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:28:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:28:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:28:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:28:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:28:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:28:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:28:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:28:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:28:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=481] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:28:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=481] Local address: 172.21.15.161, port 55736 2022-07-03T15:28:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=481] Peer address: 172.21.15.101, port 22 2022-07-03T15:28:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=481] Connection lost 2022-07-03T15:28:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=481] Aborting connection 2022-07-03T15:28:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:28:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:28:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:28:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:28:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:28:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:28:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:28:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:28:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:28:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:28:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:28:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:28:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:28:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:28:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:28:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:28:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:28:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:28:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:28:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:28:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:28:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:28:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:28:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:28:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:28:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:28:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:28:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:28:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:28:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:28:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:28:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:28:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:28:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:28:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:28:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:28:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:28:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:28:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:28:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:28:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:28:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:28:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:28:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:28:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:28:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:28:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=481] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:28:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=481] Local address: 172.21.15.161, port 55736 2022-07-03T15:28:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=481] Peer address: 172.21.15.101, port 22 2022-07-03T15:28:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=481] Connection lost 2022-07-03T15:28:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=481] Aborting connection 2022-07-03T15:28:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:33 smithi161 ceph-mon[91579]: pgmap v8683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:33 smithi188 ceph-mon[92425]: pgmap v8683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:35 smithi161 ceph-mon[91579]: pgmap v8684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:35 smithi188 ceph-mon[92425]: pgmap v8684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:37 smithi161 ceph-mon[91579]: pgmap v8685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:37 smithi188 ceph-mon[92425]: pgmap v8685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:39 smithi161 ceph-mon[91579]: pgmap v8686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:39 smithi188 ceph-mon[92425]: pgmap v8686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:41 smithi161 ceph-mon[91579]: pgmap v8687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:41 smithi188 ceph-mon[92425]: pgmap v8687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:43 smithi161 ceph-mon[91579]: pgmap v8688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:43 smithi188 ceph-mon[92425]: pgmap v8688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:45 smithi161 ceph-mon[91579]: pgmap v8689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:45 smithi188 ceph-mon[92425]: pgmap v8689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:47 smithi161 ceph-mon[91579]: pgmap v8690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:47 smithi188 ceph-mon[92425]: pgmap v8690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:48 smithi161 ceph-mon[91579]: pgmap v8691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:48 smithi188 ceph-mon[92425]: pgmap v8691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:51 smithi161 ceph-mon[91579]: pgmap v8692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:51 smithi188 ceph-mon[92425]: pgmap v8692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:53 smithi161 ceph-mon[91579]: pgmap v8693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:53 smithi188 ceph-mon[92425]: pgmap v8693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:55 smithi161 ceph-mon[91579]: pgmap v8694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:55 smithi188 ceph-mon[92425]: pgmap v8694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:57 smithi161 ceph-mon[91579]: pgmap v8695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:57 smithi188 ceph-mon[92425]: pgmap v8695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:28:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:28:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:28:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:28:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:28:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:28:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:28:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:28:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:28:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:28:59 smithi161 ceph-mon[91579]: pgmap v8696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:28:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:28:59 smithi188 ceph-mon[92425]: pgmap v8696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:01 smithi161 ceph-mon[91579]: pgmap v8697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:01 smithi188 ceph-mon[92425]: pgmap v8697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:03 smithi161 ceph-mon[91579]: pgmap v8698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:03 smithi188 ceph-mon[92425]: pgmap v8698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:05 smithi161 ceph-mon[91579]: pgmap v8699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:05 smithi188 ceph-mon[92425]: pgmap v8699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:07 smithi161 ceph-mon[91579]: pgmap v8700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:07 smithi188 ceph-mon[92425]: pgmap v8700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:09 smithi161 ceph-mon[91579]: pgmap v8701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:09 smithi188 ceph-mon[92425]: pgmap v8701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:11 smithi161 ceph-mon[91579]: pgmap v8702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:11 smithi188 ceph-mon[92425]: pgmap v8702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:12 smithi161 ceph-mon[91579]: pgmap v8703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:12 smithi188 ceph-mon[92425]: pgmap v8703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:15 smithi161 ceph-mon[91579]: pgmap v8704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:15 smithi188 ceph-mon[92425]: pgmap v8704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:17 smithi161 ceph-mon[91579]: pgmap v8705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:17 smithi188 ceph-mon[92425]: pgmap v8705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:19 smithi161 ceph-mon[91579]: pgmap v8706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:19 smithi188 ceph-mon[92425]: pgmap v8706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:21 smithi161 ceph-mon[91579]: pgmap v8707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:21 smithi188 ceph-mon[92425]: pgmap v8707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:23 smithi161 ceph-mon[91579]: pgmap v8708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:23 smithi188 ceph-mon[92425]: pgmap v8708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:25 smithi161 ceph-mon[91579]: pgmap v8709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:25 smithi188 ceph-mon[92425]: pgmap v8709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:27 smithi161 ceph-mon[91579]: pgmap v8710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:27 smithi188 ceph-mon[92425]: pgmap v8710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:29 smithi161 ceph-mon[91579]: pgmap v8711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:29 smithi188 ceph-mon[92425]: pgmap v8711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-mon[91579]: pgmap v8712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:29:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:29:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:29:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:29:31.323+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:29:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:29:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:29:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:29:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:29:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:29:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:29:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:29:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:29:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:29:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:29:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:29:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:29:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:29:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:29:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:29:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:29:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:29:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:29:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:29:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:29:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:29:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:29:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:29:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:29:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:29:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:29:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:29:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:29:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:29:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:29:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:29:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:29:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:29:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:29:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:29:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:29:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:29:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:29:31.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:29:31.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:29:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:29:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:29:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:29:31.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:29:31.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=483] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:29:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=483] Local address: 172.21.15.161, port 55740 2022-07-03T15:29:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=483] Peer address: 172.21.15.101, port 22 2022-07-03T15:29:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=483] Connection lost 2022-07-03T15:29:31.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=483] Aborting connection 2022-07-03T15:29:31.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:29:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:29:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:31 smithi188 ceph-mon[92425]: pgmap v8712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:29:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:29:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:29:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:29:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:29:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:29:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:29:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:29:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:29:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:29:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:29:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:29:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:29:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:29:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:29:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:29:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:29:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:29:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:29:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:29:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:29:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:29:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:29:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:29:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:29:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:29:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:29:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:29:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:29:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:29:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:29:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:29:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:29:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:29:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:29:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:29:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:29:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:29:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:29:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:29:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:29:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:29:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:29:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:29:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:29:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:29:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:29:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:29:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:29:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=483] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:29:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=483] Local address: 172.21.15.161, port 55740 2022-07-03T15:29:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=483] Peer address: 172.21.15.101, port 22 2022-07-03T15:29:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=483] Connection lost 2022-07-03T15:29:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=483] Aborting connection 2022-07-03T15:29:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:29:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:29:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:29:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:29:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:29:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:29:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:29:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:29:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:29:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:29:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:29:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:29:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:29:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:29:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:29:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:29:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:29:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:29:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:29:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:29:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:29:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:29:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:29:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:29:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:29:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:29:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:29:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:29:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:29:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:29:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:29:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:29:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:29:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:29:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:29:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:29:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:29:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:29:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:29:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:29:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:29:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:29:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:29:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:29:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:29:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:29:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=483] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:29:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=483] Local address: 172.21.15.161, port 55740 2022-07-03T15:29:32.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=483] Peer address: 172.21.15.101, port 22 2022-07-03T15:29:32.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=483] Connection lost 2022-07-03T15:29:32.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=483] Aborting connection 2022-07-03T15:29:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:33 smithi161 ceph-mon[91579]: pgmap v8713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:33 smithi188 ceph-mon[92425]: pgmap v8713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:35 smithi161 ceph-mon[91579]: pgmap v8714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:35 smithi188 ceph-mon[92425]: pgmap v8714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:37 smithi161 ceph-mon[91579]: pgmap v8715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:37 smithi188 ceph-mon[92425]: pgmap v8715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:38 smithi161 ceph-mon[91579]: pgmap v8716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:38 smithi188 ceph-mon[92425]: pgmap v8716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:41 smithi161 ceph-mon[91579]: pgmap v8717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:41 smithi188 ceph-mon[92425]: pgmap v8717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:43 smithi161 ceph-mon[91579]: pgmap v8718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:43 smithi188 ceph-mon[92425]: pgmap v8718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:45 smithi161 ceph-mon[91579]: pgmap v8719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:45 smithi188 ceph-mon[92425]: pgmap v8719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:47 smithi161 ceph-mon[91579]: pgmap v8720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:47 smithi188 ceph-mon[92425]: pgmap v8720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:49 smithi161 ceph-mon[91579]: pgmap v8721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:49 smithi188 ceph-mon[92425]: pgmap v8721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:51 smithi161 ceph-mon[91579]: pgmap v8722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:51 smithi188 ceph-mon[92425]: pgmap v8722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:53 smithi161 ceph-mon[91579]: pgmap v8723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:53 smithi188 ceph-mon[92425]: pgmap v8723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:55 smithi161 ceph-mon[91579]: pgmap v8724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:55 smithi188 ceph-mon[92425]: pgmap v8724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:57 smithi161 ceph-mon[91579]: pgmap v8725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:57 smithi188 ceph-mon[92425]: pgmap v8725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:29:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:29:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:29:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:29:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:29:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:29:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:29:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:29:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:29:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:29:59 smithi161 ceph-mon[91579]: pgmap v8726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:29:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:29:59 smithi188 ceph-mon[92425]: pgmap v8726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:00 smithi161 ceph-mon[91579]: pgmap v8727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:30:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:00 smithi188 ceph-mon[92425]: pgmap v8727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:30:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:03 smithi161 ceph-mon[91579]: pgmap v8728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:03 smithi188 ceph-mon[92425]: pgmap v8728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:05 smithi161 ceph-mon[91579]: pgmap v8729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:05 smithi188 ceph-mon[92425]: pgmap v8729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:07 smithi161 ceph-mon[91579]: pgmap v8730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:07 smithi188 ceph-mon[92425]: pgmap v8730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:09 smithi161 ceph-mon[91579]: pgmap v8731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:09 smithi188 ceph-mon[92425]: pgmap v8731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:11 smithi161 ceph-mon[91579]: pgmap v8732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:11 smithi188 ceph-mon[92425]: pgmap v8732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:13 smithi161 ceph-mon[91579]: pgmap v8733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:13 smithi188 ceph-mon[92425]: pgmap v8733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:15 smithi161 ceph-mon[91579]: pgmap v8734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:15 smithi188 ceph-mon[92425]: pgmap v8734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:17 smithi161 ceph-mon[91579]: pgmap v8735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:17 smithi188 ceph-mon[92425]: pgmap v8735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:19 smithi161 ceph-mon[91579]: pgmap v8736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:19 smithi188 ceph-mon[92425]: pgmap v8736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:21 smithi161 ceph-mon[91579]: pgmap v8737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:21 smithi188 ceph-mon[92425]: pgmap v8737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:23 smithi161 ceph-mon[91579]: pgmap v8738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:23 smithi188 ceph-mon[92425]: pgmap v8738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:25 smithi161 ceph-mon[91579]: pgmap v8739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:25 smithi188 ceph-mon[92425]: pgmap v8739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:26 smithi161 ceph-mon[91579]: pgmap v8740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:26 smithi188 ceph-mon[92425]: pgmap v8740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:29 smithi161 ceph-mon[91579]: pgmap v8741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:29 smithi188 ceph-mon[92425]: pgmap v8741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:31.830 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-mon[91579]: pgmap v8742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:31.852 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:31 smithi188 ceph-mon[92425]: pgmap v8742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:32.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:30:31.828+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:30:32.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:30:32.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:30:32.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:30:32.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:30:32.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:30:32.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:30:32.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:30:32.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:30:32.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:30:32.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:30:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:30:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:30:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:30:32.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:30:32.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:30:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:30:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:30:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:30:32.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:30:32.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:30:32.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:30:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:30:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:30:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:30:32.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:30:32.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:30:32.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:30:32.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:30:32.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:30:32.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:30:32.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:30:32.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:30:32.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:30:32.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:30:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:30:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:30:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:30:32.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:30:32.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:30:32.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:30:32.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:30:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:30:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:30:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:30:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=485] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:30:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=485] Local address: 172.21.15.161, port 55744 2022-07-03T15:30:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=485] Peer address: 172.21.15.101, port 22 2022-07-03T15:30:32.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=485] Connection lost 2022-07-03T15:30:32.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=485] Aborting connection 2022-07-03T15:30:32.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:30:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:30:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:30:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:30:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:30:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:30:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:30:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:30:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:30:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:30:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:30:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:30:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:30:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:30:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:30:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:30:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:30:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:30:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:30:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:30:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:30:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:30:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:30:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:30:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:30:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:30:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:30:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:30:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:30:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:30:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:30:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:30:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:30:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:30:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:30:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:30:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:30:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:30:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:30:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:30:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:30:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:30:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:30:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:30:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:30:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:30:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:30:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:30:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:30:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:30:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:30:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:30:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:30:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=485] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:30:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=485] Local address: 172.21.15.161, port 55744 2022-07-03T15:30:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=485] Peer address: 172.21.15.101, port 22 2022-07-03T15:30:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=485] Connection lost 2022-07-03T15:30:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=485] Aborting connection 2022-07-03T15:30:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:33 smithi161 ceph-mon[91579]: pgmap v8743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:30:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:30:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:30:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:30:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:30:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:30:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:30:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:30:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:30:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:30:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:30:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:30:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:30:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:30:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:30:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:30:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:30:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:30:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:30:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:30:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:30:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:30:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:30:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:30:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:30:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:30:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:30:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:30:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:30:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:30:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:30:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:30:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:30:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:30:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:30:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:30:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:30:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:30:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:30:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:30:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:30:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:30:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:30:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:30:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:30:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:30:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=485] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:30:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=485] Local address: 172.21.15.161, port 55744 2022-07-03T15:30:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=485] Peer address: 172.21.15.101, port 22 2022-07-03T15:30:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=485] Connection lost 2022-07-03T15:30:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=485] Aborting connection 2022-07-03T15:30:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:33 smithi188 ceph-mon[92425]: pgmap v8743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:35 smithi161 ceph-mon[91579]: pgmap v8744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:35 smithi188 ceph-mon[92425]: pgmap v8744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:37 smithi161 ceph-mon[91579]: pgmap v8745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:37 smithi188 ceph-mon[92425]: pgmap v8745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:39 smithi161 ceph-mon[91579]: pgmap v8746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:39 smithi188 ceph-mon[92425]: pgmap v8746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:41 smithi161 ceph-mon[91579]: pgmap v8747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:41 smithi188 ceph-mon[92425]: pgmap v8747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:43 smithi161 ceph-mon[91579]: pgmap v8748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:43 smithi188 ceph-mon[92425]: pgmap v8748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:45 smithi161 ceph-mon[91579]: pgmap v8749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:45 smithi188 ceph-mon[92425]: pgmap v8749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:47 smithi161 ceph-mon[91579]: pgmap v8750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:47 smithi188 ceph-mon[92425]: pgmap v8750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:48 smithi161 ceph-mon[91579]: pgmap v8751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:48 smithi188 ceph-mon[92425]: pgmap v8751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:51 smithi161 ceph-mon[91579]: pgmap v8752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:51 smithi188 ceph-mon[92425]: pgmap v8752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:53 smithi161 ceph-mon[91579]: pgmap v8753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:53 smithi188 ceph-mon[92425]: pgmap v8753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:55 smithi161 ceph-mon[91579]: pgmap v8754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:55 smithi188 ceph-mon[92425]: pgmap v8754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:57 smithi161 ceph-mon[91579]: pgmap v8755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:57 smithi188 ceph-mon[92425]: pgmap v8755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:30:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:30:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:30:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:30:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:30:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:30:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:30:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:30:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:30:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:30:59 smithi161 ceph-mon[91579]: pgmap v8756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:30:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:30:59 smithi188 ceph-mon[92425]: pgmap v8756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:01 smithi161 ceph-mon[91579]: pgmap v8757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:01 smithi188 ceph-mon[92425]: pgmap v8757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:03 smithi161 ceph-mon[91579]: pgmap v8758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:03 smithi188 ceph-mon[92425]: pgmap v8758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:05 smithi161 ceph-mon[91579]: pgmap v8759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:05 smithi188 ceph-mon[92425]: pgmap v8759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:06 smithi161 ceph-mon[91579]: pgmap v8760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:06 smithi188 ceph-mon[92425]: pgmap v8760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:09 smithi161 ceph-mon[91579]: pgmap v8761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:09 smithi188 ceph-mon[92425]: pgmap v8761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:11 smithi161 ceph-mon[91579]: pgmap v8762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:11 smithi188 ceph-mon[92425]: pgmap v8762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:13 smithi161 ceph-mon[91579]: pgmap v8763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:13 smithi188 ceph-mon[92425]: pgmap v8763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:15 smithi161 ceph-mon[91579]: pgmap v8764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:15 smithi188 ceph-mon[92425]: pgmap v8764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:17 smithi161 ceph-mon[91579]: pgmap v8765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:17 smithi188 ceph-mon[92425]: pgmap v8765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:19 smithi161 ceph-mon[91579]: pgmap v8766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:19 smithi188 ceph-mon[92425]: pgmap v8766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:21 smithi161 ceph-mon[91579]: pgmap v8767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:21 smithi188 ceph-mon[92425]: pgmap v8767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:23 smithi161 ceph-mon[91579]: pgmap v8768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:23 smithi188 ceph-mon[92425]: pgmap v8768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:25 smithi161 ceph-mon[91579]: pgmap v8769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:25 smithi188 ceph-mon[92425]: pgmap v8769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:27 smithi161 ceph-mon[91579]: pgmap v8770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:27 smithi188 ceph-mon[92425]: pgmap v8770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:31:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:28 smithi188 ceph-mon[92425]: pgmap v8771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:28 smithi161 ceph-mon[91579]: pgmap v8771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:31 smithi161 ceph-mon[91579]: pgmap v8772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:31 smithi188 ceph-mon[92425]: pgmap v8772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:31:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:31:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:31:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:31:32.337+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:31:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:31:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:31:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:31:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:31:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:31:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:31:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:31:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:31:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:31:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:31:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:31:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:31:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:31:32.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:31:32.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:31:32.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:31:32.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:31:32.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:31:32.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:31:32.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:31:32.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:31:32.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:31:32.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:31:32.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:31:32.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:31:32.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:31:32.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:31:32.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:31:32.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:31:32.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:31:32.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:31:32.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:31:32.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:31:32.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:31:32.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:31:32.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:31:32.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:31:32.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:31:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:31:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:31:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:31:32.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:31:32.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:31:32.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:31:32.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=487] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:31:32.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=487] Local address: 172.21.15.161, port 55748 2022-07-03T15:31:32.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=487] Peer address: 172.21.15.101, port 22 2022-07-03T15:31:32.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=487] Connection lost 2022-07-03T15:31:32.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=487] Aborting connection 2022-07-03T15:31:32.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:31:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:31:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:31:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:31:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:31:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:33 smithi161 ceph-mon[91579]: pgmap v8773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:31:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:31:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:31:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:31:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:31:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:31:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:31:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:31:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:31:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:31:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:31:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:31:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:31:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:31:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:31:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:31:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:31:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:31:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:31:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:31:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:31:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:31:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:31:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:31:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:31:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:31:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:31:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:31:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:31:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:31:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:31:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:31:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:31:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:31:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:31:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:31:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:31:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:31:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:31:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:31:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:31:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:31:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:31:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:31:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=487] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:31:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=487] Local address: 172.21.15.161, port 55748 2022-07-03T15:31:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=487] Peer address: 172.21.15.101, port 22 2022-07-03T15:31:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=487] Connection lost 2022-07-03T15:31:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=487] Aborting connection 2022-07-03T15:31:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:33 smithi188 ceph-mon[92425]: pgmap v8773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:31:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:31:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:31:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:31:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:31:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:31:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:31:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:31:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:31:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:31:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:31:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:31:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:31:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:31:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:31:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:31:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:31:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:31:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:31:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:31:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:31:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:31:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:31:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:31:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:31:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:31:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:31:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:31:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:31:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:31:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:31:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:31:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:31:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:31:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:31:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:31:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:31:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:31:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:31:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:31:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:31:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:31:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:31:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:31:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:31:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:31:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=487] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:31:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=487] Local address: 172.21.15.161, port 55748 2022-07-03T15:31:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=487] Peer address: 172.21.15.101, port 22 2022-07-03T15:31:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=487] Connection lost 2022-07-03T15:31:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=487] Aborting connection 2022-07-03T15:31:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:35 smithi161 ceph-mon[91579]: pgmap v8774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:35 smithi188 ceph-mon[92425]: pgmap v8774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:37 smithi161 ceph-mon[91579]: pgmap v8775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:37 smithi188 ceph-mon[92425]: pgmap v8775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:39 smithi161 ceph-mon[91579]: pgmap v8776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:39 smithi188 ceph-mon[92425]: pgmap v8776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:41 smithi161 ceph-mon[91579]: pgmap v8777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:41 smithi188 ceph-mon[92425]: pgmap v8777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:42 smithi188 ceph-mon[92425]: pgmap v8778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:42 smithi161 ceph-mon[91579]: pgmap v8778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:45 smithi161 ceph-mon[91579]: pgmap v8779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:45 smithi188 ceph-mon[92425]: pgmap v8779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:47 smithi161 ceph-mon[91579]: pgmap v8780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:47 smithi188 ceph-mon[92425]: pgmap v8780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:49 smithi161 ceph-mon[91579]: pgmap v8781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:49 smithi188 ceph-mon[92425]: pgmap v8781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:51 smithi161 ceph-mon[91579]: pgmap v8782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:51 smithi188 ceph-mon[92425]: pgmap v8782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:53 smithi161 ceph-mon[91579]: pgmap v8783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:53 smithi188 ceph-mon[92425]: pgmap v8783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:55 smithi161 ceph-mon[91579]: pgmap v8784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:55 smithi188 ceph-mon[92425]: pgmap v8784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:57 smithi161 ceph-mon[91579]: pgmap v8785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:57 smithi188 ceph-mon[92425]: pgmap v8785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:31:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:31:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:31:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:31:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:31:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:31:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:31:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:31:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:31:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:31:59 smithi161 ceph-mon[91579]: pgmap v8786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:31:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:31:59 smithi188 ceph-mon[92425]: pgmap v8786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:01 smithi161 ceph-mon[91579]: pgmap v8787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:01 smithi188 ceph-mon[92425]: pgmap v8787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:03 smithi161 ceph-mon[91579]: pgmap v8788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:03 smithi188 ceph-mon[92425]: pgmap v8788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:05 smithi161 ceph-mon[91579]: pgmap v8789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:05 smithi188 ceph-mon[92425]: pgmap v8789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:07 smithi161 ceph-mon[91579]: pgmap v8790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:07 smithi188 ceph-mon[92425]: pgmap v8790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:09 smithi161 ceph-mon[91579]: pgmap v8791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:09 smithi188 ceph-mon[92425]: pgmap v8791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:11 smithi161 ceph-mon[91579]: pgmap v8792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:11 smithi188 ceph-mon[92425]: pgmap v8792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:13 smithi161 ceph-mon[91579]: pgmap v8793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:13 smithi188 ceph-mon[92425]: pgmap v8793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:14 smithi161 ceph-mon[91579]: pgmap v8794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:14 smithi188 ceph-mon[92425]: pgmap v8794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:17 smithi161 ceph-mon[91579]: pgmap v8795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:17.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:17 smithi188 ceph-mon[92425]: pgmap v8795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:19 smithi161 ceph-mon[91579]: pgmap v8796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:19 smithi188 ceph-mon[92425]: pgmap v8796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:21 smithi161 ceph-mon[91579]: pgmap v8797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:21 smithi188 ceph-mon[92425]: pgmap v8797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:23 smithi161 ceph-mon[91579]: pgmap v8798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:23 smithi188 ceph-mon[92425]: pgmap v8798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:25 smithi161 ceph-mon[91579]: pgmap v8799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:25 smithi188 ceph-mon[92425]: pgmap v8799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:27 smithi161 ceph-mon[91579]: pgmap v8800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:27 smithi188 ceph-mon[92425]: pgmap v8800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:29 smithi161 ceph-mon[91579]: pgmap v8801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:29 smithi188 ceph-mon[92425]: pgmap v8801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:31 smithi161 ceph-mon[91579]: pgmap v8802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:31 smithi188 ceph-mon[92425]: pgmap v8802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:33.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:32:32.845+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:32:33.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:32:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:32:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:32:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:32:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:32:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:32:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:32:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:32:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:32:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:32:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:32:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:32:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:32:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:32:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:32:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:32:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:32:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:32:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:32:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:32:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:32:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:32:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:32:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:32:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:32:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:32:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:32:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:32:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:32:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:32:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:32:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:32:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:32:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:32:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:32:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:32:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:32:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:32:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:32:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:32:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:32:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:32:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:32:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:32:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=489] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:32:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=489] Local address: 172.21.15.161, port 55752 2022-07-03T15:32:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=489] Peer address: 172.21.15.101, port 22 2022-07-03T15:32:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=489] Connection lost 2022-07-03T15:32:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=489] Aborting connection 2022-07-03T15:32:33.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:32:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:33 smithi161 ceph-mon[91579]: pgmap v8803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:32:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:32:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:32:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:33 smithi188 ceph-mon[92425]: pgmap v8803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:32:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:32:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:32:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:32:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:32:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:32:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:32:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:32:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:32:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:32:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:32:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:32:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:32:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:32:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:32:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:32:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:32:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:32:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:32:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:32:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:32:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:32:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:32:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:32:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:32:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:32:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:32:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:32:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:32:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:32:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:32:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:32:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:32:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:32:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:32:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:32:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:32:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:32:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:32:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:32:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:32:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:32:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:32:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:32:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:32:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:32:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:32:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:32:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:32:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=489] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:32:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=489] Local address: 172.21.15.161, port 55752 2022-07-03T15:32:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=489] Peer address: 172.21.15.101, port 22 2022-07-03T15:32:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=489] Connection lost 2022-07-03T15:32:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=489] Aborting connection 2022-07-03T15:32:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:32:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:32:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:32:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:32:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:32:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:32:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:32:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:32:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:32:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:32:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:32:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:32:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:32:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:32:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:32:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:32:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:32:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:32:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:32:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:32:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:32:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:32:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:32:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:32:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:32:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:32:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:32:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:32:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:32:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:32:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:32:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:32:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:32:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:32:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:32:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:32:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:32:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:32:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:32:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:32:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:32:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:32:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:32:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:32:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:32:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:32:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=489] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:32:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=489] Local address: 172.21.15.161, port 55752 2022-07-03T15:32:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=489] Peer address: 172.21.15.101, port 22 2022-07-03T15:32:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=489] Connection lost 2022-07-03T15:32:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=489] Aborting connection 2022-07-03T15:32:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:35 smithi161 ceph-mon[91579]: pgmap v8804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:35 smithi188 ceph-mon[92425]: pgmap v8804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:37 smithi161 ceph-mon[91579]: pgmap v8805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:37 smithi188 ceph-mon[92425]: pgmap v8805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:39 smithi161 ceph-mon[91579]: pgmap v8806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:39 smithi188 ceph-mon[92425]: pgmap v8806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:41 smithi161 ceph-mon[91579]: pgmap v8807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:41 smithi188 ceph-mon[92425]: pgmap v8807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:43 smithi161 ceph-mon[91579]: pgmap v8808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:43 smithi188 ceph-mon[92425]: pgmap v8808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:45 smithi161 ceph-mon[91579]: pgmap v8809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:45 smithi188 ceph-mon[92425]: pgmap v8809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:47 smithi161 ceph-mon[91579]: pgmap v8810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:47 smithi188 ceph-mon[92425]: pgmap v8810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:49 smithi161 ceph-mon[91579]: pgmap v8811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:49 smithi188 ceph-mon[92425]: pgmap v8811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:51 smithi161 ceph-mon[91579]: pgmap v8812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:51 smithi188 ceph-mon[92425]: pgmap v8812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:53 smithi161 ceph-mon[91579]: pgmap v8813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:53 smithi188 ceph-mon[92425]: pgmap v8813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:54 smithi161 ceph-mon[91579]: pgmap v8814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:54 smithi188 ceph-mon[92425]: pgmap v8814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:57 smithi161 ceph-mon[91579]: pgmap v8815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:57 smithi188 ceph-mon[92425]: pgmap v8815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:32:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:32:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:32:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:32:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:32:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:32:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:32:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:32:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:32:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:32:59 smithi161 ceph-mon[91579]: pgmap v8816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:32:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:32:59 smithi188 ceph-mon[92425]: pgmap v8816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:01 smithi161 ceph-mon[91579]: pgmap v8817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:01 smithi188 ceph-mon[92425]: pgmap v8817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:03 smithi161 ceph-mon[91579]: pgmap v8818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:03 smithi188 ceph-mon[92425]: pgmap v8818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:04 smithi161 ceph-mon[91579]: pgmap v8819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:04 smithi188 ceph-mon[92425]: pgmap v8819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:07 smithi161 ceph-mon[91579]: pgmap v8820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:07 smithi188 ceph-mon[92425]: pgmap v8820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:09 smithi161 ceph-mon[91579]: pgmap v8821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:09 smithi188 ceph-mon[92425]: pgmap v8821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:11 smithi161 ceph-mon[91579]: pgmap v8822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:11 smithi188 ceph-mon[92425]: pgmap v8822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:13 smithi161 ceph-mon[91579]: pgmap v8823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:13 smithi188 ceph-mon[92425]: pgmap v8823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:15 smithi161 ceph-mon[91579]: pgmap v8824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:15 smithi188 ceph-mon[92425]: pgmap v8824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:17 smithi161 ceph-mon[91579]: pgmap v8825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:17 smithi188 ceph-mon[92425]: pgmap v8825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:18 smithi188 ceph-mon[92425]: pgmap v8826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:18 smithi161 ceph-mon[91579]: pgmap v8826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:21 smithi161 ceph-mon[91579]: pgmap v8827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:21 smithi188 ceph-mon[92425]: pgmap v8827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:23 smithi161 ceph-mon[91579]: pgmap v8828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:23 smithi188 ceph-mon[92425]: pgmap v8828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:25 smithi161 ceph-mon[91579]: pgmap v8829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:25 smithi188 ceph-mon[92425]: pgmap v8829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:27 smithi161 ceph-mon[91579]: pgmap v8830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:27 smithi188 ceph-mon[92425]: pgmap v8830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:29 smithi161 ceph-mon[91579]: pgmap v8831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:29 smithi188 ceph-mon[92425]: pgmap v8831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:31 smithi161 ceph-mon[91579]: pgmap v8832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:31 smithi188 ceph-mon[92425]: pgmap v8832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-mon[91579]: pgmap v8833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:33:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:33:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:33:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:33:33.345+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:33:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:33:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:33:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:33:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:33:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:33:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:33:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:33:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:33:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:33:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:33:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:33:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:33:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:33:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:33:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:33:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:33:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:33:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:33:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:33:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:33:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:33:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:33:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:33:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:33:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:33:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:33:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:33:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:33:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:33:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:33:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:33:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:33:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:33:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:33:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:33:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:33:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:33:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:33:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:33:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:33:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:33:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:33:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:33:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:33:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=491] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:33:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=491] Local address: 172.21.15.161, port 55756 2022-07-03T15:33:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=491] Peer address: 172.21.15.101, port 22 2022-07-03T15:33:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=491] Connection lost 2022-07-03T15:33:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=491] Aborting connection 2022-07-03T15:33:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:33:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:33:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:33 smithi188 ceph-mon[92425]: pgmap v8833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:33:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:33:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:33:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:33:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:33:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:33:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:33:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:33:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:33:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:33:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:33:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:33:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:33:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:33:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:33:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:33:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:33:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:33:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:33:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:33:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:33:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:33:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:33:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:33:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:33:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:33:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:33:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:33:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:33:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:33:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:33:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:33:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:33:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:33:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:33:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:33:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:33:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:33:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:33:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:33:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:33:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:33:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:33:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:33:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:33:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:33:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:33:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:33:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:33:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=491] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:33:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=491] Local address: 172.21.15.161, port 55756 2022-07-03T15:33:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=491] Peer address: 172.21.15.101, port 22 2022-07-03T15:33:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=491] Connection lost 2022-07-03T15:33:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=491] Aborting connection 2022-07-03T15:33:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:33:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:33:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:33:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:33:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:33:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:33:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:33:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:33:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:33:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:33:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:33:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:33:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:33:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:33:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:33:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:33:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:33:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:33:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:33:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:33:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:33:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:33:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:33:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:33:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:33:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:33:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:33:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:33:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:33:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:33:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:33:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:33:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:33:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:33:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:33:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:33:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:33:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:33:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:33:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:33:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:33:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:33:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:33:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:33:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:33:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:33:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=491] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:33:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=491] Local address: 172.21.15.161, port 55756 2022-07-03T15:33:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=491] Peer address: 172.21.15.101, port 22 2022-07-03T15:33:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=491] Connection lost 2022-07-03T15:33:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=491] Aborting connection 2022-07-03T15:33:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:34.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:33:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:35 smithi161 ceph-mon[91579]: pgmap v8834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:35 smithi188 ceph-mon[92425]: pgmap v8834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:37 smithi161 ceph-mon[91579]: pgmap v8835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:37 smithi188 ceph-mon[92425]: pgmap v8835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:39 smithi161 ceph-mon[91579]: pgmap v8836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:39 smithi188 ceph-mon[92425]: pgmap v8836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:41 smithi161 ceph-mon[91579]: pgmap v8837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:41 smithi188 ceph-mon[92425]: pgmap v8837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:42 smithi188 ceph-mon[92425]: pgmap v8838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:42 smithi161 ceph-mon[91579]: pgmap v8838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:45.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:45 smithi161 ceph-mon[91579]: pgmap v8839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:45 smithi188 ceph-mon[92425]: pgmap v8839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:47 smithi161 ceph-mon[91579]: pgmap v8840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:47 smithi188 ceph-mon[92425]: pgmap v8840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:49 smithi161 ceph-mon[91579]: pgmap v8841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:49 smithi188 ceph-mon[92425]: pgmap v8841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:51 smithi161 ceph-mon[91579]: pgmap v8842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:51 smithi188 ceph-mon[92425]: pgmap v8842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:53 smithi161 ceph-mon[91579]: pgmap v8843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:53 smithi188 ceph-mon[92425]: pgmap v8843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:55 smithi161 ceph-mon[91579]: pgmap v8844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:55 smithi188 ceph-mon[92425]: pgmap v8844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:56 smithi188 ceph-mon[92425]: pgmap v8845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:56 smithi161 ceph-mon[91579]: pgmap v8845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:33:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:33:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:33:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:33:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:33:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:33:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:33:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:33:59 smithi161 ceph-mon[91579]: pgmap v8846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:33:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:33:59 smithi188 ceph-mon[92425]: pgmap v8846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:01 smithi161 ceph-mon[91579]: pgmap v8847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:01 smithi188 ceph-mon[92425]: pgmap v8847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:03 smithi161 ceph-mon[91579]: pgmap v8848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:03 smithi188 ceph-mon[92425]: pgmap v8848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:05 smithi161 ceph-mon[91579]: pgmap v8849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:05 smithi188 ceph-mon[92425]: pgmap v8849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:07 smithi161 ceph-mon[91579]: pgmap v8850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:07 smithi188 ceph-mon[92425]: pgmap v8850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:09 smithi161 ceph-mon[91579]: pgmap v8851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:09 smithi188 ceph-mon[92425]: pgmap v8851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:11 smithi161 ceph-mon[91579]: pgmap v8852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:11 smithi188 ceph-mon[92425]: pgmap v8852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:13 smithi161 ceph-mon[91579]: pgmap v8853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:13 smithi188 ceph-mon[92425]: pgmap v8853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:15 smithi161 ceph-mon[91579]: pgmap v8854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:15 smithi188 ceph-mon[92425]: pgmap v8854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:17 smithi161 ceph-mon[91579]: pgmap v8855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:17 smithi188 ceph-mon[92425]: pgmap v8855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:19 smithi161 ceph-mon[91579]: pgmap v8856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:19 smithi188 ceph-mon[92425]: pgmap v8856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:21 smithi161 ceph-mon[91579]: pgmap v8857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:21 smithi188 ceph-mon[92425]: pgmap v8857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:22 smithi161 ceph-mon[91579]: pgmap v8858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:22 smithi188 ceph-mon[92425]: pgmap v8858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:25 smithi188 ceph-mon[92425]: pgmap v8859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:25 smithi161 ceph-mon[91579]: pgmap v8859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:27 smithi161 ceph-mon[91579]: pgmap v8860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:27 smithi188 ceph-mon[92425]: pgmap v8860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:29 smithi161 ceph-mon[91579]: pgmap v8861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:29 smithi188 ceph-mon[92425]: pgmap v8861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:31 smithi161 ceph-mon[91579]: pgmap v8862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:31 smithi188 ceph-mon[92425]: pgmap v8862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:33 smithi161 ceph-mon[91579]: pgmap v8863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:33 smithi188 ceph-mon[92425]: pgmap v8863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:34.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:34:34.252+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:34:34.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:34:34.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:34:34.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:34:34.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:34:34.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:34:34.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:34:34.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:34:34.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:34:34.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:34:34.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:34:34.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:34:34.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:34:34.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:34:34.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:34:34.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:34:34.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:34:34.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:34:34.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:34:34.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:34:34.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:34:34.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:34:34.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:34:34.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:34:34.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:34:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:34:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:34:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:34:34.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:34:34.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:34:34.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:34:34.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:34:34.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:34:34.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:34:34.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:34:34.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:34:34.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:34:34.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:34:34.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:34:34.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:34:34.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:34:34.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:34:34.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:34:34.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:34:34.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:34:34.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=493] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:34:34.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=493] Local address: 172.21.15.161, port 55760 2022-07-03T15:34:34.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=493] Peer address: 172.21.15.101, port 22 2022-07-03T15:34:34.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=493] Connection lost 2022-07-03T15:34:34.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=493] Aborting connection 2022-07-03T15:34:34.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:34:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:34:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:34:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:34:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:34:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:34:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:34:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:35 smithi161 ceph-mon[91579]: pgmap v8864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:34:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:34:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:34:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:34:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:34:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:34:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:34:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:34:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:34:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:34:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:34:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:34:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:34:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:34:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:34:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:34:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:34:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:34:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:34:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:34:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:34:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:34:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:34:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:34:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:34:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:34:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:34:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:34:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:34:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:34:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:34:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:34:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:34:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:34:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:34:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:34:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:34:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:34:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:34:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:34:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:34:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:34:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:34:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:34:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:34:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:34:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=493] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:34:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=493] Local address: 172.21.15.161, port 55760 2022-07-03T15:34:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=493] Peer address: 172.21.15.101, port 22 2022-07-03T15:34:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=493] Connection lost 2022-07-03T15:34:35.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=493] Aborting connection 2022-07-03T15:34:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:35 smithi188 ceph-mon[92425]: pgmap v8864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:34:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:34:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:34:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:34:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:34:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:34:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:34:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:34:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:34:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:34:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:34:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:34:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:34:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:34:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:34:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:34:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:34:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:34:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:34:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:34:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:34:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:34:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:34:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:34:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:34:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:34:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:34:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:34:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:34:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:34:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:34:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:34:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:34:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:34:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:34:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:34:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:34:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:34:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:34:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:34:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:34:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:34:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:34:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:34:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:34:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:34:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=493] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:34:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=493] Local address: 172.21.15.161, port 55760 2022-07-03T15:34:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=493] Peer address: 172.21.15.101, port 22 2022-07-03T15:34:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=493] Connection lost 2022-07-03T15:34:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=493] Aborting connection 2022-07-03T15:34:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:36 smithi161 ceph-mon[91579]: pgmap v8865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:36 smithi188 ceph-mon[92425]: pgmap v8865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:39 smithi188 ceph-mon[92425]: pgmap v8866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:39 smithi161 ceph-mon[91579]: pgmap v8866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:41 smithi161 ceph-mon[91579]: pgmap v8867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:41 smithi188 ceph-mon[92425]: pgmap v8867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:43 smithi161 ceph-mon[91579]: pgmap v8868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:43 smithi188 ceph-mon[92425]: pgmap v8868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:45 smithi161 ceph-mon[91579]: pgmap v8869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:45 smithi188 ceph-mon[92425]: pgmap v8869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:47 smithi161 ceph-mon[91579]: pgmap v8870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:47 smithi188 ceph-mon[92425]: pgmap v8870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:49 smithi161 ceph-mon[91579]: pgmap v8871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:49 smithi188 ceph-mon[92425]: pgmap v8871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:50 smithi161 ceph-mon[91579]: pgmap v8872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:50 smithi188 ceph-mon[92425]: pgmap v8872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:53 smithi188 ceph-mon[92425]: pgmap v8873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:53 smithi161 ceph-mon[91579]: pgmap v8873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:55 smithi188 ceph-mon[92425]: pgmap v8874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:55 smithi161 ceph-mon[91579]: pgmap v8874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:34:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:57 smithi161 ceph-mon[91579]: pgmap v8875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:57 smithi188 ceph-mon[92425]: pgmap v8875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:34:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:34:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:34:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:34:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:34:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:34:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:34:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:34:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:34:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:34:59 smithi161 ceph-mon[91579]: pgmap v8876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:34:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:34:59 smithi188 ceph-mon[92425]: pgmap v8876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:01 smithi161 ceph-mon[91579]: pgmap v8877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:01 smithi188 ceph-mon[92425]: pgmap v8877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:02 smithi188 ceph-mon[92425]: pgmap v8878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:02 smithi161 ceph-mon[91579]: pgmap v8878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:05 smithi188 ceph-mon[92425]: pgmap v8879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:05 smithi161 ceph-mon[91579]: pgmap v8879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:07 smithi161 ceph-mon[91579]: pgmap v8880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:07 smithi188 ceph-mon[92425]: pgmap v8880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:09 smithi161 ceph-mon[91579]: pgmap v8881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:09 smithi188 ceph-mon[92425]: pgmap v8881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:11 smithi161 ceph-mon[91579]: pgmap v8882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:11 smithi188 ceph-mon[92425]: pgmap v8882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:12 smithi188 ceph-mon[92425]: pgmap v8883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:12 smithi161 ceph-mon[91579]: pgmap v8883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:15 smithi188 ceph-mon[92425]: pgmap v8884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:15 smithi161 ceph-mon[91579]: pgmap v8884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:17 smithi161 ceph-mon[91579]: pgmap v8885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:17 smithi188 ceph-mon[92425]: pgmap v8885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:19 smithi161 ceph-mon[91579]: pgmap v8886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:19 smithi188 ceph-mon[92425]: pgmap v8886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:21 smithi161 ceph-mon[91579]: pgmap v8887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:21 smithi188 ceph-mon[92425]: pgmap v8887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:22 smithi188 ceph-mon[92425]: pgmap v8888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:22 smithi161 ceph-mon[91579]: pgmap v8888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:25 smithi188 ceph-mon[92425]: pgmap v8889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:25 smithi161 ceph-mon[91579]: pgmap v8889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:27 smithi161 ceph-mon[91579]: pgmap v8890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:27 smithi188 ceph-mon[92425]: pgmap v8890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:29 smithi161 ceph-mon[91579]: pgmap v8891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:29 smithi188 ceph-mon[92425]: pgmap v8891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:31 smithi161 ceph-mon[91579]: pgmap v8892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:31 smithi188 ceph-mon[92425]: pgmap v8892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:32 smithi188 ceph-mon[92425]: pgmap v8893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:32 smithi161 ceph-mon[91579]: pgmap v8893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:35.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:35:34.749+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:35:35.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:35:35.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:35:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:35:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:35:35.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:35:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:35:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:35:35.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:35:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:35:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:35:35.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:35:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:35:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:35:35.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:35:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:35:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:35:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:35:35.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:35:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:35:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:35:35.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:35:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:35:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:35:35.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:35:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:35:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:35:35.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:35:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:35:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:35:35.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:35:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:35:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:35:35.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:35:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:35:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:35:35.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:35:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:35:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:35:35.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:35:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:35:35.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:35:35.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:35:35.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:35:35.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:35:35.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=495] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:35:35.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=495] Local address: 172.21.15.161, port 55764 2022-07-03T15:35:35.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=495] Peer address: 172.21.15.101, port 22 2022-07-03T15:35:35.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=495] Connection lost 2022-07-03T15:35:35.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=495] Aborting connection 2022-07-03T15:35:35.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:35:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:35 smithi188 ceph-mon[92425]: pgmap v8894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:35:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:35:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:35:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:35 smithi161 ceph-mon[91579]: pgmap v8894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:35:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:35:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:35:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:35:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:35:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:35:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:35:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:35:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:35:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:35:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:35:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:35:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:35:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:35:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:35:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:35:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:35:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:35:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:35:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:35:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:35:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:35:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:35:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:35:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:35:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:35:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:35:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:35:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:35:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:35:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:35:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:35:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:35:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:35:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:35:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:35:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:35:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:35:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:35:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:35:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:35:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:35:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:35:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:35:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:35:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:35:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:35:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:35:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:35:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=495] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:35:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=495] Local address: 172.21.15.161, port 55764 2022-07-03T15:35:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=495] Peer address: 172.21.15.101, port 22 2022-07-03T15:35:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=495] Connection lost 2022-07-03T15:35:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=495] Aborting connection 2022-07-03T15:35:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:35:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:35:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:35:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:35:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:35:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:35:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:35:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:35:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:35:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:35:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:35:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:35:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:35:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:35:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:35:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:35:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:35:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:35:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:35:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:35:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:35:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:35:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:35:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:35:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:35:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:35:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:35:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:35:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:35:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:35:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:35:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:35:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:35:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:35:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:35:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:35:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:35:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:35:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:35:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:35:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:35:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:35:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:35:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:35:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:35:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:35:36.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=495] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:35:36.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=495] Local address: 172.21.15.161, port 55764 2022-07-03T15:35:36.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=495] Peer address: 172.21.15.101, port 22 2022-07-03T15:35:36.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=495] Connection lost 2022-07-03T15:35:36.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=495] Aborting connection 2022-07-03T15:35:36.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:37 smithi161 ceph-mon[91579]: pgmap v8895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:37 smithi188 ceph-mon[92425]: pgmap v8895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:38 smithi161 ceph-mon[91579]: pgmap v8896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:38 smithi188 ceph-mon[92425]: pgmap v8896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:41 smithi188 ceph-mon[92425]: pgmap v8897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:41 smithi161 ceph-mon[91579]: pgmap v8897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:43 smithi188 ceph-mon[92425]: pgmap v8898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:43 smithi161 ceph-mon[91579]: pgmap v8898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:45 smithi188 ceph-mon[92425]: pgmap v8899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:45 smithi161 ceph-mon[91579]: pgmap v8899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:47 smithi161 ceph-mon[91579]: pgmap v8900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:47 smithi188 ceph-mon[92425]: pgmap v8900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:49 smithi188 ceph-mon[92425]: pgmap v8901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:49 smithi161 ceph-mon[91579]: pgmap v8901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:50 smithi188 ceph-mon[92425]: pgmap v8902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:50 smithi161 ceph-mon[91579]: pgmap v8902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:53 smithi188 ceph-mon[92425]: pgmap v8903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:53 smithi161 ceph-mon[91579]: pgmap v8903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:55 smithi188 ceph-mon[92425]: pgmap v8904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:55 smithi161 ceph-mon[91579]: pgmap v8904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:56.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:35:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:57 smithi188 ceph-mon[92425]: pgmap v8905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:57 smithi161 ceph-mon[91579]: pgmap v8905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:35:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:35:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:35:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:35:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:35:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:35:58 smithi188 ceph-mon[92425]: pgmap v8906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:35:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:35:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:35:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:35:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:35:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:35:58 smithi161 ceph-mon[91579]: pgmap v8906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:01 smithi188 ceph-mon[92425]: pgmap v8907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:01 smithi161 ceph-mon[91579]: pgmap v8907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:03 smithi161 ceph-mon[91579]: pgmap v8908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:03 smithi188 ceph-mon[92425]: pgmap v8908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:05 smithi161 ceph-mon[91579]: pgmap v8909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:05 smithi188 ceph-mon[92425]: pgmap v8909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:07 smithi161 ceph-mon[91579]: pgmap v8910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:07 smithi188 ceph-mon[92425]: pgmap v8910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:09 smithi188 ceph-mon[92425]: pgmap v8911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:09 smithi161 ceph-mon[91579]: pgmap v8911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:10 smithi188 ceph-mon[92425]: pgmap v8912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:10.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:10 smithi161 ceph-mon[91579]: pgmap v8912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:13 smithi161 ceph-mon[91579]: pgmap v8913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:13 smithi188 ceph-mon[92425]: pgmap v8913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:15 smithi161 ceph-mon[91579]: pgmap v8914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:15 smithi188 ceph-mon[92425]: pgmap v8914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:17 smithi188 ceph-mon[92425]: pgmap v8915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:17 smithi161 ceph-mon[91579]: pgmap v8915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:18 smithi161 ceph-mon[91579]: pgmap v8916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:18 smithi188 ceph-mon[92425]: pgmap v8916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:21 smithi161 ceph-mon[91579]: pgmap v8917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:21 smithi188 ceph-mon[92425]: pgmap v8917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:23 smithi161 ceph-mon[91579]: pgmap v8918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:23 smithi188 ceph-mon[92425]: pgmap v8918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:25 smithi161 ceph-mon[91579]: pgmap v8919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:25 smithi188 ceph-mon[92425]: pgmap v8919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:27 smithi161 ceph-mon[91579]: pgmap v8920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:27 smithi188 ceph-mon[92425]: pgmap v8920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:29 smithi161 ceph-mon[91579]: pgmap v8921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:29 smithi188 ceph-mon[92425]: pgmap v8921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:31 smithi161 ceph-mon[91579]: pgmap v8922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:31 smithi188 ceph-mon[92425]: pgmap v8922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:33 smithi161 ceph-mon[91579]: pgmap v8923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:33 smithi188 ceph-mon[92425]: pgmap v8923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-mon[91579]: pgmap v8924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:36:35.253+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:36:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:36:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:36:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:36:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:36:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:36:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:36:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:36:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:36:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:36:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:36:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:36:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:36:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:36:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:36:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:36:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:36:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:36:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:36:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:36:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:36:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:36:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:36:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:36:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:36:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:36:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:36:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:36:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:36:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:36:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:36:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:36:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:36:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:36:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:36:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:36:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:36:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:36:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:36:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:36:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:36:35.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:36:35.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:36:35.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:36:35.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:36:35.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=497] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:36:35.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=497] Local address: 172.21.15.161, port 55768 2022-07-03T15:36:35.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=497] Peer address: 172.21.15.101, port 22 2022-07-03T15:36:35.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=497] Connection lost 2022-07-03T15:36:35.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=497] Aborting connection 2022-07-03T15:36:35.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:36:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:36:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:35 smithi188 ceph-mon[92425]: pgmap v8924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:36:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:36:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:36:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:36:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:36:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:36:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:36:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:36:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:36:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:36:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:36:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:36:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:36:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:36:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:36:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:36:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:36:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:36:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:36:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:36:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:36:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:36:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:36:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:36:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:36:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:36:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:36:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:36:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:36:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:36:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:36:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:36:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:36:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:36:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:36:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:36:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:36:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:36:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:36:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:36:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:36:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:36:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:36:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:36:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:36:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:36:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:36:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:36:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:36:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=497] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:36:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=497] Local address: 172.21.15.161, port 55768 2022-07-03T15:36:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=497] Peer address: 172.21.15.101, port 22 2022-07-03T15:36:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=497] Connection lost 2022-07-03T15:36:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=497] Aborting connection 2022-07-03T15:36:36.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:36:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:36:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:36:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:36:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:36:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:36:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:36:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:36:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:36:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:36:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:36:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:36:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:36:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:36:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:36:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:36:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:36:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:36:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:36:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:36:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:36:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:36:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:36:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:36:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:36:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:36:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:36:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:36:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:36:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:36:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:36:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:36:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:36:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:36:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:36:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:36:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:36:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:36:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:36:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:36:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:36:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:36:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:36:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:36:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:36:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:36:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:36:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:36:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:36:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=497] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:36:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=497] Local address: 172.21.15.161, port 55768 2022-07-03T15:36:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=497] Peer address: 172.21.15.101, port 22 2022-07-03T15:36:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=497] Connection lost 2022-07-03T15:36:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=497] Aborting connection 2022-07-03T15:36:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:37 smithi188 ceph-mon[92425]: pgmap v8925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:37.929 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:37 smithi161 ceph-mon[91579]: pgmap v8925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: pgmap v8926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:39.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 162.7M 2022-07-03T15:36:39.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 170606731: error parsing value: Value '170606731' is below minimum 939524096 2022-07-03T15:36:39.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:36:39.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:38 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:36:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:36:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: pgmap v8926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 162.7M 2022-07-03T15:36:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 170606731: error parsing value: Value '170606731' is below minimum 939524096 2022-07-03T15:36:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:36:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:36:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:38 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:36:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:41 smithi161 ceph-mon[91579]: pgmap v8927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:41 smithi188 ceph-mon[92425]: pgmap v8927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:43 smithi161 ceph-mon[91579]: pgmap v8928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:43 smithi188 ceph-mon[92425]: pgmap v8928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:45 smithi161 ceph-mon[91579]: pgmap v8929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:45 smithi188 ceph-mon[92425]: pgmap v8929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:47 smithi188 ceph-mon[92425]: pgmap v8930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:47 smithi161 ceph-mon[91579]: pgmap v8930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:48 smithi188 ceph-mon[92425]: pgmap v8931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:48 smithi161 ceph-mon[91579]: pgmap v8931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:51 smithi161 ceph-mon[91579]: pgmap v8932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:51 smithi188 ceph-mon[92425]: pgmap v8932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:53 smithi161 ceph-mon[91579]: pgmap v8933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:53 smithi188 ceph-mon[92425]: pgmap v8933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:55 smithi161 ceph-mon[91579]: pgmap v8934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:55 smithi188 ceph-mon[92425]: pgmap v8934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:36:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:57 smithi188 ceph-mon[92425]: pgmap v8935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:57 smithi161 ceph-mon[91579]: pgmap v8935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:36:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:36:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:36:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:36:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:36:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:36:58 smithi188 ceph-mon[92425]: pgmap v8936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:36:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:36:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:36:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:36:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:36:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:36:58 smithi161 ceph-mon[91579]: pgmap v8936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:01 smithi161 ceph-mon[91579]: pgmap v8937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:01 smithi188 ceph-mon[92425]: pgmap v8937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:03 smithi161 ceph-mon[91579]: pgmap v8938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:03 smithi188 ceph-mon[92425]: pgmap v8938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:05 smithi161 ceph-mon[91579]: pgmap v8939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:05 smithi188 ceph-mon[92425]: pgmap v8939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:07 smithi188 ceph-mon[92425]: pgmap v8940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:07 smithi161 ceph-mon[91579]: pgmap v8940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:08 smithi188 ceph-mon[92425]: pgmap v8941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:08 smithi161 ceph-mon[91579]: pgmap v8941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:11 smithi161 ceph-mon[91579]: pgmap v8942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:11 smithi188 ceph-mon[92425]: pgmap v8942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:13 smithi161 ceph-mon[91579]: pgmap v8943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:13 smithi188 ceph-mon[92425]: pgmap v8943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:15 smithi161 ceph-mon[91579]: pgmap v8944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:15 smithi188 ceph-mon[92425]: pgmap v8944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:17 smithi188 ceph-mon[92425]: pgmap v8945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:17 smithi161 ceph-mon[91579]: pgmap v8945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:18 smithi188 ceph-mon[92425]: pgmap v8946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:19.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:18 smithi161 ceph-mon[91579]: pgmap v8946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:21 smithi161 ceph-mon[91579]: pgmap v8947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:21 smithi188 ceph-mon[92425]: pgmap v8947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:23 smithi161 ceph-mon[91579]: pgmap v8948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:23 smithi188 ceph-mon[92425]: pgmap v8948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:25 smithi161 ceph-mon[91579]: pgmap v8949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:25 smithi188 ceph-mon[92425]: pgmap v8949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:27 smithi161 ceph-mon[91579]: pgmap v8950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:27 smithi188 ceph-mon[92425]: pgmap v8950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:29 smithi161 ceph-mon[91579]: pgmap v8951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:29 smithi188 ceph-mon[92425]: pgmap v8951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:31 smithi161 ceph-mon[91579]: pgmap v8952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:31 smithi188 ceph-mon[92425]: pgmap v8952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:33 smithi161 ceph-mon[91579]: pgmap v8953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:33 smithi188 ceph-mon[92425]: pgmap v8953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:35 smithi161 ceph-mon[91579]: pgmap v8954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:35 smithi188 ceph-mon[92425]: pgmap v8954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:37 smithi161 ceph-mon[91579]: pgmap v8955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:37 smithi188 ceph-mon[92425]: pgmap v8955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:38.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:37:38.509+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:37:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:37:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:37:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:37:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:37:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:37:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:37:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:37:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:37:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:37:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:37:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:37:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:37:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:37:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:37:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:37:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:37:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:37:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:37:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:37:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:37:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:37:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:37:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:37:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:37:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:37:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:37:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:37:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:37:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:37:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:37:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:37:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:37:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:37:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:37:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:37:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:37:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:37:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:37:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:37:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:37:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:37:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:37:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:37:38.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:37:38.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=499] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:37:38.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=499] Local address: 172.21.15.161, port 55778 2022-07-03T15:37:38.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=499] Peer address: 172.21.15.101, port 22 2022-07-03T15:37:38.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=499] Connection lost 2022-07-03T15:37:38.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=499] Aborting connection 2022-07-03T15:37:38.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:37:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:37:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: pgmap v8956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:37:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:37:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:37:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:37:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:37:39.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:37:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:37:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:37:39.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:37:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:37:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:37:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:37:39.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:37:39.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:37:39.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:37:39.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:37:39.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:37:39.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:37:39.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:37:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:37:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:37:39.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:37:39.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:37:39.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:37:39.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:37:39.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:37:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:37:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:37:39.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:37:39.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:37:39.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:37:39.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:37:39.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:37:39.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:37:39.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:37:39.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:37:39.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:37:39.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:37:39.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:37:39.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:37:39.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:37:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:37:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:37:39.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:37:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:37:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:37:39.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:37:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:37:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:37:39.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=499] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:37:39.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=499] Local address: 172.21.15.161, port 55778 2022-07-03T15:37:39.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=499] Peer address: 172.21.15.101, port 22 2022-07-03T15:37:39.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=499] Connection lost 2022-07-03T15:37:39.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=499] Aborting connection 2022-07-03T15:37:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: pgmap v8956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:37:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:37:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:37:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:37:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:37:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:37:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:37:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:37:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:37:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:37:39.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:37:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:37:39.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:37:39.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:37:39.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:37:39.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:37:39.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:37:39.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:37:39.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:37:39.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:37:39.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:37:39.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:37:39.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:37:39.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:37:39.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:37:39.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:37:39.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:37:39.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:37:39.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:37:39.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:37:39.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:37:39.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:37:39.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:37:39.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:37:39.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:37:39.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:37:39.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:37:39.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:37:39.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:37:39.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:37:39.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:37:39.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:37:39.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:37:39.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:37:39.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:37:39.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:37:39.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:37:39.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:37:39.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:37:39.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=499] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:37:39.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=499] Local address: 172.21.15.161, port 55778 2022-07-03T15:37:39.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=499] Peer address: 172.21.15.101, port 22 2022-07-03T15:37:39.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=499] Connection lost 2022-07-03T15:37:39.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=499] Aborting connection 2022-07-03T15:37:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:41 smithi161 ceph-mon[91579]: pgmap v8957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:41 smithi188 ceph-mon[92425]: pgmap v8957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:43 smithi161 ceph-mon[91579]: pgmap v8958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:43 smithi188 ceph-mon[92425]: pgmap v8958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:45 smithi161 ceph-mon[91579]: pgmap v8959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:45 smithi188 ceph-mon[92425]: pgmap v8959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:47 smithi161 ceph-mon[91579]: pgmap v8960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:47 smithi188 ceph-mon[92425]: pgmap v8960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:49 smithi161 ceph-mon[91579]: pgmap v8961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:49 smithi188 ceph-mon[92425]: pgmap v8961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:51 smithi161 ceph-mon[91579]: pgmap v8962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:51 smithi188 ceph-mon[92425]: pgmap v8962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:53 smithi161 ceph-mon[91579]: pgmap v8963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:53 smithi188 ceph-mon[92425]: pgmap v8963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:55 smithi161 ceph-mon[91579]: pgmap v8964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:55 smithi188 ceph-mon[92425]: pgmap v8964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:37:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:37:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:57 smithi161 ceph-mon[91579]: pgmap v8965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:57 smithi188 ceph-mon[92425]: pgmap v8965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:37:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:37:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:37:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:37:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:37:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:37:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:37:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:37:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:37:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:37:59 smithi161 ceph-mon[91579]: pgmap v8966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:37:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:37:59 smithi188 ceph-mon[92425]: pgmap v8966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:01 smithi161 ceph-mon[91579]: pgmap v8967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:01 smithi188 ceph-mon[92425]: pgmap v8967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:03 smithi161 ceph-mon[91579]: pgmap v8968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:03 smithi188 ceph-mon[92425]: pgmap v8968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:05 smithi161 ceph-mon[91579]: pgmap v8969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:05 smithi188 ceph-mon[92425]: pgmap v8969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:07 smithi161 ceph-mon[91579]: pgmap v8970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:07 smithi188 ceph-mon[92425]: pgmap v8970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:09 smithi161 ceph-mon[91579]: pgmap v8971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:09 smithi188 ceph-mon[92425]: pgmap v8971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:11 smithi161 ceph-mon[91579]: pgmap v8972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:11 smithi188 ceph-mon[92425]: pgmap v8972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:13 smithi161 ceph-mon[91579]: pgmap v8973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:13 smithi188 ceph-mon[92425]: pgmap v8973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:15 smithi161 ceph-mon[91579]: pgmap v8974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:15 smithi188 ceph-mon[92425]: pgmap v8974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:17 smithi161 ceph-mon[91579]: pgmap v8975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:17 smithi188 ceph-mon[92425]: pgmap v8975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:19 smithi161 ceph-mon[91579]: pgmap v8976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:19 smithi188 ceph-mon[92425]: pgmap v8976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:21 smithi161 ceph-mon[91579]: pgmap v8977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:21 smithi188 ceph-mon[92425]: pgmap v8977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:23 smithi161 ceph-mon[91579]: pgmap v8978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:23 smithi188 ceph-mon[92425]: pgmap v8978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:25 smithi161 ceph-mon[91579]: pgmap v8979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:25 smithi188 ceph-mon[92425]: pgmap v8979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:27 smithi161 ceph-mon[91579]: pgmap v8980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:27 smithi188 ceph-mon[92425]: pgmap v8980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:29 smithi161 ceph-mon[91579]: pgmap v8981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:29 smithi188 ceph-mon[92425]: pgmap v8981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:31 smithi161 ceph-mon[91579]: pgmap v8982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:31 smithi188 ceph-mon[92425]: pgmap v8982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:33 smithi161 ceph-mon[91579]: pgmap v8983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:33 smithi188 ceph-mon[92425]: pgmap v8983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:35 smithi161 ceph-mon[91579]: pgmap v8984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:35 smithi188 ceph-mon[92425]: pgmap v8984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:37 smithi161 ceph-mon[91579]: pgmap v8985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:37 smithi188 ceph-mon[92425]: pgmap v8985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:39.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:38:39.011+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:38:39.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:38:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:38:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:38:39.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:38:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:38:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:38:39.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:38:39.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:38:39.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:38:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:38:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:38:39.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:38:39.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:38:39.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:38:39.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:38:39.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:38:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:38:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:38:39.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:38:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:38:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:38:39.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:38:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:38:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:38:39.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:38:39.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:38:39.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:38:39.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:38:39.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:38:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:38:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:38:39.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:38:39.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:38:39.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:38:39.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:38:39.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:38:39.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:38:39.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:38:39.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:38:39.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:38:39.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:38:39.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:38:39.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:38:39.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:38:39.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=501] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:38:39.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=501] Local address: 172.21.15.161, port 55806 2022-07-03T15:38:39.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=501] Peer address: 172.21.15.101, port 22 2022-07-03T15:38:39.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=501] Connection lost 2022-07-03T15:38:39.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=501] Aborting connection 2022-07-03T15:38:39.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:38:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-mon[91579]: pgmap v8986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:38:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:38:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:38:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:39 smithi188 ceph-mon[92425]: pgmap v8986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:38:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:38:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:38:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:38:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:38:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:38:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:38:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:38:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:38:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:38:40.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:38:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:38:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:38:40.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:38:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:38:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:38:40.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:38:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:38:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:38:40.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:38:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:38:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:38:40.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:38:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:38:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:38:40.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:38:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:38:40.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:38:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:38:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:38:40.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:38:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:38:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:38:40.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:38:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:38:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:38:40.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:38:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:38:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:38:40.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:38:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:38:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:38:40.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:38:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:38:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:38:40.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:38:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:38:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:38:40.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:38:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=501] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:38:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=501] Local address: 172.21.15.161, port 55806 2022-07-03T15:38:40.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=501] Peer address: 172.21.15.101, port 22 2022-07-03T15:38:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=501] Connection lost 2022-07-03T15:38:40.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=501] Aborting connection 2022-07-03T15:38:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:38:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:38:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:38:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:38:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:38:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:38:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:38:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:38:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:38:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:38:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:38:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:38:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:38:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:38:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:38:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:38:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:38:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:38:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:38:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:38:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:38:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:38:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:38:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:38:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:38:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:38:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:38:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:38:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:38:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:38:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:38:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:38:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:38:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:38:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:38:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:38:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:38:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:38:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:38:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:38:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:38:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:38:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:38:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:38:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:38:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:38:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=501] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:38:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=501] Local address: 172.21.15.161, port 55806 2022-07-03T15:38:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=501] Peer address: 172.21.15.101, port 22 2022-07-03T15:38:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=501] Connection lost 2022-07-03T15:38:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=501] Aborting connection 2022-07-03T15:38:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:41 smithi161 ceph-mon[91579]: pgmap v8987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:41 smithi188 ceph-mon[92425]: pgmap v8987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:43 smithi161 ceph-mon[91579]: pgmap v8988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:43 smithi188 ceph-mon[92425]: pgmap v8988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:45 smithi161 ceph-mon[91579]: pgmap v8989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:45 smithi188 ceph-mon[92425]: pgmap v8989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:47 smithi161 ceph-mon[91579]: pgmap v8990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:47 smithi188 ceph-mon[92425]: pgmap v8990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:49 smithi161 ceph-mon[91579]: pgmap v8991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:49 smithi188 ceph-mon[92425]: pgmap v8991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:51 smithi161 ceph-mon[91579]: pgmap v8992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:51 smithi188 ceph-mon[92425]: pgmap v8992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:53 smithi161 ceph-mon[91579]: pgmap v8993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:53 smithi188 ceph-mon[92425]: pgmap v8993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:55 smithi161 ceph-mon[91579]: pgmap v8994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:55 smithi188 ceph-mon[92425]: pgmap v8994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:38:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:38:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:57 smithi161 ceph-mon[91579]: pgmap v8995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:57 smithi188 ceph-mon[92425]: pgmap v8995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:38:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:38:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:38:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:38:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:38:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:38:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:38:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:38:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:38:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:38:59 smithi161 ceph-mon[91579]: pgmap v8996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:38:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:38:59 smithi188 ceph-mon[92425]: pgmap v8996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:01 smithi161 ceph-mon[91579]: pgmap v8997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:01 smithi188 ceph-mon[92425]: pgmap v8997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:03 smithi161 ceph-mon[91579]: pgmap v8998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:03 smithi188 ceph-mon[92425]: pgmap v8998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:05 smithi161 ceph-mon[91579]: pgmap v8999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:05 smithi188 ceph-mon[92425]: pgmap v8999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:07 smithi161 ceph-mon[91579]: pgmap v9000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:07 smithi188 ceph-mon[92425]: pgmap v9000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:09 smithi161 ceph-mon[91579]: pgmap v9001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:09 smithi188 ceph-mon[92425]: pgmap v9001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:11 smithi161 ceph-mon[91579]: pgmap v9002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:11 smithi188 ceph-mon[92425]: pgmap v9002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:13 smithi188 ceph-mon[92425]: pgmap v9003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:13 smithi161 ceph-mon[91579]: pgmap v9003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:15 smithi188 ceph-mon[92425]: pgmap v9004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:15 smithi161 ceph-mon[91579]: pgmap v9004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:17 smithi188 ceph-mon[92425]: pgmap v9005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:17 smithi161 ceph-mon[91579]: pgmap v9005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:19 smithi188 ceph-mon[92425]: pgmap v9006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:19 smithi161 ceph-mon[91579]: pgmap v9006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:21 smithi188 ceph-mon[92425]: pgmap v9007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:21 smithi161 ceph-mon[91579]: pgmap v9007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:23 smithi188 ceph-mon[92425]: pgmap v9008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:23 smithi161 ceph-mon[91579]: pgmap v9008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:25 smithi188 ceph-mon[92425]: pgmap v9009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:25 smithi161 ceph-mon[91579]: pgmap v9009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:27 smithi188 ceph-mon[92425]: pgmap v9010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:27 smithi161 ceph-mon[91579]: pgmap v9010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:29 smithi188 ceph-mon[92425]: pgmap v9011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:29 smithi161 ceph-mon[91579]: pgmap v9011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:31 smithi188 ceph-mon[92425]: pgmap v9012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:31 smithi161 ceph-mon[91579]: pgmap v9012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:33 smithi161 ceph-mon[91579]: pgmap v9013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:33 smithi188 ceph-mon[92425]: pgmap v9013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:35 smithi161 ceph-mon[91579]: pgmap v9014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:35 smithi188 ceph-mon[92425]: pgmap v9014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:37 smithi161 ceph-mon[91579]: pgmap v9015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:37 smithi188 ceph-mon[92425]: pgmap v9015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-mon[91579]: pgmap v9016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:39.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:39:39.514+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:39:39.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:39:39.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:39:39.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:39:39.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:39:39.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:39:39.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:39:39.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:39:39.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:39:39.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:39:39.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:39:39.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:39:39.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:39:39.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:39:39.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:39:39.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:39:39.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:39:39.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:39:39.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:39:39.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:39:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:39:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:39:39.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:39:39.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:39:39.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:39:39.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:39:39.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:39:39.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:39:39.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:39:39.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:39:39.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:39:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:39:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:39:39.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:39:39.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:39:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:39:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:39:39.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:39:39.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:39:39.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:39:39.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:39:39.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:39:39.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:39:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:39:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:39:39.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=503] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:39:39.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=503] Local address: 172.21.15.161, port 55862 2022-07-03T15:39:39.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=503] Peer address: 172.21.15.101, port 22 2022-07-03T15:39:39.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=503] Connection lost 2022-07-03T15:39:39.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=503] Aborting connection 2022-07-03T15:39:39.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:39:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:39:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:39 smithi188 ceph-mon[92425]: pgmap v9016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:40.795 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:39:40.796 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:39:40.796 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:39:40.797 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:39:40.797 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:39:40.797 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:39:40.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:39:40.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:39:40.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:39:40.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:39:40.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:39:40.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:39:40.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:39:40.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:39:40.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:39:40.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:39:40.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:39:40.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:39:40.802 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:39:40.802 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:39:40.802 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:39:40.803 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:39:40.803 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:39:40.803 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:39:40.804 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:39:40.804 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:39:40.804 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:39:40.805 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:39:40.805 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:39:40.805 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:39:40.806 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:39:40.806 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:39:40.806 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:39:40.807 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:39:40.807 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:39:40.807 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:39:40.808 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:39:40.808 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:39:40.808 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:39:40.808 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:39:40.809 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:39:40.809 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:39:40.809 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:39:40.810 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:39:40.810 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:39:40.810 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:39:40.811 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:39:40.811 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:39:40.811 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:39:40.812 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=503] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:39:40.812 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=503] Local address: 172.21.15.161, port 55862 2022-07-03T15:39:40.812 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=503] Peer address: 172.21.15.101, port 22 2022-07-03T15:39:40.813 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=503] Connection lost 2022-07-03T15:39:40.813 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=503] Aborting connection 2022-07-03T15:39:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:39:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:39:40.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:39:40.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:39:40.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:39:40.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:39:40.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:39:40.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:39:40.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:39:40.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:39:40.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:39:40.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:39:40.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:39:40.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:39:40.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:39:40.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:39:40.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:39:40.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:39:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:39:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:39:40.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:39:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:39:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:39:40.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:39:40.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:39:40.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:39:40.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:39:40.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:39:40.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:39:40.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:39:40.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:39:40.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:39:40.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:39:40.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:39:40.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:39:40.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:39:40.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:39:40.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:39:40.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:39:40.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:39:40.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:39:40.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:39:40.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:39:40.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:39:40.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:39:40.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:39:40.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:39:40.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:39:40.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:39:40.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=503] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:39:40.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=503] Local address: 172.21.15.161, port 55862 2022-07-03T15:39:40.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=503] Peer address: 172.21.15.101, port 22 2022-07-03T15:39:40.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=503] Connection lost 2022-07-03T15:39:40.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=503] Aborting connection 2022-07-03T15:39:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:41 smithi161 ceph-mon[91579]: pgmap v9017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:41 smithi188 ceph-mon[92425]: pgmap v9017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:43 smithi161 ceph-mon[91579]: pgmap v9018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:39:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:39:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:43 smithi188 ceph-mon[92425]: pgmap v9018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:39:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:39:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:44 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T15:39:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:44 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T15:39:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:45 smithi161 ceph-mon[91579]: pgmap v9019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:45 smithi188 ceph-mon[92425]: pgmap v9019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:47 smithi161 ceph-mon[91579]: pgmap v9020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:47 smithi188 ceph-mon[92425]: pgmap v9020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:49 smithi161 ceph-mon[91579]: pgmap v9021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:49 smithi188 ceph-mon[92425]: pgmap v9021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:51 smithi161 ceph-mon[91579]: pgmap v9022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:51 smithi188 ceph-mon[92425]: pgmap v9022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:53 smithi161 ceph-mon[91579]: pgmap v9023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:53 smithi188 ceph-mon[92425]: pgmap v9023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:55 smithi161 ceph-mon[91579]: pgmap v9024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:55 smithi188 ceph-mon[92425]: pgmap v9024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:39:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:39:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:57 smithi161 ceph-mon[91579]: pgmap v9025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:57 smithi188 ceph-mon[92425]: pgmap v9025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:39:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:39:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:39:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:39:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:39:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:39:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:39:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:39:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:39:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:39:59 smithi161 ceph-mon[91579]: pgmap v9026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:39:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:39:59 smithi188 ceph-mon[92425]: pgmap v9026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:40:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:40:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:01 smithi161 ceph-mon[91579]: pgmap v9027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:01 smithi188 ceph-mon[92425]: pgmap v9027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:03 smithi161 ceph-mon[91579]: pgmap v9028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:03 smithi188 ceph-mon[92425]: pgmap v9028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:05 smithi161 ceph-mon[91579]: pgmap v9029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:05 smithi188 ceph-mon[92425]: pgmap v9029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:07 smithi161 ceph-mon[91579]: pgmap v9030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:07 smithi188 ceph-mon[92425]: pgmap v9030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:09 smithi161 ceph-mon[91579]: pgmap v9031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:09 smithi188 ceph-mon[92425]: pgmap v9031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:11 smithi161 ceph-mon[91579]: pgmap v9032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:11 smithi188 ceph-mon[92425]: pgmap v9032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:13 smithi161 ceph-mon[91579]: pgmap v9033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:13 smithi188 ceph-mon[92425]: pgmap v9033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:15 smithi161 ceph-mon[91579]: pgmap v9034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:15 smithi188 ceph-mon[92425]: pgmap v9034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:17 smithi161 ceph-mon[91579]: pgmap v9035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:17 smithi188 ceph-mon[92425]: pgmap v9035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:19 smithi161 ceph-mon[91579]: pgmap v9036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:19 smithi188 ceph-mon[92425]: pgmap v9036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:21 smithi161 ceph-mon[91579]: pgmap v9037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:21 smithi188 ceph-mon[92425]: pgmap v9037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:23 smithi161 ceph-mon[91579]: pgmap v9038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:23 smithi188 ceph-mon[92425]: pgmap v9038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:25 smithi161 ceph-mon[91579]: pgmap v9039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:25 smithi188 ceph-mon[92425]: pgmap v9039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:27 smithi161 ceph-mon[91579]: pgmap v9040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:27 smithi188 ceph-mon[92425]: pgmap v9040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:29 smithi161 ceph-mon[91579]: pgmap v9041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:29 smithi188 ceph-mon[92425]: pgmap v9041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:31 smithi161 ceph-mon[91579]: pgmap v9042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:31 smithi188 ceph-mon[92425]: pgmap v9042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:33 smithi161 ceph-mon[91579]: pgmap v9043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:33 smithi188 ceph-mon[92425]: pgmap v9043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:35 smithi161 ceph-mon[91579]: pgmap v9044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:35 smithi188 ceph-mon[92425]: pgmap v9044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:37 smithi161 ceph-mon[91579]: pgmap v9045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:37 smithi188 ceph-mon[92425]: pgmap v9045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:39 smithi161 ceph-mon[91579]: pgmap v9046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:39 smithi188 ceph-mon[92425]: pgmap v9046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:41 smithi161 ceph-mon[91579]: pgmap v9047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:41 smithi188 ceph-mon[92425]: pgmap v9047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:43.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:40:42.869+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:40:43.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:40:43.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:40:43.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:40:43.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:40:43.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:40:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:40:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:40:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:40:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:40:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:40:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:40:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:40:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:40:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:40:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:40:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:40:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:40:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:40:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:40:43.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:40:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:40:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:40:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:40:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:40:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:40:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:40:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:40:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:40:43.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:40:43.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:40:43.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:40:43.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:40:43.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:40:43.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:40:43.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:40:43.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:40:43.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:40:43.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:40:43.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:40:43.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:40:43.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:40:43.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:40:43.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:40:43.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:40:43.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=505] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:40:43.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=505] Local address: 172.21.15.161, port 55892 2022-07-03T15:40:43.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=505] Peer address: 172.21.15.101, port 22 2022-07-03T15:40:43.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=505] Connection lost 2022-07-03T15:40:43.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=505] Aborting connection 2022-07-03T15:40:43.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:40:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:40:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:43 smithi161 ceph-mon[91579]: pgmap v9048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:40:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:40:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:40:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:43 smithi188 ceph-mon[92425]: pgmap v9048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:40:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:40:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:40:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:40:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:40:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:40:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:40:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:40:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:40:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:40:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:40:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:40:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:40:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:40:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:40:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:40:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:40:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:40:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:40:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:40:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:40:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:40:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:40:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:40:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:40:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:40:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:40:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:40:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:40:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:40:44.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:40:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:40:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:40:44.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:40:44.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:40:44.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:40:44.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:40:44.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:40:44.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:40:44.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:40:44.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:40:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:40:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:40:44.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:40:44.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:40:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:40:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:40:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:40:44.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:40:44.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=505] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:40:44.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=505] Local address: 172.21.15.161, port 55892 2022-07-03T15:40:44.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=505] Peer address: 172.21.15.101, port 22 2022-07-03T15:40:44.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=505] Connection lost 2022-07-03T15:40:44.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=505] Aborting connection 2022-07-03T15:40:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:40:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:40:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:40:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:40:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:40:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:40:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:40:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:40:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:40:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:40:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:40:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:40:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:40:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:40:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:40:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:40:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:40:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:40:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:40:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:40:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:40:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:40:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:40:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:40:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:40:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:40:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:40:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:40:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:40:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:40:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:40:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:40:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:40:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:40:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:40:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:40:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:40:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:40:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:40:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:40:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:40:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:40:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:40:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:40:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:40:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:40:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=505] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:40:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=505] Local address: 172.21.15.161, port 55892 2022-07-03T15:40:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=505] Peer address: 172.21.15.101, port 22 2022-07-03T15:40:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=505] Connection lost 2022-07-03T15:40:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=505] Aborting connection 2022-07-03T15:40:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:45 smithi161 ceph-mon[91579]: pgmap v9049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:45 smithi188 ceph-mon[92425]: pgmap v9049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:47 smithi161 ceph-mon[91579]: pgmap v9050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:47 smithi188 ceph-mon[92425]: pgmap v9050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:49 smithi161 ceph-mon[91579]: pgmap v9051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:49 smithi188 ceph-mon[92425]: pgmap v9051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:51 smithi161 ceph-mon[91579]: pgmap v9052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:51 smithi188 ceph-mon[92425]: pgmap v9052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:53 smithi161 ceph-mon[91579]: pgmap v9053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:53 smithi188 ceph-mon[92425]: pgmap v9053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:55 smithi161 ceph-mon[91579]: pgmap v9054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:55 smithi188 ceph-mon[92425]: pgmap v9054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:40:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:40:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:57 smithi161 ceph-mon[91579]: pgmap v9055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:57 smithi188 ceph-mon[92425]: pgmap v9055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:40:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:40:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:40:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:40:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:40:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:40:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:40:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:40:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:40:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:40:59 smithi161 ceph-mon[91579]: pgmap v9056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:40:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:40:59 smithi188 ceph-mon[92425]: pgmap v9056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:01 smithi161 ceph-mon[91579]: pgmap v9057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:01 smithi188 ceph-mon[92425]: pgmap v9057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:03 smithi161 ceph-mon[91579]: pgmap v9058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:03 smithi188 ceph-mon[92425]: pgmap v9058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:05 smithi161 ceph-mon[91579]: pgmap v9059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:05 smithi188 ceph-mon[92425]: pgmap v9059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:07 smithi161 ceph-mon[91579]: pgmap v9060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:07 smithi188 ceph-mon[92425]: pgmap v9060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:09 smithi161 ceph-mon[91579]: pgmap v9061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:09 smithi188 ceph-mon[92425]: pgmap v9061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:11 smithi161 ceph-mon[91579]: pgmap v9062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:11 smithi188 ceph-mon[92425]: pgmap v9062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:13 smithi161 ceph-mon[91579]: pgmap v9063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:13 smithi188 ceph-mon[92425]: pgmap v9063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:41:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:15 smithi161 ceph-mon[91579]: pgmap v9064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:15 smithi188 ceph-mon[92425]: pgmap v9064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:17 smithi161 ceph-mon[91579]: pgmap v9065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:17 smithi188 ceph-mon[92425]: pgmap v9065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:19 smithi161 ceph-mon[91579]: pgmap v9066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:19 smithi188 ceph-mon[92425]: pgmap v9066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:21 smithi161 ceph-mon[91579]: pgmap v9067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:21 smithi188 ceph-mon[92425]: pgmap v9067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:23 smithi161 ceph-mon[91579]: pgmap v9068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:23 smithi188 ceph-mon[92425]: pgmap v9068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:25 smithi161 ceph-mon[91579]: pgmap v9069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:25 smithi188 ceph-mon[92425]: pgmap v9069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:27 smithi161 ceph-mon[91579]: pgmap v9070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:27 smithi188 ceph-mon[92425]: pgmap v9070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:29 smithi161 ceph-mon[91579]: pgmap v9071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:29 smithi188 ceph-mon[92425]: pgmap v9071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:31 smithi161 ceph-mon[91579]: pgmap v9072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:31 smithi188 ceph-mon[92425]: pgmap v9072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:33 smithi161 ceph-mon[91579]: pgmap v9073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:33 smithi188 ceph-mon[92425]: pgmap v9073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:35 smithi161 ceph-mon[91579]: pgmap v9074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:35 smithi188 ceph-mon[92425]: pgmap v9074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:37 smithi188 ceph-mon[92425]: pgmap v9075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:37 smithi161 ceph-mon[91579]: pgmap v9075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:39 smithi188 ceph-mon[92425]: pgmap v9076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:39 smithi161 ceph-mon[91579]: pgmap v9076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:41 smithi188 ceph-mon[92425]: pgmap v9077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:41 smithi161 ceph-mon[91579]: pgmap v9077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:43.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:41:43.377+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:41:43.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:41:43.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:41:43.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:41:43.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:41:43.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:41:43.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:41:43.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:41:43.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:41:43.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:41:43.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:41:43.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:41:43.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:41:43.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:41:43.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:41:43.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:41:43.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:41:43.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:41:43.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:41:43.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:41:43.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:41:43.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:41:43.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:41:43.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:41:43.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:41:43.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:41:43.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:41:43.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:41:43.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:41:43.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:41:43.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:41:43.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:41:43.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:41:43.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:41:43.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:41:43.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:41:43.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:41:43.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:41:43.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:41:43.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:41:43.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:41:43.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:41:43.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:41:43.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:41:43.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:41:43.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=507] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:41:43.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=507] Local address: 172.21.15.161, port 55920 2022-07-03T15:41:43.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=507] Peer address: 172.21.15.101, port 22 2022-07-03T15:41:43.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=507] Connection lost 2022-07-03T15:41:43.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=507] Aborting connection 2022-07-03T15:41:43.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:41:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:43 smithi188 ceph-mon[92425]: pgmap v9078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:41:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:41:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:41:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-mon[91579]: pgmap v9078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:41:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:41:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:41:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:41:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:41:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:41:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:41:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:41:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:41:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:41:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:41:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:41:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:41:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:41:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:41:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:41:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:41:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:41:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:41:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:41:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:41:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:41:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:41:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:41:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:41:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:41:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:41:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:41:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:41:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:41:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:41:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:41:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:41:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:41:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:41:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:41:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:41:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:41:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:41:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:41:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:41:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:41:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:41:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:41:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:41:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:41:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:41:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:41:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:41:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=507] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:41:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=507] Local address: 172.21.15.161, port 55920 2022-07-03T15:41:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=507] Peer address: 172.21.15.101, port 22 2022-07-03T15:41:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=507] Connection lost 2022-07-03T15:41:44.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=507] Aborting connection 2022-07-03T15:41:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:41:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:41:45.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:41:45.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:41:45.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:41:45.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:41:45.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:41:45.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:41:45.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:41:45.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:41:45.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:41:45.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:41:45.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:41:45.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:41:45.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:41:45.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:41:45.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:41:45.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:41:45.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:41:45.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:41:45.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:41:45.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:41:45.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:41:45.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:41:45.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:41:45.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:41:45.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:41:45.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:41:45.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:41:45.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:41:45.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:41:45.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:41:45.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:41:45.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:41:45.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:41:45.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:41:45.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:41:45.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:41:45.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:41:45.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:41:45.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:41:45.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:41:45.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:41:45.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:41:45.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:41:45.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:41:45.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=507] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:41:45.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=507] Local address: 172.21.15.161, port 55920 2022-07-03T15:41:45.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=507] Peer address: 172.21.15.101, port 22 2022-07-03T15:41:45.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=507] Connection lost 2022-07-03T15:41:45.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=507] Aborting connection 2022-07-03T15:41:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:45 smithi188 ceph-mon[92425]: pgmap v9079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:45 smithi161 ceph-mon[91579]: pgmap v9079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:46 smithi161 ceph-mon[91579]: pgmap v9080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:46 smithi188 ceph-mon[92425]: pgmap v9080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:49 smithi161 ceph-mon[91579]: pgmap v9081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:49 smithi188 ceph-mon[92425]: pgmap v9081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:51 smithi161 ceph-mon[91579]: pgmap v9082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:51 smithi188 ceph-mon[92425]: pgmap v9082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:51.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:53 smithi161 ceph-mon[91579]: pgmap v9083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:53 smithi188 ceph-mon[92425]: pgmap v9083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:55 smithi161 ceph-mon[91579]: pgmap v9084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:55 smithi188 ceph-mon[92425]: pgmap v9084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:41:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:41:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:57 smithi161 ceph-mon[91579]: pgmap v9085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:57 smithi188 ceph-mon[92425]: pgmap v9085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:41:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:41:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:41:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:41:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:41:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:41:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:41:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:41:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:41:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:41:59 smithi161 ceph-mon[91579]: pgmap v9086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:41:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:41:59 smithi188 ceph-mon[92425]: pgmap v9086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:01 smithi161 ceph-mon[91579]: pgmap v9087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:01 smithi188 ceph-mon[92425]: pgmap v9087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:03 smithi161 ceph-mon[91579]: pgmap v9088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:03 smithi188 ceph-mon[92425]: pgmap v9088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:05 smithi161 ceph-mon[91579]: pgmap v9089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:05 smithi188 ceph-mon[92425]: pgmap v9089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:07 smithi161 ceph-mon[91579]: pgmap v9090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:07 smithi188 ceph-mon[92425]: pgmap v9090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:09 smithi161 ceph-mon[91579]: pgmap v9091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:09 smithi188 ceph-mon[92425]: pgmap v9091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:11 smithi161 ceph-mon[91579]: pgmap v9092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:11 smithi188 ceph-mon[92425]: pgmap v9092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:13 smithi161 ceph-mon[91579]: pgmap v9093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:13 smithi188 ceph-mon[92425]: pgmap v9093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:15 smithi161 ceph-mon[91579]: pgmap v9094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:15 smithi188 ceph-mon[92425]: pgmap v9094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:17 smithi161 ceph-mon[91579]: pgmap v9095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:17 smithi188 ceph-mon[92425]: pgmap v9095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:19 smithi161 ceph-mon[91579]: pgmap v9096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:19 smithi188 ceph-mon[92425]: pgmap v9096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:21 smithi161 ceph-mon[91579]: pgmap v9097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:21 smithi188 ceph-mon[92425]: pgmap v9097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:23 smithi161 ceph-mon[91579]: pgmap v9098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:23 smithi188 ceph-mon[92425]: pgmap v9098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:25 smithi161 ceph-mon[91579]: pgmap v9099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:25 smithi188 ceph-mon[92425]: pgmap v9099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:27 smithi161 ceph-mon[91579]: pgmap v9100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:27 smithi188 ceph-mon[92425]: pgmap v9100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:29 smithi161 ceph-mon[91579]: pgmap v9101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:29 smithi188 ceph-mon[92425]: pgmap v9101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:31 smithi161 ceph-mon[91579]: pgmap v9102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:31 smithi188 ceph-mon[92425]: pgmap v9102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:33 smithi161 ceph-mon[91579]: pgmap v9103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:33 smithi188 ceph-mon[92425]: pgmap v9103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:35 smithi161 ceph-mon[91579]: pgmap v9104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:35 smithi188 ceph-mon[92425]: pgmap v9104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:37 smithi161 ceph-mon[91579]: pgmap v9105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:37 smithi188 ceph-mon[92425]: pgmap v9105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:39 smithi161 ceph-mon[91579]: pgmap v9106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:39 smithi188 ceph-mon[92425]: pgmap v9106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:41 smithi161 ceph-mon[91579]: pgmap v9107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:41 smithi188 ceph-mon[92425]: pgmap v9107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-mon[91579]: pgmap v9108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:43 smithi188 ceph-mon[92425]: pgmap v9108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:44.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:42:43.877+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:42:44.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:42:44.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:42:44.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:42:44.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:42:44.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:42:44.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:42:44.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:42:44.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:42:44.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:42:44.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:42:44.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:42:44.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:42:44.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:42:44.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:42:44.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:42:44.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:42:44.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:42:44.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:42:44.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:42:44.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:42:44.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:42:44.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:42:44.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:42:44.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:42:44.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:42:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:42:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:42:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:42:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:42:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:42:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:42:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:42:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:42:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:42:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:42:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:42:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:42:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:42:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:42:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:42:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:42:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:42:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:42:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:42:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=509] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:42:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=509] Local address: 172.21.15.161, port 55948 2022-07-03T15:42:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=509] Peer address: 172.21.15.101, port 22 2022-07-03T15:42:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=509] Connection lost 2022-07-03T15:42:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=509] Aborting connection 2022-07-03T15:42:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:42:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:42:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:42:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:42:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:42:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:42:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:42:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:42:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:42:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:42:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:42:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:42:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:42:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:42:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:42:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:42:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:42:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:42:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:42:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:42:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:42:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:42:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:42:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:42:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:42:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:42:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:42:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:42:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:42:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:42:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:42:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:42:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:42:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:42:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:42:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:42:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:42:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:42:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:42:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:42:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:42:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:42:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:42:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:42:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:42:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:42:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:42:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:42:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:42:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:42:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:42:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:42:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:42:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=509] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:42:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=509] Local address: 172.21.15.161, port 55948 2022-07-03T15:42:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=509] Peer address: 172.21.15.101, port 22 2022-07-03T15:42:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=509] Connection lost 2022-07-03T15:42:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=509] Aborting connection 2022-07-03T15:42:45.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:45 smithi161 ceph-mon[91579]: pgmap v9109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:42:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:42:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:42:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:42:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:42:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:42:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:42:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:42:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:42:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:42:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:42:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:42:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:42:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:42:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:42:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:42:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:42:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:42:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:42:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:42:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:42:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:42:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:42:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:42:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:42:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:42:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:42:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:42:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:42:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:42:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:42:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:42:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:42:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:42:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:42:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:42:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:42:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:42:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:42:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:42:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:42:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:42:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:42:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:42:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:42:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:42:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=509] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:42:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=509] Local address: 172.21.15.161, port 55948 2022-07-03T15:42:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=509] Peer address: 172.21.15.101, port 22 2022-07-03T15:42:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=509] Connection lost 2022-07-03T15:42:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=509] Aborting connection 2022-07-03T15:42:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:45 smithi188 ceph-mon[92425]: pgmap v9109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:47 smithi161 ceph-mon[91579]: pgmap v9110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:47 smithi188 ceph-mon[92425]: pgmap v9110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:49 smithi161 ceph-mon[91579]: pgmap v9111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:49 smithi188 ceph-mon[92425]: pgmap v9111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:51 smithi188 ceph-mon[92425]: pgmap v9112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:51 smithi161 ceph-mon[91579]: pgmap v9112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:51.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:53 smithi188 ceph-mon[92425]: pgmap v9113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:53 smithi161 ceph-mon[91579]: pgmap v9113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:55 smithi188 ceph-mon[92425]: pgmap v9114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:55 smithi161 ceph-mon[91579]: pgmap v9114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:42:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:42:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:57 smithi188 ceph-mon[92425]: pgmap v9115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:57 smithi161 ceph-mon[91579]: pgmap v9115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:42:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:42:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:42:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:42:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:42:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:42:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:42:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:42:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:42:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:42:59 smithi188 ceph-mon[92425]: pgmap v9116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:42:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:42:59 smithi161 ceph-mon[91579]: pgmap v9116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:01 smithi188 ceph-mon[92425]: pgmap v9117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:01 smithi161 ceph-mon[91579]: pgmap v9117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:03 smithi188 ceph-mon[92425]: pgmap v9118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:03 smithi161 ceph-mon[91579]: pgmap v9118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:05 smithi188 ceph-mon[92425]: pgmap v9119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:05 smithi161 ceph-mon[91579]: pgmap v9119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:07 smithi188 ceph-mon[92425]: pgmap v9120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:07 smithi161 ceph-mon[91579]: pgmap v9120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:09 smithi188 ceph-mon[92425]: pgmap v9121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:09 smithi161 ceph-mon[91579]: pgmap v9121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:11 smithi161 ceph-mon[91579]: pgmap v9122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:11 smithi188 ceph-mon[92425]: pgmap v9122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:13 smithi161 ceph-mon[91579]: pgmap v9123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:13 smithi188 ceph-mon[92425]: pgmap v9123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:15 smithi161 ceph-mon[91579]: pgmap v9124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:15 smithi188 ceph-mon[92425]: pgmap v9124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:17 smithi161 ceph-mon[91579]: pgmap v9125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:17 smithi188 ceph-mon[92425]: pgmap v9125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:19 smithi161 ceph-mon[91579]: pgmap v9126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:19 smithi188 ceph-mon[92425]: pgmap v9126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:21 smithi161 ceph-mon[91579]: pgmap v9127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:21 smithi188 ceph-mon[92425]: pgmap v9127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:21.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:23 smithi161 ceph-mon[91579]: pgmap v9128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:23 smithi188 ceph-mon[92425]: pgmap v9128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:25 smithi161 ceph-mon[91579]: pgmap v9129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:25 smithi188 ceph-mon[92425]: pgmap v9129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:27 smithi161 ceph-mon[91579]: pgmap v9130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:27 smithi188 ceph-mon[92425]: pgmap v9130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:29 smithi161 ceph-mon[91579]: pgmap v9131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:29 smithi188 ceph-mon[92425]: pgmap v9131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:31 smithi161 ceph-mon[91579]: pgmap v9132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:31 smithi188 ceph-mon[92425]: pgmap v9132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:33 smithi161 ceph-mon[91579]: pgmap v9133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:33 smithi188 ceph-mon[92425]: pgmap v9133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:35 smithi161 ceph-mon[91579]: pgmap v9134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:35 smithi188 ceph-mon[92425]: pgmap v9134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:37 smithi161 ceph-mon[91579]: pgmap v9135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:37 smithi188 ceph-mon[92425]: pgmap v9135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:39 smithi161 ceph-mon[91579]: pgmap v9136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:39 smithi188 ceph-mon[92425]: pgmap v9136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:41 smithi161 ceph-mon[91579]: pgmap v9137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:41 smithi188 ceph-mon[92425]: pgmap v9137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:43 smithi161 ceph-mon[91579]: pgmap v9138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:43 smithi188 ceph-mon[92425]: pgmap v9138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:43:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:43:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:43:44.828 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:43:44.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:43:44.829 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:43:44.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:43:44.380+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:43:44.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:43:44.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:43:44.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:43:44.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:43:44.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:43:44.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:43:44.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:43:44.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:43:44.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:43:44.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:43:44.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:43:44.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:43:44.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:43:44.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:43:44.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:43:44.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:43:44.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:43:44.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:43:44.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:43:44.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:43:44.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:43:44.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:43:44.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:43:44.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:43:44.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:43:44.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:43:44.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:43:44.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:43:44.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:43:44.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:43:44.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:43:44.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:43:44.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:43:44.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:43:44.842 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:43:44.842 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:43:44.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:43:44.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:43:44.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:43:44.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:43:44.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:43:44.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:43:44.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:43:44.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:43:44.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=511] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:43:44.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=511] Local address: 172.21.15.161, port 55976 2022-07-03T15:43:44.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=511] Peer address: 172.21.15.101, port 22 2022-07-03T15:43:44.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=511] Connection lost 2022-07-03T15:43:44.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=511] Aborting connection 2022-07-03T15:43:44.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:43:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:43:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: pgmap v9139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:43:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:43:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:43:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:43:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:43:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:43:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:43:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:43:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:43:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:43:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:43:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:43:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:43:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:43:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:43:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:43:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:43:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:43:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:43:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:43:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:43:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:43:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:43:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=511] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=511] Local address: 172.21.15.161, port 55976 2022-07-03T15:43:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=511] Peer address: 172.21.15.101, port 22 2022-07-03T15:43:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=511] Connection lost 2022-07-03T15:43:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=511] Aborting connection 2022-07-03T15:43:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: pgmap v9139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:43:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:43:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:43:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:43:45.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:43:45.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:43:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:43:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:43:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:43:45.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:43:45.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:43:45.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:43:45.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:43:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:43:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:43:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:43:45.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:43:45.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:43:45.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:43:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:43:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:43:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:43:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:43:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:43:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:43:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:43:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:43:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:43:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:43:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:43:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:43:45.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:43:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:43:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:43:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:43:45.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:43:45.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:43:45.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:43:45.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:43:45.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:43:45.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:43:45.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:43:45.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:43:45.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:43:45.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:43:45.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:43:45.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=511] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:43:45.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=511] Local address: 172.21.15.161, port 55976 2022-07-03T15:43:45.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=511] Peer address: 172.21.15.101, port 22 2022-07-03T15:43:45.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=511] Connection lost 2022-07-03T15:43:45.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=511] Aborting connection 2022-07-03T15:43:45.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:45.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:47.775 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:47 smithi188 ceph-mon[92425]: pgmap v9140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:47 smithi161 ceph-mon[91579]: pgmap v9140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:49 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T15:43:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:49 smithi188 ceph-mon[92425]: pgmap v9141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:49 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T15:43:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:43:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:49 smithi161 ceph-mon[91579]: pgmap v9141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:51 smithi161 ceph-mon[91579]: pgmap v9142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:51.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:51 smithi188 ceph-mon[92425]: pgmap v9142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:53 smithi161 ceph-mon[91579]: pgmap v9143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:53 smithi188 ceph-mon[92425]: pgmap v9143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:55 smithi161 ceph-mon[91579]: pgmap v9144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:55 smithi188 ceph-mon[92425]: pgmap v9144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:43:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:43:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:57 smithi161 ceph-mon[91579]: pgmap v9145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:57 smithi188 ceph-mon[92425]: pgmap v9145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:43:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:43:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:43:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:43:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:43:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:43:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:43:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:43:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:43:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:43:59 smithi161 ceph-mon[91579]: pgmap v9146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:43:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:43:59 smithi188 ceph-mon[92425]: pgmap v9146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:01 smithi161 ceph-mon[91579]: pgmap v9147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:01 smithi188 ceph-mon[92425]: pgmap v9147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:03 smithi161 ceph-mon[91579]: pgmap v9148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:03 smithi188 ceph-mon[92425]: pgmap v9148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:05 smithi161 ceph-mon[91579]: pgmap v9149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:05 smithi188 ceph-mon[92425]: pgmap v9149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:07 smithi161 ceph-mon[91579]: pgmap v9150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:07 smithi188 ceph-mon[92425]: pgmap v9150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:09 smithi161 ceph-mon[91579]: pgmap v9151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:09 smithi188 ceph-mon[92425]: pgmap v9151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:11 smithi161 ceph-mon[91579]: pgmap v9152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:11 smithi188 ceph-mon[92425]: pgmap v9152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:13 smithi161 ceph-mon[91579]: pgmap v9153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:13 smithi188 ceph-mon[92425]: pgmap v9153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:15 smithi161 ceph-mon[91579]: pgmap v9154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:15 smithi188 ceph-mon[92425]: pgmap v9154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:17 smithi161 ceph-mon[91579]: pgmap v9155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:17 smithi188 ceph-mon[92425]: pgmap v9155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:19 smithi161 ceph-mon[91579]: pgmap v9156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:19 smithi188 ceph-mon[92425]: pgmap v9156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:21 smithi161 ceph-mon[91579]: pgmap v9157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:21 smithi188 ceph-mon[92425]: pgmap v9157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:23 smithi161 ceph-mon[91579]: pgmap v9158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:23 smithi188 ceph-mon[92425]: pgmap v9158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:25 smithi161 ceph-mon[91579]: pgmap v9159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:25 smithi188 ceph-mon[92425]: pgmap v9159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:27 smithi161 ceph-mon[91579]: pgmap v9160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:27 smithi188 ceph-mon[92425]: pgmap v9160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:29 smithi161 ceph-mon[91579]: pgmap v9161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:29 smithi188 ceph-mon[92425]: pgmap v9161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:31 smithi188 ceph-mon[92425]: pgmap v9162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:31 smithi161 ceph-mon[91579]: pgmap v9162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:33 smithi188 ceph-mon[92425]: pgmap v9163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:33 smithi161 ceph-mon[91579]: pgmap v9163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:35 smithi188 ceph-mon[92425]: pgmap v9164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:35 smithi161 ceph-mon[91579]: pgmap v9164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:37 smithi188 ceph-mon[92425]: pgmap v9165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:37 smithi161 ceph-mon[91579]: pgmap v9165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:39 smithi188 ceph-mon[92425]: pgmap v9166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:39 smithi161 ceph-mon[91579]: pgmap v9166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:41 smithi188 ceph-mon[92425]: pgmap v9167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:41 smithi161 ceph-mon[91579]: pgmap v9167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:43 smithi188 ceph-mon[92425]: pgmap v9168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:43 smithi161 ceph-mon[91579]: pgmap v9168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:45 smithi188 ceph-mon[92425]: pgmap v9169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:45 smithi161 ceph-mon[91579]: pgmap v9169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:47 smithi188 ceph-mon[92425]: pgmap v9170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:47 smithi161 ceph-mon[91579]: pgmap v9170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:44:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:44:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:44:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:44:48.174+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:44:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:44:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:44:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:44:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:44:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:44:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:44:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:44:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:44:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:44:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:44:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:44:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:44:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:44:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:44:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:44:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:44:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:44:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:44:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:44:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:44:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:44:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:44:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:44:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:44:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:44:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:44:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:44:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:44:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:44:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:44:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:44:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:44:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:44:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:44:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:44:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:44:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:44:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:44:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:44:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:44:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:44:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:44:48.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:44:48.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:44:48.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=513] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:44:48.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=513] Local address: 172.21.15.161, port 56006 2022-07-03T15:44:48.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=513] Peer address: 172.21.15.101, port 22 2022-07-03T15:44:48.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=513] Connection lost 2022-07-03T15:44:48.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=513] Aborting connection 2022-07-03T15:44:48.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:44:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:44:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:44:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:44:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:44:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:44:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:44:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:44:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:44:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:44:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:44:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:44:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:44:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:44:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:44:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:44:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:44:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:44:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:44:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:44:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:44:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:44:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:44:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:44:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:44:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:44:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:44:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:44:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:44:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:44:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:44:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:44:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:44:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:44:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:44:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:44:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:44:49.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:44:49.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:44:49.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:44:49.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:44:49.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:44:49.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:44:49.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:44:49.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:44:49.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:44:49.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:44:49.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:44:49.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:44:49.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:44:49.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:44:49.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=513] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:44:49.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=513] Local address: 172.21.15.161, port 56006 2022-07-03T15:44:49.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=513] Peer address: 172.21.15.101, port 22 2022-07-03T15:44:49.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=513] Connection lost 2022-07-03T15:44:49.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=513] Aborting connection 2022-07-03T15:44:49.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: pgmap v9171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:49.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.906 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:44:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:44:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:44:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:44:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:44:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:44:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:44:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:44:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:44:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:44:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:44:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:44:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:44:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:44:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:44:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:44:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:44:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:44:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:44:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:44:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:44:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:44:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:44:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:44:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:44:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:44:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:44:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:44:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:44:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:44:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:44:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:44:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:44:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:44:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:44:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:44:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:44:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:44:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:44:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:44:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:44:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:44:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:44:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:44:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:44:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:44:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=513] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:44:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=513] Local address: 172.21.15.161, port 56006 2022-07-03T15:44:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=513] Peer address: 172.21.15.101, port 22 2022-07-03T15:44:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=513] Connection lost 2022-07-03T15:44:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=513] Aborting connection 2022-07-03T15:44:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: pgmap v9171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:49.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:44:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:51 smithi161 ceph-mon[91579]: pgmap v9172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:51 smithi188 ceph-mon[92425]: pgmap v9172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:53 smithi161 ceph-mon[91579]: pgmap v9173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:53 smithi188 ceph-mon[92425]: pgmap v9173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:55 smithi161 ceph-mon[91579]: pgmap v9174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:55 smithi188 ceph-mon[92425]: pgmap v9174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:44:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:44:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:57 smithi161 ceph-mon[91579]: pgmap v9175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:57 smithi188 ceph-mon[92425]: pgmap v9175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:44:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:44:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:44:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:44:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:44:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:44:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:44:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:44:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:44:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:44:59 smithi161 ceph-mon[91579]: pgmap v9176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:44:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:44:59 smithi188 ceph-mon[92425]: pgmap v9176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:01 smithi161 ceph-mon[91579]: pgmap v9177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:01 smithi188 ceph-mon[92425]: pgmap v9177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:03 smithi161 ceph-mon[91579]: pgmap v9178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:03 smithi188 ceph-mon[92425]: pgmap v9178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:05 smithi161 ceph-mon[91579]: pgmap v9179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:05 smithi188 ceph-mon[92425]: pgmap v9179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:07 smithi161 ceph-mon[91579]: pgmap v9180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:07 smithi188 ceph-mon[92425]: pgmap v9180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:09 smithi161 ceph-mon[91579]: pgmap v9181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:09 smithi188 ceph-mon[92425]: pgmap v9181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:11 smithi161 ceph-mon[91579]: pgmap v9182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:11 smithi188 ceph-mon[92425]: pgmap v9182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:13 smithi161 ceph-mon[91579]: pgmap v9183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:13 smithi188 ceph-mon[92425]: pgmap v9183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:15 smithi161 ceph-mon[91579]: pgmap v9184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:15 smithi188 ceph-mon[92425]: pgmap v9184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:17 smithi161 ceph-mon[91579]: pgmap v9185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:17 smithi188 ceph-mon[92425]: pgmap v9185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:19 smithi161 ceph-mon[91579]: pgmap v9186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:19 smithi188 ceph-mon[92425]: pgmap v9186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:21 smithi161 ceph-mon[91579]: pgmap v9187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:21 smithi188 ceph-mon[92425]: pgmap v9187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:23 smithi161 ceph-mon[91579]: pgmap v9188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:23 smithi188 ceph-mon[92425]: pgmap v9188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:25 smithi161 ceph-mon[91579]: pgmap v9189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:25 smithi188 ceph-mon[92425]: pgmap v9189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:27 smithi161 ceph-mon[91579]: pgmap v9190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:27 smithi188 ceph-mon[92425]: pgmap v9190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:29 smithi161 ceph-mon[91579]: pgmap v9191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:29 smithi188 ceph-mon[92425]: pgmap v9191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:31 smithi161 ceph-mon[91579]: pgmap v9192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:31 smithi188 ceph-mon[92425]: pgmap v9192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:33 smithi161 ceph-mon[91579]: pgmap v9193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:33 smithi188 ceph-mon[92425]: pgmap v9193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:35 smithi161 ceph-mon[91579]: pgmap v9194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:35 smithi188 ceph-mon[92425]: pgmap v9194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:37 smithi161 ceph-mon[91579]: pgmap v9195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:37 smithi188 ceph-mon[92425]: pgmap v9195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:39 smithi161 ceph-mon[91579]: pgmap v9196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:39 smithi188 ceph-mon[92425]: pgmap v9196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:41 smithi161 ceph-mon[91579]: pgmap v9197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:41 smithi188 ceph-mon[92425]: pgmap v9197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:43 smithi161 ceph-mon[91579]: pgmap v9198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:43 smithi188 ceph-mon[92425]: pgmap v9198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:45 smithi161 ceph-mon[91579]: pgmap v9199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:45 smithi188 ceph-mon[92425]: pgmap v9199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:47 smithi161 ceph-mon[91579]: pgmap v9200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:47 smithi188 ceph-mon[92425]: pgmap v9200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:49.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:45:49.093+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:45:49.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:45:49.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:45:49.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:45:49.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:45:49.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:45:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:45:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:45:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:45:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:45:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:45:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:45:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:45:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:45:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:45:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:45:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:45:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:45:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:45:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:45:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:45:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:45:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:45:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:45:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:45:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:45:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:45:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:45:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:45:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:45:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:45:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:45:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:45:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:45:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:45:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:45:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:45:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:45:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:45:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:45:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:45:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:45:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:45:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:45:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:45:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=515] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:45:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=515] Local address: 172.21.15.161, port 56034 2022-07-03T15:45:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=515] Peer address: 172.21.15.101, port 22 2022-07-03T15:45:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=515] Connection lost 2022-07-03T15:45:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=515] Aborting connection 2022-07-03T15:45:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:45:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-mon[91579]: pgmap v9201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:45:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:45:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:45:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:49 smithi188 ceph-mon[92425]: pgmap v9201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:45:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:45:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:45:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:45:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:45:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:45:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:45:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:45:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:45:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:45:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:45:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:45:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:45:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:45:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:45:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:45:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:45:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:45:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:45:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:45:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:45:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:45:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:45:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:45:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:45:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:45:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:45:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:45:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:45:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:45:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:45:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:45:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:45:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:45:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:45:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:45:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:45:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:45:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:45:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:45:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:45:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:45:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:45:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:45:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:45:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:45:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:45:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:45:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:45:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=515] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:45:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=515] Local address: 172.21.15.161, port 56034 2022-07-03T15:45:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=515] Peer address: 172.21.15.101, port 22 2022-07-03T15:45:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=515] Connection lost 2022-07-03T15:45:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=515] Aborting connection 2022-07-03T15:45:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:45:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:45:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:45:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:45:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:45:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:45:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:45:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:45:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:45:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:45:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:45:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:45:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:45:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:45:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:45:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:45:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:45:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:45:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:45:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:45:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:45:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:45:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:45:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:45:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:45:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:45:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:45:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:45:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:45:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:45:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:45:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:45:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:45:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:45:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:45:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:45:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:45:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:45:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:45:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:45:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:45:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:45:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:45:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:45:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:45:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:45:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=515] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:45:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=515] Local address: 172.21.15.161, port 56034 2022-07-03T15:45:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=515] Peer address: 172.21.15.101, port 22 2022-07-03T15:45:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=515] Connection lost 2022-07-03T15:45:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=515] Aborting connection 2022-07-03T15:45:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:51 smithi161 ceph-mon[91579]: pgmap v9202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:51 smithi188 ceph-mon[92425]: pgmap v9202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:53 smithi161 ceph-mon[91579]: pgmap v9203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:53 smithi188 ceph-mon[92425]: pgmap v9203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:55 smithi161 ceph-mon[91579]: pgmap v9204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:55 smithi188 ceph-mon[92425]: pgmap v9204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:45:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:45:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:57 smithi161 ceph-mon[91579]: pgmap v9205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:57 smithi188 ceph-mon[92425]: pgmap v9205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:45:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:45:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:45:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:45:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:45:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:45:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:45:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:45:59 smithi161 ceph-mon[91579]: pgmap v9206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:45:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:45:59 smithi188 ceph-mon[92425]: pgmap v9206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:01 smithi161 ceph-mon[91579]: pgmap v9207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:01 smithi188 ceph-mon[92425]: pgmap v9207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:03 smithi161 ceph-mon[91579]: pgmap v9208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:03 smithi188 ceph-mon[92425]: pgmap v9208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:05 smithi161 ceph-mon[91579]: pgmap v9209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:05 smithi188 ceph-mon[92425]: pgmap v9209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:07 smithi161 ceph-mon[91579]: pgmap v9210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:07 smithi188 ceph-mon[92425]: pgmap v9210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:09 smithi161 ceph-mon[91579]: pgmap v9211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:09 smithi188 ceph-mon[92425]: pgmap v9211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:11 smithi161 ceph-mon[91579]: pgmap v9212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:11 smithi188 ceph-mon[92425]: pgmap v9212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:13 smithi161 ceph-mon[91579]: pgmap v9213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:13 smithi188 ceph-mon[92425]: pgmap v9213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:15 smithi161 ceph-mon[91579]: pgmap v9214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:15 smithi188 ceph-mon[92425]: pgmap v9214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:17 smithi161 ceph-mon[91579]: pgmap v9215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:17 smithi188 ceph-mon[92425]: pgmap v9215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:19 smithi161 ceph-mon[91579]: pgmap v9216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:19 smithi188 ceph-mon[92425]: pgmap v9216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:21 smithi161 ceph-mon[91579]: pgmap v9217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:21 smithi188 ceph-mon[92425]: pgmap v9217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:23 smithi161 ceph-mon[91579]: pgmap v9218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:23 smithi188 ceph-mon[92425]: pgmap v9218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:25 smithi161 ceph-mon[91579]: pgmap v9219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:25 smithi188 ceph-mon[92425]: pgmap v9219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:27 smithi161 ceph-mon[91579]: pgmap v9220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:27 smithi188 ceph-mon[92425]: pgmap v9220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:29 smithi161 ceph-mon[91579]: pgmap v9221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:29 smithi188 ceph-mon[92425]: pgmap v9221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:31 smithi161 ceph-mon[91579]: pgmap v9222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:31 smithi188 ceph-mon[92425]: pgmap v9222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:33 smithi161 ceph-mon[91579]: pgmap v9223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:33 smithi188 ceph-mon[92425]: pgmap v9223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:35 smithi161 ceph-mon[91579]: pgmap v9224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:35 smithi188 ceph-mon[92425]: pgmap v9224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:37 smithi161 ceph-mon[91579]: pgmap v9225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:37 smithi188 ceph-mon[92425]: pgmap v9225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:39 smithi161 ceph-mon[91579]: pgmap v9226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:39 smithi188 ceph-mon[92425]: pgmap v9226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:41 smithi161 ceph-mon[91579]: pgmap v9227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:41 smithi188 ceph-mon[92425]: pgmap v9227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:43 smithi161 ceph-mon[91579]: pgmap v9228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:43 smithi188 ceph-mon[92425]: pgmap v9228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:45 smithi161 ceph-mon[91579]: pgmap v9229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:45 smithi188 ceph-mon[92425]: pgmap v9229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:47 smithi161 ceph-mon[91579]: pgmap v9230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:47 smithi188 ceph-mon[92425]: pgmap v9230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-mon[91579]: pgmap v9231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:46:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:46:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:46:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:46:49.597+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:46:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:46:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:46:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:46:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:46:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:46:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:46:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:46:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:46:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:46:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:46:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:46:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:46:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:46:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:46:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:46:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:46:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:46:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:46:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:46:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:46:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:46:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:46:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:46:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:46:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:46:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:46:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:46:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:46:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:46:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:46:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:46:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:46:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:46:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:46:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:46:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:46:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:46:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:46:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:46:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:46:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:46:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:46:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:46:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:46:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=517] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:46:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=517] Local address: 172.21.15.161, port 56062 2022-07-03T15:46:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=517] Peer address: 172.21.15.101, port 22 2022-07-03T15:46:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=517] Connection lost 2022-07-03T15:46:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=517] Aborting connection 2022-07-03T15:46:49.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:46:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:49 smithi188 ceph-mon[92425]: pgmap v9231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:46:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:46:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:46:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:46:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:46:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:46:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:46:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:46:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:46:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:46:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:46:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:46:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:46:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:46:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:46:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:46:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:46:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:46:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:46:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:46:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:46:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:46:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:46:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:46:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:46:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:46:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:46:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:46:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:46:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:46:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:46:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:46:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:46:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:46:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:46:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:46:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:46:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:46:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:46:50.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:46:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:46:50.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:46:50.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:46:50.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:46:50.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:46:50.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:46:50.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:46:50.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=517] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:46:50.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=517] Local address: 172.21.15.161, port 56062 2022-07-03T15:46:50.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=517] Peer address: 172.21.15.101, port 22 2022-07-03T15:46:50.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=517] Connection lost 2022-07-03T15:46:50.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=517] Aborting connection 2022-07-03T15:46:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:46:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:46:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:46:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:46:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:46:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:46:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:46:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:46:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:46:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:46:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:46:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:46:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:46:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:46:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:46:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:46:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:46:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:46:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:46:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:46:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:46:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:46:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:46:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:46:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:46:50.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:46:50.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:46:50.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:46:50.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:46:50.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:46:50.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:46:50.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:46:50.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:46:50.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:46:50.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:46:50.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:46:50.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:46:50.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:46:50.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:46:50.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:46:50.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:46:50.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:46:50.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:46:50.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:46:50.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:46:50.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:46:50.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=517] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:46:50.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=517] Local address: 172.21.15.161, port 56062 2022-07-03T15:46:50.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=517] Peer address: 172.21.15.101, port 22 2022-07-03T15:46:50.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=517] Connection lost 2022-07-03T15:46:50.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=517] Aborting connection 2022-07-03T15:46:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:51 smithi161 ceph-mon[91579]: pgmap v9232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:51 smithi188 ceph-mon[92425]: pgmap v9232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: pgmap v9233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 153.7M 2022-07-03T15:46:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 161211490: error parsing value: Value '161211490' is below minimum 939524096 2022-07-03T15:46:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:46:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:53 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:46:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:46:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: pgmap v9233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 153.7M 2022-07-03T15:46:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 161211490: error parsing value: Value '161211490' is below minimum 939524096 2022-07-03T15:46:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:46:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:46:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:53 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:46:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:55 smithi161 ceph-mon[91579]: pgmap v9234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:55 smithi188 ceph-mon[92425]: pgmap v9234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:46:56.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:46:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:57 smithi161 ceph-mon[91579]: pgmap v9235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:57 smithi188 ceph-mon[92425]: pgmap v9235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:46:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:46:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:46:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:46:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:46:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:46:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:46:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:46:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:46:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:46:59 smithi161 ceph-mon[91579]: pgmap v9236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:46:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:46:59 smithi188 ceph-mon[92425]: pgmap v9236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:01 smithi188 ceph-mon[92425]: pgmap v9237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:01 smithi161 ceph-mon[91579]: pgmap v9237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:03 smithi188 ceph-mon[92425]: pgmap v9238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:03 smithi161 ceph-mon[91579]: pgmap v9238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:05 smithi188 ceph-mon[92425]: pgmap v9239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:05 smithi161 ceph-mon[91579]: pgmap v9239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:07 smithi188 ceph-mon[92425]: pgmap v9240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:07 smithi161 ceph-mon[91579]: pgmap v9240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:09 smithi188 ceph-mon[92425]: pgmap v9241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:09 smithi161 ceph-mon[91579]: pgmap v9241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:11 smithi188 ceph-mon[92425]: pgmap v9242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:11 smithi161 ceph-mon[91579]: pgmap v9242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:13 smithi188 ceph-mon[92425]: pgmap v9243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:13 smithi161 ceph-mon[91579]: pgmap v9243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:15 smithi188 ceph-mon[92425]: pgmap v9244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:15 smithi161 ceph-mon[91579]: pgmap v9244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:16.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:17 smithi188 ceph-mon[92425]: pgmap v9245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:17 smithi161 ceph-mon[91579]: pgmap v9245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:19 smithi188 ceph-mon[92425]: pgmap v9246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:19 smithi161 ceph-mon[91579]: pgmap v9246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:21 smithi161 ceph-mon[91579]: pgmap v9247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:21 smithi188 ceph-mon[92425]: pgmap v9247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:23 smithi161 ceph-mon[91579]: pgmap v9248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:23 smithi188 ceph-mon[92425]: pgmap v9248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:25 smithi161 ceph-mon[91579]: pgmap v9249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:25 smithi188 ceph-mon[92425]: pgmap v9249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:27 smithi161 ceph-mon[91579]: pgmap v9250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:27 smithi188 ceph-mon[92425]: pgmap v9250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:29 smithi161 ceph-mon[91579]: pgmap v9251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:29 smithi188 ceph-mon[92425]: pgmap v9251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:31 smithi161 ceph-mon[91579]: pgmap v9252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:31 smithi188 ceph-mon[92425]: pgmap v9252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:33 smithi161 ceph-mon[91579]: pgmap v9253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:33 smithi188 ceph-mon[92425]: pgmap v9253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:35 smithi161 ceph-mon[91579]: pgmap v9254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:35 smithi188 ceph-mon[92425]: pgmap v9254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:37 smithi161 ceph-mon[91579]: pgmap v9255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:37 smithi188 ceph-mon[92425]: pgmap v9255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:39 smithi161 ceph-mon[91579]: pgmap v9256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:39 smithi188 ceph-mon[92425]: pgmap v9256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:41 smithi161 ceph-mon[91579]: pgmap v9257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:41 smithi188 ceph-mon[92425]: pgmap v9257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:43 smithi161 ceph-mon[91579]: pgmap v9258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:43 smithi188 ceph-mon[92425]: pgmap v9258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:45 smithi161 ceph-mon[91579]: pgmap v9259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:45 smithi188 ceph-mon[92425]: pgmap v9259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:47 smithi161 ceph-mon[91579]: pgmap v9260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:47 smithi188 ceph-mon[92425]: pgmap v9260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:49 smithi161 ceph-mon[91579]: pgmap v9261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:49 smithi188 ceph-mon[92425]: pgmap v9261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:51 smithi161 ceph-mon[91579]: pgmap v9262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:51 smithi188 ceph-mon[92425]: pgmap v9262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:47:52.747+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:47:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:47:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:47:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:47:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:47:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:47:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:47:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:47:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:47:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:47:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:47:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:47:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:47:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:47:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:47:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:47:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:47:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:47:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:47:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:47:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:47:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:47:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:47:53.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:47:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:47:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:47:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:47:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:47:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:47:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:47:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:47:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:47:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:47:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:47:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:47:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:47:53.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:47:53.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:47:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:47:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:47:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:47:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:47:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:47:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:47:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:47:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=519] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:47:53.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=519] Local address: 172.21.15.161, port 56092 2022-07-03T15:47:53.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=519] Peer address: 172.21.15.101, port 22 2022-07-03T15:47:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=519] Connection lost 2022-07-03T15:47:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=519] Aborting connection 2022-07-03T15:47:53.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:47:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:47:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: pgmap v9263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:47:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:47:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:47:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:47:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:47:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:47:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:47:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:47:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:47:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:47:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:47:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:47:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:47:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:47:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:47:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:47:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:47:53.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:47:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:47:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:47:53.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:47:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:47:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:47:53.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:47:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:47:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:47:53.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:47:53.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:47:53.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:47:53.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:47:53.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:47:53.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:47:53.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:47:53.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:47:53.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:47:53.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:47:53.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:47:53.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:47:53.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:47:53.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:47:53.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:47:53.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:47:53.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:47:53.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:47:53.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:47:53.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:47:53.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:47:53.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:47:53.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:47:53.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=519] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:47:53.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=519] Local address: 172.21.15.161, port 56092 2022-07-03T15:47:53.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=519] Peer address: 172.21.15.101, port 22 2022-07-03T15:47:53.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=519] Connection lost 2022-07-03T15:47:53.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=519] Aborting connection 2022-07-03T15:47:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: pgmap v9263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:47:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:47:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:47:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:47:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:47:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:47:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:47:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:47:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:47:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:47:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:47:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:47:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:47:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:47:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:47:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:47:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:47:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:47:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:47:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:47:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:47:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:47:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:47:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:47:53.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:47:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:47:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:47:53.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:47:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:47:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:47:53.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:47:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:47:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:47:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:47:53.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:47:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:47:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:47:53.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:47:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:47:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:47:53.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:47:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:47:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:47:53.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:47:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:47:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:47:53.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:47:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:47:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:47:53.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=519] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:47:53.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=519] Local address: 172.21.15.161, port 56092 2022-07-03T15:47:53.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=519] Peer address: 172.21.15.101, port 22 2022-07-03T15:47:53.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=519] Connection lost 2022-07-03T15:47:53.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=519] Aborting connection 2022-07-03T15:47:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:55 smithi161 ceph-mon[91579]: pgmap v9264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:55 smithi188 ceph-mon[92425]: pgmap v9264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:47:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:47:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:57 smithi161 ceph-mon[91579]: pgmap v9265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:57 smithi188 ceph-mon[92425]: pgmap v9265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:47:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:47:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:47:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:47:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:47:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:47:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:47:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:47:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:47:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:47:59 smithi161 ceph-mon[91579]: pgmap v9266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:47:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:47:59 smithi188 ceph-mon[92425]: pgmap v9266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:01 smithi161 ceph-mon[91579]: pgmap v9267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:01 smithi188 ceph-mon[92425]: pgmap v9267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:03 smithi161 ceph-mon[91579]: pgmap v9268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:03 smithi188 ceph-mon[92425]: pgmap v9268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:05 smithi161 ceph-mon[91579]: pgmap v9269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:05 smithi188 ceph-mon[92425]: pgmap v9269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:07 smithi161 ceph-mon[91579]: pgmap v9270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:07 smithi188 ceph-mon[92425]: pgmap v9270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:09 smithi161 ceph-mon[91579]: pgmap v9271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:09 smithi188 ceph-mon[92425]: pgmap v9271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:11 smithi161 ceph-mon[91579]: pgmap v9272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:11 smithi188 ceph-mon[92425]: pgmap v9272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:13 smithi161 ceph-mon[91579]: pgmap v9273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:13 smithi188 ceph-mon[92425]: pgmap v9273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:15 smithi161 ceph-mon[91579]: pgmap v9274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:15 smithi188 ceph-mon[92425]: pgmap v9274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:17 smithi161 ceph-mon[91579]: pgmap v9275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:17 smithi188 ceph-mon[92425]: pgmap v9275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:19 smithi161 ceph-mon[91579]: pgmap v9276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:19 smithi188 ceph-mon[92425]: pgmap v9276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:21 smithi161 ceph-mon[91579]: pgmap v9277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:21 smithi188 ceph-mon[92425]: pgmap v9277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:23 smithi161 ceph-mon[91579]: pgmap v9278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:23 smithi188 ceph-mon[92425]: pgmap v9278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:25 smithi161 ceph-mon[91579]: pgmap v9279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:25 smithi188 ceph-mon[92425]: pgmap v9279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:27 smithi161 ceph-mon[91579]: pgmap v9280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:27 smithi188 ceph-mon[92425]: pgmap v9280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:29 smithi161 ceph-mon[91579]: pgmap v9281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:29 smithi188 ceph-mon[92425]: pgmap v9281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:31 smithi161 ceph-mon[91579]: pgmap v9282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:31 smithi188 ceph-mon[92425]: pgmap v9282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:33 smithi161 ceph-mon[91579]: pgmap v9283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:33 smithi188 ceph-mon[92425]: pgmap v9283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:35 smithi161 ceph-mon[91579]: pgmap v9284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:35 smithi188 ceph-mon[92425]: pgmap v9284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:37 smithi161 ceph-mon[91579]: pgmap v9285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:37 smithi188 ceph-mon[92425]: pgmap v9285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:39 smithi161 ceph-mon[91579]: pgmap v9286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:39 smithi188 ceph-mon[92425]: pgmap v9286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:41 smithi161 ceph-mon[91579]: pgmap v9287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:41 smithi188 ceph-mon[92425]: pgmap v9287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:43 smithi161 ceph-mon[91579]: pgmap v9288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:43 smithi188 ceph-mon[92425]: pgmap v9288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:45 smithi161 ceph-mon[91579]: pgmap v9289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:45 smithi188 ceph-mon[92425]: pgmap v9289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:46.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:47 smithi161 ceph-mon[91579]: pgmap v9290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:47 smithi188 ceph-mon[92425]: pgmap v9290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:49 smithi161 ceph-mon[91579]: pgmap v9291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:49 smithi188 ceph-mon[92425]: pgmap v9291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:51 smithi161 ceph-mon[91579]: pgmap v9292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:51 smithi188 ceph-mon[92425]: pgmap v9292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:53.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:48:53.251+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:48:53.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:48:53.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:48:53.565 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:48:53.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:48:53.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:48:53.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:48:53.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:48:53.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:48:53.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:48:53.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:48:53.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:48:53.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:48:53.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:48:53.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:48:53.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:48:53.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:48:53.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:48:53.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:48:53.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:48:53.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:48:53.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:48:53.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:48:53.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:48:53.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:48:53.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:48:53.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:48:53.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:48:53.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:48:53.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:48:53.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:48:53.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:48:53.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:48:53.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:48:53.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:48:53.577 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:48:53.577 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:48:53.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:48:53.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:48:53.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:48:53.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:48:53.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:48:53.580 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:48:53.580 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:48:53.581 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:48:53.581 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=521] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:48:53.582 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=521] Local address: 172.21.15.161, port 56120 2022-07-03T15:48:53.582 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=521] Peer address: 172.21.15.101, port 22 2022-07-03T15:48:53.582 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=521] Connection lost 2022-07-03T15:48:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=521] Aborting connection 2022-07-03T15:48:53.583 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:48:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-mon[91579]: pgmap v9293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:48:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:48:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:48:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:53 smithi188 ceph-mon[92425]: pgmap v9293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:48:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:48:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:48:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:48:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:48:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:48:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:48:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:48:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:48:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:48:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:48:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:48:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:48:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:48:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:48:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:48:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:48:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:48:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:48:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:48:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:48:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:48:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:48:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:48:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:48:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:48:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:48:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:48:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:48:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:48:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:48:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:48:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:48:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:48:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:48:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:48:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:48:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:48:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:48:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:48:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:48:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:48:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:48:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:48:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:48:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=521] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:48:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=521] Local address: 172.21.15.161, port 56120 2022-07-03T15:48:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=521] Peer address: 172.21.15.101, port 22 2022-07-03T15:48:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=521] Connection lost 2022-07-03T15:48:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=521] Aborting connection 2022-07-03T15:48:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:48:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:48:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:48:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:48:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:48:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:48:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:48:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:48:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:48:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:48:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:48:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:48:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:48:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:48:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:48:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:48:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:48:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:48:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:48:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:48:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:48:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:48:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:48:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:48:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:48:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:48:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:48:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:48:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:48:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:48:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:48:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:48:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:48:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:48:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:48:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:48:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:48:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:48:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:48:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:48:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:48:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:48:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:48:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:48:54.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:48:54.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:48:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=521] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:48:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=521] Local address: 172.21.15.161, port 56120 2022-07-03T15:48:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=521] Peer address: 172.21.15.101, port 22 2022-07-03T15:48:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=521] Connection lost 2022-07-03T15:48:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=521] Aborting connection 2022-07-03T15:48:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:55 smithi161 ceph-mon[91579]: pgmap v9294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:55 smithi188 ceph-mon[92425]: pgmap v9294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:48:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:48:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:57 smithi161 ceph-mon[91579]: pgmap v9295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:57 smithi188 ceph-mon[92425]: pgmap v9295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:48:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:48:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:48:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:48:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:48:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:48:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:48:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:48:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:48:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:48:59 smithi161 ceph-mon[91579]: pgmap v9296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:48:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:48:59 smithi188 ceph-mon[92425]: pgmap v9296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:01 smithi161 ceph-mon[91579]: pgmap v9297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:01 smithi188 ceph-mon[92425]: pgmap v9297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:03 smithi161 ceph-mon[91579]: pgmap v9298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:03 smithi188 ceph-mon[92425]: pgmap v9298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:05 smithi161 ceph-mon[91579]: pgmap v9299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:05 smithi188 ceph-mon[92425]: pgmap v9299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:07 smithi161 ceph-mon[91579]: pgmap v9300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:07 smithi188 ceph-mon[92425]: pgmap v9300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:09 smithi161 ceph-mon[91579]: pgmap v9301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:09 smithi188 ceph-mon[92425]: pgmap v9301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:11 smithi161 ceph-mon[91579]: pgmap v9302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:11 smithi188 ceph-mon[92425]: pgmap v9302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:13 smithi161 ceph-mon[91579]: pgmap v9303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:13 smithi188 ceph-mon[92425]: pgmap v9303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:15 smithi161 ceph-mon[91579]: pgmap v9304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:15 smithi188 ceph-mon[92425]: pgmap v9304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:17 smithi161 ceph-mon[91579]: pgmap v9305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:17 smithi188 ceph-mon[92425]: pgmap v9305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:19 smithi161 ceph-mon[91579]: pgmap v9306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:19 smithi188 ceph-mon[92425]: pgmap v9306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:21 smithi161 ceph-mon[91579]: pgmap v9307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:21 smithi188 ceph-mon[92425]: pgmap v9307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:23 smithi161 ceph-mon[91579]: pgmap v9308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:23 smithi188 ceph-mon[92425]: pgmap v9308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:25 smithi161 ceph-mon[91579]: pgmap v9309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:25 smithi188 ceph-mon[92425]: pgmap v9309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:27 smithi161 ceph-mon[91579]: pgmap v9310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:27 smithi188 ceph-mon[92425]: pgmap v9310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:29 smithi161 ceph-mon[91579]: pgmap v9311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:29 smithi188 ceph-mon[92425]: pgmap v9311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:31 smithi161 ceph-mon[91579]: pgmap v9312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:31 smithi188 ceph-mon[92425]: pgmap v9312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:33 smithi188 ceph-mon[92425]: pgmap v9313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:33 smithi161 ceph-mon[91579]: pgmap v9313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:35 smithi188 ceph-mon[92425]: pgmap v9314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:35 smithi161 ceph-mon[91579]: pgmap v9314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:36 smithi161 ceph-mon[91579]: pgmap v9315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:37.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:36 smithi188 ceph-mon[92425]: pgmap v9315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:39 smithi161 ceph-mon[91579]: pgmap v9316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:39 smithi188 ceph-mon[92425]: pgmap v9316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:41 smithi161 ceph-mon[91579]: pgmap v9317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:41 smithi188 ceph-mon[92425]: pgmap v9317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:43 smithi161 ceph-mon[91579]: pgmap v9318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:43 smithi188 ceph-mon[92425]: pgmap v9318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:45 smithi161 ceph-mon[91579]: pgmap v9319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:45 smithi188 ceph-mon[92425]: pgmap v9319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:46 smithi161 ceph-mon[91579]: pgmap v9320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:46 smithi188 ceph-mon[92425]: pgmap v9320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:49 smithi161 ceph-mon[91579]: pgmap v9321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:49 smithi188 ceph-mon[92425]: pgmap v9321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:51 smithi161 ceph-mon[91579]: pgmap v9322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:51 smithi188 ceph-mon[92425]: pgmap v9322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:53.756 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-mon[91579]: pgmap v9323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:53.764 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:53 smithi188 ceph-mon[92425]: pgmap v9323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:54.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:49:53.754+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:49:54.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:49:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:49:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:49:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:49:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:49:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:49:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:49:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:49:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:49:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:49:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:49:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:49:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:49:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:49:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:49:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:49:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:49:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:49:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:49:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:49:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:49:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:49:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:49:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:49:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:49:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:49:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:49:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:49:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:49:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:49:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:49:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:49:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:49:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:49:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:49:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:49:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:49:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:49:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:49:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:49:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:49:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:49:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:49:54.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:49:54.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=523] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:49:54.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=523] Local address: 172.21.15.161, port 56148 2022-07-03T15:49:54.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=523] Peer address: 172.21.15.101, port 22 2022-07-03T15:49:54.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=523] Connection lost 2022-07-03T15:49:54.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=523] Aborting connection 2022-07-03T15:49:54.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:49:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:49:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:49:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:49:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:49:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:49:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:49:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:49:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:49:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:49:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:49:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:49:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:49:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:49:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:49:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:49:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:49:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:49:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:49:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:49:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:49:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:49:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:49:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:49:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:49:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:49:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:49:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:49:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:49:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:49:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:49:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:49:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:49:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:49:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:49:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:49:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:49:54.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:49:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:49:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:49:54.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:49:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:49:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:49:54.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:49:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:49:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:49:54.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:49:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:49:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:49:54.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:49:54.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=523] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:49:54.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=523] Local address: 172.21.15.161, port 56148 2022-07-03T15:49:54.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=523] Peer address: 172.21.15.101, port 22 2022-07-03T15:49:54.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=523] Connection lost 2022-07-03T15:49:54.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=523] Aborting connection 2022-07-03T15:49:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:49:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:49:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:49:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:49:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:49:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:49:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:49:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:49:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:49:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:49:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:49:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:49:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:49:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:49:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:49:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:49:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:49:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:49:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:49:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:49:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:49:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:49:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:49:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:49:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:49:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:49:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:49:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:49:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:49:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:49:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:49:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:49:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:49:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:49:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:49:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:49:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:49:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:49:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:49:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:49:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:49:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:49:54.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:49:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:49:54.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:49:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:49:54.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:49:54.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:49:54.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:49:54.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=523] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:49:54.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=523] Local address: 172.21.15.161, port 56148 2022-07-03T15:49:54.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=523] Peer address: 172.21.15.101, port 22 2022-07-03T15:49:54.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=523] Connection lost 2022-07-03T15:49:54.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=523] Aborting connection 2022-07-03T15:49:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:55 smithi161 ceph-mon[91579]: pgmap v9324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:55 smithi188 ceph-mon[92425]: pgmap v9324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:56 smithi161 ceph-mon[91579]: pgmap v9325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:49:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:49:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:56 smithi188 ceph-mon[92425]: pgmap v9325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:49:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:49:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:49:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:49:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:49:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:49:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:49:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:49:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:49:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:49:59 smithi188 ceph-mon[92425]: pgmap v9326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:49:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:49:59 smithi161 ceph-mon[91579]: pgmap v9326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:50:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T15:50:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:01 smithi188 ceph-mon[92425]: pgmap v9327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:01 smithi161 ceph-mon[91579]: pgmap v9327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:03 smithi161 ceph-mon[91579]: pgmap v9328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:03 smithi188 ceph-mon[92425]: pgmap v9328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:05 smithi161 ceph-mon[91579]: pgmap v9329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:05 smithi188 ceph-mon[92425]: pgmap v9329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:06 smithi161 ceph-mon[91579]: pgmap v9330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:06 smithi188 ceph-mon[92425]: pgmap v9330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:09 smithi161 ceph-mon[91579]: pgmap v9331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:09 smithi188 ceph-mon[92425]: pgmap v9331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:11 smithi161 ceph-mon[91579]: pgmap v9332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:11 smithi188 ceph-mon[92425]: pgmap v9332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:13 smithi161 ceph-mon[91579]: pgmap v9333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:13 smithi188 ceph-mon[92425]: pgmap v9333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:15 smithi161 ceph-mon[91579]: pgmap v9334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:15.972 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:15 smithi188 ceph-mon[92425]: pgmap v9334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:16 smithi161 ceph-mon[91579]: pgmap v9335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:16 smithi188 ceph-mon[92425]: pgmap v9335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:19 smithi161 ceph-mon[91579]: pgmap v9336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:19 smithi188 ceph-mon[92425]: pgmap v9336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:21 smithi161 ceph-mon[91579]: pgmap v9337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:21 smithi188 ceph-mon[92425]: pgmap v9337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:23 smithi161 ceph-mon[91579]: pgmap v9338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:23 smithi188 ceph-mon[92425]: pgmap v9338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:25 smithi161 ceph-mon[91579]: pgmap v9339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:25 smithi188 ceph-mon[92425]: pgmap v9339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:26 smithi161 ceph-mon[91579]: pgmap v9340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:26 smithi188 ceph-mon[92425]: pgmap v9340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:29 smithi161 ceph-mon[91579]: pgmap v9341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:29 smithi188 ceph-mon[92425]: pgmap v9341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:31 smithi161 ceph-mon[91579]: pgmap v9342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:31 smithi188 ceph-mon[92425]: pgmap v9342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:33 smithi161 ceph-mon[91579]: pgmap v9343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:33 smithi188 ceph-mon[92425]: pgmap v9343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:35 smithi161 ceph-mon[91579]: pgmap v9344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:35 smithi188 ceph-mon[92425]: pgmap v9344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:36 smithi161 ceph-mon[91579]: pgmap v9345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:36 smithi188 ceph-mon[92425]: pgmap v9345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:39 smithi161 ceph-mon[91579]: pgmap v9346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:39 smithi188 ceph-mon[92425]: pgmap v9346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:41 smithi161 ceph-mon[91579]: pgmap v9347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:41 smithi188 ceph-mon[92425]: pgmap v9347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:43 smithi161 ceph-mon[91579]: pgmap v9348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:43 smithi188 ceph-mon[92425]: pgmap v9348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:45 smithi161 ceph-mon[91579]: pgmap v9349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:45 smithi188 ceph-mon[92425]: pgmap v9349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:46 smithi161 ceph-mon[91579]: pgmap v9350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:46 smithi188 ceph-mon[92425]: pgmap v9350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:49 smithi161 ceph-mon[91579]: pgmap v9351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:49 smithi188 ceph-mon[92425]: pgmap v9351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:51 smithi161 ceph-mon[91579]: pgmap v9352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:51 smithi188 ceph-mon[92425]: pgmap v9352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:53 smithi161 ceph-mon[91579]: pgmap v9353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:53 smithi188 ceph-mon[92425]: pgmap v9353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:50:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:50:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:50:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:50:54.253+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:50:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:50:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:50:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:50:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:50:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:50:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:50:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:50:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:50:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:50:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:50:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:50:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:50:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:50:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:50:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:50:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:50:54.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:50:54.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:50:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:50:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:50:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:50:54.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:50:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:50:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:50:54.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:50:54.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:50:54.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:50:54.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:50:54.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:50:54.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:50:54.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:50:54.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:50:54.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:50:54.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:50:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:50:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:50:54.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:50:54.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:50:54.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:50:54.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:50:54.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:50:54.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:50:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:50:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:50:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=525] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:50:54.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=525] Local address: 172.21.15.161, port 56176 2022-07-03T15:50:54.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=525] Peer address: 172.21.15.101, port 22 2022-07-03T15:50:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=525] Connection lost 2022-07-03T15:50:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=525] Aborting connection 2022-07-03T15:50:54.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:50:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:50:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:50:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:50:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:50:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:50:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:50:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:50:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:50:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:50:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:50:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:50:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:50:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:50:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:50:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:50:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:50:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:50:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:50:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:50:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:50:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:50:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:50:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:50:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:50:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:50:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:50:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:50:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:50:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:50:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:50:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:50:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:50:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:50:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:50:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:50:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:50:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:50:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:50:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:50:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:50:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:50:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:50:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:50:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:50:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:50:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:50:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:50:55.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:50:55.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:50:55.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:50:55.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=525] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:50:55.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=525] Local address: 172.21.15.161, port 56176 2022-07-03T15:50:55.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=525] Peer address: 172.21.15.101, port 22 2022-07-03T15:50:55.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=525] Connection lost 2022-07-03T15:50:55.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=525] Aborting connection 2022-07-03T15:50:55.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:55 smithi161 ceph-mon[91579]: pgmap v9354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:50:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:50:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:50:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:50:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:50:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:50:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:50:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:50:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:50:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:50:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:50:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:50:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:50:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:50:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:50:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:50:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:50:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:50:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:50:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:50:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:50:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:50:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:50:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:50:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:50:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:50:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:50:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:50:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:50:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:50:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:50:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:50:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:50:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:50:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:50:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:50:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:50:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:50:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:50:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:50:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:50:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:50:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:50:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:50:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:50:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:50:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=525] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:50:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=525] Local address: 172.21.15.161, port 56176 2022-07-03T15:50:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=525] Peer address: 172.21.15.101, port 22 2022-07-03T15:50:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=525] Connection lost 2022-07-03T15:50:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=525] Aborting connection 2022-07-03T15:50:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:55 smithi188 ceph-mon[92425]: pgmap v9354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:56 smithi161 ceph-mon[91579]: pgmap v9355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:50:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:50:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:56 smithi188 ceph-mon[92425]: pgmap v9355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:50:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:50:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:50:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:50:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:50:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:50:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:50:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:50:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:50:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:50:59 smithi188 ceph-mon[92425]: pgmap v9356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:50:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:50:59 smithi161 ceph-mon[91579]: pgmap v9356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:01 smithi161 ceph-mon[91579]: pgmap v9357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:01 smithi188 ceph-mon[92425]: pgmap v9357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:03 smithi161 ceph-mon[91579]: pgmap v9358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:03 smithi188 ceph-mon[92425]: pgmap v9358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:05 smithi161 ceph-mon[91579]: pgmap v9359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:05 smithi188 ceph-mon[92425]: pgmap v9359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:06 smithi161 ceph-mon[91579]: pgmap v9360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:06 smithi188 ceph-mon[92425]: pgmap v9360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:09 smithi161 ceph-mon[91579]: pgmap v9361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:09 smithi188 ceph-mon[92425]: pgmap v9361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:11 smithi161 ceph-mon[91579]: pgmap v9362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:11 smithi188 ceph-mon[92425]: pgmap v9362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:13 smithi161 ceph-mon[91579]: pgmap v9363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:13 smithi188 ceph-mon[92425]: pgmap v9363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:15 smithi161 ceph-mon[91579]: pgmap v9364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:15 smithi188 ceph-mon[92425]: pgmap v9364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:16 smithi161 ceph-mon[91579]: pgmap v9365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:16 smithi188 ceph-mon[92425]: pgmap v9365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:19 smithi161 ceph-mon[91579]: pgmap v9366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:19 smithi188 ceph-mon[92425]: pgmap v9366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:21 smithi161 ceph-mon[91579]: pgmap v9367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:21 smithi188 ceph-mon[92425]: pgmap v9367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:23 smithi161 ceph-mon[91579]: pgmap v9368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:23 smithi188 ceph-mon[92425]: pgmap v9368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:25 smithi161 ceph-mon[91579]: pgmap v9369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:25 smithi188 ceph-mon[92425]: pgmap v9369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:26 smithi161 ceph-mon[91579]: pgmap v9370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:26 smithi188 ceph-mon[92425]: pgmap v9370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:29 smithi161 ceph-mon[91579]: pgmap v9371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:29 smithi188 ceph-mon[92425]: pgmap v9371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:31 smithi161 ceph-mon[91579]: pgmap v9372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:31 smithi188 ceph-mon[92425]: pgmap v9372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:33 smithi161 ceph-mon[91579]: pgmap v9373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:33 smithi188 ceph-mon[92425]: pgmap v9373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:35 smithi161 ceph-mon[91579]: pgmap v9374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:35 smithi188 ceph-mon[92425]: pgmap v9374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:36 smithi161 ceph-mon[91579]: pgmap v9375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:36 smithi188 ceph-mon[92425]: pgmap v9375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:39 smithi161 ceph-mon[91579]: pgmap v9376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:39 smithi188 ceph-mon[92425]: pgmap v9376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:41 smithi161 ceph-mon[91579]: pgmap v9377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:41 smithi188 ceph-mon[92425]: pgmap v9377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:43 smithi161 ceph-mon[91579]: pgmap v9378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:43 smithi188 ceph-mon[92425]: pgmap v9378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:45 smithi161 ceph-mon[91579]: pgmap v9379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:45 smithi188 ceph-mon[92425]: pgmap v9379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:46 smithi161 ceph-mon[91579]: pgmap v9380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:46 smithi188 ceph-mon[92425]: pgmap v9380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T15:51:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:49 smithi161 ceph-mon[91579]: pgmap v9381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:49 smithi188 ceph-mon[92425]: pgmap v9381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:51 smithi161 ceph-mon[91579]: pgmap v9382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:51 smithi188 ceph-mon[92425]: pgmap v9382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:53 smithi161 ceph-mon[91579]: pgmap v9383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:53 smithi188 ceph-mon[92425]: pgmap v9383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:55.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:51:54.760+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:51:55.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:51:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:51:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:51:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:51:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:51:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:51:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:51:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:51:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:51:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:51:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:51:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:51:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:51:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:51:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:51:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:51:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:51:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:51:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:51:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:51:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:51:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:51:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:51:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:51:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:51:55.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:51:55.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:51:55.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:51:55.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:51:55.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:51:55.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:51:55.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:51:55.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:51:55.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:51:55.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:51:55.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:51:55.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:51:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:51:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:51:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:51:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:51:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:51:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:51:55.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:51:55.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=527] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:51:55.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=527] Local address: 172.21.15.161, port 56204 2022-07-03T15:51:55.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=527] Peer address: 172.21.15.101, port 22 2022-07-03T15:51:55.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=527] Connection lost 2022-07-03T15:51:55.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=527] Aborting connection 2022-07-03T15:51:55.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:51:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: pgmap v9384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:51:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:51:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:51:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:51:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:51:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:51:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:51:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:51:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:51:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:51:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:51:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:51:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:51:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:51:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:51:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:51:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:51:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:51:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:51:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:51:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:51:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:51:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:51:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:51:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:51:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:51:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:51:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:51:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:51:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:51:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:51:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:51:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:51:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:51:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:51:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:51:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:51:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:51:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:51:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:51:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:51:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:51:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:51:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:51:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:51:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:51:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:51:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:51:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:51:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=527] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:51:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=527] Local address: 172.21.15.161, port 56204 2022-07-03T15:51:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=527] Peer address: 172.21.15.101, port 22 2022-07-03T15:51:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=527] Connection lost 2022-07-03T15:51:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=527] Aborting connection 2022-07-03T15:51:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: pgmap v9384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:51:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:51:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:51:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:51:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:51:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:51:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:51:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:51:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:51:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:51:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:51:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:51:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:51:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:51:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:51:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:51:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:51:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:51:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:51:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:51:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:51:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:51:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:51:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:51:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:51:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:51:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:51:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:51:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:51:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:51:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:51:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:51:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:51:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:51:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:51:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:51:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:51:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:51:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:51:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:51:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:51:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:51:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:51:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:51:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:51:55.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:51:55.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:51:55.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:51:55.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:51:55.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=527] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:51:55.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=527] Local address: 172.21.15.161, port 56204 2022-07-03T15:51:55.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=527] Peer address: 172.21.15.101, port 22 2022-07-03T15:51:55.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=527] Connection lost 2022-07-03T15:51:55.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=527] Aborting connection 2022-07-03T15:51:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:56 smithi161 ceph-mon[91579]: pgmap v9385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:51:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:51:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:56 smithi188 ceph-mon[92425]: pgmap v9385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:51:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:51:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:51:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:51:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:51:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:51:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:51:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:51:59 smithi188 ceph-mon[92425]: pgmap v9386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:51:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:51:59 smithi161 ceph-mon[91579]: pgmap v9386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:01 smithi161 ceph-mon[91579]: pgmap v9387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:01 smithi188 ceph-mon[92425]: pgmap v9387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:03 smithi161 ceph-mon[91579]: pgmap v9388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:03 smithi188 ceph-mon[92425]: pgmap v9388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:05 smithi161 ceph-mon[91579]: pgmap v9389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:05 smithi188 ceph-mon[92425]: pgmap v9389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:06 smithi161 ceph-mon[91579]: pgmap v9390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:06 smithi188 ceph-mon[92425]: pgmap v9390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:09 smithi161 ceph-mon[91579]: pgmap v9391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:09 smithi188 ceph-mon[92425]: pgmap v9391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:11 smithi161 ceph-mon[91579]: pgmap v9392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:11 smithi188 ceph-mon[92425]: pgmap v9392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:13 smithi161 ceph-mon[91579]: pgmap v9393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:13 smithi188 ceph-mon[92425]: pgmap v9393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:15 smithi161 ceph-mon[91579]: pgmap v9394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:15 smithi188 ceph-mon[92425]: pgmap v9394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:16 smithi161 ceph-mon[91579]: pgmap v9395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:16 smithi188 ceph-mon[92425]: pgmap v9395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:19 smithi161 ceph-mon[91579]: pgmap v9396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:19 smithi188 ceph-mon[92425]: pgmap v9396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:21 smithi161 ceph-mon[91579]: pgmap v9397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:21 smithi188 ceph-mon[92425]: pgmap v9397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:23 smithi161 ceph-mon[91579]: pgmap v9398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:23 smithi188 ceph-mon[92425]: pgmap v9398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:25 smithi161 ceph-mon[91579]: pgmap v9399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:25 smithi188 ceph-mon[92425]: pgmap v9399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:26 smithi161 ceph-mon[91579]: pgmap v9400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:26 smithi188 ceph-mon[92425]: pgmap v9400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:29 smithi161 ceph-mon[91579]: pgmap v9401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:29 smithi188 ceph-mon[92425]: pgmap v9401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:31 smithi161 ceph-mon[91579]: pgmap v9402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:31 smithi188 ceph-mon[92425]: pgmap v9402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:33 smithi161 ceph-mon[91579]: pgmap v9403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:33 smithi188 ceph-mon[92425]: pgmap v9403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:35 smithi161 ceph-mon[91579]: pgmap v9404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:35 smithi188 ceph-mon[92425]: pgmap v9404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:37.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:36 smithi161 ceph-mon[91579]: pgmap v9405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:37.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:36 smithi188 ceph-mon[92425]: pgmap v9405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:39 smithi161 ceph-mon[91579]: pgmap v9406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:39 smithi188 ceph-mon[92425]: pgmap v9406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:41 smithi161 ceph-mon[91579]: pgmap v9407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:41 smithi188 ceph-mon[92425]: pgmap v9407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:43 smithi161 ceph-mon[91579]: pgmap v9408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:43 smithi188 ceph-mon[92425]: pgmap v9408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:45 smithi161 ceph-mon[91579]: pgmap v9409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:45 smithi188 ceph-mon[92425]: pgmap v9409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:46 smithi161 ceph-mon[91579]: pgmap v9410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:46 smithi188 ceph-mon[92425]: pgmap v9410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:49 smithi161 ceph-mon[91579]: pgmap v9411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:49 smithi188 ceph-mon[92425]: pgmap v9411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:51 smithi161 ceph-mon[91579]: pgmap v9412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:51 smithi188 ceph-mon[92425]: pgmap v9412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:53 smithi161 ceph-mon[91579]: pgmap v9413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:53 smithi188 ceph-mon[92425]: pgmap v9413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-mon[91579]: pgmap v9414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:52:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:52:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:52:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:52:55.259+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:52:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:52:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:52:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:52:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:52:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:52:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:52:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:52:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:52:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:52:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:52:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:52:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:52:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:52:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:52:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:52:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:52:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:52:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:52:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:52:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:52:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:52:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:52:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:52:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:52:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:52:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:52:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:52:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:52:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:52:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:52:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:52:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:52:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:52:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:52:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:52:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:52:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:52:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:52:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:52:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:52:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:52:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:52:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:52:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:52:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=529] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:52:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=529] Local address: 172.21.15.161, port 56232 2022-07-03T15:52:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=529] Peer address: 172.21.15.101, port 22 2022-07-03T15:52:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=529] Connection lost 2022-07-03T15:52:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=529] Aborting connection 2022-07-03T15:52:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:52:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:52:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:55 smithi188 ceph-mon[92425]: pgmap v9414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:52:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:52:55.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:52:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:52:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:52:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:52:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:52:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:52:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:52:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:52:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:52:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:52:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:52:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:52:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:52:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:52:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:52:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:52:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:52:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:52:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:52:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:52:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:52:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:52:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:52:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:52:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:52:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:52:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:52:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:52:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:52:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:52:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:52:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:52:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:52:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:52:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:52:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:52:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:52:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:52:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:52:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:52:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:52:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:52:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:52:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:52:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:52:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:52:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:52:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=529] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:52:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=529] Local address: 172.21.15.161, port 56232 2022-07-03T15:52:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=529] Peer address: 172.21.15.101, port 22 2022-07-03T15:52:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=529] Connection lost 2022-07-03T15:52:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=529] Aborting connection 2022-07-03T15:52:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:52:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:52:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:52:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:52:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:52:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:52:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:52:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:52:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:52:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:52:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:52:56.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:52:56.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:52:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:52:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:52:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:52:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:52:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:52:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:52:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:52:56.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:52:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:52:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:52:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:52:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:52:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:52:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:52:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:52:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:52:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:52:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:52:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:52:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:52:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:52:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:52:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:52:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:52:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:52:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:52:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:52:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:52:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:52:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:52:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:52:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:52:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:52:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=529] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:52:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=529] Local address: 172.21.15.161, port 56232 2022-07-03T15:52:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=529] Peer address: 172.21.15.101, port 22 2022-07-03T15:52:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=529] Connection lost 2022-07-03T15:52:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=529] Aborting connection 2022-07-03T15:52:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:52:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:52:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:57 smithi161 ceph-mon[91579]: pgmap v9415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:57 smithi188 ceph-mon[92425]: pgmap v9415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:52:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:52:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:52:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:52:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:52:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:52:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:52:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:52:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:52:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:52:59 smithi161 ceph-mon[91579]: pgmap v9416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:52:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:52:59 smithi188 ceph-mon[92425]: pgmap v9416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:01 smithi161 ceph-mon[91579]: pgmap v9417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:01 smithi188 ceph-mon[92425]: pgmap v9417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:03 smithi161 ceph-mon[91579]: pgmap v9418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:03 smithi188 ceph-mon[92425]: pgmap v9418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:05 smithi161 ceph-mon[91579]: pgmap v9419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:05 smithi188 ceph-mon[92425]: pgmap v9419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:06 smithi161 ceph-mon[91579]: pgmap v9420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:06 smithi188 ceph-mon[92425]: pgmap v9420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:09 smithi161 ceph-mon[91579]: pgmap v9421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:09 smithi188 ceph-mon[92425]: pgmap v9421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:11 smithi161 ceph-mon[91579]: pgmap v9422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:11.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:11 smithi188 ceph-mon[92425]: pgmap v9422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:11.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:13 smithi161 ceph-mon[91579]: pgmap v9423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:13 smithi188 ceph-mon[92425]: pgmap v9423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:15 smithi161 ceph-mon[91579]: pgmap v9424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:15 smithi188 ceph-mon[92425]: pgmap v9424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:17.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:16 smithi161 ceph-mon[91579]: pgmap v9425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:16 smithi188 ceph-mon[92425]: pgmap v9425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:19 smithi161 ceph-mon[91579]: pgmap v9426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:19 smithi188 ceph-mon[92425]: pgmap v9426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:21 smithi161 ceph-mon[91579]: pgmap v9427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:21 smithi188 ceph-mon[92425]: pgmap v9427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:23 smithi161 ceph-mon[91579]: pgmap v9428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:23 smithi188 ceph-mon[92425]: pgmap v9428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:25 smithi161 ceph-mon[91579]: pgmap v9429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:25 smithi188 ceph-mon[92425]: pgmap v9429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:26 smithi161 ceph-mon[91579]: pgmap v9430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:26 smithi188 ceph-mon[92425]: pgmap v9430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:29 smithi161 ceph-mon[91579]: pgmap v9431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:29 smithi188 ceph-mon[92425]: pgmap v9431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:31 smithi161 ceph-mon[91579]: pgmap v9432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:31 smithi188 ceph-mon[92425]: pgmap v9432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:33 smithi161 ceph-mon[91579]: pgmap v9433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:33 smithi188 ceph-mon[92425]: pgmap v9433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:35 smithi161 ceph-mon[91579]: pgmap v9434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:35 smithi188 ceph-mon[92425]: pgmap v9434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:36 smithi161 ceph-mon[91579]: pgmap v9435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:36 smithi188 ceph-mon[92425]: pgmap v9435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:39 smithi161 ceph-mon[91579]: pgmap v9436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:39 smithi188 ceph-mon[92425]: pgmap v9436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:41 smithi161 ceph-mon[91579]: pgmap v9437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:41 smithi188 ceph-mon[92425]: pgmap v9437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:43 smithi161 ceph-mon[91579]: pgmap v9438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:43 smithi188 ceph-mon[92425]: pgmap v9438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:45 smithi161 ceph-mon[91579]: pgmap v9439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:45 smithi188 ceph-mon[92425]: pgmap v9439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:46 smithi161 ceph-mon[91579]: pgmap v9440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:46 smithi188 ceph-mon[92425]: pgmap v9440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:49 smithi161 ceph-mon[91579]: pgmap v9441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:49 smithi188 ceph-mon[92425]: pgmap v9441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:51 smithi161 ceph-mon[91579]: pgmap v9442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:51 smithi188 ceph-mon[92425]: pgmap v9442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:53 smithi161 ceph-mon[91579]: pgmap v9443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:53 smithi188 ceph-mon[92425]: pgmap v9443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:55.760 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-mon[91579]: pgmap v9444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:55.769 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:55 smithi188 ceph-mon[92425]: pgmap v9444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:56.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:53:55.757+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:53:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:53:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:53:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:53:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:53:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:53:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:53:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:53:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:53:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:53:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:53:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:53:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:53:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:53:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:53:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:53:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:53:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:53:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:53:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:53:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:53:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:53:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:53:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:53:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:53:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:53:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:53:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:53:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:53:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:53:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:53:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:53:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:53:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:53:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:53:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:53:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:53:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:53:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:53:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:53:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:53:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:53:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:53:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:53:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:53:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=531] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:53:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=531] Local address: 172.21.15.161, port 56260 2022-07-03T15:53:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=531] Peer address: 172.21.15.101, port 22 2022-07-03T15:53:56.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=531] Connection lost 2022-07-03T15:53:56.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=531] Aborting connection 2022-07-03T15:53:56.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:53:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:53:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:53:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:53:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:53:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:53:56.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:53:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:53:56.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:53:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:53:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:53:56.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:53:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:53:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:53:56.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:53:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:53:56.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:53:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:53:56.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:53:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:53:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:53:56.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:53:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:53:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:53:56.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:53:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:53:56.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:53:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:53:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:53:56.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:53:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:53:56.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:53:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:53:56.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:53:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:53:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:53:56.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:53:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:53:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:53:56.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:53:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:53:56.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:53:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:53:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:53:56.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:53:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:53:56.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:53:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:53:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:53:56.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:53:56.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=531] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:53:56.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=531] Local address: 172.21.15.161, port 56260 2022-07-03T15:53:56.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=531] Peer address: 172.21.15.101, port 22 2022-07-03T15:53:56.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=531] Connection lost 2022-07-03T15:53:56.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=531] Aborting connection 2022-07-03T15:53:56.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:56.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:56.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:53:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:53:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:53:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:53:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:53:56.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:53:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:53:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:53:56.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:53:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:53:56.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:53:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:53:56.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:53:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:53:56.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:53:56.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:53:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:53:56.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:53:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:53:56.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:53:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:53:56.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:53:56.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:53:56.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:53:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:53:56.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:53:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:53:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:53:56.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:53:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:53:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:53:56.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:53:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:53:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:53:56.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:53:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:53:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:53:56.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:53:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:53:56.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:53:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:53:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:53:56.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:53:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:53:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:53:56.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:53:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:53:56.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:53:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:53:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=531] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:53:56.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=531] Local address: 172.21.15.161, port 56260 2022-07-03T15:53:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=531] Peer address: 172.21.15.101, port 22 2022-07-03T15:53:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=531] Connection lost 2022-07-03T15:53:56.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=531] Aborting connection 2022-07-03T15:53:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:53:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:53:56.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.938 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:56.939 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:53:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:57 smithi161 ceph-mon[91579]: pgmap v9445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:57 smithi188 ceph-mon[92425]: pgmap v9445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:53:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:53:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:53:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:53:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:53:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:53:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:53:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:53:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:53:59.894 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:53:59 smithi161 ceph-mon[91579]: pgmap v9446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:53:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:53:59 smithi188 ceph-mon[92425]: pgmap v9446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:01 smithi161 ceph-mon[91579]: pgmap v9447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:01 smithi188 ceph-mon[92425]: pgmap v9447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:03 smithi161 ceph-mon[91579]: pgmap v9448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:03 smithi188 ceph-mon[92425]: pgmap v9448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:05 smithi161 ceph-mon[91579]: pgmap v9449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:05 smithi188 ceph-mon[92425]: pgmap v9449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:06 smithi188 ceph-mon[92425]: pgmap v9450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:06 smithi161 ceph-mon[91579]: pgmap v9450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:09 smithi161 ceph-mon[91579]: pgmap v9451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:09 smithi188 ceph-mon[92425]: pgmap v9451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:11 smithi161 ceph-mon[91579]: pgmap v9452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:11 smithi188 ceph-mon[92425]: pgmap v9452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:13.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:13 smithi161 ceph-mon[91579]: pgmap v9453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:13.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:13 smithi188 ceph-mon[92425]: pgmap v9453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:15 smithi161 ceph-mon[91579]: pgmap v9454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:15 smithi188 ceph-mon[92425]: pgmap v9454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:16 smithi188 ceph-mon[92425]: pgmap v9455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:16 smithi161 ceph-mon[91579]: pgmap v9455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:19 smithi161 ceph-mon[91579]: pgmap v9456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:19 smithi188 ceph-mon[92425]: pgmap v9456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:21 smithi161 ceph-mon[91579]: pgmap v9457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:21 smithi188 ceph-mon[92425]: pgmap v9457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:23 smithi161 ceph-mon[91579]: pgmap v9458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:23 smithi188 ceph-mon[92425]: pgmap v9458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:25 smithi161 ceph-mon[91579]: pgmap v9459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:25 smithi188 ceph-mon[92425]: pgmap v9459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:26 smithi188 ceph-mon[92425]: pgmap v9460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:26 smithi161 ceph-mon[91579]: pgmap v9460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:29 smithi161 ceph-mon[91579]: pgmap v9461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:29 smithi188 ceph-mon[92425]: pgmap v9461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:31 smithi161 ceph-mon[91579]: pgmap v9462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:31 smithi188 ceph-mon[92425]: pgmap v9462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:33 smithi161 ceph-mon[91579]: pgmap v9463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:33 smithi188 ceph-mon[92425]: pgmap v9463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:35 smithi161 ceph-mon[91579]: pgmap v9464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:35 smithi188 ceph-mon[92425]: pgmap v9464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:36 smithi188 ceph-mon[92425]: pgmap v9465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:36 smithi161 ceph-mon[91579]: pgmap v9465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:39 smithi161 ceph-mon[91579]: pgmap v9466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:39 smithi188 ceph-mon[92425]: pgmap v9466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:41 smithi161 ceph-mon[91579]: pgmap v9467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:41 smithi188 ceph-mon[92425]: pgmap v9467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:43 smithi161 ceph-mon[91579]: pgmap v9468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:43 smithi188 ceph-mon[92425]: pgmap v9468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:45 smithi161 ceph-mon[91579]: pgmap v9469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:45 smithi188 ceph-mon[92425]: pgmap v9469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:46 smithi188 ceph-mon[92425]: pgmap v9470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:46 smithi161 ceph-mon[91579]: pgmap v9470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:49 smithi161 ceph-mon[91579]: pgmap v9471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:49 smithi188 ceph-mon[92425]: pgmap v9471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:51 smithi161 ceph-mon[91579]: pgmap v9472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:51 smithi188 ceph-mon[92425]: pgmap v9472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:53 smithi161 ceph-mon[91579]: pgmap v9473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:53 smithi188 ceph-mon[92425]: pgmap v9473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:55 smithi161 ceph-mon[91579]: pgmap v9474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:55 smithi188 ceph-mon[92425]: pgmap v9474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: pgmap v9475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:54:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:54:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:54:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:54:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:54:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:54:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:54:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:54:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:54:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:54:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:54:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:54:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:54:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:54:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:54:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:54:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:54:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:54:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:54:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:54:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:54:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:54:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:54:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:54:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:54:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:54:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:54:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:54:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:54:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:54:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:54:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:54:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:54:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:54:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:54:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:54:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:54:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:54:57.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:54:57.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:54:57.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:54:57.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:54:57.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:54:57.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:54:57.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:54:57.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=533] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:54:57.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=533] Local address: 172.21.15.161, port 56314 2022-07-03T15:54:57.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=533] Peer address: 172.21.15.101, port 22 2022-07-03T15:54:57.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=533] Connection lost 2022-07-03T15:54:57.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=533] Aborting connection 2022-07-03T15:54:57.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:54:56.687+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:54:57.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:54:57.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:54:57.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:54:57.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:54:57.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:54:57.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:54:57.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:54:57.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:54:57.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:54:57.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:54:57.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:54:57.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:54:57.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:54:57.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:54:57.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:54:57.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:54:57.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:54:57.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:54:57.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:54:57.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:54:57.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:54:57.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:54:57.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:54:57.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:54:57.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:54:57.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:54:57.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:54:57.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:54:57.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:54:57.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:54:57.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:54:57.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:54:57.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:54:57.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:54:57.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:54:57.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:54:57.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:54:57.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:54:57.167 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:54:57.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:54:57.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:54:57.168 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:54:57.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:54:57.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:54:57.169 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=533] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:54:57.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=533] Local address: 172.21.15.161, port 56314 2022-07-03T15:54:57.170 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=533] Peer address: 172.21.15.101, port 22 2022-07-03T15:54:57.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=533] Connection lost 2022-07-03T15:54:57.172 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=533] Aborting connection 2022-07-03T15:54:57.173 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:54:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:54:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:54:57.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:54:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: pgmap v9475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:54:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:54:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:54:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:54:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:54:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:54:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:54:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:54:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:54:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:54:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:54:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:54:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:54:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:54:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:54:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:54:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:54:57.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:54:57.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:54:57.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:54:57.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:54:57.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:54:57.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:54:57.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:54:57.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:54:57.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:54:57.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:54:57.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:54:57.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:54:57.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:54:57.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:54:57.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:54:57.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:54:57.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:54:57.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:54:57.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:54:57.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:54:57.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:54:57.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:54:57.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:54:57.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:54:57.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:54:57.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:54:57.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:54:57.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:54:57.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:54:57.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:54:57.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:54:57.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:54:57.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:54:57.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=533] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:54:57.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=533] Local address: 172.21.15.161, port 56314 2022-07-03T15:54:57.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=533] Peer address: 172.21.15.101, port 22 2022-07-03T15:54:57.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=533] Connection lost 2022-07-03T15:54:57.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=533] Aborting connection 2022-07-03T15:54:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:54:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:54:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:54:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:54:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:54:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:54:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:54:59 smithi161 ceph-mon[91579]: pgmap v9476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:54:59 smithi188 ceph-mon[92425]: pgmap v9476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:01 smithi161 ceph-mon[91579]: pgmap v9477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:01 smithi188 ceph-mon[92425]: pgmap v9477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:03 smithi161 ceph-mon[91579]: pgmap v9478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:03 smithi188 ceph-mon[92425]: pgmap v9478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:05 smithi161 ceph-mon[91579]: pgmap v9479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:05 smithi188 ceph-mon[92425]: pgmap v9479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:06 smithi188 ceph-mon[92425]: pgmap v9480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:06 smithi161 ceph-mon[91579]: pgmap v9480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:09 smithi161 ceph-mon[91579]: pgmap v9481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:09 smithi188 ceph-mon[92425]: pgmap v9481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:11 smithi161 ceph-mon[91579]: pgmap v9482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:11 smithi188 ceph-mon[92425]: pgmap v9482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:13 smithi161 ceph-mon[91579]: pgmap v9483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:13 smithi188 ceph-mon[92425]: pgmap v9483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:15 smithi161 ceph-mon[91579]: pgmap v9484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:15 smithi188 ceph-mon[92425]: pgmap v9484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:17.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:16 smithi188 ceph-mon[92425]: pgmap v9485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:16 smithi161 ceph-mon[91579]: pgmap v9485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:19 smithi161 ceph-mon[91579]: pgmap v9486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:19 smithi188 ceph-mon[92425]: pgmap v9486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:21 smithi161 ceph-mon[91579]: pgmap v9487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:21 smithi188 ceph-mon[92425]: pgmap v9487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:23 smithi161 ceph-mon[91579]: pgmap v9488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:23 smithi188 ceph-mon[92425]: pgmap v9488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:25 smithi161 ceph-mon[91579]: pgmap v9489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:25 smithi188 ceph-mon[92425]: pgmap v9489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:26 smithi188 ceph-mon[92425]: pgmap v9490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:26 smithi161 ceph-mon[91579]: pgmap v9490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:29 smithi161 ceph-mon[91579]: pgmap v9491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:29 smithi188 ceph-mon[92425]: pgmap v9491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:31 smithi161 ceph-mon[91579]: pgmap v9492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:31 smithi188 ceph-mon[92425]: pgmap v9492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:33 smithi161 ceph-mon[91579]: pgmap v9493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:33 smithi188 ceph-mon[92425]: pgmap v9493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:35 smithi161 ceph-mon[91579]: pgmap v9494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:35 smithi188 ceph-mon[92425]: pgmap v9494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:37.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:36 smithi188 ceph-mon[92425]: pgmap v9495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:36 smithi161 ceph-mon[91579]: pgmap v9495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:39 smithi161 ceph-mon[91579]: pgmap v9496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:39 smithi188 ceph-mon[92425]: pgmap v9496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:41 smithi161 ceph-mon[91579]: pgmap v9497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:41 smithi188 ceph-mon[92425]: pgmap v9497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:43 smithi161 ceph-mon[91579]: pgmap v9498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:43 smithi188 ceph-mon[92425]: pgmap v9498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:45 smithi161 ceph-mon[91579]: pgmap v9499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:45 smithi188 ceph-mon[92425]: pgmap v9499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:46 smithi188 ceph-mon[92425]: pgmap v9500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:46 smithi161 ceph-mon[91579]: pgmap v9500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:49 smithi161 ceph-mon[91579]: pgmap v9501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:49 smithi188 ceph-mon[92425]: pgmap v9501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:51 smithi161 ceph-mon[91579]: pgmap v9502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:51 smithi188 ceph-mon[92425]: pgmap v9502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:53 smithi161 ceph-mon[91579]: pgmap v9503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:53 smithi188 ceph-mon[92425]: pgmap v9503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:55 smithi161 ceph-mon[91579]: pgmap v9504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:55 smithi188 ceph-mon[92425]: pgmap v9504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:56 smithi188 ceph-mon[92425]: pgmap v9505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:57.193 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:55:57.194 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:55:57.195 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:56 smithi161 ceph-mon[91579]: pgmap v9505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:55:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:55:57.191+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:55:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:55:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:55:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:55:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:55:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:55:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:55:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:55:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:55:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:55:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:55:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:55:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:55:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:55:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:55:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:55:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:55:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:55:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:55:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:55:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:55:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:55:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:55:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:55:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:55:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:55:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:55:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:55:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:55:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:55:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:55:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:55:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:55:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:55:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:55:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:55:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:55:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:55:57.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:55:57.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:55:57.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:55:57.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:55:57.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:55:57.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:55:57.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:55:57.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=535] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:55:57.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=535] Local address: 172.21.15.161, port 56342 2022-07-03T15:55:57.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=535] Peer address: 172.21.15.101, port 22 2022-07-03T15:55:57.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=535] Connection lost 2022-07-03T15:55:57.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=535] Aborting connection 2022-07-03T15:55:57.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:55:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:55:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:55:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:55:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:55:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:55:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:55:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:55:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:55:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:55:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:55:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:55:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:55:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:55:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:55:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:55:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:55:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:55:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:55:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:55:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:55:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:55:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:55:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:55:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:55:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:55:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:55:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:55:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:55:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:55:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=535] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:55:58.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=535] Local address: 172.21.15.161, port 56342 2022-07-03T15:55:58.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=535] Peer address: 172.21.15.101, port 22 2022-07-03T15:55:58.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=535] Connection lost 2022-07-03T15:55:58.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=535] Aborting connection 2022-07-03T15:55:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:55:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:55:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:55:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:55:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:55:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:55:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:55:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:55:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:55:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:55:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:55:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:55:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:55:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:55:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:55:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:55:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:55:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:55:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:55:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:55:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:55:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:55:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:55:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:55:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:55:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:55:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:55:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:55:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:55:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:55:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:55:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:55:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:55:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:55:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:55:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:55:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:55:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:55:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:55:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:55:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:55:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:55:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:55:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:55:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:55:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:55:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:55:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=535] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:55:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=535] Local address: 172.21.15.161, port 56342 2022-07-03T15:55:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=535] Peer address: 172.21.15.101, port 22 2022-07-03T15:55:58.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=535] Connection lost 2022-07-03T15:55:58.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=535] Aborting connection 2022-07-03T15:55:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:55:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:55:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:55:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:55:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:55:58 smithi161 ceph-mon[91579]: pgmap v9506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:55:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:55:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:55:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:55:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:55:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:55:58 smithi188 ceph-mon[92425]: pgmap v9506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:01 smithi161 ceph-mon[91579]: pgmap v9507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:01 smithi188 ceph-mon[92425]: pgmap v9507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:03 smithi161 ceph-mon[91579]: pgmap v9508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:03 smithi188 ceph-mon[92425]: pgmap v9508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:05 smithi161 ceph-mon[91579]: pgmap v9509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:05 smithi188 ceph-mon[92425]: pgmap v9509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:06 smithi188 ceph-mon[92425]: pgmap v9510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:06 smithi161 ceph-mon[91579]: pgmap v9510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:09 smithi161 ceph-mon[91579]: pgmap v9511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:09 smithi188 ceph-mon[92425]: pgmap v9511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:11 smithi161 ceph-mon[91579]: pgmap v9512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:11 smithi188 ceph-mon[92425]: pgmap v9512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:13 smithi161 ceph-mon[91579]: pgmap v9513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:13 smithi188 ceph-mon[92425]: pgmap v9513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:15 smithi161 ceph-mon[91579]: pgmap v9514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:15 smithi188 ceph-mon[92425]: pgmap v9514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:16 smithi188 ceph-mon[92425]: pgmap v9515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:16 smithi161 ceph-mon[91579]: pgmap v9515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:19 smithi161 ceph-mon[91579]: pgmap v9516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:19 smithi188 ceph-mon[92425]: pgmap v9516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:21 smithi161 ceph-mon[91579]: pgmap v9517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:21 smithi188 ceph-mon[92425]: pgmap v9517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:23 smithi161 ceph-mon[91579]: pgmap v9518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:23 smithi188 ceph-mon[92425]: pgmap v9518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:25 smithi161 ceph-mon[91579]: pgmap v9519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:25 smithi188 ceph-mon[92425]: pgmap v9519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:26 smithi188 ceph-mon[92425]: pgmap v9520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:26 smithi161 ceph-mon[91579]: pgmap v9520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:29 smithi161 ceph-mon[91579]: pgmap v9521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:29 smithi188 ceph-mon[92425]: pgmap v9521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:31 smithi161 ceph-mon[91579]: pgmap v9522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:31 smithi188 ceph-mon[92425]: pgmap v9522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:33 smithi161 ceph-mon[91579]: pgmap v9523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:33 smithi188 ceph-mon[92425]: pgmap v9523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:35 smithi161 ceph-mon[91579]: pgmap v9524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:35 smithi188 ceph-mon[92425]: pgmap v9524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:36 smithi161 ceph-mon[91579]: pgmap v9525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:36 smithi188 ceph-mon[92425]: pgmap v9525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:39 smithi161 ceph-mon[91579]: pgmap v9526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:39 smithi188 ceph-mon[92425]: pgmap v9526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:41 smithi161 ceph-mon[91579]: pgmap v9527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:41 smithi188 ceph-mon[92425]: pgmap v9527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:43 smithi161 ceph-mon[91579]: pgmap v9528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:43 smithi188 ceph-mon[92425]: pgmap v9528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:45 smithi161 ceph-mon[91579]: pgmap v9529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:45 smithi188 ceph-mon[92425]: pgmap v9529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:46 smithi161 ceph-mon[91579]: pgmap v9530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:46 smithi188 ceph-mon[92425]: pgmap v9530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:49 smithi161 ceph-mon[91579]: pgmap v9531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:49 smithi188 ceph-mon[92425]: pgmap v9531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:51 smithi161 ceph-mon[91579]: pgmap v9532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:51 smithi188 ceph-mon[92425]: pgmap v9532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:53 smithi161 ceph-mon[91579]: pgmap v9533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:53 smithi188 ceph-mon[92425]: pgmap v9533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:55 smithi161 ceph-mon[91579]: pgmap v9534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:55 smithi188 ceph-mon[92425]: pgmap v9534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:56 smithi161 ceph-mon[91579]: pgmap v9535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:56:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:56:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:56 smithi188 ceph-mon[92425]: pgmap v9535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:56:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:56:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:56:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:56:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:56:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:56:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:56:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:56:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:56:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:56:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:56:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:56:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:56:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:56:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:56:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:56:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:56:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:56:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:56:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:56:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:56:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:56:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:56:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:56:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:56:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:56:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:56:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:56:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:56:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:56:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:56:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:56:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:56:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:56:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:56:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:56:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:56:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:56:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:56:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:56:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:56:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:56:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:56:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:56:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:56:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:56:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:56:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:56:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:56:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:56:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=537] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:56:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=537] Local address: 172.21.15.161, port 56370 2022-07-03T15:56:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=537] Peer address: 172.21.15.101, port 22 2022-07-03T15:56:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=537] Connection lost 2022-07-03T15:56:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=537] Aborting connection 2022-07-03T15:56:58.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:56:57.683+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:56:58.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:56:58.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:56:58.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:56:58.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:56:58.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:56:58.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:56:58.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:56:58.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:56:58.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:56:58.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:56:58.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:56:58.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:56:58.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:56:58.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:56:58.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:56:58.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:56:58.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:56:58.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:56:58.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:56:58.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:56:58.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:56:58.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:56:58.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:56:58.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:56:58.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:56:58.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:56:58.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:56:58.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:56:58.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:56:58.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:56:58.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:56:58.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:56:58.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:56:58.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:56:58.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:56:58.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:56:58.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:56:58.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:56:58.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:56:58.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:56:58.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:56:58.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:56:58.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:56:58.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:56:58.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=537] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:56:58.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=537] Local address: 172.21.15.161, port 56370 2022-07-03T15:56:58.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=537] Peer address: 172.21.15.101, port 22 2022-07-03T15:56:58.165 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=537] Connection lost 2022-07-03T15:56:58.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=537] Aborting connection 2022-07-03T15:56:58.166 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:56:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:56:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:56:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:56:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:56:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:56:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:56:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:56:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:56:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:56:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:56:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:56:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:56:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:56:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:56:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:56:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:56:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:56:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:56:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:56:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:56:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:56:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:56:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:56:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:56:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:56:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:56:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:56:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:56:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:56:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:56:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:56:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:56:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:56:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:56:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:56:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:56:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:56:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:56:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:56:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:56:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:56:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:56:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:56:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:56:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:56:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:56:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:56:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:56:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:56:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=537] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:56:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=537] Local address: 172.21.15.161, port 56370 2022-07-03T15:56:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=537] Peer address: 172.21.15.101, port 22 2022-07-03T15:56:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=537] Connection lost 2022-07-03T15:56:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=537] Aborting connection 2022-07-03T15:56:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:56:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:56:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:56:58 smithi161 ceph-mon[91579]: pgmap v9536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:56:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:56:58 smithi188 ceph-mon[92425]: pgmap v9536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: pgmap v9537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 145.2M 2022-07-03T15:57:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 152353120: error parsing value: Value '152353120' is below minimum 939524096 2022-07-03T15:57:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T15:57:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: pgmap v9537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 145.2M 2022-07-03T15:57:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 152353120: error parsing value: Value '152353120' is below minimum 939524096 2022-07-03T15:57:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T15:57:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:57:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:57:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T15:57:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T15:57:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:03 smithi161 ceph-mon[91579]: pgmap v9538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:03 smithi188 ceph-mon[92425]: pgmap v9538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:05 smithi161 ceph-mon[91579]: pgmap v9539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:05 smithi188 ceph-mon[92425]: pgmap v9539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:06 smithi161 ceph-mon[91579]: pgmap v9540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:06 smithi188 ceph-mon[92425]: pgmap v9540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:09 smithi161 ceph-mon[91579]: pgmap v9541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:09 smithi188 ceph-mon[92425]: pgmap v9541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:11 smithi161 ceph-mon[91579]: pgmap v9542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:11 smithi188 ceph-mon[92425]: pgmap v9542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:13 smithi161 ceph-mon[91579]: pgmap v9543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:13 smithi188 ceph-mon[92425]: pgmap v9543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:15 smithi161 ceph-mon[91579]: pgmap v9544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:15 smithi188 ceph-mon[92425]: pgmap v9544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:16 smithi161 ceph-mon[91579]: pgmap v9545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:16 smithi188 ceph-mon[92425]: pgmap v9545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:19 smithi161 ceph-mon[91579]: pgmap v9546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:19 smithi188 ceph-mon[92425]: pgmap v9546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:21 smithi161 ceph-mon[91579]: pgmap v9547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:21 smithi188 ceph-mon[92425]: pgmap v9547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:23 smithi161 ceph-mon[91579]: pgmap v9548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:23 smithi188 ceph-mon[92425]: pgmap v9548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:25 smithi161 ceph-mon[91579]: pgmap v9549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:25 smithi188 ceph-mon[92425]: pgmap v9549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:26 smithi161 ceph-mon[91579]: pgmap v9550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:26 smithi188 ceph-mon[92425]: pgmap v9550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:29 smithi161 ceph-mon[91579]: pgmap v9551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:29 smithi188 ceph-mon[92425]: pgmap v9551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:31 smithi161 ceph-mon[91579]: pgmap v9552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:31 smithi188 ceph-mon[92425]: pgmap v9552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:33 smithi161 ceph-mon[91579]: pgmap v9553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:33 smithi188 ceph-mon[92425]: pgmap v9553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:35 smithi161 ceph-mon[91579]: pgmap v9554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:35 smithi188 ceph-mon[92425]: pgmap v9554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:36 smithi161 ceph-mon[91579]: pgmap v9555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:36 smithi188 ceph-mon[92425]: pgmap v9555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:39 smithi161 ceph-mon[91579]: pgmap v9556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:39 smithi188 ceph-mon[92425]: pgmap v9556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:41 smithi161 ceph-mon[91579]: pgmap v9557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:41 smithi188 ceph-mon[92425]: pgmap v9557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:43 smithi161 ceph-mon[91579]: pgmap v9558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:43 smithi188 ceph-mon[92425]: pgmap v9558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:45 smithi161 ceph-mon[91579]: pgmap v9559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:45 smithi188 ceph-mon[92425]: pgmap v9559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:46 smithi161 ceph-mon[91579]: pgmap v9560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:46 smithi188 ceph-mon[92425]: pgmap v9560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:49 smithi161 ceph-mon[91579]: pgmap v9561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:49 smithi188 ceph-mon[92425]: pgmap v9561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:51 smithi161 ceph-mon[91579]: pgmap v9562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:51 smithi188 ceph-mon[92425]: pgmap v9562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:53 smithi161 ceph-mon[91579]: pgmap v9563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:53 smithi188 ceph-mon[92425]: pgmap v9563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:55 smithi161 ceph-mon[91579]: pgmap v9564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:55 smithi188 ceph-mon[92425]: pgmap v9564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:56 smithi161 ceph-mon[91579]: pgmap v9565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:57:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:57:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:56 smithi188 ceph-mon[92425]: pgmap v9565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:57:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:57:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:57:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:57:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:57:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:57:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:57:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:57:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:57:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:57:59 smithi161 ceph-mon[91579]: pgmap v9566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:57:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:57:59 smithi188 ceph-mon[92425]: pgmap v9566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:01.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:58:00.939+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:58:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:58:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:58:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:58:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:58:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:58:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:58:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:58:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:58:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:58:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:58:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:58:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:58:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:58:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:58:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:58:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:58:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:58:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:58:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:58:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:58:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:58:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:58:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:58:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:58:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:58:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:58:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:58:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:58:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:58:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:58:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:58:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:58:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:58:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:58:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:58:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:58:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:58:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:58:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:58:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:58:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:58:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:58:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:58:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:58:01.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=539] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:58:01.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=539] Local address: 172.21.15.161, port 56398 2022-07-03T15:58:01.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=539] Peer address: 172.21.15.101, port 22 2022-07-03T15:58:01.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=539] Connection lost 2022-07-03T15:58:01.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=539] Aborting connection 2022-07-03T15:58:01.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:58:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:58:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: pgmap v9567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:58:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:58:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:58:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: pgmap v9567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:58:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:58:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:58:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:58:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:58:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:58:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:58:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:58:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:58:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:58:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:58:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:58:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:58:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:58:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:58:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:58:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:58:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:58:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:58:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:58:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:58:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:58:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:58:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:58:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:58:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:58:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:58:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:58:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:58:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:58:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:58:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:58:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:58:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:58:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:58:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:58:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:58:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:58:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:58:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:58:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:58:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:58:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:58:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:58:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:58:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:58:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:58:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:58:02.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:58:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=539] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:58:02.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=539] Local address: 172.21.15.161, port 56398 2022-07-03T15:58:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=539] Peer address: 172.21.15.101, port 22 2022-07-03T15:58:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=539] Connection lost 2022-07-03T15:58:02.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=539] Aborting connection 2022-07-03T15:58:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:58:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:58:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:58:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:58:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:58:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:58:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:58:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:58:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:58:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:58:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:58:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:58:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:58:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:58:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:58:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:58:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:58:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:58:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:58:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:58:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:58:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:58:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:58:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:58:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:58:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:58:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:58:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:58:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:58:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:58:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:58:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:58:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:58:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:58:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:58:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:58:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:58:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:58:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:58:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:58:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:58:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:58:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:58:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:58:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:58:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:58:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=539] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:58:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=539] Local address: 172.21.15.161, port 56398 2022-07-03T15:58:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=539] Peer address: 172.21.15.101, port 22 2022-07-03T15:58:02.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=539] Connection lost 2022-07-03T15:58:02.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=539] Aborting connection 2022-07-03T15:58:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:03 smithi161 ceph-mon[91579]: pgmap v9568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:03 smithi188 ceph-mon[92425]: pgmap v9568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:05 smithi161 ceph-mon[91579]: pgmap v9569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:05 smithi188 ceph-mon[92425]: pgmap v9569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:06 smithi161 ceph-mon[91579]: pgmap v9570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:06 smithi188 ceph-mon[92425]: pgmap v9570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:09 smithi161 ceph-mon[91579]: pgmap v9571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:09 smithi188 ceph-mon[92425]: pgmap v9571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:11 smithi161 ceph-mon[91579]: pgmap v9572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:11 smithi188 ceph-mon[92425]: pgmap v9572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:13 smithi161 ceph-mon[91579]: pgmap v9573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:13 smithi188 ceph-mon[92425]: pgmap v9573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:15 smithi161 ceph-mon[91579]: pgmap v9574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:15 smithi188 ceph-mon[92425]: pgmap v9574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:16 smithi161 ceph-mon[91579]: pgmap v9575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:16 smithi188 ceph-mon[92425]: pgmap v9575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:19 smithi161 ceph-mon[91579]: pgmap v9576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:19 smithi188 ceph-mon[92425]: pgmap v9576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:21 smithi161 ceph-mon[91579]: pgmap v9577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:21 smithi188 ceph-mon[92425]: pgmap v9577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:23 smithi161 ceph-mon[91579]: pgmap v9578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:23 smithi188 ceph-mon[92425]: pgmap v9578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:25 smithi161 ceph-mon[91579]: pgmap v9579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:25 smithi188 ceph-mon[92425]: pgmap v9579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:26 smithi161 ceph-mon[91579]: pgmap v9580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:26 smithi188 ceph-mon[92425]: pgmap v9580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:29 smithi161 ceph-mon[91579]: pgmap v9581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:29 smithi188 ceph-mon[92425]: pgmap v9581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:31 smithi161 ceph-mon[91579]: pgmap v9582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:31 smithi188 ceph-mon[92425]: pgmap v9582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:33 smithi161 ceph-mon[91579]: pgmap v9583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:33 smithi188 ceph-mon[92425]: pgmap v9583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:35 smithi161 ceph-mon[91579]: pgmap v9584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:35 smithi188 ceph-mon[92425]: pgmap v9584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:36 smithi161 ceph-mon[91579]: pgmap v9585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:36 smithi188 ceph-mon[92425]: pgmap v9585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:39 smithi161 ceph-mon[91579]: pgmap v9586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:39 smithi188 ceph-mon[92425]: pgmap v9586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:41 smithi161 ceph-mon[91579]: pgmap v9587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:41 smithi188 ceph-mon[92425]: pgmap v9587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:43 smithi161 ceph-mon[91579]: pgmap v9588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:43 smithi188 ceph-mon[92425]: pgmap v9588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:45 smithi161 ceph-mon[91579]: pgmap v9589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:45 smithi188 ceph-mon[92425]: pgmap v9589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:46 smithi161 ceph-mon[91579]: pgmap v9590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:46 smithi188 ceph-mon[92425]: pgmap v9590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:49 smithi161 ceph-mon[91579]: pgmap v9591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:49 smithi188 ceph-mon[92425]: pgmap v9591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:51 smithi161 ceph-mon[91579]: pgmap v9592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:51 smithi188 ceph-mon[92425]: pgmap v9592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:53 smithi161 ceph-mon[91579]: pgmap v9593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:53 smithi188 ceph-mon[92425]: pgmap v9593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:55 smithi161 ceph-mon[91579]: pgmap v9594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:55 smithi188 ceph-mon[92425]: pgmap v9594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:56 smithi161 ceph-mon[91579]: pgmap v9595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:58:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:58:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:56 smithi188 ceph-mon[92425]: pgmap v9595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:58:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:58:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:58:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:58:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:58:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:58:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:58:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:58:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:58:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:58:59 smithi161 ceph-mon[91579]: pgmap v9596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:58:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:58:59 smithi188 ceph-mon[92425]: pgmap v9596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: pgmap v9597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:59:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:59:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:59:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:59:01.443+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:59:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:59:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:59:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T15:59:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:59:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:59:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:59:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T15:59:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:59:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T15:59:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:59:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T15:59:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:59:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:59:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T15:59:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:59:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T15:59:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:59:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T15:59:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:59:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T15:59:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:59:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:59:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:59:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:59:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:59:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T15:59:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:59:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:59:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:59:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T15:59:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:59:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T15:59:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:59:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:59:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:59:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T15:59:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:59:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T15:59:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:59:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T15:59:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:59:01.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T15:59:01.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:59:01.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:59:01.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=541] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:59:01.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=541] Local address: 172.21.15.161, port 56428 2022-07-03T15:59:01.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=541] Peer address: 172.21.15.101, port 22 2022-07-03T15:59:01.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=541] Connection lost 2022-07-03T15:59:01.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=541] Aborting connection 2022-07-03T15:59:01.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 15:59:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T15:59:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: pgmap v9597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T15:59:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T15:59:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T15:59:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:59:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:59:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:59:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:59:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T15:59:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:59:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:59:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:59:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T15:59:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:59:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T15:59:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:59:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T15:59:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:59:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:59:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:59:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T15:59:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T15:59:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T15:59:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:59:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T15:59:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:59:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:59:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:59:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:59:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:59:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:59:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:59:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:59:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:59:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T15:59:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:59:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T15:59:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:59:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:59:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:59:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:59:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:59:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T15:59:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:59:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:59:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:59:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T15:59:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:59:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:59:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=541] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:59:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=541] Local address: 172.21.15.161, port 56428 2022-07-03T15:59:02.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=541] Peer address: 172.21.15.101, port 22 2022-07-03T15:59:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=541] Connection lost 2022-07-03T15:59:02.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=541] Aborting connection 2022-07-03T15:59:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T15:59:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T15:59:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:59:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T15:59:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T15:59:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T15:59:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T15:59:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T15:59:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T15:59:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T15:59:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T15:59:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T15:59:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T15:59:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T15:59:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:59:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T15:59:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T15:59:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T15:59:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T15:59:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T15:59:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T15:59:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T15:59:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T15:59:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T15:59:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T15:59:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T15:59:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T15:59:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T15:59:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T15:59:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T15:59:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T15:59:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T15:59:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T15:59:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T15:59:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T15:59:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T15:59:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T15:59:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T15:59:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T15:59:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T15:59:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T15:59:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T15:59:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T15:59:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T15:59:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T15:59:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T15:59:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=541] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T15:59:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=541] Local address: 172.21.15.161, port 56428 2022-07-03T15:59:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=541] Peer address: 172.21.15.101, port 22 2022-07-03T15:59:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=541] Connection lost 2022-07-03T15:59:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=541] Aborting connection 2022-07-03T15:59:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:03 smithi161 ceph-mon[91579]: pgmap v9598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:03 smithi188 ceph-mon[92425]: pgmap v9598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:05 smithi161 ceph-mon[91579]: pgmap v9599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:05 smithi188 ceph-mon[92425]: pgmap v9599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:06 smithi161 ceph-mon[91579]: pgmap v9600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:06 smithi188 ceph-mon[92425]: pgmap v9600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:09 smithi161 ceph-mon[91579]: pgmap v9601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:09 smithi188 ceph-mon[92425]: pgmap v9601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:11 smithi161 ceph-mon[91579]: pgmap v9602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:11 smithi188 ceph-mon[92425]: pgmap v9602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:13 smithi161 ceph-mon[91579]: pgmap v9603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:13 smithi188 ceph-mon[92425]: pgmap v9603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:15 smithi161 ceph-mon[91579]: pgmap v9604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:15 smithi188 ceph-mon[92425]: pgmap v9604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:16 smithi161 ceph-mon[91579]: pgmap v9605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:16 smithi188 ceph-mon[92425]: pgmap v9605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:19 smithi161 ceph-mon[91579]: pgmap v9606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:19 smithi188 ceph-mon[92425]: pgmap v9606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:21 smithi161 ceph-mon[91579]: pgmap v9607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:21 smithi188 ceph-mon[92425]: pgmap v9607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:23 smithi161 ceph-mon[91579]: pgmap v9608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:23 smithi188 ceph-mon[92425]: pgmap v9608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:25 smithi161 ceph-mon[91579]: pgmap v9609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:25 smithi188 ceph-mon[92425]: pgmap v9609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:26 smithi161 ceph-mon[91579]: pgmap v9610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:26 smithi188 ceph-mon[92425]: pgmap v9610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:29 smithi161 ceph-mon[91579]: pgmap v9611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:29 smithi188 ceph-mon[92425]: pgmap v9611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:31 smithi161 ceph-mon[91579]: pgmap v9612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:31 smithi188 ceph-mon[92425]: pgmap v9612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:33 smithi161 ceph-mon[91579]: pgmap v9613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:33 smithi188 ceph-mon[92425]: pgmap v9613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:35 smithi161 ceph-mon[91579]: pgmap v9614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:35 smithi188 ceph-mon[92425]: pgmap v9614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:36 smithi161 ceph-mon[91579]: pgmap v9615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:36 smithi188 ceph-mon[92425]: pgmap v9615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:39 smithi161 ceph-mon[91579]: pgmap v9616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:39 smithi188 ceph-mon[92425]: pgmap v9616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:41 smithi161 ceph-mon[91579]: pgmap v9617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:41 smithi188 ceph-mon[92425]: pgmap v9617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:43 smithi161 ceph-mon[91579]: pgmap v9618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:43 smithi188 ceph-mon[92425]: pgmap v9618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:45 smithi161 ceph-mon[91579]: pgmap v9619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:45 smithi188 ceph-mon[92425]: pgmap v9619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:46 smithi161 ceph-mon[91579]: pgmap v9620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:46 smithi188 ceph-mon[92425]: pgmap v9620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:49 smithi161 ceph-mon[91579]: pgmap v9621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:49 smithi188 ceph-mon[92425]: pgmap v9621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:51 smithi161 ceph-mon[91579]: pgmap v9622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:51 smithi188 ceph-mon[92425]: pgmap v9622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:53 smithi161 ceph-mon[91579]: pgmap v9623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:53 smithi188 ceph-mon[92425]: pgmap v9623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:55 smithi161 ceph-mon[91579]: pgmap v9624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:55 smithi188 ceph-mon[92425]: pgmap v9624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:56 smithi161 ceph-mon[91579]: pgmap v9625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T15:59:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T15:59:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:56 smithi188 ceph-mon[92425]: pgmap v9625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T15:59:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:59:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:59:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:59:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:59:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:59:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T15:59:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:59:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T15:59:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 15:59:59 smithi161 ceph-mon[91579]: pgmap v9626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T15:59:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 15:59:59 smithi188 ceph-mon[92425]: pgmap v9626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:00:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:00:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-mon[91579]: pgmap v9627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:01 smithi188 ceph-mon[92425]: pgmap v9627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:02.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:00:01.944+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:00:02.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:00:02.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:00:02.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:00:02.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:00:02.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:00:02.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:00:02.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:00:02.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:00:02.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:00:02.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:00:02.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:00:02.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:00:02.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:00:02.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:00:02.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:00:02.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:00:02.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:00:02.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:00:02.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:00:02.214 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:00:02.214 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:00:02.214 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:00:02.215 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:00:02.215 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:00:02.215 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:00:02.216 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:00:02.216 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:00:02.217 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:00:02.217 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:00:02.217 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:00:02.218 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:00:02.218 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:00:02.218 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:00:02.219 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:00:02.219 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:00:02.219 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:00:02.220 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:00:02.220 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:00:02.220 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:00:02.221 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:00:02.221 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:00:02.221 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:00:02.222 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:00:02.222 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:00:02.223 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=543] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:00:02.223 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=543] Local address: 172.21.15.161, port 56456 2022-07-03T16:00:02.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=543] Peer address: 172.21.15.101, port 22 2022-07-03T16:00:02.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=543] Connection lost 2022-07-03T16:00:02.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=543] Aborting connection 2022-07-03T16:00:02.225 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:00:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:00:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:00:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:00:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:00:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:00:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:00:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:00:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:00:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:00:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:00:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:00:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:00:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:00:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:00:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:00:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:00:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:00:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:00:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:00:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:00:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:00:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:00:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:00:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:00:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:00:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:00:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:00:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:00:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:00:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:00:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:00:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:00:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:00:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:00:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:00:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:00:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:00:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:00:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:00:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:00:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:00:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:00:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:00:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:00:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:00:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:00:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:00:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:00:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:00:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:00:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:00:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:00:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=543] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:00:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=543] Local address: 172.21.15.161, port 56456 2022-07-03T16:00:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=543] Peer address: 172.21.15.101, port 22 2022-07-03T16:00:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=543] Connection lost 2022-07-03T16:00:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=543] Aborting connection 2022-07-03T16:00:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:03 smithi161 ceph-mon[91579]: pgmap v9628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:00:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:00:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:00:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:00:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:00:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:00:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:00:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:00:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:00:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:00:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:00:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:00:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:00:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:00:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:00:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:00:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:00:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:00:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:00:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:00:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:00:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:00:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:00:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:00:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:00:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:00:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:00:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:00:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:00:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:00:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:00:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:00:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:00:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:00:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:00:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:00:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:00:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:00:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:00:03.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:00:03.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:00:03.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:00:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:00:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:00:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=543] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:00:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=543] Local address: 172.21.15.161, port 56456 2022-07-03T16:00:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=543] Peer address: 172.21.15.101, port 22 2022-07-03T16:00:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=543] Connection lost 2022-07-03T16:00:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=543] Aborting connection 2022-07-03T16:00:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:03 smithi188 ceph-mon[92425]: pgmap v9628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:05 smithi161 ceph-mon[91579]: pgmap v9629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:05 smithi188 ceph-mon[92425]: pgmap v9629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:06 smithi161 ceph-mon[91579]: pgmap v9630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:06 smithi188 ceph-mon[92425]: pgmap v9630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:09 smithi161 ceph-mon[91579]: pgmap v9631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:09 smithi188 ceph-mon[92425]: pgmap v9631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:11 smithi161 ceph-mon[91579]: pgmap v9632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:11 smithi188 ceph-mon[92425]: pgmap v9632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:13 smithi161 ceph-mon[91579]: pgmap v9633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:13 smithi188 ceph-mon[92425]: pgmap v9633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:15 smithi161 ceph-mon[91579]: pgmap v9634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:15 smithi188 ceph-mon[92425]: pgmap v9634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:16 smithi161 ceph-mon[91579]: pgmap v9635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:16 smithi188 ceph-mon[92425]: pgmap v9635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:19 smithi161 ceph-mon[91579]: pgmap v9636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:19 smithi188 ceph-mon[92425]: pgmap v9636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:21 smithi161 ceph-mon[91579]: pgmap v9637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:21.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:21 smithi188 ceph-mon[92425]: pgmap v9637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:23.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:23 smithi161 ceph-mon[91579]: pgmap v9638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:23.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:23 smithi188 ceph-mon[92425]: pgmap v9638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:25 smithi161 ceph-mon[91579]: pgmap v9639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:25 smithi188 ceph-mon[92425]: pgmap v9639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:26 smithi161 ceph-mon[91579]: pgmap v9640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:26 smithi188 ceph-mon[92425]: pgmap v9640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:29 smithi161 ceph-mon[91579]: pgmap v9641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:29 smithi188 ceph-mon[92425]: pgmap v9641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:31 smithi161 ceph-mon[91579]: pgmap v9642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:31 smithi188 ceph-mon[92425]: pgmap v9642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:33 smithi161 ceph-mon[91579]: pgmap v9643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:33 smithi188 ceph-mon[92425]: pgmap v9643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:35 smithi161 ceph-mon[91579]: pgmap v9644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:35 smithi188 ceph-mon[92425]: pgmap v9644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:36 smithi161 ceph-mon[91579]: pgmap v9645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:36 smithi188 ceph-mon[92425]: pgmap v9645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:39 smithi161 ceph-mon[91579]: pgmap v9646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:39 smithi188 ceph-mon[92425]: pgmap v9646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:41 smithi161 ceph-mon[91579]: pgmap v9647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:41 smithi188 ceph-mon[92425]: pgmap v9647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:43 smithi188 ceph-mon[92425]: pgmap v9648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:43 smithi161 ceph-mon[91579]: pgmap v9648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:45 smithi188 ceph-mon[92425]: pgmap v9649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:45 smithi161 ceph-mon[91579]: pgmap v9649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:46 smithi161 ceph-mon[91579]: pgmap v9650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:46 smithi188 ceph-mon[92425]: pgmap v9650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:49 smithi161 ceph-mon[91579]: pgmap v9651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:49 smithi188 ceph-mon[92425]: pgmap v9651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:51 smithi188 ceph-mon[92425]: pgmap v9652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:51 smithi161 ceph-mon[91579]: pgmap v9652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:52.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:53 smithi188 ceph-mon[92425]: pgmap v9653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:53 smithi161 ceph-mon[91579]: pgmap v9653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:55 smithi188 ceph-mon[92425]: pgmap v9654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:55 smithi161 ceph-mon[91579]: pgmap v9654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:56 smithi161 ceph-mon[91579]: pgmap v9655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:00:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:00:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:56 smithi188 ceph-mon[92425]: pgmap v9655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:00:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:00:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:00:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:00:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:00:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:00:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:00:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:00:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:00:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:00:59 smithi161 ceph-mon[91579]: pgmap v9656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:00:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:00:59 smithi188 ceph-mon[92425]: pgmap v9656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:01 smithi188 ceph-mon[92425]: pgmap v9657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:01.948 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:01 smithi161 ceph-mon[91579]: pgmap v9657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:01.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:01.949 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:01:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:01:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:01:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:01:02.454+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:01:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:01:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:01:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:01:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:01:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:01:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:01:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:01:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:01:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:01:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:01:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:01:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:01:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:01:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:01:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:01:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:01:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:01:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:01:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:01:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:01:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:01:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:01:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:01:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:01:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:01:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:01:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:01:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:01:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:01:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:01:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:01:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:01:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:01:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:01:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:01:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:01:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:01:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:01:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:01:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:01:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:01:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:01:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:01:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:01:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=545] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:01:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=545] Local address: 172.21.15.161, port 56484 2022-07-03T16:01:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=545] Peer address: 172.21.15.101, port 22 2022-07-03T16:01:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=545] Connection lost 2022-07-03T16:01:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=545] Aborting connection 2022-07-03T16:01:02.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:01:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:01:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:01:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:01:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:01:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:01:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:01:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:01:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:01:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:01:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:01:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:01:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:01:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:01:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:01:03.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:01:03.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:01:03.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:01:03.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:01:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:01:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:01:03.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:01:03.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:01:03.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:01:03.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:01:03.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:01:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:01:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:01:03.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:01:03.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:01:03.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:01:03.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:01:03.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:01:03.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:01:03.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:01:03.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:01:03.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:01:03.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:01:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:01:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:01:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:01:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:01:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:01:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:01:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:01:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:01:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:01:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:01:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:01:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:01:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:01:03.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=545] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:01:03.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=545] Local address: 172.21.15.161, port 56484 2022-07-03T16:01:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=545] Peer address: 172.21.15.101, port 22 2022-07-03T16:01:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=545] Connection lost 2022-07-03T16:01:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=545] Aborting connection 2022-07-03T16:01:03.940 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:03 smithi188 ceph-mon[92425]: pgmap v9658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:01:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:01:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:01:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:01:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:01:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:01:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:01:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:01:04.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:01:04.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:01:04.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:01:04.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:01:04.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:01:04.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:01:04.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:01:04.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:01:04.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:01:04.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:01:04.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:01:04.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:01:04.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:01:04.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:01:04.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:01:04.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:01:04.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:01:04.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:01:04.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:01:04.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:01:04.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:01:04.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:01:04.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:01:04.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:01:04.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:01:04.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:01:04.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:01:04.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:01:04.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:01:04.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:01:04.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:01:04.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:01:04.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:01:04.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:01:04.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:01:04.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:01:04.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:01:04.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:01:04.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=545] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:01:04.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=545] Local address: 172.21.15.161, port 56484 2022-07-03T16:01:04.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=545] Peer address: 172.21.15.101, port 22 2022-07-03T16:01:04.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=545] Connection lost 2022-07-03T16:01:04.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=545] Aborting connection 2022-07-03T16:01:04.152 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:03 smithi161 ceph-mon[91579]: pgmap v9658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:05 smithi188 ceph-mon[92425]: pgmap v9659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:05 smithi161 ceph-mon[91579]: pgmap v9659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:06 smithi161 ceph-mon[91579]: pgmap v9660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:06 smithi188 ceph-mon[92425]: pgmap v9660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:09 smithi188 ceph-mon[92425]: pgmap v9661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:09 smithi161 ceph-mon[91579]: pgmap v9661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:11 smithi188 ceph-mon[92425]: pgmap v9662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:11 smithi161 ceph-mon[91579]: pgmap v9662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:13 smithi188 ceph-mon[92425]: pgmap v9663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:13 smithi161 ceph-mon[91579]: pgmap v9663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:15 smithi188 ceph-mon[92425]: pgmap v9664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:15 smithi161 ceph-mon[91579]: pgmap v9664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:16 smithi161 ceph-mon[91579]: pgmap v9665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:17.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:17.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:16 smithi188 ceph-mon[92425]: pgmap v9665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:19 smithi188 ceph-mon[92425]: pgmap v9666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:19 smithi161 ceph-mon[91579]: pgmap v9666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:21 smithi188 ceph-mon[92425]: pgmap v9667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:21.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:21 smithi161 ceph-mon[91579]: pgmap v9667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:23 smithi188 ceph-mon[92425]: pgmap v9668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:23 smithi161 ceph-mon[91579]: pgmap v9668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:25 smithi188 ceph-mon[92425]: pgmap v9669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:25 smithi161 ceph-mon[91579]: pgmap v9669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:26 smithi161 ceph-mon[91579]: pgmap v9670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:26 smithi188 ceph-mon[92425]: pgmap v9670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:29 smithi188 ceph-mon[92425]: pgmap v9671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:29 smithi161 ceph-mon[91579]: pgmap v9671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:31 smithi188 ceph-mon[92425]: pgmap v9672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:31 smithi161 ceph-mon[91579]: pgmap v9672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:01:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:33 smithi188 ceph-mon[92425]: pgmap v9673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:33 smithi161 ceph-mon[91579]: pgmap v9673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:35 smithi188 ceph-mon[92425]: pgmap v9674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:35 smithi161 ceph-mon[91579]: pgmap v9674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:36 smithi161 ceph-mon[91579]: pgmap v9675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:36 smithi188 ceph-mon[92425]: pgmap v9675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:39 smithi188 ceph-mon[92425]: pgmap v9676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:39 smithi161 ceph-mon[91579]: pgmap v9676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:41 smithi188 ceph-mon[92425]: pgmap v9677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:41 smithi161 ceph-mon[91579]: pgmap v9677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:43 smithi188 ceph-mon[92425]: pgmap v9678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:43 smithi161 ceph-mon[91579]: pgmap v9678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:45 smithi188 ceph-mon[92425]: pgmap v9679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:45 smithi161 ceph-mon[91579]: pgmap v9679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:46 smithi161 ceph-mon[91579]: pgmap v9680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:46 smithi188 ceph-mon[92425]: pgmap v9680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:49 smithi188 ceph-mon[92425]: pgmap v9681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:49 smithi161 ceph-mon[91579]: pgmap v9681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:51 smithi188 ceph-mon[92425]: pgmap v9682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:51 smithi161 ceph-mon[91579]: pgmap v9682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:53 smithi188 ceph-mon[92425]: pgmap v9683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:53 smithi161 ceph-mon[91579]: pgmap v9683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:55 smithi188 ceph-mon[92425]: pgmap v9684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:55 smithi161 ceph-mon[91579]: pgmap v9684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:56 smithi161 ceph-mon[91579]: pgmap v9685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:01:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:01:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:56 smithi188 ceph-mon[92425]: pgmap v9685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:01:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:01:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:01:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:01:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:01:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:01:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:01:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:01:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:01:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:01:59 smithi161 ceph-mon[91579]: pgmap v9686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:01:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:01:59 smithi188 ceph-mon[92425]: pgmap v9686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:01 smithi188 ceph-mon[92425]: pgmap v9687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:01 smithi161 ceph-mon[91579]: pgmap v9687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:03.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:02:02.960+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:02:03.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:02:03.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:02:03.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:02:03.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:02:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:02:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:02:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:02:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:02:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:02:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:02:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:02:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:02:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:02:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:02:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:02:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:02:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:02:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:02:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:02:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:02:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:02:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:02:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:02:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:02:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:02:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:02:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:02:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:02:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:02:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:02:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:02:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:02:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:02:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:02:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:02:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:02:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:02:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:02:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:02:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:02:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:02:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:02:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:02:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:02:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=547] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:02:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=547] Local address: 172.21.15.161, port 56512 2022-07-03T16:02:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=547] Peer address: 172.21.15.101, port 22 2022-07-03T16:02:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=547] Connection lost 2022-07-03T16:02:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=547] Aborting connection 2022-07-03T16:02:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:02:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:03 smithi188 ceph-mon[92425]: pgmap v9688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:02:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:02:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:02:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:03 smithi161 ceph-mon[91579]: pgmap v9688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:02:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:02:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:02:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:02:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:02:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:02:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:02:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:02:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:02:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:02:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:02:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:02:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:02:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:02:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:02:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:02:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:02:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:02:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:02:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:02:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:02:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:02:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:02:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:02:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:02:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:02:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:02:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:02:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:02:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:02:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:02:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:02:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:02:04.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:02:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:02:04.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:02:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:02:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:02:04.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:02:04.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:02:04.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:02:04.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:02:04.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:02:04.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:02:04.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:02:04.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:02:04.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:02:04.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:02:04.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:02:04.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=547] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:02:04.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=547] Local address: 172.21.15.161, port 56512 2022-07-03T16:02:04.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=547] Peer address: 172.21.15.101, port 22 2022-07-03T16:02:04.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=547] Connection lost 2022-07-03T16:02:04.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=547] Aborting connection 2022-07-03T16:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:02:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:02:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:02:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:02:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:02:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:02:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:02:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:02:05.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:02:05.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:02:05.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:02:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:02:05.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:02:05.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:02:05.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:02:05.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:02:05.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:02:05.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:02:05.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:02:05.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:02:05.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:02:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:02:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:02:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:02:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:02:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:02:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:02:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:02:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:02:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:02:05.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:02:05.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:02:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:02:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:02:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:02:05.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:02:05.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:02:05.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:02:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:02:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:02:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:02:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=547] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:02:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=547] Local address: 172.21.15.161, port 56512 2022-07-03T16:02:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=547] Peer address: 172.21.15.101, port 22 2022-07-03T16:02:05.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=547] Connection lost 2022-07-03T16:02:05.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=547] Aborting connection 2022-07-03T16:02:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:05 smithi188 ceph-mon[92425]: pgmap v9689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:05 smithi161 ceph-mon[91579]: pgmap v9689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:06 smithi161 ceph-mon[91579]: pgmap v9690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:06 smithi188 ceph-mon[92425]: pgmap v9690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:09 smithi188 ceph-mon[92425]: pgmap v9691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:09 smithi161 ceph-mon[91579]: pgmap v9691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:11 smithi188 ceph-mon[92425]: pgmap v9692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:11.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:11 smithi161 ceph-mon[91579]: pgmap v9692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:13 smithi188 ceph-mon[92425]: pgmap v9693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:13 smithi161 ceph-mon[91579]: pgmap v9693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:15 smithi188 ceph-mon[92425]: pgmap v9694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:15 smithi161 ceph-mon[91579]: pgmap v9694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:16 smithi161 ceph-mon[91579]: pgmap v9695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:16 smithi188 ceph-mon[92425]: pgmap v9695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:19 smithi188 ceph-mon[92425]: pgmap v9696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:19 smithi161 ceph-mon[91579]: pgmap v9696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:21.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:21 smithi188 ceph-mon[92425]: pgmap v9697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:21.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:21.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:21 smithi161 ceph-mon[91579]: pgmap v9697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:23 smithi188 ceph-mon[92425]: pgmap v9698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:23 smithi161 ceph-mon[91579]: pgmap v9698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:25 smithi161 ceph-mon[91579]: pgmap v9699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:25 smithi188 ceph-mon[92425]: pgmap v9699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:26 smithi161 ceph-mon[91579]: pgmap v9700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:26 smithi188 ceph-mon[92425]: pgmap v9700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:29 smithi188 ceph-mon[92425]: pgmap v9701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:29 smithi161 ceph-mon[91579]: pgmap v9701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:31 smithi188 ceph-mon[92425]: pgmap v9702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:31 smithi161 ceph-mon[91579]: pgmap v9702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:33 smithi161 ceph-mon[91579]: pgmap v9703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:33 smithi188 ceph-mon[92425]: pgmap v9703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:35 smithi161 ceph-mon[91579]: pgmap v9704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:35 smithi188 ceph-mon[92425]: pgmap v9704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:36 smithi161 ceph-mon[91579]: pgmap v9705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:37.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:37.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:36 smithi188 ceph-mon[92425]: pgmap v9705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:39 smithi188 ceph-mon[92425]: pgmap v9706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:39 smithi161 ceph-mon[91579]: pgmap v9706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:41 smithi161 ceph-mon[91579]: pgmap v9707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:41 smithi188 ceph-mon[92425]: pgmap v9707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:43 smithi161 ceph-mon[91579]: pgmap v9708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:43 smithi188 ceph-mon[92425]: pgmap v9708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:45 smithi161 ceph-mon[91579]: pgmap v9709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:45 smithi188 ceph-mon[92425]: pgmap v9709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:46 smithi161 ceph-mon[91579]: pgmap v9710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:46 smithi188 ceph-mon[92425]: pgmap v9710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:49 smithi161 ceph-mon[91579]: pgmap v9711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:49 smithi188 ceph-mon[92425]: pgmap v9711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:51 smithi161 ceph-mon[91579]: pgmap v9712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:52.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:51 smithi188 ceph-mon[92425]: pgmap v9712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:53 smithi161 ceph-mon[91579]: pgmap v9713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:53 smithi188 ceph-mon[92425]: pgmap v9713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:55 smithi161 ceph-mon[91579]: pgmap v9714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:55 smithi188 ceph-mon[92425]: pgmap v9714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:56 smithi161 ceph-mon[91579]: pgmap v9715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:02:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:02:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:57 smithi188 ceph-mon[92425]: pgmap v9715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:02:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:02:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:02:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:02:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:02:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:02:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:02:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:02:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:02:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:02:59 smithi161 ceph-mon[91579]: pgmap v9716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:02:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:02:59 smithi188 ceph-mon[92425]: pgmap v9716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:01 smithi161 ceph-mon[91579]: pgmap v9717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:01 smithi188 ceph-mon[92425]: pgmap v9717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-mon[91579]: pgmap v9718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:03:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:03:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:03:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:03:03.462+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:03:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:03:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:03:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:03:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:03:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:03:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:03:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:03:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:03:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:03:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:03:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:03:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:03:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:03:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:03:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:03:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:03:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:03:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:03:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:03:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:03:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:03:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:03:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:03:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:03:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:03:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:03:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:03:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:03:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:03:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:03:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:03:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:03:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:03:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:03:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:03:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:03:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:03:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:03:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:03:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:03:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:03:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:03:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:03:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:03:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=549] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:03:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=549] Local address: 172.21.15.161, port 56540 2022-07-03T16:03:03.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=549] Peer address: 172.21.15.101, port 22 2022-07-03T16:03:03.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=549] Connection lost 2022-07-03T16:03:03.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=549] Aborting connection 2022-07-03T16:03:03.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:03:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:03 smithi188 ceph-mon[92425]: pgmap v9718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:03:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:03:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:03:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:03:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:03:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:03:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:03:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:03:05.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:03:05.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:03:05.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:03:05.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:03:05.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:03:05.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:03:05.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:03:05.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:03:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:03:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:03:05.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:03:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:03:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:03:05.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:03:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:03:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:03:05.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:03:05.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:03:05.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:03:05.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:03:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:03:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:03:05.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:03:05.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:03:05.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:03:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:03:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:03:05.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:03:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:03:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:03:05.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:03:05.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:03:05.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:03:05.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:03:05.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:03:05.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:03:05.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:03:05.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:03:05.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:03:05.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:03:05.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:03:05.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=549] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:03:05.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=549] Local address: 172.21.15.161, port 56540 2022-07-03T16:03:05.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=549] Peer address: 172.21.15.101, port 22 2022-07-03T16:03:05.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=549] Connection lost 2022-07-03T16:03:05.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=549] Aborting connection 2022-07-03T16:03:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:03:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:03:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:03:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:03:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:03:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:03:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:03:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:03:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:03:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:03:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:03:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:03:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:03:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:03:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:03:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:03:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:03:05.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:03:05.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:03:05.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:03:05.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:03:05.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:03:05.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:03:05.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:03:05.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:03:05.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:03:05.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:03:05.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:03:05.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:03:05.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:03:05.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:03:05.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:03:05.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:03:05.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:03:05.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:03:05.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:03:05.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:03:05.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:03:05.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:03:05.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:03:05.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:03:05.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:03:05.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:03:05.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:03:05.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:03:05.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:03:05.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=549] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:03:05.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=549] Local address: 172.21.15.161, port 56540 2022-07-03T16:03:05.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=549] Peer address: 172.21.15.101, port 22 2022-07-03T16:03:05.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=549] Connection lost 2022-07-03T16:03:05.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=549] Aborting connection 2022-07-03T16:03:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:05 smithi161 ceph-mon[91579]: pgmap v9719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:05 smithi188 ceph-mon[92425]: pgmap v9719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:07 smithi161 ceph-mon[91579]: pgmap v9720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:07 smithi188 ceph-mon[92425]: pgmap v9720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:09 smithi161 ceph-mon[91579]: pgmap v9721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:09 smithi188 ceph-mon[92425]: pgmap v9721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:11 smithi161 ceph-mon[91579]: pgmap v9722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:11 smithi188 ceph-mon[92425]: pgmap v9722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:13 smithi161 ceph-mon[91579]: pgmap v9723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:13 smithi188 ceph-mon[92425]: pgmap v9723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:15 smithi161 ceph-mon[91579]: pgmap v9724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:15 smithi188 ceph-mon[92425]: pgmap v9724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:17 smithi161 ceph-mon[91579]: pgmap v9725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:17 smithi188 ceph-mon[92425]: pgmap v9725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:19 smithi161 ceph-mon[91579]: pgmap v9726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:19 smithi188 ceph-mon[92425]: pgmap v9726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:21 smithi161 ceph-mon[91579]: pgmap v9727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:21 smithi188 ceph-mon[92425]: pgmap v9727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:23 smithi161 ceph-mon[91579]: pgmap v9728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:23 smithi188 ceph-mon[92425]: pgmap v9728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:25 smithi161 ceph-mon[91579]: pgmap v9729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:25 smithi188 ceph-mon[92425]: pgmap v9729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:27 smithi161 ceph-mon[91579]: pgmap v9730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:27 smithi188 ceph-mon[92425]: pgmap v9730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:29 smithi161 ceph-mon[91579]: pgmap v9731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:29 smithi188 ceph-mon[92425]: pgmap v9731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:31.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:03:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:03:31.100+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T16:03:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:31 smithi161 ceph-mon[91579]: pgmap v9732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:31 smithi188 ceph-mon[92425]: pgmap v9732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:03:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:33 smithi161 ceph-mon[91579]: pgmap v9733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:33 smithi188 ceph-mon[92425]: pgmap v9733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:35 smithi161 ceph-mon[91579]: pgmap v9734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:35 smithi188 ceph-mon[92425]: pgmap v9734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:37 smithi161 ceph-mon[91579]: pgmap v9735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:37 smithi188 ceph-mon[92425]: pgmap v9735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:39 smithi161 ceph-mon[91579]: pgmap v9736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:39 smithi188 ceph-mon[92425]: pgmap v9736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:41 smithi161 ceph-mon[91579]: pgmap v9737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:41 smithi188 ceph-mon[92425]: pgmap v9737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:43 smithi161 ceph-mon[91579]: pgmap v9738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:43 smithi188 ceph-mon[92425]: pgmap v9738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:45 smithi161 ceph-mon[91579]: pgmap v9739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:45 smithi188 ceph-mon[92425]: pgmap v9739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:47 smithi161 ceph-mon[91579]: pgmap v9740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:47 smithi188 ceph-mon[92425]: pgmap v9740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:49 smithi161 ceph-mon[91579]: pgmap v9741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:49 smithi188 ceph-mon[92425]: pgmap v9741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:51 smithi161 ceph-mon[91579]: pgmap v9742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:51 smithi188 ceph-mon[92425]: pgmap v9742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:53 smithi161 ceph-mon[91579]: pgmap v9743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:53 smithi188 ceph-mon[92425]: pgmap v9743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:55 smithi161 ceph-mon[91579]: pgmap v9744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:55 smithi188 ceph-mon[92425]: pgmap v9744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:57 smithi161 ceph-mon[91579]: pgmap v9745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:03:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:57 smithi188 ceph-mon[92425]: pgmap v9745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:03:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:03:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:03:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:03:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:03:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:03:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:03:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:03:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:03:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:03:59 smithi161 ceph-mon[91579]: pgmap v9746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:03:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:03:59 smithi188 ceph-mon[92425]: pgmap v9746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:01 smithi161 ceph-mon[91579]: pgmap v9747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:01 smithi188 ceph-mon[92425]: pgmap v9747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:03.966 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-mon[91579]: pgmap v9748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:03.977 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:03 smithi188 ceph-mon[92425]: pgmap v9748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:04:03.963+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:04:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:04:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:04:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:04:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:04:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:04:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:04:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:04:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:04:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:04:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:04:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:04:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:04:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:04:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:04:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:04:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:04:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:04:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:04:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:04:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:04:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:04:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:04:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:04:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:04:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:04:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:04:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:04:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:04:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:04:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:04:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:04:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:04:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:04:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:04:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:04:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:04:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:04:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:04:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:04:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:04:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:04:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:04:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:04:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:04:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=551] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:04:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=551] Local address: 172.21.15.161, port 56556 2022-07-03T16:04:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=551] Peer address: 172.21.15.101, port 22 2022-07-03T16:04:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=551] Connection lost 2022-07-03T16:04:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=551] Aborting connection 2022-07-03T16:04:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:04:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:04:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:04:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:04:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:04:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:04:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:04:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:04:05.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:05.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:04:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:04:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:04:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:04:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:04:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:04:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:04:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:04:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:04:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:04:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:04:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:04:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:04:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:04:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:04:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:04:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:04:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:04:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:04:06.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:04:06.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:04:06.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:04:06.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:04:06.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:04:06.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:04:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:04:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:04:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:04:06.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:04:06.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:04:06.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:04:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:04:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:04:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:04:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:04:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:04:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:04:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:04:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:04:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:04:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:04:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:04:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:04:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:04:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:04:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:04:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:04:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=551] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:04:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=551] Local address: 172.21.15.161, port 56556 2022-07-03T16:04:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=551] Peer address: 172.21.15.101, port 22 2022-07-03T16:04:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=551] Connection lost 2022-07-03T16:04:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=551] Aborting connection 2022-07-03T16:04:06.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:05 smithi161 ceph-mon[91579]: pgmap v9749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:04:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:04:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:04:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:04:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:04:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:04:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:04:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:04:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:04:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:04:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:04:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:04:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:04:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:04:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:04:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:04:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:04:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:04:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:04:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:04:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:04:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:04:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:04:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:04:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:04:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:04:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:04:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:04:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:04:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:04:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:04:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:04:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:04:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:04:06.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:04:06.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:04:06.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:04:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:04:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:04:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:04:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:04:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:04:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:04:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:04:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:04:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:04:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=551] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:04:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=551] Local address: 172.21.15.161, port 56556 2022-07-03T16:04:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=551] Peer address: 172.21.15.101, port 22 2022-07-03T16:04:06.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=551] Connection lost 2022-07-03T16:04:06.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=551] Aborting connection 2022-07-03T16:04:06.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:05 smithi188 ceph-mon[92425]: pgmap v9749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:07 smithi161 ceph-mon[91579]: pgmap v9750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:07 smithi188 ceph-mon[92425]: pgmap v9750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:09 smithi161 ceph-mon[91579]: pgmap v9751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:09 smithi188 ceph-mon[92425]: pgmap v9751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:11 smithi161 ceph-mon[91579]: pgmap v9752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:11 smithi188 ceph-mon[92425]: pgmap v9752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:13 smithi161 ceph-mon[91579]: pgmap v9753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:13 smithi188 ceph-mon[92425]: pgmap v9753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:15 smithi161 ceph-mon[91579]: pgmap v9754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:15 smithi188 ceph-mon[92425]: pgmap v9754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:17 smithi161 ceph-mon[91579]: pgmap v9755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:17 smithi188 ceph-mon[92425]: pgmap v9755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:19 smithi161 ceph-mon[91579]: pgmap v9756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:19 smithi188 ceph-mon[92425]: pgmap v9756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:21 smithi161 ceph-mon[91579]: pgmap v9757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:21 smithi188 ceph-mon[92425]: pgmap v9757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:23 smithi161 ceph-mon[91579]: pgmap v9758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:23 smithi188 ceph-mon[92425]: pgmap v9758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:25 smithi161 ceph-mon[91579]: pgmap v9759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:25 smithi188 ceph-mon[92425]: pgmap v9759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:27 smithi161 ceph-mon[91579]: pgmap v9760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:27 smithi188 ceph-mon[92425]: pgmap v9760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:29 smithi161 ceph-mon[91579]: pgmap v9761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:29 smithi188 ceph-mon[92425]: pgmap v9761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:31 smithi161 ceph-mon[91579]: pgmap v9762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:31 smithi188 ceph-mon[92425]: pgmap v9762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:33 smithi161 ceph-mon[91579]: pgmap v9763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:33 smithi188 ceph-mon[92425]: pgmap v9763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:35 smithi161 ceph-mon[91579]: pgmap v9764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:35 smithi188 ceph-mon[92425]: pgmap v9764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:37 smithi161 ceph-mon[91579]: pgmap v9765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:37 smithi188 ceph-mon[92425]: pgmap v9765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:39 smithi161 ceph-mon[91579]: pgmap v9766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:39 smithi188 ceph-mon[92425]: pgmap v9766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:41 smithi161 ceph-mon[91579]: pgmap v9767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:41 smithi188 ceph-mon[92425]: pgmap v9767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:43 smithi161 ceph-mon[91579]: pgmap v9768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:43 smithi188 ceph-mon[92425]: pgmap v9768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:45 smithi161 ceph-mon[91579]: pgmap v9769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:45 smithi188 ceph-mon[92425]: pgmap v9769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:47 smithi161 ceph-mon[91579]: pgmap v9770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:47 smithi188 ceph-mon[92425]: pgmap v9770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:49 smithi161 ceph-mon[91579]: pgmap v9771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:49 smithi188 ceph-mon[92425]: pgmap v9771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:51 smithi161 ceph-mon[91579]: pgmap v9772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:51 smithi188 ceph-mon[92425]: pgmap v9772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:53 smithi161 ceph-mon[91579]: pgmap v9773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:53 smithi188 ceph-mon[92425]: pgmap v9773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:55 smithi161 ceph-mon[91579]: pgmap v9774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:55 smithi188 ceph-mon[92425]: pgmap v9774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:57 smithi161 ceph-mon[91579]: pgmap v9775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:04:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:57 smithi188 ceph-mon[92425]: pgmap v9775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:04:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:04:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:04:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:04:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:04:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:04:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:04:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:04:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:04:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:04:59 smithi161 ceph-mon[91579]: pgmap v9776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:04:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:04:59 smithi188 ceph-mon[92425]: pgmap v9776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:01 smithi161 ceph-mon[91579]: pgmap v9777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:01 smithi188 ceph-mon[92425]: pgmap v9777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:03 smithi161 ceph-mon[91579]: pgmap v9778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:03 smithi188 ceph-mon[92425]: pgmap v9778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:05:04.893+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:05:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:05:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:05:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:05:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:05:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:05:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:05:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:05:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:05:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:05:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:05:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:05:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:05:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:05:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:05:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:05:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:05:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:05:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:05:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:05:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:05:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:05:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:05:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:05:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:05:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:05:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:05:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:05:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:05:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:05:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:05:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:05:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:05:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:05:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:05:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:05:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:05:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:05:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:05:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:05:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:05:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:05:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:05:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:05:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:05:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=553] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:05:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=553] Local address: 172.21.15.161, port 56560 2022-07-03T16:05:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=553] Peer address: 172.21.15.101, port 22 2022-07-03T16:05:05.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=553] Connection lost 2022-07-03T16:05:05.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=553] Aborting connection 2022-07-03T16:05:05.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:05:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:05:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: pgmap v9779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:05:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:05:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:05:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:05:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:05:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:05:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:05:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:05:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:05:06.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:05:06.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:05:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:05:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:05:06.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:05:06.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:05:06.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:05:06.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:05:06.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:05:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:05:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:05:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:05:06.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:05:06.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:05:06.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:05:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:05:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:05:06.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:05:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:05:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:05:06.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:05:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:05:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:05:06.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:05:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:05:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:05:06.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:05:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:05:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:05:06.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:05:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:05:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:05:06.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:05:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:05:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:05:06.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:05:06.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:05:06.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:05:06.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:05:06.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:05:06.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=553] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:05:06.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=553] Local address: 172.21.15.161, port 56560 2022-07-03T16:05:06.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=553] Peer address: 172.21.15.101, port 22 2022-07-03T16:05:06.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=553] Connection lost 2022-07-03T16:05:06.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=553] Aborting connection 2022-07-03T16:05:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: pgmap v9779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:05:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:05:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:05:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:05:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:05:06.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:05:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:05:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:05:06.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:05:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:05:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:05:06.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:05:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:05:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:05:06.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:05:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:05:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:05:06.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:05:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:05:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:05:06.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:05:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:05:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:05:06.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:05:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:05:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:05:06.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:05:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:05:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:05:06.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:05:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:05:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:05:06.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:05:06.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:05:06.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:05:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:05:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:05:06.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:05:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:05:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:05:06.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:05:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:05:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:05:06.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:05:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:05:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:05:06.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:05:06.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:05:06.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:05:06.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=553] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:05:06.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=553] Local address: 172.21.15.161, port 56560 2022-07-03T16:05:06.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=553] Peer address: 172.21.15.101, port 22 2022-07-03T16:05:06.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=553] Connection lost 2022-07-03T16:05:06.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=553] Aborting connection 2022-07-03T16:05:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:07 smithi161 ceph-mon[91579]: pgmap v9780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:07 smithi188 ceph-mon[92425]: pgmap v9780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:09 smithi161 ceph-mon[91579]: pgmap v9781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:09 smithi188 ceph-mon[92425]: pgmap v9781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:11 smithi161 ceph-mon[91579]: pgmap v9782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:11 smithi188 ceph-mon[92425]: pgmap v9782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:13 smithi161 ceph-mon[91579]: pgmap v9783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:13 smithi188 ceph-mon[92425]: pgmap v9783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:15 smithi161 ceph-mon[91579]: pgmap v9784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:15 smithi188 ceph-mon[92425]: pgmap v9784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:17 smithi161 ceph-mon[91579]: pgmap v9785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:17 smithi188 ceph-mon[92425]: pgmap v9785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:19 smithi161 ceph-mon[91579]: pgmap v9786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:19 smithi188 ceph-mon[92425]: pgmap v9786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:21 smithi161 ceph-mon[91579]: pgmap v9787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:21 smithi188 ceph-mon[92425]: pgmap v9787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:23 smithi161 ceph-mon[91579]: pgmap v9788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:23 smithi188 ceph-mon[92425]: pgmap v9788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:25 smithi161 ceph-mon[91579]: pgmap v9789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:25 smithi188 ceph-mon[92425]: pgmap v9789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:27 smithi161 ceph-mon[91579]: pgmap v9790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:27 smithi188 ceph-mon[92425]: pgmap v9790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:29 smithi161 ceph-mon[91579]: pgmap v9791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:29 smithi188 ceph-mon[92425]: pgmap v9791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:31 smithi161 ceph-mon[91579]: pgmap v9792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:31 smithi188 ceph-mon[92425]: pgmap v9792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:33 smithi161 ceph-mon[91579]: pgmap v9793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:33 smithi188 ceph-mon[92425]: pgmap v9793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:35 smithi161 ceph-mon[91579]: pgmap v9794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:35 smithi188 ceph-mon[92425]: pgmap v9794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:37 smithi161 ceph-mon[91579]: pgmap v9795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:37 smithi188 ceph-mon[92425]: pgmap v9795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:39 smithi161 ceph-mon[91579]: pgmap v9796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:39 smithi188 ceph-mon[92425]: pgmap v9796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:41 smithi161 ceph-mon[91579]: pgmap v9797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:41 smithi188 ceph-mon[92425]: pgmap v9797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:43 smithi161 ceph-mon[91579]: pgmap v9798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:43 smithi188 ceph-mon[92425]: pgmap v9798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:45 smithi161 ceph-mon[91579]: pgmap v9799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:45 smithi188 ceph-mon[92425]: pgmap v9799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:47 smithi161 ceph-mon[91579]: pgmap v9800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:47 smithi188 ceph-mon[92425]: pgmap v9800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:49 smithi161 ceph-mon[91579]: pgmap v9801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:49 smithi188 ceph-mon[92425]: pgmap v9801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:51 smithi161 ceph-mon[91579]: pgmap v9802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:51 smithi188 ceph-mon[92425]: pgmap v9802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:52.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:53 smithi161 ceph-mon[91579]: pgmap v9803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:53 smithi188 ceph-mon[92425]: pgmap v9803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:55 smithi161 ceph-mon[91579]: pgmap v9804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:55 smithi188 ceph-mon[92425]: pgmap v9804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:57 smithi161 ceph-mon[91579]: pgmap v9805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:05:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:57 smithi188 ceph-mon[92425]: pgmap v9805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:05:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:05:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:05:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:05:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:05:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:05:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:05:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:05:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:05:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:05:59 smithi161 ceph-mon[91579]: pgmap v9806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:05:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:05:59 smithi188 ceph-mon[92425]: pgmap v9806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:01 smithi161 ceph-mon[91579]: pgmap v9807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:01 smithi188 ceph-mon[92425]: pgmap v9807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:03 smithi161 ceph-mon[91579]: pgmap v9808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:03 smithi188 ceph-mon[92425]: pgmap v9808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:05.732 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:06:05.399+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:06:05.733 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:06:05.733 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:06:05.734 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:06:05.734 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:06:05.735 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:06:05.735 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:06:05.736 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:06:05.736 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:06:05.737 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:06:05.737 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:06:05.737 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:06:05.738 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:06:05.738 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:06:05.738 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:06:05.739 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:06:05.739 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:06:05.739 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:06:05.740 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:06:05.741 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:06:05.741 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:06:05.741 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:06:05.742 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:06:05.742 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:06:05.742 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:06:05.743 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:06:05.743 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:06:05.744 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:06:05.745 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:06:05.745 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:06:05.745 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:06:05.746 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:06:05.746 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:06:05.747 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:06:05.747 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:06:05.748 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:06:05.748 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:06:05.749 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:06:05.749 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:06:05.750 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:06:05.750 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:06:05.751 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:06:05.752 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:06:05.752 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:06:05.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:06:05.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=555] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:06:05.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=555] Local address: 172.21.15.161, port 56564 2022-07-03T16:06:05.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=555] Peer address: 172.21.15.101, port 22 2022-07-03T16:06:05.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=555] Connection lost 2022-07-03T16:06:05.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=555] Aborting connection 2022-07-03T16:06:05.755 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:06:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-mon[91579]: pgmap v9809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:06:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:06:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:06:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:05 smithi188 ceph-mon[92425]: pgmap v9809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:06:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:06:06.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:06:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:06:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:06:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:06:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:06:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:06:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:06:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:06:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:06:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:06:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:06:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:06:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:06:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:06:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:06:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:06:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:06:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:06:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:06:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:06:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:06:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:06:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:06:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:06:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:06:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:06:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:06:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:06:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:06:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:06:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:06:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:06:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:06:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:06:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:06:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:06:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:06:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:06:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:06:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:06:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:06:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:06:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:06:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:06:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:06:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:06:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:06:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=555] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:06:07.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=555] Local address: 172.21.15.161, port 56564 2022-07-03T16:06:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=555] Peer address: 172.21.15.101, port 22 2022-07-03T16:06:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=555] Connection lost 2022-07-03T16:06:07.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=555] Aborting connection 2022-07-03T16:06:07.154 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:06:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:06:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:06:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:06:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:06:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:06:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:06:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:06:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:06:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:06:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:06:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:06:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:06:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:06:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:06:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:06:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:06:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:06:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:06:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:06:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:06:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:06:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:06:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:06:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:06:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:06:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:06:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:06:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:06:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:06:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:06:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:06:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:06:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:06:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:06:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:06:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:06:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:06:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:06:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:06:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:06:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:06:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:06:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:06:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:06:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:06:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=555] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:06:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=555] Local address: 172.21.15.161, port 56564 2022-07-03T16:06:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=555] Peer address: 172.21.15.101, port 22 2022-07-03T16:06:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=555] Connection lost 2022-07-03T16:06:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=555] Aborting connection 2022-07-03T16:06:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:07 smithi161 ceph-mon[91579]: pgmap v9810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:07 smithi188 ceph-mon[92425]: pgmap v9810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:09 smithi161 ceph-mon[91579]: pgmap v9811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:09 smithi188 ceph-mon[92425]: pgmap v9811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:11 smithi161 ceph-mon[91579]: pgmap v9812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:11 smithi188 ceph-mon[92425]: pgmap v9812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:13 smithi161 ceph-mon[91579]: pgmap v9813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:13 smithi188 ceph-mon[92425]: pgmap v9813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:15 smithi161 ceph-mon[91579]: pgmap v9814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:15 smithi188 ceph-mon[92425]: pgmap v9814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:17 smithi161 ceph-mon[91579]: pgmap v9815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:17 smithi188 ceph-mon[92425]: pgmap v9815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:19 smithi161 ceph-mon[91579]: pgmap v9816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:19 smithi188 ceph-mon[92425]: pgmap v9816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:21 smithi161 ceph-mon[91579]: pgmap v9817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:21 smithi188 ceph-mon[92425]: pgmap v9817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:23 smithi161 ceph-mon[91579]: pgmap v9818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:23 smithi188 ceph-mon[92425]: pgmap v9818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:25 smithi161 ceph-mon[91579]: pgmap v9819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:25 smithi188 ceph-mon[92425]: pgmap v9819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:27 smithi161 ceph-mon[91579]: pgmap v9820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:27 smithi188 ceph-mon[92425]: pgmap v9820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:29 smithi161 ceph-mon[91579]: pgmap v9821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:29 smithi188 ceph-mon[92425]: pgmap v9821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:31 smithi161 ceph-mon[91579]: pgmap v9822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:31 smithi188 ceph-mon[92425]: pgmap v9822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:33 smithi161 ceph-mon[91579]: pgmap v9823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:33 smithi188 ceph-mon[92425]: pgmap v9823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:35 smithi161 ceph-mon[91579]: pgmap v9824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:35 smithi188 ceph-mon[92425]: pgmap v9824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:37 smithi161 ceph-mon[91579]: pgmap v9825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:37 smithi188 ceph-mon[92425]: pgmap v9825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:39 smithi161 ceph-mon[91579]: pgmap v9826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:39 smithi188 ceph-mon[92425]: pgmap v9826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:41 smithi161 ceph-mon[91579]: pgmap v9827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:41 smithi188 ceph-mon[92425]: pgmap v9827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:43 smithi161 ceph-mon[91579]: pgmap v9828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:43 smithi188 ceph-mon[92425]: pgmap v9828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:45 smithi161 ceph-mon[91579]: pgmap v9829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:45 smithi188 ceph-mon[92425]: pgmap v9829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:47 smithi161 ceph-mon[91579]: pgmap v9830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:47 smithi188 ceph-mon[92425]: pgmap v9830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:49 smithi161 ceph-mon[91579]: pgmap v9831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:49 smithi188 ceph-mon[92425]: pgmap v9831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:51 smithi161 ceph-mon[91579]: pgmap v9832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:51 smithi188 ceph-mon[92425]: pgmap v9832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:53 smithi161 ceph-mon[91579]: pgmap v9833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:53 smithi188 ceph-mon[92425]: pgmap v9833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:55 smithi161 ceph-mon[91579]: pgmap v9834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:55 smithi188 ceph-mon[92425]: pgmap v9834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:57 smithi161 ceph-mon[91579]: pgmap v9835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:06:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:57 smithi188 ceph-mon[92425]: pgmap v9835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:06:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:06:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:06:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:06:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:06:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:06:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:06:59 smithi161 ceph-mon[91579]: pgmap v9836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:06:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:06:59 smithi188 ceph-mon[92425]: pgmap v9836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:01 smithi161 ceph-mon[91579]: pgmap v9837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:01 smithi188 ceph-mon[92425]: pgmap v9837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:03 smithi161 ceph-mon[91579]: pgmap v9838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:03 smithi188 ceph-mon[92425]: pgmap v9838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-mon[91579]: pgmap v9839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:07:05.888+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:07:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:07:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:07:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:07:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:07:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:07:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:07:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:07:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:07:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:07:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:07:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:07:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:07:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:07:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:07:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:07:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:07:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:07:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:07:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:07:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:07:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:07:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:07:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:07:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:07:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:07:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:07:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:07:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:07:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:07:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:07:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:07:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:07:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:07:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:07:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:07:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:07:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:07:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:07:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:07:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:07:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:07:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:07:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:07:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:07:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=557] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:07:06.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=557] Local address: 172.21.15.161, port 56568 2022-07-03T16:07:06.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=557] Peer address: 172.21.15.101, port 22 2022-07-03T16:07:06.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=557] Connection lost 2022-07-03T16:07:06.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=557] Aborting connection 2022-07-03T16:07:06.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:07:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:07:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:05 smithi188 ceph-mon[92425]: pgmap v9839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:07:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:07:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:07:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:07:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:07:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:07:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:07:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:07:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:07:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:07:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:07:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:07:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:07:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:07:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:07:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:07:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:07:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:07:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:07:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:07:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:07:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:07:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:07:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:07:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:07:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:07:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:07:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:07:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:07:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:07:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:07:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:07:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:07:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:07:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:07:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:07:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:07:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:07:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:07:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:07:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:07:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:07:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:07:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:07:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:07:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:07:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:07:07.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:07:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:07:07.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=557] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:07:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=557] Local address: 172.21.15.161, port 56568 2022-07-03T16:07:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=557] Peer address: 172.21.15.101, port 22 2022-07-03T16:07:07.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=557] Connection lost 2022-07-03T16:07:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=557] Aborting connection 2022-07-03T16:07:07.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:07:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:07:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:07:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:07:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:07:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:07:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:07:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:07:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:07:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:07:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:07:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:07:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:07:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:07:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:07:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:07:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:07:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:07:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:07:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:07:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:07:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:07:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:07:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:07:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:07:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:07:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:07:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:07:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:07:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:07:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:07:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:07:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:07:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:07:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:07:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:07:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:07:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:07:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:07:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:07:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:07:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:07:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:07:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:07:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:07:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:07:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:07:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:07:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:07:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=557] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:07:07.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=557] Local address: 172.21.15.161, port 56568 2022-07-03T16:07:07.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=557] Peer address: 172.21.15.101, port 22 2022-07-03T16:07:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=557] Connection lost 2022-07-03T16:07:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=557] Aborting connection 2022-07-03T16:07:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:07 smithi161 ceph-mon[91579]: pgmap v9840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:07 smithi188 ceph-mon[92425]: pgmap v9840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: pgmap v9841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:07:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: pgmap v9841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:09.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:09.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:07:10.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 136.3M 2022-07-03T16:07:10.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 142957879: error parsing value: Value '142957879' is below minimum 939524096 2022-07-03T16:07:10.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:07:10.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:07:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 136.3M 2022-07-03T16:07:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 142957879: error parsing value: Value '142957879' is below minimum 939524096 2022-07-03T16:07:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:07:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:07:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:11 smithi188 ceph-mon[92425]: pgmap v9842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:11 smithi161 ceph-mon[91579]: pgmap v9842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:13 smithi161 ceph-mon[91579]: pgmap v9843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:13 smithi188 ceph-mon[92425]: pgmap v9843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:15 smithi161 ceph-mon[91579]: pgmap v9844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:15 smithi188 ceph-mon[92425]: pgmap v9844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:17 smithi161 ceph-mon[91579]: pgmap v9845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:17 smithi188 ceph-mon[92425]: pgmap v9845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:19 smithi161 ceph-mon[91579]: pgmap v9846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:19 smithi188 ceph-mon[92425]: pgmap v9846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:21 smithi161 ceph-mon[91579]: pgmap v9847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:21 smithi188 ceph-mon[92425]: pgmap v9847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:23 smithi161 ceph-mon[91579]: pgmap v9848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:23 smithi188 ceph-mon[92425]: pgmap v9848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:25 smithi161 ceph-mon[91579]: pgmap v9849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:25 smithi188 ceph-mon[92425]: pgmap v9849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:27 smithi161 ceph-mon[91579]: pgmap v9850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:27 smithi188 ceph-mon[92425]: pgmap v9850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:29 smithi161 ceph-mon[91579]: pgmap v9851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:29 smithi188 ceph-mon[92425]: pgmap v9851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:31 smithi161 ceph-mon[91579]: pgmap v9852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:31 smithi188 ceph-mon[92425]: pgmap v9852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:33 smithi161 ceph-mon[91579]: pgmap v9853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:33 smithi188 ceph-mon[92425]: pgmap v9853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:35 smithi161 ceph-mon[91579]: pgmap v9854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:35 smithi188 ceph-mon[92425]: pgmap v9854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:37 smithi161 ceph-mon[91579]: pgmap v9855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:37 smithi188 ceph-mon[92425]: pgmap v9855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:39 smithi161 ceph-mon[91579]: pgmap v9856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:39 smithi188 ceph-mon[92425]: pgmap v9856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:41 smithi161 ceph-mon[91579]: pgmap v9857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:41 smithi188 ceph-mon[92425]: pgmap v9857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:43 smithi161 ceph-mon[91579]: pgmap v9858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:43 smithi188 ceph-mon[92425]: pgmap v9858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:45 smithi161 ceph-mon[91579]: pgmap v9859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:45 smithi188 ceph-mon[92425]: pgmap v9859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:47 smithi161 ceph-mon[91579]: pgmap v9860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:47 smithi188 ceph-mon[92425]: pgmap v9860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:49 smithi161 ceph-mon[91579]: pgmap v9861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:49 smithi188 ceph-mon[92425]: pgmap v9861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:51 smithi161 ceph-mon[91579]: pgmap v9862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:51 smithi188 ceph-mon[92425]: pgmap v9862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:53 smithi161 ceph-mon[91579]: pgmap v9863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:53 smithi188 ceph-mon[92425]: pgmap v9863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:55 smithi161 ceph-mon[91579]: pgmap v9864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:55 smithi188 ceph-mon[92425]: pgmap v9864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:57 smithi161 ceph-mon[91579]: pgmap v9865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:07:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:57 smithi188 ceph-mon[92425]: pgmap v9865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:07:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:07:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:07:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:07:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:07:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:07:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:07:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:07:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:07:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:07:59 smithi161 ceph-mon[91579]: pgmap v9866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:07:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:07:59 smithi188 ceph-mon[92425]: pgmap v9866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:01 smithi161 ceph-mon[91579]: pgmap v9867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:01 smithi188 ceph-mon[92425]: pgmap v9867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:03 smithi161 ceph-mon[91579]: pgmap v9868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:03 smithi188 ceph-mon[92425]: pgmap v9868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:05 smithi161 ceph-mon[91579]: pgmap v9869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:05 smithi188 ceph-mon[92425]: pgmap v9869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:07 smithi161 ceph-mon[91579]: pgmap v9870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:07 smithi188 ceph-mon[92425]: pgmap v9870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:09.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:08:09.140+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:08:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:08:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:08:09.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:08:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:08:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:08:09.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:08:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:08:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:08:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:08:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:08:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:08:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:08:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:08:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:08:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:08:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:08:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:08:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:08:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:08:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:08:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:08:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:08:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:08:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:08:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:08:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:08:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:08:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:08:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:08:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:08:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:08:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:08:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:08:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:08:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:08:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:08:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:08:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:08:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:08:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:08:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:08:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:08:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:08:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:08:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=559] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:08:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=559] Local address: 172.21.15.161, port 56572 2022-07-03T16:08:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=559] Peer address: 172.21.15.101, port 22 2022-07-03T16:08:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=559] Connection lost 2022-07-03T16:08:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=559] Aborting connection 2022-07-03T16:08:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:08:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-mon[91579]: pgmap v9871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:08:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:08:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:08:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:09 smithi188 ceph-mon[92425]: pgmap v9871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:08:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:08:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:08:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:08:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:08:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:08:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:08:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:08:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:08:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:08:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:08:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:08:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:08:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:08:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:08:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:08:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:08:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:08:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:08:11.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:08:11.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:08:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:08:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:08:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:08:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:08:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:08:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:08:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:08:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:08:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:08:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:08:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:08:11.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:08:11.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:08:11.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:08:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:08:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:08:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:08:11.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:08:11.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:08:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:08:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:08:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:08:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:08:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:08:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:08:11.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:08:11.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:08:11.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:08:11.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=559] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:08:11.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=559] Local address: 172.21.15.161, port 56572 2022-07-03T16:08:11.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=559] Peer address: 172.21.15.101, port 22 2022-07-03T16:08:11.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=559] Connection lost 2022-07-03T16:08:11.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=559] Aborting connection 2022-07-03T16:08:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:08:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:08:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:08:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:08:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:08:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:08:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:08:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:08:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:08:11.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:08:11.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:08:11.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:08:11.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:08:11.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:08:11.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:08:11.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:08:11.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:08:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:08:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:08:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:08:11.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:08:11.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:08:11.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:08:11.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:08:11.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:08:11.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:08:11.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:08:11.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:08:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:08:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:08:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:08:11.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:08:11.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:08:11.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:08:11.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:08:11.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:08:11.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:08:11.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:08:11.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:08:11.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:08:11.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:08:11.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:08:11.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:08:11.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:08:11.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:08:11.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:08:11.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=559] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:08:11.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=559] Local address: 172.21.15.161, port 56572 2022-07-03T16:08:11.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=559] Peer address: 172.21.15.101, port 22 2022-07-03T16:08:11.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=559] Connection lost 2022-07-03T16:08:11.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=559] Aborting connection 2022-07-03T16:08:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:11 smithi161 ceph-mon[91579]: pgmap v9872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:11 smithi188 ceph-mon[92425]: pgmap v9872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:13 smithi161 ceph-mon[91579]: pgmap v9873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:13 smithi188 ceph-mon[92425]: pgmap v9873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:15 smithi161 ceph-mon[91579]: pgmap v9874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:15 smithi188 ceph-mon[92425]: pgmap v9874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:17 smithi161 ceph-mon[91579]: pgmap v9875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:17 smithi188 ceph-mon[92425]: pgmap v9875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:19 smithi161 ceph-mon[91579]: pgmap v9876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:19 smithi188 ceph-mon[92425]: pgmap v9876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:21 smithi161 ceph-mon[91579]: pgmap v9877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:21 smithi188 ceph-mon[92425]: pgmap v9877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:23 smithi161 ceph-mon[91579]: pgmap v9878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:23 smithi188 ceph-mon[92425]: pgmap v9878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:25 smithi161 ceph-mon[91579]: pgmap v9879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:25 smithi188 ceph-mon[92425]: pgmap v9879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:27 smithi161 ceph-mon[91579]: pgmap v9880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:27 smithi188 ceph-mon[92425]: pgmap v9880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:29 smithi161 ceph-mon[91579]: pgmap v9881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:29 smithi188 ceph-mon[92425]: pgmap v9881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:31 smithi161 ceph-mon[91579]: pgmap v9882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:31 smithi188 ceph-mon[92425]: pgmap v9882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:33 smithi161 ceph-mon[91579]: pgmap v9883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:33 smithi188 ceph-mon[92425]: pgmap v9883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:35 smithi161 ceph-mon[91579]: pgmap v9884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:35 smithi188 ceph-mon[92425]: pgmap v9884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:37 smithi161 ceph-mon[91579]: pgmap v9885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:37 smithi188 ceph-mon[92425]: pgmap v9885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:39 smithi161 ceph-mon[91579]: pgmap v9886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:39 smithi188 ceph-mon[92425]: pgmap v9886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:41 smithi161 ceph-mon[91579]: pgmap v9887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:41 smithi188 ceph-mon[92425]: pgmap v9887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:42.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:43 smithi161 ceph-mon[91579]: pgmap v9888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:43 smithi188 ceph-mon[92425]: pgmap v9888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:45 smithi161 ceph-mon[91579]: pgmap v9889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:45 smithi188 ceph-mon[92425]: pgmap v9889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:47 smithi161 ceph-mon[91579]: pgmap v9890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:47 smithi188 ceph-mon[92425]: pgmap v9890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:49 smithi161 ceph-mon[91579]: pgmap v9891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:49 smithi188 ceph-mon[92425]: pgmap v9891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:51 smithi161 ceph-mon[91579]: pgmap v9892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:51 smithi188 ceph-mon[92425]: pgmap v9892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:53 smithi161 ceph-mon[91579]: pgmap v9893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:53 smithi188 ceph-mon[92425]: pgmap v9893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:55 smithi161 ceph-mon[91579]: pgmap v9894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:55 smithi188 ceph-mon[92425]: pgmap v9894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:57 smithi161 ceph-mon[91579]: pgmap v9895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:08:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:57 smithi188 ceph-mon[92425]: pgmap v9895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:08:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:08:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:08:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:08:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:08:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:08:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:08:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:08:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:08:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:08:59 smithi161 ceph-mon[91579]: pgmap v9896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:08:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:08:59 smithi188 ceph-mon[92425]: pgmap v9896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:01 smithi161 ceph-mon[91579]: pgmap v9897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:01 smithi188 ceph-mon[92425]: pgmap v9897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:03 smithi161 ceph-mon[91579]: pgmap v9898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:03 smithi188 ceph-mon[92425]: pgmap v9898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:05 smithi161 ceph-mon[91579]: pgmap v9899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:05 smithi188 ceph-mon[92425]: pgmap v9899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:07 smithi161 ceph-mon[91579]: pgmap v9900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:07 smithi188 ceph-mon[92425]: pgmap v9900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:09 smithi188 ceph-mon[92425]: pgmap v9901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:09:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:09:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:09:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-mon[91579]: pgmap v9901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:09:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:09:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:09:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:09:09.639+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:09:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:09:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:09:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:09:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:09:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:09:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:09:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:09:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:09:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:09:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:09:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:09:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:09:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:09:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:09:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:09:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:09:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:09:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:09:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:09:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:09:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:09:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:09:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:09:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:09:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:09:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:09:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:09:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:09:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:09:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:09:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:09:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:09:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:09:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:09:10.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:09:10.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:09:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:09:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:09:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:09:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:09:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:09:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:09:10.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:09:10.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:09:10.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=561] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:09:10.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=561] Local address: 172.21.15.161, port 56604 2022-07-03T16:09:10.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=561] Peer address: 172.21.15.101, port 22 2022-07-03T16:09:10.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=561] Connection lost 2022-07-03T16:09:10.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=561] Aborting connection 2022-07-03T16:09:10.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:09:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:09:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:09:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:09:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:09:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:09:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:09:11.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:09:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:09:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:09:11.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:09:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:09:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:09:11.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:09:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:09:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:09:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:09:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:09:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:09:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:09:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:09:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:09:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:09:11.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:09:11.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:09:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:09:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:09:11.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:09:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:09:11.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:09:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:09:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:09:11.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:09:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:09:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:09:11.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:09:11.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:09:11.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:09:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:09:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:09:11.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:09:11.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:09:11.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:09:11.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:09:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:09:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:09:11.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:09:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:09:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=561] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:09:11.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=561] Local address: 172.21.15.161, port 56604 2022-07-03T16:09:11.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=561] Peer address: 172.21.15.101, port 22 2022-07-03T16:09:11.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=561] Connection lost 2022-07-03T16:09:11.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=561] Aborting connection 2022-07-03T16:09:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:09:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:09:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:09:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:09:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:09:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:09:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:09:11.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:09:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:09:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:09:11.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:09:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:09:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:09:11.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:09:11.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:09:11.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:09:11.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:09:11.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:09:11.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:09:11.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:09:11.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:09:11.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:09:11.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:09:11.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:09:11.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:09:11.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:09:11.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:09:11.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:09:11.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:09:11.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:09:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:09:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:09:11.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:09:11.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:09:11.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:09:11.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:09:11.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:09:11.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:09:11.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:09:11.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:09:11.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:09:11.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:09:11.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:09:11.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:09:11.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:09:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:09:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=561] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:09:11.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=561] Local address: 172.21.15.161, port 56604 2022-07-03T16:09:11.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=561] Peer address: 172.21.15.101, port 22 2022-07-03T16:09:11.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=561] Connection lost 2022-07-03T16:09:11.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=561] Aborting connection 2022-07-03T16:09:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:11 smithi161 ceph-mon[91579]: pgmap v9902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:11 smithi188 ceph-mon[92425]: pgmap v9902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:13 smithi161 ceph-mon[91579]: pgmap v9903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:13 smithi188 ceph-mon[92425]: pgmap v9903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:15 smithi161 ceph-mon[91579]: pgmap v9904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:15 smithi188 ceph-mon[92425]: pgmap v9904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:17 smithi161 ceph-mon[91579]: pgmap v9905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:17 smithi188 ceph-mon[92425]: pgmap v9905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:19 smithi161 ceph-mon[91579]: pgmap v9906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:19 smithi188 ceph-mon[92425]: pgmap v9906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:21 smithi161 ceph-mon[91579]: pgmap v9907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:21 smithi188 ceph-mon[92425]: pgmap v9907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:23 smithi161 ceph-mon[91579]: pgmap v9908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:23 smithi188 ceph-mon[92425]: pgmap v9908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:25 smithi161 ceph-mon[91579]: pgmap v9909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:25 smithi188 ceph-mon[92425]: pgmap v9909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:27 smithi161 ceph-mon[91579]: pgmap v9910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:27 smithi188 ceph-mon[92425]: pgmap v9910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:29 smithi161 ceph-mon[91579]: pgmap v9911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:29 smithi188 ceph-mon[92425]: pgmap v9911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:31 smithi161 ceph-mon[91579]: pgmap v9912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:31 smithi188 ceph-mon[92425]: pgmap v9912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:33 smithi161 ceph-mon[91579]: pgmap v9913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:34.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:33 smithi188 ceph-mon[92425]: pgmap v9913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:35 smithi161 ceph-mon[91579]: pgmap v9914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:35 smithi188 ceph-mon[92425]: pgmap v9914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:37 smithi161 ceph-mon[91579]: pgmap v9915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:37 smithi188 ceph-mon[92425]: pgmap v9915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:39 smithi161 ceph-mon[91579]: pgmap v9916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:39 smithi188 ceph-mon[92425]: pgmap v9916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:41 smithi161 ceph-mon[91579]: pgmap v9917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:41 smithi188 ceph-mon[92425]: pgmap v9917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:43 smithi161 ceph-mon[91579]: pgmap v9918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:43 smithi188 ceph-mon[92425]: pgmap v9918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:45 smithi161 ceph-mon[91579]: pgmap v9919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:45 smithi188 ceph-mon[92425]: pgmap v9919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:47 smithi161 ceph-mon[91579]: pgmap v9920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:47 smithi188 ceph-mon[92425]: pgmap v9920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:49 smithi161 ceph-mon[91579]: pgmap v9921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:49 smithi188 ceph-mon[92425]: pgmap v9921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:51 smithi161 ceph-mon[91579]: pgmap v9922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:51 smithi188 ceph-mon[92425]: pgmap v9922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:53 smithi161 ceph-mon[91579]: pgmap v9923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:53 smithi188 ceph-mon[92425]: pgmap v9923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:55 smithi161 ceph-mon[91579]: pgmap v9924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:55 smithi188 ceph-mon[92425]: pgmap v9924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:57 smithi161 ceph-mon[91579]: pgmap v9925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:09:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:57 smithi188 ceph-mon[92425]: pgmap v9925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:09:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:09:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:09:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:09:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:09:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:09:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:09:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:09:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:09:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:09:59 smithi161 ceph-mon[91579]: pgmap v9926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:09:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:09:59 smithi188 ceph-mon[92425]: pgmap v9926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:10:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:10:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:01 smithi161 ceph-mon[91579]: pgmap v9927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:01 smithi188 ceph-mon[92425]: pgmap v9927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:03 smithi161 ceph-mon[91579]: pgmap v9928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:03 smithi188 ceph-mon[92425]: pgmap v9928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:05 smithi161 ceph-mon[91579]: pgmap v9929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:05 smithi188 ceph-mon[92425]: pgmap v9929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:07 smithi161 ceph-mon[91579]: pgmap v9930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:07 smithi188 ceph-mon[92425]: pgmap v9930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:09 smithi161 ceph-mon[91579]: pgmap v9931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:10.152 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:09 smithi188 ceph-mon[92425]: pgmap v9931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:10.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:10:10.141+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:10:10.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:10:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:10:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:10:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:10:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:10:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:10:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:10:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:10:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:10:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:10:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:10:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:10:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:10:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:10:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:10:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:10:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:10:10.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:10:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:10:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:10:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:10:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:10:10.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:10:10.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:10:10.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:10:10.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:10:10.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:10:10.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:10:10.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:10:10.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:10:10.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:10:10.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:10:10.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:10:10.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:10:10.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:10:10.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:10:10.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:10:10.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:10:10.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:10:10.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:10:10.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:10:10.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:10:10.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:10:10.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:10:10.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=563] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:10:10.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=563] Local address: 172.21.15.161, port 56608 2022-07-03T16:10:10.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=563] Peer address: 172.21.15.101, port 22 2022-07-03T16:10:10.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=563] Connection lost 2022-07-03T16:10:10.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=563] Aborting connection 2022-07-03T16:10:10.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:10:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:10:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:10:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:10:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:10:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:10:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:10:12.018 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:10:12.019 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:10:12.019 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:10:12.019 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:10:12.020 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:10:12.020 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:10:12.020 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:10:12.021 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:10:12.021 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:10:12.021 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:10:12.022 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:10:12.022 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:10:12.022 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:10:12.023 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:10:12.023 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:10:12.023 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:10:12.024 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:10:12.024 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:10:12.025 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:10:12.025 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:10:12.025 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:10:12.026 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:10:12.026 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:10:12.026 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:10:12.027 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:10:12.027 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:10:12.027 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:10:12.028 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:10:12.028 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:10:12.028 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:10:12.029 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:10:12.029 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:10:12.029 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:10:12.029 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:10:12.030 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:10:12.030 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:10:12.031 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:10:12.031 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:10:12.031 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:10:12.031 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:10:12.032 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:10:12.032 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:10:12.032 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:10:12.033 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:10:12.033 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:10:12.033 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:10:12.034 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=563] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:10:12.034 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=563] Local address: 172.21.15.161, port 56608 2022-07-03T16:10:12.034 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=563] Peer address: 172.21.15.101, port 22 2022-07-03T16:10:12.035 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=563] Connection lost 2022-07-03T16:10:12.036 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=563] Aborting connection 2022-07-03T16:10:12.036 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:11 smithi161 ceph-mon[91579]: pgmap v9932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:12.036 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:10:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:10:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:10:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:10:12.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:10:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:10:12.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:10:12.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:10:12.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:10:12.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:10:12.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:10:12.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:10:12.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:10:12.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:10:12.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:10:12.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:10:12.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:10:12.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:10:12.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:10:12.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:10:12.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:10:12.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:10:12.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:10:12.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:10:12.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:10:12.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:10:12.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:10:12.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:10:12.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:10:12.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:10:12.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:10:12.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:10:12.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:10:12.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:10:12.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:10:12.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:10:12.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:10:12.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:10:12.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:10:12.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:10:12.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:10:12.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:10:12.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:10:12.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:10:12.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:10:12.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:10:12.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=563] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:10:12.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=563] Local address: 172.21.15.161, port 56608 2022-07-03T16:10:12.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=563] Peer address: 172.21.15.101, port 22 2022-07-03T16:10:12.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=563] Connection lost 2022-07-03T16:10:12.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=563] Aborting connection 2022-07-03T16:10:12.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:11 smithi188 ceph-mon[92425]: pgmap v9932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:12.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:13 smithi161 ceph-mon[91579]: pgmap v9933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:10:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:10:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:13 smithi188 ceph-mon[92425]: pgmap v9933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:10:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:10:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:14 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T16:10:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:14 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T16:10:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:15 smithi161 ceph-mon[91579]: pgmap v9934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:15 smithi188 ceph-mon[92425]: pgmap v9934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:17 smithi161 ceph-mon[91579]: pgmap v9935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:17 smithi188 ceph-mon[92425]: pgmap v9935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:19 smithi161 ceph-mon[91579]: pgmap v9936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:19 smithi188 ceph-mon[92425]: pgmap v9936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:21 smithi161 ceph-mon[91579]: pgmap v9937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:21 smithi188 ceph-mon[92425]: pgmap v9937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:23 smithi161 ceph-mon[91579]: pgmap v9938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:23 smithi188 ceph-mon[92425]: pgmap v9938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:25 smithi161 ceph-mon[91579]: pgmap v9939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:25 smithi188 ceph-mon[92425]: pgmap v9939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:27 smithi161 ceph-mon[91579]: pgmap v9940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:27 smithi188 ceph-mon[92425]: pgmap v9940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:29 smithi161 ceph-mon[91579]: pgmap v9941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:29 smithi188 ceph-mon[92425]: pgmap v9941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:31 smithi161 ceph-mon[91579]: pgmap v9942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:31 smithi188 ceph-mon[92425]: pgmap v9942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:33 smithi161 ceph-mon[91579]: pgmap v9943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:33 smithi188 ceph-mon[92425]: pgmap v9943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:35 smithi161 ceph-mon[91579]: pgmap v9944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:35 smithi188 ceph-mon[92425]: pgmap v9944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:37 smithi161 ceph-mon[91579]: pgmap v9945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:37 smithi188 ceph-mon[92425]: pgmap v9945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:40.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:39 smithi161 ceph-mon[91579]: pgmap v9946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:39 smithi188 ceph-mon[92425]: pgmap v9946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:41 smithi161 ceph-mon[91579]: pgmap v9947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:41 smithi188 ceph-mon[92425]: pgmap v9947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:43 smithi161 ceph-mon[91579]: pgmap v9948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:43 smithi188 ceph-mon[92425]: pgmap v9948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:45 smithi161 ceph-mon[91579]: pgmap v9949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:45 smithi188 ceph-mon[92425]: pgmap v9949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:47 smithi161 ceph-mon[91579]: pgmap v9950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:47 smithi188 ceph-mon[92425]: pgmap v9950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:49 smithi161 ceph-mon[91579]: pgmap v9951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:49 smithi188 ceph-mon[92425]: pgmap v9951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:51 smithi161 ceph-mon[91579]: pgmap v9952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:51 smithi188 ceph-mon[92425]: pgmap v9952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:53 smithi161 ceph-mon[91579]: pgmap v9953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:53 smithi188 ceph-mon[92425]: pgmap v9953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:55 smithi161 ceph-mon[91579]: pgmap v9954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:55 smithi188 ceph-mon[92425]: pgmap v9954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:57 smithi161 ceph-mon[91579]: pgmap v9955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:10:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:57 smithi188 ceph-mon[92425]: pgmap v9955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:10:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:10:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:10:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:10:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:10:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:10:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:10:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:10:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:10:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:10:59 smithi161 ceph-mon[91579]: pgmap v9956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:10:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:10:59 smithi188 ceph-mon[92425]: pgmap v9956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:01 smithi161 ceph-mon[91579]: pgmap v9957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:01 smithi188 ceph-mon[92425]: pgmap v9957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:03 smithi161 ceph-mon[91579]: pgmap v9958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:03 smithi188 ceph-mon[92425]: pgmap v9958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:05 smithi161 ceph-mon[91579]: pgmap v9959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:05 smithi188 ceph-mon[92425]: pgmap v9959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:07 smithi161 ceph-mon[91579]: pgmap v9960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:07 smithi188 ceph-mon[92425]: pgmap v9960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:09 smithi161 ceph-mon[91579]: pgmap v9961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:09 smithi188 ceph-mon[92425]: pgmap v9961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:11 smithi161 ceph-mon[91579]: pgmap v9962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:11 smithi188 ceph-mon[92425]: pgmap v9962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:13.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:11:13.435+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:11:13.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:11:13.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:11:13.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:11:13.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:11:13.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:11:13.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:11:13.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:11:13.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:11:13.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:11:13.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:11:13.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:11:13.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:11:13.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:11:13.835 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:11:13.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:11:13.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:11:13.836 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:11:13.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:11:13.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:11:13.837 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:11:13.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:11:13.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:11:13.838 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:11:13.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:11:13.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:11:13.839 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:11:13.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:11:13.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:11:13.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:11:13.840 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:11:13.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:11:13.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:11:13.841 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:11:13.842 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:11:13.842 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:11:13.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:11:13.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:11:13.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:11:13.843 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:11:13.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:11:13.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:11:13.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:11:13.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:11:13.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:11:13.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=565] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:11:13.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=565] Local address: 172.21.15.161, port 56612 2022-07-03T16:11:13.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=565] Peer address: 172.21.15.101, port 22 2022-07-03T16:11:13.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=565] Connection lost 2022-07-03T16:11:13.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=565] Aborting connection 2022-07-03T16:11:13.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:11:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-mon[91579]: pgmap v9963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:11:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:11:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:11:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:13 smithi188 ceph-mon[92425]: pgmap v9963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:11:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:11:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:11:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:11:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:11:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:11:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:11:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:11:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:11:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:11:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:11:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:11:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:11:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:11:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:11:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:11:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:11:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:11:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:11:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:11:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:11:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:11:15.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:11:15.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:11:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:11:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:11:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:11:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:11:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:11:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:11:15.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:11:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:11:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:11:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:11:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:11:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:11:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:11:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:11:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:11:15.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:11:15.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:11:15.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:11:15.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:11:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:11:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:11:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:11:15.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:11:15.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:11:15.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:11:15.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=565] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:11:15.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=565] Local address: 172.21.15.161, port 56612 2022-07-03T16:11:15.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=565] Peer address: 172.21.15.101, port 22 2022-07-03T16:11:15.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=565] Connection lost 2022-07-03T16:11:15.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=565] Aborting connection 2022-07-03T16:11:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:11:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:11:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:11:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:11:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:11:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:11:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:11:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:11:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:11:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:11:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:11:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:11:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:11:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:11:15.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:11:15.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:11:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:11:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:11:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:11:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:11:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:11:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:11:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:11:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:11:15.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:11:15.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:11:15.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:11:15.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:11:15.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:11:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:11:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:11:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:11:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:11:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:11:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:11:15.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:11:15.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:11:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:11:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:11:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:11:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:11:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:11:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:11:15.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:11:15.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:11:15.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:11:15.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=565] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:11:15.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=565] Local address: 172.21.15.161, port 56612 2022-07-03T16:11:15.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=565] Peer address: 172.21.15.101, port 22 2022-07-03T16:11:15.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=565] Connection lost 2022-07-03T16:11:15.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=565] Aborting connection 2022-07-03T16:11:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:15 smithi161 ceph-mon[91579]: pgmap v9964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:15 smithi188 ceph-mon[92425]: pgmap v9964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:17 smithi161 ceph-mon[91579]: pgmap v9965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:17 smithi188 ceph-mon[92425]: pgmap v9965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:11:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:19 smithi161 ceph-mon[91579]: pgmap v9966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:19 smithi188 ceph-mon[92425]: pgmap v9966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:21 smithi161 ceph-mon[91579]: pgmap v9967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:21 smithi188 ceph-mon[92425]: pgmap v9967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:23 smithi161 ceph-mon[91579]: pgmap v9968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:23 smithi188 ceph-mon[92425]: pgmap v9968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:25 smithi161 ceph-mon[91579]: pgmap v9969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:25 smithi188 ceph-mon[92425]: pgmap v9969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:27 smithi161 ceph-mon[91579]: pgmap v9970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:27 smithi188 ceph-mon[92425]: pgmap v9970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:29 smithi161 ceph-mon[91579]: pgmap v9971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:29 smithi188 ceph-mon[92425]: pgmap v9971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:31 smithi161 ceph-mon[91579]: pgmap v9972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:31 smithi188 ceph-mon[92425]: pgmap v9972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:33 smithi161 ceph-mon[91579]: pgmap v9973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:33 smithi188 ceph-mon[92425]: pgmap v9973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:35 smithi161 ceph-mon[91579]: pgmap v9974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:35 smithi188 ceph-mon[92425]: pgmap v9974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:37 smithi161 ceph-mon[91579]: pgmap v9975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:37 smithi188 ceph-mon[92425]: pgmap v9975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:39 smithi161 ceph-mon[91579]: pgmap v9976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:39 smithi188 ceph-mon[92425]: pgmap v9976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:41 smithi161 ceph-mon[91579]: pgmap v9977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:41 smithi188 ceph-mon[92425]: pgmap v9977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:43 smithi161 ceph-mon[91579]: pgmap v9978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:43 smithi188 ceph-mon[92425]: pgmap v9978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:45 smithi161 ceph-mon[91579]: pgmap v9979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:45 smithi188 ceph-mon[92425]: pgmap v9979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:47 smithi161 ceph-mon[91579]: pgmap v9980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:47 smithi188 ceph-mon[92425]: pgmap v9980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:49 smithi161 ceph-mon[91579]: pgmap v9981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:49 smithi188 ceph-mon[92425]: pgmap v9981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:51 smithi161 ceph-mon[91579]: pgmap v9982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:51 smithi188 ceph-mon[92425]: pgmap v9982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:53 smithi161 ceph-mon[91579]: pgmap v9983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:53 smithi188 ceph-mon[92425]: pgmap v9983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:55 smithi161 ceph-mon[91579]: pgmap v9984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:55 smithi188 ceph-mon[92425]: pgmap v9984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:57 smithi161 ceph-mon[91579]: pgmap v9985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:11:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:57 smithi188 ceph-mon[92425]: pgmap v9985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:11:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:11:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:11:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:11:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:11:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:11:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:11:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:11:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:11:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:11:59 smithi161 ceph-mon[91579]: pgmap v9986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:11:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:11:59 smithi188 ceph-mon[92425]: pgmap v9986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:01 smithi161 ceph-mon[91579]: pgmap v9987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:01 smithi188 ceph-mon[92425]: pgmap v9987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:03 smithi161 ceph-mon[91579]: pgmap v9988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:03 smithi188 ceph-mon[92425]: pgmap v9988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:05 smithi161 ceph-mon[91579]: pgmap v9989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:05 smithi188 ceph-mon[92425]: pgmap v9989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:07 smithi188 ceph-mon[92425]: pgmap v9990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:07 smithi161 ceph-mon[91579]: pgmap v9990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:09 smithi161 ceph-mon[91579]: pgmap v9991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:09 smithi188 ceph-mon[92425]: pgmap v9991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:11 smithi161 ceph-mon[91579]: pgmap v9992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:11 smithi188 ceph-mon[92425]: pgmap v9992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-mon[91579]: pgmap v9993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:14.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:12:13.940+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:12:14.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:12:14.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:12:14.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:12:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:12:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:12:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:12:14.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:12:14.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:12:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:12:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:12:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:12:14.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:12:14.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:12:14.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:12:14.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:12:14.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:12:14.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:12:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:12:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:12:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:12:14.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:12:14.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:12:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:12:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:12:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:12:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:12:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:12:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:12:14.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:12:14.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:12:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:12:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:12:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:12:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:12:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:12:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:12:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:12:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:12:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:12:14.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:12:14.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:12:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:12:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:12:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:12:14.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=567] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:12:14.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=567] Local address: 172.21.15.161, port 56616 2022-07-03T16:12:14.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=567] Peer address: 172.21.15.101, port 22 2022-07-03T16:12:14.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=567] Connection lost 2022-07-03T16:12:14.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=567] Aborting connection 2022-07-03T16:12:14.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:12:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:12:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:13 smithi188 ceph-mon[92425]: pgmap v9993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:12:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:12:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:12:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:12:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:12:15.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:12:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:12:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:12:15.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:12:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:12:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:12:15.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:12:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:12:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:12:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:12:15.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:12:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:12:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:12:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:12:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:12:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:12:15.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:12:15.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:12:15.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:12:15.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:12:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:12:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:12:15.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:12:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:12:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:12:15.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:12:15.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:12:15.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:12:15.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:12:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:12:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:12:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:12:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:12:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:12:15.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:12:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:12:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:12:15.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:12:15.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:12:15.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:12:15.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:12:15.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:12:15.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:12:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:12:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=567] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:12:15.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=567] Local address: 172.21.15.161, port 56616 2022-07-03T16:12:15.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=567] Peer address: 172.21.15.101, port 22 2022-07-03T16:12:15.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=567] Connection lost 2022-07-03T16:12:15.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=567] Aborting connection 2022-07-03T16:12:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:12:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:12:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:12:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:12:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:12:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:12:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:12:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:12:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:12:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:12:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:12:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:12:15.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:12:15.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:12:15.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:12:15.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:12:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:12:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:12:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:12:15.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:12:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:12:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:12:15.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:12:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:12:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:12:15.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:12:15.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:12:15.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:12:15.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:12:15.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:12:15.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:12:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:12:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:12:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:12:15.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:12:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:12:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:12:15.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:12:15.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:12:15.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:12:15.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:12:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:12:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:12:15.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:12:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:12:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:12:15.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:12:15.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:12:15.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:12:15.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=567] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:12:15.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=567] Local address: 172.21.15.161, port 56616 2022-07-03T16:12:15.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=567] Peer address: 172.21.15.101, port 22 2022-07-03T16:12:15.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=567] Connection lost 2022-07-03T16:12:15.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=567] Aborting connection 2022-07-03T16:12:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:15 smithi161 ceph-mon[91579]: pgmap v9994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:15 smithi188 ceph-mon[92425]: pgmap v9994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:17 smithi161 ceph-mon[91579]: pgmap v9995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:17 smithi188 ceph-mon[92425]: pgmap v9995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:19 smithi161 ceph-mon[91579]: pgmap v9996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:19 smithi188 ceph-mon[92425]: pgmap v9996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:21 smithi161 ceph-mon[91579]: pgmap v9997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:21 smithi188 ceph-mon[92425]: pgmap v9997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:23 smithi161 ceph-mon[91579]: pgmap v9998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:23 smithi188 ceph-mon[92425]: pgmap v9998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:25 smithi161 ceph-mon[91579]: pgmap v9999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:25 smithi188 ceph-mon[92425]: pgmap v9999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:27 smithi188 ceph-mon[92425]: pgmap v10000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:27 smithi161 ceph-mon[91579]: pgmap v10000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:29 smithi161 ceph-mon[91579]: pgmap v10001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:29 smithi188 ceph-mon[92425]: pgmap v10001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:31 smithi161 ceph-mon[91579]: pgmap v10002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:31 smithi188 ceph-mon[92425]: pgmap v10002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:33 smithi161 ceph-mon[91579]: pgmap v10003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:33 smithi188 ceph-mon[92425]: pgmap v10003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:35 smithi161 ceph-mon[91579]: pgmap v10004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:35 smithi188 ceph-mon[92425]: pgmap v10004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:37 smithi188 ceph-mon[92425]: pgmap v10005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:37 smithi161 ceph-mon[91579]: pgmap v10005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:39 smithi161 ceph-mon[91579]: pgmap v10006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:39 smithi188 ceph-mon[92425]: pgmap v10006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:41 smithi161 ceph-mon[91579]: pgmap v10007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:41 smithi188 ceph-mon[92425]: pgmap v10007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:43 smithi161 ceph-mon[91579]: pgmap v10008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:43 smithi188 ceph-mon[92425]: pgmap v10008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:45 smithi161 ceph-mon[91579]: pgmap v10009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:45 smithi188 ceph-mon[92425]: pgmap v10009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:47 smithi188 ceph-mon[92425]: pgmap v10010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:47 smithi161 ceph-mon[91579]: pgmap v10010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:49 smithi161 ceph-mon[91579]: pgmap v10011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:49 smithi188 ceph-mon[92425]: pgmap v10011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:51 smithi161 ceph-mon[91579]: pgmap v10012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:52.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:51 smithi188 ceph-mon[92425]: pgmap v10012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:53 smithi161 ceph-mon[91579]: pgmap v10013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:53 smithi188 ceph-mon[92425]: pgmap v10013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:55 smithi161 ceph-mon[91579]: pgmap v10014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:55 smithi188 ceph-mon[92425]: pgmap v10014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:57 smithi188 ceph-mon[92425]: pgmap v10015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:12:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:57 smithi161 ceph-mon[91579]: pgmap v10015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:12:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:12:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:12:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:12:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:12:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:12:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:12:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:12:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:12:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:12:59 smithi161 ceph-mon[91579]: pgmap v10016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:12:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:12:59 smithi188 ceph-mon[92425]: pgmap v10016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:01 smithi161 ceph-mon[91579]: pgmap v10017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:01 smithi188 ceph-mon[92425]: pgmap v10017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:03 smithi161 ceph-mon[91579]: pgmap v10018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:03 smithi188 ceph-mon[92425]: pgmap v10018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:05 smithi161 ceph-mon[91579]: pgmap v10019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:05 smithi188 ceph-mon[92425]: pgmap v10019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:07 smithi188 ceph-mon[92425]: pgmap v10020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:07 smithi161 ceph-mon[91579]: pgmap v10020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:09 smithi161 ceph-mon[91579]: pgmap v10021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:09 smithi188 ceph-mon[92425]: pgmap v10021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:11 smithi161 ceph-mon[91579]: pgmap v10022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:11 smithi188 ceph-mon[92425]: pgmap v10022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:13 smithi161 ceph-mon[91579]: pgmap v10023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:13 smithi188 ceph-mon[92425]: pgmap v10023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:14.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:13:14.438+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:13:14.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:13:14.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:13:14.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:13:14.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:13:14.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:13:14.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:13:14.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:13:14.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:13:14.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:13:14.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:13:14.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:13:14.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:13:14.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:13:14.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:13:14.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:13:14.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:13:14.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:13:14.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:13:14.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:13:14.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:13:14.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:13:14.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:13:14.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:13:14.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:13:14.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:13:14.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:13:14.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:13:14.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:13:14.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:13:14.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:13:14.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:13:14.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:13:14.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:13:14.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:13:14.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:13:14.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:13:14.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:13:14.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:13:14.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:13:14.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:13:14.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:13:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:13:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:13:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:13:14.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=569] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:13:14.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=569] Local address: 172.21.15.161, port 56620 2022-07-03T16:13:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=569] Peer address: 172.21.15.101, port 22 2022-07-03T16:13:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=569] Connection lost 2022-07-03T16:13:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=569] Aborting connection 2022-07-03T16:13:14.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:13:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:13:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:13:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:13:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:13:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:13:15.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:13:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:13:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:13:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:13:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:13:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:13:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:13:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:13:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:13:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:13:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:13:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:13:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:13:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:13:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:13:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:13:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:13:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:13:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:13:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:13:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:13:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:13:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:13:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:13:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:13:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:13:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:13:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:13:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:13:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:13:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:13:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:13:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:13:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:13:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:13:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:13:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:13:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:13:16.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:13:16.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:13:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:13:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:13:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:13:16.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:13:16.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:13:16.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:13:16.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:13:16.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=569] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:13:16.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=569] Local address: 172.21.15.161, port 56620 2022-07-03T16:13:16.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=569] Peer address: 172.21.15.101, port 22 2022-07-03T16:13:16.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=569] Connection lost 2022-07-03T16:13:16.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=569] Aborting connection 2022-07-03T16:13:16.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:15 smithi161 ceph-mon[91579]: pgmap v10024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:13:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:13:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:13:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:13:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:13:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:13:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:13:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:13:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:13:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:13:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:13:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:13:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:13:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:13:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:13:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:13:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:13:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:13:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:13:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:13:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:13:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:13:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:13:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:13:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:13:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:13:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:13:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:13:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:13:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:13:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:13:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:13:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:13:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:13:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:13:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:13:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:13:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:13:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:13:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:13:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:13:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:13:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:13:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:13:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:13:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:13:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=569] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:13:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=569] Local address: 172.21.15.161, port 56620 2022-07-03T16:13:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=569] Peer address: 172.21.15.101, port 22 2022-07-03T16:13:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=569] Connection lost 2022-07-03T16:13:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=569] Aborting connection 2022-07-03T16:13:16.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:15 smithi188 ceph-mon[92425]: pgmap v10024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:17 smithi188 ceph-mon[92425]: pgmap v10025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:17 smithi161 ceph-mon[91579]: pgmap v10025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:19 smithi161 ceph-mon[91579]: pgmap v10026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:19 smithi188 ceph-mon[92425]: pgmap v10026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:21 smithi161 ceph-mon[91579]: pgmap v10027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:21 smithi188 ceph-mon[92425]: pgmap v10027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:23 smithi161 ceph-mon[91579]: pgmap v10028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:23 smithi188 ceph-mon[92425]: pgmap v10028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:25 smithi161 ceph-mon[91579]: pgmap v10029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:25 smithi188 ceph-mon[92425]: pgmap v10029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:27 smithi188 ceph-mon[92425]: pgmap v10030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:27 smithi161 ceph-mon[91579]: pgmap v10030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:29 smithi161 ceph-mon[91579]: pgmap v10031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:29 smithi188 ceph-mon[92425]: pgmap v10031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:31 smithi161 ceph-mon[91579]: pgmap v10032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:31 smithi188 ceph-mon[92425]: pgmap v10032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:33 smithi161 ceph-mon[91579]: pgmap v10033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:33 smithi188 ceph-mon[92425]: pgmap v10033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:35 smithi161 ceph-mon[91579]: pgmap v10034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:35 smithi188 ceph-mon[92425]: pgmap v10034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:37 smithi188 ceph-mon[92425]: pgmap v10035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:37 smithi161 ceph-mon[91579]: pgmap v10035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:39 smithi161 ceph-mon[91579]: pgmap v10036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:39 smithi188 ceph-mon[92425]: pgmap v10036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:41 smithi161 ceph-mon[91579]: pgmap v10037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:41 smithi188 ceph-mon[92425]: pgmap v10037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:43 smithi161 ceph-mon[91579]: pgmap v10038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:43 smithi188 ceph-mon[92425]: pgmap v10038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:45 smithi161 ceph-mon[91579]: pgmap v10039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:45 smithi188 ceph-mon[92425]: pgmap v10039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:47 smithi188 ceph-mon[92425]: pgmap v10040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:47 smithi161 ceph-mon[91579]: pgmap v10040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:49 smithi161 ceph-mon[91579]: pgmap v10041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:49 smithi188 ceph-mon[92425]: pgmap v10041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:51 smithi161 ceph-mon[91579]: pgmap v10042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:51 smithi188 ceph-mon[92425]: pgmap v10042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:53 smithi161 ceph-mon[91579]: pgmap v10043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:53 smithi188 ceph-mon[92425]: pgmap v10043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:55 smithi161 ceph-mon[91579]: pgmap v10044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:55 smithi188 ceph-mon[92425]: pgmap v10044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:57 smithi188 ceph-mon[92425]: pgmap v10045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:13:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:57 smithi161 ceph-mon[91579]: pgmap v10045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:13:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:13:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:13:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:13:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:13:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:13:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:13:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:13:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:13:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:13:59 smithi161 ceph-mon[91579]: pgmap v10046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:13:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:13:59 smithi188 ceph-mon[92425]: pgmap v10046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:01 smithi161 ceph-mon[91579]: pgmap v10047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:01 smithi188 ceph-mon[92425]: pgmap v10047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:03 smithi161 ceph-mon[91579]: pgmap v10048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:03 smithi188 ceph-mon[92425]: pgmap v10048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:05 smithi161 ceph-mon[91579]: pgmap v10049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:05 smithi188 ceph-mon[92425]: pgmap v10049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:07 smithi188 ceph-mon[92425]: pgmap v10050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:07 smithi161 ceph-mon[91579]: pgmap v10050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:09 smithi161 ceph-mon[91579]: pgmap v10051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:09 smithi188 ceph-mon[92425]: pgmap v10051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:11 smithi161 ceph-mon[91579]: pgmap v10052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:11 smithi188 ceph-mon[92425]: pgmap v10052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:13 smithi188 ceph-mon[92425]: pgmap v10053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:13 smithi161 ceph-mon[91579]: pgmap v10053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:15.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:14:14.939+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:14:15.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:14:15.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:14:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:14:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:14:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:14:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:14:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:14:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:14:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:14:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:14:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:14:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:14:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:14:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:14:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:14:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:14:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:14:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:14:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:14:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:14:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:14:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:14:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:14:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:14:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:14:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:14:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:14:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:14:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:14:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:14:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:14:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:14:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:14:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:14:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:14:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:14:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:14:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:14:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:14:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:14:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:14:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:14:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:14:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:14:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=571] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:14:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=571] Local address: 172.21.15.161, port 56624 2022-07-03T16:14:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=571] Peer address: 172.21.15.101, port 22 2022-07-03T16:14:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=571] Connection lost 2022-07-03T16:14:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=571] Aborting connection 2022-07-03T16:14:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:14:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:14:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: pgmap v10054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:14:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:14:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:14:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:14:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:14:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:14:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:14:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:14:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:14:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:14:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:14:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:14:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:14:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:14:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:14:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:14:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:14:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:14:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:14:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:14:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:14:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:14:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:14:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:14:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:14:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:14:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:14:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:14:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:14:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:14:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:14:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:14:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:14:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:14:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:14:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:14:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:14:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:14:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:14:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:14:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:14:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:14:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:14:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:14:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:14:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:14:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:14:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:14:16.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:14:16.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=571] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:14:16.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=571] Local address: 172.21.15.161, port 56624 2022-07-03T16:14:16.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=571] Peer address: 172.21.15.101, port 22 2022-07-03T16:14:16.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=571] Connection lost 2022-07-03T16:14:16.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=571] Aborting connection 2022-07-03T16:14:16.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: pgmap v10054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:14:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:14:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:14:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:14:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:14:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:14:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:14:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:14:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:14:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:14:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:14:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:14:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:14:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:14:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:14:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:14:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:14:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:14:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:14:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:14:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:14:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:14:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:14:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:14:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:14:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:14:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:14:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:14:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:14:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:14:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:14:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:14:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:14:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:14:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:14:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:14:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:14:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:14:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:14:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:14:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:14:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:14:16.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:14:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:14:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:14:16.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:14:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:14:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:14:16.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:14:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=571] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:14:16.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=571] Local address: 172.21.15.161, port 56624 2022-07-03T16:14:16.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=571] Peer address: 172.21.15.101, port 22 2022-07-03T16:14:16.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=571] Connection lost 2022-07-03T16:14:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=571] Aborting connection 2022-07-03T16:14:16.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:16.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:17.265 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:17.266 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:17 smithi188 ceph-mon[92425]: pgmap v10055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:17 smithi161 ceph-mon[91579]: pgmap v10055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:19 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T16:14:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:19 smithi161 ceph-mon[91579]: pgmap v10056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:19 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T16:14:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:19 smithi188 ceph-mon[92425]: pgmap v10056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:14:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:21 smithi161 ceph-mon[91579]: pgmap v10057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:21 smithi188 ceph-mon[92425]: pgmap v10057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:23 smithi188 ceph-mon[92425]: pgmap v10058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:23 smithi161 ceph-mon[91579]: pgmap v10058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:25 smithi188 ceph-mon[92425]: pgmap v10059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:25 smithi161 ceph-mon[91579]: pgmap v10059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:27 smithi188 ceph-mon[92425]: pgmap v10060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:27 smithi161 ceph-mon[91579]: pgmap v10060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:29 smithi188 ceph-mon[92425]: pgmap v10061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:29 smithi161 ceph-mon[91579]: pgmap v10061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:31 smithi188 ceph-mon[92425]: pgmap v10062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:31 smithi161 ceph-mon[91579]: pgmap v10062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:33 smithi188 ceph-mon[92425]: pgmap v10063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:33 smithi161 ceph-mon[91579]: pgmap v10063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:35 smithi188 ceph-mon[92425]: pgmap v10064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:35 smithi161 ceph-mon[91579]: pgmap v10064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:37 smithi188 ceph-mon[92425]: pgmap v10065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:37 smithi161 ceph-mon[91579]: pgmap v10065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:39 smithi188 ceph-mon[92425]: pgmap v10066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:39 smithi161 ceph-mon[91579]: pgmap v10066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:41 smithi188 ceph-mon[92425]: pgmap v10067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:41 smithi161 ceph-mon[91579]: pgmap v10067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:43 smithi188 ceph-mon[92425]: pgmap v10068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:43 smithi161 ceph-mon[91579]: pgmap v10068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:45 smithi188 ceph-mon[92425]: pgmap v10069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:45 smithi161 ceph-mon[91579]: pgmap v10069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:47 smithi161 ceph-mon[91579]: pgmap v10070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:47 smithi188 ceph-mon[92425]: pgmap v10070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:49 smithi188 ceph-mon[92425]: pgmap v10071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:49 smithi161 ceph-mon[91579]: pgmap v10071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:51 smithi188 ceph-mon[92425]: pgmap v10072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:52.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:51 smithi161 ceph-mon[91579]: pgmap v10072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:53 smithi188 ceph-mon[92425]: pgmap v10073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:53 smithi161 ceph-mon[91579]: pgmap v10073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:55 smithi188 ceph-mon[92425]: pgmap v10074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:55 smithi161 ceph-mon[91579]: pgmap v10074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:57 smithi161 ceph-mon[91579]: pgmap v10075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:14:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:57 smithi188 ceph-mon[92425]: pgmap v10075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:14:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:14:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:14:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:14:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:14:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:14:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:14:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:14:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:14:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:14:59 smithi161 ceph-mon[91579]: pgmap v10076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:14:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:14:59 smithi188 ceph-mon[92425]: pgmap v10076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:01 smithi188 ceph-mon[92425]: pgmap v10077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:01 smithi161 ceph-mon[91579]: pgmap v10077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:03 smithi188 ceph-mon[92425]: pgmap v10078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:03 smithi161 ceph-mon[91579]: pgmap v10078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:05 smithi188 ceph-mon[92425]: pgmap v10079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:05 smithi161 ceph-mon[91579]: pgmap v10079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:07 smithi161 ceph-mon[91579]: pgmap v10080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:07 smithi188 ceph-mon[92425]: pgmap v10080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:09 smithi188 ceph-mon[92425]: pgmap v10081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:09 smithi161 ceph-mon[91579]: pgmap v10081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:11 smithi188 ceph-mon[92425]: pgmap v10082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:11 smithi161 ceph-mon[91579]: pgmap v10082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:13 smithi188 ceph-mon[92425]: pgmap v10083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:13 smithi161 ceph-mon[91579]: pgmap v10083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:15 smithi188 ceph-mon[92425]: pgmap v10084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:15 smithi161 ceph-mon[91579]: pgmap v10084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:17 smithi161 ceph-mon[91579]: pgmap v10085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:17 smithi188 ceph-mon[92425]: pgmap v10085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:15:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:15:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:15:19.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:15:18.857+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:15:19.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:15:19.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:15:19.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:15:19.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:15:19.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:15:19.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:15:19.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:15:19.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:15:19.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:15:19.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:15:19.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:15:19.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:15:19.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:15:19.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:15:19.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:15:19.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:15:19.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:15:19.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:15:19.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:15:19.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:15:19.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:15:19.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:15:19.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:15:19.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:15:19.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:15:19.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:15:19.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:15:19.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:15:19.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:15:19.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:15:19.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:15:19.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:15:19.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:15:19.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:15:19.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:15:19.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:15:19.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:15:19.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:15:19.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:15:19.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:15:19.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:15:19.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:15:19.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:15:19.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:15:19.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=573] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:15:19.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=573] Local address: 172.21.15.161, port 56628 2022-07-03T16:15:19.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=573] Peer address: 172.21.15.101, port 22 2022-07-03T16:15:19.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=573] Connection lost 2022-07-03T16:15:19.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=573] Aborting connection 2022-07-03T16:15:19.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:15:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:15:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:15:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:15:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:15:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: pgmap v10086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:15:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:15:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:15:20.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:15:20.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:15:20.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:15:20.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:15:20.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:15:20.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:15:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:15:20.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:15:20.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:15:20.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:15:20.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:15:20.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:15:20.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:15:20.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:15:20.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:15:20.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:15:20.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:15:20.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:15:20.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:15:20.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:15:20.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:15:20.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:15:20.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:15:20.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:15:20.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:15:20.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:15:20.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:15:20.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:15:20.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:15:20.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:15:20.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:15:20.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:15:20.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:15:20.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:15:20.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:15:20.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:15:20.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:15:20.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:15:20.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:15:20.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:15:20.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:15:20.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=573] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:15:20.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=573] Local address: 172.21.15.161, port 56628 2022-07-03T16:15:20.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=573] Peer address: 172.21.15.101, port 22 2022-07-03T16:15:20.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=573] Connection lost 2022-07-03T16:15:20.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=573] Aborting connection 2022-07-03T16:15:20.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: pgmap v10086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:15:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:15:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:15:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:15:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:15:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:15:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:15:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:15:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:15:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:15:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:15:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:15:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:15:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:15:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:15:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:15:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:15:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:15:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:15:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:15:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:15:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:15:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:15:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:15:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:15:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:15:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:15:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:15:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:15:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:15:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:15:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:15:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:15:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:15:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:15:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:15:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:15:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:15:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:15:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:15:20.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:15:20.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:15:20.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:15:20.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:15:20.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:15:20.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:15:20.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=573] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:15:20.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=573] Local address: 172.21.15.161, port 56628 2022-07-03T16:15:20.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=573] Peer address: 172.21.15.101, port 22 2022-07-03T16:15:20.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=573] Connection lost 2022-07-03T16:15:20.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=573] Aborting connection 2022-07-03T16:15:20.400 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:20.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:15:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:21 smithi188 ceph-mon[92425]: pgmap v10087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:21 smithi161 ceph-mon[91579]: pgmap v10087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:23 smithi188 ceph-mon[92425]: pgmap v10088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:23 smithi161 ceph-mon[91579]: pgmap v10088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:25 smithi188 ceph-mon[92425]: pgmap v10089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:25 smithi161 ceph-mon[91579]: pgmap v10089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:27 smithi161 ceph-mon[91579]: pgmap v10090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:27 smithi188 ceph-mon[92425]: pgmap v10090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:29 smithi188 ceph-mon[92425]: pgmap v10091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:29 smithi161 ceph-mon[91579]: pgmap v10091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:31 smithi188 ceph-mon[92425]: pgmap v10092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:31 smithi161 ceph-mon[91579]: pgmap v10092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:33 smithi188 ceph-mon[92425]: pgmap v10093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:33 smithi161 ceph-mon[91579]: pgmap v10093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:35 smithi188 ceph-mon[92425]: pgmap v10094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:35 smithi161 ceph-mon[91579]: pgmap v10094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:37 smithi161 ceph-mon[91579]: pgmap v10095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:37 smithi188 ceph-mon[92425]: pgmap v10095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:39 smithi188 ceph-mon[92425]: pgmap v10096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:39 smithi161 ceph-mon[91579]: pgmap v10096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:41 smithi188 ceph-mon[92425]: pgmap v10097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:41 smithi161 ceph-mon[91579]: pgmap v10097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:43 smithi188 ceph-mon[92425]: pgmap v10098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:43 smithi161 ceph-mon[91579]: pgmap v10098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:45 smithi188 ceph-mon[92425]: pgmap v10099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:45 smithi161 ceph-mon[91579]: pgmap v10099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:47 smithi161 ceph-mon[91579]: pgmap v10100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:47 smithi188 ceph-mon[92425]: pgmap v10100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:49 smithi188 ceph-mon[92425]: pgmap v10101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:49 smithi161 ceph-mon[91579]: pgmap v10101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:51 smithi188 ceph-mon[92425]: pgmap v10102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:51 smithi161 ceph-mon[91579]: pgmap v10102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:53 smithi188 ceph-mon[92425]: pgmap v10103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:53 smithi161 ceph-mon[91579]: pgmap v10103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:55 smithi188 ceph-mon[92425]: pgmap v10104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:55 smithi161 ceph-mon[91579]: pgmap v10104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:57 smithi161 ceph-mon[91579]: pgmap v10105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:15:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:57 smithi188 ceph-mon[92425]: pgmap v10105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:15:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:15:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:15:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:15:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:15:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:15:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:15:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:15:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:15:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:15:59 smithi161 ceph-mon[91579]: pgmap v10106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:15:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:15:59 smithi188 ceph-mon[92425]: pgmap v10106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:01 smithi188 ceph-mon[92425]: pgmap v10107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:01 smithi161 ceph-mon[91579]: pgmap v10107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:03 smithi188 ceph-mon[92425]: pgmap v10108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:03 smithi161 ceph-mon[91579]: pgmap v10108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:05 smithi161 ceph-mon[91579]: pgmap v10109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:05 smithi188 ceph-mon[92425]: pgmap v10109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:07 smithi161 ceph-mon[91579]: pgmap v10110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:07 smithi188 ceph-mon[92425]: pgmap v10110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:09 smithi188 ceph-mon[92425]: pgmap v10111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:09 smithi161 ceph-mon[91579]: pgmap v10111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:11 smithi161 ceph-mon[91579]: pgmap v10112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:11 smithi188 ceph-mon[92425]: pgmap v10112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:13 smithi161 ceph-mon[91579]: pgmap v10113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:13 smithi188 ceph-mon[92425]: pgmap v10113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:15 smithi161 ceph-mon[91579]: pgmap v10114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:15 smithi188 ceph-mon[92425]: pgmap v10114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:17 smithi161 ceph-mon[91579]: pgmap v10115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:17 smithi188 ceph-mon[92425]: pgmap v10115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-mon[91579]: pgmap v10116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:16:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:16:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:16:20.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:16:19.779+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:16:20.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:16:20.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:16:20.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:16:20.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:16:20.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:16:20.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:16:20.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:16:20.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:16:20.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:16:20.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:16:20.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:16:20.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:16:20.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:16:20.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:16:20.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:16:20.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:16:20.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:16:20.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:16:20.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:16:20.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:16:20.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:16:20.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:16:20.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:16:20.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:16:20.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:16:20.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:16:20.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:16:20.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:16:20.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:16:20.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:16:20.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:16:20.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:16:20.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:16:20.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:16:20.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:16:20.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:16:20.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:16:20.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:16:20.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:16:20.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:16:20.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:16:20.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:16:20.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:16:20.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:16:20.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=575] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:16:20.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=575] Local address: 172.21.15.161, port 56632 2022-07-03T16:16:20.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=575] Peer address: 172.21.15.101, port 22 2022-07-03T16:16:20.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=575] Connection lost 2022-07-03T16:16:20.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=575] Aborting connection 2022-07-03T16:16:20.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:16:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:16:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:19 smithi188 ceph-mon[92425]: pgmap v10116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:16:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:16:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:16:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:16:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:16:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:16:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:16:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:16:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:16:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:16:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:16:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:16:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:16:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:16:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:16:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:16:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:16:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:16:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:16:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:16:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:16:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:16:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:16:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:16:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:16:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:16:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:16:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:16:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:16:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:16:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:16:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:16:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:16:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:16:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:16:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:16:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:16:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:16:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:16:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:16:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:16:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:16:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:16:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:16:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:16:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:16:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:16:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:16:21.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:16:21.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=575] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:16:21.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=575] Local address: 172.21.15.161, port 56632 2022-07-03T16:16:21.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=575] Peer address: 172.21.15.101, port 22 2022-07-03T16:16:21.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=575] Connection lost 2022-07-03T16:16:21.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=575] Aborting connection 2022-07-03T16:16:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:16:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:16:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:16:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:16:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:16:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:16:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:16:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:16:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:16:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:16:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:16:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:16:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:16:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:16:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:16:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:16:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:16:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:16:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:16:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:16:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:16:21.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:16:21.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:16:21.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:16:21.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:16:21.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:16:21.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:16:21.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:16:21.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:16:21.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:16:21.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:16:21.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:16:21.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:16:21.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:16:21.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:16:21.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:16:21.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:16:21.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:16:21.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:16:21.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:16:21.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:16:21.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:16:21.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:16:21.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:16:21.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:16:21.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:16:21.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=575] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:16:21.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=575] Local address: 172.21.15.161, port 56632 2022-07-03T16:16:21.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=575] Peer address: 172.21.15.101, port 22 2022-07-03T16:16:21.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=575] Connection lost 2022-07-03T16:16:21.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=575] Aborting connection 2022-07-03T16:16:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:21 smithi161 ceph-mon[91579]: pgmap v10117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:21 smithi188 ceph-mon[92425]: pgmap v10117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:23 smithi161 ceph-mon[91579]: pgmap v10118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:23 smithi188 ceph-mon[92425]: pgmap v10118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:25 smithi161 ceph-mon[91579]: pgmap v10119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:25 smithi188 ceph-mon[92425]: pgmap v10119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:27 smithi161 ceph-mon[91579]: pgmap v10120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:27 smithi188 ceph-mon[92425]: pgmap v10120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:29 smithi161 ceph-mon[91579]: pgmap v10121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:29 smithi188 ceph-mon[92425]: pgmap v10121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:31 smithi161 ceph-mon[91579]: pgmap v10122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:31 smithi188 ceph-mon[92425]: pgmap v10122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:33 smithi161 ceph-mon[91579]: pgmap v10123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:33 smithi188 ceph-mon[92425]: pgmap v10123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:35 smithi161 ceph-mon[91579]: pgmap v10124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:35 smithi188 ceph-mon[92425]: pgmap v10124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:37 smithi161 ceph-mon[91579]: pgmap v10125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:37 smithi188 ceph-mon[92425]: pgmap v10125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:39 smithi161 ceph-mon[91579]: pgmap v10126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:39 smithi188 ceph-mon[92425]: pgmap v10126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:41 smithi161 ceph-mon[91579]: pgmap v10127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:41 smithi188 ceph-mon[92425]: pgmap v10127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:43 smithi161 ceph-mon[91579]: pgmap v10128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:43 smithi188 ceph-mon[92425]: pgmap v10128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:45 smithi161 ceph-mon[91579]: pgmap v10129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:45 smithi188 ceph-mon[92425]: pgmap v10129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:47 smithi161 ceph-mon[91579]: pgmap v10130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:47 smithi188 ceph-mon[92425]: pgmap v10130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:49 smithi161 ceph-mon[91579]: pgmap v10131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:49 smithi188 ceph-mon[92425]: pgmap v10131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:51 smithi161 ceph-mon[91579]: pgmap v10132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:51 smithi188 ceph-mon[92425]: pgmap v10132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:53 smithi161 ceph-mon[91579]: pgmap v10133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:53 smithi188 ceph-mon[92425]: pgmap v10133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:55 smithi161 ceph-mon[91579]: pgmap v10134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:55 smithi188 ceph-mon[92425]: pgmap v10134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:57 smithi161 ceph-mon[91579]: pgmap v10135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:16:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:57 smithi188 ceph-mon[92425]: pgmap v10135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:16:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:16:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:16:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:16:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:16:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:16:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:16:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:16:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:16:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:16:59 smithi161 ceph-mon[91579]: pgmap v10136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:16:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:16:59 smithi188 ceph-mon[92425]: pgmap v10136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:01 smithi161 ceph-mon[91579]: pgmap v10137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:01 smithi188 ceph-mon[92425]: pgmap v10137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:03 smithi161 ceph-mon[91579]: pgmap v10138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:03 smithi188 ceph-mon[92425]: pgmap v10138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:05 smithi161 ceph-mon[91579]: pgmap v10139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:05 smithi188 ceph-mon[92425]: pgmap v10139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:07 smithi161 ceph-mon[91579]: pgmap v10140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:07 smithi188 ceph-mon[92425]: pgmap v10140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:09 smithi161 ceph-mon[91579]: pgmap v10141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:09 smithi188 ceph-mon[92425]: pgmap v10141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:11 smithi161 ceph-mon[91579]: pgmap v10142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:11 smithi188 ceph-mon[92425]: pgmap v10142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:13 smithi161 ceph-mon[91579]: pgmap v10143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:13 smithi188 ceph-mon[92425]: pgmap v10143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:15 smithi161 ceph-mon[91579]: pgmap v10144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:15 smithi188 ceph-mon[92425]: pgmap v10144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:17 smithi161 ceph-mon[91579]: pgmap v10145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:17 smithi188 ceph-mon[92425]: pgmap v10145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:20.288 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:19 smithi161 ceph-mon[91579]: pgmap v10146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:20.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:19 smithi188 ceph-mon[92425]: pgmap v10146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:20.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:17:20.285+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:17:20.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:17:20.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:17:20.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:17:20.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:17:20.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:17:20.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:17:20.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:17:20.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:17:20.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:17:20.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:17:20.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:17:20.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:17:20.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:17:20.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:17:20.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:17:20.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:17:20.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:17:20.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:17:20.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:17:20.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:17:20.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:17:20.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:17:20.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:17:20.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:17:20.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:17:20.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:17:20.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:17:20.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:17:20.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:17:20.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:17:20.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:17:20.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:17:20.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:17:20.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:17:20.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:17:20.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:17:20.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:17:20.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:17:20.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:17:20.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:17:20.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:17:20.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:17:20.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:17:20.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:17:20.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=577] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:17:20.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=577] Local address: 172.21.15.161, port 56636 2022-07-03T16:17:20.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=577] Peer address: 172.21.15.101, port 22 2022-07-03T16:17:20.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=577] Connection lost 2022-07-03T16:17:20.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=577] Aborting connection 2022-07-03T16:17:20.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:17:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:17:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:17:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:17:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:17:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:17:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:17:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:17:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:17:22.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:17:22.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:17:22.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:17:22.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:17:22.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:17:22.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:17:22.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:17:22.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:17:22.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:17:22.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:17:22.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:17:22.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:17:22.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:17:22.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:17:22.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:17:22.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:17:22.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:17:22.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:17:22.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:17:22.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:17:22.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:17:22.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:17:22.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:17:22.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:17:22.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:17:22.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:17:22.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:17:22.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:17:22.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:17:22.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:17:22.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:17:22.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:17:22.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:17:22.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:17:22.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:17:22.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:17:22.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:17:22.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:17:22.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:17:22.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:17:22.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:17:22.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:17:22.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:17:22.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:17:22.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=577] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:17:22.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=577] Local address: 172.21.15.161, port 56636 2022-07-03T16:17:22.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=577] Peer address: 172.21.15.101, port 22 2022-07-03T16:17:22.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=577] Connection lost 2022-07-03T16:17:22.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=577] Aborting connection 2022-07-03T16:17:22.414 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:21 smithi161 ceph-mon[91579]: pgmap v10147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:22.415 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:17:22.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:17:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:17:22.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:17:22.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:17:22.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:17:22.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:17:22.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:17:22.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:17:22.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:17:22.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:17:22.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:17:22.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:17:22.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:17:22.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:17:22.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:17:22.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:17:22.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:17:22.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:17:22.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:17:22.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:17:22.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:17:22.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:17:22.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:17:22.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:17:22.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:17:22.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:17:22.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:17:22.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:17:22.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:17:22.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:17:22.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:17:22.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:17:22.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:17:22.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:17:22.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:17:22.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:17:22.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:17:22.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:17:22.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:17:22.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:17:22.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:17:22.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:17:22.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:17:22.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:17:22.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:17:22.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=577] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:17:22.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=577] Local address: 172.21.15.161, port 56636 2022-07-03T16:17:22.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=577] Peer address: 172.21.15.101, port 22 2022-07-03T16:17:22.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=577] Connection lost 2022-07-03T16:17:22.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=577] Aborting connection 2022-07-03T16:17:22.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:21 smithi188 ceph-mon[92425]: pgmap v10147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:22.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: pgmap v10148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: pgmap v10148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:17:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:24.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:17:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:25 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 127.8M 2022-07-03T16:17:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:25 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 134099509: error parsing value: Value '134099509' is below minimum 939524096 2022-07-03T16:17:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:25 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:17:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:25 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:17:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:25 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 127.8M 2022-07-03T16:17:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:25 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 134099509: error parsing value: Value '134099509' is below minimum 939524096 2022-07-03T16:17:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:25 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:17:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:25 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:17:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:26 smithi161 ceph-mon[91579]: pgmap v10149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:26 smithi188 ceph-mon[92425]: pgmap v10149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:27 smithi161 ceph-mon[91579]: pgmap v10150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:27 smithi188 ceph-mon[92425]: pgmap v10150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:29 smithi161 ceph-mon[91579]: pgmap v10151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:29 smithi188 ceph-mon[92425]: pgmap v10151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:31 smithi161 ceph-mon[91579]: pgmap v10152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:31 smithi188 ceph-mon[92425]: pgmap v10152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:33 smithi161 ceph-mon[91579]: pgmap v10153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:33 smithi188 ceph-mon[92425]: pgmap v10153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:35 smithi161 ceph-mon[91579]: pgmap v10154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:35 smithi188 ceph-mon[92425]: pgmap v10154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:37 smithi161 ceph-mon[91579]: pgmap v10155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:37 smithi188 ceph-mon[92425]: pgmap v10155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:39 smithi161 ceph-mon[91579]: pgmap v10156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:39 smithi188 ceph-mon[92425]: pgmap v10156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:41 smithi161 ceph-mon[91579]: pgmap v10157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:41 smithi188 ceph-mon[92425]: pgmap v10157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:43 smithi161 ceph-mon[91579]: pgmap v10158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:43 smithi188 ceph-mon[92425]: pgmap v10158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:45 smithi161 ceph-mon[91579]: pgmap v10159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:45 smithi188 ceph-mon[92425]: pgmap v10159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:47 smithi161 ceph-mon[91579]: pgmap v10160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:47 smithi188 ceph-mon[92425]: pgmap v10160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:49 smithi161 ceph-mon[91579]: pgmap v10161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:49 smithi188 ceph-mon[92425]: pgmap v10161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:51 smithi161 ceph-mon[91579]: pgmap v10162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:51 smithi188 ceph-mon[92425]: pgmap v10162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:53 smithi161 ceph-mon[91579]: pgmap v10163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:53 smithi188 ceph-mon[92425]: pgmap v10163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:55 smithi161 ceph-mon[91579]: pgmap v10164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:55 smithi188 ceph-mon[92425]: pgmap v10164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:58.125 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:57 smithi161 ceph-mon[91579]: pgmap v10165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:58.126 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:17:58.127 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:57 smithi188 ceph-mon[92425]: pgmap v10165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:17:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:17:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:17:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:17:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:17:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:17:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:17:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:17:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:17:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:17:59 smithi161 ceph-mon[91579]: pgmap v10166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:17:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:17:59 smithi188 ceph-mon[92425]: pgmap v10166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:01 smithi161 ceph-mon[91579]: pgmap v10167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:01 smithi188 ceph-mon[92425]: pgmap v10167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:03 smithi161 ceph-mon[91579]: pgmap v10168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:03 smithi188 ceph-mon[92425]: pgmap v10168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:05 smithi161 ceph-mon[91579]: pgmap v10169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:05 smithi188 ceph-mon[92425]: pgmap v10169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:07 smithi161 ceph-mon[91579]: pgmap v10170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:07 smithi188 ceph-mon[92425]: pgmap v10170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:09 smithi161 ceph-mon[91579]: pgmap v10171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:09 smithi188 ceph-mon[92425]: pgmap v10171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:11 smithi161 ceph-mon[91579]: pgmap v10172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:11 smithi188 ceph-mon[92425]: pgmap v10172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:13 smithi161 ceph-mon[91579]: pgmap v10173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:13 smithi188 ceph-mon[92425]: pgmap v10173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:15 smithi161 ceph-mon[91579]: pgmap v10174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:15 smithi188 ceph-mon[92425]: pgmap v10174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:17 smithi161 ceph-mon[91579]: pgmap v10175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:17 smithi188 ceph-mon[92425]: pgmap v10175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:19 smithi161 ceph-mon[91579]: pgmap v10176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:19 smithi188 ceph-mon[92425]: pgmap v10176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:21 smithi161 ceph-mon[91579]: pgmap v10177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:21 smithi188 ceph-mon[92425]: pgmap v10177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:23.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:18:23.530+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:18:23.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:18:23.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:18:23.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:18:23.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:18:23.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:18:23.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:18:23.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:18:23.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:18:23.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:18:23.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:18:23.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:18:23.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:18:23.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:18:23.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:18:23.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:18:23.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:18:23.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:18:23.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:18:23.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:18:23.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:18:23.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:18:23.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:18:23.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:18:23.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:18:23.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:18:23.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:18:23.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:18:23.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:18:23.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:18:23.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:18:23.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:18:23.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:18:23.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:18:23.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:18:23.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:18:23.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:18:23.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:18:23.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:18:23.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:18:23.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:18:23.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:18:23.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:18:23.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:18:23.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:18:23.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=579] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:18:23.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=579] Local address: 172.21.15.161, port 56640 2022-07-03T16:18:23.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=579] Peer address: 172.21.15.101, port 22 2022-07-03T16:18:23.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=579] Connection lost 2022-07-03T16:18:23.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=579] Aborting connection 2022-07-03T16:18:23.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:18:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-mon[91579]: pgmap v10178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:18:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:18:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:18:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:23 smithi188 ceph-mon[92425]: pgmap v10178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:18:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:18:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:18:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:18:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:18:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:18:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:18:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:18:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:18:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:18:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:18:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:18:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:18:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:18:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:18:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:18:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:18:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:18:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:18:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:18:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:18:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:18:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:18:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:18:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:18:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:18:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:18:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:18:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:18:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:18:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:18:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:18:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:18:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:18:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:18:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:18:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:18:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:18:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:18:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:18:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:18:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:18:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:18:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:18:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:18:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:18:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:18:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:18:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:18:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=579] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:18:25.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=579] Local address: 172.21.15.161, port 56640 2022-07-03T16:18:25.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=579] Peer address: 172.21.15.101, port 22 2022-07-03T16:18:25.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=579] Connection lost 2022-07-03T16:18:25.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=579] Aborting connection 2022-07-03T16:18:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:18:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:18:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:18:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:18:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:18:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:18:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:18:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:18:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:18:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:18:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:18:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:18:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:18:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:18:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:18:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:18:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:18:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:18:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:18:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:18:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:18:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:18:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:18:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:18:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:18:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:18:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:18:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:18:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:18:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:18:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:18:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:18:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:18:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:18:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:18:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:18:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:18:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:18:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:18:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:18:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:18:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:18:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:18:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:18:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:18:25.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:18:25.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=579] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:18:25.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=579] Local address: 172.21.15.161, port 56640 2022-07-03T16:18:25.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=579] Peer address: 172.21.15.101, port 22 2022-07-03T16:18:25.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=579] Connection lost 2022-07-03T16:18:25.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=579] Aborting connection 2022-07-03T16:18:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:25 smithi161 ceph-mon[91579]: pgmap v10179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:25 smithi188 ceph-mon[92425]: pgmap v10179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:27 smithi161 ceph-mon[91579]: pgmap v10180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:27 smithi188 ceph-mon[92425]: pgmap v10180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:29 smithi161 ceph-mon[91579]: pgmap v10181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:29 smithi188 ceph-mon[92425]: pgmap v10181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:31 smithi161 ceph-mon[91579]: pgmap v10182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:31 smithi188 ceph-mon[92425]: pgmap v10182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:33 smithi161 ceph-mon[91579]: pgmap v10183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:33 smithi188 ceph-mon[92425]: pgmap v10183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:35 smithi161 ceph-mon[91579]: pgmap v10184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:35 smithi188 ceph-mon[92425]: pgmap v10184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:37.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:37.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:37 smithi161 ceph-mon[91579]: pgmap v10185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:37 smithi188 ceph-mon[92425]: pgmap v10185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:39 smithi161 ceph-mon[91579]: pgmap v10186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:39 smithi188 ceph-mon[92425]: pgmap v10186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:41 smithi161 ceph-mon[91579]: pgmap v10187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:41 smithi188 ceph-mon[92425]: pgmap v10187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:43 smithi161 ceph-mon[91579]: pgmap v10188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:43 smithi188 ceph-mon[92425]: pgmap v10188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:45 smithi161 ceph-mon[91579]: pgmap v10189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:45 smithi188 ceph-mon[92425]: pgmap v10189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:47 smithi161 ceph-mon[91579]: pgmap v10190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:47 smithi188 ceph-mon[92425]: pgmap v10190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:49 smithi161 ceph-mon[91579]: pgmap v10191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:49 smithi188 ceph-mon[92425]: pgmap v10191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:51 smithi161 ceph-mon[91579]: pgmap v10192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:51 smithi188 ceph-mon[92425]: pgmap v10192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:53 smithi161 ceph-mon[91579]: pgmap v10193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:53 smithi188 ceph-mon[92425]: pgmap v10193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:18:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:55 smithi161 ceph-mon[91579]: pgmap v10194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:55 smithi188 ceph-mon[92425]: pgmap v10194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:18:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:18:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:57 smithi161 ceph-mon[91579]: pgmap v10195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:57 smithi188 ceph-mon[92425]: pgmap v10195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:18:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:18:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:18:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:18:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:18:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:18:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:18:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:18:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:19:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:00 smithi161 ceph-mon[91579]: pgmap v10196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:00 smithi188 ceph-mon[92425]: pgmap v10196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:02 smithi161 ceph-mon[91579]: pgmap v10197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:02 smithi188 ceph-mon[92425]: pgmap v10197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:04 smithi161 ceph-mon[91579]: pgmap v10198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:04 smithi188 ceph-mon[92425]: pgmap v10198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:06 smithi161 ceph-mon[91579]: pgmap v10199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:06 smithi188 ceph-mon[92425]: pgmap v10199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:08 smithi161 ceph-mon[91579]: pgmap v10200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:08 smithi188 ceph-mon[92425]: pgmap v10200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:10 smithi161 ceph-mon[91579]: pgmap v10201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:10 smithi188 ceph-mon[92425]: pgmap v10201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:12 smithi161 ceph-mon[91579]: pgmap v10202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:12 smithi188 ceph-mon[92425]: pgmap v10202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:14 smithi161 ceph-mon[91579]: pgmap v10203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:14 smithi188 ceph-mon[92425]: pgmap v10203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:16 smithi161 ceph-mon[91579]: pgmap v10204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:16 smithi188 ceph-mon[92425]: pgmap v10204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:18 smithi161 ceph-mon[91579]: pgmap v10205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:18 smithi188 ceph-mon[92425]: pgmap v10205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:20 smithi161 ceph-mon[91579]: pgmap v10206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:20 smithi188 ceph-mon[92425]: pgmap v10206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:22 smithi161 ceph-mon[91579]: pgmap v10207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:22 smithi188 ceph-mon[92425]: pgmap v10207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:24 smithi161 ceph-mon[91579]: pgmap v10208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:19:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:19:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:19:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:24 smithi188 ceph-mon[92425]: pgmap v10208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:19:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:19:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:19:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:19:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:19:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:26 smithi161 ceph-mon[91579]: pgmap v10209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:26 smithi188 ceph-mon[92425]: pgmap v10209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:28 smithi161 ceph-mon[91579]: pgmap v10210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:28 smithi188 ceph-mon[92425]: pgmap v10210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:30 smithi161 ceph-mon[91579]: pgmap v10211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:30 smithi188 ceph-mon[92425]: pgmap v10211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:32 smithi161 ceph-mon[91579]: pgmap v10212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:32 smithi188 ceph-mon[92425]: pgmap v10212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:34 smithi161 ceph-mon[91579]: pgmap v10213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:34 smithi188 ceph-mon[92425]: pgmap v10213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:35 smithi161 ceph-mon[91579]: pgmap v10214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:35 smithi188 ceph-mon[92425]: pgmap v10214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:37 smithi161 ceph-mon[91579]: pgmap v10215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:37 smithi188 ceph-mon[92425]: pgmap v10215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:39 smithi161 ceph-mon[91579]: pgmap v10216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:39 smithi188 ceph-mon[92425]: pgmap v10216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:41 smithi161 ceph-mon[91579]: pgmap v10217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:41 smithi188 ceph-mon[92425]: pgmap v10217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:43 smithi161 ceph-mon[91579]: pgmap v10218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:43 smithi188 ceph-mon[92425]: pgmap v10218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:45 smithi161 ceph-mon[91579]: pgmap v10219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:45 smithi188 ceph-mon[92425]: pgmap v10219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:47 smithi161 ceph-mon[91579]: pgmap v10220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:47 smithi188 ceph-mon[92425]: pgmap v10220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:49 smithi161 ceph-mon[91579]: pgmap v10221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:49 smithi188 ceph-mon[92425]: pgmap v10221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:51 smithi161 ceph-mon[91579]: pgmap v10222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:51 smithi188 ceph-mon[92425]: pgmap v10222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:52.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:54.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-mon[91579]: pgmap v10223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:54.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:19:54.030+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:19:54.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:19:54.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:19:54.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:19:54.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:19:54.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:19:54.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:19:54.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:19:54.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:19:54.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:19:54.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:19:54.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:19:54.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T16:19:54.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:19:54.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:19:54.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:19:54.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:19:54.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:19:54.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:19:54.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:19:54.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:19:54.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:19:54.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:19:54.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:19:54.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:19:54.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:19:54.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:19:54.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:19:54.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:19:54.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:19:54.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:19:54.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:19:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:19:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:19:54.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:19:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:19:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:19:54.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:19:54.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:19:54.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:19:54.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:19:54.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:19:54.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:19:54.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T16:19:54.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:19:54.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=581] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:19:54.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=581] Local address: 172.21.15.161, port 56666 2022-07-03T16:19:54.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=581] Peer address: 172.21.15.101, port 22 2022-07-03T16:19:54.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1560] Received exit status 0 2022-07-03T16:19:54.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1560] Received channel close 2022-07-03T16:19:54.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1560] Channel closed 2022-07-03T16:19:54.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1560] Received exit status 0 2022-07-03T16:19:54.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1560] Received channel close 2022-07-03T16:19:54.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1560] Channel closed 2022-07-03T16:19:54.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1561] Requesting new SSH session 2022-07-03T16:19:54.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1561] Requesting new SSH session 2022-07-03T16:19:54.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1561] Command: which python3 2022-07-03T16:19:54.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1561] Command: which python3 2022-07-03T16:19:54.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1561] Received exit status 0 2022-07-03T16:19:54.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1561] Received channel close 2022-07-03T16:19:54.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1561] Channel closed 2022-07-03T16:19:54.417 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1561] Received exit status 0 2022-07-03T16:19:54.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1561] Received channel close 2022-07-03T16:19:54.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1562] Requesting new SSH session 2022-07-03T16:19:54.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1561] Channel closed 2022-07-03T16:19:54.419 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1562] Requesting new SSH session 2022-07-03T16:19:54.419 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1562] Command: sudo true 2022-07-03T16:19:54.420 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1562] Command: sudo true 2022-07-03T16:19:54.421 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1562] Received exit status 0 2022-07-03T16:19:54.421 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1562] Received channel close 2022-07-03T16:19:54.422 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1562] Channel closed 2022-07-03T16:19:54.423 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1563] Requesting new SSH session 2022-07-03T16:19:54.424 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1562] Received exit status 0 2022-07-03T16:19:54.424 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1562] Received channel close 2022-07-03T16:19:54.425 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:54.425 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1562] Channel closed 2022-07-03T16:19:54.426 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1563] Requesting new SSH session 2022-07-03T16:19:54.426 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:54.426 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1563] Received exit status 0 2022-07-03T16:19:54.427 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1563] Received channel close 2022-07-03T16:19:54.427 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1, chan=1563] Channel closed 2022-07-03T16:19:54.428 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1563] Received exit status 0 2022-07-03T16:19:54.428 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1563] Received channel close 2022-07-03T16:19:54.429 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=2, chan=1563] Channel closed 2022-07-03T16:19:54.429 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=581] Login timeout expired 2022-07-03T16:19:54.430 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=581] Aborting connection 2022-07-03T16:19:54.430 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:19:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:19:54.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:54 smithi188 ceph-mon[92425]: pgmap v10223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:19:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:19:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:19:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:19:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:19:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:19:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:19:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:19:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:19:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:19:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:19:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T16:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:19:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:19:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:19:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:19:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:19:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:19:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:19:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:19:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:19:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:19:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:19:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:19:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:19:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:19:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:19:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:19:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:19:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:19:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:19:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:19:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:19:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:19:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:19:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:19:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:19:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:19:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:19:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T16:19:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:19:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=581] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:19:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=581] Local address: 172.21.15.161, port 56666 2022-07-03T16:19:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=581] Peer address: 172.21.15.101, port 22 2022-07-03T16:19:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1560] Received exit status 0 2022-07-03T16:19:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1560] Received channel close 2022-07-03T16:19:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1560] Channel closed 2022-07-03T16:19:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1560] Received exit status 0 2022-07-03T16:19:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1560] Received channel close 2022-07-03T16:19:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1560] Channel closed 2022-07-03T16:19:55.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1561] Requesting new SSH session 2022-07-03T16:19:55.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1561] Requesting new SSH session 2022-07-03T16:19:55.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1561] Command: which python3 2022-07-03T16:19:55.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1561] Command: which python3 2022-07-03T16:19:55.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1561] Received exit status 0 2022-07-03T16:19:55.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1561] Received channel close 2022-07-03T16:19:55.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1561] Channel closed 2022-07-03T16:19:55.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1561] Received exit status 0 2022-07-03T16:19:55.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1561] Received channel close 2022-07-03T16:19:55.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1562] Requesting new SSH session 2022-07-03T16:19:55.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1561] Channel closed 2022-07-03T16:19:55.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1562] Requesting new SSH session 2022-07-03T16:19:55.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1562] Command: sudo true 2022-07-03T16:19:55.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1562] Command: sudo true 2022-07-03T16:19:55.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1562] Received exit status 0 2022-07-03T16:19:55.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1562] Received channel close 2022-07-03T16:19:55.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1562] Channel closed 2022-07-03T16:19:55.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1563] Requesting new SSH session 2022-07-03T16:19:55.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1562] Received exit status 0 2022-07-03T16:19:55.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1562] Received channel close 2022-07-03T16:19:55.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:55.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1562] Channel closed 2022-07-03T16:19:55.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1563] Requesting new SSH session 2022-07-03T16:19:55.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:55.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1563] Received exit status 0 2022-07-03T16:19:55.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1563] Received channel close 2022-07-03T16:19:55.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1, chan=1563] Channel closed 2022-07-03T16:19:55.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1563] Received exit status 0 2022-07-03T16:19:55.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1563] Received channel close 2022-07-03T16:19:55.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=2, chan=1563] Channel closed 2022-07-03T16:19:55.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=581] Login timeout expired 2022-07-03T16:19:55.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=581] Aborting connection 2022-07-03T16:19:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:19:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:19:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:19:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:19:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:19:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:19:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:19:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:19:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:19:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:19:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:19:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:19:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Login timeout expired 2022-07-03T16:19:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:19:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:19:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:19:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:19:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:19:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:19:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:19:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:19:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:19:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:19:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:19:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:19:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:19:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:19:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:19:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:19:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:19:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:19:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:19:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:19:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:19:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:19:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:19:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:19:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:19:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:19:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:19:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:19:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:19:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:19:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Login timeout expired 2022-07-03T16:19:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:19:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=581] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:19:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=581] Local address: 172.21.15.161, port 56666 2022-07-03T16:19:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=581] Peer address: 172.21.15.101, port 22 2022-07-03T16:19:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1560] Received exit status 0 2022-07-03T16:19:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1560] Received channel close 2022-07-03T16:19:55.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1560] Channel closed 2022-07-03T16:19:55.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1560] Received exit status 0 2022-07-03T16:19:55.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1560] Received channel close 2022-07-03T16:19:55.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1560] Channel closed 2022-07-03T16:19:55.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1561] Requesting new SSH session 2022-07-03T16:19:55.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1561] Requesting new SSH session 2022-07-03T16:19:55.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1561] Command: which python3 2022-07-03T16:19:55.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1561] Command: which python3 2022-07-03T16:19:55.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1561] Received exit status 0 2022-07-03T16:19:55.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1561] Received channel close 2022-07-03T16:19:55.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1561] Channel closed 2022-07-03T16:19:55.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1561] Received exit status 0 2022-07-03T16:19:55.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1561] Received channel close 2022-07-03T16:19:55.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1562] Requesting new SSH session 2022-07-03T16:19:55.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1561] Channel closed 2022-07-03T16:19:55.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1562] Requesting new SSH session 2022-07-03T16:19:55.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1562] Command: sudo true 2022-07-03T16:19:55.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1562] Command: sudo true 2022-07-03T16:19:55.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1562] Received exit status 0 2022-07-03T16:19:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1562] Received channel close 2022-07-03T16:19:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1562] Channel closed 2022-07-03T16:19:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1563] Requesting new SSH session 2022-07-03T16:19:55.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1562] Received exit status 0 2022-07-03T16:19:55.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1562] Received channel close 2022-07-03T16:19:55.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:55.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1562] Channel closed 2022-07-03T16:19:55.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1563] Requesting new SSH session 2022-07-03T16:19:55.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1563] Command: /usr/bin/python3 /var/lib/ceph/23a851ae-fab9-11ec-842c-001a4aab830c/cephadm.3408b27e6d88c3e133a285cc572aced721a581cd857bb22ca58ab7c8207bfd31 gather-facts 2022-07-03T16:19:55.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1563] Received exit status 0 2022-07-03T16:19:55.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1563] Received channel close 2022-07-03T16:19:55.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1, chan=1563] Channel closed 2022-07-03T16:19:55.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1563] Received exit status 0 2022-07-03T16:19:55.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1563] Received channel close 2022-07-03T16:19:55.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=2, chan=1563] Channel closed 2022-07-03T16:19:55.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=581] Login timeout expired 2022-07-03T16:19:55.451 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=581] Aborting connection 2022-07-03T16:19:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:56 smithi161 ceph-mon[91579]: pgmap v10224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:56 smithi188 ceph-mon[92425]: pgmap v10224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:19:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:19:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:58 smithi161 ceph-mon[91579]: pgmap v10225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:58 smithi188 ceph-mon[92425]: pgmap v10225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:19:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:19:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:19:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:19:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:19:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:19:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:19:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:19:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:20:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:00 smithi161 ceph-mon[91579]: pgmap v10226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:20:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:00 smithi188 ceph-mon[92425]: pgmap v10226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:20:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:02 smithi161 ceph-mon[91579]: pgmap v10227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:02 smithi188 ceph-mon[92425]: pgmap v10227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:04 smithi161 ceph-mon[91579]: pgmap v10228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:04 smithi188 ceph-mon[92425]: pgmap v10228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:06 smithi161 ceph-mon[91579]: pgmap v10229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:06 smithi188 ceph-mon[92425]: pgmap v10229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:08 smithi161 ceph-mon[91579]: pgmap v10230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:08 smithi188 ceph-mon[92425]: pgmap v10230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:10 smithi161 ceph-mon[91579]: pgmap v10231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:10 smithi188 ceph-mon[92425]: pgmap v10231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:12 smithi161 ceph-mon[91579]: pgmap v10232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:12 smithi188 ceph-mon[92425]: pgmap v10232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:14 smithi161 ceph-mon[91579]: pgmap v10233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:14 smithi188 ceph-mon[92425]: pgmap v10233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:16 smithi161 ceph-mon[91579]: pgmap v10234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:16 smithi188 ceph-mon[92425]: pgmap v10234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:18 smithi161 ceph-mon[91579]: pgmap v10235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:18 smithi188 ceph-mon[92425]: pgmap v10235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:20 smithi161 ceph-mon[91579]: pgmap v10236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:20 smithi188 ceph-mon[92425]: pgmap v10236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:22 smithi161 ceph-mon[91579]: pgmap v10237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:22 smithi188 ceph-mon[92425]: pgmap v10237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:24 smithi161 ceph-mon[91579]: pgmap v10238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:24 smithi188 ceph-mon[92425]: pgmap v10238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:26 smithi161 ceph-mon[91579]: pgmap v10239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:26 smithi188 ceph-mon[92425]: pgmap v10239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:28 smithi161 ceph-mon[91579]: pgmap v10240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:28 smithi188 ceph-mon[92425]: pgmap v10240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:30 smithi161 ceph-mon[91579]: pgmap v10241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:30 smithi188 ceph-mon[92425]: pgmap v10241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:32 smithi161 ceph-mon[91579]: pgmap v10242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:32 smithi188 ceph-mon[92425]: pgmap v10242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:34 smithi161 ceph-mon[91579]: pgmap v10243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:34 smithi188 ceph-mon[92425]: pgmap v10243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:35 smithi161 ceph-mon[91579]: pgmap v10244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:35 smithi188 ceph-mon[92425]: pgmap v10244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:37 smithi161 ceph-mon[91579]: pgmap v10245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:37 smithi188 ceph-mon[92425]: pgmap v10245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:39 smithi161 ceph-mon[91579]: pgmap v10246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:39 smithi188 ceph-mon[92425]: pgmap v10246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:41 smithi161 ceph-mon[91579]: pgmap v10247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:41 smithi188 ceph-mon[92425]: pgmap v10247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:43 smithi161 ceph-mon[91579]: pgmap v10248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:43 smithi188 ceph-mon[92425]: pgmap v10248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:46.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:46 smithi161 ceph-mon[91579]: pgmap v10249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:46 smithi188 ceph-mon[92425]: pgmap v10249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:48.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:48 smithi161 ceph-mon[91579]: pgmap v10250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:48 smithi188 ceph-mon[92425]: pgmap v10250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:50 smithi161 ceph-mon[91579]: pgmap v10251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:50 smithi188 ceph-mon[92425]: pgmap v10251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:52 smithi161 ceph-mon[91579]: pgmap v10252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:52 smithi188 ceph-mon[92425]: pgmap v10252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:54 smithi161 ceph-mon[91579]: pgmap v10253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:54 smithi188 ceph-mon[92425]: pgmap v10253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:20:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:20:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:20:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:20:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:20:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:20:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:20:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:56 smithi161 ceph-mon[91579]: pgmap v10254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:56 smithi188 ceph-mon[92425]: pgmap v10254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:20:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:20:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:58 smithi161 ceph-mon[91579]: pgmap v10255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:58 smithi188 ceph-mon[92425]: pgmap v10255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:20:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:20:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:20:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:20:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:20:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:20:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:20:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:20:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:21:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:00 smithi161 ceph-mon[91579]: pgmap v10256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:00 smithi188 ceph-mon[92425]: pgmap v10256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:02 smithi161 ceph-mon[91579]: pgmap v10257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:02 smithi188 ceph-mon[92425]: pgmap v10257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:04 smithi161 ceph-mon[91579]: pgmap v10258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:04 smithi188 ceph-mon[92425]: pgmap v10258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:06 smithi161 ceph-mon[91579]: pgmap v10259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:06 smithi188 ceph-mon[92425]: pgmap v10259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:08 smithi161 ceph-mon[91579]: pgmap v10260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:08 smithi188 ceph-mon[92425]: pgmap v10260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:10 smithi161 ceph-mon[91579]: pgmap v10261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:10 smithi188 ceph-mon[92425]: pgmap v10261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:12 smithi161 ceph-mon[91579]: pgmap v10262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:12 smithi188 ceph-mon[92425]: pgmap v10262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:14 smithi161 ceph-mon[91579]: pgmap v10263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:14 smithi188 ceph-mon[92425]: pgmap v10263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:16 smithi161 ceph-mon[91579]: pgmap v10264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:16 smithi188 ceph-mon[92425]: pgmap v10264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:18 smithi161 ceph-mon[91579]: pgmap v10265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:18 smithi188 ceph-mon[92425]: pgmap v10265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:20 smithi161 ceph-mon[91579]: pgmap v10266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:20 smithi188 ceph-mon[92425]: pgmap v10266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:22 smithi161 ceph-mon[91579]: pgmap v10267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:22.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:22 smithi188 ceph-mon[92425]: pgmap v10267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-mon[91579]: pgmap v10268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:21:24.066+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:21:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:21:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:21:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:21:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:21:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:21:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:21:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:21:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:21:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:21:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:21:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:21:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:21:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:21:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:21:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:21:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:21:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:21:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:21:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:21:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:21:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:21:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:21:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:21:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:21:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:21:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:21:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:21:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:21:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:21:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:21:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:21:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:21:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:21:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:21:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:21:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:21:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:21:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:21:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:21:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:21:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:21:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:21:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:21:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:21:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=583] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:21:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=583] Local address: 172.21.15.161, port 56718 2022-07-03T16:21:24.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=583] Peer address: 172.21.15.101, port 22 2022-07-03T16:21:24.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=583] Connection lost 2022-07-03T16:21:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=583] Aborting connection 2022-07-03T16:21:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:21:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:21:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:24 smithi188 ceph-mon[92425]: pgmap v10268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:21:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:21:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:21:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:21:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:21:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:21:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:21:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:21:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:21:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:21:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:21:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:21:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:21:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:21:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:21:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:21:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:21:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:21:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:21:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:21:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:21:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:21:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:21:25.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:21:25.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:21:25.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:21:25.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:21:25.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:21:25.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:21:25.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:21:25.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:21:25.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:21:25.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:21:25.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:21:25.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:21:25.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:21:25.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:21:25.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:21:25.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:21:25.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:21:25.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:21:25.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:21:25.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:21:25.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:21:25.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:21:25.416 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:21:25.417 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=583] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:21:25.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=583] Local address: 172.21.15.161, port 56718 2022-07-03T16:21:25.418 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=583] Peer address: 172.21.15.101, port 22 2022-07-03T16:21:25.419 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=583] Connection lost 2022-07-03T16:21:25.420 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=583] Aborting connection 2022-07-03T16:21:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:21:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:21:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:21:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:21:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:21:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:21:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:21:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:21:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:21:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:21:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:21:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:21:25.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:21:25.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:21:25.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:21:25.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:21:25.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:21:25.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:21:25.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:21:25.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:21:25.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:21:25.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:21:25.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:21:25.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:21:25.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:21:25.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:21:25.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:21:25.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:21:25.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:21:25.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:21:25.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:21:25.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:21:25.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:21:25.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:21:25.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:21:25.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:21:25.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:21:25.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:21:25.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:21:25.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:21:25.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:21:25.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:21:25.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:21:25.446 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:21:25.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:21:25.447 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:21:25.448 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=583] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:21:25.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=583] Local address: 172.21.15.161, port 56718 2022-07-03T16:21:25.449 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=583] Peer address: 172.21.15.101, port 22 2022-07-03T16:21:25.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=583] Connection lost 2022-07-03T16:21:25.450 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=583] Aborting connection 2022-07-03T16:21:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:26 smithi161 ceph-mon[91579]: pgmap v10269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:26 smithi188 ceph-mon[92425]: pgmap v10269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:27 smithi161 ceph-mon[91579]: pgmap v10270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:27 smithi188 ceph-mon[92425]: pgmap v10270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:30 smithi161 ceph-mon[91579]: pgmap v10271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:30 smithi188 ceph-mon[92425]: pgmap v10271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:32 smithi161 ceph-mon[91579]: pgmap v10272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:32 smithi188 ceph-mon[92425]: pgmap v10272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:34 smithi161 ceph-mon[91579]: pgmap v10273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:34 smithi188 ceph-mon[92425]: pgmap v10273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:36 smithi161 ceph-mon[91579]: pgmap v10274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:36 smithi188 ceph-mon[92425]: pgmap v10274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:38 smithi161 ceph-mon[91579]: pgmap v10275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:38 smithi188 ceph-mon[92425]: pgmap v10275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:40 smithi161 ceph-mon[91579]: pgmap v10276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:40 smithi188 ceph-mon[92425]: pgmap v10276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:42 smithi161 ceph-mon[91579]: pgmap v10277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:42 smithi188 ceph-mon[92425]: pgmap v10277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:44 smithi161 ceph-mon[91579]: pgmap v10278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:44 smithi188 ceph-mon[92425]: pgmap v10278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:46 smithi161 ceph-mon[91579]: pgmap v10279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:46 smithi188 ceph-mon[92425]: pgmap v10279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:48 smithi161 ceph-mon[91579]: pgmap v10280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:48 smithi188 ceph-mon[92425]: pgmap v10280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:50 smithi161 ceph-mon[91579]: pgmap v10281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:50 smithi188 ceph-mon[92425]: pgmap v10281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:52 smithi161 ceph-mon[91579]: pgmap v10282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:52 smithi188 ceph-mon[92425]: pgmap v10282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:21:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:54 smithi161 ceph-mon[91579]: pgmap v10283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:54 smithi188 ceph-mon[92425]: pgmap v10283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:21:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:56 smithi161 ceph-mon[91579]: pgmap v10284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:56 smithi188 ceph-mon[92425]: pgmap v10284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:57.464 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:57.465 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:21:57.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:21:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:58 smithi161 ceph-mon[91579]: pgmap v10285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:58 smithi188 ceph-mon[92425]: pgmap v10285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:21:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:21:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:21:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:21:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:21:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:21:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:21:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:21:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:22:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:00 smithi161 ceph-mon[91579]: pgmap v10286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:00 smithi188 ceph-mon[92425]: pgmap v10286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:02 smithi161 ceph-mon[91579]: pgmap v10287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:02 smithi188 ceph-mon[92425]: pgmap v10287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:04 smithi161 ceph-mon[91579]: pgmap v10288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:04 smithi188 ceph-mon[92425]: pgmap v10288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:06 smithi161 ceph-mon[91579]: pgmap v10289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:06 smithi188 ceph-mon[92425]: pgmap v10289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:08 smithi161 ceph-mon[91579]: pgmap v10290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:08 smithi188 ceph-mon[92425]: pgmap v10290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:10 smithi161 ceph-mon[91579]: pgmap v10291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:10 smithi188 ceph-mon[92425]: pgmap v10291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:12 smithi161 ceph-mon[91579]: pgmap v10292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:12 smithi188 ceph-mon[92425]: pgmap v10292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:14 smithi161 ceph-mon[91579]: pgmap v10293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:14 smithi188 ceph-mon[92425]: pgmap v10293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:16 smithi161 ceph-mon[91579]: pgmap v10294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:16 smithi188 ceph-mon[92425]: pgmap v10294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:18.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:18 smithi161 ceph-mon[91579]: pgmap v10295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:18.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:18 smithi188 ceph-mon[92425]: pgmap v10295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:19 smithi161 ceph-mon[91579]: pgmap v10296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:19 smithi188 ceph-mon[92425]: pgmap v10296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:22 smithi161 ceph-mon[91579]: pgmap v10297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:22 smithi188 ceph-mon[92425]: pgmap v10297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:24.106 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-mon[91579]: pgmap v10298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:24.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:22:24.101+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:22:24.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:22:24.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:22:24.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:22:24.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:22:24.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:22:24.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:22:24.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:22:24.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:22:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:22:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:22:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:22:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:22:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:22:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:22:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:22:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:22:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:22:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:22:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:22:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:22:24.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:22:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:22:24.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:22:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:22:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:22:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:22:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:22:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:22:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:22:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:22:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:22:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:22:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:22:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:22:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:22:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:22:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:22:24.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:22:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:22:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:22:24.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:22:24.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:22:24.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:22:24.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:22:24.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=585] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:22:24.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=585] Local address: 172.21.15.161, port 56746 2022-07-03T16:22:24.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=585] Peer address: 172.21.15.101, port 22 2022-07-03T16:22:24.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=585] Connection lost 2022-07-03T16:22:24.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=585] Aborting connection 2022-07-03T16:22:24.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:22:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:24 smithi188 ceph-mon[92425]: pgmap v10298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:22:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:22:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:22:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:22:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:22:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:22:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:22:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:22:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:22:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:22:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:22:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:22:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:22:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:22:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:22:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:22:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:22:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:22:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:22:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:22:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:22:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:22:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:22:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:22:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:22:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:22:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:22:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:22:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:22:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:22:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:22:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:22:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:22:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:22:26.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:22:26.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:22:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:22:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:22:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:22:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:22:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:22:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:22:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:22:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:22:26.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:22:26.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:22:26.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:22:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:22:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:22:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:22:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:22:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:22:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=585] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:22:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=585] Local address: 172.21.15.161, port 56746 2022-07-03T16:22:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=585] Peer address: 172.21.15.101, port 22 2022-07-03T16:22:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=585] Connection lost 2022-07-03T16:22:26.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=585] Aborting connection 2022-07-03T16:22:26.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:26 smithi161 ceph-mon[91579]: pgmap v10299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:22:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:22:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:22:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:22:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:22:26.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:22:26.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:22:26.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:22:26.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:22:26.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:22:26.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:22:26.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:22:26.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:22:26.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:22:26.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:22:26.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:22:26.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:22:26.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:22:26.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:22:26.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:22:26.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:22:26.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:22:26.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:22:26.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:22:26.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:22:26.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:22:26.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:22:26.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:22:26.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:22:26.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:22:26.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:22:26.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:22:26.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:22:26.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:22:26.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:22:26.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:22:26.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:22:26.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:22:26.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:22:26.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:22:26.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:22:26.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:22:26.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:22:26.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:22:26.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:22:26.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:22:26.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=585] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:22:26.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=585] Local address: 172.21.15.161, port 56746 2022-07-03T16:22:26.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=585] Peer address: 172.21.15.101, port 22 2022-07-03T16:22:26.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=585] Connection lost 2022-07-03T16:22:26.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=585] Aborting connection 2022-07-03T16:22:26.442 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:26 smithi188 ceph-mon[92425]: pgmap v10299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:28 smithi161 ceph-mon[91579]: pgmap v10300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:28 smithi188 ceph-mon[92425]: pgmap v10300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:30 smithi161 ceph-mon[91579]: pgmap v10301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:30 smithi188 ceph-mon[92425]: pgmap v10301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:32 smithi161 ceph-mon[91579]: pgmap v10302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:32 smithi188 ceph-mon[92425]: pgmap v10302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:34 smithi161 ceph-mon[91579]: pgmap v10303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:34 smithi188 ceph-mon[92425]: pgmap v10303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:36 smithi161 ceph-mon[91579]: pgmap v10304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:36 smithi188 ceph-mon[92425]: pgmap v10304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:38 smithi161 ceph-mon[91579]: pgmap v10305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:38 smithi188 ceph-mon[92425]: pgmap v10305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:40 smithi161 ceph-mon[91579]: pgmap v10306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:40 smithi188 ceph-mon[92425]: pgmap v10306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:42 smithi161 ceph-mon[91579]: pgmap v10307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:42 smithi188 ceph-mon[92425]: pgmap v10307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:44 smithi161 ceph-mon[91579]: pgmap v10308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:44 smithi188 ceph-mon[92425]: pgmap v10308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:46 smithi161 ceph-mon[91579]: pgmap v10309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:46 smithi188 ceph-mon[92425]: pgmap v10309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:48 smithi161 ceph-mon[91579]: pgmap v10310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:48 smithi188 ceph-mon[92425]: pgmap v10310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:50 smithi161 ceph-mon[91579]: pgmap v10311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:50.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:50 smithi188 ceph-mon[92425]: pgmap v10311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:52 smithi161 ceph-mon[91579]: pgmap v10312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:52 smithi188 ceph-mon[92425]: pgmap v10312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:54 smithi161 ceph-mon[91579]: pgmap v10313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:54 smithi188 ceph-mon[92425]: pgmap v10313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:22:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:56 smithi161 ceph-mon[91579]: pgmap v10314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:56.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:22:56.193+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T16:22:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:56 smithi188 ceph-mon[92425]: pgmap v10314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:22:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:22:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:58 smithi161 ceph-mon[91579]: pgmap v10315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:58 smithi188 ceph-mon[92425]: pgmap v10315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:22:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:22:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:22:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:22:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:22:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:22:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:22:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:22:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:23:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:00 smithi161 ceph-mon[91579]: pgmap v10316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:00 smithi188 ceph-mon[92425]: pgmap v10316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:01 smithi161 ceph-mon[91579]: pgmap v10317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:01 smithi188 ceph-mon[92425]: pgmap v10317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:03 smithi161 ceph-mon[91579]: pgmap v10318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:03 smithi188 ceph-mon[92425]: pgmap v10318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:06 smithi161 ceph-mon[91579]: pgmap v10319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:06 smithi188 ceph-mon[92425]: pgmap v10319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:07 smithi161 ceph-mon[91579]: pgmap v10320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:07 smithi188 ceph-mon[92425]: pgmap v10320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:10 smithi161 ceph-mon[91579]: pgmap v10321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:10 smithi188 ceph-mon[92425]: pgmap v10321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:12 smithi161 ceph-mon[91579]: pgmap v10322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:12 smithi188 ceph-mon[92425]: pgmap v10322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:14 smithi161 ceph-mon[91579]: pgmap v10323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:14.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:14 smithi188 ceph-mon[92425]: pgmap v10323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:16 smithi161 ceph-mon[91579]: pgmap v10324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:16 smithi188 ceph-mon[92425]: pgmap v10324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:17 smithi161 ceph-mon[91579]: pgmap v10325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:17 smithi188 ceph-mon[92425]: pgmap v10325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:20 smithi161 ceph-mon[91579]: pgmap v10326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:20 smithi188 ceph-mon[92425]: pgmap v10326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:22 smithi161 ceph-mon[91579]: pgmap v10327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:22 smithi188 ceph-mon[92425]: pgmap v10327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-mon[91579]: pgmap v10328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:24 smithi188 ceph-mon[92425]: pgmap v10328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:24.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:23:24.607+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:23:24.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:23:24.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:23:24.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:23:24.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:23:24.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:23:24.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:23:24.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:23:24.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:23:24.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:23:24.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:23:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:23:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:23:24.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:23:24.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:23:24.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:23:24.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:23:24.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:23:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:23:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:23:24.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:23:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:23:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:23:24.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:23:24.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:23:24.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:23:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:23:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:23:24.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:23:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:23:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:23:24.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:23:24.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:23:24.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:23:24.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:23:24.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:23:24.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:23:24.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:23:24.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:23:24.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:23:24.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:23:24.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:23:24.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:23:24.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:23:24.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:23:24.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=587] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:23:24.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=587] Local address: 172.21.15.161, port 56764 2022-07-03T16:23:24.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=587] Peer address: 172.21.15.101, port 22 2022-07-03T16:23:24.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=587] Connection lost 2022-07-03T16:23:24.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=587] Aborting connection 2022-07-03T16:23:24.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:23:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:23:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:23:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:23:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:23:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:23:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:23:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:23:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:23:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:23:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:23:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:23:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:23:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:23:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:23:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:23:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:23:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:23:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:23:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:23:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:23:26.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:23:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:23:26.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:23:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:23:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:23:26.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:23:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:23:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:23:26.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:23:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:23:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:23:26.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:23:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:23:26.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:23:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:23:26.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:23:26.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:23:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:23:26.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:23:26.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:23:26.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:23:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:23:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:23:26.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:23:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:23:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:23:26.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:23:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:23:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:23:26.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:23:26.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:23:26.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:23:26.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:23:26.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=587] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:23:26.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=587] Local address: 172.21.15.161, port 56764 2022-07-03T16:23:26.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=587] Peer address: 172.21.15.101, port 22 2022-07-03T16:23:26.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=587] Connection lost 2022-07-03T16:23:26.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=587] Aborting connection 2022-07-03T16:23:26.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:26 smithi161 ceph-mon[91579]: pgmap v10329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:23:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:23:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:23:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:23:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:23:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:23:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:23:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:23:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:23:26.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:23:26.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:23:26.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:23:26.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:23:26.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:23:26.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:23:26.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:23:26.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:23:26.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:23:26.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:23:26.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:23:26.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:23:26.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:23:26.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:23:26.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:23:26.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:23:26.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:23:26.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:23:26.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:23:26.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:23:26.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:23:26.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:23:26.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:23:26.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:23:26.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:23:26.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:23:26.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:23:26.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:23:26.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:23:26.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:23:26.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:23:26.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:23:26.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:23:26.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:23:26.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:23:26.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:23:26.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:23:26.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=587] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:23:26.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=587] Local address: 172.21.15.161, port 56764 2022-07-03T16:23:26.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=587] Peer address: 172.21.15.101, port 22 2022-07-03T16:23:26.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=587] Connection lost 2022-07-03T16:23:26.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=587] Aborting connection 2022-07-03T16:23:26.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:26 smithi188 ceph-mon[92425]: pgmap v10329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:27 smithi161 ceph-mon[91579]: pgmap v10330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:27 smithi188 ceph-mon[92425]: pgmap v10330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:30 smithi161 ceph-mon[91579]: pgmap v10331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:30 smithi188 ceph-mon[92425]: pgmap v10331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:32 smithi161 ceph-mon[91579]: pgmap v10332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:32 smithi188 ceph-mon[92425]: pgmap v10332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:34 smithi161 ceph-mon[91579]: pgmap v10333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:34 smithi188 ceph-mon[92425]: pgmap v10333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:36 smithi161 ceph-mon[91579]: pgmap v10334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:36 smithi188 ceph-mon[92425]: pgmap v10334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:37 smithi161 ceph-mon[91579]: pgmap v10335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:37 smithi188 ceph-mon[92425]: pgmap v10335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:40 smithi161 ceph-mon[91579]: pgmap v10336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:40 smithi188 ceph-mon[92425]: pgmap v10336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:42 smithi161 ceph-mon[91579]: pgmap v10337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:42 smithi188 ceph-mon[92425]: pgmap v10337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:44 smithi161 ceph-mon[91579]: pgmap v10338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:44 smithi188 ceph-mon[92425]: pgmap v10338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:46 smithi161 ceph-mon[91579]: pgmap v10339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:46 smithi188 ceph-mon[92425]: pgmap v10339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:47 smithi161 ceph-mon[91579]: pgmap v10340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:47 smithi188 ceph-mon[92425]: pgmap v10340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:50 smithi161 ceph-mon[91579]: pgmap v10341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:50 smithi188 ceph-mon[92425]: pgmap v10341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:52 smithi161 ceph-mon[91579]: pgmap v10342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:52 smithi188 ceph-mon[92425]: pgmap v10342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:54 smithi161 ceph-mon[91579]: pgmap v10343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:54 smithi188 ceph-mon[92425]: pgmap v10343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:56 smithi161 ceph-mon[91579]: pgmap v10344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:56 smithi188 ceph-mon[92425]: pgmap v10344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:57 smithi161 ceph-mon[91579]: pgmap v10345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:23:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:57 smithi188 ceph-mon[92425]: pgmap v10345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:23:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:23:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:23:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:23:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:23:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:23:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:23:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:23:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:23:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:23:59 smithi161 ceph-mon[91579]: pgmap v10346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:23:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:23:59 smithi188 ceph-mon[92425]: pgmap v10346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:02 smithi161 ceph-mon[91579]: pgmap v10347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:02 smithi188 ceph-mon[92425]: pgmap v10347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:04 smithi161 ceph-mon[91579]: pgmap v10348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:04 smithi188 ceph-mon[92425]: pgmap v10348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:06 smithi161 ceph-mon[91579]: pgmap v10349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:06 smithi188 ceph-mon[92425]: pgmap v10349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:07 smithi161 ceph-mon[91579]: pgmap v10350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:07 smithi188 ceph-mon[92425]: pgmap v10350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:10 smithi161 ceph-mon[91579]: pgmap v10351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:10 smithi188 ceph-mon[92425]: pgmap v10351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:12 smithi161 ceph-mon[91579]: pgmap v10352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:12 smithi188 ceph-mon[92425]: pgmap v10352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:14 smithi161 ceph-mon[91579]: pgmap v10353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:14 smithi188 ceph-mon[92425]: pgmap v10353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:16 smithi161 ceph-mon[91579]: pgmap v10354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:16 smithi188 ceph-mon[92425]: pgmap v10354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:17 smithi161 ceph-mon[91579]: pgmap v10355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:17 smithi188 ceph-mon[92425]: pgmap v10355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:20 smithi161 ceph-mon[91579]: pgmap v10356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:20 smithi188 ceph-mon[92425]: pgmap v10356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:22 smithi161 ceph-mon[91579]: pgmap v10357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:22 smithi188 ceph-mon[92425]: pgmap v10357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:24 smithi161 ceph-mon[91579]: pgmap v10358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:24 smithi188 ceph-mon[92425]: pgmap v10358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:25.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:24:25.104+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:24:25.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:24:25.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:24:25.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:24:25.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:24:25.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:24:25.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:24:25.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:24:25.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:24:25.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:24:25.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:24:25.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:24:25.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:24:25.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:24:25.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:24:25.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:24:25.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:24:25.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:24:25.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:24:25.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:24:25.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:24:25.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:24:25.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:24:25.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:24:25.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:24:25.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:24:25.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:24:25.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:24:25.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:24:25.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:24:25.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:24:25.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:24:25.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:24:25.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:24:25.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:24:25.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:24:25.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:24:25.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:24:25.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:24:25.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:24:25.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:24:25.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:24:25.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:24:25.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:24:25.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:24:25.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=589] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:24:25.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=589] Local address: 172.21.15.161, port 56796 2022-07-03T16:24:25.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=589] Peer address: 172.21.15.101, port 22 2022-07-03T16:24:25.415 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=589] Connection lost 2022-07-03T16:24:25.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=589] Aborting connection 2022-07-03T16:24:25.416 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:24:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:24:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: pgmap v10359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:24:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:24:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:24:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:24:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: pgmap v10359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:24:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:24:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:24:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:24:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:24:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:24:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:24:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:24:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:24:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:24:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:24:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:24:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:24:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:24:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:24:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:24:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:24:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:24:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:24:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:24:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:24:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:24:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:24:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:24:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:24:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:24:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:24:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:24:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:24:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:24:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:24:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:24:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:24:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:24:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:24:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:24:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:24:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:24:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:24:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:24:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:24:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:24:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:24:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:24:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=589] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:24:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=589] Local address: 172.21.15.161, port 56796 2022-07-03T16:24:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=589] Peer address: 172.21.15.101, port 22 2022-07-03T16:24:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=589] Connection lost 2022-07-03T16:24:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=589] Aborting connection 2022-07-03T16:24:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:24:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:24:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:24:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:24:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:24:27.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:24:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:24:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:24:27.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:24:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:24:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:24:27.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:24:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:24:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:24:27.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:24:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:24:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:24:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:24:27.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:24:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:24:27.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:24:27.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:24:27.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:24:27.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:24:27.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:24:27.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:24:27.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:24:27.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:24:27.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:24:27.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:24:27.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:24:27.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:24:27.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:24:27.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:24:27.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:24:27.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:24:27.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:24:27.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:24:27.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:24:27.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:24:27.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:24:27.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:24:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:24:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:24:27.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:24:27.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=589] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:24:27.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=589] Local address: 172.21.15.161, port 56796 2022-07-03T16:24:27.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=589] Peer address: 172.21.15.101, port 22 2022-07-03T16:24:27.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=589] Connection lost 2022-07-03T16:24:27.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=589] Aborting connection 2022-07-03T16:24:27.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:28 smithi161 ceph-mon[91579]: pgmap v10360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:28 smithi188 ceph-mon[92425]: pgmap v10360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:30 smithi161 ceph-mon[91579]: pgmap v10361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:30 smithi188 ceph-mon[92425]: pgmap v10361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:32 smithi161 ceph-mon[91579]: pgmap v10362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:32 smithi188 ceph-mon[92425]: pgmap v10362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:34 smithi161 ceph-mon[91579]: pgmap v10363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:34 smithi188 ceph-mon[92425]: pgmap v10363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:36 smithi161 ceph-mon[91579]: pgmap v10364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:36 smithi188 ceph-mon[92425]: pgmap v10364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:37 smithi161 ceph-mon[91579]: pgmap v10365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:37 smithi188 ceph-mon[92425]: pgmap v10365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:40 smithi161 ceph-mon[91579]: pgmap v10366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:40 smithi188 ceph-mon[92425]: pgmap v10366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:42 smithi161 ceph-mon[91579]: pgmap v10367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:42 smithi188 ceph-mon[92425]: pgmap v10367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:44 smithi161 ceph-mon[91579]: pgmap v10368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:44 smithi188 ceph-mon[92425]: pgmap v10368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:46 smithi161 ceph-mon[91579]: pgmap v10369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:46.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:46 smithi188 ceph-mon[92425]: pgmap v10369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:47 smithi161 ceph-mon[91579]: pgmap v10370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:47 smithi188 ceph-mon[92425]: pgmap v10370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:50 smithi161 ceph-mon[91579]: pgmap v10371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:50 smithi188 ceph-mon[92425]: pgmap v10371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:52 smithi161 ceph-mon[91579]: pgmap v10372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:52 smithi188 ceph-mon[92425]: pgmap v10372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:54 smithi161 ceph-mon[91579]: pgmap v10373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:54 smithi188 ceph-mon[92425]: pgmap v10373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:56 smithi161 ceph-mon[91579]: pgmap v10374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:56 smithi188 ceph-mon[92425]: pgmap v10374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:57 smithi161 ceph-mon[91579]: pgmap v10375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:24:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:57 smithi188 ceph-mon[92425]: pgmap v10375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:24:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:24:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:24:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:24:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:24:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:24:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:24:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:24:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:24:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:24:59 smithi161 ceph-mon[91579]: pgmap v10376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:24:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:24:59 smithi188 ceph-mon[92425]: pgmap v10376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:02 smithi161 ceph-mon[91579]: pgmap v10377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:02 smithi188 ceph-mon[92425]: pgmap v10377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:04 smithi161 ceph-mon[91579]: pgmap v10378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:04 smithi188 ceph-mon[92425]: pgmap v10378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:06 smithi161 ceph-mon[91579]: pgmap v10379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:06 smithi188 ceph-mon[92425]: pgmap v10379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:07 smithi161 ceph-mon[91579]: pgmap v10380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:07.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:07 smithi188 ceph-mon[92425]: pgmap v10380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:10 smithi161 ceph-mon[91579]: pgmap v10381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:10 smithi188 ceph-mon[92425]: pgmap v10381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:12 smithi161 ceph-mon[91579]: pgmap v10382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:12 smithi188 ceph-mon[92425]: pgmap v10382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:14 smithi161 ceph-mon[91579]: pgmap v10383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:14.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:14 smithi188 ceph-mon[92425]: pgmap v10383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:16 smithi161 ceph-mon[91579]: pgmap v10384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:16 smithi188 ceph-mon[92425]: pgmap v10384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:17 smithi161 ceph-mon[91579]: pgmap v10385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:17 smithi188 ceph-mon[92425]: pgmap v10385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:20 smithi161 ceph-mon[91579]: pgmap v10386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:20 smithi188 ceph-mon[92425]: pgmap v10386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:22 smithi161 ceph-mon[91579]: pgmap v10387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:22.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:22.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:22 smithi188 ceph-mon[92425]: pgmap v10387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:22.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:24 smithi161 ceph-mon[91579]: pgmap v10388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:24 smithi188 ceph-mon[92425]: pgmap v10388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-mon[91579]: pgmap v10389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:25:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:25:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:25:26.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:25:26.033+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:25:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:25:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:25:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:25:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:25:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:25:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:25:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:25:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:25:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:25:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:25:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:25:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:25:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:25:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:25:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:25:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:25:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:25:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:25:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:25:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:25:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:25:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:25:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:25:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:25:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:25:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:25:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:25:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:25:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:25:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:25:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:25:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:25:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:25:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:25:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:25:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:25:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:25:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:25:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:25:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:25:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:25:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:25:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:25:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:25:26.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=591] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:25:26.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=591] Local address: 172.21.15.161, port 56800 2022-07-03T16:25:26.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=591] Peer address: 172.21.15.101, port 22 2022-07-03T16:25:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=591] Connection lost 2022-07-03T16:25:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=591] Aborting connection 2022-07-03T16:25:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:25:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:25:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:26 smithi188 ceph-mon[92425]: pgmap v10389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:25:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:25:26.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:25:27.091 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:25:27.092 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:25:27.092 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:25:27.093 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:25:27.093 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:25:27.093 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:25:27.094 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:25:27.095 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:25:27.095 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:25:27.095 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:25:27.096 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:25:27.096 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:25:27.097 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:25:27.097 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:25:27.097 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:25:27.098 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:25:27.098 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:25:27.098 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:25:27.099 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:25:27.099 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:25:27.099 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:25:27.100 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:25:27.100 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:25:27.101 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:25:27.101 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:25:27.101 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:25:27.102 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:25:27.102 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:25:27.103 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:25:27.103 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:25:27.103 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:25:27.104 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:25:27.104 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:25:27.105 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:25:27.105 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:25:27.105 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:25:27.106 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:25:27.106 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:25:27.106 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:25:27.107 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:25:27.107 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:25:27.107 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:25:27.108 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:25:27.108 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:25:27.109 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:25:27.109 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:25:27.110 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=591] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:25:27.110 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=591] Local address: 172.21.15.161, port 56800 2022-07-03T16:25:27.110 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=591] Peer address: 172.21.15.101, port 22 2022-07-03T16:25:27.111 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=591] Connection lost 2022-07-03T16:25:27.111 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=591] Aborting connection 2022-07-03T16:25:27.112 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:25:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:25:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:25:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:25:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:25:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:25:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:25:27.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:25:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:25:27.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:25:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:25:27.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:25:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:25:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:25:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:25:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:25:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:25:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:25:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:25:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:25:27.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:25:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:25:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:25:27.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:25:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:25:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:25:27.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:25:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:25:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:25:27.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:25:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:25:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:25:27.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:25:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:25:27.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:25:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:25:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:25:27.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:25:27.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:25:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:25:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:25:27.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:25:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:25:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:25:27.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:25:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:25:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=591] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:25:27.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=591] Local address: 172.21.15.161, port 56800 2022-07-03T16:25:27.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=591] Peer address: 172.21.15.101, port 22 2022-07-03T16:25:27.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=591] Connection lost 2022-07-03T16:25:27.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=591] Aborting connection 2022-07-03T16:25:27.400 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:28 smithi161 ceph-mon[91579]: pgmap v10390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:28 smithi188 ceph-mon[92425]: pgmap v10390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:30 smithi161 ceph-mon[91579]: pgmap v10391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:30 smithi188 ceph-mon[92425]: pgmap v10391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:32 smithi161 ceph-mon[91579]: pgmap v10392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:32 smithi188 ceph-mon[92425]: pgmap v10392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:34 smithi161 ceph-mon[91579]: pgmap v10393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:34 smithi188 ceph-mon[92425]: pgmap v10393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:36 smithi161 ceph-mon[91579]: pgmap v10394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:36 smithi188 ceph-mon[92425]: pgmap v10394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:37 smithi161 ceph-mon[91579]: pgmap v10395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:37 smithi188 ceph-mon[92425]: pgmap v10395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:40 smithi161 ceph-mon[91579]: pgmap v10396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:40 smithi188 ceph-mon[92425]: pgmap v10396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:42 smithi161 ceph-mon[91579]: pgmap v10397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:42 smithi188 ceph-mon[92425]: pgmap v10397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:44 smithi161 ceph-mon[91579]: pgmap v10398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:44 smithi188 ceph-mon[92425]: pgmap v10398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:46 smithi161 ceph-mon[91579]: pgmap v10399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:46 smithi188 ceph-mon[92425]: pgmap v10399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:47 smithi161 ceph-mon[91579]: pgmap v10400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:47 smithi188 ceph-mon[92425]: pgmap v10400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:50 smithi161 ceph-mon[91579]: pgmap v10401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:50 smithi188 ceph-mon[92425]: pgmap v10401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:52 smithi161 ceph-mon[91579]: pgmap v10402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:52 smithi188 ceph-mon[92425]: pgmap v10402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:54 smithi161 ceph-mon[91579]: pgmap v10403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:54 smithi188 ceph-mon[92425]: pgmap v10403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:56 smithi161 ceph-mon[91579]: pgmap v10404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:56 smithi188 ceph-mon[92425]: pgmap v10404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:57 smithi161 ceph-mon[91579]: pgmap v10405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:25:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:57 smithi188 ceph-mon[92425]: pgmap v10405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:25:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:25:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:25:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:25:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:25:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:25:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:25:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:25:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:25:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:25:59 smithi161 ceph-mon[91579]: pgmap v10406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:25:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:25:59 smithi188 ceph-mon[92425]: pgmap v10406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:02 smithi161 ceph-mon[91579]: pgmap v10407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:02 smithi188 ceph-mon[92425]: pgmap v10407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:04 smithi161 ceph-mon[91579]: pgmap v10408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:04 smithi188 ceph-mon[92425]: pgmap v10408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:06 smithi161 ceph-mon[91579]: pgmap v10409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:06 smithi188 ceph-mon[92425]: pgmap v10409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:07 smithi161 ceph-mon[91579]: pgmap v10410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:07 smithi188 ceph-mon[92425]: pgmap v10410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:10 smithi161 ceph-mon[91579]: pgmap v10411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:10 smithi188 ceph-mon[92425]: pgmap v10411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:12 smithi161 ceph-mon[91579]: pgmap v10412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:12 smithi188 ceph-mon[92425]: pgmap v10412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:14 smithi161 ceph-mon[91579]: pgmap v10413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:14 smithi188 ceph-mon[92425]: pgmap v10413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:16 smithi161 ceph-mon[91579]: pgmap v10414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:16 smithi188 ceph-mon[92425]: pgmap v10414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:17 smithi161 ceph-mon[91579]: pgmap v10415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:17 smithi188 ceph-mon[92425]: pgmap v10415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:20 smithi161 ceph-mon[91579]: pgmap v10416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:20 smithi188 ceph-mon[92425]: pgmap v10416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:22 smithi161 ceph-mon[91579]: pgmap v10417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:22 smithi188 ceph-mon[92425]: pgmap v10417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:24 smithi161 ceph-mon[91579]: pgmap v10418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:24 smithi188 ceph-mon[92425]: pgmap v10418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-mon[91579]: pgmap v10419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:26 smithi188 ceph-mon[92425]: pgmap v10419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:26:26.528+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:26:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:26:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:26:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:26:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:26:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:26:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:26:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:26:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:26:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:26:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:26:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:26:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:26:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:26:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:26:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:26:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:26:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:26:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:26:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:26:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:26:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:26:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:26:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:26:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:26:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:26:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:26:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:26:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:26:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:26:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:26:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:26:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:26:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:26:26.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:26:26.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:26:26.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:26:26.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:26:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:26:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:26:26.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:26:26.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:26:26.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:26:26.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:26:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:26:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=593] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:26:26.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=593] Local address: 172.21.15.161, port 56804 2022-07-03T16:26:26.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=593] Peer address: 172.21.15.101, port 22 2022-07-03T16:26:26.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=593] Connection lost 2022-07-03T16:26:26.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=593] Aborting connection 2022-07-03T16:26:26.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:26:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:26:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:26:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:26:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:26:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:26:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:26:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:26:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:26:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:26:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:26:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:26:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:26:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:26:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:26:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:26:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:26:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:26:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:26:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:26:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:26:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:26:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:26:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:26:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:26:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:26:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:26:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:26:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:26:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:26:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:26:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:26:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:26:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:26:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:26:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:26:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:26:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:26:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:26:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:26:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:26:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:26:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:26:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:26:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:26:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:26:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:26:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:26:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:26:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:26:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:26:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=593] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:26:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=593] Local address: 172.21.15.161, port 56804 2022-07-03T16:26:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=593] Peer address: 172.21.15.101, port 22 2022-07-03T16:26:27.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=593] Connection lost 2022-07-03T16:26:27.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=593] Aborting connection 2022-07-03T16:26:27.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:27.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:27 smithi161 ceph-mon[91579]: pgmap v10420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:26:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:26:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:26:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:26:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:26:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:26:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:26:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:26:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:26:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:26:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:26:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:26:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:26:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:26:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:26:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:26:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:26:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:26:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:26:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:26:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:26:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:26:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:26:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:26:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:26:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:26:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:26:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:26:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:26:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:26:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:26:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:26:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:26:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:26:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:26:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:26:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:26:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:26:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:26:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:26:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:26:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:26:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:26:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:26:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:26:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:26:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:26:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:26:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:26:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=593] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:26:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=593] Local address: 172.21.15.161, port 56804 2022-07-03T16:26:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=593] Peer address: 172.21.15.101, port 22 2022-07-03T16:26:27.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=593] Connection lost 2022-07-03T16:26:27.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=593] Aborting connection 2022-07-03T16:26:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:27.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:27 smithi188 ceph-mon[92425]: pgmap v10420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:30 smithi161 ceph-mon[91579]: pgmap v10421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:30 smithi188 ceph-mon[92425]: pgmap v10421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:32 smithi161 ceph-mon[91579]: pgmap v10422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:32 smithi188 ceph-mon[92425]: pgmap v10422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:34 smithi161 ceph-mon[91579]: pgmap v10423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:34 smithi188 ceph-mon[92425]: pgmap v10423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:36 smithi161 ceph-mon[91579]: pgmap v10424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:36 smithi188 ceph-mon[92425]: pgmap v10424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:37 smithi161 ceph-mon[91579]: pgmap v10425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:37 smithi188 ceph-mon[92425]: pgmap v10425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:40 smithi161 ceph-mon[91579]: pgmap v10426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:40 smithi188 ceph-mon[92425]: pgmap v10426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:42 smithi161 ceph-mon[91579]: pgmap v10427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:42 smithi188 ceph-mon[92425]: pgmap v10427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:44 smithi161 ceph-mon[91579]: pgmap v10428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:44 smithi188 ceph-mon[92425]: pgmap v10428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:46 smithi161 ceph-mon[91579]: pgmap v10429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:46 smithi188 ceph-mon[92425]: pgmap v10429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:47 smithi161 ceph-mon[91579]: pgmap v10430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:47 smithi188 ceph-mon[92425]: pgmap v10430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:50 smithi161 ceph-mon[91579]: pgmap v10431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:50 smithi188 ceph-mon[92425]: pgmap v10431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:52 smithi161 ceph-mon[91579]: pgmap v10432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:52 smithi188 ceph-mon[92425]: pgmap v10432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:54 smithi161 ceph-mon[91579]: pgmap v10433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:54 smithi188 ceph-mon[92425]: pgmap v10433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:56 smithi161 ceph-mon[91579]: pgmap v10434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:56 smithi188 ceph-mon[92425]: pgmap v10434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:57 smithi161 ceph-mon[91579]: pgmap v10435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:26:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:57 smithi188 ceph-mon[92425]: pgmap v10435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:26:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:26:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:26:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:26:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:26:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:26:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:26:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:26:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:26:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:26:59 smithi161 ceph-mon[91579]: pgmap v10436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:26:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:26:59 smithi188 ceph-mon[92425]: pgmap v10436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:02 smithi161 ceph-mon[91579]: pgmap v10437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:02 smithi188 ceph-mon[92425]: pgmap v10437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:04 smithi161 ceph-mon[91579]: pgmap v10438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:04 smithi188 ceph-mon[92425]: pgmap v10438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:06 smithi161 ceph-mon[91579]: pgmap v10439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:06 smithi188 ceph-mon[92425]: pgmap v10439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:07 smithi161 ceph-mon[91579]: pgmap v10440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:07 smithi188 ceph-mon[92425]: pgmap v10440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:10 smithi161 ceph-mon[91579]: pgmap v10441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:10 smithi188 ceph-mon[92425]: pgmap v10441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:12 smithi161 ceph-mon[91579]: pgmap v10442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:12 smithi188 ceph-mon[92425]: pgmap v10442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:14 smithi161 ceph-mon[91579]: pgmap v10443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:14 smithi188 ceph-mon[92425]: pgmap v10443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:16 smithi161 ceph-mon[91579]: pgmap v10444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:16 smithi188 ceph-mon[92425]: pgmap v10444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:17 smithi161 ceph-mon[91579]: pgmap v10445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:17 smithi188 ceph-mon[92425]: pgmap v10445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:20 smithi161 ceph-mon[91579]: pgmap v10446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:20 smithi188 ceph-mon[92425]: pgmap v10446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:22 smithi161 ceph-mon[91579]: pgmap v10447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:22.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:22 smithi188 ceph-mon[92425]: pgmap v10447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:24 smithi161 ceph-mon[91579]: pgmap v10448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:24 smithi188 ceph-mon[92425]: pgmap v10448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:26 smithi161 ceph-mon[91579]: pgmap v10449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:26 smithi188 ceph-mon[92425]: pgmap v10449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:27.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:27:27.030+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:27:27.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:27:27.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:27:27.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:27:27.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:27:27.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:27:27.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:27:27.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:27:27.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:27:27.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:27:27.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:27:27.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:27:27.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:27:27.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:27:27.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:27:27.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:27:27.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:27:27.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:27:27.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:27:27.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:27:27.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:27:27.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:27:27.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:27:27.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:27:27.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:27:27.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:27:27.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:27:27.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:27:27.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:27:27.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:27:27.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:27:27.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:27:27.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:27:27.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:27:27.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:27:27.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:27:27.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:27:27.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:27:27.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:27:27.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:27:27.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:27:27.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:27:27.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:27:27.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:27:27.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:27:27.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=595] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:27:27.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=595] Local address: 172.21.15.161, port 56808 2022-07-03T16:27:27.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=595] Peer address: 172.21.15.101, port 22 2022-07-03T16:27:27.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=595] Connection lost 2022-07-03T16:27:27.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=595] Aborting connection 2022-07-03T16:27:27.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:27:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:27:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:27:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:27:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:27:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:27:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:27:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:27:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:27:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:27:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:27:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:27:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:27:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:27:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:27:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:27:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:27:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:27:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:27:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:27:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:27:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:27:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:27:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:27:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:27:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:27:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:27:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:27:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:27:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:27:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:27:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:27:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:27:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:27:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:27:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:27:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:27:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:27:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:27:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:27:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:27:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:27:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:27:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:27:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:27:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:27:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:27:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:27:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:27:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:27:27.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=595] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:27:27.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=595] Local address: 172.21.15.161, port 56808 2022-07-03T16:27:27.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=595] Peer address: 172.21.15.101, port 22 2022-07-03T16:27:27.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=595] Connection lost 2022-07-03T16:27:27.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=595] Aborting connection 2022-07-03T16:27:27.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:27 smithi161 ceph-mon[91579]: pgmap v10450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:27:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:27:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:27:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:27:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:27:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:27:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:27:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:27:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:27:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:27:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:27:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:27:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:27:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:27:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:27:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:27:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:27:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:27:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:27:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:27:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:27:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:27:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:27:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:27:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:27:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:27:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:27:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:27:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:27:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:27:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:27:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:27:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:27:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:27:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:27:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:27:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:27:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:27:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:27:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:27:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:27:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:27:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:27:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:27:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:27:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:27:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:27:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:27:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:27:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=595] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:27:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=595] Local address: 172.21.15.161, port 56808 2022-07-03T16:27:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=595] Peer address: 172.21.15.101, port 22 2022-07-03T16:27:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=595] Connection lost 2022-07-03T16:27:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=595] Aborting connection 2022-07-03T16:27:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:27 smithi188 ceph-mon[92425]: pgmap v10450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:30 smithi161 ceph-mon[91579]: pgmap v10451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:30 smithi188 ceph-mon[92425]: pgmap v10451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:27:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:27:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 119.1M 2022-07-03T16:27:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 124972703: error parsing value: Value '124972703' is below minimum 939524096 2022-07-03T16:27:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:27:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:27:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: pgmap v10452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 119.1M 2022-07-03T16:27:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 124972703: error parsing value: Value '124972703' is below minimum 939524096 2022-07-03T16:27:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:27:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:27:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: pgmap v10452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:34 smithi161 ceph-mon[91579]: pgmap v10453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:34 smithi188 ceph-mon[92425]: pgmap v10453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:36 smithi188 ceph-mon[92425]: pgmap v10454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:36 smithi161 ceph-mon[91579]: pgmap v10454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:37 smithi161 ceph-mon[91579]: pgmap v10455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:37 smithi188 ceph-mon[92425]: pgmap v10455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:40 smithi161 ceph-mon[91579]: pgmap v10456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:40 smithi188 ceph-mon[92425]: pgmap v10456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:42 smithi161 ceph-mon[91579]: pgmap v10457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:42 smithi188 ceph-mon[92425]: pgmap v10457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:44 smithi161 ceph-mon[91579]: pgmap v10458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:44 smithi188 ceph-mon[92425]: pgmap v10458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:46 smithi188 ceph-mon[92425]: pgmap v10459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:46 smithi161 ceph-mon[91579]: pgmap v10459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:47 smithi161 ceph-mon[91579]: pgmap v10460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:47 smithi188 ceph-mon[92425]: pgmap v10460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:50 smithi161 ceph-mon[91579]: pgmap v10461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:50 smithi188 ceph-mon[92425]: pgmap v10461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:52 smithi161 ceph-mon[91579]: pgmap v10462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:52 smithi188 ceph-mon[92425]: pgmap v10462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:54 smithi188 ceph-mon[92425]: pgmap v10463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:54 smithi161 ceph-mon[91579]: pgmap v10463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:55 smithi161 ceph-mon[91579]: pgmap v10464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:55 smithi188 ceph-mon[92425]: pgmap v10464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:57 smithi161 ceph-mon[91579]: pgmap v10465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:27:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:57 smithi188 ceph-mon[92425]: pgmap v10465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:27:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:27:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:27:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:27:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:27:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:27:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:27:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:27:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:27:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:27:59 smithi161 ceph-mon[91579]: pgmap v10466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:27:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:27:59 smithi188 ceph-mon[92425]: pgmap v10466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:02 smithi161 ceph-mon[91579]: pgmap v10467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:02 smithi188 ceph-mon[92425]: pgmap v10467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:04 smithi188 ceph-mon[92425]: pgmap v10468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:04 smithi161 ceph-mon[91579]: pgmap v10468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:06 smithi188 ceph-mon[92425]: pgmap v10469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:06 smithi161 ceph-mon[91579]: pgmap v10469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:07 smithi161 ceph-mon[91579]: pgmap v10470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:07 smithi188 ceph-mon[92425]: pgmap v10470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:10 smithi188 ceph-mon[92425]: pgmap v10471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:10 smithi161 ceph-mon[91579]: pgmap v10471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:12 smithi188 ceph-mon[92425]: pgmap v10472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:12 smithi161 ceph-mon[91579]: pgmap v10472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:13 smithi161 ceph-mon[91579]: pgmap v10473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:13 smithi188 ceph-mon[92425]: pgmap v10473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:16 smithi188 ceph-mon[92425]: pgmap v10474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:16 smithi161 ceph-mon[91579]: pgmap v10474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:17 smithi161 ceph-mon[91579]: pgmap v10475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:17 smithi188 ceph-mon[92425]: pgmap v10475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:20 smithi188 ceph-mon[92425]: pgmap v10476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:20 smithi161 ceph-mon[91579]: pgmap v10476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:22 smithi188 ceph-mon[92425]: pgmap v10477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:22.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:22 smithi161 ceph-mon[91579]: pgmap v10477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:23 smithi188 ceph-mon[92425]: pgmap v10478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:23 smithi161 ceph-mon[91579]: pgmap v10478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:26 smithi188 ceph-mon[92425]: pgmap v10479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:26 smithi161 ceph-mon[91579]: pgmap v10479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:27 smithi161 ceph-mon[91579]: pgmap v10480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:27 smithi188 ceph-mon[92425]: pgmap v10480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:30 smithi188 ceph-mon[92425]: pgmap v10481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-mon[91579]: pgmap v10481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:28:30.285+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:28:30.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:28:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:28:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:28:30.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:28:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:28:30.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:28:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:28:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:28:30.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:28:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:28:30.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:28:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:28:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:28:30.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:28:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:28:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:28:30.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:28:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:28:30.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:28:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:28:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:28:30.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:28:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:28:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:28:30.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:28:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:28:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:28:30.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:28:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:28:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:28:30.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:28:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:28:30.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:28:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:28:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:28:30.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:28:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:28:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:28:30.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:28:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:28:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:28:30.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:28:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:28:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:28:30.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=597] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:28:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=597] Local address: 172.21.15.161, port 56812 2022-07-03T16:28:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=597] Peer address: 172.21.15.101, port 22 2022-07-03T16:28:30.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=597] Connection lost 2022-07-03T16:28:30.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=597] Aborting connection 2022-07-03T16:28:30.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:28:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:28:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:28:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:28:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:28:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:28:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:28:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:28:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:28:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:28:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:28:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:28:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:28:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:28:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:28:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:28:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:28:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:28:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:28:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:28:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:28:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:28:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:28:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:28:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:28:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:28:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:28:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:28:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:28:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:28:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:28:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:28:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:28:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:28:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:28:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:28:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:28:31.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:28:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:28:31.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:28:31.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:28:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:28:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:28:31.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:28:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:28:31.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:28:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:28:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:28:31.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:28:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:28:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:28:31.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=597] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:28:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=597] Local address: 172.21.15.161, port 56812 2022-07-03T16:28:31.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=597] Peer address: 172.21.15.101, port 22 2022-07-03T16:28:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=597] Connection lost 2022-07-03T16:28:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=597] Aborting connection 2022-07-03T16:28:31.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:31 smithi188 ceph-mon[92425]: pgmap v10482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:28:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:28:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:28:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:28:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:28:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:28:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:28:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:28:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:28:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:28:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:28:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:28:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:28:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:28:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:28:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:28:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:28:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:28:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:28:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:28:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:28:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:28:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:28:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:28:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:28:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:28:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:28:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:28:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:28:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:28:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:28:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:28:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:28:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:28:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:28:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:28:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:28:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:28:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:28:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:28:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:28:31.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:28:31.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:28:31.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:28:31.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:28:31.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:28:31.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:28:31.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:28:31.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:28:31.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=597] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:28:31.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=597] Local address: 172.21.15.161, port 56812 2022-07-03T16:28:31.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=597] Peer address: 172.21.15.101, port 22 2022-07-03T16:28:31.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=597] Connection lost 2022-07-03T16:28:31.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=597] Aborting connection 2022-07-03T16:28:31.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:31 smithi161 ceph-mon[91579]: pgmap v10482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:33 smithi161 ceph-mon[91579]: pgmap v10483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:33 smithi188 ceph-mon[92425]: pgmap v10483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:36 smithi188 ceph-mon[92425]: pgmap v10484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:36 smithi161 ceph-mon[91579]: pgmap v10484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:37 smithi161 ceph-mon[91579]: pgmap v10485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:37 smithi188 ceph-mon[92425]: pgmap v10485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:40 smithi188 ceph-mon[92425]: pgmap v10486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:40 smithi161 ceph-mon[91579]: pgmap v10486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:41 smithi188 ceph-mon[92425]: pgmap v10487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:41 smithi161 ceph-mon[91579]: pgmap v10487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:43 smithi161 ceph-mon[91579]: pgmap v10488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:43 smithi188 ceph-mon[92425]: pgmap v10488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:45 smithi161 ceph-mon[91579]: pgmap v10489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:45 smithi188 ceph-mon[92425]: pgmap v10489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:47 smithi161 ceph-mon[91579]: pgmap v10490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:47 smithi188 ceph-mon[92425]: pgmap v10490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:49 smithi161 ceph-mon[91579]: pgmap v10491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:49 smithi188 ceph-mon[92425]: pgmap v10491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:51 smithi161 ceph-mon[91579]: pgmap v10492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:51 smithi188 ceph-mon[92425]: pgmap v10492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:53 smithi161 ceph-mon[91579]: pgmap v10493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:53 smithi188 ceph-mon[92425]: pgmap v10493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:55 smithi161 ceph-mon[91579]: pgmap v10494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:55 smithi188 ceph-mon[92425]: pgmap v10494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:57 smithi161 ceph-mon[91579]: pgmap v10495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:28:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:57 smithi188 ceph-mon[92425]: pgmap v10495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:28:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:28:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:28:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:28:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:28:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:28:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:28:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:28:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:28:59 smithi161 ceph-mon[91579]: pgmap v10496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:28:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:28:59 smithi188 ceph-mon[92425]: pgmap v10496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:01 smithi161 ceph-mon[91579]: pgmap v10497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:01 smithi188 ceph-mon[92425]: pgmap v10497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:03 smithi161 ceph-mon[91579]: pgmap v10498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:03 smithi188 ceph-mon[92425]: pgmap v10498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:05 smithi161 ceph-mon[91579]: pgmap v10499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:05 smithi188 ceph-mon[92425]: pgmap v10499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:07 smithi161 ceph-mon[91579]: pgmap v10500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:07 smithi188 ceph-mon[92425]: pgmap v10500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:09 smithi161 ceph-mon[91579]: pgmap v10501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:09 smithi188 ceph-mon[92425]: pgmap v10501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:11 smithi161 ceph-mon[91579]: pgmap v10502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:11 smithi188 ceph-mon[92425]: pgmap v10502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:13 smithi161 ceph-mon[91579]: pgmap v10503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:13 smithi188 ceph-mon[92425]: pgmap v10503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:15 smithi161 ceph-mon[91579]: pgmap v10504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:15 smithi188 ceph-mon[92425]: pgmap v10504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:17 smithi161 ceph-mon[91579]: pgmap v10505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:17 smithi188 ceph-mon[92425]: pgmap v10505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:19 smithi161 ceph-mon[91579]: pgmap v10506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:19 smithi188 ceph-mon[92425]: pgmap v10506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:21 smithi161 ceph-mon[91579]: pgmap v10507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:21 smithi188 ceph-mon[92425]: pgmap v10507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:23.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:23 smithi161 ceph-mon[91579]: pgmap v10508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:23 smithi188 ceph-mon[92425]: pgmap v10508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:25 smithi161 ceph-mon[91579]: pgmap v10509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:25 smithi188 ceph-mon[92425]: pgmap v10509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:27 smithi161 ceph-mon[91579]: pgmap v10510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:27 smithi188 ceph-mon[92425]: pgmap v10510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:29 smithi161 ceph-mon[91579]: pgmap v10511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:29 smithi188 ceph-mon[92425]: pgmap v10511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:29:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:29:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:29:31.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:29:30.785+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:29:31.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:29:31.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:29:31.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:29:31.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:29:31.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:29:31.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:29:31.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:29:31.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:29:31.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:29:31.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:29:31.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:29:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:29:31.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:29:31.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:29:31.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:29:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:29:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:29:31.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:29:31.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:29:31.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:29:31.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:29:31.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:29:31.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:29:31.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:29:31.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:29:31.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:29:31.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:29:31.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:29:31.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:29:31.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:29:31.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:29:31.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:29:31.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:29:31.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:29:31.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:29:31.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:29:31.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:29:31.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:29:31.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:29:31.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:29:31.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:29:31.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:29:31.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:29:31.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:29:31.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=599] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:29:31.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=599] Local address: 172.21.15.161, port 56816 2022-07-03T16:29:31.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=599] Peer address: 172.21.15.101, port 22 2022-07-03T16:29:31.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=599] Connection lost 2022-07-03T16:29:31.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=599] Aborting connection 2022-07-03T16:29:31.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:29:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:29:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:29:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:29:31.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:29:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:29:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:29:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:29:32.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:29:32.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:29:32.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:29:32.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:29:32.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:29:32.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:29:32.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:29:32.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:29:32.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:29:32.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:29:32.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:29:32.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:29:32.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:29:32.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:29:32.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:29:32.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:29:32.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:29:32.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:29:32.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:29:32.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:29:32.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:29:32.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:29:32.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:29:32.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:29:32.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:29:32.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:29:32.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:29:32.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:29:32.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:29:32.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:29:32.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:29:32.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:29:32.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:29:32.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:29:32.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:29:32.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:29:32.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:29:32.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:29:32.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:29:32.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:29:32.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:29:32.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:29:32.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:29:32.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=599] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:29:32.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=599] Local address: 172.21.15.161, port 56816 2022-07-03T16:29:32.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=599] Peer address: 172.21.15.101, port 22 2022-07-03T16:29:32.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=599] Connection lost 2022-07-03T16:29:32.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=599] Aborting connection 2022-07-03T16:29:32.158 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:31 smithi161 ceph-mon[91579]: pgmap v10512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:29:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:29:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:29:32.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:29:32.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:29:32.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:29:32.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:29:32.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:29:32.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:29:32.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:29:32.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:29:32.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:29:32.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:29:32.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:29:32.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:29:32.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:29:32.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:29:32.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:29:32.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:29:32.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:29:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:29:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:29:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:29:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:29:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:29:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:29:32.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:29:32.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:29:32.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:29:32.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:29:32.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:29:32.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:29:32.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:29:32.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:29:32.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:29:32.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:29:32.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:29:32.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:29:32.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:29:32.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:29:32.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:29:32.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:29:32.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:29:32.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:29:32.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:29:32.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:29:32.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=599] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:29:32.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=599] Local address: 172.21.15.161, port 56816 2022-07-03T16:29:32.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=599] Peer address: 172.21.15.101, port 22 2022-07-03T16:29:32.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=599] Connection lost 2022-07-03T16:29:32.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=599] Aborting connection 2022-07-03T16:29:32.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:31 smithi188 ceph-mon[92425]: pgmap v10512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:33 smithi161 ceph-mon[91579]: pgmap v10513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:33 smithi188 ceph-mon[92425]: pgmap v10513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:35 smithi161 ceph-mon[91579]: pgmap v10514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:35 smithi188 ceph-mon[92425]: pgmap v10514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:37 smithi161 ceph-mon[91579]: pgmap v10515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:37 smithi188 ceph-mon[92425]: pgmap v10515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:39 smithi161 ceph-mon[91579]: pgmap v10516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:39 smithi188 ceph-mon[92425]: pgmap v10516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:41 smithi161 ceph-mon[91579]: pgmap v10517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:41 smithi188 ceph-mon[92425]: pgmap v10517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:43 smithi161 ceph-mon[91579]: pgmap v10518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:43 smithi188 ceph-mon[92425]: pgmap v10518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:45 smithi161 ceph-mon[91579]: pgmap v10519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:45 smithi188 ceph-mon[92425]: pgmap v10519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:47 smithi161 ceph-mon[91579]: pgmap v10520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:47 smithi188 ceph-mon[92425]: pgmap v10520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:49 smithi161 ceph-mon[91579]: pgmap v10521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:49 smithi188 ceph-mon[92425]: pgmap v10521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:51 smithi161 ceph-mon[91579]: pgmap v10522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:51 smithi188 ceph-mon[92425]: pgmap v10522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:53 smithi161 ceph-mon[91579]: pgmap v10523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:53 smithi188 ceph-mon[92425]: pgmap v10523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:55 smithi161 ceph-mon[91579]: pgmap v10524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:55 smithi188 ceph-mon[92425]: pgmap v10524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:57 smithi161 ceph-mon[91579]: pgmap v10525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:29:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:57 smithi188 ceph-mon[92425]: pgmap v10525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:29:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:29:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:29:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:29:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:29:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:29:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:29:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:29:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:29:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:29:59 smithi161 ceph-mon[91579]: pgmap v10526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:29:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:29:59 smithi188 ceph-mon[92425]: pgmap v10526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:30:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:30:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:01 smithi188 ceph-mon[92425]: pgmap v10527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:01 smithi161 ceph-mon[91579]: pgmap v10527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:03 smithi188 ceph-mon[92425]: pgmap v10528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:03 smithi161 ceph-mon[91579]: pgmap v10528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:05 smithi161 ceph-mon[91579]: pgmap v10529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:05 smithi188 ceph-mon[92425]: pgmap v10529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:07 smithi161 ceph-mon[91579]: pgmap v10530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:07 smithi188 ceph-mon[92425]: pgmap v10530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:09 smithi161 ceph-mon[91579]: pgmap v10531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:09 smithi188 ceph-mon[92425]: pgmap v10531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:11 smithi161 ceph-mon[91579]: pgmap v10532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:11 smithi188 ceph-mon[92425]: pgmap v10532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:13 smithi161 ceph-mon[91579]: pgmap v10533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:13 smithi188 ceph-mon[92425]: pgmap v10533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:15 smithi161 ceph-mon[91579]: pgmap v10534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:15 smithi188 ceph-mon[92425]: pgmap v10534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:17 smithi161 ceph-mon[91579]: pgmap v10535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:17 smithi188 ceph-mon[92425]: pgmap v10535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:19 smithi161 ceph-mon[91579]: pgmap v10536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:19 smithi188 ceph-mon[92425]: pgmap v10536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:21 smithi161 ceph-mon[91579]: pgmap v10537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:21 smithi188 ceph-mon[92425]: pgmap v10537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:23 smithi161 ceph-mon[91579]: pgmap v10538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:23 smithi188 ceph-mon[92425]: pgmap v10538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:25 smithi161 ceph-mon[91579]: pgmap v10539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:25 smithi188 ceph-mon[92425]: pgmap v10539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:27 smithi161 ceph-mon[91579]: pgmap v10540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:27 smithi188 ceph-mon[92425]: pgmap v10540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:29 smithi161 ceph-mon[91579]: pgmap v10541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:29 smithi188 ceph-mon[92425]: pgmap v10541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-mon[91579]: pgmap v10542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:30:31.285+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:30:31.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:30:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:30:31.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:30:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:30:31.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:30:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:30:31.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:30:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:30:31.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:30:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:30:31.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:30:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:30:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:30:31.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:30:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:30:31.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:30:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:30:31.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:30:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:30:31.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:30:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:30:31.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:30:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:30:31.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:30:31.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:30:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:30:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:30:31.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:30:31.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:30:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:30:31.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:30:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:30:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:30:31.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:30:31.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:30:31.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:30:31.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:30:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:30:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:30:31.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:30:31.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:30:31.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:30:31.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:30:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:30:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=601] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:30:31.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=601] Local address: 172.21.15.161, port 56820 2022-07-03T16:30:31.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=601] Peer address: 172.21.15.101, port 22 2022-07-03T16:30:31.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=601] Connection lost 2022-07-03T16:30:31.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=601] Aborting connection 2022-07-03T16:30:31.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:30:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:30:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:31 smithi188 ceph-mon[92425]: pgmap v10542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:30:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:30:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:30:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:30:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:30:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:30:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:30:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:30:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:30:32.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:30:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:30:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:30:32.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:30:32.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:30:32.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:30:32.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:30:32.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:30:32.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:30:32.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:30:32.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:30:32.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:30:32.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:30:32.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:30:32.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:30:32.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:30:32.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:30:32.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:30:32.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:30:32.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:30:32.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:30:32.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:30:32.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:30:32.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:30:32.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:30:32.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:30:32.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:30:32.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:30:32.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:30:32.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:30:32.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:30:32.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:30:32.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:30:32.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:30:32.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:30:32.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:30:32.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:30:32.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:30:32.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:30:32.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:30:32.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=601] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:30:32.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=601] Local address: 172.21.15.161, port 56820 2022-07-03T16:30:32.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=601] Peer address: 172.21.15.101, port 22 2022-07-03T16:30:32.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=601] Connection lost 2022-07-03T16:30:32.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=601] Aborting connection 2022-07-03T16:30:32.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:30:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:30:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:30:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:30:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:30:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:30:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:30:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:30:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:30:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:30:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:30:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:30:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:30:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:30:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:30:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:30:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:30:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:30:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:30:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:30:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:30:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:30:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:30:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:30:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:30:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:30:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:30:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:30:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:30:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:30:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:30:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:30:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:30:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:30:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:30:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:30:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:30:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:30:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:30:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:30:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:30:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:30:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:30:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:30:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:30:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:30:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:30:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:30:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:30:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=601] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:30:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=601] Local address: 172.21.15.161, port 56820 2022-07-03T16:30:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=601] Peer address: 172.21.15.101, port 22 2022-07-03T16:30:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=601] Connection lost 2022-07-03T16:30:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=601] Aborting connection 2022-07-03T16:30:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:33 smithi161 ceph-mon[91579]: pgmap v10543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:33 smithi188 ceph-mon[92425]: pgmap v10543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:35 smithi161 ceph-mon[91579]: pgmap v10544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:35 smithi188 ceph-mon[92425]: pgmap v10544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:37 smithi188 ceph-mon[92425]: pgmap v10545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:37 smithi161 ceph-mon[91579]: pgmap v10545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:39 smithi161 ceph-mon[91579]: pgmap v10546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:39 smithi188 ceph-mon[92425]: pgmap v10546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:41 smithi161 ceph-mon[91579]: pgmap v10547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:41 smithi188 ceph-mon[92425]: pgmap v10547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:43 smithi188 ceph-mon[92425]: pgmap v10548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:43 smithi161 ceph-mon[91579]: pgmap v10548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:45 smithi161 ceph-mon[91579]: pgmap v10549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:45 smithi188 ceph-mon[92425]: pgmap v10549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:47 smithi188 ceph-mon[92425]: pgmap v10550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:47 smithi161 ceph-mon[91579]: pgmap v10550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:49 smithi161 ceph-mon[91579]: pgmap v10551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:49 smithi188 ceph-mon[92425]: pgmap v10551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:51 smithi161 ceph-mon[91579]: pgmap v10552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:51 smithi188 ceph-mon[92425]: pgmap v10552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:53 smithi188 ceph-mon[92425]: pgmap v10553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:53 smithi161 ceph-mon[91579]: pgmap v10553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:55 smithi161 ceph-mon[91579]: pgmap v10554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:55 smithi188 ceph-mon[92425]: pgmap v10554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:57 smithi161 ceph-mon[91579]: pgmap v10555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:30:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:57 smithi188 ceph-mon[92425]: pgmap v10555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:30:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:30:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:30:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:30:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:30:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:30:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:30:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:30:58.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:30:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:30:59 smithi188 ceph-mon[92425]: pgmap v10556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:30:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:30:59 smithi161 ceph-mon[91579]: pgmap v10556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:01 smithi161 ceph-mon[91579]: pgmap v10557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:01 smithi188 ceph-mon[92425]: pgmap v10557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:03 smithi188 ceph-mon[92425]: pgmap v10558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:03 smithi161 ceph-mon[91579]: pgmap v10558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:05 smithi161 ceph-mon[91579]: pgmap v10559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:05 smithi188 ceph-mon[92425]: pgmap v10559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:07 smithi188 ceph-mon[92425]: pgmap v10560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:07 smithi161 ceph-mon[91579]: pgmap v10560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:09 smithi161 ceph-mon[91579]: pgmap v10561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:09 smithi188 ceph-mon[92425]: pgmap v10561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:11 smithi161 ceph-mon[91579]: pgmap v10562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:11 smithi188 ceph-mon[92425]: pgmap v10562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:13 smithi188 ceph-mon[92425]: pgmap v10563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:13 smithi161 ceph-mon[91579]: pgmap v10563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:15 smithi161 ceph-mon[91579]: pgmap v10564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:15 smithi188 ceph-mon[92425]: pgmap v10564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:17 smithi188 ceph-mon[92425]: pgmap v10565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:17 smithi161 ceph-mon[91579]: pgmap v10565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:19 smithi161 ceph-mon[91579]: pgmap v10566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:19 smithi188 ceph-mon[92425]: pgmap v10566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:21 smithi161 ceph-mon[91579]: pgmap v10567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:21 smithi188 ceph-mon[92425]: pgmap v10567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:23 smithi188 ceph-mon[92425]: pgmap v10568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:23 smithi161 ceph-mon[91579]: pgmap v10568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:25 smithi161 ceph-mon[91579]: pgmap v10569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:25 smithi188 ceph-mon[92425]: pgmap v10569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:27 smithi188 ceph-mon[92425]: pgmap v10570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:27 smithi161 ceph-mon[91579]: pgmap v10570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:29 smithi161 ceph-mon[91579]: pgmap v10571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:29 smithi188 ceph-mon[92425]: pgmap v10571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-mon[91579]: pgmap v10572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:31 smithi188 ceph-mon[92425]: pgmap v10572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:32.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:31:31.795+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:31:32.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:31:32.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:31:32.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:31:32.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:31:32.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:31:32.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:31:32.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:31:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:31:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:31:32.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:31:32.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:31:32.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:31:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:31:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:31:32.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:31:32.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:31:32.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:31:32.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:31:32.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:31:32.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:31:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:31:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:31:32.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:31:32.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:31:32.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:31:32.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:31:32.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:31:32.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:31:32.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:31:32.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:31:32.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:31:32.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:31:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:31:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:31:32.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:31:32.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:31:32.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:31:32.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:31:32.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:31:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:31:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:31:32.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:31:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:31:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:31:32.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=603] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:31:32.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=603] Local address: 172.21.15.161, port 56824 2022-07-03T16:31:32.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=603] Peer address: 172.21.15.101, port 22 2022-07-03T16:31:32.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=603] Connection lost 2022-07-03T16:31:32.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=603] Aborting connection 2022-07-03T16:31:32.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:31:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:31:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:31:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:31:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:31:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:31:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:31:32.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:31:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:31:32.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:31:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:31:32.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:31:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:31:32.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:31:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:31:32.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:31:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:31:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:31:32.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:31:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:31:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:31:32.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:31:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:31:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:31:32.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:31:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:31:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:31:32.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:31:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:31:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:31:32.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:31:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:31:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:31:32.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:31:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:31:32.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:31:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:31:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:31:32.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:31:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:31:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:31:32.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:31:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:31:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:31:32.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:31:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:31:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:31:32.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:31:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:31:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:31:32.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:31:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=603] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:31:32.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=603] Local address: 172.21.15.161, port 56824 2022-07-03T16:31:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=603] Peer address: 172.21.15.101, port 22 2022-07-03T16:31:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=603] Connection lost 2022-07-03T16:31:32.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=603] Aborting connection 2022-07-03T16:31:32.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:31:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:31:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:31:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:31:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:31:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:31:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:31:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:31:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:31:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:31:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:31:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:31:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:31:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:31:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:31:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:31:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:31:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:31:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:31:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:31:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:31:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:31:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:31:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:31:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:31:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:31:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:31:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:31:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:31:32.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:31:32.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:31:32.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:31:32.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:31:32.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:31:32.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:31:32.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:31:32.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:31:32.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:31:32.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:31:32.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:31:32.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:31:32.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:31:32.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:31:32.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:31:32.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:31:32.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:31:32.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:31:32.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:31:32.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:31:32.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=603] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:31:32.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=603] Local address: 172.21.15.161, port 56824 2022-07-03T16:31:32.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=603] Peer address: 172.21.15.101, port 22 2022-07-03T16:31:32.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=603] Connection lost 2022-07-03T16:31:32.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=603] Aborting connection 2022-07-03T16:31:32.911 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:33 smithi188 ceph-mon[92425]: pgmap v10573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:33 smithi161 ceph-mon[91579]: pgmap v10573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:35 smithi161 ceph-mon[91579]: pgmap v10574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:35 smithi188 ceph-mon[92425]: pgmap v10574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:37 smithi188 ceph-mon[92425]: pgmap v10575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:37 smithi161 ceph-mon[91579]: pgmap v10575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:39 smithi161 ceph-mon[91579]: pgmap v10576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:39 smithi188 ceph-mon[92425]: pgmap v10576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:41 smithi161 ceph-mon[91579]: pgmap v10577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:41 smithi188 ceph-mon[92425]: pgmap v10577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:43 smithi188 ceph-mon[92425]: pgmap v10578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:43 smithi161 ceph-mon[91579]: pgmap v10578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:45 smithi161 ceph-mon[91579]: pgmap v10579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:45 smithi188 ceph-mon[92425]: pgmap v10579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:47 smithi188 ceph-mon[92425]: pgmap v10580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:47 smithi161 ceph-mon[91579]: pgmap v10580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:49 smithi161 ceph-mon[91579]: pgmap v10581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:49 smithi188 ceph-mon[92425]: pgmap v10581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:51 smithi161 ceph-mon[91579]: pgmap v10582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:51 smithi188 ceph-mon[92425]: pgmap v10582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:53 smithi188 ceph-mon[92425]: pgmap v10583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:53 smithi161 ceph-mon[91579]: pgmap v10583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:55 smithi161 ceph-mon[91579]: pgmap v10584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:55 smithi188 ceph-mon[92425]: pgmap v10584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:57 smithi188 ceph-mon[92425]: pgmap v10585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:31:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:57 smithi161 ceph-mon[91579]: pgmap v10585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:31:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:31:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:31:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:31:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:31:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:31:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:31:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:31:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:31:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:31:59 smithi188 ceph-mon[92425]: pgmap v10586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:31:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:31:59 smithi161 ceph-mon[91579]: pgmap v10586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:01 smithi161 ceph-mon[91579]: pgmap v10587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:01 smithi188 ceph-mon[92425]: pgmap v10587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:03 smithi188 ceph-mon[92425]: pgmap v10588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:03 smithi161 ceph-mon[91579]: pgmap v10588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:05 smithi161 ceph-mon[91579]: pgmap v10589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:05 smithi188 ceph-mon[92425]: pgmap v10589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:07 smithi188 ceph-mon[92425]: pgmap v10590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:07 smithi161 ceph-mon[91579]: pgmap v10590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:09 smithi161 ceph-mon[91579]: pgmap v10591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:09 smithi188 ceph-mon[92425]: pgmap v10591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:11 smithi161 ceph-mon[91579]: pgmap v10592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:11 smithi188 ceph-mon[92425]: pgmap v10592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:13 smithi188 ceph-mon[92425]: pgmap v10593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:13 smithi161 ceph-mon[91579]: pgmap v10593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:15 smithi161 ceph-mon[91579]: pgmap v10594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:15 smithi188 ceph-mon[92425]: pgmap v10594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:17 smithi188 ceph-mon[92425]: pgmap v10595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:17 smithi161 ceph-mon[91579]: pgmap v10595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:19 smithi161 ceph-mon[91579]: pgmap v10596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:19 smithi188 ceph-mon[92425]: pgmap v10596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:21 smithi161 ceph-mon[91579]: pgmap v10597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:21 smithi188 ceph-mon[92425]: pgmap v10597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:23 smithi188 ceph-mon[92425]: pgmap v10598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:23 smithi161 ceph-mon[91579]: pgmap v10598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:25 smithi161 ceph-mon[91579]: pgmap v10599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:25.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:25 smithi188 ceph-mon[92425]: pgmap v10599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:27 smithi188 ceph-mon[92425]: pgmap v10600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:27 smithi161 ceph-mon[91579]: pgmap v10600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:29 smithi161 ceph-mon[91579]: pgmap v10601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:29 smithi188 ceph-mon[92425]: pgmap v10601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:31 smithi161 ceph-mon[91579]: pgmap v10602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:31 smithi188 ceph-mon[92425]: pgmap v10602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:32:32.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:32:32.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:32:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:32:32.299+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:32:32.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:32:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:32:32.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:32:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:32:32.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:32:32.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:32:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:32:32.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:32:32.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:32:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:32:32.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:32:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:32:32.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:32:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:32:32.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:32:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:32:32.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:32:32.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:32:32.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:32:32.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:32:32.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:32:32.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:32:32.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:32:32.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:32:32.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:32:32.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:32:32.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:32:32.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:32:32.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:32:32.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:32:32.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:32:32.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:32:32.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:32:32.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:32:32.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:32:32.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:32:32.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:32:32.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:32:32.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:32:32.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:32:32.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:32:32.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:32:32.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:32:32.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:32:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=605] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:32:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=605] Local address: 172.21.15.161, port 56828 2022-07-03T16:32:32.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=605] Peer address: 172.21.15.101, port 22 2022-07-03T16:32:32.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=605] Connection lost 2022-07-03T16:32:32.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=605] Aborting connection 2022-07-03T16:32:32.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:32:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:32:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:32:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:32:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:32:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:32:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:32:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:32:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:32:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:32:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:32:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:32:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:32:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:32:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:32:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:32:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:32:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:32:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:32:33.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:32:33.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:32:33.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:32:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:32:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:32:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:32:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:32:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:32:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:32:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:32:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:32:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:32:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:32:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:32:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:32:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:32:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:32:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:32:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:32:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:32:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:32:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:32:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:32:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:32:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:32:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:32:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:32:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:32:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:32:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:32:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:32:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:32:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=605] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:32:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=605] Local address: 172.21.15.161, port 56828 2022-07-03T16:32:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=605] Peer address: 172.21.15.101, port 22 2022-07-03T16:32:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=605] Connection lost 2022-07-03T16:32:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=605] Aborting connection 2022-07-03T16:32:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:33 smithi188 ceph-mon[92425]: pgmap v10603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:32:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:32:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:32:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:32:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:32:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:32:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:32:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:32:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:32:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:32:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:32:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:32:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:32:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:32:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:32:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:32:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:32:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:32:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:32:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:32:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:32:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:32:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:32:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:32:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:32:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:32:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:32:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:32:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:32:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:32:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:32:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:32:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:32:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:32:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:32:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:32:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:32:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:32:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:32:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:32:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:32:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:32:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:32:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:32:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:32:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:32:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=605] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:32:33.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=605] Local address: 172.21.15.161, port 56828 2022-07-03T16:32:33.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=605] Peer address: 172.21.15.101, port 22 2022-07-03T16:32:33.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=605] Connection lost 2022-07-03T16:32:33.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=605] Aborting connection 2022-07-03T16:32:33.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:33 smithi161 ceph-mon[91579]: pgmap v10603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:35 smithi161 ceph-mon[91579]: pgmap v10604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:35 smithi188 ceph-mon[92425]: pgmap v10604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:37 smithi188 ceph-mon[92425]: pgmap v10605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:37 smithi161 ceph-mon[91579]: pgmap v10605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:39 smithi161 ceph-mon[91579]: pgmap v10606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:39 smithi188 ceph-mon[92425]: pgmap v10606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:41 smithi161 ceph-mon[91579]: pgmap v10607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:41 smithi188 ceph-mon[92425]: pgmap v10607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:43 smithi188 ceph-mon[92425]: pgmap v10608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:43 smithi161 ceph-mon[91579]: pgmap v10608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:45 smithi161 ceph-mon[91579]: pgmap v10609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:45 smithi188 ceph-mon[92425]: pgmap v10609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:47 smithi188 ceph-mon[92425]: pgmap v10610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:47 smithi161 ceph-mon[91579]: pgmap v10610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:49 smithi161 ceph-mon[91579]: pgmap v10611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:49 smithi188 ceph-mon[92425]: pgmap v10611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:51 smithi161 ceph-mon[91579]: pgmap v10612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:51 smithi188 ceph-mon[92425]: pgmap v10612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:53 smithi188 ceph-mon[92425]: pgmap v10613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:53 smithi161 ceph-mon[91579]: pgmap v10613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:55 smithi161 ceph-mon[91579]: pgmap v10614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:55 smithi188 ceph-mon[92425]: pgmap v10614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:57 smithi188 ceph-mon[92425]: pgmap v10615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:32:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:57 smithi161 ceph-mon[91579]: pgmap v10615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:32:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:32:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:32:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:32:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:32:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:32:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:32:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:32:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:32:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:32:59 smithi161 ceph-mon[91579]: pgmap v10616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:32:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:32:59 smithi188 ceph-mon[92425]: pgmap v10616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:01 smithi161 ceph-mon[91579]: pgmap v10617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:01 smithi188 ceph-mon[92425]: pgmap v10617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:03 smithi161 ceph-mon[91579]: pgmap v10618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:03 smithi188 ceph-mon[92425]: pgmap v10618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:05 smithi161 ceph-mon[91579]: pgmap v10619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:05 smithi188 ceph-mon[92425]: pgmap v10619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:07 smithi188 ceph-mon[92425]: pgmap v10620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:07 smithi161 ceph-mon[91579]: pgmap v10620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:09 smithi161 ceph-mon[91579]: pgmap v10621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:09 smithi188 ceph-mon[92425]: pgmap v10621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:11 smithi161 ceph-mon[91579]: pgmap v10622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:11 smithi188 ceph-mon[92425]: pgmap v10622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:13 smithi161 ceph-mon[91579]: pgmap v10623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:13 smithi188 ceph-mon[92425]: pgmap v10623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:15 smithi161 ceph-mon[91579]: pgmap v10624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:15 smithi188 ceph-mon[92425]: pgmap v10624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:17 smithi161 ceph-mon[91579]: pgmap v10625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:17 smithi188 ceph-mon[92425]: pgmap v10625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:19 smithi161 ceph-mon[91579]: pgmap v10626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:19 smithi188 ceph-mon[92425]: pgmap v10626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:21 smithi161 ceph-mon[91579]: pgmap v10627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:21 smithi188 ceph-mon[92425]: pgmap v10627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:23 smithi161 ceph-mon[91579]: pgmap v10628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:23 smithi188 ceph-mon[92425]: pgmap v10628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:25 smithi161 ceph-mon[91579]: pgmap v10629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:25 smithi188 ceph-mon[92425]: pgmap v10629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:27 smithi161 ceph-mon[91579]: pgmap v10630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:27 smithi188 ceph-mon[92425]: pgmap v10630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:29 smithi161 ceph-mon[91579]: pgmap v10631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:29 smithi188 ceph-mon[92425]: pgmap v10631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:31 smithi161 ceph-mon[91579]: pgmap v10632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:31 smithi188 ceph-mon[92425]: pgmap v10632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:32.794 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:33:32.790+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:33:33.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:33:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:33:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:33:33.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:33:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:33:33.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:33:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:33:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:33:33.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:33:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:33:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:33:33.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:33:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:33:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:33:33.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:33:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:33:33.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:33:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:33:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:33:33.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:33:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:33:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:33:33.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:33:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:33:33.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:33:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:33:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:33:33.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:33:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:33:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:33:33.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:33:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:33:33.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:33:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:33:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:33:33.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:33:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:33:33.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:33:33.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:33:33.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:33:33.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:33:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:33:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:33:33.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:33:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=607] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:33:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=607] Local address: 172.21.15.161, port 56832 2022-07-03T16:33:33.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=607] Peer address: 172.21.15.101, port 22 2022-07-03T16:33:33.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=607] Connection lost 2022-07-03T16:33:33.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=607] Aborting connection 2022-07-03T16:33:33.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:33:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:33:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:33:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:33:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:33:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:33:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:33:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:33:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:33:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:33:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:33:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:33:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:33:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:33:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:33:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:33:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:33:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:33:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:33:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:33:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:33:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:33:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:33:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:33:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:33:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:33:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:33:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:33:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:33:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:33:33.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:33:33.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:33:33.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:33:33.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:33:33.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:33:33.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:33:33.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:33:33.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:33:33.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:33:33.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:33:33.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:33:33.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:33:33.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:33:33.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:33:33.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:33:33.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:33:33.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:33:33.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:33:33.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:33:33.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:33:33.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:33:33.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=607] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:33:33.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=607] Local address: 172.21.15.161, port 56832 2022-07-03T16:33:33.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=607] Peer address: 172.21.15.101, port 22 2022-07-03T16:33:33.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=607] Connection lost 2022-07-03T16:33:33.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=607] Aborting connection 2022-07-03T16:33:33.915 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:33 smithi161 ceph-mon[91579]: pgmap v10633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:33:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:33:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:33:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:33:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:33:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:33:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:33:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:33:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:33:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:33:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:33:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:33:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:33:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:33:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:33:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:33:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:33:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:33:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:33:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:33:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:33:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:33:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:33:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:33:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:33:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:33:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:33:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:33:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:33:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:33:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:33:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:33:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:33:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:33:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:33:33.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:33:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:33:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:33:33.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:33:33.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:33:33.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:33:33.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:33:33.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:33:33.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:33:33.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:33:33.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:33:33.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:33:33.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:33:33.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:33:33.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=607] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:33:33.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=607] Local address: 172.21.15.161, port 56832 2022-07-03T16:33:33.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=607] Peer address: 172.21.15.101, port 22 2022-07-03T16:33:33.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=607] Connection lost 2022-07-03T16:33:33.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=607] Aborting connection 2022-07-03T16:33:33.942 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:33 smithi188 ceph-mon[92425]: pgmap v10633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:35 smithi161 ceph-mon[91579]: pgmap v10634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:35 smithi188 ceph-mon[92425]: pgmap v10634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:37 smithi161 ceph-mon[91579]: pgmap v10635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:37 smithi188 ceph-mon[92425]: pgmap v10635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:39 smithi161 ceph-mon[91579]: pgmap v10636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:39 smithi188 ceph-mon[92425]: pgmap v10636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:41 smithi161 ceph-mon[91579]: pgmap v10637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:41 smithi188 ceph-mon[92425]: pgmap v10637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:43 smithi161 ceph-mon[91579]: pgmap v10638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:43 smithi188 ceph-mon[92425]: pgmap v10638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:45 smithi161 ceph-mon[91579]: pgmap v10639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:45 smithi188 ceph-mon[92425]: pgmap v10639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:47 smithi161 ceph-mon[91579]: pgmap v10640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:47 smithi188 ceph-mon[92425]: pgmap v10640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:49 smithi161 ceph-mon[91579]: pgmap v10641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:49 smithi188 ceph-mon[92425]: pgmap v10641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:51 smithi161 ceph-mon[91579]: pgmap v10642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:51 smithi188 ceph-mon[92425]: pgmap v10642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:53 smithi161 ceph-mon[91579]: pgmap v10643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:53 smithi188 ceph-mon[92425]: pgmap v10643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:55 smithi161 ceph-mon[91579]: pgmap v10644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:55 smithi188 ceph-mon[92425]: pgmap v10644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:57 smithi161 ceph-mon[91579]: pgmap v10645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:33:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:57 smithi188 ceph-mon[92425]: pgmap v10645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:33:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:33:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:33:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:33:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:33:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:33:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:33:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:33:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:33:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:33:59 smithi161 ceph-mon[91579]: pgmap v10646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:33:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:33:59 smithi188 ceph-mon[92425]: pgmap v10646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:01 smithi161 ceph-mon[91579]: pgmap v10647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:01 smithi188 ceph-mon[92425]: pgmap v10647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:03 smithi161 ceph-mon[91579]: pgmap v10648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:03 smithi188 ceph-mon[92425]: pgmap v10648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:05 smithi161 ceph-mon[91579]: pgmap v10649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:05 smithi188 ceph-mon[92425]: pgmap v10649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:07 smithi161 ceph-mon[91579]: pgmap v10650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:07 smithi188 ceph-mon[92425]: pgmap v10650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:09 smithi161 ceph-mon[91579]: pgmap v10651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:09 smithi188 ceph-mon[92425]: pgmap v10651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:11 smithi161 ceph-mon[91579]: pgmap v10652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:11 smithi188 ceph-mon[92425]: pgmap v10652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:14 smithi161 ceph-mon[91579]: pgmap v10653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:14 smithi188 ceph-mon[92425]: pgmap v10653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:16 smithi161 ceph-mon[91579]: pgmap v10654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:16 smithi188 ceph-mon[92425]: pgmap v10654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:18 smithi161 ceph-mon[91579]: pgmap v10655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:18 smithi188 ceph-mon[92425]: pgmap v10655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:20 smithi161 ceph-mon[91579]: pgmap v10656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:20 smithi188 ceph-mon[92425]: pgmap v10656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:22 smithi161 ceph-mon[91579]: pgmap v10657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:22 smithi188 ceph-mon[92425]: pgmap v10657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:24 smithi161 ceph-mon[91579]: pgmap v10658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:24 smithi188 ceph-mon[92425]: pgmap v10658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:26 smithi161 ceph-mon[91579]: pgmap v10659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:26 smithi188 ceph-mon[92425]: pgmap v10659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:28 smithi161 ceph-mon[91579]: pgmap v10660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:28 smithi188 ceph-mon[92425]: pgmap v10660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:30 smithi161 ceph-mon[91579]: pgmap v10661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:30 smithi188 ceph-mon[92425]: pgmap v10661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:32 smithi161 ceph-mon[91579]: pgmap v10662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:32 smithi188 ceph-mon[92425]: pgmap v10662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:34:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:34:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:34:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:34:33.293+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:34:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:34:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:34:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:34:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:34:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:34:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:34:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:34:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:34:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:34:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:34:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:34:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:34:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:34:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:34:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:34:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:34:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:34:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:34:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:34:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:34:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:34:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:34:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:34:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:34:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:34:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:34:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:34:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:34:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:34:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:34:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:34:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:34:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:34:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:34:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:34:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:34:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:34:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:34:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:34:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:34:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:34:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:34:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:34:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:34:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=609] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:34:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=609] Local address: 172.21.15.161, port 56836 2022-07-03T16:34:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=609] Peer address: 172.21.15.101, port 22 2022-07-03T16:34:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=609] Connection lost 2022-07-03T16:34:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=609] Aborting connection 2022-07-03T16:34:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:34:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:34:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:34:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:34:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:34:35.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: pgmap v10663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:35.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:34:35.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:34:35.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:34:35.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:34:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:34:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:34:35.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:34:35.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:34:35.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:34:35.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:34:35.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:34:35.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:34:35.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:34:35.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:34:35.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:34:35.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:34:35.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:34:35.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:34:35.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:34:35.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:34:35.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:34:35.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:34:35.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:34:35.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:34:35.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:34:35.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:34:35.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:34:35.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:34:35.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:34:35.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:34:35.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:34:35.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:34:35.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:34:35.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:34:35.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:34:35.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:34:35.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:34:35.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:34:35.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:34:35.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:34:35.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:34:35.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:34:35.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:34:35.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:34:35.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:34:35.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:34:35.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=609] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:34:35.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=609] Local address: 172.21.15.161, port 56836 2022-07-03T16:34:35.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=609] Peer address: 172.21.15.101, port 22 2022-07-03T16:34:35.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=609] Connection lost 2022-07-03T16:34:35.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=609] Aborting connection 2022-07-03T16:34:35.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: pgmap v10663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:34:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:34:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:34:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:34:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:34:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:34:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:34:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:34:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:34:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:34:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:34:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:34:35.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:34:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:34:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:34:35.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:34:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:34:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:34:35.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:34:35.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:34:35.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:34:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:34:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:34:35.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:34:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:34:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:34:35.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:34:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:34:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:34:35.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:34:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:34:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:34:35.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:34:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:34:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:34:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:34:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:34:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:34:35.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:34:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:34:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:34:35.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:34:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:34:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:34:35.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:34:35.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:34:35.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=609] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:34:35.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=609] Local address: 172.21.15.161, port 56836 2022-07-03T16:34:35.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=609] Peer address: 172.21.15.101, port 22 2022-07-03T16:34:35.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=609] Connection lost 2022-07-03T16:34:35.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=609] Aborting connection 2022-07-03T16:34:35.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:35.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:34:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:36 smithi161 ceph-mon[91579]: pgmap v10664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:37.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:36 smithi188 ceph-mon[92425]: pgmap v10664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:38 smithi161 ceph-mon[91579]: pgmap v10665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:38 smithi188 ceph-mon[92425]: pgmap v10665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:40 smithi161 ceph-mon[91579]: pgmap v10666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:40 smithi188 ceph-mon[92425]: pgmap v10666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:42 smithi161 ceph-mon[91579]: pgmap v10667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:42 smithi188 ceph-mon[92425]: pgmap v10667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:44 smithi161 ceph-mon[91579]: pgmap v10668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:44 smithi188 ceph-mon[92425]: pgmap v10668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:46 smithi161 ceph-mon[91579]: pgmap v10669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:46 smithi188 ceph-mon[92425]: pgmap v10669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:48 smithi161 ceph-mon[91579]: pgmap v10670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:48 smithi188 ceph-mon[92425]: pgmap v10670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:50 smithi161 ceph-mon[91579]: pgmap v10671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:50 smithi188 ceph-mon[92425]: pgmap v10671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:52 smithi161 ceph-mon[91579]: pgmap v10672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:52 smithi188 ceph-mon[92425]: pgmap v10672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:55.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:54 smithi161 ceph-mon[91579]: pgmap v10673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:54 smithi188 ceph-mon[92425]: pgmap v10673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:34:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:56 smithi161 ceph-mon[91579]: pgmap v10674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:56 smithi188 ceph-mon[92425]: pgmap v10674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:34:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:58 smithi161 ceph-mon[91579]: pgmap v10675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:34:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:34:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:34:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:58 smithi188 ceph-mon[92425]: pgmap v10675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:34:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:34:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:34:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:34:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:34:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:35:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:00 smithi161 ceph-mon[91579]: pgmap v10676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:00 smithi188 ceph-mon[92425]: pgmap v10676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:02 smithi161 ceph-mon[91579]: pgmap v10677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:02 smithi188 ceph-mon[92425]: pgmap v10677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:04 smithi161 ceph-mon[91579]: pgmap v10678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:05.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:04 smithi188 ceph-mon[92425]: pgmap v10678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:06 smithi161 ceph-mon[91579]: pgmap v10679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:06 smithi188 ceph-mon[92425]: pgmap v10679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:08 smithi161 ceph-mon[91579]: pgmap v10680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:08 smithi188 ceph-mon[92425]: pgmap v10680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:10 smithi161 ceph-mon[91579]: pgmap v10681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:10 smithi188 ceph-mon[92425]: pgmap v10681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:12 smithi161 ceph-mon[91579]: pgmap v10682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:12 smithi188 ceph-mon[92425]: pgmap v10682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:15.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:14 smithi161 ceph-mon[91579]: pgmap v10683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:14 smithi188 ceph-mon[92425]: pgmap v10683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:16 smithi161 ceph-mon[91579]: pgmap v10684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:16 smithi188 ceph-mon[92425]: pgmap v10684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:18 smithi161 ceph-mon[91579]: pgmap v10685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:19.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:18 smithi188 ceph-mon[92425]: pgmap v10685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:20 smithi161 ceph-mon[91579]: pgmap v10686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:20 smithi188 ceph-mon[92425]: pgmap v10686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:22 smithi161 ceph-mon[91579]: pgmap v10687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:22 smithi188 ceph-mon[92425]: pgmap v10687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:24 smithi161 ceph-mon[91579]: pgmap v10688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:24 smithi188 ceph-mon[92425]: pgmap v10688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:26 smithi161 ceph-mon[91579]: pgmap v10689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:26 smithi188 ceph-mon[92425]: pgmap v10689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:28 smithi161 ceph-mon[91579]: pgmap v10690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:28 smithi188 ceph-mon[92425]: pgmap v10690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:30 smithi161 ceph-mon[91579]: pgmap v10691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:30 smithi188 ceph-mon[92425]: pgmap v10691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:32 smithi161 ceph-mon[91579]: pgmap v10692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:32 smithi188 ceph-mon[92425]: pgmap v10692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:34.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:35:34.213+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:35:34.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:35:34.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:35:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:35:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:35:34.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:35:34.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:35:34.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:35:34.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:35:34.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:35:34.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:35:34.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:35:34.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:35:34.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:35:34.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:35:34.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:35:34.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:35:34.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:35:34.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:35:34.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:35:34.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:35:34.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:35:34.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:35:34.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:35:34.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:35:34.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:35:34.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:35:34.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:35:34.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:35:34.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:35:34.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:35:34.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:35:34.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:35:34.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:35:34.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:35:34.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:35:34.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:35:34.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:35:34.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:35:34.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:35:34.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:35:34.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:35:34.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:35:34.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:35:34.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:35:34.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=611] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:35:34.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=611] Local address: 172.21.15.161, port 56840 2022-07-03T16:35:34.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=611] Peer address: 172.21.15.101, port 22 2022-07-03T16:35:34.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=611] Connection lost 2022-07-03T16:35:34.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=611] Aborting connection 2022-07-03T16:35:34.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:35:35.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-mon[91579]: pgmap v10693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:35:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:35:35.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:35:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:34 smithi188 ceph-mon[92425]: pgmap v10693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:35:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:35:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:35:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:35:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:35:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:35:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:35:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:35:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:35:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:35:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:35:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:35:36.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:35:36.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:35:36.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:35:36.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:35:36.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:35:36.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:35:36.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:35:36.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:35:36.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:35:36.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:35:36.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:35:36.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:35:36.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:35:36.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:35:36.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:35:36.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:35:36.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:35:36.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:35:36.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:35:36.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:35:36.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:35:36.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:35:36.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:35:36.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:35:36.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:35:36.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:35:36.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:35:36.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:35:36.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:35:36.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:35:36.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:35:36.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:35:36.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:35:36.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:35:36.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:35:36.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:35:36.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:35:36.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=611] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:35:36.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=611] Local address: 172.21.15.161, port 56840 2022-07-03T16:35:36.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=611] Peer address: 172.21.15.101, port 22 2022-07-03T16:35:36.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=611] Connection lost 2022-07-03T16:35:36.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=611] Aborting connection 2022-07-03T16:35:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:35:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:35:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:35:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:35:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:35:36.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:35:36.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:35:36.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:35:36.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:35:36.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:35:36.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:35:36.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:35:36.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:35:36.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:35:36.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:35:36.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:35:36.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:35:36.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:35:36.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:35:36.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:35:36.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:35:36.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:35:36.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:35:36.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:35:36.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:35:36.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:35:36.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:35:36.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:35:36.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:35:36.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:35:36.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:35:36.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:35:36.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:35:36.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:35:36.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:35:36.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:35:36.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:35:36.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:35:36.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:35:36.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:35:36.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:35:36.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:35:36.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:35:36.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:35:36.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:35:36.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:35:36.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=611] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:35:36.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=611] Local address: 172.21.15.161, port 56840 2022-07-03T16:35:36.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=611] Peer address: 172.21.15.101, port 22 2022-07-03T16:35:36.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=611] Connection lost 2022-07-03T16:35:36.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=611] Aborting connection 2022-07-03T16:35:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:36 smithi161 ceph-mon[91579]: pgmap v10694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:36 smithi188 ceph-mon[92425]: pgmap v10694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:38 smithi161 ceph-mon[91579]: pgmap v10695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:38 smithi188 ceph-mon[92425]: pgmap v10695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:40 smithi161 ceph-mon[91579]: pgmap v10696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:40 smithi188 ceph-mon[92425]: pgmap v10696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:42 smithi161 ceph-mon[91579]: pgmap v10697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:42 smithi188 ceph-mon[92425]: pgmap v10697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:44 smithi161 ceph-mon[91579]: pgmap v10698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:44 smithi188 ceph-mon[92425]: pgmap v10698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:46 smithi161 ceph-mon[91579]: pgmap v10699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:46 smithi188 ceph-mon[92425]: pgmap v10699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:48 smithi161 ceph-mon[91579]: pgmap v10700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:48 smithi188 ceph-mon[92425]: pgmap v10700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:50 smithi188 ceph-mon[92425]: pgmap v10701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:50 smithi161 ceph-mon[91579]: pgmap v10701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:52 smithi188 ceph-mon[92425]: pgmap v10702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:53.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:52 smithi161 ceph-mon[91579]: pgmap v10702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:54 smithi188 ceph-mon[92425]: pgmap v10703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:54 smithi161 ceph-mon[91579]: pgmap v10703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:35:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:56 smithi188 ceph-mon[92425]: pgmap v10704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:56 smithi161 ceph-mon[91579]: pgmap v10704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:35:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:58 smithi188 ceph-mon[92425]: pgmap v10705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:35:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:35:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:35:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:35:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:58 smithi161 ceph-mon[91579]: pgmap v10705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:35:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:35:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:35:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:35:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:36:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:00 smithi188 ceph-mon[92425]: pgmap v10706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:00 smithi161 ceph-mon[91579]: pgmap v10706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:02 smithi188 ceph-mon[92425]: pgmap v10707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:02 smithi161 ceph-mon[91579]: pgmap v10707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:04 smithi188 ceph-mon[92425]: pgmap v10708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:04 smithi161 ceph-mon[91579]: pgmap v10708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:06 smithi188 ceph-mon[92425]: pgmap v10709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:06 smithi161 ceph-mon[91579]: pgmap v10709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:08 smithi161 ceph-mon[91579]: pgmap v10710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:08 smithi188 ceph-mon[92425]: pgmap v10710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:10 smithi161 ceph-mon[91579]: pgmap v10711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:10 smithi188 ceph-mon[92425]: pgmap v10711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:12 smithi161 ceph-mon[91579]: pgmap v10712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:12 smithi188 ceph-mon[92425]: pgmap v10712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:14 smithi161 ceph-mon[91579]: pgmap v10713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:14 smithi188 ceph-mon[92425]: pgmap v10713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:16 smithi161 ceph-mon[91579]: pgmap v10714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:16 smithi188 ceph-mon[92425]: pgmap v10714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:18 smithi161 ceph-mon[91579]: pgmap v10715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:18 smithi188 ceph-mon[92425]: pgmap v10715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:20 smithi161 ceph-mon[91579]: pgmap v10716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:20 smithi188 ceph-mon[92425]: pgmap v10716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:22 smithi161 ceph-mon[91579]: pgmap v10717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:22 smithi188 ceph-mon[92425]: pgmap v10717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:24 smithi161 ceph-mon[91579]: pgmap v10718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:24 smithi188 ceph-mon[92425]: pgmap v10718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:26 smithi161 ceph-mon[91579]: pgmap v10719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:26 smithi188 ceph-mon[92425]: pgmap v10719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:28 smithi161 ceph-mon[91579]: pgmap v10720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:28 smithi188 ceph-mon[92425]: pgmap v10720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:30 smithi161 ceph-mon[91579]: pgmap v10721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:30 smithi188 ceph-mon[92425]: pgmap v10721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:32 smithi161 ceph-mon[91579]: pgmap v10722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:32 smithi188 ceph-mon[92425]: pgmap v10722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:34.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:36:34.712+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:36:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:36:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:36:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:36:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:36:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:36:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:36:34.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:36:34.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:36:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:36:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:36:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:36:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:36:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:36:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:36:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:36:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:36:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:36:34.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:36:34.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:36:34.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:36:34.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:36:34.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:36:34.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:36:34.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:36:34.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:36:34.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:36:34.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:36:34.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:36:34.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:36:34.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:36:34.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:36:34.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:36:34.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:36:34.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:36:34.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:36:34.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:36:34.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:36:34.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:36:34.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:36:34.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:36:34.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:36:34.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:36:34.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:36:34.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:36:34.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=613] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:36:34.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=613] Local address: 172.21.15.161, port 56844 2022-07-03T16:36:34.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=613] Peer address: 172.21.15.101, port 22 2022-07-03T16:36:34.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=613] Connection lost 2022-07-03T16:36:34.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=613] Aborting connection 2022-07-03T16:36:34.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:36:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:34 smithi188 ceph-mon[92425]: pgmap v10723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:36:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:36:35.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:36:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-mon[91579]: pgmap v10723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:36:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:36:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:36:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:36:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:36:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:36:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:36:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:36:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:36:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:36:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:36:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:36:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:36:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:36:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:36:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:36:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:36:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:36:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:36:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:36:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:36:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:36:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:36:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:36:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:36:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:36:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:36:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:36:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:36:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:36:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:36:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:36:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:36:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:36:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:36:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:36:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:36:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:36:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:36:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:36:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:36:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:36:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:36:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:36:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:36:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:36:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:36:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:36:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:36:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=613] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:36:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=613] Local address: 172.21.15.161, port 56844 2022-07-03T16:36:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=613] Peer address: 172.21.15.101, port 22 2022-07-03T16:36:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=613] Connection lost 2022-07-03T16:36:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=613] Aborting connection 2022-07-03T16:36:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:36:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:36:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:36:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:36:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:36:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:36:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:36:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:36:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:36:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:36:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:36:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:36:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:36:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:36:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:36:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:36:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:36:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:36:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:36:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:36:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:36:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:36:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:36:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:36:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:36:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:36:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:36:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:36:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:36:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:36:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:36:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:36:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:36:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:36:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:36:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:36:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:36:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:36:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:36:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:36:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:36:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:36:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:36:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:36:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:36:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:36:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=613] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:36:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=613] Local address: 172.21.15.161, port 56844 2022-07-03T16:36:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=613] Peer address: 172.21.15.101, port 22 2022-07-03T16:36:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=613] Connection lost 2022-07-03T16:36:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=613] Aborting connection 2022-07-03T16:36:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:36 smithi161 ceph-mon[91579]: pgmap v10724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:36 smithi188 ceph-mon[92425]: pgmap v10724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:38 smithi161 ceph-mon[91579]: pgmap v10725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:38 smithi188 ceph-mon[92425]: pgmap v10725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:40 smithi161 ceph-mon[91579]: pgmap v10726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:40 smithi188 ceph-mon[92425]: pgmap v10726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:42 smithi161 ceph-mon[91579]: pgmap v10727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:42 smithi188 ceph-mon[92425]: pgmap v10727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:44 smithi161 ceph-mon[91579]: pgmap v10728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:44 smithi188 ceph-mon[92425]: pgmap v10728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:46 smithi161 ceph-mon[91579]: pgmap v10729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:46 smithi188 ceph-mon[92425]: pgmap v10729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:49 smithi161 ceph-mon[91579]: pgmap v10730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:49 smithi188 ceph-mon[92425]: pgmap v10730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:51 smithi161 ceph-mon[91579]: pgmap v10731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:51 smithi188 ceph-mon[92425]: pgmap v10731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:53 smithi161 ceph-mon[91579]: pgmap v10732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:53 smithi188 ceph-mon[92425]: pgmap v10732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:55 smithi161 ceph-mon[91579]: pgmap v10733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:55 smithi188 ceph-mon[92425]: pgmap v10733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:36:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:57 smithi161 ceph-mon[91579]: pgmap v10734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:57 smithi188 ceph-mon[92425]: pgmap v10734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:36:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:59 smithi161 ceph-mon[91579]: pgmap v10735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:36:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:36:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:36:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:36:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:59 smithi188 ceph-mon[92425]: pgmap v10735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:36:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:36:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:36:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:36:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:37:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:01 smithi161 ceph-mon[91579]: pgmap v10736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:01 smithi188 ceph-mon[92425]: pgmap v10736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:03 smithi161 ceph-mon[91579]: pgmap v10737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:03 smithi188 ceph-mon[92425]: pgmap v10737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:05 smithi161 ceph-mon[91579]: pgmap v10738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:05 smithi188 ceph-mon[92425]: pgmap v10738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:07 smithi161 ceph-mon[91579]: pgmap v10739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:07 smithi188 ceph-mon[92425]: pgmap v10739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:09 smithi161 ceph-mon[91579]: pgmap v10740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:09 smithi188 ceph-mon[92425]: pgmap v10740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:11 smithi161 ceph-mon[91579]: pgmap v10741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:11 smithi188 ceph-mon[92425]: pgmap v10741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:13 smithi161 ceph-mon[91579]: pgmap v10742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:13 smithi188 ceph-mon[92425]: pgmap v10742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:15 smithi161 ceph-mon[91579]: pgmap v10743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:15 smithi188 ceph-mon[92425]: pgmap v10743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:17 smithi161 ceph-mon[91579]: pgmap v10744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:17 smithi188 ceph-mon[92425]: pgmap v10744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:18.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:19 smithi161 ceph-mon[91579]: pgmap v10745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:19 smithi188 ceph-mon[92425]: pgmap v10745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:21 smithi161 ceph-mon[91579]: pgmap v10746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:21 smithi188 ceph-mon[92425]: pgmap v10746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:23 smithi161 ceph-mon[91579]: pgmap v10747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:23 smithi188 ceph-mon[92425]: pgmap v10747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:25 smithi161 ceph-mon[91579]: pgmap v10748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:25 smithi188 ceph-mon[92425]: pgmap v10748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:27 smithi161 ceph-mon[91579]: pgmap v10749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:27 smithi188 ceph-mon[92425]: pgmap v10749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:29 smithi161 ceph-mon[91579]: pgmap v10750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:29 smithi188 ceph-mon[92425]: pgmap v10750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:31 smithi161 ceph-mon[91579]: pgmap v10751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:31 smithi188 ceph-mon[92425]: pgmap v10751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:33 smithi161 ceph-mon[91579]: pgmap v10752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:33 smithi188 ceph-mon[92425]: pgmap v10752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:33.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-mon[91579]: pgmap v10753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:37:35.218+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:37:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:37:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:37:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:37:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:37:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:37:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:37:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:37:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:37:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:37:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:37:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:37:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:37:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:37:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:37:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:37:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:37:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:37:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:37:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:37:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:37:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:37:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:37:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:37:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:37:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:37:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:37:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:37:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:37:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:37:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:37:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:37:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:37:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:37:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:37:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:37:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:37:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:37:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:37:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:37:35.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:37:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:37:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:37:35.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:37:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:37:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=615] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:37:35.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=615] Local address: 172.21.15.161, port 56848 2022-07-03T16:37:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=615] Peer address: 172.21.15.101, port 22 2022-07-03T16:37:35.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=615] Connection lost 2022-07-03T16:37:35.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=615] Aborting connection 2022-07-03T16:37:35.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:37:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:37:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:35 smithi188 ceph-mon[92425]: pgmap v10753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:37:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:37:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:37:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:37:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:37:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:37:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:37:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:37:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:37:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:37:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:37:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:37:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:37:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:37:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:37:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:37:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:37:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:37:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:37:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:37:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:37:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:37:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:37:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:37:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:37:37.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:37:37.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:37:37.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:37:37.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:37:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:37:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:37:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:37:37.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:37:37.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:37:37.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:37:37.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:37:37.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:37:37.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:37:37.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:37:37.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:37:37.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:37:37.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:37:37.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:37:37.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:37:37.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:37:37.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:37:37.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:37:37.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:37:37.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:37:37.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:37:37.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:37:37.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:37:37.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:37:37.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=615] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:37:37.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=615] Local address: 172.21.15.161, port 56848 2022-07-03T16:37:37.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=615] Peer address: 172.21.15.101, port 22 2022-07-03T16:37:37.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=615] Connection lost 2022-07-03T16:37:37.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=615] Aborting connection 2022-07-03T16:37:37.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:37 smithi161 ceph-mon[91579]: pgmap v10754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:37.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:37:37.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:37:37.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:37:37.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:37:37.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:37:37.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:37:37.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:37:37.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:37:37.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:37:37.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:37:37.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:37:37.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:37:37.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:37:37.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:37:37.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:37:37.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:37:37.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:37:37.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:37:37.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:37:37.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:37:37.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:37:37.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:37:37.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:37:37.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:37:37.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:37:37.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:37:37.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:37:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:37:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:37:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:37:37.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:37:37.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:37:37.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:37:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:37:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:37:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:37:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:37:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:37:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:37:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:37:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:37:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:37:37.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:37:37.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:37:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:37:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=615] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:37:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=615] Local address: 172.21.15.161, port 56848 2022-07-03T16:37:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=615] Peer address: 172.21.15.101, port 22 2022-07-03T16:37:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=615] Connection lost 2022-07-03T16:37:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=615] Aborting connection 2022-07-03T16:37:37.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:37 smithi188 ceph-mon[92425]: pgmap v10754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:37:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: pgmap v10755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: pgmap v10755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:37:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 110.7M 2022-07-03T16:37:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 116114333: error parsing value: Value '116114333' is below minimum 939524096 2022-07-03T16:37:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:37:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:37:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 110.7M 2022-07-03T16:37:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 116114333: error parsing value: Value '116114333' is below minimum 939524096 2022-07-03T16:37:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:37:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:37:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:41 smithi161 ceph-mon[91579]: pgmap v10756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:41 smithi188 ceph-mon[92425]: pgmap v10756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:43 smithi161 ceph-mon[91579]: pgmap v10757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:43 smithi188 ceph-mon[92425]: pgmap v10757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:45 smithi161 ceph-mon[91579]: pgmap v10758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:45 smithi188 ceph-mon[92425]: pgmap v10758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:47 smithi161 ceph-mon[91579]: pgmap v10759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:47 smithi188 ceph-mon[92425]: pgmap v10759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:49 smithi161 ceph-mon[91579]: pgmap v10760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:49 smithi188 ceph-mon[92425]: pgmap v10760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:51 smithi161 ceph-mon[91579]: pgmap v10761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:51 smithi188 ceph-mon[92425]: pgmap v10761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:53 smithi188 ceph-mon[92425]: pgmap v10762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:53 smithi161 ceph-mon[91579]: pgmap v10762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:55 smithi188 ceph-mon[92425]: pgmap v10763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:55 smithi161 ceph-mon[91579]: pgmap v10763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:37:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:57 smithi188 ceph-mon[92425]: pgmap v10764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:57 smithi161 ceph-mon[91579]: pgmap v10764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:37:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:59 smithi188 ceph-mon[92425]: pgmap v10765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:37:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:37:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:37:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:37:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:59 smithi161 ceph-mon[91579]: pgmap v10765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:37:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:37:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:38:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:01 smithi188 ceph-mon[92425]: pgmap v10766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:01 smithi161 ceph-mon[91579]: pgmap v10766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:03 smithi188 ceph-mon[92425]: pgmap v10767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:03 smithi161 ceph-mon[91579]: pgmap v10767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:05 smithi188 ceph-mon[92425]: pgmap v10768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:05 smithi161 ceph-mon[91579]: pgmap v10768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:07 smithi188 ceph-mon[92425]: pgmap v10769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:07 smithi161 ceph-mon[91579]: pgmap v10769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:09 smithi161 ceph-mon[91579]: pgmap v10770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:09 smithi188 ceph-mon[92425]: pgmap v10770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:11 smithi161 ceph-mon[91579]: pgmap v10771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:11 smithi188 ceph-mon[92425]: pgmap v10771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:13 smithi161 ceph-mon[91579]: pgmap v10772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:13 smithi188 ceph-mon[92425]: pgmap v10772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:15 smithi161 ceph-mon[91579]: pgmap v10773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:15 smithi188 ceph-mon[92425]: pgmap v10773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:17 smithi161 ceph-mon[91579]: pgmap v10774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:17 smithi188 ceph-mon[92425]: pgmap v10774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:19 smithi161 ceph-mon[91579]: pgmap v10775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:19 smithi188 ceph-mon[92425]: pgmap v10775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:21 smithi161 ceph-mon[91579]: pgmap v10776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:21 smithi188 ceph-mon[92425]: pgmap v10776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:23 smithi161 ceph-mon[91579]: pgmap v10777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:23 smithi188 ceph-mon[92425]: pgmap v10777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:25 smithi161 ceph-mon[91579]: pgmap v10778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:25 smithi188 ceph-mon[92425]: pgmap v10778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:27 smithi161 ceph-mon[91579]: pgmap v10779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:27 smithi188 ceph-mon[92425]: pgmap v10779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:29 smithi161 ceph-mon[91579]: pgmap v10780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:29 smithi188 ceph-mon[92425]: pgmap v10780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:31 smithi161 ceph-mon[91579]: pgmap v10781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:31 smithi188 ceph-mon[92425]: pgmap v10781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:32 smithi161 ceph-mon[91579]: pgmap v10782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:32 smithi188 ceph-mon[92425]: pgmap v10782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:34 smithi161 ceph-mon[91579]: pgmap v10783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:34 smithi188 ceph-mon[92425]: pgmap v10783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:36 smithi161 ceph-mon[91579]: pgmap v10784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:36 smithi188 ceph-mon[92425]: pgmap v10784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-mon[91579]: pgmap v10785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:38:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:38:38.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:38:38.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:38:38.479+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:38:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:38:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:38:38.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:38:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:38:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:38:38.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:38:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:38:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:38:38.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:38:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:38:38.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:38:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:38:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:38:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:38:38.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:38:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:38:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:38:38.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:38:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:38:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:38:38.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:38:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:38:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:38:38.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:38:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:38:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:38:38.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:38:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:38:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:38:38.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:38:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:38:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:38:38.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:38:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:38:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:38:38.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:38:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:38:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:38:38.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:38:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:38:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:38:38.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:38:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:38:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:38:38.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=617] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:38:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=617] Local address: 172.21.15.161, port 56870 2022-07-03T16:38:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=617] Peer address: 172.21.15.101, port 22 2022-07-03T16:38:38.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=617] Connection lost 2022-07-03T16:38:38.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=617] Aborting connection 2022-07-03T16:38:38.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:38:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:38:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:38 smithi188 ceph-mon[92425]: pgmap v10785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:38:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:38:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:38:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:38:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:38:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:38:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:38:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:38:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:38:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:38:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:38:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:38:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:38:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:38:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:38:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:38:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:38:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:38:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:38:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:38:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:38:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:38:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:38:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:38:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:38:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:38:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:38:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:38:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:38:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:38:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:38:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:38:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:38:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:38:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:38:39.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:38:39.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:38:39.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:38:39.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:38:39.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:38:39.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:38:39.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:38:39.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:38:39.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:38:39.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:38:39.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:38:39.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:38:39.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:38:39.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:38:39.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=617] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:38:39.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=617] Local address: 172.21.15.161, port 56870 2022-07-03T16:38:39.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=617] Peer address: 172.21.15.101, port 22 2022-07-03T16:38:39.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=617] Connection lost 2022-07-03T16:38:39.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=617] Aborting connection 2022-07-03T16:38:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:38:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:38:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:38:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:38:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:38:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:38:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:38:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:38:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:38:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:38:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:38:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:38:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:38:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:38:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:38:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:38:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:38:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:38:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:38:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:38:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:38:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:38:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:38:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:38:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:38:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:38:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:38:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:38:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:38:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:38:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:38:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:38:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:38:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:38:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:38:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:38:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:38:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:38:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:38:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:38:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:38:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:38:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:38:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:38:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:38:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:38:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=617] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:38:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=617] Local address: 172.21.15.161, port 56870 2022-07-03T16:38:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=617] Peer address: 172.21.15.101, port 22 2022-07-03T16:38:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=617] Connection lost 2022-07-03T16:38:39.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=617] Aborting connection 2022-07-03T16:38:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:40 smithi161 ceph-mon[91579]: pgmap v10786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:40.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:40 smithi188 ceph-mon[92425]: pgmap v10786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:42 smithi161 ceph-mon[91579]: pgmap v10787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:42 smithi188 ceph-mon[92425]: pgmap v10787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:44 smithi161 ceph-mon[91579]: pgmap v10788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:44 smithi188 ceph-mon[92425]: pgmap v10788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:46 smithi161 ceph-mon[91579]: pgmap v10789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:46.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:46.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:46 smithi188 ceph-mon[92425]: pgmap v10789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:48 smithi161 ceph-mon[91579]: pgmap v10790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:48 smithi188 ceph-mon[92425]: pgmap v10790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:50 smithi161 ceph-mon[91579]: pgmap v10791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:50 smithi188 ceph-mon[92425]: pgmap v10791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:52 smithi161 ceph-mon[91579]: pgmap v10792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:52 smithi188 ceph-mon[92425]: pgmap v10792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:54 smithi161 ceph-mon[91579]: pgmap v10793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:54 smithi188 ceph-mon[92425]: pgmap v10793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:38:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:56 smithi161 ceph-mon[91579]: pgmap v10794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:56 smithi188 ceph-mon[92425]: pgmap v10794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:38:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:38:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:58 smithi161 ceph-mon[91579]: pgmap v10795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:38:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:38:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:38:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:38:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:58 smithi188 ceph-mon[92425]: pgmap v10795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:38:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:38:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:38:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:38:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:39:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:00 smithi161 ceph-mon[91579]: pgmap v10796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:00 smithi188 ceph-mon[92425]: pgmap v10796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:02 smithi161 ceph-mon[91579]: pgmap v10797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:02 smithi188 ceph-mon[92425]: pgmap v10797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:04 smithi161 ceph-mon[91579]: pgmap v10798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:04 smithi188 ceph-mon[92425]: pgmap v10798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:06 smithi161 ceph-mon[91579]: pgmap v10799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:06 smithi188 ceph-mon[92425]: pgmap v10799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:08 smithi161 ceph-mon[91579]: pgmap v10800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:08 smithi188 ceph-mon[92425]: pgmap v10800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:10 smithi161 ceph-mon[91579]: pgmap v10801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:10 smithi188 ceph-mon[92425]: pgmap v10801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:12 smithi161 ceph-mon[91579]: pgmap v10802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:12 smithi188 ceph-mon[92425]: pgmap v10802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:14 smithi161 ceph-mon[91579]: pgmap v10803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:14 smithi188 ceph-mon[92425]: pgmap v10803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:16 smithi161 ceph-mon[91579]: pgmap v10804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:16.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:16 smithi188 ceph-mon[92425]: pgmap v10804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:18 smithi161 ceph-mon[91579]: pgmap v10805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:18 smithi188 ceph-mon[92425]: pgmap v10805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:20 smithi161 ceph-mon[91579]: pgmap v10806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:20 smithi188 ceph-mon[92425]: pgmap v10806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:22 smithi161 ceph-mon[91579]: pgmap v10807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:22 smithi188 ceph-mon[92425]: pgmap v10807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:24 smithi161 ceph-mon[91579]: pgmap v10808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:24 smithi188 ceph-mon[92425]: pgmap v10808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:26 smithi161 ceph-mon[91579]: pgmap v10809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:26.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:26 smithi188 ceph-mon[92425]: pgmap v10809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:28 smithi188 ceph-mon[92425]: pgmap v10810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:28 smithi161 ceph-mon[91579]: pgmap v10810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:30 smithi188 ceph-mon[92425]: pgmap v10811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:30 smithi161 ceph-mon[91579]: pgmap v10811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:32 smithi188 ceph-mon[92425]: pgmap v10812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:32 smithi161 ceph-mon[91579]: pgmap v10812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:34 smithi188 ceph-mon[92425]: pgmap v10813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:34 smithi161 ceph-mon[91579]: pgmap v10813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:36 smithi188 ceph-mon[92425]: pgmap v10814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:36 smithi161 ceph-mon[91579]: pgmap v10814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:38 smithi188 ceph-mon[92425]: pgmap v10815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:38.984 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-mon[91579]: pgmap v10815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:39:38.981+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:39:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:39:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:39:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:39:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:39:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:39:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:39:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:39:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:39:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:39:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:39:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:39:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:39:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:39:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:39:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:39:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:39:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:39:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:39:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:39:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:39:39.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:39:39.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:39:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:39:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:39:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:39:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:39:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:39:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:39:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:39:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:39:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:39:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:39:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:39:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:39:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:39:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:39:39.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:39:39.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:39:39.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:39:39.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:39:39.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:39:39.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:39:39.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:39:39.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:39:39.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=619] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:39:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=619] Local address: 172.21.15.161, port 56924 2022-07-03T16:39:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=619] Peer address: 172.21.15.101, port 22 2022-07-03T16:39:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=619] Connection lost 2022-07-03T16:39:39.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=619] Aborting connection 2022-07-03T16:39:39.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:39:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:39:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:39:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:39:40.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:39:40.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:39:40.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:39:40.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:39:40.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:39:40.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:39:40.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:39:40.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:39:40.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:39:40.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:39:40.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:39:40.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:39:40.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:39:40.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:39:40.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:39:40.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:39:40.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:39:40.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:39:40.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:39:40.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:39:40.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:39:40.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:39:40.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:39:40.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:39:40.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:39:40.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:39:40.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:39:40.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:39:40.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:39:40.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:39:40.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:39:40.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:39:40.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:39:40.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:39:40.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:39:40.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:39:40.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:39:40.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:39:40.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:39:40.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:39:40.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:39:40.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:39:40.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:39:40.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:39:40.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:39:40.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=619] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:39:40.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=619] Local address: 172.21.15.161, port 56924 2022-07-03T16:39:40.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=619] Peer address: 172.21.15.101, port 22 2022-07-03T16:39:40.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=619] Connection lost 2022-07-03T16:39:40.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=619] Aborting connection 2022-07-03T16:39:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:39:40.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:39:40.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:39:40.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:39:40.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:39:40.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:39:40.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:39:40.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:39:40.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:39:40.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:39:40.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:39:40.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:39:40.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:39:40.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:39:40.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:39:40.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:39:40.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:39:40.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:39:40.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:39:40.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:39:40.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:39:40.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:39:40.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:39:40.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:39:40.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:39:40.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:39:40.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:39:40.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:39:40.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:39:40.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:39:40.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:39:40.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:39:40.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:39:40.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:39:40.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:39:40.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:39:40.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:39:40.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:39:40.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:39:40.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:39:40.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:39:40.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:39:40.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:39:40.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:39:40.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:39:40.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:39:40.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:39:40.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:39:40.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:39:40.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=619] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:39:40.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=619] Local address: 172.21.15.161, port 56924 2022-07-03T16:39:40.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=619] Peer address: 172.21.15.101, port 22 2022-07-03T16:39:40.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=619] Connection lost 2022-07-03T16:39:40.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=619] Aborting connection 2022-07-03T16:39:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:40 smithi161 ceph-mon[91579]: pgmap v10816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:40 smithi188 ceph-mon[92425]: pgmap v10816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:42 smithi161 ceph-mon[91579]: pgmap v10817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:42 smithi188 ceph-mon[92425]: pgmap v10817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:44 smithi161 ceph-mon[91579]: pgmap v10818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:44 smithi188 ceph-mon[92425]: pgmap v10818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:46 smithi161 ceph-mon[91579]: pgmap v10819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:46 smithi188 ceph-mon[92425]: pgmap v10819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:48 smithi161 ceph-mon[91579]: pgmap v10820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:48 smithi188 ceph-mon[92425]: pgmap v10820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:50 smithi161 ceph-mon[91579]: pgmap v10821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:50 smithi188 ceph-mon[92425]: pgmap v10821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:52 smithi161 ceph-mon[91579]: pgmap v10822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:52 smithi188 ceph-mon[92425]: pgmap v10822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:54 smithi161 ceph-mon[91579]: pgmap v10823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:55.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:54 smithi188 ceph-mon[92425]: pgmap v10823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:39:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:56 smithi161 ceph-mon[91579]: pgmap v10824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:56 smithi188 ceph-mon[92425]: pgmap v10824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:39:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:39:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:58 smithi161 ceph-mon[91579]: pgmap v10825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:39:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:39:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:39:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:39:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:58 smithi188 ceph-mon[92425]: pgmap v10825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:39:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:39:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:39:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:39:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:40:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:00 smithi161 ceph-mon[91579]: pgmap v10826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:40:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:00 smithi188 ceph-mon[92425]: pgmap v10826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:40:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:02 smithi161 ceph-mon[91579]: pgmap v10827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:02 smithi188 ceph-mon[92425]: pgmap v10827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:04 smithi161 ceph-mon[91579]: pgmap v10828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:04 smithi188 ceph-mon[92425]: pgmap v10828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:06 smithi161 ceph-mon[91579]: pgmap v10829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:06 smithi188 ceph-mon[92425]: pgmap v10829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:09.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:08 smithi161 ceph-mon[91579]: pgmap v10830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:08 smithi188 ceph-mon[92425]: pgmap v10830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:10 smithi161 ceph-mon[91579]: pgmap v10831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:10 smithi188 ceph-mon[92425]: pgmap v10831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:12.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:12 smithi161 ceph-mon[91579]: pgmap v10832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:12 smithi188 ceph-mon[92425]: pgmap v10832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:14 smithi161 ceph-mon[91579]: pgmap v10833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:15.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:14 smithi188 ceph-mon[92425]: pgmap v10833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:16 smithi161 ceph-mon[91579]: pgmap v10834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:16 smithi188 ceph-mon[92425]: pgmap v10834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:17.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:18 smithi161 ceph-mon[91579]: pgmap v10835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:18 smithi188 ceph-mon[92425]: pgmap v10835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:20 smithi161 ceph-mon[91579]: pgmap v10836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:20 smithi188 ceph-mon[92425]: pgmap v10836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:22 smithi161 ceph-mon[91579]: pgmap v10837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:22 smithi188 ceph-mon[92425]: pgmap v10837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:24 smithi161 ceph-mon[91579]: pgmap v10838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:24 smithi188 ceph-mon[92425]: pgmap v10838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:26 smithi161 ceph-mon[91579]: pgmap v10839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:26 smithi188 ceph-mon[92425]: pgmap v10839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:28 smithi161 ceph-mon[91579]: pgmap v10840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:28 smithi188 ceph-mon[92425]: pgmap v10840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:30 smithi161 ceph-mon[91579]: pgmap v10841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:30 smithi188 ceph-mon[92425]: pgmap v10841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:32 smithi161 ceph-mon[91579]: pgmap v10842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:32 smithi188 ceph-mon[92425]: pgmap v10842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:34 smithi161 ceph-mon[91579]: pgmap v10843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:34 smithi188 ceph-mon[92425]: pgmap v10843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:36 smithi161 ceph-mon[91579]: pgmap v10844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:36 smithi188 ceph-mon[92425]: pgmap v10844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:38.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:39.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:38 smithi161 ceph-mon[91579]: pgmap v10845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:38 smithi188 ceph-mon[92425]: pgmap v10845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:39.813 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:40:39.481+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:40:39.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:40:39.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:40:39.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:40:39.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:40:39.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:40:39.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:40:39.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:40:39.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:40:39.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:40:39.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:40:39.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:40:39.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:40:39.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:40:39.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:40:39.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:40:39.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:40:39.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:40:39.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:40:39.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:40:39.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:40:39.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:40:39.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:40:39.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:40:39.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:40:39.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:40:39.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:40:39.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:40:39.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:40:39.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:40:39.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:40:39.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:40:39.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:40:39.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:40:39.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:40:39.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:40:39.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:40:39.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:40:39.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:40:39.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:40:39.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:40:39.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:40:39.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:40:39.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:40:39.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:40:39.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=621] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:40:39.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=621] Local address: 172.21.15.161, port 56952 2022-07-03T16:40:39.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=621] Peer address: 172.21.15.101, port 22 2022-07-03T16:40:39.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=621] Connection lost 2022-07-03T16:40:39.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=621] Aborting connection 2022-07-03T16:40:39.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:40:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:40:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:40:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:40:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:40:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:40:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:40:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:40 smithi161 ceph-mon[91579]: pgmap v10846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:40:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:40:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:40:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:40:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:40:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:40:41.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:40:41.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:40:41.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:40:41.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:40:41.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:40:41.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:40:41.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:40:41.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:40:41.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:40:41.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:40:41.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:40:41.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:40:41.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:40:41.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:40:41.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:40:41.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:40:41.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:40:41.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:40:41.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:40:41.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:40:41.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:40:41.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:40:41.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:40:41.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:40:41.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:40:41.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:40:41.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:40:41.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:40:41.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:40:41.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:40:41.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:40:41.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:40:41.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:40:41.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:40:41.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:40:41.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:40:41.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:40:41.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:40:41.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:40:41.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:40:41.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=621] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:40:41.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=621] Local address: 172.21.15.161, port 56952 2022-07-03T16:40:41.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=621] Peer address: 172.21.15.101, port 22 2022-07-03T16:40:41.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=621] Connection lost 2022-07-03T16:40:41.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=621] Aborting connection 2022-07-03T16:40:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:40 smithi188 ceph-mon[92425]: pgmap v10846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:40:41.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:40:41.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:40:41.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:40:41.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:40:41.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:40:41.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:40:41.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:40:41.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:40:41.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:40:41.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:40:41.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:40:41.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:40:41.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:40:41.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:40:41.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:40:41.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:40:41.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:40:41.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:40:41.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:40:41.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:40:41.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:40:41.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:40:41.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:40:41.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:40:41.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:40:41.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:40:41.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:40:41.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:40:41.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:40:41.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:40:41.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:40:41.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:40:41.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:40:41.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:40:41.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:40:41.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:40:41.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:40:41.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:40:41.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:40:41.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:40:41.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:40:41.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:40:41.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:40:41.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:40:41.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:40:41.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=621] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:40:41.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=621] Local address: 172.21.15.161, port 56952 2022-07-03T16:40:41.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=621] Peer address: 172.21.15.101, port 22 2022-07-03T16:40:41.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=621] Connection lost 2022-07-03T16:40:41.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=621] Aborting connection 2022-07-03T16:40:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:42 smithi161 ceph-mon[91579]: pgmap v10847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:40:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:40:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:42 smithi188 ceph-mon[92425]: pgmap v10847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:40:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:40:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:43 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T16:40:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:43 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T16:40:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:44 smithi161 ceph-mon[91579]: pgmap v10848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:44 smithi188 ceph-mon[92425]: pgmap v10848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:46 smithi161 ceph-mon[91579]: pgmap v10849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:46 smithi188 ceph-mon[92425]: pgmap v10849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:48 smithi161 ceph-mon[91579]: pgmap v10850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:48 smithi188 ceph-mon[92425]: pgmap v10850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:50 smithi161 ceph-mon[91579]: pgmap v10851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:50 smithi188 ceph-mon[92425]: pgmap v10851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:52 smithi161 ceph-mon[91579]: pgmap v10852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:52 smithi188 ceph-mon[92425]: pgmap v10852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:40:53.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:54 smithi161 ceph-mon[91579]: pgmap v10853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:54 smithi188 ceph-mon[92425]: pgmap v10853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:40:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:56 smithi161 ceph-mon[91579]: pgmap v10854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:40:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:56 smithi188 ceph-mon[92425]: pgmap v10854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:40:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:40:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:40:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:58 smithi161 ceph-mon[91579]: pgmap v10855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:40:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:40:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:40:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:40:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:40:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:58 smithi188 ceph-mon[92425]: pgmap v10855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:40:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:40:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:40:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:40:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:41:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:00 smithi161 ceph-mon[91579]: pgmap v10856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:00 smithi188 ceph-mon[92425]: pgmap v10856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:02 smithi161 ceph-mon[91579]: pgmap v10857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:41:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:02 smithi188 ceph-mon[92425]: pgmap v10857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:41:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:04 smithi161 ceph-mon[91579]: pgmap v10858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:04 smithi188 ceph-mon[92425]: pgmap v10858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:06 smithi161 ceph-mon[91579]: pgmap v10859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:06 smithi188 ceph-mon[92425]: pgmap v10859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:08 smithi188 ceph-mon[92425]: pgmap v10860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:08 smithi161 ceph-mon[91579]: pgmap v10860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:10 smithi188 ceph-mon[92425]: pgmap v10861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:10 smithi161 ceph-mon[91579]: pgmap v10861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:12 smithi188 ceph-mon[92425]: pgmap v10862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:12 smithi161 ceph-mon[91579]: pgmap v10862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:14 smithi188 ceph-mon[92425]: pgmap v10863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:14 smithi161 ceph-mon[91579]: pgmap v10863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:16 smithi188 ceph-mon[92425]: pgmap v10864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:16 smithi161 ceph-mon[91579]: pgmap v10864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:18.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:18 smithi188 ceph-mon[92425]: pgmap v10865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:18 smithi161 ceph-mon[91579]: pgmap v10865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:20 smithi188 ceph-mon[92425]: pgmap v10866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:20 smithi161 ceph-mon[91579]: pgmap v10866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:22 smithi161 ceph-mon[91579]: pgmap v10867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:22 smithi188 ceph-mon[92425]: pgmap v10867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:24 smithi161 ceph-mon[91579]: pgmap v10868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:24 smithi188 ceph-mon[92425]: pgmap v10868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:26 smithi161 ceph-mon[91579]: pgmap v10869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:26 smithi188 ceph-mon[92425]: pgmap v10869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:28 smithi161 ceph-mon[91579]: pgmap v10870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:28 smithi188 ceph-mon[92425]: pgmap v10870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:30 smithi161 ceph-mon[91579]: pgmap v10871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:30 smithi188 ceph-mon[92425]: pgmap v10871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:32 smithi161 ceph-mon[91579]: pgmap v10872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:32 smithi188 ceph-mon[92425]: pgmap v10872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:34 smithi161 ceph-mon[91579]: pgmap v10873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:34 smithi188 ceph-mon[92425]: pgmap v10873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:36 smithi161 ceph-mon[91579]: pgmap v10874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:36 smithi188 ceph-mon[92425]: pgmap v10874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:38 smithi161 ceph-mon[91579]: pgmap v10875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:38 smithi188 ceph-mon[92425]: pgmap v10875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:40 smithi161 ceph-mon[91579]: pgmap v10876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:40 smithi188 ceph-mon[92425]: pgmap v10876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-mon[91579]: pgmap v10877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:41:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:41:43.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:41:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:41:42.772+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:41:43.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:41:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:41:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:41:43.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:41:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:41:43.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:41:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:41:43.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:41:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:41:43.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:41:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:41:43.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:41:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:41:43.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:41:43.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:41:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:41:43.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:41:43.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:41:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:41:43.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:41:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:41:43.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:41:43.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:41:43.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:41:43.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:41:43.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:41:43.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:41:43.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:41:43.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:41:43.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:41:43.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:41:43.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:41:43.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:41:43.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:41:43.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:41:43.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:41:43.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:41:43.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:41:43.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:41:43.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:41:43.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:41:43.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:41:43.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:41:43.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:41:43.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=623] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:41:43.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=623] Local address: 172.21.15.161, port 56982 2022-07-03T16:41:43.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=623] Peer address: 172.21.15.101, port 22 2022-07-03T16:41:43.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=623] Connection lost 2022-07-03T16:41:43.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=623] Aborting connection 2022-07-03T16:41:43.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:41:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:41:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:42 smithi188 ceph-mon[92425]: pgmap v10877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:41:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:41:43.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:41:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:41:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:41:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:41:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:41:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:41:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:41:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:41:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:41:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:41:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:41:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:41:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:41:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:41:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:41:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:41:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:41:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:41:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:41:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:41:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:41:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:41:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:41:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:41:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:41:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:41:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:41:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:41:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:41:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:41:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:41:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:41:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:41:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:41:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:41:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:41:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:41:44.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:41:44.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:41:44.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:41:44.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:41:44.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:41:44.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:41:44.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:41:44.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:41:44.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:41:44.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:41:44.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=623] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:41:44.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=623] Local address: 172.21.15.161, port 56982 2022-07-03T16:41:44.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=623] Peer address: 172.21.15.101, port 22 2022-07-03T16:41:44.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=623] Connection lost 2022-07-03T16:41:44.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=623] Aborting connection 2022-07-03T16:41:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:41:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:41:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:41:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:41:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:41:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:41:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:41:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:41:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:41:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:41:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:41:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:41:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:41:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:41:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:41:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:41:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:41:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:41:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:41:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:41:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:41:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:41:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:41:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:41:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:41:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:41:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:41:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:41:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:41:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:41:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:41:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:41:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:41:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:41:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:41:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:41:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:41:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:41:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:41:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:41:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:41:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:41:44.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:41:44.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:41:44.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:41:44.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:41:44.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=623] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:41:44.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=623] Local address: 172.21.15.161, port 56982 2022-07-03T16:41:44.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=623] Peer address: 172.21.15.101, port 22 2022-07-03T16:41:44.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=623] Connection lost 2022-07-03T16:41:44.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=623] Aborting connection 2022-07-03T16:41:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:44 smithi161 ceph-mon[91579]: pgmap v10878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:44 smithi188 ceph-mon[92425]: pgmap v10878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:46 smithi161 ceph-mon[91579]: pgmap v10879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:46 smithi188 ceph-mon[92425]: pgmap v10879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:48 smithi161 ceph-mon[91579]: pgmap v10880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:48 smithi188 ceph-mon[92425]: pgmap v10880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:50 smithi161 ceph-mon[91579]: pgmap v10881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:50 smithi188 ceph-mon[92425]: pgmap v10881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:52 smithi161 ceph-mon[91579]: pgmap v10882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:52 smithi188 ceph-mon[92425]: pgmap v10882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:54 smithi161 ceph-mon[91579]: pgmap v10883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:54 smithi188 ceph-mon[92425]: pgmap v10883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:41:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:57 smithi161 ceph-mon[91579]: pgmap v10884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:57 smithi188 ceph-mon[92425]: pgmap v10884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:41:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:41:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:59 smithi161 ceph-mon[91579]: pgmap v10885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:41:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:41:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:41:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:41:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:59 smithi188 ceph-mon[92425]: pgmap v10885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:41:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:41:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:41:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:41:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:42:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:01 smithi161 ceph-mon[91579]: pgmap v10886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:01 smithi188 ceph-mon[92425]: pgmap v10886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:03 smithi161 ceph-mon[91579]: pgmap v10887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:03 smithi188 ceph-mon[92425]: pgmap v10887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:05 smithi161 ceph-mon[91579]: pgmap v10888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:05 smithi188 ceph-mon[92425]: pgmap v10888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:07 smithi161 ceph-mon[91579]: pgmap v10889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:07 smithi188 ceph-mon[92425]: pgmap v10889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:09 smithi161 ceph-mon[91579]: pgmap v10890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:09 smithi188 ceph-mon[92425]: pgmap v10890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:11 smithi161 ceph-mon[91579]: pgmap v10891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:11 smithi188 ceph-mon[92425]: pgmap v10891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:13 smithi161 ceph-mon[91579]: pgmap v10892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:13 smithi188 ceph-mon[92425]: pgmap v10892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:15 smithi161 ceph-mon[91579]: pgmap v10893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:15 smithi188 ceph-mon[92425]: pgmap v10893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:17 smithi161 ceph-mon[91579]: pgmap v10894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:17 smithi188 ceph-mon[92425]: pgmap v10894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:19 smithi161 ceph-mon[91579]: pgmap v10895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:19 smithi188 ceph-mon[92425]: pgmap v10895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:21 smithi161 ceph-mon[91579]: pgmap v10896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:21 smithi188 ceph-mon[92425]: pgmap v10896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:23 smithi161 ceph-mon[91579]: pgmap v10897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:23 smithi188 ceph-mon[92425]: pgmap v10897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:25 smithi161 ceph-mon[91579]: pgmap v10898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:25 smithi188 ceph-mon[92425]: pgmap v10898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:27 smithi161 ceph-mon[91579]: pgmap v10899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:27 smithi188 ceph-mon[92425]: pgmap v10899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:29 smithi161 ceph-mon[91579]: pgmap v10900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:29 smithi188 ceph-mon[92425]: pgmap v10900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:31 smithi161 ceph-mon[91579]: pgmap v10901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:31 smithi188 ceph-mon[92425]: pgmap v10901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:33 smithi161 ceph-mon[91579]: pgmap v10902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:33 smithi188 ceph-mon[92425]: pgmap v10902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:35 smithi161 ceph-mon[91579]: pgmap v10903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:35 smithi188 ceph-mon[92425]: pgmap v10903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:37 smithi161 ceph-mon[91579]: pgmap v10904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:37 smithi188 ceph-mon[92425]: pgmap v10904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:39 smithi161 ceph-mon[91579]: pgmap v10905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:39 smithi188 ceph-mon[92425]: pgmap v10905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:41 smithi161 ceph-mon[91579]: pgmap v10906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:41 smithi188 ceph-mon[92425]: pgmap v10906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:43.338 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-mon[91579]: pgmap v10907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:43.338 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:43.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:42:43.283+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:42:43.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:42:43.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:42:43.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:42:43.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:42:43.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:42:43.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:42:43.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:42:43.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:42:43.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:42:43.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:42:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:42:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:42:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:42:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:42:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:42:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:42:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:42:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:42:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:42:43.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:42:43.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:42:43.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:42:43.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:42:43.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:42:43.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:42:43.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:42:43.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:42:43.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:42:43.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:42:43.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:42:43.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:42:43.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:42:43.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:42:43.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:42:43.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:42:43.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:42:43.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:42:43.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:42:43.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:42:43.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:42:43.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:42:43.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:42:43.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:42:43.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:42:43.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=625] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:42:43.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=625] Local address: 172.21.15.161, port 57010 2022-07-03T16:42:43.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=625] Peer address: 172.21.15.101, port 22 2022-07-03T16:42:43.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=625] Connection lost 2022-07-03T16:42:43.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=625] Aborting connection 2022-07-03T16:42:43.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:42:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:42:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:43 smithi188 ceph-mon[92425]: pgmap v10907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:42:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:42:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:42:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:42:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:42:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:42:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:42:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:42:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:42:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:42:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:42:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:42:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:42:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:42:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:42:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:42:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:42:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:42:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:42:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:42:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:42:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:42:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:42:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:42:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:42:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:42:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:42:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:42:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:42:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:42:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:42:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:42:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:42:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:42:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:42:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:42:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:42:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:42:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:42:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:42:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:42:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:42:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:42:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:42:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:42:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:42:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:42:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:42:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:42:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:42:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:42:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:42:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=625] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:42:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=625] Local address: 172.21.15.161, port 57010 2022-07-03T16:42:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=625] Peer address: 172.21.15.101, port 22 2022-07-03T16:42:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=625] Connection lost 2022-07-03T16:42:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=625] Aborting connection 2022-07-03T16:42:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:45 smithi161 ceph-mon[91579]: pgmap v10908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:42:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:42:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:42:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:42:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:42:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:42:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:42:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:42:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:42:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:42:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:42:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:42:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:42:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:42:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:42:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:42:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:42:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:42:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:42:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:42:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:42:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:42:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:42:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:42:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:42:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:42:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:42:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:42:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:42:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:42:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:42:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:42:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:42:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:42:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:42:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:42:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:42:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:42:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:42:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:42:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:42:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:42:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:42:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:42:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:42:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:42:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=625] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:42:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=625] Local address: 172.21.15.161, port 57010 2022-07-03T16:42:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=625] Peer address: 172.21.15.101, port 22 2022-07-03T16:42:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=625] Connection lost 2022-07-03T16:42:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=625] Aborting connection 2022-07-03T16:42:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:45 smithi188 ceph-mon[92425]: pgmap v10908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:47 smithi161 ceph-mon[91579]: pgmap v10909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:47 smithi188 ceph-mon[92425]: pgmap v10909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:49 smithi188 ceph-mon[92425]: pgmap v10910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:49 smithi161 ceph-mon[91579]: pgmap v10910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:51 smithi188 ceph-mon[92425]: pgmap v10911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:51 smithi161 ceph-mon[91579]: pgmap v10911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:53 smithi188 ceph-mon[92425]: pgmap v10912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:53 smithi161 ceph-mon[91579]: pgmap v10912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:55 smithi188 ceph-mon[92425]: pgmap v10913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:55 smithi161 ceph-mon[91579]: pgmap v10913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:42:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:57 smithi188 ceph-mon[92425]: pgmap v10914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:57 smithi161 ceph-mon[91579]: pgmap v10914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:42:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:42:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:59 smithi188 ceph-mon[92425]: pgmap v10915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:42:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:42:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:42:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:42:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:59 smithi161 ceph-mon[91579]: pgmap v10915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:42:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:42:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:42:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:42:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:43:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:01 smithi188 ceph-mon[92425]: pgmap v10916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:01 smithi161 ceph-mon[91579]: pgmap v10916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:03 smithi161 ceph-mon[91579]: pgmap v10917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:03 smithi188 ceph-mon[92425]: pgmap v10917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:05 smithi161 ceph-mon[91579]: pgmap v10918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:05 smithi188 ceph-mon[92425]: pgmap v10918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:07 smithi161 ceph-mon[91579]: pgmap v10919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:07 smithi188 ceph-mon[92425]: pgmap v10919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:09 smithi161 ceph-mon[91579]: pgmap v10920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:09 smithi188 ceph-mon[92425]: pgmap v10920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:11 smithi161 ceph-mon[91579]: pgmap v10921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:11 smithi188 ceph-mon[92425]: pgmap v10921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:13 smithi161 ceph-mon[91579]: pgmap v10922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:13 smithi188 ceph-mon[92425]: pgmap v10922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:15 smithi161 ceph-mon[91579]: pgmap v10923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:15 smithi188 ceph-mon[92425]: pgmap v10923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:17 smithi161 ceph-mon[91579]: pgmap v10924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:17 smithi188 ceph-mon[92425]: pgmap v10924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:19.390 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:19 smithi188 ceph-mon[92425]: pgmap v10925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:19 smithi161 ceph-mon[91579]: pgmap v10925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:21.291 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:21 smithi188 ceph-mon[92425]: pgmap v10926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:21 smithi161 ceph-mon[91579]: pgmap v10926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:23 smithi161 ceph-mon[91579]: pgmap v10927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:23 smithi188 ceph-mon[92425]: pgmap v10927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:25 smithi161 ceph-mon[91579]: pgmap v10928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:25 smithi188 ceph-mon[92425]: pgmap v10928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:27 smithi161 ceph-mon[91579]: pgmap v10929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:27 smithi188 ceph-mon[92425]: pgmap v10929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:29 smithi161 ceph-mon[91579]: pgmap v10930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:29 smithi188 ceph-mon[92425]: pgmap v10930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:31 smithi161 ceph-mon[91579]: pgmap v10931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:31 smithi188 ceph-mon[92425]: pgmap v10931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:33 smithi161 ceph-mon[91579]: pgmap v10932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:33 smithi188 ceph-mon[92425]: pgmap v10932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:35 smithi161 ceph-mon[91579]: pgmap v10933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:35 smithi188 ceph-mon[92425]: pgmap v10933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:37 smithi161 ceph-mon[91579]: pgmap v10934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:37 smithi188 ceph-mon[92425]: pgmap v10934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:39 smithi161 ceph-mon[91579]: pgmap v10935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:39 smithi188 ceph-mon[92425]: pgmap v10935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:41 smithi161 ceph-mon[91579]: pgmap v10936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:41 smithi188 ceph-mon[92425]: pgmap v10936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-mon[91579]: pgmap v10937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:43 smithi188 ceph-mon[92425]: pgmap v10937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:44.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:43:43.806+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:43:44.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:43:44.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:43:44.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:43:44.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:43:44.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:43:44.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:43:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:43:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:43:44.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:43:44.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:43:44.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:43:44.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:43:44.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:43:44.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:43:44.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:43:44.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:43:44.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:43:44.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:43:44.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:43:44.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:43:44.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:43:44.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:43:44.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:43:44.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:43:44.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:43:44.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:43:44.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:43:44.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:43:44.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:43:44.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:43:44.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:43:44.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:43:44.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:43:44.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:43:44.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:43:44.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:43:44.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:43:44.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:43:44.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:43:44.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:43:44.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:43:44.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:43:44.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:43:44.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:43:44.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=627] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:43:44.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=627] Local address: 172.21.15.161, port 57038 2022-07-03T16:43:44.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=627] Peer address: 172.21.15.101, port 22 2022-07-03T16:43:44.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=627] Connection lost 2022-07-03T16:43:44.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=627] Aborting connection 2022-07-03T16:43:44.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:43:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:43:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:43:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:43:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:43:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:43:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:43:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:43:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:45 smithi161 ceph-mon[91579]: pgmap v10938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:43:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:43:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:43:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:43:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:43:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:43:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:43:45.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:43:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:43:45.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:43:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:43:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:43:45.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:43:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:43:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:43:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:43:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:43:45.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:43:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:43:45.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:43:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:43:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:43:45.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:43:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:43:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:43:45.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:43:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:43:45.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:43:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:43:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:43:45.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:43:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:43:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:43:45.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:43:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:43:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:43:45.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:43:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:43:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:43:45.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:43:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:43:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:43:45.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:43:45.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:43:45.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:43:45.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:43:45.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=627] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:43:45.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=627] Local address: 172.21.15.161, port 57038 2022-07-03T16:43:45.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=627] Peer address: 172.21.15.101, port 22 2022-07-03T16:43:45.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=627] Connection lost 2022-07-03T16:43:45.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=627] Aborting connection 2022-07-03T16:43:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:45 smithi188 ceph-mon[92425]: pgmap v10938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:43:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:43:45.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:43:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:43:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:43:45.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:43:45.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:43:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:43:45.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:43:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:43:45.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:43:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:43:45.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:43:45.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:43:45.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:43:45.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:43:45.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:43:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:43:45.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:43:45.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:43:45.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:43:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:43:45.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:43:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:43:45.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:43:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:43:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:43:45.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:43:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:43:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:43:45.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=627] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:43:45.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=627] Local address: 172.21.15.161, port 57038 2022-07-03T16:43:45.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=627] Peer address: 172.21.15.101, port 22 2022-07-03T16:43:45.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=627] Connection lost 2022-07-03T16:43:45.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=627] Aborting connection 2022-07-03T16:43:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:47 smithi161 ceph-mon[91579]: pgmap v10939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:47 smithi188 ceph-mon[92425]: pgmap v10939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:49 smithi161 ceph-mon[91579]: pgmap v10940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:49 smithi188 ceph-mon[92425]: pgmap v10940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:51 smithi161 ceph-mon[91579]: pgmap v10941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:51 smithi188 ceph-mon[92425]: pgmap v10941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:52 smithi161 ceph-mon[91579]: pgmap v10942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:52 smithi188 ceph-mon[92425]: pgmap v10942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:54 smithi161 ceph-mon[91579]: pgmap v10943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:54 smithi188 ceph-mon[92425]: pgmap v10943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:43:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:56 smithi161 ceph-mon[91579]: pgmap v10944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:56.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:56 smithi188 ceph-mon[92425]: pgmap v10944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:43:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:43:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:58 smithi161 ceph-mon[91579]: pgmap v10945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:43:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:43:58.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:43:58.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:43:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:58 smithi188 ceph-mon[92425]: pgmap v10945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:43:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:43:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:43:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:43:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:44:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:00 smithi161 ceph-mon[91579]: pgmap v10946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:00 smithi188 ceph-mon[92425]: pgmap v10946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:02 smithi161 ceph-mon[91579]: pgmap v10947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:02 smithi188 ceph-mon[92425]: pgmap v10947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:04 smithi161 ceph-mon[91579]: pgmap v10948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:04 smithi188 ceph-mon[92425]: pgmap v10948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:06 smithi161 ceph-mon[91579]: pgmap v10949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:06 smithi188 ceph-mon[92425]: pgmap v10949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:08 smithi161 ceph-mon[91579]: pgmap v10950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:08 smithi188 ceph-mon[92425]: pgmap v10950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:10 smithi161 ceph-mon[91579]: pgmap v10951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:10.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:10 smithi188 ceph-mon[92425]: pgmap v10951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:12 smithi161 ceph-mon[91579]: pgmap v10952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:12 smithi188 ceph-mon[92425]: pgmap v10952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:14 smithi161 ceph-mon[91579]: pgmap v10953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:14 smithi188 ceph-mon[92425]: pgmap v10953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:16 smithi161 ceph-mon[91579]: pgmap v10954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:16.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:16.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:16 smithi188 ceph-mon[92425]: pgmap v10954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:18 smithi161 ceph-mon[91579]: pgmap v10955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:18.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:18 smithi188 ceph-mon[92425]: pgmap v10955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:20 smithi161 ceph-mon[91579]: pgmap v10956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:20.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:20 smithi188 ceph-mon[92425]: pgmap v10956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:22 smithi161 ceph-mon[91579]: pgmap v10957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:22 smithi188 ceph-mon[92425]: pgmap v10957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:24 smithi161 ceph-mon[91579]: pgmap v10958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:24 smithi188 ceph-mon[92425]: pgmap v10958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:26 smithi161 ceph-mon[91579]: pgmap v10959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:26.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:26 smithi188 ceph-mon[92425]: pgmap v10959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:28 smithi161 ceph-mon[91579]: pgmap v10960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:28 smithi188 ceph-mon[92425]: pgmap v10960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:30 smithi188 ceph-mon[92425]: pgmap v10961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:30 smithi161 ceph-mon[91579]: pgmap v10961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:32 smithi188 ceph-mon[92425]: pgmap v10962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:32 smithi161 ceph-mon[91579]: pgmap v10962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:34 smithi188 ceph-mon[92425]: pgmap v10963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:34 smithi161 ceph-mon[91579]: pgmap v10963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:36 smithi188 ceph-mon[92425]: pgmap v10964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:36 smithi161 ceph-mon[91579]: pgmap v10964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:37.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:38 smithi188 ceph-mon[92425]: pgmap v10965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:38 smithi161 ceph-mon[91579]: pgmap v10965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:40 smithi188 ceph-mon[92425]: pgmap v10966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:40 smithi161 ceph-mon[91579]: pgmap v10966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:42 smithi161 ceph-mon[91579]: pgmap v10967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:42 smithi188 ceph-mon[92425]: pgmap v10967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:44:44.316+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:44:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:44:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:44:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:44:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:44:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:44:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:44:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:44:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:44:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:44:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:44:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:44:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:44:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:44:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:44:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:44:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:44:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:44:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:44:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:44:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:44:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:44:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:44:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:44:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:44:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:44:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:44:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:44:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:44:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:44:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:44:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:44:44.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:44:44.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:44:44.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:44:44.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:44:44.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:44:44.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:44:44.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:44:44.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:44:44.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:44:44.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:44:44.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:44:44.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:44:44.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:44:44.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=629] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:44:44.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=629] Local address: 172.21.15.161, port 57066 2022-07-03T16:44:44.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=629] Peer address: 172.21.15.101, port 22 2022-07-03T16:44:44.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=629] Connection lost 2022-07-03T16:44:44.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=629] Aborting connection 2022-07-03T16:44:44.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:44:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-mon[91579]: pgmap v10968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:44:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:44:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:44:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:44 smithi188 ceph-mon[92425]: pgmap v10968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:44:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:44:45.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:44:46.003 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:44:46.004 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:44:46.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:44:46.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:44:46.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:44:46.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:44:46.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:44:46.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:44:46.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:44:46.007 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:44:46.007 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:44:46.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:44:46.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:44:46.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:44:46.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:44:46.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:44:46.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:44:46.010 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:44:46.010 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:44:46.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:44:46.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:44:46.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:44:46.012 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:44:46.012 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:44:46.013 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:44:46.013 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:44:46.014 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:44:46.014 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:44:46.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:44:46.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:44:46.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:44:46.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:44:46.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:44:46.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:44:46.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:44:46.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:44:46.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:44:46.018 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:44:46.018 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:44:46.019 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:44:46.019 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:44:46.019 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:44:46.020 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:44:46.020 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:44:46.020 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:44:46.021 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:44:46.021 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=629] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:44:46.021 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=629] Local address: 172.21.15.161, port 57066 2022-07-03T16:44:46.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=629] Peer address: 172.21.15.101, port 22 2022-07-03T16:44:46.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=629] Connection lost 2022-07-03T16:44:46.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=629] Aborting connection 2022-07-03T16:44:46.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.024 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.025 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:44:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:44:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:44:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:44:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:44:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:44:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:44:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:44:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:44:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:44:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:44:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:44:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:44:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:44:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:44:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:44:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:44:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:44:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:44:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:44:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:44:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:44:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:44:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:44:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:44:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:44:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:44:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:44:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:44:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:44:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:44:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:44:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:44:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:44:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:44:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:44:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:44:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:44:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:44:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:44:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:44:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:44:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:44:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:44:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:44:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:44:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=629] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:44:46.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=629] Local address: 172.21.15.161, port 57066 2022-07-03T16:44:46.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=629] Peer address: 172.21.15.101, port 22 2022-07-03T16:44:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=629] Connection lost 2022-07-03T16:44:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=629] Aborting connection 2022-07-03T16:44:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:46 smithi161 ceph-mon[91579]: pgmap v10969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:46 smithi188 ceph-mon[92425]: pgmap v10969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:48.048 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:49 smithi161 ceph-mon[91579]: pgmap v10970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:49 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T16:44:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:49 smithi188 ceph-mon[92425]: pgmap v10970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:49 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T16:44:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:44:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:51 smithi161 ceph-mon[91579]: pgmap v10971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:51 smithi188 ceph-mon[92425]: pgmap v10971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:53 smithi161 ceph-mon[91579]: pgmap v10972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:53 smithi188 ceph-mon[92425]: pgmap v10972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:55 smithi161 ceph-mon[91579]: pgmap v10973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:55 smithi188 ceph-mon[92425]: pgmap v10973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:44:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:57 smithi161 ceph-mon[91579]: pgmap v10974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:57 smithi188 ceph-mon[92425]: pgmap v10974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:44:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:44:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:59 smithi161 ceph-mon[91579]: pgmap v10975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:44:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:44:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:44:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:44:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:59 smithi188 ceph-mon[92425]: pgmap v10975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:44:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:44:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:44:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:44:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:45:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:01 smithi161 ceph-mon[91579]: pgmap v10976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:01 smithi188 ceph-mon[92425]: pgmap v10976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:03 smithi161 ceph-mon[91579]: pgmap v10977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:03 smithi188 ceph-mon[92425]: pgmap v10977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:05 smithi161 ceph-mon[91579]: pgmap v10978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:05 smithi188 ceph-mon[92425]: pgmap v10978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:07 smithi161 ceph-mon[91579]: pgmap v10979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:07 smithi188 ceph-mon[92425]: pgmap v10979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:09 smithi161 ceph-mon[91579]: pgmap v10980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:09 smithi188 ceph-mon[92425]: pgmap v10980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:11 smithi161 ceph-mon[91579]: pgmap v10981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:11 smithi188 ceph-mon[92425]: pgmap v10981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:13 smithi161 ceph-mon[91579]: pgmap v10982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:13 smithi188 ceph-mon[92425]: pgmap v10982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:15 smithi161 ceph-mon[91579]: pgmap v10983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:15 smithi188 ceph-mon[92425]: pgmap v10983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:17 smithi188 ceph-mon[92425]: pgmap v10984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:17 smithi161 ceph-mon[91579]: pgmap v10984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:19 smithi188 ceph-mon[92425]: pgmap v10985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:19 smithi161 ceph-mon[91579]: pgmap v10985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:21 smithi188 ceph-mon[92425]: pgmap v10986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:21 smithi161 ceph-mon[91579]: pgmap v10986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:23 smithi188 ceph-mon[92425]: pgmap v10987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:23 smithi161 ceph-mon[91579]: pgmap v10987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:25 smithi188 ceph-mon[92425]: pgmap v10988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:25 smithi161 ceph-mon[91579]: pgmap v10988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:27 smithi188 ceph-mon[92425]: pgmap v10989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:27 smithi161 ceph-mon[91579]: pgmap v10989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:29 smithi161 ceph-mon[91579]: pgmap v10990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:29 smithi188 ceph-mon[92425]: pgmap v10990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:31 smithi161 ceph-mon[91579]: pgmap v10991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:31 smithi188 ceph-mon[92425]: pgmap v10991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:33 smithi161 ceph-mon[91579]: pgmap v10992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:33 smithi188 ceph-mon[92425]: pgmap v10992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:35 smithi161 ceph-mon[91579]: pgmap v10993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:35 smithi188 ceph-mon[92425]: pgmap v10993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:37 smithi161 ceph-mon[91579]: pgmap v10994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:37 smithi188 ceph-mon[92425]: pgmap v10994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:39 smithi161 ceph-mon[91579]: pgmap v10995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:39 smithi188 ceph-mon[92425]: pgmap v10995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:41 smithi161 ceph-mon[91579]: pgmap v10996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:41 smithi188 ceph-mon[92425]: pgmap v10996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:43 smithi161 ceph-mon[91579]: pgmap v10997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:43 smithi188 ceph-mon[92425]: pgmap v10997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:45 smithi161 ceph-mon[91579]: pgmap v10998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:45 smithi188 ceph-mon[92425]: pgmap v10998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:47 smithi161 ceph-mon[91579]: pgmap v10999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:47 smithi188 ceph-mon[92425]: pgmap v10999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:45:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:45:48.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:45:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:45:48.108+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:45:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:45:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:45:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:45:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:45:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:45:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:45:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:45:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:45:48.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:45:48.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:45:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:45:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:45:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:45:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:45:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:45:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:45:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:45:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:45:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:45:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:45:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:45:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:45:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:45:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:45:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:45:48.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:45:48.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:45:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:45:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:45:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:45:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:45:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:45:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:45:48.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:45:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:45:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:45:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:45:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:45:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:45:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:45:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:45:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:45:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:45:48.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:45:48.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=631] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:45:48.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=631] Local address: 172.21.15.161, port 57096 2022-07-03T16:45:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=631] Peer address: 172.21.15.101, port 22 2022-07-03T16:45:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=631] Connection lost 2022-07-03T16:45:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=631] Aborting connection 2022-07-03T16:45:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:45:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:45:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:45:48.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:45:48.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:45:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: pgmap v11000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:45:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:45:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:45:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:45:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:45:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:45:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:45:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:45:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:45:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:45:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:45:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:45:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:45:49.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:45:49.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:45:49.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:45:49.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:45:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:45:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:45:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:45:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:45:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:45:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:45:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:45:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:45:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:45:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:45:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:45:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:45:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:45:49.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:45:49.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:45:49.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:45:49.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:45:49.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:45:49.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:45:49.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:45:49.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:45:49.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:45:49.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:45:49.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:45:49.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:45:49.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:45:49.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:45:49.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:45:49.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:45:49.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=631] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:45:49.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=631] Local address: 172.21.15.161, port 57096 2022-07-03T16:45:49.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=631] Peer address: 172.21.15.101, port 22 2022-07-03T16:45:49.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=631] Connection lost 2022-07-03T16:45:49.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=631] Aborting connection 2022-07-03T16:45:49.664 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.665 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.665 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.666 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: pgmap v11000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:45:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:45:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:45:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:45:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:45:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:45:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:45:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:45:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:45:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:45:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:45:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:45:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:45:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:45:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:45:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:45:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:45:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:45:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:45:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:45:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:45:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:45:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:45:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:45:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:45:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:45:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:45:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:45:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:45:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:45:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:45:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:45:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:45:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:45:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:45:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:45:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:45:49.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:45:49.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:45:49.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:45:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:45:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:45:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:45:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:45:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:45:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:45:49.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=631] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:45:49.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=631] Local address: 172.21.15.161, port 57096 2022-07-03T16:45:49.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=631] Peer address: 172.21.15.101, port 22 2022-07-03T16:45:49.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=631] Connection lost 2022-07-03T16:45:49.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=631] Aborting connection 2022-07-03T16:45:49.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:49.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:45:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:51 smithi161 ceph-mon[91579]: pgmap v11001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:51 smithi188 ceph-mon[92425]: pgmap v11001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:53 smithi161 ceph-mon[91579]: pgmap v11002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:53 smithi188 ceph-mon[92425]: pgmap v11002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:55 smithi161 ceph-mon[91579]: pgmap v11003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:55 smithi188 ceph-mon[92425]: pgmap v11003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:45:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:57 smithi161 ceph-mon[91579]: pgmap v11004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:57 smithi188 ceph-mon[92425]: pgmap v11004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:45:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:45:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:59 smithi161 ceph-mon[91579]: pgmap v11005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:45:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:45:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:45:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:45:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:59 smithi188 ceph-mon[92425]: pgmap v11005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:45:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:45:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:45:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:45:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:46:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:01 smithi161 ceph-mon[91579]: pgmap v11006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:01 smithi188 ceph-mon[92425]: pgmap v11006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:03 smithi161 ceph-mon[91579]: pgmap v11007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:03 smithi188 ceph-mon[92425]: pgmap v11007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:05 smithi161 ceph-mon[91579]: pgmap v11008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:05 smithi188 ceph-mon[92425]: pgmap v11008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:07 smithi161 ceph-mon[91579]: pgmap v11009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:07 smithi188 ceph-mon[92425]: pgmap v11009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:09 smithi161 ceph-mon[91579]: pgmap v11010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:09 smithi188 ceph-mon[92425]: pgmap v11010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:11 smithi161 ceph-mon[91579]: pgmap v11011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:11 smithi188 ceph-mon[92425]: pgmap v11011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:13 smithi161 ceph-mon[91579]: pgmap v11012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:13.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:13 smithi188 ceph-mon[92425]: pgmap v11012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:13.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:15 smithi161 ceph-mon[91579]: pgmap v11013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:15 smithi188 ceph-mon[92425]: pgmap v11013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:17 smithi161 ceph-mon[91579]: pgmap v11014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:17 smithi188 ceph-mon[92425]: pgmap v11014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:19 smithi161 ceph-mon[91579]: pgmap v11015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:19 smithi188 ceph-mon[92425]: pgmap v11015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:21 smithi161 ceph-mon[91579]: pgmap v11016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:21 smithi188 ceph-mon[92425]: pgmap v11016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:23 smithi161 ceph-mon[91579]: pgmap v11017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:23 smithi188 ceph-mon[92425]: pgmap v11017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:25 smithi161 ceph-mon[91579]: pgmap v11018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:25 smithi188 ceph-mon[92425]: pgmap v11018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:27 smithi161 ceph-mon[91579]: pgmap v11019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:27 smithi188 ceph-mon[92425]: pgmap v11019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:29 smithi161 ceph-mon[91579]: pgmap v11020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:29 smithi188 ceph-mon[92425]: pgmap v11020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:30 smithi161 ceph-mon[91579]: pgmap v11021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:30 smithi188 ceph-mon[92425]: pgmap v11021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:32 smithi161 ceph-mon[91579]: pgmap v11022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:32 smithi188 ceph-mon[92425]: pgmap v11022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:34 smithi161 ceph-mon[91579]: pgmap v11023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:34 smithi188 ceph-mon[92425]: pgmap v11023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:36 smithi161 ceph-mon[91579]: pgmap v11024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:36 smithi188 ceph-mon[92425]: pgmap v11024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:38 smithi161 ceph-mon[91579]: pgmap v11025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:38 smithi188 ceph-mon[92425]: pgmap v11025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:40 smithi161 ceph-mon[91579]: pgmap v11026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:40 smithi188 ceph-mon[92425]: pgmap v11026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:41.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:46:41.377+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T16:46:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T16:46:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:42 smithi161 ceph-mon[91579]: pgmap v11027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:42 smithi188 ceph-mon[92425]: pgmap v11027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:44 smithi161 ceph-mon[91579]: pgmap v11028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:44 smithi188 ceph-mon[92425]: pgmap v11028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:46.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:46 smithi188 ceph-mon[92425]: pgmap v11029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:46 smithi161 ceph-mon[91579]: pgmap v11029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:48 smithi188 ceph-mon[92425]: pgmap v11030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:49.022 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:48 smithi161 ceph-mon[91579]: pgmap v11030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:49.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:46:49.020+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:46:49.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:46:49.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:46:49.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:46:49.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:46:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:46:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:46:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:46:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:46:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:46:49.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:46:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:46:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:46:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:46:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:46:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:46:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:46:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:46:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:46:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:46:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:46:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:46:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:46:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:46:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:46:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:46:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:46:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:46:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:46:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:46:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:46:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:46:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:46:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:46:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:46:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:46:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:46:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:46:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:46:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:46:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:46:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:46:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:46:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:46:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:46:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=633] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:46:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=633] Local address: 172.21.15.161, port 57122 2022-07-03T16:46:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=633] Peer address: 172.21.15.101, port 22 2022-07-03T16:46:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=633] Connection lost 2022-07-03T16:46:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=633] Aborting connection 2022-07-03T16:46:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:46:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:46:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:46:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:46:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:46:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:46:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:46:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:46:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:46:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:46:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:46:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:46:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:46:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:46:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:46:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:46:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:46:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:46:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:46:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:46:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:46:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:46:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:46:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:46:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:46:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:46:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:46:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:46:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:46:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:46:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:46:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:46:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:46:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:46:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:46:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:46:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:46:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:46:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:46:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:46:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:46:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:46:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:46:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:46:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:46:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:46:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:46:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=633] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:46:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=633] Local address: 172.21.15.161, port 57122 2022-07-03T16:46:49.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=633] Peer address: 172.21.15.101, port 22 2022-07-03T16:46:49.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=633] Connection lost 2022-07-03T16:46:49.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=633] Aborting connection 2022-07-03T16:46:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:46:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:46:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:46:50.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:46:50.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:46:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:46:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:46:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:46:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:46:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:46:50.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:46:50.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:46:50.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:46:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:46:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:46:50.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:46:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:46:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:46:50.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:46:50.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:46:50.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:46:50.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:46:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:46:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:46:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:46:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:46:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:46:50.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:46:50.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:46:50.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:46:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:46:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:46:50.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:46:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:46:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:46:50.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:46:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:46:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:46:50.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:46:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:46:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:46:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:46:50.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:46:50.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:46:50.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:46:50.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:46:50.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:46:50.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:46:50.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:46:50.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=633] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:46:50.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=633] Local address: 172.21.15.161, port 57122 2022-07-03T16:46:50.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=633] Peer address: 172.21.15.101, port 22 2022-07-03T16:46:50.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=633] Connection lost 2022-07-03T16:46:50.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=633] Aborting connection 2022-07-03T16:46:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:50 smithi188 ceph-mon[92425]: pgmap v11031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:50 smithi161 ceph-mon[91579]: pgmap v11031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:52 smithi188 ceph-mon[92425]: pgmap v11032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:52 smithi161 ceph-mon[91579]: pgmap v11032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:54 smithi188 ceph-mon[92425]: pgmap v11033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:54 smithi161 ceph-mon[91579]: pgmap v11033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:46:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:56 smithi188 ceph-mon[92425]: pgmap v11034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:56 smithi161 ceph-mon[91579]: pgmap v11034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:58 smithi161 ceph-mon[91579]: pgmap v11035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:46:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:46:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:46:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:46:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:58 smithi188 ceph-mon[92425]: pgmap v11035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:46:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:46:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:46:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:46:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:47:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:00 smithi161 ceph-mon[91579]: pgmap v11036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:00 smithi188 ceph-mon[92425]: pgmap v11036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:02 smithi161 ceph-mon[91579]: pgmap v11037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:02 smithi188 ceph-mon[92425]: pgmap v11037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:04 smithi161 ceph-mon[91579]: pgmap v11038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:04 smithi188 ceph-mon[92425]: pgmap v11038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:06 smithi161 ceph-mon[91579]: pgmap v11039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:06 smithi188 ceph-mon[92425]: pgmap v11039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:08 smithi161 ceph-mon[91579]: pgmap v11040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:08 smithi188 ceph-mon[92425]: pgmap v11040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:10 smithi161 ceph-mon[91579]: pgmap v11041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:10 smithi188 ceph-mon[92425]: pgmap v11041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:12 smithi161 ceph-mon[91579]: pgmap v11042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:12 smithi188 ceph-mon[92425]: pgmap v11042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:14 smithi161 ceph-mon[91579]: pgmap v11043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:15.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:14 smithi188 ceph-mon[92425]: pgmap v11043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:16 smithi161 ceph-mon[91579]: pgmap v11044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:16 smithi188 ceph-mon[92425]: pgmap v11044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:18 smithi161 ceph-mon[91579]: pgmap v11045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:18 smithi188 ceph-mon[92425]: pgmap v11045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:20 smithi161 ceph-mon[91579]: pgmap v11046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:20 smithi188 ceph-mon[92425]: pgmap v11046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:22 smithi161 ceph-mon[91579]: pgmap v11047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:22 smithi188 ceph-mon[92425]: pgmap v11047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:24 smithi161 ceph-mon[91579]: pgmap v11048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:24 smithi188 ceph-mon[92425]: pgmap v11048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:26 smithi161 ceph-mon[91579]: pgmap v11049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:27.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:26 smithi188 ceph-mon[92425]: pgmap v11049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:28 smithi161 ceph-mon[91579]: pgmap v11050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:28 smithi188 ceph-mon[92425]: pgmap v11050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:30 smithi161 ceph-mon[91579]: pgmap v11051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:30 smithi188 ceph-mon[92425]: pgmap v11051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:32 smithi161 ceph-mon[91579]: pgmap v11052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:32 smithi188 ceph-mon[92425]: pgmap v11052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:34 smithi161 ceph-mon[91579]: pgmap v11053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:34 smithi188 ceph-mon[92425]: pgmap v11053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:36 smithi161 ceph-mon[91579]: pgmap v11054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:36 smithi188 ceph-mon[92425]: pgmap v11054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:38 smithi161 ceph-mon[91579]: pgmap v11055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:38 smithi188 ceph-mon[92425]: pgmap v11055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:40 smithi161 ceph-mon[91579]: pgmap v11056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:40 smithi188 ceph-mon[92425]: pgmap v11056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:42 smithi161 ceph-mon[91579]: pgmap v11057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:42 smithi188 ceph-mon[92425]: pgmap v11057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:44 smithi161 ceph-mon[91579]: pgmap v11058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:44 smithi188 ceph-mon[92425]: pgmap v11058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:46 smithi161 ceph-mon[91579]: pgmap v11059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:46 smithi188 ceph-mon[92425]: pgmap v11059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:48 smithi161 ceph-mon[91579]: pgmap v11060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:48 smithi188 ceph-mon[92425]: pgmap v11060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:49.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:47:49.519+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:47:49.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:47:49.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:47:49.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:47:49.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:47:49.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:47:49.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:47:49.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:47:49.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:47:49.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:47:49.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:47:49.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:47:49.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:47:49.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:47:49.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:47:49.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:47:49.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:47:49.798 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:47:49.799 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:47:49.799 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:47:49.800 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:47:49.800 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:47:49.801 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:47:49.802 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:47:49.802 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:47:49.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:47:49.804 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:47:49.805 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:47:49.805 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:47:49.806 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:47:49.807 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:47:49.808 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:47:49.809 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:47:49.809 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:47:49.810 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:47:49.811 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:47:49.811 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:47:49.812 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:47:49.813 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:47:49.813 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:47:49.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:47:49.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:47:49.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:47:49.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:47:49.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:47:49.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=635] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:47:49.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=635] Local address: 172.21.15.161, port 57126 2022-07-03T16:47:49.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=635] Peer address: 172.21.15.101, port 22 2022-07-03T16:47:49.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=635] Connection lost 2022-07-03T16:47:49.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=635] Aborting connection 2022-07-03T16:47:49.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:47:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:47:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:47:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:47:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:47:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:47:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:47:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:50 smithi161 ceph-mon[91579]: pgmap v11061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:47:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:47:51.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:47:51.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:47:51.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:47:51.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:47:51.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:47:51.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:47:51.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:47:51.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:47:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:47:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:47:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:47:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:47:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:47:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:47:51.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:47:51.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:47:51.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:47:51.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:47:51.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:47:51.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:47:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:47:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:47:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:47:51.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:47:51.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:47:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:47:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:47:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:47:51.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:47:51.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:47:51.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:47:51.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:47:51.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:47:51.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:47:51.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:47:51.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:47:51.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:47:51.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:47:51.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:47:51.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:47:51.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:47:51.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:47:51.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:47:51.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:47:51.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=635] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:47:51.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=635] Local address: 172.21.15.161, port 57126 2022-07-03T16:47:51.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=635] Peer address: 172.21.15.101, port 22 2022-07-03T16:47:51.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=635] Connection lost 2022-07-03T16:47:51.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=635] Aborting connection 2022-07-03T16:47:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:50 smithi188 ceph-mon[92425]: pgmap v11061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:47:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:47:51.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:47:51.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:47:51.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:47:51.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:47:51.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:47:51.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:47:51.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:47:51.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:47:51.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:47:51.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:47:51.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:47:51.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:47:51.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:47:51.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:47:51.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:47:51.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:47:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:47:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:47:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:47:51.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:47:51.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:47:51.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:47:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:47:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:47:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:47:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:47:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:47:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:47:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:47:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:47:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:47:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:47:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:47:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:47:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:47:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:47:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:47:51.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:47:51.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:47:51.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:47:51.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:47:51.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:47:51.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:47:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:47:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=635] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:47:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=635] Local address: 172.21.15.161, port 57126 2022-07-03T16:47:51.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=635] Peer address: 172.21.15.101, port 22 2022-07-03T16:47:51.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=635] Connection lost 2022-07-03T16:47:51.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=635] Aborting connection 2022-07-03T16:47:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: pgmap v11062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: pgmap v11062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:47:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:47:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 101.2M 2022-07-03T16:47:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 106182222: error parsing value: Value '106182222' is below minimum 939524096 2022-07-03T16:47:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:47:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:47:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 101.2M 2022-07-03T16:47:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 106182222: error parsing value: Value '106182222' is below minimum 939524096 2022-07-03T16:47:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:47:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:47:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:55 smithi161 ceph-mon[91579]: pgmap v11063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:55 smithi188 ceph-mon[92425]: pgmap v11063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:47:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:57 smithi161 ceph-mon[91579]: pgmap v11064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:57 smithi188 ceph-mon[92425]: pgmap v11064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:47:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:59 smithi161 ceph-mon[91579]: pgmap v11065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:47:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:47:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:47:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:47:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:59 smithi188 ceph-mon[92425]: pgmap v11065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:47:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:47:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:47:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:47:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:48:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:01 smithi161 ceph-mon[91579]: pgmap v11066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:01 smithi188 ceph-mon[92425]: pgmap v11066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:03 smithi161 ceph-mon[91579]: pgmap v11067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:03 smithi188 ceph-mon[92425]: pgmap v11067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:05 smithi161 ceph-mon[91579]: pgmap v11068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:05 smithi188 ceph-mon[92425]: pgmap v11068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:07 smithi161 ceph-mon[91579]: pgmap v11069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:07 smithi188 ceph-mon[92425]: pgmap v11069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:09 smithi161 ceph-mon[91579]: pgmap v11070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:09 smithi188 ceph-mon[92425]: pgmap v11070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:11 smithi161 ceph-mon[91579]: pgmap v11071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:11 smithi188 ceph-mon[92425]: pgmap v11071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:13 smithi161 ceph-mon[91579]: pgmap v11072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:13 smithi188 ceph-mon[92425]: pgmap v11072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:15 smithi161 ceph-mon[91579]: pgmap v11073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:15 smithi188 ceph-mon[92425]: pgmap v11073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:17 smithi161 ceph-mon[91579]: pgmap v11074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:17 smithi188 ceph-mon[92425]: pgmap v11074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:19 smithi161 ceph-mon[91579]: pgmap v11075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:19 smithi188 ceph-mon[92425]: pgmap v11075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:21 smithi161 ceph-mon[91579]: pgmap v11076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:21 smithi188 ceph-mon[92425]: pgmap v11076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:23 smithi161 ceph-mon[91579]: pgmap v11077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:23 smithi188 ceph-mon[92425]: pgmap v11077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:25 smithi161 ceph-mon[91579]: pgmap v11078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:25 smithi188 ceph-mon[92425]: pgmap v11078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:27 smithi161 ceph-mon[91579]: pgmap v11079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:27 smithi188 ceph-mon[92425]: pgmap v11079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:29 smithi161 ceph-mon[91579]: pgmap v11080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:29 smithi188 ceph-mon[92425]: pgmap v11080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:31 smithi161 ceph-mon[91579]: pgmap v11081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:31 smithi188 ceph-mon[92425]: pgmap v11081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:33 smithi161 ceph-mon[91579]: pgmap v11082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:33 smithi188 ceph-mon[92425]: pgmap v11082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:35 smithi161 ceph-mon[91579]: pgmap v11083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:35 smithi188 ceph-mon[92425]: pgmap v11083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:37 smithi161 ceph-mon[91579]: pgmap v11084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:37 smithi188 ceph-mon[92425]: pgmap v11084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:39 smithi161 ceph-mon[91579]: pgmap v11085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:39 smithi188 ceph-mon[92425]: pgmap v11085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:41 smithi161 ceph-mon[91579]: pgmap v11086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:41 smithi188 ceph-mon[92425]: pgmap v11086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:43 smithi161 ceph-mon[91579]: pgmap v11087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:43 smithi188 ceph-mon[92425]: pgmap v11087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:45 smithi161 ceph-mon[91579]: pgmap v11088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:45 smithi188 ceph-mon[92425]: pgmap v11088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:47 smithi161 ceph-mon[91579]: pgmap v11089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:47 smithi188 ceph-mon[92425]: pgmap v11089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:49 smithi161 ceph-mon[91579]: pgmap v11090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:49 smithi188 ceph-mon[92425]: pgmap v11090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:51 smithi161 ceph-mon[91579]: pgmap v11091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:51 smithi188 ceph-mon[92425]: pgmap v11091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:48:52.769+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:48:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:48:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:48:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:48:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:48:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:48:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:48:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:48:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:48:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:48:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:48:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:48:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:48:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:48:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:48:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:48:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:48:53.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:48:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:48:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:48:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:48:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:48:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:48:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:48:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:48:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:48:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:48:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:48:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:48:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:48:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:48:53.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:48:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:48:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:48:53.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:48:53.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:48:53.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:48:53.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:48:53.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:48:53.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:48:53.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:48:53.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:48:53.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:48:53.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:48:53.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:48:53.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=637] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:48:53.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=637] Local address: 172.21.15.161, port 57130 2022-07-03T16:48:53.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=637] Peer address: 172.21.15.101, port 22 2022-07-03T16:48:53.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=637] Connection lost 2022-07-03T16:48:53.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=637] Aborting connection 2022-07-03T16:48:53.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:48:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:48:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:53 smithi161 ceph-mon[91579]: pgmap v11092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:48:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:48:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:48:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:53 smithi188 ceph-mon[92425]: pgmap v11092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:48:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:48:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:48:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:48:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:48:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:48:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:48:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:48:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:48:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:48:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:48:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:48:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:48:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:48:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:48:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:48:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:48:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:48:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:48:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:48:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:48:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:48:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:48:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:48:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:48:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:48:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:48:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:48:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:48:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:48:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:48:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:48:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:48:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:48:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:48:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:48:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:48:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:48:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:48:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:48:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:48:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:48:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:48:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:48:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:48:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:48:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:48:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:48:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:48:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=637] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:48:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=637] Local address: 172.21.15.161, port 57130 2022-07-03T16:48:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=637] Peer address: 172.21.15.101, port 22 2022-07-03T16:48:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=637] Connection lost 2022-07-03T16:48:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=637] Aborting connection 2022-07-03T16:48:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:48:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:48:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:48:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:48:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:48:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:48:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:48:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:48:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:48:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:48:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:48:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:48:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:48:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:48:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:48:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:48:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:48:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:48:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:48:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:48:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:48:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:48:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:48:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:48:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:48:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:48:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:48:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:48:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:48:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:48:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:48:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:48:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:48:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:48:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:48:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:48:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:48:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:48:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:48:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:48:54.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:48:54.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:48:54.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:48:54.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:48:54.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:48:54.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:48:54.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=637] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:48:54.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=637] Local address: 172.21.15.161, port 57130 2022-07-03T16:48:54.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=637] Peer address: 172.21.15.101, port 22 2022-07-03T16:48:54.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=637] Connection lost 2022-07-03T16:48:54.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=637] Aborting connection 2022-07-03T16:48:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:55 smithi161 ceph-mon[91579]: pgmap v11093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:55.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:55 smithi188 ceph-mon[92425]: pgmap v11093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:48:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:57 smithi161 ceph-mon[91579]: pgmap v11094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:57 smithi188 ceph-mon[92425]: pgmap v11094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:48:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:59 smithi161 ceph-mon[91579]: pgmap v11095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:48:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:48:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:48:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:48:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:59 smithi188 ceph-mon[92425]: pgmap v11095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:48:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:48:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:48:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:48:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:49:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:01 smithi161 ceph-mon[91579]: pgmap v11096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:01 smithi188 ceph-mon[92425]: pgmap v11096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:03 smithi161 ceph-mon[91579]: pgmap v11097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:03 smithi188 ceph-mon[92425]: pgmap v11097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:05 smithi161 ceph-mon[91579]: pgmap v11098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:05 smithi188 ceph-mon[92425]: pgmap v11098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:07 smithi161 ceph-mon[91579]: pgmap v11099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:07 smithi188 ceph-mon[92425]: pgmap v11099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:09 smithi161 ceph-mon[91579]: pgmap v11100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:09 smithi188 ceph-mon[92425]: pgmap v11100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:11 smithi161 ceph-mon[91579]: pgmap v11101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:11 smithi188 ceph-mon[92425]: pgmap v11101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:12 smithi188 ceph-mon[92425]: pgmap v11102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:12.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:12 smithi161 ceph-mon[91579]: pgmap v11102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:14 smithi188 ceph-mon[92425]: pgmap v11103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:14 smithi161 ceph-mon[91579]: pgmap v11103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:16 smithi188 ceph-mon[92425]: pgmap v11104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:16 smithi161 ceph-mon[91579]: pgmap v11104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:18 smithi188 ceph-mon[92425]: pgmap v11105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:18 smithi161 ceph-mon[91579]: pgmap v11105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:20 smithi188 ceph-mon[92425]: pgmap v11106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:20 smithi161 ceph-mon[91579]: pgmap v11106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:22 smithi188 ceph-mon[92425]: pgmap v11107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:22 smithi161 ceph-mon[91579]: pgmap v11107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:24 smithi188 ceph-mon[92425]: pgmap v11108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:24 smithi161 ceph-mon[91579]: pgmap v11108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:26 smithi188 ceph-mon[92425]: pgmap v11109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:26 smithi161 ceph-mon[91579]: pgmap v11109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:28 smithi161 ceph-mon[91579]: pgmap v11110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:28 smithi188 ceph-mon[92425]: pgmap v11110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:30 smithi161 ceph-mon[91579]: pgmap v11111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:30 smithi188 ceph-mon[92425]: pgmap v11111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:32 smithi161 ceph-mon[91579]: pgmap v11112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:32 smithi188 ceph-mon[92425]: pgmap v11112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:34 smithi161 ceph-mon[91579]: pgmap v11113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:34 smithi188 ceph-mon[92425]: pgmap v11113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:36 smithi161 ceph-mon[91579]: pgmap v11114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:37.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:36 smithi188 ceph-mon[92425]: pgmap v11114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:38 smithi161 ceph-mon[91579]: pgmap v11115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:38 smithi188 ceph-mon[92425]: pgmap v11115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:40 smithi161 ceph-mon[91579]: pgmap v11116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:40 smithi188 ceph-mon[92425]: pgmap v11116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:42 smithi161 ceph-mon[91579]: pgmap v11117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:42 smithi188 ceph-mon[92425]: pgmap v11117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:45.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:44 smithi161 ceph-mon[91579]: pgmap v11118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:44 smithi188 ceph-mon[92425]: pgmap v11118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:46 smithi161 ceph-mon[91579]: pgmap v11119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:46 smithi188 ceph-mon[92425]: pgmap v11119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:48 smithi161 ceph-mon[91579]: pgmap v11120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:48 smithi188 ceph-mon[92425]: pgmap v11120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:50 smithi161 ceph-mon[91579]: pgmap v11121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:50 smithi188 ceph-mon[92425]: pgmap v11121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:52 smithi161 ceph-mon[91579]: pgmap v11122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:53.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:52 smithi188 ceph-mon[92425]: pgmap v11122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:53.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:49:53.271+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:49:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:49:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:49:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:49:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:49:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:49:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:49:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:49:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:49:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:49:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:49:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:49:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:49:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:49:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:49:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:49:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:49:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:49:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:49:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:49:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:49:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:49:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:49:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:49:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:49:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:49:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:49:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:49:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:49:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:49:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:49:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:49:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:49:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:49:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:49:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:49:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:49:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:49:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:49:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:49:53.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:49:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:49:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:49:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:49:53.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:49:53.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=639] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:49:53.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=639] Local address: 172.21.15.161, port 57134 2022-07-03T16:49:53.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=639] Peer address: 172.21.15.101, port 22 2022-07-03T16:49:53.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=639] Connection lost 2022-07-03T16:49:53.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=639] Aborting connection 2022-07-03T16:49:53.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:49:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:49:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:49:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:49:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:49:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:49:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:49:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:49:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:49:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:49:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:49:54.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:49:54.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:49:54.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:49:54.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:49:54.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:49:54.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:49:54.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:49:54.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:49:54.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:49:54.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:49:54.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:49:54.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:49:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:49:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:49:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:49:54.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:49:54.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:49:54.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:49:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:49:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:49:54.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:49:54.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:49:54.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:49:54.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:49:54.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:49:54.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:49:54.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:49:54.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:49:54.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:49:54.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:49:54.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:49:54.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:49:54.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:49:54.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:49:54.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:49:54.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:49:54.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:49:54.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:49:54.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:49:54.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=639] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:49:54.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=639] Local address: 172.21.15.161, port 57134 2022-07-03T16:49:54.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=639] Peer address: 172.21.15.101, port 22 2022-07-03T16:49:54.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=639] Connection lost 2022-07-03T16:49:54.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=639] Aborting connection 2022-07-03T16:49:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:49:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:49:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:49:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:49:54.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:49:54.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:49:54.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:49:54.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:49:54.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:49:54.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:49:54.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:49:54.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:49:54.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:49:54.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:49:54.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:49:54.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:49:54.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:49:54.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:49:54.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:49:54.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:49:54.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:49:54.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:49:54.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:49:54.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:49:54.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:49:54.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:49:54.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:49:54.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:49:54.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:49:54.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:49:54.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:49:54.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:49:54.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:49:54.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:49:54.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:49:54.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:49:54.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:49:54.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:49:54.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:49:54.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:49:54.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:49:54.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:49:54.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:49:54.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:49:54.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:49:54.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:49:54.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:49:54.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:49:54.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:49:54.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=639] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:49:54.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=639] Local address: 172.21.15.161, port 57134 2022-07-03T16:49:54.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=639] Peer address: 172.21.15.101, port 22 2022-07-03T16:49:54.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=639] Connection lost 2022-07-03T16:49:54.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=639] Aborting connection 2022-07-03T16:49:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:54 smithi161 ceph-mon[91579]: pgmap v11123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:54 smithi188 ceph-mon[92425]: pgmap v11123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:49:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:56 smithi161 ceph-mon[91579]: pgmap v11124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:56 smithi188 ceph-mon[92425]: pgmap v11124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:49:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:58 smithi161 ceph-mon[91579]: pgmap v11125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:49:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:49:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:49:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:49:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:58 smithi188 ceph-mon[92425]: pgmap v11125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:49:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:49:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:49:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:49:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:50:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:00 smithi161 ceph-mon[91579]: pgmap v11126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:00 smithi188 ceph-mon[92425]: pgmap v11126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T16:50:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:02 smithi161 ceph-mon[91579]: pgmap v11127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:02 smithi188 ceph-mon[92425]: pgmap v11127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:04 smithi161 ceph-mon[91579]: pgmap v11128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:04 smithi188 ceph-mon[92425]: pgmap v11128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:06 smithi161 ceph-mon[91579]: pgmap v11129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:06 smithi188 ceph-mon[92425]: pgmap v11129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:08 smithi161 ceph-mon[91579]: pgmap v11130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:08 smithi188 ceph-mon[92425]: pgmap v11130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:10 smithi161 ceph-mon[91579]: pgmap v11131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:11.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:10 smithi188 ceph-mon[92425]: pgmap v11131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:12 smithi161 ceph-mon[91579]: pgmap v11132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:12 smithi188 ceph-mon[92425]: pgmap v11132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:14 smithi161 ceph-mon[91579]: pgmap v11133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:14 smithi188 ceph-mon[92425]: pgmap v11133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:16 smithi161 ceph-mon[91579]: pgmap v11134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:16 smithi188 ceph-mon[92425]: pgmap v11134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:18.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:18.252 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:18 smithi161 ceph-mon[91579]: pgmap v11135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:18 smithi188 ceph-mon[92425]: pgmap v11135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:20 smithi161 ceph-mon[91579]: pgmap v11136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:20 smithi188 ceph-mon[92425]: pgmap v11136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:22 smithi161 ceph-mon[91579]: pgmap v11137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:22 smithi188 ceph-mon[92425]: pgmap v11137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:24 smithi161 ceph-mon[91579]: pgmap v11138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:24 smithi188 ceph-mon[92425]: pgmap v11138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:26 smithi161 ceph-mon[91579]: pgmap v11139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:26 smithi188 ceph-mon[92425]: pgmap v11139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:28 smithi161 ceph-mon[91579]: pgmap v11140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:28 smithi188 ceph-mon[92425]: pgmap v11140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:30 smithi161 ceph-mon[91579]: pgmap v11141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:30 smithi188 ceph-mon[92425]: pgmap v11141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:32 smithi161 ceph-mon[91579]: pgmap v11142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:32 smithi188 ceph-mon[92425]: pgmap v11142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:35.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:34 smithi161 ceph-mon[91579]: pgmap v11143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:35.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:34 smithi188 ceph-mon[92425]: pgmap v11143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:36 smithi161 ceph-mon[91579]: pgmap v11144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:36 smithi188 ceph-mon[92425]: pgmap v11144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:38 smithi161 ceph-mon[91579]: pgmap v11145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:38 smithi188 ceph-mon[92425]: pgmap v11145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:40 smithi161 ceph-mon[91579]: pgmap v11146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:40 smithi188 ceph-mon[92425]: pgmap v11146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:42 smithi161 ceph-mon[91579]: pgmap v11147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:42 smithi188 ceph-mon[92425]: pgmap v11147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:44 smithi161 ceph-mon[91579]: pgmap v11148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:44 smithi188 ceph-mon[92425]: pgmap v11148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:46 smithi161 ceph-mon[91579]: pgmap v11149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:46 smithi188 ceph-mon[92425]: pgmap v11149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:48 smithi161 ceph-mon[91579]: pgmap v11150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:48 smithi188 ceph-mon[92425]: pgmap v11150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:50 smithi161 ceph-mon[91579]: pgmap v11151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:50 smithi188 ceph-mon[92425]: pgmap v11151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:52 smithi161 ceph-mon[91579]: pgmap v11152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:52 smithi188 ceph-mon[92425]: pgmap v11152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:53.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:50:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:50:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:50:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:50:53.770+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:50:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:50:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:50:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:50:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:50:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:50:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:50:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:50:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:50:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:50:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:50:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:50:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:50:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:50:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:50:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:50:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:50:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:50:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:50:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:50:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:50:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:50:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:50:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:50:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:50:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:50:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:50:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:50:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:50:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:50:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:50:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:50:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:50:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:50:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:50:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:50:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:50:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:50:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:50:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:50:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:50:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:50:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:50:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:50:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:50:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=641] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:50:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=641] Local address: 172.21.15.161, port 57138 2022-07-03T16:50:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=641] Peer address: 172.21.15.101, port 22 2022-07-03T16:50:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=641] Connection lost 2022-07-03T16:50:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=641] Aborting connection 2022-07-03T16:50:54.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:50:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:50:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:50:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:50:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:50:55.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:54 smithi161 ceph-mon[91579]: pgmap v11153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:55.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:50:55.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:50:55.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:50:55.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:50:55.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:50:55.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:50:55.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:50:55.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:50:55.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:50:55.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:50:55.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:50:55.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:50:55.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:50:55.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:50:55.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:50:55.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:50:55.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:50:55.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:50:55.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:50:55.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:50:55.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:50:55.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:50:55.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:50:55.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:50:55.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:50:55.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:50:55.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:50:55.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:50:55.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:50:55.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:50:55.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:50:55.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:50:55.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:50:55.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:50:55.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:50:55.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:50:55.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:50:55.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:50:55.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:50:55.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:50:55.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:50:55.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:50:55.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:50:55.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:50:55.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:50:55.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:50:55.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=641] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:50:55.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=641] Local address: 172.21.15.161, port 57138 2022-07-03T16:50:55.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=641] Peer address: 172.21.15.101, port 22 2022-07-03T16:50:55.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=641] Connection lost 2022-07-03T16:50:55.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=641] Aborting connection 2022-07-03T16:50:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:54 smithi188 ceph-mon[92425]: pgmap v11153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:50:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:50:55.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:50:55.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:50:55.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:50:55.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:50:55.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:50:55.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:50:55.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:50:55.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:50:55.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:50:55.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:50:55.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:50:55.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:50:55.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:50:55.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:50:55.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:50:55.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:50:55.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:50:55.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:50:55.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:50:55.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:50:55.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:50:55.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:50:55.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:50:55.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:50:55.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:50:55.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:50:55.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:50:55.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:50:55.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:50:55.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:50:55.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:50:55.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:50:55.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:50:55.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:50:55.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:50:55.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:50:55.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:50:55.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:50:55.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:50:55.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:50:55.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:50:55.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:50:55.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:50:55.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:50:55.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:50:55.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=641] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:50:55.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=641] Local address: 172.21.15.161, port 57138 2022-07-03T16:50:55.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=641] Peer address: 172.21.15.101, port 22 2022-07-03T16:50:55.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=641] Connection lost 2022-07-03T16:50:55.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=641] Aborting connection 2022-07-03T16:50:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:56 smithi161 ceph-mon[91579]: pgmap v11154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:56 smithi188 ceph-mon[92425]: pgmap v11154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:50:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:58 smithi161 ceph-mon[91579]: pgmap v11155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:50:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:50:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:50:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:58 smithi188 ceph-mon[92425]: pgmap v11155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:50:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:50:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:51:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:00 smithi161 ceph-mon[91579]: pgmap v11156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:00 smithi188 ceph-mon[92425]: pgmap v11156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:02 smithi161 ceph-mon[91579]: pgmap v11157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:02 smithi188 ceph-mon[92425]: pgmap v11157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:04 smithi161 ceph-mon[91579]: pgmap v11158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:05.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:04 smithi188 ceph-mon[92425]: pgmap v11158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:06 smithi161 ceph-mon[91579]: pgmap v11159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:06 smithi188 ceph-mon[92425]: pgmap v11159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:08 smithi188 ceph-mon[92425]: pgmap v11160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:08 smithi161 ceph-mon[91579]: pgmap v11160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:10 smithi188 ceph-mon[92425]: pgmap v11161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:10 smithi161 ceph-mon[91579]: pgmap v11161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:12 smithi188 ceph-mon[92425]: pgmap v11162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:12 smithi161 ceph-mon[91579]: pgmap v11162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:14 smithi188 ceph-mon[92425]: pgmap v11163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:14 smithi161 ceph-mon[91579]: pgmap v11163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:17.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:16 smithi188 ceph-mon[92425]: pgmap v11164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:16 smithi161 ceph-mon[91579]: pgmap v11164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:18 smithi188 ceph-mon[92425]: pgmap v11165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:18 smithi161 ceph-mon[91579]: pgmap v11165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:20 smithi188 ceph-mon[92425]: pgmap v11166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:20 smithi161 ceph-mon[91579]: pgmap v11166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:22 smithi188 ceph-mon[92425]: pgmap v11167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:22 smithi161 ceph-mon[91579]: pgmap v11167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:24 smithi161 ceph-mon[91579]: pgmap v11168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:24 smithi188 ceph-mon[92425]: pgmap v11168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:26 smithi161 ceph-mon[91579]: pgmap v11169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:26 smithi188 ceph-mon[92425]: pgmap v11169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:28 smithi161 ceph-mon[91579]: pgmap v11170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:28 smithi188 ceph-mon[92425]: pgmap v11170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:30 smithi161 ceph-mon[91579]: pgmap v11171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:30 smithi188 ceph-mon[92425]: pgmap v11171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:32 smithi161 ceph-mon[91579]: pgmap v11172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:32 smithi188 ceph-mon[92425]: pgmap v11172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T16:51:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:34 smithi161 ceph-mon[91579]: pgmap v11173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:34 smithi188 ceph-mon[92425]: pgmap v11173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:36 smithi161 ceph-mon[91579]: pgmap v11174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:36 smithi188 ceph-mon[92425]: pgmap v11174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:38 smithi161 ceph-mon[91579]: pgmap v11175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:38 smithi188 ceph-mon[92425]: pgmap v11175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:40 smithi161 ceph-mon[91579]: pgmap v11176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:40 smithi188 ceph-mon[92425]: pgmap v11176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:42 smithi161 ceph-mon[91579]: pgmap v11177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:42 smithi188 ceph-mon[92425]: pgmap v11177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:44 smithi161 ceph-mon[91579]: pgmap v11178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:44 smithi188 ceph-mon[92425]: pgmap v11178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:46 smithi161 ceph-mon[91579]: pgmap v11179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:46 smithi188 ceph-mon[92425]: pgmap v11179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:48 smithi161 ceph-mon[91579]: pgmap v11180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:48 smithi188 ceph-mon[92425]: pgmap v11180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:50 smithi161 ceph-mon[91579]: pgmap v11181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:50 smithi188 ceph-mon[92425]: pgmap v11181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:52 smithi161 ceph-mon[91579]: pgmap v11182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:52 smithi188 ceph-mon[92425]: pgmap v11182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:51:54.277+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:51:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:51:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:51:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:51:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:51:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:51:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:51:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:51:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:51:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:51:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:51:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:51:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:51:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:51:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:51:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:51:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:51:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:51:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:51:54.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:51:54.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:51:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:51:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:51:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:51:54.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:51:54.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:51:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:51:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:51:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:51:54.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:51:54.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:51:54.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:51:54.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:51:54.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:51:54.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:51:54.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:51:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:51:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:51:54.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:51:54.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:51:54.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:51:54.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:51:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:51:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:51:54.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:51:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=643] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:51:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=643] Local address: 172.21.15.161, port 57142 2022-07-03T16:51:54.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=643] Peer address: 172.21.15.101, port 22 2022-07-03T16:51:54.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=643] Connection lost 2022-07-03T16:51:54.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=643] Aborting connection 2022-07-03T16:51:54.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:51:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: pgmap v11183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:51:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:51:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:51:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:51:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:51:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:51:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:51:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:51:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:51:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:51:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:51:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:51:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:51:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:51:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:51:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:51:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:51:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:51:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:51:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:51:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:51:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:51:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:51:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:51:55.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:51:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:51:55.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:51:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:51:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:51:55.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:51:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:51:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:51:55.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:51:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:51:55.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:51:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:51:55.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:51:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:51:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:51:55.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:51:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:51:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:51:55.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:51:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:51:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:51:55.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:51:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:51:55.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:51:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:51:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=643] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:51:55.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=643] Local address: 172.21.15.161, port 57142 2022-07-03T16:51:55.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=643] Peer address: 172.21.15.101, port 22 2022-07-03T16:51:55.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=643] Connection lost 2022-07-03T16:51:55.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=643] Aborting connection 2022-07-03T16:51:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: pgmap v11183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:51:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:51:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:51:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:51:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:51:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:51:55.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:51:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:51:55.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:51:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:51:55.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:51:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:51:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:51:55.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:51:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:51:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:51:55.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:51:55.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:51:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:51:55.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:51:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:51:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:51:55.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:51:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:51:55.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:51:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:51:55.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:51:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:51:55.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:51:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:51:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:51:55.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:51:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:51:55.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:51:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:51:55.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:51:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:51:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:51:55.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:51:55.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:51:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:51:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:51:55.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:51:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:51:55.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:51:55.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:51:55.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:51:55.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:51:55.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:51:55.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:51:55.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=643] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:51:55.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=643] Local address: 172.21.15.161, port 57142 2022-07-03T16:51:55.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=643] Peer address: 172.21.15.101, port 22 2022-07-03T16:51:55.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=643] Connection lost 2022-07-03T16:51:55.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=643] Aborting connection 2022-07-03T16:51:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:56 smithi161 ceph-mon[91579]: pgmap v11184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:56 smithi188 ceph-mon[92425]: pgmap v11184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:51:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:58 smithi161 ceph-mon[91579]: pgmap v11185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:51:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:51:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:51:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:51:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:58 smithi188 ceph-mon[92425]: pgmap v11185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:51:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:51:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:51:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:51:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:52:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:00 smithi161 ceph-mon[91579]: pgmap v11186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:00 smithi188 ceph-mon[92425]: pgmap v11186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:02 smithi161 ceph-mon[91579]: pgmap v11187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:02 smithi188 ceph-mon[92425]: pgmap v11187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:04 smithi161 ceph-mon[91579]: pgmap v11188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:04 smithi188 ceph-mon[92425]: pgmap v11188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:07 smithi161 ceph-mon[91579]: pgmap v11189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:07 smithi188 ceph-mon[92425]: pgmap v11189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:09 smithi161 ceph-mon[91579]: pgmap v11190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:09 smithi188 ceph-mon[92425]: pgmap v11190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:11 smithi161 ceph-mon[91579]: pgmap v11191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:11 smithi188 ceph-mon[92425]: pgmap v11191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:13 smithi161 ceph-mon[91579]: pgmap v11192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:13 smithi188 ceph-mon[92425]: pgmap v11192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:15 smithi161 ceph-mon[91579]: pgmap v11193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:15 smithi188 ceph-mon[92425]: pgmap v11193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:17 smithi161 ceph-mon[91579]: pgmap v11194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:17 smithi188 ceph-mon[92425]: pgmap v11194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:19 smithi161 ceph-mon[91579]: pgmap v11195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:19 smithi188 ceph-mon[92425]: pgmap v11195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:21 smithi161 ceph-mon[91579]: pgmap v11196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:21 smithi188 ceph-mon[92425]: pgmap v11196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:23 smithi161 ceph-mon[91579]: pgmap v11197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:23 smithi188 ceph-mon[92425]: pgmap v11197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:25 smithi161 ceph-mon[91579]: pgmap v11198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:25 smithi188 ceph-mon[92425]: pgmap v11198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:27 smithi161 ceph-mon[91579]: pgmap v11199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:27 smithi188 ceph-mon[92425]: pgmap v11199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:29 smithi161 ceph-mon[91579]: pgmap v11200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:29 smithi188 ceph-mon[92425]: pgmap v11200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:31 smithi161 ceph-mon[91579]: pgmap v11201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:31 smithi188 ceph-mon[92425]: pgmap v11201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:33 smithi161 ceph-mon[91579]: pgmap v11202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:33 smithi188 ceph-mon[92425]: pgmap v11202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:35 smithi161 ceph-mon[91579]: pgmap v11203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:35 smithi188 ceph-mon[92425]: pgmap v11203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:37 smithi161 ceph-mon[91579]: pgmap v11204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:37 smithi188 ceph-mon[92425]: pgmap v11204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:39 smithi161 ceph-mon[91579]: pgmap v11205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:39 smithi188 ceph-mon[92425]: pgmap v11205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:41 smithi161 ceph-mon[91579]: pgmap v11206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:41 smithi188 ceph-mon[92425]: pgmap v11206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:43 smithi161 ceph-mon[91579]: pgmap v11207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:43 smithi188 ceph-mon[92425]: pgmap v11207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:45 smithi161 ceph-mon[91579]: pgmap v11208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:45 smithi188 ceph-mon[92425]: pgmap v11208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:47 smithi161 ceph-mon[91579]: pgmap v11209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:47 smithi188 ceph-mon[92425]: pgmap v11209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:49 smithi161 ceph-mon[91579]: pgmap v11210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:49 smithi188 ceph-mon[92425]: pgmap v11210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:51 smithi161 ceph-mon[91579]: pgmap v11211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:51 smithi188 ceph-mon[92425]: pgmap v11211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:53 smithi161 ceph-mon[91579]: pgmap v11212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:53 smithi188 ceph-mon[92425]: pgmap v11212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:55.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:52:54.779+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:52:55.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:52:55.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:52:55.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:52:55.123 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:52:55.123 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:52:55.123 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:52:55.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:52:55.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:52:55.124 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:52:55.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:52:55.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:52:55.125 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:52:55.126 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:52:55.126 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:52:55.126 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:52:55.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:52:55.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:52:55.127 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:52:55.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:52:55.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:52:55.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:52:55.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:52:55.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:52:55.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:52:55.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:52:55.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:52:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:52:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:52:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:52:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:52:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:52:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:52:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:52:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:52:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:52:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:52:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:52:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:52:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:52:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:52:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:52:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:52:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:52:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:52:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=645] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:52:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=645] Local address: 172.21.15.161, port 57146 2022-07-03T16:52:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=645] Peer address: 172.21.15.101, port 22 2022-07-03T16:52:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=645] Connection lost 2022-07-03T16:52:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=645] Aborting connection 2022-07-03T16:52:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:52:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:52:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:55 smithi161 ceph-mon[91579]: pgmap v11213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:52:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:52:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:52:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:55 smithi188 ceph-mon[92425]: pgmap v11213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:52:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:52:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:52:55.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:52:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:52:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:52:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:52:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:52:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:52:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:52:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:52:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:52:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:52:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:52:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:52:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:52:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:52:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:52:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:52:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:52:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:52:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:52:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:52:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:52:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:52:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:52:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:52:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:52:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:52:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:52:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:52:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:52:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:52:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:52:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:52:56.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:52:56.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:52:56.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:52:56.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:52:56.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:52:56.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:52:56.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:52:56.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:52:56.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:52:56.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:52:56.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:52:56.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:52:56.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:52:56.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:52:56.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:52:56.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=645] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:52:56.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=645] Local address: 172.21.15.161, port 57146 2022-07-03T16:52:56.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=645] Peer address: 172.21.15.101, port 22 2022-07-03T16:52:56.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=645] Connection lost 2022-07-03T16:52:56.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=645] Aborting connection 2022-07-03T16:52:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:52:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:52:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:52:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:52:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:52:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:52:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:52:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:52:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:52:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:52:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:52:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:52:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:52:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:52:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:52:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:52:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:52:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:52:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:52:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:52:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:52:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:52:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:52:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:52:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:52:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:52:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:52:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:52:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:52:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:52:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:52:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:52:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:52:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:52:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:52:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:52:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:52:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:52:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:52:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:52:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:52:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:52:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:52:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:52:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:52:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:52:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=645] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:52:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=645] Local address: 172.21.15.161, port 57146 2022-07-03T16:52:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=645] Peer address: 172.21.15.101, port 22 2022-07-03T16:52:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=645] Connection lost 2022-07-03T16:52:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=645] Aborting connection 2022-07-03T16:52:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:57 smithi161 ceph-mon[91579]: pgmap v11214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:57 smithi188 ceph-mon[92425]: pgmap v11214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:52:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:59 smithi188 ceph-mon[92425]: pgmap v11215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:52:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:52:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:52:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:52:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:59 smithi161 ceph-mon[91579]: pgmap v11215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:52:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:52:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:52:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:52:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:53:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:01 smithi188 ceph-mon[92425]: pgmap v11216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:01 smithi161 ceph-mon[91579]: pgmap v11216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:03 smithi188 ceph-mon[92425]: pgmap v11217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:03 smithi161 ceph-mon[91579]: pgmap v11217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:05 smithi188 ceph-mon[92425]: pgmap v11218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:05 smithi161 ceph-mon[91579]: pgmap v11218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:07 smithi188 ceph-mon[92425]: pgmap v11219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:07 smithi161 ceph-mon[91579]: pgmap v11219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:09 smithi188 ceph-mon[92425]: pgmap v11220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:09 smithi161 ceph-mon[91579]: pgmap v11220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:11 smithi161 ceph-mon[91579]: pgmap v11221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:11 smithi188 ceph-mon[92425]: pgmap v11221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:13 smithi161 ceph-mon[91579]: pgmap v11222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:13 smithi188 ceph-mon[92425]: pgmap v11222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:15 smithi161 ceph-mon[91579]: pgmap v11223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:15 smithi188 ceph-mon[92425]: pgmap v11223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:17 smithi161 ceph-mon[91579]: pgmap v11224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:17 smithi188 ceph-mon[92425]: pgmap v11224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:19 smithi161 ceph-mon[91579]: pgmap v11225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:19 smithi188 ceph-mon[92425]: pgmap v11225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:21 smithi161 ceph-mon[91579]: pgmap v11226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:21 smithi188 ceph-mon[92425]: pgmap v11226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:23 smithi161 ceph-mon[91579]: pgmap v11227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:23 smithi188 ceph-mon[92425]: pgmap v11227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:25 smithi161 ceph-mon[91579]: pgmap v11228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:25 smithi188 ceph-mon[92425]: pgmap v11228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:27 smithi161 ceph-mon[91579]: pgmap v11229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:27 smithi188 ceph-mon[92425]: pgmap v11229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:29 smithi161 ceph-mon[91579]: pgmap v11230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:29 smithi188 ceph-mon[92425]: pgmap v11230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:31 smithi161 ceph-mon[91579]: pgmap v11231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:31 smithi188 ceph-mon[92425]: pgmap v11231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:33 smithi161 ceph-mon[91579]: pgmap v11232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:33 smithi188 ceph-mon[92425]: pgmap v11232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:35 smithi161 ceph-mon[91579]: pgmap v11233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:35 smithi188 ceph-mon[92425]: pgmap v11233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:37 smithi161 ceph-mon[91579]: pgmap v11234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:37 smithi188 ceph-mon[92425]: pgmap v11234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:39 smithi161 ceph-mon[91579]: pgmap v11235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:39 smithi188 ceph-mon[92425]: pgmap v11235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:41 smithi161 ceph-mon[91579]: pgmap v11236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:41 smithi188 ceph-mon[92425]: pgmap v11236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:43 smithi161 ceph-mon[91579]: pgmap v11237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:43 smithi188 ceph-mon[92425]: pgmap v11237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:45 smithi161 ceph-mon[91579]: pgmap v11238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:45 smithi188 ceph-mon[92425]: pgmap v11238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:47 smithi161 ceph-mon[91579]: pgmap v11239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:47 smithi188 ceph-mon[92425]: pgmap v11239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:49 smithi161 ceph-mon[91579]: pgmap v11240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:49 smithi188 ceph-mon[92425]: pgmap v11240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:51.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:51 smithi161 ceph-mon[91579]: pgmap v11241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:51.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:51 smithi188 ceph-mon[92425]: pgmap v11241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:53 smithi161 ceph-mon[91579]: pgmap v11242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:53 smithi188 ceph-mon[92425]: pgmap v11242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-mon[91579]: pgmap v11243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:53:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:53:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:53:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:53:55.284+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:53:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:53:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:53:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:53:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:53:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:53:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:53:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:53:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:53:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:53:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:53:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:53:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:53:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:53:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:53:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:53:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:53:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:53:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:53:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:53:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:53:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:53:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:53:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:53:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:53:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:53:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:53:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:53:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:53:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:53:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:53:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:53:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:53:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:53:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:53:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:53:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:53:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:53:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:53:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:53:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:53:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:53:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:53:55.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:53:55.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:53:55.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=647] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:53:55.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=647] Local address: 172.21.15.161, port 57150 2022-07-03T16:53:55.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=647] Peer address: 172.21.15.101, port 22 2022-07-03T16:53:55.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=647] Connection lost 2022-07-03T16:53:55.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=647] Aborting connection 2022-07-03T16:53:55.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:53:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:53:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:55 smithi188 ceph-mon[92425]: pgmap v11243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:53:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:53:55.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:53:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:53:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:53:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:53:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:53:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:53:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:53:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:53:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:53:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:53:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:53:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:53:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:53:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:53:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:53:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:53:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:53:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:53:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:53:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:53:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:53:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:53:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:53:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:53:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:53:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:53:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:53:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:53:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:53:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:53:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:53:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:53:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:53:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:53:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:53:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:53:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:53:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:53:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:53:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:53:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:53:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:53:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:53:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:53:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:53:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:53:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:53:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=647] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:53:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=647] Local address: 172.21.15.161, port 57150 2022-07-03T16:53:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=647] Peer address: 172.21.15.101, port 22 2022-07-03T16:53:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=647] Connection lost 2022-07-03T16:53:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=647] Aborting connection 2022-07-03T16:53:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:53:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:53:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:53:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:53:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:53:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:53:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:53:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:53:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:53:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:53:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:53:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:53:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:53:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:53:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:53:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:53:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:53:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:53:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:53:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:53:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:53:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:53:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:53:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:53:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:53:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:53:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:53:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:53:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:53:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:53:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:53:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:53:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:53:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:53:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:53:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:53:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:53:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:53:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:53:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:53:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:53:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:53:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:53:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:53:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:53:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:53:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=647] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:53:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=647] Local address: 172.21.15.161, port 57150 2022-07-03T16:53:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=647] Peer address: 172.21.15.101, port 22 2022-07-03T16:53:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=647] Connection lost 2022-07-03T16:53:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=647] Aborting connection 2022-07-03T16:53:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:57 smithi161 ceph-mon[91579]: pgmap v11244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:57 smithi188 ceph-mon[92425]: pgmap v11244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:53:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:53:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:59 smithi161 ceph-mon[91579]: pgmap v11245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:53:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:53:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:53:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:53:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:59 smithi188 ceph-mon[92425]: pgmap v11245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:53:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:53:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:53:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:53:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:54:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:01 smithi161 ceph-mon[91579]: pgmap v11246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:01 smithi188 ceph-mon[92425]: pgmap v11246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:03 smithi161 ceph-mon[91579]: pgmap v11247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:03 smithi188 ceph-mon[92425]: pgmap v11247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:05 smithi161 ceph-mon[91579]: pgmap v11248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:05 smithi188 ceph-mon[92425]: pgmap v11248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:07 smithi161 ceph-mon[91579]: pgmap v11249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:07 smithi188 ceph-mon[92425]: pgmap v11249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:09 smithi161 ceph-mon[91579]: pgmap v11250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:09 smithi188 ceph-mon[92425]: pgmap v11250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:11 smithi161 ceph-mon[91579]: pgmap v11251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:11 smithi188 ceph-mon[92425]: pgmap v11251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:13 smithi161 ceph-mon[91579]: pgmap v11252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:13 smithi188 ceph-mon[92425]: pgmap v11252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:15 smithi161 ceph-mon[91579]: pgmap v11253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:15 smithi188 ceph-mon[92425]: pgmap v11253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:17 smithi161 ceph-mon[91579]: pgmap v11254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:17 smithi188 ceph-mon[92425]: pgmap v11254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:19 smithi161 ceph-mon[91579]: pgmap v11255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:19 smithi188 ceph-mon[92425]: pgmap v11255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:21 smithi161 ceph-mon[91579]: pgmap v11256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:21 smithi188 ceph-mon[92425]: pgmap v11256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:23 smithi161 ceph-mon[91579]: pgmap v11257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:23 smithi188 ceph-mon[92425]: pgmap v11257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:25 smithi161 ceph-mon[91579]: pgmap v11258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:25 smithi188 ceph-mon[92425]: pgmap v11258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:27 smithi161 ceph-mon[91579]: pgmap v11259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:27 smithi188 ceph-mon[92425]: pgmap v11259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:29 smithi161 ceph-mon[91579]: pgmap v11260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:29 smithi188 ceph-mon[92425]: pgmap v11260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:31 smithi161 ceph-mon[91579]: pgmap v11261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:31 smithi188 ceph-mon[92425]: pgmap v11261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:33 smithi161 ceph-mon[91579]: pgmap v11262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:33 smithi188 ceph-mon[92425]: pgmap v11262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:35 smithi161 ceph-mon[91579]: pgmap v11263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:35 smithi188 ceph-mon[92425]: pgmap v11263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:37 smithi161 ceph-mon[91579]: pgmap v11264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:37 smithi188 ceph-mon[92425]: pgmap v11264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:39 smithi161 ceph-mon[91579]: pgmap v11265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:39 smithi188 ceph-mon[92425]: pgmap v11265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:41 smithi161 ceph-mon[91579]: pgmap v11266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:41 smithi188 ceph-mon[92425]: pgmap v11266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:43 smithi161 ceph-mon[91579]: pgmap v11267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:43 smithi188 ceph-mon[92425]: pgmap v11267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:45 smithi161 ceph-mon[91579]: pgmap v11268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:45 smithi188 ceph-mon[92425]: pgmap v11268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:47 smithi161 ceph-mon[91579]: pgmap v11269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:47 smithi188 ceph-mon[92425]: pgmap v11269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:49 smithi161 ceph-mon[91579]: pgmap v11270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:49 smithi188 ceph-mon[92425]: pgmap v11270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:51 smithi161 ceph-mon[91579]: pgmap v11271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:51 smithi188 ceph-mon[92425]: pgmap v11271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:53 smithi161 ceph-mon[91579]: pgmap v11272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:53 smithi188 ceph-mon[92425]: pgmap v11272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-mon[91579]: pgmap v11273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:55 smithi188 ceph-mon[92425]: pgmap v11273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:54:55.782+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:54:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:54:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:54:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:54:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:54:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:54:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:54:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:54:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:54:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:54:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:54:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:54:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:54:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:54:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:54:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:54:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:54:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:54:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:54:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:54:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:54:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:54:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:54:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:54:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:54:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:54:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:54:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:54:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:54:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:54:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:54:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:54:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:54:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:54:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:54:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:54:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:54:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:54:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:54:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:54:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:54:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:54:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:54:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:54:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:54:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=649] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:54:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=649] Local address: 172.21.15.161, port 57182 2022-07-03T16:54:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=649] Peer address: 172.21.15.101, port 22 2022-07-03T16:54:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=649] Connection lost 2022-07-03T16:54:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=649] Aborting connection 2022-07-03T16:54:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:54:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:54:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:54:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:54:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:54:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:54:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:54:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:54:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:54:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:57 smithi161 ceph-mon[91579]: pgmap v11274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:54:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:54:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:54:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:54:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:54:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:54:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:54:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:54:57.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:54:57.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:54:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:54:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:54:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:54:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:54:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:54:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:54:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:54:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:54:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:54:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:54:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:54:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:54:57.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:54:57.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:54:57.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:54:57.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:54:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:54:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:54:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:54:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:54:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:54:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:54:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:54:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:54:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:54:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:54:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:54:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:54:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:54:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:54:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:54:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:54:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:54:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:54:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:54:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:54:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=649] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:54:57.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=649] Local address: 172.21.15.161, port 57182 2022-07-03T16:54:57.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=649] Peer address: 172.21.15.101, port 22 2022-07-03T16:54:57.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=649] Connection lost 2022-07-03T16:54:57.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=649] Aborting connection 2022-07-03T16:54:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:57 smithi188 ceph-mon[92425]: pgmap v11274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:54:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:54:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:54:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:54:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:54:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:54:57.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:54:57.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:54:57.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:54:57.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:54:57.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:54:57.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:54:57.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:54:57.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:54:57.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:54:57.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:54:57.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:54:57.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:54:57.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:54:57.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:54:57.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:54:57.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:54:57.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:54:57.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:54:57.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:54:57.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:54:57.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:54:57.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:54:57.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:54:57.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:54:57.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:54:57.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:54:57.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:54:57.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:54:57.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:54:57.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:54:57.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:54:57.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:54:57.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:54:57.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:54:57.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:54:57.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:54:57.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:54:57.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:54:57.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:54:57.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:54:57.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:54:57.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=649] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:54:57.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=649] Local address: 172.21.15.161, port 57182 2022-07-03T16:54:57.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=649] Peer address: 172.21.15.101, port 22 2022-07-03T16:54:57.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=649] Connection lost 2022-07-03T16:54:57.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=649] Aborting connection 2022-07-03T16:54:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:59 smithi188 ceph-mon[92425]: pgmap v11275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:54:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:59 smithi161 ceph-mon[91579]: pgmap v11275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:54:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:54:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:54:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:54:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:55:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:00 smithi188 ceph-mon[92425]: pgmap v11276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:00 smithi161 ceph-mon[91579]: pgmap v11276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:02 smithi161 ceph-mon[91579]: pgmap v11277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:02 smithi188 ceph-mon[92425]: pgmap v11277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:04 smithi161 ceph-mon[91579]: pgmap v11278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:04 smithi188 ceph-mon[92425]: pgmap v11278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:06 smithi161 ceph-mon[91579]: pgmap v11279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:06 smithi188 ceph-mon[92425]: pgmap v11279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:08 smithi161 ceph-mon[91579]: pgmap v11280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:08 smithi188 ceph-mon[92425]: pgmap v11280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:10 smithi161 ceph-mon[91579]: pgmap v11281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:10 smithi188 ceph-mon[92425]: pgmap v11281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:12 smithi161 ceph-mon[91579]: pgmap v11282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:12 smithi188 ceph-mon[92425]: pgmap v11282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:14 smithi161 ceph-mon[91579]: pgmap v11283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:14 smithi188 ceph-mon[92425]: pgmap v11283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:16 smithi161 ceph-mon[91579]: pgmap v11284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:16 smithi188 ceph-mon[92425]: pgmap v11284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:18 smithi161 ceph-mon[91579]: pgmap v11285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:18 smithi188 ceph-mon[92425]: pgmap v11285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:20 smithi161 ceph-mon[91579]: pgmap v11286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:21.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:20 smithi188 ceph-mon[92425]: pgmap v11286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:22 smithi161 ceph-mon[91579]: pgmap v11287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:22 smithi188 ceph-mon[92425]: pgmap v11287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:24 smithi161 ceph-mon[91579]: pgmap v11288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:24 smithi188 ceph-mon[92425]: pgmap v11288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:26 smithi161 ceph-mon[91579]: pgmap v11289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:26 smithi188 ceph-mon[92425]: pgmap v11289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:28 smithi161 ceph-mon[91579]: pgmap v11290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:28 smithi188 ceph-mon[92425]: pgmap v11290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:30 smithi161 ceph-mon[91579]: pgmap v11291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:30 smithi188 ceph-mon[92425]: pgmap v11291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:32 smithi161 ceph-mon[91579]: pgmap v11292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:32 smithi188 ceph-mon[92425]: pgmap v11292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:35.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:34 smithi161 ceph-mon[91579]: pgmap v11293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:35.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:34 smithi188 ceph-mon[92425]: pgmap v11293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:37.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:36 smithi161 ceph-mon[91579]: pgmap v11294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:36 smithi188 ceph-mon[92425]: pgmap v11294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:38 smithi161 ceph-mon[91579]: pgmap v11295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:38 smithi188 ceph-mon[92425]: pgmap v11295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:40 smithi161 ceph-mon[91579]: pgmap v11296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:40 smithi188 ceph-mon[92425]: pgmap v11296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:42 smithi161 ceph-mon[91579]: pgmap v11297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:42 smithi188 ceph-mon[92425]: pgmap v11297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:44 smithi161 ceph-mon[91579]: pgmap v11298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:44 smithi188 ceph-mon[92425]: pgmap v11298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:46 smithi161 ceph-mon[91579]: pgmap v11299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:46 smithi188 ceph-mon[92425]: pgmap v11299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:48 smithi161 ceph-mon[91579]: pgmap v11300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:49.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:48 smithi188 ceph-mon[92425]: pgmap v11300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:50 smithi161 ceph-mon[91579]: pgmap v11301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:50 smithi188 ceph-mon[92425]: pgmap v11301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:52 smithi161 ceph-mon[91579]: pgmap v11302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:52 smithi188 ceph-mon[92425]: pgmap v11302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:54 smithi161 ceph-mon[91579]: pgmap v11303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:54 smithi188 ceph-mon[92425]: pgmap v11303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-mon[91579]: pgmap v11304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:55:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:55:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:55:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:55:56.707+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:55:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:55:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:55:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:55:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:55:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:55:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:55:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:55:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:55:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:55:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:55:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:55:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:55:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:55:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:55:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:55:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:55:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:55:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:55:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:55:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:55:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:55:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:55:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:55:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:55:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:55:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:55:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:55:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:55:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:55:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:55:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:55:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:55:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:55:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:55:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:55:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:55:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:55:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:55:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:55:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:55:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:55:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:55:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:55:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:55:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=651] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:55:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=651] Local address: 172.21.15.161, port 57186 2022-07-03T16:55:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=651] Peer address: 172.21.15.101, port 22 2022-07-03T16:55:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=651] Connection lost 2022-07-03T16:55:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=651] Aborting connection 2022-07-03T16:55:57.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:55:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:55:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:56 smithi188 ceph-mon[92425]: pgmap v11304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:55:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:55:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:55:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:55:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:55:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:55:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:55:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:55:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:55:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:55:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:55:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:55:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:55:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:55:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:55:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:55:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:55:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:55:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:55:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:55:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:55:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:55:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:55:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:55:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:55:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:55:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:55:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:55:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:55:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:55:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:55:58.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:55:58.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:55:58.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:55:58.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:55:58.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:55:58.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:55:58.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:55:58.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:55:58.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:55:58.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:55:58.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:55:58.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:55:58.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:55:58.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:55:58.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:55:58.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:55:58.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:55:58.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:55:58.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:55:58.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=651] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:55:58.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=651] Local address: 172.21.15.161, port 57186 2022-07-03T16:55:58.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=651] Peer address: 172.21.15.101, port 22 2022-07-03T16:55:58.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=651] Connection lost 2022-07-03T16:55:58.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=651] Aborting connection 2022-07-03T16:55:58.161 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:55:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:55:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:55:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:55:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:55:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:55:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:55:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:55:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:55:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:55:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:55:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:55:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:55:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:55:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:55:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:55:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:55:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:55:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:55:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:55:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:55:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:55:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:55:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:55:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:55:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:55:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:55:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:55:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:55:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:55:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:55:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:55:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=651] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:55:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=651] Local address: 172.21.15.161, port 57186 2022-07-03T16:55:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=651] Peer address: 172.21.15.101, port 22 2022-07-03T16:55:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=651] Connection lost 2022-07-03T16:55:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=651] Aborting connection 2022-07-03T16:55:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:55:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:58 smithi161 ceph-mon[91579]: pgmap v11305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:55:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:55:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:55:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:58 smithi188 ceph-mon[92425]: pgmap v11305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:55:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:55:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:56:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:00 smithi161 ceph-mon[91579]: pgmap v11306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:00 smithi188 ceph-mon[92425]: pgmap v11306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:02 smithi161 ceph-mon[91579]: pgmap v11307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:02 smithi188 ceph-mon[92425]: pgmap v11307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:04 smithi161 ceph-mon[91579]: pgmap v11308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:04 smithi188 ceph-mon[92425]: pgmap v11308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:06 smithi161 ceph-mon[91579]: pgmap v11309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:06 smithi188 ceph-mon[92425]: pgmap v11309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:08 smithi161 ceph-mon[91579]: pgmap v11310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:08 smithi188 ceph-mon[92425]: pgmap v11310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:10 smithi161 ceph-mon[91579]: pgmap v11311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:10 smithi188 ceph-mon[92425]: pgmap v11311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:12 smithi161 ceph-mon[91579]: pgmap v11312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:12 smithi188 ceph-mon[92425]: pgmap v11312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:14 smithi161 ceph-mon[91579]: pgmap v11313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:14 smithi188 ceph-mon[92425]: pgmap v11313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:17.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:16 smithi161 ceph-mon[91579]: pgmap v11314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:16 smithi188 ceph-mon[92425]: pgmap v11314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:18 smithi161 ceph-mon[91579]: pgmap v11315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:19.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:18 smithi188 ceph-mon[92425]: pgmap v11315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:20 smithi161 ceph-mon[91579]: pgmap v11316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:20 smithi188 ceph-mon[92425]: pgmap v11316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:22 smithi161 ceph-mon[91579]: pgmap v11317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:22 smithi188 ceph-mon[92425]: pgmap v11317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:24 smithi161 ceph-mon[91579]: pgmap v11318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:25.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:24 smithi188 ceph-mon[92425]: pgmap v11318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:27.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:26 smithi161 ceph-mon[91579]: pgmap v11319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:26 smithi188 ceph-mon[92425]: pgmap v11319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:28 smithi188 ceph-mon[92425]: pgmap v11320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:28 smithi161 ceph-mon[91579]: pgmap v11320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:30 smithi188 ceph-mon[92425]: pgmap v11321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:30 smithi161 ceph-mon[91579]: pgmap v11321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:32 smithi188 ceph-mon[92425]: pgmap v11322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:32 smithi161 ceph-mon[91579]: pgmap v11322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:34 smithi188 ceph-mon[92425]: pgmap v11323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:34 smithi161 ceph-mon[91579]: pgmap v11323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:36 smithi188 ceph-mon[92425]: pgmap v11324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:36 smithi161 ceph-mon[91579]: pgmap v11324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:38 smithi188 ceph-mon[92425]: pgmap v11325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:38 smithi161 ceph-mon[91579]: pgmap v11325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:40 smithi188 ceph-mon[92425]: pgmap v11326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:40 smithi161 ceph-mon[91579]: pgmap v11326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:42 smithi188 ceph-mon[92425]: pgmap v11327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:42 smithi161 ceph-mon[91579]: pgmap v11327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:44 smithi161 ceph-mon[91579]: pgmap v11328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:44 smithi188 ceph-mon[92425]: pgmap v11328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:46 smithi161 ceph-mon[91579]: pgmap v11329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:46 smithi188 ceph-mon[92425]: pgmap v11329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:48 smithi161 ceph-mon[91579]: pgmap v11330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:48 smithi188 ceph-mon[92425]: pgmap v11330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:50 smithi161 ceph-mon[91579]: pgmap v11331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:50 smithi188 ceph-mon[92425]: pgmap v11331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:52 smithi161 ceph-mon[91579]: pgmap v11332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:52 smithi188 ceph-mon[92425]: pgmap v11332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:54 smithi161 ceph-mon[91579]: pgmap v11333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:54 smithi188 ceph-mon[92425]: pgmap v11333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:57.213 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:56 smithi161 ceph-mon[91579]: pgmap v11334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:57.222 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:56 smithi188 ceph-mon[92425]: pgmap v11334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:56:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:56:57.210+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:56:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:56:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:56:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:56:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:56:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:56:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:56:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:56:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:56:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:56:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:56:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:56:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:56:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:56:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:56:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:56:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:56:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:56:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:56:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:56:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:56:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:56:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:56:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:56:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:56:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:56:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:56:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:56:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:56:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:56:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:56:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:56:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:56:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:56:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:56:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:56:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:56:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:56:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:56:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:56:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:56:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:56:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:56:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:56:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:56:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=653] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:56:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=653] Local address: 172.21.15.161, port 57190 2022-07-03T16:56:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=653] Peer address: 172.21.15.101, port 22 2022-07-03T16:56:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=653] Connection lost 2022-07-03T16:56:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=653] Aborting connection 2022-07-03T16:56:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:56:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:56:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:56:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:56:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:56:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:56:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:56:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:56:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:56:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:56:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:56:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:56:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:56:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:56:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:56:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:56:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:56:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:56:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:56:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:56:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:56:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:56:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:56:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:56:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:56:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:56:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:56:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:56:58.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:56:58.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:56:58.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:56:58.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:56:58.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:56:58.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:56:58.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:56:58.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:56:58.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:56:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:56:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:56:58.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:56:58.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:56:58.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:56:58.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:56:58.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:56:58.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:56:58.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:56:58.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:56:58.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:56:58.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:56:58.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:56:58.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=653] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:56:58.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=653] Local address: 172.21.15.161, port 57190 2022-07-03T16:56:58.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=653] Peer address: 172.21.15.101, port 22 2022-07-03T16:56:58.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=653] Connection lost 2022-07-03T16:56:58.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=653] Aborting connection 2022-07-03T16:56:58.411 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:56:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:56:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:56:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:56:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:56:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:56:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:56:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:56:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:56:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:56:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:56:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:56:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:56:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:56:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:56:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:56:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:56:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:56:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:56:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:56:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:56:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:56:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:56:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:56:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:56:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:56:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:56:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:56:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:56:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:56:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:56:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:56:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:56:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:56:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:56:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:56:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:56:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:56:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:56:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:56:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:56:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:56:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:56:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:56:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:56:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:56:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:56:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:56:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:56:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=653] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:56:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=653] Local address: 172.21.15.161, port 57190 2022-07-03T16:56:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=653] Peer address: 172.21.15.101, port 22 2022-07-03T16:56:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=653] Connection lost 2022-07-03T16:56:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=653] Aborting connection 2022-07-03T16:56:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:56:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:58 smithi161 ceph-mon[91579]: pgmap v11335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:56:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:56:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:58 smithi188 ceph-mon[92425]: pgmap v11335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:57:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:00 smithi161 ceph-mon[91579]: pgmap v11336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:01.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:00 smithi188 ceph-mon[92425]: pgmap v11336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:02 smithi161 ceph-mon[91579]: pgmap v11337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:02 smithi188 ceph-mon[92425]: pgmap v11337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:04 smithi161 ceph-mon[91579]: pgmap v11338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:04 smithi188 ceph-mon[92425]: pgmap v11338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:06 smithi161 ceph-mon[91579]: pgmap v11339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:06 smithi188 ceph-mon[92425]: pgmap v11339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:08 smithi161 ceph-mon[91579]: pgmap v11340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:08 smithi188 ceph-mon[92425]: pgmap v11340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:10 smithi161 ceph-mon[91579]: pgmap v11341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:10 smithi188 ceph-mon[92425]: pgmap v11341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:12 smithi161 ceph-mon[91579]: pgmap v11342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:12 smithi188 ceph-mon[92425]: pgmap v11342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:14 smithi161 ceph-mon[91579]: pgmap v11343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:14 smithi188 ceph-mon[92425]: pgmap v11343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:16 smithi161 ceph-mon[91579]: pgmap v11344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:16 smithi188 ceph-mon[92425]: pgmap v11344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:18 smithi161 ceph-mon[91579]: pgmap v11345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:18 smithi188 ceph-mon[92425]: pgmap v11345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:20 smithi161 ceph-mon[91579]: pgmap v11346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:20 smithi188 ceph-mon[92425]: pgmap v11346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:22 smithi161 ceph-mon[91579]: pgmap v11347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:22 smithi188 ceph-mon[92425]: pgmap v11347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:24 smithi161 ceph-mon[91579]: pgmap v11348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:24 smithi188 ceph-mon[92425]: pgmap v11348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:27 smithi161 ceph-mon[91579]: pgmap v11349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:27 smithi188 ceph-mon[92425]: pgmap v11349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:29 smithi161 ceph-mon[91579]: pgmap v11350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:29 smithi188 ceph-mon[92425]: pgmap v11350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:31 smithi161 ceph-mon[91579]: pgmap v11351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:31 smithi188 ceph-mon[92425]: pgmap v11351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:33 smithi161 ceph-mon[91579]: pgmap v11352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:33 smithi188 ceph-mon[92425]: pgmap v11352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:35 smithi161 ceph-mon[91579]: pgmap v11353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:35 smithi188 ceph-mon[92425]: pgmap v11353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:37 smithi161 ceph-mon[91579]: pgmap v11354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:37 smithi188 ceph-mon[92425]: pgmap v11354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:39 smithi161 ceph-mon[91579]: pgmap v11355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:39 smithi188 ceph-mon[92425]: pgmap v11355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:41 smithi161 ceph-mon[91579]: pgmap v11356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:41 smithi188 ceph-mon[92425]: pgmap v11356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:43 smithi161 ceph-mon[91579]: pgmap v11357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:43 smithi188 ceph-mon[92425]: pgmap v11357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:45 smithi161 ceph-mon[91579]: pgmap v11358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:45 smithi188 ceph-mon[92425]: pgmap v11358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:47 smithi161 ceph-mon[91579]: pgmap v11359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:47 smithi188 ceph-mon[92425]: pgmap v11359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:49 smithi161 ceph-mon[91579]: pgmap v11360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:49 smithi188 ceph-mon[92425]: pgmap v11360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:51 smithi161 ceph-mon[91579]: pgmap v11361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:51 smithi188 ceph-mon[92425]: pgmap v11361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:53 smithi161 ceph-mon[91579]: pgmap v11362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:53 smithi188 ceph-mon[92425]: pgmap v11362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:55 smithi161 ceph-mon[91579]: pgmap v11363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:55 smithi188 ceph-mon[92425]: pgmap v11363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-mon[91579]: pgmap v11364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:57 smithi188 ceph-mon[92425]: pgmap v11364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:57:58.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:57:57.704+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:57:58.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:57:58.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:57:58.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:57:58.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:57:58.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:57:58.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:57:58.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:57:58.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:57:58.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:57:58.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:57:58.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:57:58.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:57:58.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:57:58.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:57:58.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:57:58.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:57:58.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:57:58.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:57:58.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:57:58.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:57:58.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:57:58.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:57:58.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:57:58.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:57:58.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:57:58.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:57:58.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:57:58.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:57:58.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:57:58.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:57:58.080 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:57:58.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:57:58.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:57:58.081 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:57:58.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:57:58.082 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:57:58.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:57:58.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:57:58.083 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:57:58.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:57:58.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:57:58.084 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:57:58.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:57:58.085 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:57:58.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=655] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:57:58.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=655] Local address: 172.21.15.161, port 57194 2022-07-03T16:57:58.086 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=655] Peer address: 172.21.15.101, port 22 2022-07-03T16:57:58.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=655] Connection lost 2022-07-03T16:57:58.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=655] Aborting connection 2022-07-03T16:57:58.087 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:57:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:57:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:57:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:57:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:57:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:57:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:57:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:57:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:57:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: pgmap v11365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:57:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:57:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:57:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:57:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:57:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:57:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:57:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:57:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:57:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:57:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:57:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:57:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:57:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:57:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:57:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:57:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:57:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:57:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:57:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:57:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:57:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:57:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:57:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:57:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:57:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:57:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:57:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:57:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:57:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:57:59.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:57:59.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:57:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:57:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:57:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:57:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:57:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:57:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:57:59.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:57:59.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:57:59.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:57:59.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:57:59.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:57:59.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:57:59.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:57:59.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:57:59.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=655] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:57:59.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=655] Local address: 172.21.15.161, port 57194 2022-07-03T16:57:59.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=655] Peer address: 172.21.15.101, port 22 2022-07-03T16:57:59.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=655] Connection lost 2022-07-03T16:57:59.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=655] Aborting connection 2022-07-03T16:57:59.410 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:57:59.410 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:57:59.411 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:57:59.411 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:57:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: pgmap v11365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:57:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:57:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:57:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:57:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:57:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:57:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:57:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:57:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:57:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:57:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:57:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:57:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:57:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:57:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:57:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:57:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:57:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:57:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:57:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:57:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:57:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:57:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:57:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:57:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:57:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:57:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:57:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:57:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:57:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:57:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:57:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:57:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:57:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:57:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:57:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:57:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:57:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:57:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:57:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:57:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:57:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:57:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:57:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:57:59.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:57:59.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:57:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:57:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=655] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:57:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=655] Local address: 172.21.15.161, port 57194 2022-07-03T16:57:59.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=655] Peer address: 172.21.15.101, port 22 2022-07-03T16:57:59.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=655] Connection lost 2022-07-03T16:57:59.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=655] Aborting connection 2022-07-03T16:57:59.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:57:59.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:57:59.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:57:59.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:58:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: pgmap v11366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: pgmap v11366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:01.735 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.735 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.735 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T16:58:01.736 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.737 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.738 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.739 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.740 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.741 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.742 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:01.742 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T16:58:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 95042k 2022-07-03T16:58:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 97323852: error parsing value: Value '97323852' is below minimum 939524096 2022-07-03T16:58:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:58:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:58:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 95042k 2022-07-03T16:58:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 97323852: error parsing value: Value '97323852' is below minimum 939524096 2022-07-03T16:58:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 196.2M 2022-07-03T16:58:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 205754572: error parsing value: Value '205754572' is below minimum 939524096 2022-07-03T16:58:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:03 smithi161 ceph-mon[91579]: pgmap v11367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:03 smithi188 ceph-mon[92425]: pgmap v11367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:05 smithi161 ceph-mon[91579]: pgmap v11368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:05 smithi188 ceph-mon[92425]: pgmap v11368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:07 smithi161 ceph-mon[91579]: pgmap v11369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:07 smithi188 ceph-mon[92425]: pgmap v11369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:09 smithi161 ceph-mon[91579]: pgmap v11370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:09 smithi188 ceph-mon[92425]: pgmap v11370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:11 smithi161 ceph-mon[91579]: pgmap v11371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:11 smithi188 ceph-mon[92425]: pgmap v11371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:13 smithi161 ceph-mon[91579]: pgmap v11372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:13 smithi188 ceph-mon[92425]: pgmap v11372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:15 smithi161 ceph-mon[91579]: pgmap v11373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:15 smithi188 ceph-mon[92425]: pgmap v11373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:17 smithi161 ceph-mon[91579]: pgmap v11374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:17 smithi188 ceph-mon[92425]: pgmap v11374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:19 smithi188 ceph-mon[92425]: pgmap v11375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:19 smithi161 ceph-mon[91579]: pgmap v11375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:21 smithi188 ceph-mon[92425]: pgmap v11376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:21 smithi161 ceph-mon[91579]: pgmap v11376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:23 smithi188 ceph-mon[92425]: pgmap v11377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:23 smithi161 ceph-mon[91579]: pgmap v11377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:25 smithi188 ceph-mon[92425]: pgmap v11378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:25 smithi161 ceph-mon[91579]: pgmap v11378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:27 smithi188 ceph-mon[92425]: pgmap v11379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:27 smithi161 ceph-mon[91579]: pgmap v11379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:29 smithi188 ceph-mon[92425]: pgmap v11380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:29 smithi161 ceph-mon[91579]: pgmap v11380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:31 smithi188 ceph-mon[92425]: pgmap v11381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:31 smithi161 ceph-mon[91579]: pgmap v11381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:33 smithi188 ceph-mon[92425]: pgmap v11382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:33 smithi161 ceph-mon[91579]: pgmap v11382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:35 smithi188 ceph-mon[92425]: pgmap v11383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:35 smithi161 ceph-mon[91579]: pgmap v11383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:36 smithi161 ceph-mon[91579]: pgmap v11384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:36 smithi188 ceph-mon[92425]: pgmap v11384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:38 smithi161 ceph-mon[91579]: pgmap v11385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:38 smithi188 ceph-mon[92425]: pgmap v11385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:39.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:40 smithi161 ceph-mon[91579]: pgmap v11386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:40 smithi188 ceph-mon[92425]: pgmap v11386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:42 smithi161 ceph-mon[91579]: pgmap v11387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:42 smithi188 ceph-mon[92425]: pgmap v11387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:45.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:44 smithi161 ceph-mon[91579]: pgmap v11388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:44 smithi188 ceph-mon[92425]: pgmap v11388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:46 smithi161 ceph-mon[91579]: pgmap v11389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:46 smithi188 ceph-mon[92425]: pgmap v11389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:48 smithi161 ceph-mon[91579]: pgmap v11390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:49.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:48 smithi188 ceph-mon[92425]: pgmap v11390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:51.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:50 smithi161 ceph-mon[91579]: pgmap v11391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:50 smithi188 ceph-mon[92425]: pgmap v11391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:52 smithi161 ceph-mon[91579]: pgmap v11392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:53.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:52 smithi188 ceph-mon[92425]: pgmap v11392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:53.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:54 smithi161 ceph-mon[91579]: pgmap v11393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:54 smithi188 ceph-mon[92425]: pgmap v11393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:56 smithi161 ceph-mon[91579]: pgmap v11394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:56 smithi188 ceph-mon[92425]: pgmap v11394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:58:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:58:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:58 smithi161 ceph-mon[91579]: pgmap v11395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:58:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:58:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:58:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:58:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:58 smithi188 ceph-mon[92425]: pgmap v11395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:58:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:58:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:58:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:58:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:59:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-mon[91579]: pgmap v11396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:01.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:59:00.957+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:59:01.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:59:01.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:59:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T16:59:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:59:01.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:59:01.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:59:01.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T16:59:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:59:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T16:59:01.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:59:01.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T16:59:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:59:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:59:01.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T16:59:01.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:59:01.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T16:59:01.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:59:01.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T16:59:01.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:59:01.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T16:59:01.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:59:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:59:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:59:01.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:59:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:59:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T16:59:01.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:59:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:59:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:59:01.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T16:59:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:59:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T16:59:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:59:01.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:59:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:59:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T16:59:01.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:59:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T16:59:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:59:01.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T16:59:01.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:59:01.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T16:59:01.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:59:01.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:59:01.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=657] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:59:01.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=657] Local address: 172.21.15.161, port 57198 2022-07-03T16:59:01.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=657] Peer address: 172.21.15.101, port 22 2022-07-03T16:59:01.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=657] Connection lost 2022-07-03T16:59:01.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=657] Aborting connection 2022-07-03T16:59:01.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 16:59:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T16:59:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:00 smithi188 ceph-mon[92425]: pgmap v11396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:59:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:59:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:59:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:59:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:59:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:59:02.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:59:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T16:59:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:59:02.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:59:02.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:59:02.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T16:59:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:59:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T16:59:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:59:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T16:59:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:59:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:59:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:59:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T16:59:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T16:59:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:59:02.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T16:59:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:59:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T16:59:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:59:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:59:02.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:59:02.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:59:02.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:59:02.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:59:02.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:59:02.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:59:02.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:59:02.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T16:59:02.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:59:02.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T16:59:02.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:59:02.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:59:02.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:59:02.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:59:02.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:59:02.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T16:59:02.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:59:02.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:59:02.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:59:02.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T16:59:02.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:59:02.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:59:02.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=657] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:59:02.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=657] Local address: 172.21.15.161, port 57198 2022-07-03T16:59:02.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=657] Peer address: 172.21.15.101, port 22 2022-07-03T16:59:02.163 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=657] Connection lost 2022-07-03T16:59:02.164 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=657] Aborting connection 2022-07-03T16:59:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T16:59:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T16:59:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T16:59:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T16:59:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T16:59:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:59:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T16:59:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T16:59:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T16:59:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T16:59:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T16:59:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T16:59:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T16:59:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T16:59:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T16:59:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T16:59:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T16:59:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:59:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T16:59:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T16:59:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T16:59:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T16:59:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T16:59:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T16:59:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T16:59:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T16:59:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T16:59:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T16:59:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T16:59:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T16:59:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T16:59:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T16:59:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T16:59:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T16:59:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T16:59:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T16:59:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T16:59:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T16:59:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T16:59:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T16:59:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T16:59:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T16:59:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T16:59:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T16:59:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T16:59:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T16:59:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T16:59:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T16:59:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T16:59:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=657] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T16:59:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=657] Local address: 172.21.15.161, port 57198 2022-07-03T16:59:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=657] Peer address: 172.21.15.101, port 22 2022-07-03T16:59:02.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=657] Connection lost 2022-07-03T16:59:02.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=657] Aborting connection 2022-07-03T16:59:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:02 smithi161 ceph-mon[91579]: pgmap v11397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:02 smithi188 ceph-mon[92425]: pgmap v11397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:05.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:04 smithi161 ceph-mon[91579]: pgmap v11398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:05.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:04 smithi188 ceph-mon[92425]: pgmap v11398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:06 smithi161 ceph-mon[91579]: pgmap v11399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:06 smithi188 ceph-mon[92425]: pgmap v11399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:08 smithi161 ceph-mon[91579]: pgmap v11400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:08 smithi188 ceph-mon[92425]: pgmap v11400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:11.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:10 smithi161 ceph-mon[91579]: pgmap v11401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:10 smithi188 ceph-mon[92425]: pgmap v11401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:12 smithi161 ceph-mon[91579]: pgmap v11402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:12 smithi188 ceph-mon[92425]: pgmap v11402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:15.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:14 smithi161 ceph-mon[91579]: pgmap v11403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:14 smithi188 ceph-mon[92425]: pgmap v11403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:16 smithi161 ceph-mon[91579]: pgmap v11404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:17.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:16 smithi188 ceph-mon[92425]: pgmap v11404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:18 smithi161 ceph-mon[91579]: pgmap v11405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:18 smithi188 ceph-mon[92425]: pgmap v11405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:21.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:20 smithi161 ceph-mon[91579]: pgmap v11406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:20 smithi188 ceph-mon[92425]: pgmap v11406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:22 smithi161 ceph-mon[91579]: pgmap v11407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:22 smithi188 ceph-mon[92425]: pgmap v11407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:24 smithi161 ceph-mon[91579]: pgmap v11408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:24 smithi188 ceph-mon[92425]: pgmap v11408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:26 smithi161 ceph-mon[91579]: pgmap v11409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:26 smithi188 ceph-mon[92425]: pgmap v11409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:28 smithi188 ceph-mon[92425]: pgmap v11410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:28 smithi161 ceph-mon[91579]: pgmap v11410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:30 smithi188 ceph-mon[92425]: pgmap v11411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:30 smithi161 ceph-mon[91579]: pgmap v11411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:32 smithi188 ceph-mon[92425]: pgmap v11412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:32 smithi161 ceph-mon[91579]: pgmap v11412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:35.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:34 smithi188 ceph-mon[92425]: pgmap v11413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:34 smithi161 ceph-mon[91579]: pgmap v11413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:36 smithi188 ceph-mon[92425]: pgmap v11414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:36 smithi161 ceph-mon[91579]: pgmap v11414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:38 smithi188 ceph-mon[92425]: pgmap v11415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:38 smithi161 ceph-mon[91579]: pgmap v11415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:40 smithi188 ceph-mon[92425]: pgmap v11416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:40 smithi161 ceph-mon[91579]: pgmap v11416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:42 smithi188 ceph-mon[92425]: pgmap v11417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:43.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:42 smithi161 ceph-mon[91579]: pgmap v11417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:43.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:44 smithi161 ceph-mon[91579]: pgmap v11418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:44 smithi188 ceph-mon[92425]: pgmap v11418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:46 smithi161 ceph-mon[91579]: pgmap v11419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:46 smithi188 ceph-mon[92425]: pgmap v11419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:48 smithi161 ceph-mon[91579]: pgmap v11420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:48 smithi188 ceph-mon[92425]: pgmap v11420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:50 smithi161 ceph-mon[91579]: pgmap v11421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:50 smithi188 ceph-mon[92425]: pgmap v11421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:52 smithi161 ceph-mon[91579]: pgmap v11422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:52 smithi188 ceph-mon[92425]: pgmap v11422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:54 smithi161 ceph-mon[91579]: pgmap v11423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:54 smithi188 ceph-mon[92425]: pgmap v11423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:56 smithi161 ceph-mon[91579]: pgmap v11424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:56 smithi188 ceph-mon[92425]: pgmap v11424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T16:59:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T16:59:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:58 smithi161 ceph-mon[91579]: pgmap v11425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:59:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:59:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:59:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 16:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:59:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:58 smithi188 ceph-mon[92425]: pgmap v11425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T16:59:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:59:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T16:59:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T16:59:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 16:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:00:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:00 smithi161 ceph-mon[91579]: pgmap v11426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:00:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:00 smithi188 ceph-mon[92425]: pgmap v11426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:00:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:00:01.459+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:00:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:00:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:00:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:00:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:00:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:00:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:00:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:00:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:00:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:00:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:00:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:00:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:00:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:00:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:00:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:00:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:00:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:00:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:00:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:00:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:00:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:00:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:00:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:00:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:00:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:00:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:00:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:00:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:00:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:00:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:00:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:00:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:00:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:00:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:00:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:00:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:00:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:00:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:00:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:00:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:00:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:00:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:00:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:00:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:00:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=659] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:00:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=659] Local address: 172.21.15.161, port 57202 2022-07-03T17:00:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=659] Peer address: 172.21.15.101, port 22 2022-07-03T17:00:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=659] Connection lost 2022-07-03T17:00:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=659] Aborting connection 2022-07-03T17:00:01.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:00:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:00:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:00:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:00:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:00:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:00:02.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:00:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:00:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:00:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:00:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:00:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:00:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:00:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:00:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:00:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:00:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:00:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:00:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:00:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:00:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:00:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:00:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:00:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:00:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:00:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:00:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:00:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:00:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:00:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:00:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:00:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:00:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:00:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:00:03.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:00:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:00:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:00:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:00:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:00:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:00:03.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:00:03.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:00:03.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:00:03.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:00:03.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:00:03.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:00:03.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:00:03.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:00:03.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:00:03.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:00:03.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:00:03.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:00:03.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:00:03.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=659] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:00:03.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=659] Local address: 172.21.15.161, port 57202 2022-07-03T17:00:03.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=659] Peer address: 172.21.15.101, port 22 2022-07-03T17:00:03.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=659] Connection lost 2022-07-03T17:00:03.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=659] Aborting connection 2022-07-03T17:00:03.406 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:02 smithi161 ceph-mon[91579]: pgmap v11427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:03.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:00:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:00:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:00:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:00:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:00:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:00:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:00:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:00:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:00:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:00:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:00:03.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:00:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:00:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:00:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:00:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:00:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:00:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:00:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:00:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:00:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:00:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:00:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:00:03.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:00:03.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:00:03.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:00:03.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:00:03.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:00:03.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:00:03.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:00:03.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:00:03.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:00:03.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:00:03.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:00:03.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:00:03.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:00:03.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:00:03.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:00:03.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:00:03.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:00:03.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:00:03.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:00:03.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:00:03.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:00:03.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:00:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:00:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=659] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:00:03.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=659] Local address: 172.21.15.161, port 57202 2022-07-03T17:00:03.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=659] Peer address: 172.21.15.101, port 22 2022-07-03T17:00:03.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=659] Connection lost 2022-07-03T17:00:03.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=659] Aborting connection 2022-07-03T17:00:03.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:02 smithi188 ceph-mon[92425]: pgmap v11427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:03.445 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:04 smithi161 ceph-mon[91579]: pgmap v11428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:04 smithi188 ceph-mon[92425]: pgmap v11428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:06 smithi161 ceph-mon[91579]: pgmap v11429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:06 smithi188 ceph-mon[92425]: pgmap v11429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:08 smithi161 ceph-mon[91579]: pgmap v11430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:08 smithi188 ceph-mon[92425]: pgmap v11430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:10 smithi161 ceph-mon[91579]: pgmap v11431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:10 smithi188 ceph-mon[92425]: pgmap v11431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:12 smithi161 ceph-mon[91579]: pgmap v11432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:12 smithi188 ceph-mon[92425]: pgmap v11432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:13.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:14 smithi161 ceph-mon[91579]: pgmap v11433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:14 smithi188 ceph-mon[92425]: pgmap v11433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:16 smithi161 ceph-mon[91579]: pgmap v11434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:16 smithi188 ceph-mon[92425]: pgmap v11434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:18 smithi161 ceph-mon[91579]: pgmap v11435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:18 smithi188 ceph-mon[92425]: pgmap v11435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:20 smithi161 ceph-mon[91579]: pgmap v11436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:20 smithi188 ceph-mon[92425]: pgmap v11436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:22 smithi161 ceph-mon[91579]: pgmap v11437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:22 smithi188 ceph-mon[92425]: pgmap v11437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:23.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:24 smithi161 ceph-mon[91579]: pgmap v11438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:24 smithi188 ceph-mon[92425]: pgmap v11438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:26 smithi161 ceph-mon[91579]: pgmap v11439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:26 smithi188 ceph-mon[92425]: pgmap v11439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:29 smithi161 ceph-mon[91579]: pgmap v11440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:29 smithi188 ceph-mon[92425]: pgmap v11440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:31 smithi161 ceph-mon[91579]: pgmap v11441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:31 smithi188 ceph-mon[92425]: pgmap v11441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:33 smithi161 ceph-mon[91579]: pgmap v11442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:33.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:33 smithi188 ceph-mon[92425]: pgmap v11442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:35 smithi161 ceph-mon[91579]: pgmap v11443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:35 smithi188 ceph-mon[92425]: pgmap v11443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:37 smithi161 ceph-mon[91579]: pgmap v11444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:37 smithi188 ceph-mon[92425]: pgmap v11444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:39 smithi161 ceph-mon[91579]: pgmap v11445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:39 smithi188 ceph-mon[92425]: pgmap v11445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:41 smithi161 ceph-mon[91579]: pgmap v11446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:41 smithi188 ceph-mon[92425]: pgmap v11446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:43 smithi161 ceph-mon[91579]: pgmap v11447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:43 smithi188 ceph-mon[92425]: pgmap v11447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:45 smithi161 ceph-mon[91579]: pgmap v11448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:45 smithi188 ceph-mon[92425]: pgmap v11448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:47 smithi161 ceph-mon[91579]: pgmap v11449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:47 smithi188 ceph-mon[92425]: pgmap v11449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:49 smithi161 ceph-mon[91579]: pgmap v11450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:49 smithi188 ceph-mon[92425]: pgmap v11450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:51 smithi161 ceph-mon[91579]: pgmap v11451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:51 smithi188 ceph-mon[92425]: pgmap v11451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:53 smithi161 ceph-mon[91579]: pgmap v11452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:53 smithi188 ceph-mon[92425]: pgmap v11452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:55 smithi161 ceph-mon[91579]: pgmap v11453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:55 smithi188 ceph-mon[92425]: pgmap v11453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:57 smithi161 ceph-mon[91579]: pgmap v11454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:57 smithi188 ceph-mon[92425]: pgmap v11454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:00:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:00:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:59 smithi161 ceph-mon[91579]: pgmap v11455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:00:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:00:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:00:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:00:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:59 smithi188 ceph-mon[92425]: pgmap v11455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:00:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:00:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:00:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:00:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:01:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-mon[91579]: pgmap v11456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:01 smithi188 ceph-mon[92425]: pgmap v11456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:02.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:01:01.965+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:01:02.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:01:02.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:01:02.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:01:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:01:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:01:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:01:02.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:01:02.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:01:02.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:01:02.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:01:02.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:01:02.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:01:02.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:01:02.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:01:02.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:01:02.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:01:02.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:01:02.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:01:02.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:01:02.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:01:02.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:01:02.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:01:02.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:01:02.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:01:02.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:01:02.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:01:02.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:01:02.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:01:02.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:01:02.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:01:02.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:01:02.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:01:02.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:01:02.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:01:02.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:01:02.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:01:02.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:01:02.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:01:02.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:01:02.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:01:02.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:01:02.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:01:02.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:01:02.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:01:02.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=661] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:01:02.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=661] Local address: 172.21.15.161, port 57206 2022-07-03T17:01:02.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=661] Peer address: 172.21.15.101, port 22 2022-07-03T17:01:02.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=661] Connection lost 2022-07-03T17:01:02.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=661] Aborting connection 2022-07-03T17:01:02.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:01:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:01:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: pgmap v11457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:01:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:01:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:01:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:01:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:01:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:01:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:01:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:01:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:01:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:01:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:01:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:01:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:01:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:01:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:01:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:01:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:01:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:01:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:01:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:01:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:01:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:01:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:01:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:01:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:01:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:01:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:01:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:01:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:01:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:01:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:01:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:01:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:01:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:01:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:01:03.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:01:03.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:01:03.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:01:03.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:01:03.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:01:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:01:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:01:03.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:01:03.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:01:03.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:01:03.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:01:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:01:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:01:03.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:01:03.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=661] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:01:03.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=661] Local address: 172.21.15.161, port 57206 2022-07-03T17:01:03.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=661] Peer address: 172.21.15.101, port 22 2022-07-03T17:01:03.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=661] Connection lost 2022-07-03T17:01:03.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=661] Aborting connection 2022-07-03T17:01:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: pgmap v11457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:03.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:01:03.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:01:03.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:01:03.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:01:03.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:01:03.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:01:03.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:01:03.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:01:03.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:01:03.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:01:03.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:01:03.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:01:03.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:01:03.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:01:03.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:01:03.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:01:03.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:01:03.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:01:03.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:01:03.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:01:03.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:01:03.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:01:03.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:01:03.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:01:03.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:01:03.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:01:03.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:01:03.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:01:03.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:01:03.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:01:03.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:01:03.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:01:03.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:01:03.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:01:03.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:01:03.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:01:03.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:01:03.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:01:03.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:01:03.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:01:03.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:01:03.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:01:03.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:01:03.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:01:03.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:01:03.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:01:03.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:01:03.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:01:03.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:01:03.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=661] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:01:03.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=661] Local address: 172.21.15.161, port 57206 2022-07-03T17:01:03.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=661] Peer address: 172.21.15.101, port 22 2022-07-03T17:01:03.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=661] Connection lost 2022-07-03T17:01:03.202 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=661] Aborting connection 2022-07-03T17:01:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:05.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:04 smithi161 ceph-mon[91579]: pgmap v11458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:05.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:04 smithi188 ceph-mon[92425]: pgmap v11458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:06 smithi161 ceph-mon[91579]: pgmap v11459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:06 smithi188 ceph-mon[92425]: pgmap v11459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:08 smithi161 ceph-mon[91579]: pgmap v11460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:09.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:08 smithi188 ceph-mon[92425]: pgmap v11460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:11.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:10 smithi161 ceph-mon[91579]: pgmap v11461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:10 smithi188 ceph-mon[92425]: pgmap v11461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:12 smithi161 ceph-mon[91579]: pgmap v11462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:12 smithi188 ceph-mon[92425]: pgmap v11462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:15.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:14 smithi161 ceph-mon[91579]: pgmap v11463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:14 smithi188 ceph-mon[92425]: pgmap v11463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:16 smithi161 ceph-mon[91579]: pgmap v11464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:17.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:16 smithi188 ceph-mon[92425]: pgmap v11464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:01:19.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:18 smithi161 ceph-mon[91579]: pgmap v11465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:19.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:18 smithi188 ceph-mon[92425]: pgmap v11465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:19.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:20 smithi161 ceph-mon[91579]: pgmap v11466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:21.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:20 smithi188 ceph-mon[92425]: pgmap v11466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:22 smithi161 ceph-mon[91579]: pgmap v11467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:22 smithi188 ceph-mon[92425]: pgmap v11467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:24 smithi161 ceph-mon[91579]: pgmap v11468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:24 smithi188 ceph-mon[92425]: pgmap v11468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:26 smithi161 ceph-mon[91579]: pgmap v11469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:26 smithi188 ceph-mon[92425]: pgmap v11469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:28 smithi188 ceph-mon[92425]: pgmap v11470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:28 smithi161 ceph-mon[91579]: pgmap v11470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:30 smithi188 ceph-mon[92425]: pgmap v11471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:30 smithi161 ceph-mon[91579]: pgmap v11471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:32 smithi188 ceph-mon[92425]: pgmap v11472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:32 smithi161 ceph-mon[91579]: pgmap v11472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:35.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:34 smithi188 ceph-mon[92425]: pgmap v11473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:34 smithi161 ceph-mon[91579]: pgmap v11473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:37.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:36 smithi188 ceph-mon[92425]: pgmap v11474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:36 smithi161 ceph-mon[91579]: pgmap v11474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:38 smithi161 ceph-mon[91579]: pgmap v11475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:38 smithi188 ceph-mon[92425]: pgmap v11475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:40 smithi161 ceph-mon[91579]: pgmap v11476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:40 smithi188 ceph-mon[92425]: pgmap v11476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:42 smithi161 ceph-mon[91579]: pgmap v11477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:42 smithi188 ceph-mon[92425]: pgmap v11477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:44 smithi161 ceph-mon[91579]: pgmap v11478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:44 smithi188 ceph-mon[92425]: pgmap v11478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:46 smithi161 ceph-mon[91579]: pgmap v11479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:46 smithi188 ceph-mon[92425]: pgmap v11479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:48 smithi161 ceph-mon[91579]: pgmap v11480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:48 smithi188 ceph-mon[92425]: pgmap v11480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:50 smithi161 ceph-mon[91579]: pgmap v11481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:50 smithi188 ceph-mon[92425]: pgmap v11481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:52 smithi161 ceph-mon[91579]: pgmap v11482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:52 smithi188 ceph-mon[92425]: pgmap v11482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:54 smithi161 ceph-mon[91579]: pgmap v11483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:54 smithi188 ceph-mon[92425]: pgmap v11483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:56 smithi161 ceph-mon[91579]: pgmap v11484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:56 smithi188 ceph-mon[92425]: pgmap v11484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:01:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:01:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:01:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:58 smithi161 ceph-mon[91579]: pgmap v11485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:01:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:01:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:01:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:01:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:58 smithi188 ceph-mon[92425]: pgmap v11485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:01:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:01:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:01:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:01:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:02:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:00 smithi161 ceph-mon[91579]: pgmap v11486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:00 smithi188 ceph-mon[92425]: pgmap v11486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:02:02.469+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:02:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:02:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:02:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:02:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:02:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:02:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:02:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:02:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:02:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:02:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:02:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:02:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:02:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:02:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:02:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:02:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:02:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:02:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:02:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:02:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:02:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:02:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:02:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:02:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:02:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:02:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:02:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:02:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:02:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:02:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:02:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:02:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:02:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:02:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:02:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:02:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:02:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:02:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:02:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:02:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:02:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:02:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:02:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:02:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:02:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=663] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:02:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=663] Local address: 172.21.15.161, port 57220 2022-07-03T17:02:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=663] Peer address: 172.21.15.101, port 22 2022-07-03T17:02:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=663] Connection lost 2022-07-03T17:02:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=663] Aborting connection 2022-07-03T17:02:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:02:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-mon[91579]: pgmap v11487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:02:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:02:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:02:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:02 smithi188 ceph-mon[92425]: pgmap v11487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:02:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:02:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:02:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:02:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:02:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:02:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:02:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:02:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:02:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:02:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:02:04.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:02:04.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:02:04.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:02:04.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:02:04.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:02:04.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:02:04.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:02:04.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:02:04.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:02:04.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:02:04.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:02:04.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:02:04.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:02:04.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:02:04.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:02:04.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:02:04.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:02:04.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:02:04.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:02:04.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:02:04.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:02:04.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:02:04.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:02:04.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:02:04.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:02:04.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:02:04.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:02:04.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:02:04.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:02:04.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:02:04.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:02:04.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:02:04.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:02:04.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:02:04.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:02:04.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:02:04.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:02:04.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:02:04.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=663] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:02:04.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=663] Local address: 172.21.15.161, port 57220 2022-07-03T17:02:04.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=663] Peer address: 172.21.15.101, port 22 2022-07-03T17:02:04.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=663] Connection lost 2022-07-03T17:02:04.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=663] Aborting connection 2022-07-03T17:02:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:02:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:02:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:02:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:02:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:02:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:02:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:02:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:02:04.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:02:04.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:02:04.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:02:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:02:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:02:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:02:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:02:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:02:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:02:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:02:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:02:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:02:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:02:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:02:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:02:04.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:02:04.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:02:04.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:02:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:02:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:02:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:02:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:02:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:02:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:02:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:02:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:02:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:02:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:02:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:02:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:02:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:02:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:02:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:02:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:02:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:02:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:02:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:02:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:02:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=663] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:02:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=663] Local address: 172.21.15.161, port 57220 2022-07-03T17:02:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=663] Peer address: 172.21.15.101, port 22 2022-07-03T17:02:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=663] Connection lost 2022-07-03T17:02:04.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=663] Aborting connection 2022-07-03T17:02:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:04 smithi161 ceph-mon[91579]: pgmap v11488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:04 smithi188 ceph-mon[92425]: pgmap v11488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:06 smithi161 ceph-mon[91579]: pgmap v11489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:06 smithi188 ceph-mon[92425]: pgmap v11489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:08 smithi161 ceph-mon[91579]: pgmap v11490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:08 smithi188 ceph-mon[92425]: pgmap v11490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:10 smithi161 ceph-mon[91579]: pgmap v11491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:10 smithi188 ceph-mon[92425]: pgmap v11491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:13 smithi161 ceph-mon[91579]: pgmap v11492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:13 smithi188 ceph-mon[92425]: pgmap v11492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:15 smithi161 ceph-mon[91579]: pgmap v11493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:15 smithi188 ceph-mon[92425]: pgmap v11493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:17 smithi161 ceph-mon[91579]: pgmap v11494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:17 smithi188 ceph-mon[92425]: pgmap v11494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:19 smithi161 ceph-mon[91579]: pgmap v11495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:19 smithi188 ceph-mon[92425]: pgmap v11495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:21 smithi161 ceph-mon[91579]: pgmap v11496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:21 smithi188 ceph-mon[92425]: pgmap v11496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:22.424 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:23 smithi161 ceph-mon[91579]: pgmap v11497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:23 smithi188 ceph-mon[92425]: pgmap v11497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:25 smithi161 ceph-mon[91579]: pgmap v11498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:25 smithi188 ceph-mon[92425]: pgmap v11498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:27 smithi161 ceph-mon[91579]: pgmap v11499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:27 smithi188 ceph-mon[92425]: pgmap v11499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:29 smithi161 ceph-mon[91579]: pgmap v11500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:29 smithi188 ceph-mon[92425]: pgmap v11500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:31 smithi161 ceph-mon[91579]: pgmap v11501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:31 smithi188 ceph-mon[92425]: pgmap v11501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:33 smithi161 ceph-mon[91579]: pgmap v11502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:33.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:33 smithi188 ceph-mon[92425]: pgmap v11502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:35 smithi161 ceph-mon[91579]: pgmap v11503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:35 smithi188 ceph-mon[92425]: pgmap v11503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:37 smithi161 ceph-mon[91579]: pgmap v11504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:37 smithi188 ceph-mon[92425]: pgmap v11504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:39 smithi161 ceph-mon[91579]: pgmap v11505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:39 smithi188 ceph-mon[92425]: pgmap v11505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:41 smithi161 ceph-mon[91579]: pgmap v11506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:41 smithi188 ceph-mon[92425]: pgmap v11506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:43 smithi161 ceph-mon[91579]: pgmap v11507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:43 smithi188 ceph-mon[92425]: pgmap v11507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:45 smithi161 ceph-mon[91579]: pgmap v11508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:45 smithi188 ceph-mon[92425]: pgmap v11508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:47 smithi161 ceph-mon[91579]: pgmap v11509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:47 smithi188 ceph-mon[92425]: pgmap v11509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:49 smithi161 ceph-mon[91579]: pgmap v11510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:49 smithi188 ceph-mon[92425]: pgmap v11510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:51 smithi161 ceph-mon[91579]: pgmap v11511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:51 smithi188 ceph-mon[92425]: pgmap v11511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:53 smithi161 ceph-mon[91579]: pgmap v11512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:53 smithi188 ceph-mon[92425]: pgmap v11512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:55 smithi161 ceph-mon[91579]: pgmap v11513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:55 smithi188 ceph-mon[92425]: pgmap v11513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:57 smithi161 ceph-mon[91579]: pgmap v11514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:57 smithi188 ceph-mon[92425]: pgmap v11514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:02:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:02:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:02:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:59 smithi188 ceph-mon[92425]: pgmap v11515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:02:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:59 smithi161 ceph-mon[91579]: pgmap v11515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:02:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:02:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:02:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:02:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:03:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:01 smithi188 ceph-mon[92425]: pgmap v11516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:01 smithi161 ceph-mon[91579]: pgmap v11516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:03 smithi161 ceph-mon[91579]: pgmap v11517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:03:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:03:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:03:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:03 smithi188 ceph-mon[92425]: pgmap v11517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:03:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:03:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:03:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:05 smithi188 ceph-mon[92425]: pgmap v11518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:05 smithi161 ceph-mon[91579]: pgmap v11518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:07 smithi188 ceph-mon[92425]: pgmap v11519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:07 smithi161 ceph-mon[91579]: pgmap v11519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:09 smithi188 ceph-mon[92425]: pgmap v11520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:09 smithi161 ceph-mon[91579]: pgmap v11520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:11 smithi161 ceph-mon[91579]: pgmap v11521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:11 smithi188 ceph-mon[92425]: pgmap v11521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:13 smithi161 ceph-mon[91579]: pgmap v11522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:13 smithi188 ceph-mon[92425]: pgmap v11522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:15 smithi161 ceph-mon[91579]: pgmap v11523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:15 smithi188 ceph-mon[92425]: pgmap v11523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:17 smithi161 ceph-mon[91579]: pgmap v11524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:17 smithi188 ceph-mon[92425]: pgmap v11524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:19 smithi161 ceph-mon[91579]: pgmap v11525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:19 smithi188 ceph-mon[92425]: pgmap v11525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:21 smithi161 ceph-mon[91579]: pgmap v11526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:21 smithi188 ceph-mon[92425]: pgmap v11526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:23 smithi161 ceph-mon[91579]: pgmap v11527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:23 smithi188 ceph-mon[92425]: pgmap v11527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:25 smithi161 ceph-mon[91579]: pgmap v11528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:25 smithi188 ceph-mon[92425]: pgmap v11528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:27 smithi161 ceph-mon[91579]: pgmap v11529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:27 smithi188 ceph-mon[92425]: pgmap v11529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:29 smithi161 ceph-mon[91579]: pgmap v11530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:29 smithi188 ceph-mon[92425]: pgmap v11530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:31 smithi161 ceph-mon[91579]: pgmap v11531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:31 smithi188 ceph-mon[92425]: pgmap v11531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:33 smithi161 ceph-mon[91579]: pgmap v11532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:03:32.967+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:03:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:03:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:03:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:03:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:03:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:03:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:03:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:03:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:03:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:03:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:03:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:03:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:03:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:03:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:03:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:03:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:03:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:03:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:03:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:03:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:03:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:03:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:03:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:03:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:03:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:03:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:03:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:03:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:03:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:03:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:03:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:03:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:03:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:03:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:03:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:03:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:03:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:03:33.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:03:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:03:33.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:03:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:03:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:03:33.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:03:33.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:03:33.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=665] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:03:33.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=665] Local address: 172.21.15.161, port 57260 2022-07-03T17:03:33.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=665] Peer address: 172.21.15.101, port 22 2022-07-03T17:03:33.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=665] Connection lost 2022-07-03T17:03:33.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=665] Aborting connection 2022-07-03T17:03:33.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:03:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:03:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:33 smithi188 ceph-mon[92425]: pgmap v11532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:03:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:03:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:03:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:03:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:03:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:03:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:03:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:03:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:03:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:03:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:03:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:03:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:03:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:03:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:03:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:03:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:03:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:03:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:03:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:03:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:03:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:03:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:03:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:03:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:03:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:03:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:03:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:03:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:03:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:03:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:03:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:03:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:03:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:03:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:03:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:03:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:03:34.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:03:34.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:03:34.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:03:34.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:03:34.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:03:34.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:03:34.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:03:34.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:03:34.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:03:34.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=665] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:03:34.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=665] Local address: 172.21.15.161, port 57260 2022-07-03T17:03:34.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=665] Peer address: 172.21.15.101, port 22 2022-07-03T17:03:34.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=665] Connection lost 2022-07-03T17:03:34.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=665] Aborting connection 2022-07-03T17:03:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:03:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:03:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:03:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:03:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:03:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:03:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:03:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:03:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:03:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:03:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:03:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:03:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:03:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:03:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:03:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:03:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:03:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:03:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:03:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:03:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:03:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:03:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:03:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:03:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:03:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:03:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:03:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:03:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:03:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:03:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:03:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:03:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:03:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:03:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:03:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:03:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:03:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:03:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:03:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:03:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:03:34.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:03:34.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:03:34.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:03:34.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:03:34.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:03:34.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=665] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:03:34.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=665] Local address: 172.21.15.161, port 57260 2022-07-03T17:03:34.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=665] Peer address: 172.21.15.101, port 22 2022-07-03T17:03:34.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=665] Connection lost 2022-07-03T17:03:34.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=665] Aborting connection 2022-07-03T17:03:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:35 smithi161 ceph-mon[91579]: pgmap v11533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:35 smithi188 ceph-mon[92425]: pgmap v11533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:37 smithi161 ceph-mon[91579]: pgmap v11534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:37 smithi188 ceph-mon[92425]: pgmap v11534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:39 smithi161 ceph-mon[91579]: pgmap v11535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:39 smithi188 ceph-mon[92425]: pgmap v11535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:41 smithi161 ceph-mon[91579]: pgmap v11536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:41 smithi188 ceph-mon[92425]: pgmap v11536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:43 smithi161 ceph-mon[91579]: pgmap v11537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:43 smithi188 ceph-mon[92425]: pgmap v11537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:45 smithi161 ceph-mon[91579]: pgmap v11538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:45 smithi188 ceph-mon[92425]: pgmap v11538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:47 smithi161 ceph-mon[91579]: pgmap v11539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:47 smithi188 ceph-mon[92425]: pgmap v11539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:49 smithi161 ceph-mon[91579]: pgmap v11540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:49 smithi188 ceph-mon[92425]: pgmap v11540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:51 smithi161 ceph-mon[91579]: pgmap v11541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:51 smithi188 ceph-mon[92425]: pgmap v11541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:53 smithi161 ceph-mon[91579]: pgmap v11542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:53 smithi188 ceph-mon[92425]: pgmap v11542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:55 smithi161 ceph-mon[91579]: pgmap v11543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:55 smithi188 ceph-mon[92425]: pgmap v11543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:57 smithi161 ceph-mon[91579]: pgmap v11544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:57 smithi188 ceph-mon[92425]: pgmap v11544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:03:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:03:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:03:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:59 smithi161 ceph-mon[91579]: pgmap v11545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:03:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:03:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:03:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:03:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:59 smithi188 ceph-mon[92425]: pgmap v11545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:03:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:03:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:03:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:03:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:04:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:01 smithi161 ceph-mon[91579]: pgmap v11546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:01 smithi188 ceph-mon[92425]: pgmap v11546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:03 smithi161 ceph-mon[91579]: pgmap v11547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:03 smithi188 ceph-mon[92425]: pgmap v11547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:03.727 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:05 smithi161 ceph-mon[91579]: pgmap v11548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:05 smithi188 ceph-mon[92425]: pgmap v11548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:07 smithi161 ceph-mon[91579]: pgmap v11549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:07.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:07 smithi188 ceph-mon[92425]: pgmap v11549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:09 smithi161 ceph-mon[91579]: pgmap v11550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:09 smithi188 ceph-mon[92425]: pgmap v11550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:11 smithi161 ceph-mon[91579]: pgmap v11551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:11 smithi188 ceph-mon[92425]: pgmap v11551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:13 smithi161 ceph-mon[91579]: pgmap v11552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:13 smithi188 ceph-mon[92425]: pgmap v11552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:15 smithi161 ceph-mon[91579]: pgmap v11553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:15 smithi188 ceph-mon[92425]: pgmap v11553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:17 smithi161 ceph-mon[91579]: pgmap v11554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:17 smithi188 ceph-mon[92425]: pgmap v11554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:19 smithi161 ceph-mon[91579]: pgmap v11555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:19 smithi188 ceph-mon[92425]: pgmap v11555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:21 smithi161 ceph-mon[91579]: pgmap v11556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:21 smithi188 ceph-mon[92425]: pgmap v11556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:23 smithi161 ceph-mon[91579]: pgmap v11557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:23 smithi188 ceph-mon[92425]: pgmap v11557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:25 smithi161 ceph-mon[91579]: pgmap v11558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:25 smithi188 ceph-mon[92425]: pgmap v11558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:27 smithi161 ceph-mon[91579]: pgmap v11559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:27 smithi188 ceph-mon[92425]: pgmap v11559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:29 smithi161 ceph-mon[91579]: pgmap v11560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:29 smithi188 ceph-mon[92425]: pgmap v11560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:31 smithi161 ceph-mon[91579]: pgmap v11561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:31 smithi188 ceph-mon[92425]: pgmap v11561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:33.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:04:33.005+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:04:33.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:04:33.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:04:33.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:04:33.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:04:33.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:04:33.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:04:33.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:04:33.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:04:33.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:04:33.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:04:33.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:04:33.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:04:33.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:04:33.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:04:33.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:04:33.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:04:33.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:04:33.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:04:33.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:04:33.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:04:33.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:04:33.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:04:33.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:04:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:04:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:04:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:04:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:04:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:04:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:04:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:04:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:04:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:04:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:04:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:04:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:04:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:04:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:04:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:04:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:04:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:04:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:04:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:04:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:04:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:04:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=667] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:04:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=667] Local address: 172.21.15.161, port 57288 2022-07-03T17:04:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=667] Peer address: 172.21.15.101, port 22 2022-07-03T17:04:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=667] Connection lost 2022-07-03T17:04:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=667] Aborting connection 2022-07-03T17:04:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:04:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-mon[91579]: pgmap v11562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:04:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:04:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:04:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:33 smithi188 ceph-mon[92425]: pgmap v11562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:04:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:04:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:04:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:04:34.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:04:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:04:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:04:34.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:04:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:04:34.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:04:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:04:34.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:04:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:04:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:04:34.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:04:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:04:34.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:04:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:04:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:04:34.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:04:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:04:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:04:34.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:04:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:04:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:04:34.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:04:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:04:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:04:34.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:04:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:04:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:04:34.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:04:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:04:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:04:34.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:04:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:04:34.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:04:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:04:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:04:34.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:04:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:04:34.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:04:34.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:04:34.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:04:34.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:04:34.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:04:34.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:04:34.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:04:34.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:04:34.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=667] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:04:34.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=667] Local address: 172.21.15.161, port 57288 2022-07-03T17:04:34.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=667] Peer address: 172.21.15.101, port 22 2022-07-03T17:04:34.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=667] Connection lost 2022-07-03T17:04:34.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=667] Aborting connection 2022-07-03T17:04:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:04:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:04:34.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:04:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:04:34.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:04:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:04:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:04:34.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:04:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:04:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:04:34.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:04:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:04:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:04:34.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:04:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:04:34.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:04:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:04:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:04:34.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:04:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:04:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:04:34.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:04:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:04:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:04:34.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:04:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:04:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:04:34.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:04:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:04:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:04:34.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:04:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:04:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:04:34.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:04:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:04:34.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:04:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:04:34.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:04:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:04:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:04:34.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:04:34.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:04:34.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:04:34.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:04:34.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:04:34.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:04:34.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=667] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:04:34.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=667] Local address: 172.21.15.161, port 57288 2022-07-03T17:04:34.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=667] Peer address: 172.21.15.101, port 22 2022-07-03T17:04:34.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=667] Connection lost 2022-07-03T17:04:34.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=667] Aborting connection 2022-07-03T17:04:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:35 smithi161 ceph-mon[91579]: pgmap v11563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:35 smithi188 ceph-mon[92425]: pgmap v11563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:37 smithi161 ceph-mon[91579]: pgmap v11564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:37 smithi188 ceph-mon[92425]: pgmap v11564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:39 smithi188 ceph-mon[92425]: pgmap v11565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:39 smithi161 ceph-mon[91579]: pgmap v11565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:41 smithi188 ceph-mon[92425]: pgmap v11566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:41 smithi161 ceph-mon[91579]: pgmap v11566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:43 smithi188 ceph-mon[92425]: pgmap v11567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:43 smithi161 ceph-mon[91579]: pgmap v11567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:45 smithi188 ceph-mon[92425]: pgmap v11568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:45 smithi161 ceph-mon[91579]: pgmap v11568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:47 smithi188 ceph-mon[92425]: pgmap v11569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:47 smithi161 ceph-mon[91579]: pgmap v11569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:49 smithi188 ceph-mon[92425]: pgmap v11570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:49 smithi161 ceph-mon[91579]: pgmap v11570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:51 smithi188 ceph-mon[92425]: pgmap v11571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:51 smithi161 ceph-mon[91579]: pgmap v11571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:53 smithi188 ceph-mon[92425]: pgmap v11572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:53 smithi161 ceph-mon[91579]: pgmap v11572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:55 smithi161 ceph-mon[91579]: pgmap v11573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:55 smithi188 ceph-mon[92425]: pgmap v11573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:57 smithi161 ceph-mon[91579]: pgmap v11574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:57 smithi188 ceph-mon[92425]: pgmap v11574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:04:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:04:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:59 smithi161 ceph-mon[91579]: pgmap v11575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:04:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:04:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:04:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:04:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:59 smithi188 ceph-mon[92425]: pgmap v11575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:04:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:04:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:04:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:04:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:05:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:01 smithi161 ceph-mon[91579]: pgmap v11576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:01 smithi188 ceph-mon[92425]: pgmap v11576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:03 smithi161 ceph-mon[91579]: pgmap v11577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:03 smithi188 ceph-mon[92425]: pgmap v11577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:05 smithi161 ceph-mon[91579]: pgmap v11578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:05 smithi188 ceph-mon[92425]: pgmap v11578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:07 smithi161 ceph-mon[91579]: pgmap v11579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:07 smithi188 ceph-mon[92425]: pgmap v11579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:08.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:09 smithi161 ceph-mon[91579]: pgmap v11580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:09 smithi188 ceph-mon[92425]: pgmap v11580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:11 smithi161 ceph-mon[91579]: pgmap v11581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:11 smithi188 ceph-mon[92425]: pgmap v11581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:13 smithi161 ceph-mon[91579]: pgmap v11582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:13 smithi188 ceph-mon[92425]: pgmap v11582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:13.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:15 smithi161 ceph-mon[91579]: pgmap v11583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:15 smithi188 ceph-mon[92425]: pgmap v11583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:16.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:17 smithi161 ceph-mon[91579]: pgmap v11584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:17 smithi188 ceph-mon[92425]: pgmap v11584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:18.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:19 smithi161 ceph-mon[91579]: pgmap v11585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:19 smithi188 ceph-mon[92425]: pgmap v11585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:20 smithi161 ceph-mon[91579]: pgmap v11586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:20 smithi188 ceph-mon[92425]: pgmap v11586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:22 smithi161 ceph-mon[91579]: pgmap v11587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:22 smithi188 ceph-mon[92425]: pgmap v11587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:24 smithi161 ceph-mon[91579]: pgmap v11588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:24 smithi188 ceph-mon[92425]: pgmap v11588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:26 smithi161 ceph-mon[91579]: pgmap v11589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:26 smithi188 ceph-mon[92425]: pgmap v11589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:28 smithi161 ceph-mon[91579]: pgmap v11590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:28 smithi188 ceph-mon[92425]: pgmap v11590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:30 smithi161 ceph-mon[91579]: pgmap v11591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:31.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:30 smithi188 ceph-mon[92425]: pgmap v11591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:32 smithi188 ceph-mon[92425]: pgmap v11592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:33.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:32 smithi161 ceph-mon[91579]: pgmap v11592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:33.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:05:33.501+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:05:33.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:05:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:05:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:05:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:05:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:05:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:05:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:05:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:05:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:05:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:05:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:05:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:05:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:05:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:05:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:05:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:05:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:05:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:05:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:05:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:05:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:05:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:05:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:05:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:05:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:05:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:05:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:05:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:05:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:05:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:05:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:05:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:05:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:05:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:05:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:05:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:05:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:05:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:05:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:05:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:05:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:05:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:05:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:05:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:05:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=669] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:05:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=669] Local address: 172.21.15.161, port 57316 2022-07-03T17:05:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=669] Peer address: 172.21.15.101, port 22 2022-07-03T17:05:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=669] Connection lost 2022-07-03T17:05:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=669] Aborting connection 2022-07-03T17:05:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:05:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:05:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:05:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:05:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:05:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:05:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:05:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:05:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:05:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:05:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:05:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:05:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:05:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:05:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:05:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:05:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:05:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:05:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:05:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:05:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:05:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:05:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:05:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:05:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:05:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:05:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:05:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:05:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:05:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:05:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:05:35.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:05:35.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:05:35.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:05:35.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:05:35.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:05:35.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:05:35.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:05:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:05:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:05:35.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:05:35.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:05:35.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:05:35.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:05:35.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:05:35.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:05:35.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:05:35.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:05:35.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:05:35.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:05:35.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:05:35.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:05:35.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:05:35.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=669] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:05:35.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=669] Local address: 172.21.15.161, port 57316 2022-07-03T17:05:35.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=669] Peer address: 172.21.15.101, port 22 2022-07-03T17:05:35.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=669] Connection lost 2022-07-03T17:05:35.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=669] Aborting connection 2022-07-03T17:05:35.415 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: pgmap v11593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:35.415 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.416 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.416 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.417 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:05:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:05:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:05:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:05:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:05:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:05:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:05:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:05:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:05:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:05:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:05:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:05:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:05:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:05:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:05:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:05:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:05:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:05:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:05:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:05:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:05:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:05:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:05:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:05:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:05:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:05:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:05:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:05:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:05:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:05:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:05:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:05:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:05:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:05:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:05:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:05:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:05:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:05:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:05:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:05:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:05:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:05:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:05:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:05:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:05:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:05:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=669] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:05:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=669] Local address: 172.21.15.161, port 57316 2022-07-03T17:05:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=669] Peer address: 172.21.15.101, port 22 2022-07-03T17:05:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=669] Connection lost 2022-07-03T17:05:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=669] Aborting connection 2022-07-03T17:05:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: pgmap v11593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:05:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:36 smithi161 ceph-mon[91579]: pgmap v11594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:36 smithi188 ceph-mon[92425]: pgmap v11594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:38 smithi161 ceph-mon[91579]: pgmap v11595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:38 smithi188 ceph-mon[92425]: pgmap v11595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:40 smithi161 ceph-mon[91579]: pgmap v11596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:40 smithi188 ceph-mon[92425]: pgmap v11596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:42 smithi161 ceph-mon[91579]: pgmap v11597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:42 smithi188 ceph-mon[92425]: pgmap v11597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:44 smithi161 ceph-mon[91579]: pgmap v11598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:44 smithi188 ceph-mon[92425]: pgmap v11598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:46 smithi161 ceph-mon[91579]: pgmap v11599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:46 smithi188 ceph-mon[92425]: pgmap v11599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:48 smithi161 ceph-mon[91579]: pgmap v11600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:48 smithi188 ceph-mon[92425]: pgmap v11600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:50 smithi161 ceph-mon[91579]: pgmap v11601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:50 smithi188 ceph-mon[92425]: pgmap v11601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:52 smithi161 ceph-mon[91579]: pgmap v11602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:52 smithi188 ceph-mon[92425]: pgmap v11602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:54 smithi161 ceph-mon[91579]: pgmap v11603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:54 smithi188 ceph-mon[92425]: pgmap v11603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:56 smithi161 ceph-mon[91579]: pgmap v11604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:56 smithi188 ceph-mon[92425]: pgmap v11604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:05:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:05:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:05:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:58 smithi161 ceph-mon[91579]: pgmap v11605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:05:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:05:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:05:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:05:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:58 smithi188 ceph-mon[92425]: pgmap v11605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:05:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:05:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:05:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:05:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:06:01.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:00 smithi161 ceph-mon[91579]: pgmap v11606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:01.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:00 smithi188 ceph-mon[92425]: pgmap v11606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:02 smithi161 ceph-mon[91579]: pgmap v11607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:02 smithi188 ceph-mon[92425]: pgmap v11607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:04 smithi161 ceph-mon[91579]: pgmap v11608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:04 smithi188 ceph-mon[92425]: pgmap v11608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:07 smithi161 ceph-mon[91579]: pgmap v11609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:07 smithi188 ceph-mon[92425]: pgmap v11609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:09 smithi161 ceph-mon[91579]: pgmap v11610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:09 smithi188 ceph-mon[92425]: pgmap v11610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:11 smithi161 ceph-mon[91579]: pgmap v11611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:11 smithi188 ceph-mon[92425]: pgmap v11611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:13 smithi161 ceph-mon[91579]: pgmap v11612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:13 smithi188 ceph-mon[92425]: pgmap v11612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:13.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:15 smithi161 ceph-mon[91579]: pgmap v11613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:15 smithi188 ceph-mon[92425]: pgmap v11613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:17 smithi161 ceph-mon[91579]: pgmap v11614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:17.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:17 smithi188 ceph-mon[92425]: pgmap v11614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:17.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:19 smithi161 ceph-mon[91579]: pgmap v11615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:19 smithi188 ceph-mon[92425]: pgmap v11615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:21 smithi161 ceph-mon[91579]: pgmap v11616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:21 smithi188 ceph-mon[92425]: pgmap v11616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:23 smithi161 ceph-mon[91579]: pgmap v11617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:23 smithi188 ceph-mon[92425]: pgmap v11617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:25 smithi161 ceph-mon[91579]: pgmap v11618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:25 smithi188 ceph-mon[92425]: pgmap v11618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:27 smithi161 ceph-mon[91579]: pgmap v11619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:27.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:27 smithi188 ceph-mon[92425]: pgmap v11619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:28.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:28.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:29 smithi161 ceph-mon[91579]: pgmap v11620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:29 smithi188 ceph-mon[92425]: pgmap v11620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:31 smithi161 ceph-mon[91579]: pgmap v11621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:31 smithi188 ceph-mon[92425]: pgmap v11621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:33 smithi161 ceph-mon[91579]: pgmap v11622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:33.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:33 smithi188 ceph-mon[92425]: pgmap v11622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:06:34.432+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:06:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:06:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:06:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:06:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:06:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:06:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:06:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:06:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:06:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:06:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:06:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:06:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:06:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:06:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:06:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:06:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:06:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:06:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:06:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:06:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:06:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:06:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:06:34.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:06:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:06:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:06:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:06:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:06:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:06:34.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:06:34.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:06:34.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:06:34.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:06:34.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:06:34.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:06:34.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:06:34.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:06:34.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:06:34.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:06:34.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:06:34.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:06:34.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:06:34.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:06:34.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:06:34.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:06:34.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=671] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:06:34.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=671] Local address: 172.21.15.161, port 57344 2022-07-03T17:06:34.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=671] Peer address: 172.21.15.101, port 22 2022-07-03T17:06:34.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=671] Connection lost 2022-07-03T17:06:34.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=671] Aborting connection 2022-07-03T17:06:34.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:06:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: pgmap v11623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:06:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:06:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:06:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:06:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:06:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:06:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:06:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:06:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:06:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:06:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:06:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:06:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:06:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:06:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:06:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:06:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:06:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:06:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:06:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:06:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:06:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:06:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:06:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:06:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:06:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:06:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:06:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:06:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:06:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:06:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:06:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:06:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:06:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:06:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:06:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:06:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:06:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:06:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:06:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:06:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:06:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:06:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:06:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:06:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:06:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:06:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:06:35.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:06:35.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:06:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=671] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:06:35.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=671] Local address: 172.21.15.161, port 57344 2022-07-03T17:06:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=671] Peer address: 172.21.15.101, port 22 2022-07-03T17:06:35.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=671] Connection lost 2022-07-03T17:06:35.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=671] Aborting connection 2022-07-03T17:06:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: pgmap v11623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:06:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:06:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:06:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:06:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:06:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:06:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:06:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:06:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:06:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:06:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:06:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:06:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:06:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:06:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:06:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:06:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:06:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:06:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:06:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:06:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:06:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:06:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:06:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:06:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:06:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:06:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:06:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:06:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:06:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:06:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:06:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:06:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:06:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:06:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:06:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:06:35.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:06:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:06:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:06:35.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:06:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:06:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:06:35.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:06:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:06:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:06:35.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:06:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:06:35.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:06:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:06:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=671] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:06:35.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=671] Local address: 172.21.15.161, port 57344 2022-07-03T17:06:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=671] Peer address: 172.21.15.101, port 22 2022-07-03T17:06:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=671] Connection lost 2022-07-03T17:06:35.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=671] Aborting connection 2022-07-03T17:06:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:37 smithi161 ceph-mon[91579]: pgmap v11624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:37 smithi188 ceph-mon[92425]: pgmap v11624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:39 smithi161 ceph-mon[91579]: pgmap v11625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:39 smithi188 ceph-mon[92425]: pgmap v11625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:41 smithi161 ceph-mon[91579]: pgmap v11626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:41 smithi188 ceph-mon[92425]: pgmap v11626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:43 smithi161 ceph-mon[91579]: pgmap v11627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:43.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:43 smithi188 ceph-mon[92425]: pgmap v11627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:45 smithi161 ceph-mon[91579]: pgmap v11628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:45 smithi188 ceph-mon[92425]: pgmap v11628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:47 smithi161 ceph-mon[91579]: pgmap v11629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:47 smithi188 ceph-mon[92425]: pgmap v11629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:49 smithi161 ceph-mon[91579]: pgmap v11630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:49 smithi188 ceph-mon[92425]: pgmap v11630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:51 smithi161 ceph-mon[91579]: pgmap v11631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:51 smithi188 ceph-mon[92425]: pgmap v11631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:53 smithi161 ceph-mon[91579]: pgmap v11632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:53 smithi188 ceph-mon[92425]: pgmap v11632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:55 smithi161 ceph-mon[91579]: pgmap v11633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:55 smithi188 ceph-mon[92425]: pgmap v11633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:57 smithi188 ceph-mon[92425]: pgmap v11634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:57 smithi161 ceph-mon[91579]: pgmap v11634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:06:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:06:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:06:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:59 smithi188 ceph-mon[92425]: pgmap v11635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:06:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:06:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:06:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:06:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:59 smithi161 ceph-mon[91579]: pgmap v11635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:06:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:06:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:06:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:06:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:07:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:01 smithi188 ceph-mon[92425]: pgmap v11636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:01 smithi161 ceph-mon[91579]: pgmap v11636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:03 smithi188 ceph-mon[92425]: pgmap v11637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:03 smithi161 ceph-mon[91579]: pgmap v11637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:05 smithi188 ceph-mon[92425]: pgmap v11638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:05 smithi161 ceph-mon[91579]: pgmap v11638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:07 smithi188 ceph-mon[92425]: pgmap v11639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:07 smithi161 ceph-mon[91579]: pgmap v11639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:09 smithi161 ceph-mon[91579]: pgmap v11640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:09 smithi188 ceph-mon[92425]: pgmap v11640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:11 smithi161 ceph-mon[91579]: pgmap v11641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:11 smithi188 ceph-mon[92425]: pgmap v11641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:13 smithi161 ceph-mon[91579]: pgmap v11642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:13 smithi188 ceph-mon[92425]: pgmap v11642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:15 smithi161 ceph-mon[91579]: pgmap v11643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:15 smithi188 ceph-mon[92425]: pgmap v11643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:17 smithi161 ceph-mon[91579]: pgmap v11644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:17 smithi188 ceph-mon[92425]: pgmap v11644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:19 smithi161 ceph-mon[91579]: pgmap v11645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:19 smithi188 ceph-mon[92425]: pgmap v11645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:21 smithi161 ceph-mon[91579]: pgmap v11646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:21 smithi188 ceph-mon[92425]: pgmap v11646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:23 smithi161 ceph-mon[91579]: pgmap v11647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:23 smithi188 ceph-mon[92425]: pgmap v11647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:25 smithi161 ceph-mon[91579]: pgmap v11648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:25 smithi188 ceph-mon[92425]: pgmap v11648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:27 smithi161 ceph-mon[91579]: pgmap v11649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:27 smithi188 ceph-mon[92425]: pgmap v11649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:29 smithi161 ceph-mon[91579]: pgmap v11650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:29 smithi188 ceph-mon[92425]: pgmap v11650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:31 smithi161 ceph-mon[91579]: pgmap v11651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:31 smithi188 ceph-mon[92425]: pgmap v11651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:33 smithi161 ceph-mon[91579]: pgmap v11652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:33 smithi188 ceph-mon[92425]: pgmap v11652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:35.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:07:34.926+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:07:35.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:07:35.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:07:35.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:07:35.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:07:35.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:07:35.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:07:35.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:07:35.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:07:35.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:07:35.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:07:35.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:07:35.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:07:35.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:07:35.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:07:35.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:07:35.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:07:35.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:07:35.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:07:35.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:07:35.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:07:35.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:07:35.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:07:35.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:07:35.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:07:35.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:07:35.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:07:35.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:07:35.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:07:35.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:07:35.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:07:35.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:07:35.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:07:35.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:07:35.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:07:35.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:07:35.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:07:35.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:07:35.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:07:35.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:07:35.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:07:35.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:07:35.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:07:35.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:07:35.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:07:35.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=673] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:07:35.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=673] Local address: 172.21.15.161, port 57372 2022-07-03T17:07:35.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=673] Peer address: 172.21.15.101, port 22 2022-07-03T17:07:35.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=673] Connection lost 2022-07-03T17:07:35.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=673] Aborting connection 2022-07-03T17:07:35.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:07:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:07:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:35 smithi161 ceph-mon[91579]: pgmap v11653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:07:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:07:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:07:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:35 smithi188 ceph-mon[92425]: pgmap v11653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:07:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:07:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:07:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:07:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:07:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:07:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:07:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:07:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:07:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:07:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:07:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:07:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:07:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:07:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:07:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:07:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:07:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:07:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:07:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:07:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:07:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:07:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:07:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:07:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:07:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:07:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:07:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:07:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:07:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:07:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:07:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:07:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:07:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:07:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:07:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:07:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:07:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:07:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:07:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:07:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:07:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:07:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:07:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:07:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:07:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:07:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:07:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:07:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:07:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=673] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:07:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=673] Local address: 172.21.15.161, port 57372 2022-07-03T17:07:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=673] Peer address: 172.21.15.101, port 22 2022-07-03T17:07:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=673] Connection lost 2022-07-03T17:07:36.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=673] Aborting connection 2022-07-03T17:07:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:07:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:07:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:07:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:07:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:07:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:07:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:07:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:07:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:07:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:07:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:07:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:07:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:07:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:07:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:07:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:07:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:07:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:07:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:07:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:07:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:07:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:07:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:07:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:07:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:07:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:07:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:07:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:07:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:07:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:07:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:07:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:07:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:07:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:07:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:07:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:07:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:07:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:07:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:07:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:07:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:07:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:07:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:07:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:07:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:07:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:07:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=673] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:07:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=673] Local address: 172.21.15.161, port 57372 2022-07-03T17:07:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=673] Peer address: 172.21.15.101, port 22 2022-07-03T17:07:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=673] Connection lost 2022-07-03T17:07:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=673] Aborting connection 2022-07-03T17:07:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:37 smithi161 ceph-mon[91579]: pgmap v11654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:37 smithi188 ceph-mon[92425]: pgmap v11654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:39 smithi161 ceph-mon[91579]: pgmap v11655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:39 smithi188 ceph-mon[92425]: pgmap v11655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:41 smithi161 ceph-mon[91579]: pgmap v11656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:41 smithi188 ceph-mon[92425]: pgmap v11656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:43 smithi161 ceph-mon[91579]: pgmap v11657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:43 smithi188 ceph-mon[92425]: pgmap v11657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:45 smithi161 ceph-mon[91579]: pgmap v11658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:45 smithi188 ceph-mon[92425]: pgmap v11658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:47 smithi161 ceph-mon[91579]: pgmap v11659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:47 smithi188 ceph-mon[92425]: pgmap v11659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:49 smithi161 ceph-mon[91579]: pgmap v11660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:49 smithi188 ceph-mon[92425]: pgmap v11660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:51 smithi161 ceph-mon[91579]: pgmap v11661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:51 smithi188 ceph-mon[92425]: pgmap v11661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:53 smithi161 ceph-mon[91579]: pgmap v11662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:53 smithi188 ceph-mon[92425]: pgmap v11662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:55 smithi161 ceph-mon[91579]: pgmap v11663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:55 smithi188 ceph-mon[92425]: pgmap v11663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:57 smithi161 ceph-mon[91579]: pgmap v11664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:57 smithi188 ceph-mon[92425]: pgmap v11664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:07:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:07:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:07:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:59 smithi161 ceph-mon[91579]: pgmap v11665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:07:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:07:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:07:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:07:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:59 smithi188 ceph-mon[92425]: pgmap v11665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:07:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:07:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:07:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:07:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:08:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:01 smithi161 ceph-mon[91579]: pgmap v11666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:01 smithi188 ceph-mon[92425]: pgmap v11666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:03 smithi161 ceph-mon[91579]: pgmap v11667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:03 smithi188 ceph-mon[92425]: pgmap v11667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:05 smithi161 ceph-mon[91579]: pgmap v11668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:05 smithi188 ceph-mon[92425]: pgmap v11668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:07 smithi161 ceph-mon[91579]: pgmap v11669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:07 smithi188 ceph-mon[92425]: pgmap v11669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:09 smithi161 ceph-mon[91579]: pgmap v11670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:09 smithi188 ceph-mon[92425]: pgmap v11670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:11 smithi161 ceph-mon[91579]: pgmap v11671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:11 smithi188 ceph-mon[92425]: pgmap v11671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:13 smithi161 ceph-mon[91579]: pgmap v11672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:13 smithi188 ceph-mon[92425]: pgmap v11672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:15 smithi161 ceph-mon[91579]: pgmap v11673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:15 smithi188 ceph-mon[92425]: pgmap v11673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:17 smithi161 ceph-mon[91579]: pgmap v11674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:17 smithi188 ceph-mon[92425]: pgmap v11674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:19 smithi161 ceph-mon[91579]: pgmap v11675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:19 smithi188 ceph-mon[92425]: pgmap v11675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:21 smithi161 ceph-mon[91579]: pgmap v11676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:21 smithi188 ceph-mon[92425]: pgmap v11676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:23 smithi161 ceph-mon[91579]: pgmap v11677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:23.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:23.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:23 smithi188 ceph-mon[92425]: pgmap v11677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:25 smithi161 ceph-mon[91579]: pgmap v11678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:25 smithi188 ceph-mon[92425]: pgmap v11678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:27 smithi161 ceph-mon[91579]: pgmap v11679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:27 smithi188 ceph-mon[92425]: pgmap v11679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:29 smithi161 ceph-mon[91579]: pgmap v11680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:29 smithi188 ceph-mon[92425]: pgmap v11680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:31 smithi161 ceph-mon[91579]: pgmap v11681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:31 smithi188 ceph-mon[92425]: pgmap v11681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:33 smithi161 ceph-mon[91579]: pgmap v11682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:33 smithi188 ceph-mon[92425]: pgmap v11682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:35 smithi188 ceph-mon[92425]: pgmap v11683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-mon[91579]: pgmap v11683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:08:35.427+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:08:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:08:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:08:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:08:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:08:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:08:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:08:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:08:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:08:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:08:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:08:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:08:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:08:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:08:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:08:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:08:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:08:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:08:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:08:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:08:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:08:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:08:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:08:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:08:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:08:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:08:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:08:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:08:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:08:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:08:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:08:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:08:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:08:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:08:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:08:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:08:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:08:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:08:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:08:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:08:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:08:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:08:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:08:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:08:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:08:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=675] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:08:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=675] Local address: 172.21.15.161, port 57400 2022-07-03T17:08:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=675] Peer address: 172.21.15.101, port 22 2022-07-03T17:08:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=675] Connection lost 2022-07-03T17:08:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=675] Aborting connection 2022-07-03T17:08:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:08:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:08:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:08:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:08:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:08:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:08:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:08:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:08:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:08:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:08:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:08:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:08:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:08:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:08:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:08:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:08:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:08:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:08:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:08:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:08:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:08:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:08:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:08:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:08:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:08:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:08:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:08:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:08:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:08:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:08:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:08:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:08:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:08:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:08:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:08:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:08:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:08:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:08:36.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:08:36.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:08:36.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:08:36.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:08:36.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:08:36.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:08:36.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:08:36.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:08:36.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:08:36.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:08:36.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:08:36.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:08:36.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:08:36.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=675] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:08:36.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=675] Local address: 172.21.15.161, port 57400 2022-07-03T17:08:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=675] Peer address: 172.21.15.101, port 22 2022-07-03T17:08:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=675] Connection lost 2022-07-03T17:08:36.698 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=675] Aborting connection 2022-07-03T17:08:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:08:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:08:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:08:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:08:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:08:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:08:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:08:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:08:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:08:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:08:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:08:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:08:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:08:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:08:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:08:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:08:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:08:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:08:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:08:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:08:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:08:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:08:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:08:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:08:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:08:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:08:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:08:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:08:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:08:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:08:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:08:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:08:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:08:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:08:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:08:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:08:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:08:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:08:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:08:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:08:36.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:08:36.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:08:36.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:08:36.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:08:36.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:08:36.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:08:36.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:08:36.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:08:36.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:08:36.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=675] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:08:36.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=675] Local address: 172.21.15.161, port 57400 2022-07-03T17:08:36.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=675] Peer address: 172.21.15.101, port 22 2022-07-03T17:08:36.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=675] Connection lost 2022-07-03T17:08:36.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=675] Aborting connection 2022-07-03T17:08:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:37 smithi188 ceph-mon[92425]: pgmap v11684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:37 smithi161 ceph-mon[91579]: pgmap v11684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:38.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:08:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: pgmap v11685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: pgmap v11685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:08:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 85605k 2022-07-03T17:08:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 87660175: error parsing value: Value '87660175' is below minimum 939524096 2022-07-03T17:08:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:40 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 192.6M 2022-07-03T17:08:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:40 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 201996476: error parsing value: Value '201996476' is below minimum 939524096 2022-07-03T17:08:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 85605k 2022-07-03T17:08:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 87660175: error parsing value: Value '87660175' is below minimum 939524096 2022-07-03T17:08:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:40 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 192.6M 2022-07-03T17:08:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:40 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 201996476: error parsing value: Value '201996476' is below minimum 939524096 2022-07-03T17:08:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:41 smithi188 ceph-mon[92425]: pgmap v11686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:41 smithi161 ceph-mon[91579]: pgmap v11686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:43 smithi161 ceph-mon[91579]: pgmap v11687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:43 smithi188 ceph-mon[92425]: pgmap v11687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:43.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:45 smithi161 ceph-mon[91579]: pgmap v11688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:45 smithi188 ceph-mon[92425]: pgmap v11688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:47 smithi161 ceph-mon[91579]: pgmap v11689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:47 smithi188 ceph-mon[92425]: pgmap v11689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:49 smithi161 ceph-mon[91579]: pgmap v11690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:49 smithi188 ceph-mon[92425]: pgmap v11690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:51 smithi161 ceph-mon[91579]: pgmap v11691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:51 smithi188 ceph-mon[92425]: pgmap v11691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:53 smithi161 ceph-mon[91579]: pgmap v11692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:53 smithi188 ceph-mon[92425]: pgmap v11692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:55 smithi161 ceph-mon[91579]: pgmap v11693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:55 smithi188 ceph-mon[92425]: pgmap v11693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:57 smithi161 ceph-mon[91579]: pgmap v11694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:57 smithi188 ceph-mon[92425]: pgmap v11694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:08:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:08:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:08:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:59 smithi161 ceph-mon[91579]: pgmap v11695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:08:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:08:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:08:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:08:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:59 smithi188 ceph-mon[92425]: pgmap v11695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:08:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:08:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:08:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:08:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:09:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:01 smithi161 ceph-mon[91579]: pgmap v11696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:01 smithi188 ceph-mon[92425]: pgmap v11696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:03 smithi161 ceph-mon[91579]: pgmap v11697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:03 smithi188 ceph-mon[92425]: pgmap v11697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:05 smithi161 ceph-mon[91579]: pgmap v11698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:05 smithi188 ceph-mon[92425]: pgmap v11698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:07 smithi161 ceph-mon[91579]: pgmap v11699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:07.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:07 smithi188 ceph-mon[92425]: pgmap v11699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:08.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:08.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:09 smithi161 ceph-mon[91579]: pgmap v11700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:09 smithi188 ceph-mon[92425]: pgmap v11700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:11 smithi161 ceph-mon[91579]: pgmap v11701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:11 smithi188 ceph-mon[92425]: pgmap v11701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:13 smithi161 ceph-mon[91579]: pgmap v11702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:13 smithi188 ceph-mon[92425]: pgmap v11702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:15 smithi161 ceph-mon[91579]: pgmap v11703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:15 smithi188 ceph-mon[92425]: pgmap v11703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:17 smithi161 ceph-mon[91579]: pgmap v11704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:17 smithi188 ceph-mon[92425]: pgmap v11704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:17.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:18.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:18.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:19 smithi161 ceph-mon[91579]: pgmap v11705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:19 smithi188 ceph-mon[92425]: pgmap v11705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:21 smithi161 ceph-mon[91579]: pgmap v11706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:21 smithi188 ceph-mon[92425]: pgmap v11706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:23 smithi161 ceph-mon[91579]: pgmap v11707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:23 smithi188 ceph-mon[92425]: pgmap v11707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:23.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:25 smithi161 ceph-mon[91579]: pgmap v11708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:25 smithi188 ceph-mon[92425]: pgmap v11708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:26 smithi161 ceph-mon[91579]: pgmap v11709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:26.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:26 smithi188 ceph-mon[92425]: pgmap v11709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:26.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:28 smithi188 ceph-mon[92425]: pgmap v11710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:28 smithi161 ceph-mon[91579]: pgmap v11710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:30 smithi161 ceph-mon[91579]: pgmap v11711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:30 smithi188 ceph-mon[92425]: pgmap v11711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:32 smithi161 ceph-mon[91579]: pgmap v11712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:32 smithi188 ceph-mon[92425]: pgmap v11712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:34 smithi161 ceph-mon[91579]: pgmap v11713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:34 smithi188 ceph-mon[92425]: pgmap v11713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:36 smithi161 ceph-mon[91579]: pgmap v11714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:36 smithi188 ceph-mon[92425]: pgmap v11714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-mon[91579]: pgmap v11715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:39.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:39.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:09:39.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:09:39.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:09:39.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:09:38.683+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:09:39.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:09:39.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:09:39.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:09:39.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:09:39.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:09:39.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:09:39.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:09:39.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:09:39.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:09:39.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:09:39.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:09:39.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:09:39.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:09:39.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:09:39.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:09:39.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:09:39.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:09:39.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:09:39.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:09:39.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:09:39.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:09:39.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:09:39.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:09:39.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:09:39.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:09:39.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:09:39.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:09:39.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:09:39.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:09:39.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:09:39.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:09:39.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:09:39.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:09:39.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:09:39.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:09:39.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:09:39.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:09:39.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:09:39.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:09:39.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:09:39.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:09:39.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:09:39.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:09:39.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:09:39.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=677] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:09:39.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=677] Local address: 172.21.15.161, port 57456 2022-07-03T17:09:39.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=677] Peer address: 172.21.15.101, port 22 2022-07-03T17:09:39.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=677] Connection lost 2022-07-03T17:09:39.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=677] Aborting connection 2022-07-03T17:09:39.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:09:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:38 smithi188 ceph-mon[92425]: pgmap v11715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:39.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:09:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:09:39.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:09:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:09:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:09:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:09:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:09:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:09:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:09:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:09:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:09:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:09:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:09:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:09:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:09:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:09:40.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:09:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:09:40.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:09:40.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:09:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:09:40.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:09:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:09:40.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:09:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:09:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:09:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:09:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:09:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:09:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:09:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:09:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:09:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:09:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:09:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:09:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:09:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:09:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:09:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:09:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:09:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:09:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:09:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:09:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:09:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:09:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:09:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:09:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:09:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:09:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=677] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:09:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=677] Local address: 172.21.15.161, port 57456 2022-07-03T17:09:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=677] Peer address: 172.21.15.101, port 22 2022-07-03T17:09:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=677] Connection lost 2022-07-03T17:09:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=677] Aborting connection 2022-07-03T17:09:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:09:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:09:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:09:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:09:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:09:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:09:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:09:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:09:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:09:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:09:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:09:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:09:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:09:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:09:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:09:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:09:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:09:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:09:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:09:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:09:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:09:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:09:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:09:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:09:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:09:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:09:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:09:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:09:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:09:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:09:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:09:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:09:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:09:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:09:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:09:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:09:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:09:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=677] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:09:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=677] Local address: 172.21.15.161, port 57456 2022-07-03T17:09:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=677] Peer address: 172.21.15.101, port 22 2022-07-03T17:09:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=677] Connection lost 2022-07-03T17:09:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=677] Aborting connection 2022-07-03T17:09:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:40 smithi161 ceph-mon[91579]: pgmap v11716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:40 smithi188 ceph-mon[92425]: pgmap v11716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:42 smithi161 ceph-mon[91579]: pgmap v11717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:43.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:42 smithi188 ceph-mon[92425]: pgmap v11717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:44 smithi161 ceph-mon[91579]: pgmap v11718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:44 smithi188 ceph-mon[92425]: pgmap v11718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:46 smithi161 ceph-mon[91579]: pgmap v11719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:46 smithi188 ceph-mon[92425]: pgmap v11719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:48 smithi161 ceph-mon[91579]: pgmap v11720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:48 smithi188 ceph-mon[92425]: pgmap v11720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:50 smithi161 ceph-mon[91579]: pgmap v11721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:50 smithi188 ceph-mon[92425]: pgmap v11721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:52 smithi161 ceph-mon[91579]: pgmap v11722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:52 smithi188 ceph-mon[92425]: pgmap v11722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:54 smithi161 ceph-mon[91579]: pgmap v11723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:54 smithi188 ceph-mon[92425]: pgmap v11723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:56 smithi161 ceph-mon[91579]: pgmap v11724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:56 smithi188 ceph-mon[92425]: pgmap v11724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:09:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:09:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: pgmap v11725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:09:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:09:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: pgmap v11725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:09:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:09:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:09:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:09:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:09:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:09:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:10:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:00 smithi161 ceph-mon[91579]: pgmap v11726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:10:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:00 smithi188 ceph-mon[92425]: pgmap v11726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:10:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:02 smithi161 ceph-mon[91579]: pgmap v11727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:02 smithi188 ceph-mon[92425]: pgmap v11727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:04 smithi161 ceph-mon[91579]: pgmap v11728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:04 smithi188 ceph-mon[92425]: pgmap v11728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:06 smithi161 ceph-mon[91579]: pgmap v11729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:06 smithi188 ceph-mon[92425]: pgmap v11729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:08 smithi161 ceph-mon[91579]: pgmap v11730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:08 smithi188 ceph-mon[92425]: pgmap v11730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:11 smithi161 ceph-mon[91579]: pgmap v11731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:11 smithi188 ceph-mon[92425]: pgmap v11731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:13 smithi161 ceph-mon[91579]: pgmap v11732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:13 smithi188 ceph-mon[92425]: pgmap v11732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:15.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:15 smithi188 ceph-mon[92425]: pgmap v11733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:15 smithi161 ceph-mon[91579]: pgmap v11733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:17 smithi161 ceph-mon[91579]: pgmap v11734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:17 smithi188 ceph-mon[92425]: pgmap v11734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:19 smithi161 ceph-mon[91579]: pgmap v11735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:19 smithi188 ceph-mon[92425]: pgmap v11735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:21 smithi161 ceph-mon[91579]: pgmap v11736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:21 smithi188 ceph-mon[92425]: pgmap v11736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:23 smithi161 ceph-mon[91579]: pgmap v11737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:23 smithi188 ceph-mon[92425]: pgmap v11737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:25 smithi161 ceph-mon[91579]: pgmap v11738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:25 smithi188 ceph-mon[92425]: pgmap v11738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:27.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:27 smithi161 ceph-mon[91579]: pgmap v11739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:27.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:27 smithi188 ceph-mon[92425]: pgmap v11739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:27.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:29 smithi161 ceph-mon[91579]: pgmap v11740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:29.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:29 smithi188 ceph-mon[92425]: pgmap v11740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:31 smithi161 ceph-mon[91579]: pgmap v11741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:31 smithi188 ceph-mon[92425]: pgmap v11741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:33 smithi161 ceph-mon[91579]: pgmap v11742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:33 smithi188 ceph-mon[92425]: pgmap v11742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:35 smithi161 ceph-mon[91579]: pgmap v11743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:35 smithi188 ceph-mon[92425]: pgmap v11743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:37 smithi161 ceph-mon[91579]: pgmap v11744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:37 smithi188 ceph-mon[92425]: pgmap v11744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-mon[91579]: pgmap v11745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:10:39.184+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:10:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:10:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:10:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:10:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:10:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:10:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:10:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:10:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:10:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:10:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:10:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:10:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:10:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:10:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:10:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:10:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:10:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:10:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:10:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:10:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:10:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:10:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:10:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:10:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:10:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:10:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:10:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:10:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:10:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:10:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:10:39.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:10:39.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:10:39.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:10:39.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:10:39.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:10:39.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=679] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:10:39.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=679] Local address: 172.21.15.161, port 57484 2022-07-03T17:10:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=679] Peer address: 172.21.15.101, port 22 2022-07-03T17:10:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=679] Connection lost 2022-07-03T17:10:39.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=679] Aborting connection 2022-07-03T17:10:39.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:10:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:39 smithi188 ceph-mon[92425]: pgmap v11745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:40.394 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:10:40.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:10:40.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:10:40.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:10:40.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:10:40.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:10:40.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:10:40.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:10:40.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:10:40.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:10:40.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:10:40.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:10:40.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:10:40.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:10:40.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:10:40.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:10:40.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:10:40.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:10:40.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:10:40.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:10:40.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:10:40.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:10:40.413 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=679] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:10:40.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=679] Local address: 172.21.15.161, port 57484 2022-07-03T17:10:40.414 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=679] Peer address: 172.21.15.101, port 22 2022-07-03T17:10:40.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=679] Connection lost 2022-07-03T17:10:40.415 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=679] Aborting connection 2022-07-03T17:10:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:10:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:10:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:10:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:10:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:10:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:10:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:10:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:10:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:10:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:10:40.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:10:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:10:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:10:40.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:10:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:10:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:10:40.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:10:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:10:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:10:40.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:10:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:10:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:10:40.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:10:40.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:10:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:10:40.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:10:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:10:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:10:40.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:10:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:10:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:10:40.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:10:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:10:40.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:10:40.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=679] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=679] Local address: 172.21.15.161, port 57484 2022-07-03T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=679] Peer address: 172.21.15.101, port 22 2022-07-03T17:10:40.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=679] Connection lost 2022-07-03T17:10:40.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=679] Aborting connection 2022-07-03T17:10:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:41 smithi161 ceph-mon[91579]: pgmap v11746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:41 smithi188 ceph-mon[92425]: pgmap v11746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:43 smithi161 ceph-mon[91579]: pgmap v11747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:43 smithi188 ceph-mon[92425]: pgmap v11747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:45 smithi161 ceph-mon[91579]: pgmap v11748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:45 smithi188 ceph-mon[92425]: pgmap v11748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:47 smithi161 ceph-mon[91579]: pgmap v11749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:47 smithi188 ceph-mon[92425]: pgmap v11749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:10:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:49 smithi161 ceph-mon[91579]: pgmap v11750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:49 smithi188 ceph-mon[92425]: pgmap v11750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:51 smithi161 ceph-mon[91579]: pgmap v11751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:51 smithi188 ceph-mon[92425]: pgmap v11751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:53 smithi161 ceph-mon[91579]: pgmap v11752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:53 smithi188 ceph-mon[92425]: pgmap v11752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:55 smithi161 ceph-mon[91579]: pgmap v11753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:55 smithi188 ceph-mon[92425]: pgmap v11753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:57 smithi161 ceph-mon[91579]: pgmap v11754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:57 smithi188 ceph-mon[92425]: pgmap v11754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:10:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:10:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: pgmap v11755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:10:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:10:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:10:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: pgmap v11755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:10:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:10:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:10:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:10:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:11:01.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:01 smithi161 ceph-mon[91579]: pgmap v11756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:01.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:01 smithi188 ceph-mon[92425]: pgmap v11756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:03 smithi161 ceph-mon[91579]: pgmap v11757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:03 smithi188 ceph-mon[92425]: pgmap v11757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:05 smithi161 ceph-mon[91579]: pgmap v11758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:05 smithi188 ceph-mon[92425]: pgmap v11758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:07 smithi188 ceph-mon[92425]: pgmap v11759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:07 smithi161 ceph-mon[91579]: pgmap v11759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:09 smithi188 ceph-mon[92425]: pgmap v11760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:09 smithi161 ceph-mon[91579]: pgmap v11760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:11 smithi188 ceph-mon[92425]: pgmap v11761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:11 smithi161 ceph-mon[91579]: pgmap v11761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:13 smithi188 ceph-mon[92425]: pgmap v11762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:13 smithi161 ceph-mon[91579]: pgmap v11762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:15 smithi188 ceph-mon[92425]: pgmap v11763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:15 smithi161 ceph-mon[91579]: pgmap v11763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:17 smithi188 ceph-mon[92425]: pgmap v11764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:17 smithi161 ceph-mon[91579]: pgmap v11764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:19 smithi188 ceph-mon[92425]: pgmap v11765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:19 smithi161 ceph-mon[91579]: pgmap v11765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:21 smithi188 ceph-mon[92425]: pgmap v11766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:21 smithi161 ceph-mon[91579]: pgmap v11766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:23 smithi161 ceph-mon[91579]: pgmap v11767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:23 smithi188 ceph-mon[92425]: pgmap v11767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:25 smithi161 ceph-mon[91579]: pgmap v11768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:25 smithi188 ceph-mon[92425]: pgmap v11768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:27 smithi161 ceph-mon[91579]: pgmap v11769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:27 smithi188 ceph-mon[92425]: pgmap v11769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:29 smithi161 ceph-mon[91579]: pgmap v11770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:29 smithi188 ceph-mon[92425]: pgmap v11770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:31 smithi161 ceph-mon[91579]: pgmap v11771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:31 smithi188 ceph-mon[92425]: pgmap v11771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:32.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:33 smithi161 ceph-mon[91579]: pgmap v11772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:33 smithi188 ceph-mon[92425]: pgmap v11772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:35 smithi161 ceph-mon[91579]: pgmap v11773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:35 smithi188 ceph-mon[92425]: pgmap v11773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:37 smithi161 ceph-mon[91579]: pgmap v11774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:37 smithi188 ceph-mon[92425]: pgmap v11774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-mon[91579]: pgmap v11775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:39 smithi188 ceph-mon[92425]: pgmap v11775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:40.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:11:39.695+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:11:40.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:11:40.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:11:40.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:11:40.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:11:40.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:11:40.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:11:40.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:11:40.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:11:40.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:11:40.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:11:40.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:11:40.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:11:40.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:11:40.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:11:40.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:11:40.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:11:40.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:11:40.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:11:40.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:11:40.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:11:40.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:11:40.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:11:40.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:11:40.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:11:40.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:11:40.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:11:40.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:11:40.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:11:40.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:11:40.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:11:40.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:11:40.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:11:40.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:11:40.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:11:40.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:11:40.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:11:40.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:11:40.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:11:40.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:11:40.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:11:40.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:11:40.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:11:40.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:11:40.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:11:40.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=681] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:11:40.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=681] Local address: 172.21.15.161, port 57512 2022-07-03T17:11:40.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=681] Peer address: 172.21.15.101, port 22 2022-07-03T17:11:40.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=681] Connection lost 2022-07-03T17:11:40.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=681] Aborting connection 2022-07-03T17:11:40.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:11:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:11:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:11:40.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:11:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:11:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:11:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:11:41.297 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:11:41.298 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:11:41.298 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:11:41.299 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:11:41.299 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:11:41.299 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:11:41.300 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:11:41.300 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:11:41.301 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:11:41.301 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:11:41.301 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:11:41.302 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:11:41.302 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:11:41.303 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:11:41.303 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:11:41.304 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:11:41.304 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:11:41.304 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:11:41.305 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:11:41.305 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:11:41.306 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:11:41.306 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:11:41.306 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:11:41.307 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:11:41.307 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:11:41.308 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:11:41.308 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:11:41.308 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:11:41.309 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:11:41.309 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:11:41.310 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:11:41.311 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:11:41.311 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:11:41.312 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:11:41.313 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:11:41.313 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:11:41.313 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:11:41.314 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:11:41.314 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:11:41.315 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:11:41.315 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:11:41.315 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:11:41.316 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:11:41.316 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:11:41.317 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:11:41.317 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:11:41.318 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=681] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:11:41.318 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=681] Local address: 172.21.15.161, port 57512 2022-07-03T17:11:41.319 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=681] Peer address: 172.21.15.101, port 22 2022-07-03T17:11:41.319 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=681] Connection lost 2022-07-03T17:11:41.320 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=681] Aborting connection 2022-07-03T17:11:41.320 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:41 smithi161 ceph-mon[91579]: pgmap v11776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:11:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:11:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:11:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:11:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:11:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:11:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:11:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:11:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:11:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:11:41.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:11:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:11:41.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:11:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:11:41.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:11:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:11:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:11:41.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:11:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:11:41.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:11:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:11:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:11:41.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:11:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:11:41.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:11:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:11:41.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:11:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:11:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:11:41.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:11:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:11:41.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:11:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:11:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:11:41.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:11:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:11:41.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:11:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:11:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:11:41.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:11:41.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:11:41.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:11:41.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:11:41.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:11:41.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:11:41.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:11:41.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=681] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:11:41.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=681] Local address: 172.21.15.161, port 57512 2022-07-03T17:11:41.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=681] Peer address: 172.21.15.101, port 22 2022-07-03T17:11:41.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=681] Connection lost 2022-07-03T17:11:41.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=681] Aborting connection 2022-07-03T17:11:41.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:41 smithi188 ceph-mon[92425]: pgmap v11776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:42.622 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:43 smithi161 ceph-mon[91579]: pgmap v11777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:11:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:11:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:43 smithi188 ceph-mon[92425]: pgmap v11777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:11:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:11:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:44 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T17:11:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:44 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T17:11:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:45 smithi161 ceph-mon[91579]: pgmap v11778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:45 smithi188 ceph-mon[92425]: pgmap v11778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:47 smithi161 ceph-mon[91579]: pgmap v11779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:47 smithi188 ceph-mon[92425]: pgmap v11779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:49 smithi161 ceph-mon[91579]: pgmap v11780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:49 smithi188 ceph-mon[92425]: pgmap v11780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:51 smithi161 ceph-mon[91579]: pgmap v11781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:51 smithi188 ceph-mon[92425]: pgmap v11781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:53 smithi161 ceph-mon[91579]: pgmap v11782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:53 smithi188 ceph-mon[92425]: pgmap v11782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:55 smithi161 ceph-mon[91579]: pgmap v11783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:55 smithi188 ceph-mon[92425]: pgmap v11783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:57 smithi161 ceph-mon[91579]: pgmap v11784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:57 smithi188 ceph-mon[92425]: pgmap v11784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:11:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:11:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:59 smithi161 ceph-mon[91579]: pgmap v11785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:11:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:11:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:11:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:11:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:59 smithi188 ceph-mon[92425]: pgmap v11785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:11:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:11:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:11:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:11:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:12:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:01 smithi161 ceph-mon[91579]: pgmap v11786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:01 smithi188 ceph-mon[92425]: pgmap v11786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:03 smithi161 ceph-mon[91579]: pgmap v11787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:03 smithi188 ceph-mon[92425]: pgmap v11787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:05 smithi161 ceph-mon[91579]: pgmap v11788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:05 smithi188 ceph-mon[92425]: pgmap v11788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:07 smithi161 ceph-mon[91579]: pgmap v11789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:07.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:07 smithi188 ceph-mon[92425]: pgmap v11789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:09 smithi161 ceph-mon[91579]: pgmap v11790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:09 smithi188 ceph-mon[92425]: pgmap v11790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:11 smithi161 ceph-mon[91579]: pgmap v11791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:11 smithi188 ceph-mon[92425]: pgmap v11791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:13 smithi161 ceph-mon[91579]: pgmap v11792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:13 smithi188 ceph-mon[92425]: pgmap v11792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:15 smithi161 ceph-mon[91579]: pgmap v11793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:15 smithi188 ceph-mon[92425]: pgmap v11793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:17 smithi161 ceph-mon[91579]: pgmap v11794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:17 smithi188 ceph-mon[92425]: pgmap v11794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:19.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:19 smithi161 ceph-mon[91579]: pgmap v11795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:19.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:19 smithi188 ceph-mon[92425]: pgmap v11795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:21 smithi161 ceph-mon[91579]: pgmap v11796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:21 smithi188 ceph-mon[92425]: pgmap v11796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:23 smithi161 ceph-mon[91579]: pgmap v11797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:23 smithi188 ceph-mon[92425]: pgmap v11797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:25 smithi161 ceph-mon[91579]: pgmap v11798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:25 smithi188 ceph-mon[92425]: pgmap v11798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:27 smithi161 ceph-mon[91579]: pgmap v11799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:27 smithi188 ceph-mon[92425]: pgmap v11799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:29 smithi161 ceph-mon[91579]: pgmap v11800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:29 smithi188 ceph-mon[92425]: pgmap v11800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:31 smithi161 ceph-mon[91579]: pgmap v11801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:31 smithi188 ceph-mon[92425]: pgmap v11801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:33 smithi161 ceph-mon[91579]: pgmap v11802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:33 smithi188 ceph-mon[92425]: pgmap v11802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:35 smithi161 ceph-mon[91579]: pgmap v11803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:35 smithi188 ceph-mon[92425]: pgmap v11803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:37 smithi161 ceph-mon[91579]: pgmap v11804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:37 smithi188 ceph-mon[92425]: pgmap v11804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:39 smithi161 ceph-mon[91579]: pgmap v11805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:39 smithi188 ceph-mon[92425]: pgmap v11805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:41 smithi161 ceph-mon[91579]: pgmap v11806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:41 smithi188 ceph-mon[92425]: pgmap v11806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:43.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:12:42.967+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:12:43.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:12:43.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:12:43.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:12:43.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:12:43.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:12:43.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:12:43.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:12:43.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:12:43.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:12:43.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:12:43.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:12:43.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:12:43.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:12:43.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:12:43.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:12:43.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:12:43.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:12:43.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:12:43.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:12:43.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:12:43.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:12:43.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:12:43.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:12:43.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:12:43.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:12:43.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:12:43.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:12:43.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:12:43.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:12:43.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:12:43.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:12:43.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:12:43.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:12:43.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:12:43.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:12:43.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:12:43.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:12:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:12:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:12:43.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:12:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:12:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:12:43.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:12:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:12:43.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=683] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:12:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=683] Local address: 172.21.15.161, port 57542 2022-07-03T17:12:43.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=683] Peer address: 172.21.15.101, port 22 2022-07-03T17:12:43.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=683] Connection lost 2022-07-03T17:12:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=683] Aborting connection 2022-07-03T17:12:43.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:12:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:43 smithi161 ceph-mon[91579]: pgmap v11807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:12:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:12:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:12:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:43 smithi188 ceph-mon[92425]: pgmap v11807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:43.751 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:12:43.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:12:43.752 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:12:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:12:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:12:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:12:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:12:44.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:12:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:12:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:12:44.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:12:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:12:44.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:12:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:12:44.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:12:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:12:44.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:12:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:12:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:12:44.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:12:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:12:44.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:12:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:12:44.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:12:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:12:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:12:44.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:12:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:12:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:12:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:12:44.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:12:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:12:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:12:44.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:12:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:12:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:12:44.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:12:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:12:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:12:44.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:12:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:12:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:12:44.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:12:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:12:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:12:44.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:12:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:12:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:12:44.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:12:44.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=683] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:12:44.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=683] Local address: 172.21.15.161, port 57542 2022-07-03T17:12:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=683] Peer address: 172.21.15.101, port 22 2022-07-03T17:12:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=683] Connection lost 2022-07-03T17:12:44.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=683] Aborting connection 2022-07-03T17:12:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:12:44.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:12:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:12:44.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:12:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:12:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:12:44.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:12:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:12:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:12:44.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:12:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:12:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:12:44.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:12:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:12:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:12:44.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:12:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:12:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:12:44.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:12:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:12:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:12:44.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:12:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:12:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:12:44.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:12:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:12:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:12:44.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:12:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:12:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:12:44.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:12:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:12:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:12:44.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:12:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:12:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:12:44.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:12:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:12:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:12:44.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:12:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:12:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:12:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:12:44.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:12:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:12:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:12:44.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=683] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:12:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=683] Local address: 172.21.15.161, port 57542 2022-07-03T17:12:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=683] Peer address: 172.21.15.101, port 22 2022-07-03T17:12:44.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=683] Connection lost 2022-07-03T17:12:44.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=683] Aborting connection 2022-07-03T17:12:45.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:45 smithi161 ceph-mon[91579]: pgmap v11808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:45 smithi188 ceph-mon[92425]: pgmap v11808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:47.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:47 smithi161 ceph-mon[91579]: pgmap v11809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:47.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:47 smithi188 ceph-mon[92425]: pgmap v11809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:49 smithi161 ceph-mon[91579]: pgmap v11810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:49 smithi188 ceph-mon[92425]: pgmap v11810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:51 smithi161 ceph-mon[91579]: pgmap v11811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:51 smithi188 ceph-mon[92425]: pgmap v11811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:53 smithi161 ceph-mon[91579]: pgmap v11812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:53 smithi188 ceph-mon[92425]: pgmap v11812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:55 smithi161 ceph-mon[91579]: pgmap v11813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:55 smithi188 ceph-mon[92425]: pgmap v11813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:57 smithi161 ceph-mon[91579]: pgmap v11814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:57 smithi188 ceph-mon[92425]: pgmap v11814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:12:57.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:12:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:59 smithi161 ceph-mon[91579]: pgmap v11815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:12:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:12:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:12:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:12:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:59 smithi188 ceph-mon[92425]: pgmap v11815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:12:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:12:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:12:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:12:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:13:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:01 smithi161 ceph-mon[91579]: pgmap v11816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:01 smithi188 ceph-mon[92425]: pgmap v11816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:03 smithi161 ceph-mon[91579]: pgmap v11817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:03 smithi188 ceph-mon[92425]: pgmap v11817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:05 smithi161 ceph-mon[91579]: pgmap v11818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:05 smithi188 ceph-mon[92425]: pgmap v11818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:07 smithi188 ceph-mon[92425]: pgmap v11819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:07 smithi161 ceph-mon[91579]: pgmap v11819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:09 smithi188 ceph-mon[92425]: pgmap v11820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:09 smithi161 ceph-mon[91579]: pgmap v11820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:11 smithi188 ceph-mon[92425]: pgmap v11821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:11.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:11 smithi161 ceph-mon[91579]: pgmap v11821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:12.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:13.675 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:13 smithi188 ceph-mon[92425]: pgmap v11822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:13 smithi161 ceph-mon[91579]: pgmap v11822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:15 smithi188 ceph-mon[92425]: pgmap v11823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:15 smithi161 ceph-mon[91579]: pgmap v11823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:17 smithi188 ceph-mon[92425]: pgmap v11824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:17 smithi161 ceph-mon[91579]: pgmap v11824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:19 smithi188 ceph-mon[92425]: pgmap v11825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:19 smithi161 ceph-mon[91579]: pgmap v11825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:21 smithi188 ceph-mon[92425]: pgmap v11826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:21 smithi161 ceph-mon[91579]: pgmap v11826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:23 smithi188 ceph-mon[92425]: pgmap v11827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:23 smithi161 ceph-mon[91579]: pgmap v11827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:25 smithi161 ceph-mon[91579]: pgmap v11828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:25 smithi188 ceph-mon[92425]: pgmap v11828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:27 smithi161 ceph-mon[91579]: pgmap v11829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:27 smithi188 ceph-mon[92425]: pgmap v11829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:29 smithi161 ceph-mon[91579]: pgmap v11830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:29 smithi188 ceph-mon[92425]: pgmap v11830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:31 smithi161 ceph-mon[91579]: pgmap v11831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:31 smithi188 ceph-mon[92425]: pgmap v11831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:33 smithi161 ceph-mon[91579]: pgmap v11832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:33 smithi188 ceph-mon[92425]: pgmap v11832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:35 smithi161 ceph-mon[91579]: pgmap v11833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:35 smithi188 ceph-mon[92425]: pgmap v11833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:37 smithi161 ceph-mon[91579]: pgmap v11834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:37 smithi188 ceph-mon[92425]: pgmap v11834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:39 smithi161 ceph-mon[91579]: pgmap v11835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:39 smithi188 ceph-mon[92425]: pgmap v11835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:41 smithi161 ceph-mon[91579]: pgmap v11836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:41 smithi188 ceph-mon[92425]: pgmap v11836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-mon[91579]: pgmap v11837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:43.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:13:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:13:43.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:13:43.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:13:43.479+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:13:43.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:13:43.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:13:43.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:13:43.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:13:43.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:13:43.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:13:43.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:13:43.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:13:43.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:13:43.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:13:43.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:13:43.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:13:43.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:13:43.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:13:43.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:13:43.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:13:43.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:13:43.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:13:43.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:13:43.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:13:43.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:13:43.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:13:43.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:13:43.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:13:43.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:13:43.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:13:43.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:13:43.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:13:43.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:13:43.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:13:43.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:13:43.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:13:43.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:13:43.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:13:43.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:13:43.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:13:43.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:13:43.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:13:43.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:13:43.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:13:43.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:13:43.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:13:43.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:13:43.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:13:43.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=685] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:13:43.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=685] Local address: 172.21.15.161, port 57570 2022-07-03T17:13:43.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=685] Peer address: 172.21.15.101, port 22 2022-07-03T17:13:43.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=685] Connection lost 2022-07-03T17:13:43.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=685] Aborting connection 2022-07-03T17:13:43.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:13:43.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:43 smithi188 ceph-mon[92425]: pgmap v11837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:43.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:13:43.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:13:43.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:13:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:13:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:13:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:13:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:13:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:13:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:13:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:13:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:13:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:13:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:13:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:13:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:13:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:13:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:13:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:13:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:13:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:13:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:13:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:13:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:13:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:13:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:13:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:13:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:13:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:13:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:13:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:13:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:13:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:13:44.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:13:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:13:44.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:13:44.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:13:44.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:13:44.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:13:44.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:13:44.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:13:44.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:13:44.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:13:44.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:13:44.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:13:44.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:13:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:13:44.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:13:44.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:13:44.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:13:44.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=685] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:13:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=685] Local address: 172.21.15.161, port 57570 2022-07-03T17:13:44.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=685] Peer address: 172.21.15.101, port 22 2022-07-03T17:13:44.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=685] Connection lost 2022-07-03T17:13:44.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=685] Aborting connection 2022-07-03T17:13:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:13:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:13:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:13:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:13:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:13:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:13:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:13:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:13:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:13:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:13:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:13:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:13:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:13:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:13:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:13:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:13:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:13:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:13:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:13:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:13:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:13:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:13:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:13:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:13:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:13:44.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:13:44.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:13:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:13:44.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:13:44.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:13:44.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:13:44.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:13:44.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:13:44.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:13:44.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:13:44.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:13:44.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:13:44.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:13:44.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:13:44.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:13:44.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=685] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:13:44.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=685] Local address: 172.21.15.161, port 57570 2022-07-03T17:13:44.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=685] Peer address: 172.21.15.101, port 22 2022-07-03T17:13:44.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=685] Connection lost 2022-07-03T17:13:44.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=685] Aborting connection 2022-07-03T17:13:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:45 smithi161 ceph-mon[91579]: pgmap v11838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:45 smithi188 ceph-mon[92425]: pgmap v11838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:47 smithi161 ceph-mon[91579]: pgmap v11839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:47 smithi188 ceph-mon[92425]: pgmap v11839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:49 smithi161 ceph-mon[91579]: pgmap v11840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:49 smithi188 ceph-mon[92425]: pgmap v11840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:51 smithi161 ceph-mon[91579]: pgmap v11841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:51 smithi188 ceph-mon[92425]: pgmap v11841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:53 smithi161 ceph-mon[91579]: pgmap v11842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:53 smithi188 ceph-mon[92425]: pgmap v11842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:55 smithi161 ceph-mon[91579]: pgmap v11843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:55 smithi188 ceph-mon[92425]: pgmap v11843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:57 smithi161 ceph-mon[91579]: pgmap v11844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:57 smithi188 ceph-mon[92425]: pgmap v11844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:13:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:13:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:59 smithi161 ceph-mon[91579]: pgmap v11845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:13:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:13:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:13:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:13:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:59 smithi188 ceph-mon[92425]: pgmap v11845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:13:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:13:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:13:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:13:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:14:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:01 smithi161 ceph-mon[91579]: pgmap v11846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:01 smithi188 ceph-mon[92425]: pgmap v11846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:03 smithi161 ceph-mon[91579]: pgmap v11847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:03 smithi188 ceph-mon[92425]: pgmap v11847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:05 smithi161 ceph-mon[91579]: pgmap v11848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:05 smithi188 ceph-mon[92425]: pgmap v11848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:07 smithi161 ceph-mon[91579]: pgmap v11849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:07 smithi188 ceph-mon[92425]: pgmap v11849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:09 smithi161 ceph-mon[91579]: pgmap v11850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:09 smithi188 ceph-mon[92425]: pgmap v11850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:11 smithi161 ceph-mon[91579]: pgmap v11851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:11 smithi188 ceph-mon[92425]: pgmap v11851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:13 smithi161 ceph-mon[91579]: pgmap v11852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:13 smithi188 ceph-mon[92425]: pgmap v11852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:15 smithi161 ceph-mon[91579]: pgmap v11853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:15 smithi188 ceph-mon[92425]: pgmap v11853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:17 smithi161 ceph-mon[91579]: pgmap v11854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:17 smithi188 ceph-mon[92425]: pgmap v11854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:19 smithi161 ceph-mon[91579]: pgmap v11855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:19 smithi188 ceph-mon[92425]: pgmap v11855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:21 smithi161 ceph-mon[91579]: pgmap v11856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:21 smithi188 ceph-mon[92425]: pgmap v11856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:23 smithi161 ceph-mon[91579]: pgmap v11857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:23 smithi188 ceph-mon[92425]: pgmap v11857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:25 smithi161 ceph-mon[91579]: pgmap v11858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:25 smithi188 ceph-mon[92425]: pgmap v11858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:27 smithi161 ceph-mon[91579]: pgmap v11859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:27 smithi188 ceph-mon[92425]: pgmap v11859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:29 smithi161 ceph-mon[91579]: pgmap v11860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:29 smithi188 ceph-mon[92425]: pgmap v11860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:31 smithi161 ceph-mon[91579]: pgmap v11861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:31 smithi188 ceph-mon[92425]: pgmap v11861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:33 smithi161 ceph-mon[91579]: pgmap v11862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:33 smithi188 ceph-mon[92425]: pgmap v11862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:34 smithi161 ceph-mon[91579]: pgmap v11863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:34 smithi188 ceph-mon[92425]: pgmap v11863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:36 smithi161 ceph-mon[91579]: pgmap v11864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:36 smithi188 ceph-mon[92425]: pgmap v11864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:38 smithi161 ceph-mon[91579]: pgmap v11865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:38 smithi188 ceph-mon[92425]: pgmap v11865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:40 smithi161 ceph-mon[91579]: pgmap v11866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:40 smithi188 ceph-mon[92425]: pgmap v11866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-mon[91579]: pgmap v11867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:43 smithi188 ceph-mon[92425]: pgmap v11867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:14:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:14:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:14:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:14:43.983+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:14:44.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:14:44.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:14:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:14:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:14:44.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:14:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:14:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:14:44.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:14:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:14:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:14:44.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:14:44.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:14:44.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:14:44.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:14:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:14:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:14:44.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:14:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:14:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:14:44.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:14:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:14:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:14:44.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:14:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:14:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:14:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:14:44.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:14:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:14:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:14:44.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:14:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:14:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:14:44.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:14:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:14:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:14:44.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:14:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:14:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:14:44.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:14:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:14:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:14:44.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:14:44.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:14:44.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:14:44.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=687] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:14:44.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=687] Local address: 172.21.15.161, port 57598 2022-07-03T17:14:44.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=687] Peer address: 172.21.15.101, port 22 2022-07-03T17:14:44.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=687] Connection lost 2022-07-03T17:14:44.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=687] Aborting connection 2022-07-03T17:14:44.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:14:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:14:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:14:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:14:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:45 smithi161 ceph-mon[91579]: pgmap v11868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:14:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:14:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:14:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:14:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:14:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:14:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:14:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:14:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:14:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:14:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:14:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:14:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:14:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:14:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:14:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:14:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:14:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:14:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:14:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:14:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:14:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:14:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:14:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:14:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:14:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:14:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:14:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:14:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:14:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:14:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:14:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:14:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:14:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:14:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:14:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:14:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:14:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:14:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:14:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:14:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:14:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:14:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:14:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:14:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:14:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:14:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=687] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:14:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=687] Local address: 172.21.15.161, port 57598 2022-07-03T17:14:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=687] Peer address: 172.21.15.101, port 22 2022-07-03T17:14:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=687] Connection lost 2022-07-03T17:14:45.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=687] Aborting connection 2022-07-03T17:14:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:45 smithi188 ceph-mon[92425]: pgmap v11868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:14:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:14:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:14:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:14:45.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:14:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:14:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:14:45.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:14:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:14:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:14:45.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:14:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:14:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:14:45.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:14:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:14:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:14:45.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:14:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:14:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:14:45.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:14:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:14:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:14:45.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:14:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:14:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:14:45.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:14:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:14:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:14:45.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:14:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:14:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:14:45.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:14:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:14:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:14:45.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:14:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:14:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:14:45.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:14:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:14:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:14:45.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:14:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:14:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:14:45.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:14:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:14:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:14:45.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=687] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:14:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=687] Local address: 172.21.15.161, port 57598 2022-07-03T17:14:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=687] Peer address: 172.21.15.101, port 22 2022-07-03T17:14:45.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=687] Connection lost 2022-07-03T17:14:45.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=687] Aborting connection 2022-07-03T17:14:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:47 smithi161 ceph-mon[91579]: pgmap v11869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:47 smithi188 ceph-mon[92425]: pgmap v11869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:49 smithi161 ceph-mon[91579]: pgmap v11870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:49 smithi188 ceph-mon[92425]: pgmap v11870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:51 smithi161 ceph-mon[91579]: pgmap v11871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:51 smithi188 ceph-mon[92425]: pgmap v11871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:53 smithi161 ceph-mon[91579]: pgmap v11872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:53 smithi188 ceph-mon[92425]: pgmap v11872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:55 smithi161 ceph-mon[91579]: pgmap v11873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:55 smithi188 ceph-mon[92425]: pgmap v11873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:57 smithi161 ceph-mon[91579]: pgmap v11874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:57 smithi188 ceph-mon[92425]: pgmap v11874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:14:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:14:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:59 smithi161 ceph-mon[91579]: pgmap v11875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:14:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:14:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:14:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:14:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:59 smithi188 ceph-mon[92425]: pgmap v11875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:14:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:14:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:14:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:14:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:15:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:01 smithi161 ceph-mon[91579]: pgmap v11876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:01 smithi188 ceph-mon[92425]: pgmap v11876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:03 smithi161 ceph-mon[91579]: pgmap v11877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:03 smithi188 ceph-mon[92425]: pgmap v11877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:05 smithi161 ceph-mon[91579]: pgmap v11878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:05 smithi188 ceph-mon[92425]: pgmap v11878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:07 smithi161 ceph-mon[91579]: pgmap v11879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:07 smithi188 ceph-mon[92425]: pgmap v11879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:09 smithi161 ceph-mon[91579]: pgmap v11880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:09 smithi188 ceph-mon[92425]: pgmap v11880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:11 smithi161 ceph-mon[91579]: pgmap v11881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:11 smithi188 ceph-mon[92425]: pgmap v11881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:13.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:13 smithi161 ceph-mon[91579]: pgmap v11882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:13 smithi188 ceph-mon[92425]: pgmap v11882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:15 smithi161 ceph-mon[91579]: pgmap v11883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:15 smithi188 ceph-mon[92425]: pgmap v11883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:17 smithi161 ceph-mon[91579]: pgmap v11884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:17 smithi188 ceph-mon[92425]: pgmap v11884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:19 smithi161 ceph-mon[91579]: pgmap v11885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:19 smithi188 ceph-mon[92425]: pgmap v11885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:21 smithi161 ceph-mon[91579]: pgmap v11886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:21 smithi188 ceph-mon[92425]: pgmap v11886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:23 smithi161 ceph-mon[91579]: pgmap v11887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:23 smithi188 ceph-mon[92425]: pgmap v11887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:25 smithi161 ceph-mon[91579]: pgmap v11888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:25 smithi188 ceph-mon[92425]: pgmap v11888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:27 smithi161 ceph-mon[91579]: pgmap v11889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:27 smithi188 ceph-mon[92425]: pgmap v11889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:29 smithi161 ceph-mon[91579]: pgmap v11890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:29 smithi188 ceph-mon[92425]: pgmap v11890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:31 smithi161 ceph-mon[91579]: pgmap v11891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:31 smithi188 ceph-mon[92425]: pgmap v11891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:33 smithi161 ceph-mon[91579]: pgmap v11892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:33 smithi188 ceph-mon[92425]: pgmap v11892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:35 smithi161 ceph-mon[91579]: pgmap v11893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:35 smithi188 ceph-mon[92425]: pgmap v11893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:37 smithi161 ceph-mon[91579]: pgmap v11894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:37 smithi188 ceph-mon[92425]: pgmap v11894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:39 smithi161 ceph-mon[91579]: pgmap v11895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:39 smithi188 ceph-mon[92425]: pgmap v11895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:41 smithi161 ceph-mon[91579]: pgmap v11896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:41 smithi188 ceph-mon[92425]: pgmap v11896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:43 smithi188 ceph-mon[92425]: pgmap v11897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:43 smithi161 ceph-mon[91579]: pgmap v11897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:44.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:15:44.482+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:15:44.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:15:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:15:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:15:44.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:15:44.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:15:44.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:15:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:15:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:15:44.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:15:44.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:15:44.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:15:44.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:15:44.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:15:44.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:15:44.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:15:44.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:15:44.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:15:44.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:15:44.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:15:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:15:44.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:15:44.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:15:44.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:15:44.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:15:44.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:15:44.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:15:44.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:15:44.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:15:44.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:15:44.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:15:44.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:15:44.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:15:44.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:15:44.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:15:44.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:15:44.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:15:44.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:15:44.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:15:44.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:15:44.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:15:44.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:15:44.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:15:44.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:15:44.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:15:44.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=689] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:15:44.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=689] Local address: 172.21.15.161, port 57626 2022-07-03T17:15:44.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=689] Peer address: 172.21.15.101, port 22 2022-07-03T17:15:44.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=689] Connection lost 2022-07-03T17:15:44.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=689] Aborting connection 2022-07-03T17:15:44.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:15:45.298 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: pgmap v11898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:15:45.299 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:15:45.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:15:45.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:15:45.300 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:15:45.301 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:15:45.301 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:15:45.302 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:15:45.302 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:15:45.302 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:15:45.303 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:15:45.303 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:15:45.303 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:15:45.304 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:15:45.305 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:15:45.305 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:15:45.306 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:15:45.306 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:15:45.307 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:15:45.307 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:15:45.308 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:15:45.309 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:15:45.309 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:15:45.310 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:15:45.310 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:15:45.311 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:15:45.311 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:15:45.311 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:15:45.312 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:15:45.312 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:15:45.313 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:15:45.313 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:15:45.313 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:15:45.314 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:15:45.314 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:15:45.314 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:15:45.315 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:15:45.315 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:15:45.315 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:15:45.316 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:15:45.316 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:15:45.317 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:15:45.317 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:15:45.317 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:15:45.318 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:15:45.318 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:15:45.319 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:15:45.319 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:15:45.319 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:15:45.320 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=689] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:15:45.320 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=689] Local address: 172.21.15.161, port 57626 2022-07-03T17:15:45.320 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=689] Peer address: 172.21.15.101, port 22 2022-07-03T17:15:45.321 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=689] Connection lost 2022-07-03T17:15:45.321 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=689] Aborting connection 2022-07-03T17:15:45.321 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.322 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.323 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: pgmap v11898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:15:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:15:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:15:45.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:15:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:15:45.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:15:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:15:45.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:15:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:15:45.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:15:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:15:45.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:15:45.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:15:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:15:45.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:15:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:15:45.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:15:45.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:15:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:15:45.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:15:45.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:15:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:15:45.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:15:45.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:15:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:15:45.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:15:45.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:15:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:15:45.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:15:45.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:15:45.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:15:45.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:15:45.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:15:45.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:15:45.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:15:45.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:15:45.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:15:45.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:15:45.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:15:45.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:15:45.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:15:45.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:15:45.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:15:45.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:15:45.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:15:45.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:15:45.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:15:45.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:15:45.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:15:45.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=689] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:15:45.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=689] Local address: 172.21.15.161, port 57626 2022-07-03T17:15:45.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=689] Peer address: 172.21.15.101, port 22 2022-07-03T17:15:45.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=689] Connection lost 2022-07-03T17:15:45.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=689] Aborting connection 2022-07-03T17:15:45.412 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.413 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.414 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:45.414 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:47 smithi188 ceph-mon[92425]: pgmap v11899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:47 smithi161 ceph-mon[91579]: pgmap v11899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:49 smithi188 ceph-mon[92425]: pgmap v11900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:49 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T17:15:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:49 smithi161 ceph-mon[91579]: pgmap v11900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:49 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T17:15:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:15:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:51 smithi188 ceph-mon[92425]: pgmap v11901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:51 smithi161 ceph-mon[91579]: pgmap v11901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:53 smithi161 ceph-mon[91579]: pgmap v11902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:53 smithi188 ceph-mon[92425]: pgmap v11902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:55 smithi161 ceph-mon[91579]: pgmap v11903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:55 smithi188 ceph-mon[92425]: pgmap v11903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:57 smithi161 ceph-mon[91579]: pgmap v11904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:57 smithi188 ceph-mon[92425]: pgmap v11904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:15:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:15:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:59 smithi161 ceph-mon[91579]: pgmap v11905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:15:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:15:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:15:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:15:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:59 smithi188 ceph-mon[92425]: pgmap v11905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:15:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:15:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:15:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:15:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:16:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:01 smithi161 ceph-mon[91579]: pgmap v11906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:01 smithi188 ceph-mon[92425]: pgmap v11906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:03 smithi161 ceph-mon[91579]: pgmap v11907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:03 smithi188 ceph-mon[92425]: pgmap v11907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:05 smithi161 ceph-mon[91579]: pgmap v11908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:05 smithi188 ceph-mon[92425]: pgmap v11908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:07 smithi161 ceph-mon[91579]: pgmap v11909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:07 smithi188 ceph-mon[92425]: pgmap v11909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:09 smithi161 ceph-mon[91579]: pgmap v11910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:09 smithi188 ceph-mon[92425]: pgmap v11910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:11 smithi161 ceph-mon[91579]: pgmap v11911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:11 smithi188 ceph-mon[92425]: pgmap v11911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:13 smithi161 ceph-mon[91579]: pgmap v11912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:13 smithi188 ceph-mon[92425]: pgmap v11912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:15 smithi161 ceph-mon[91579]: pgmap v11913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:15 smithi188 ceph-mon[92425]: pgmap v11913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:17 smithi161 ceph-mon[91579]: pgmap v11914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:17 smithi188 ceph-mon[92425]: pgmap v11914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:19 smithi161 ceph-mon[91579]: pgmap v11915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:19 smithi188 ceph-mon[92425]: pgmap v11915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:21 smithi161 ceph-mon[91579]: pgmap v11916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:21 smithi188 ceph-mon[92425]: pgmap v11916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:23 smithi161 ceph-mon[91579]: pgmap v11917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:23 smithi188 ceph-mon[92425]: pgmap v11917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:25 smithi161 ceph-mon[91579]: pgmap v11918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:25 smithi188 ceph-mon[92425]: pgmap v11918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:27 smithi161 ceph-mon[91579]: pgmap v11919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:27 smithi188 ceph-mon[92425]: pgmap v11919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:29 smithi161 ceph-mon[91579]: pgmap v11920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:29 smithi188 ceph-mon[92425]: pgmap v11920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:31 smithi161 ceph-mon[91579]: pgmap v11921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:31 smithi188 ceph-mon[92425]: pgmap v11921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:33 smithi161 ceph-mon[91579]: pgmap v11922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:33 smithi188 ceph-mon[92425]: pgmap v11922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:35 smithi161 ceph-mon[91579]: pgmap v11923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:35 smithi188 ceph-mon[92425]: pgmap v11923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:37 smithi161 ceph-mon[91579]: pgmap v11924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:37 smithi188 ceph-mon[92425]: pgmap v11924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:39 smithi161 ceph-mon[91579]: pgmap v11925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:39 smithi188 ceph-mon[92425]: pgmap v11925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:41 smithi161 ceph-mon[91579]: pgmap v11926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:41 smithi188 ceph-mon[92425]: pgmap v11926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:43 smithi161 ceph-mon[91579]: pgmap v11927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:43 smithi188 ceph-mon[92425]: pgmap v11927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:45 smithi161 ceph-mon[91579]: pgmap v11928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:45 smithi188 ceph-mon[92425]: pgmap v11928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:47 smithi161 ceph-mon[91579]: pgmap v11929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:47 smithi188 ceph-mon[92425]: pgmap v11929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:48.588 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:16:48.589 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:16:48.589 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:16:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:16:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:16:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:16:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:16:48.202+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:16:48.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:16:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:16:48.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:16:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:16:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:16:48.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:16:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:16:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:16:48.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:16:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:16:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:16:48.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:16:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:16:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:16:48.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:16:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:16:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:16:48.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:16:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:16:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:16:48.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:16:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:16:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:16:48.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:16:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:16:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:16:48.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:16:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:16:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:16:48.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:16:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:16:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:16:48.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:16:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:16:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:16:48.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:16:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:16:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:16:48.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:16:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:16:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:16:48.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:16:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:16:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:16:48.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=691] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:16:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=691] Local address: 172.21.15.161, port 57656 2022-07-03T17:16:48.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=691] Peer address: 172.21.15.101, port 22 2022-07-03T17:16:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=691] Connection lost 2022-07-03T17:16:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=691] Aborting connection 2022-07-03T17:16:48.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:16:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:16:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: pgmap v11930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:16:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:16:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:16:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:16:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:16:49.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:16:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:16:49.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:16:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:16:49.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:16:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:16:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:16:49.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:16:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:16:49.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:16:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:16:49.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:16:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:16:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:16:49.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:16:49.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:16:49.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:16:49.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:16:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:16:49.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:16:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:16:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:16:49.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:16:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:16:49.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:16:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:16:49.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:16:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:16:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:16:49.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:16:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:16:49.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:16:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:16:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:16:49.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:16:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:16:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:16:49.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:16:49.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:16:49.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:16:49.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:16:49.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=691] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:16:49.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=691] Local address: 172.21.15.161, port 57656 2022-07-03T17:16:49.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=691] Peer address: 172.21.15.101, port 22 2022-07-03T17:16:49.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=691] Connection lost 2022-07-03T17:16:49.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=691] Aborting connection 2022-07-03T17:16:49.654 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.654 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.655 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.655 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: pgmap v11930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:16:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:16:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:16:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:16:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:16:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:16:49.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:16:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:16:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:16:49.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:16:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:16:49.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:16:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:16:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:16:49.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:16:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:16:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:16:49.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:16:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:16:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:16:49.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:16:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:16:49.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:16:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:16:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:16:49.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:16:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:16:49.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:16:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:16:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:16:49.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:16:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:16:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:16:49.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:16:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:16:49.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:16:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:16:49.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:16:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:16:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:16:49.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:16:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:16:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:16:49.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:16:49.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:16:49.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:16:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=691] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:16:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=691] Local address: 172.21.15.161, port 57656 2022-07-03T17:16:49.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=691] Peer address: 172.21.15.101, port 22 2022-07-03T17:16:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=691] Connection lost 2022-07-03T17:16:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=691] Aborting connection 2022-07-03T17:16:49.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:49.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:16:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:51 smithi161 ceph-mon[91579]: pgmap v11931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:51 smithi188 ceph-mon[92425]: pgmap v11931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:53 smithi161 ceph-mon[91579]: pgmap v11932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:53 smithi188 ceph-mon[92425]: pgmap v11932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:55 smithi161 ceph-mon[91579]: pgmap v11933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:55 smithi188 ceph-mon[92425]: pgmap v11933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:57 smithi161 ceph-mon[91579]: pgmap v11934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:57 smithi188 ceph-mon[92425]: pgmap v11934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:16:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:16:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:59 smithi161 ceph-mon[91579]: pgmap v11935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:16:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:16:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:16:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:16:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:59 smithi188 ceph-mon[92425]: pgmap v11935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:16:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:16:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:16:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:16:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:17:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:01 smithi161 ceph-mon[91579]: pgmap v11936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:01 smithi188 ceph-mon[92425]: pgmap v11936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:03 smithi161 ceph-mon[91579]: pgmap v11937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:03 smithi188 ceph-mon[92425]: pgmap v11937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:05 smithi161 ceph-mon[91579]: pgmap v11938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:05 smithi188 ceph-mon[92425]: pgmap v11938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:07 smithi161 ceph-mon[91579]: pgmap v11939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:07.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:07 smithi188 ceph-mon[92425]: pgmap v11939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:07.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:09 smithi161 ceph-mon[91579]: pgmap v11940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:09 smithi188 ceph-mon[92425]: pgmap v11940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:11 smithi161 ceph-mon[91579]: pgmap v11941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:11 smithi188 ceph-mon[92425]: pgmap v11941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:13 smithi161 ceph-mon[91579]: pgmap v11942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:13 smithi188 ceph-mon[92425]: pgmap v11942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:15 smithi161 ceph-mon[91579]: pgmap v11943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:15 smithi188 ceph-mon[92425]: pgmap v11943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:17 smithi161 ceph-mon[91579]: pgmap v11944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:17 smithi188 ceph-mon[92425]: pgmap v11944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:19 smithi161 ceph-mon[91579]: pgmap v11945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:19 smithi188 ceph-mon[92425]: pgmap v11945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:21 smithi161 ceph-mon[91579]: pgmap v11946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:21 smithi188 ceph-mon[92425]: pgmap v11946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:23 smithi161 ceph-mon[91579]: pgmap v11947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:23 smithi188 ceph-mon[92425]: pgmap v11947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:25 smithi161 ceph-mon[91579]: pgmap v11948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:25 smithi188 ceph-mon[92425]: pgmap v11948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:27 smithi161 ceph-mon[91579]: pgmap v11949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:27 smithi188 ceph-mon[92425]: pgmap v11949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:29 smithi161 ceph-mon[91579]: pgmap v11950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:29 smithi188 ceph-mon[92425]: pgmap v11950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:31 smithi188 ceph-mon[92425]: pgmap v11951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:31 smithi161 ceph-mon[91579]: pgmap v11951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:33 smithi188 ceph-mon[92425]: pgmap v11952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:33 smithi161 ceph-mon[91579]: pgmap v11952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:35 smithi188 ceph-mon[92425]: pgmap v11953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:35 smithi161 ceph-mon[91579]: pgmap v11953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:37 smithi188 ceph-mon[92425]: pgmap v11954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:37 smithi161 ceph-mon[91579]: pgmap v11954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:39 smithi188 ceph-mon[92425]: pgmap v11955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:39 smithi161 ceph-mon[91579]: pgmap v11955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:41 smithi188 ceph-mon[92425]: pgmap v11956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:41 smithi161 ceph-mon[91579]: pgmap v11956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:43 smithi188 ceph-mon[92425]: pgmap v11957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:43 smithi161 ceph-mon[91579]: pgmap v11957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:45 smithi188 ceph-mon[92425]: pgmap v11958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:45 smithi161 ceph-mon[91579]: pgmap v11958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:47 smithi161 ceph-mon[91579]: pgmap v11959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:47 smithi188 ceph-mon[92425]: pgmap v11959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:49.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:17:49.104+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:17:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:17:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:17:49.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:17:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:17:49.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:17:49.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:17:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:17:49.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:17:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:17:49.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:17:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:17:49.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:17:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:17:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:17:49.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:17:49.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:17:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:17:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:17:49.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:17:49.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:17:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:17:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:17:49.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:17:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:17:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:17:49.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:17:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:17:49.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:17:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:17:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:17:49.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:17:49.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:17:49.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:17:49.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:17:49.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:17:49.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:17:49.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:17:49.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:17:49.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:17:49.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:17:49.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:17:49.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:17:49.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:17:49.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:17:49.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=693] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:17:49.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=693] Local address: 172.21.15.161, port 57684 2022-07-03T17:17:49.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=693] Peer address: 172.21.15.101, port 22 2022-07-03T17:17:49.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=693] Connection lost 2022-07-03T17:17:49.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=693] Aborting connection 2022-07-03T17:17:49.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:17:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-mon[91579]: pgmap v11960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:17:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:17:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:17:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:49 smithi188 ceph-mon[92425]: pgmap v11960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:17:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:17:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:17:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:17:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:17:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:17:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:17:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:17:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:17:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:17:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:17:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:17:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:17:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:17:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:17:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:17:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:17:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:17:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:17:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:17:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:17:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:17:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:17:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:17:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:17:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:17:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:17:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:17:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:17:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:17:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:17:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:17:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:17:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:17:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:17:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:17:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:17:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:17:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:17:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:17:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:17:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:17:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:17:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:17:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:17:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:17:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:17:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:17:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:17:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=693] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:17:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=693] Local address: 172.21.15.161, port 57684 2022-07-03T17:17:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=693] Peer address: 172.21.15.101, port 22 2022-07-03T17:17:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=693] Connection lost 2022-07-03T17:17:50.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=693] Aborting connection 2022-07-03T17:17:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:17:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:17:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:17:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:17:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:17:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:17:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:17:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:17:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:17:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:17:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:17:50.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:17:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:17:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:17:50.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:17:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:17:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:17:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:17:50.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:17:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:17:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:17:50.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:17:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:17:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:17:50.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:17:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:17:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:17:50.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:17:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:17:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:17:50.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:17:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:17:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:17:50.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:17:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:17:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:17:50.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:17:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:17:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:17:50.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:17:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:17:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:17:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:17:50.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:17:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:17:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:17:50.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=693] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:17:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=693] Local address: 172.21.15.161, port 57684 2022-07-03T17:17:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=693] Peer address: 172.21.15.101, port 22 2022-07-03T17:17:50.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=693] Connection lost 2022-07-03T17:17:50.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=693] Aborting connection 2022-07-03T17:17:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:51 smithi161 ceph-mon[91579]: pgmap v11961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:51 smithi188 ceph-mon[92425]: pgmap v11961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:53 smithi161 ceph-mon[91579]: pgmap v11962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:53 smithi188 ceph-mon[92425]: pgmap v11962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:55 smithi161 ceph-mon[91579]: pgmap v11963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:55 smithi188 ceph-mon[92425]: pgmap v11963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:57 smithi161 ceph-mon[91579]: pgmap v11964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:57 smithi188 ceph-mon[92425]: pgmap v11964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:17:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:17:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:59 smithi161 ceph-mon[91579]: pgmap v11965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:17:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:17:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:17:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:17:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:59 smithi188 ceph-mon[92425]: pgmap v11965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:17:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:17:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:17:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:17:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:18:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:01 smithi161 ceph-mon[91579]: pgmap v11966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:01 smithi188 ceph-mon[92425]: pgmap v11966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:03 smithi161 ceph-mon[91579]: pgmap v11967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:03 smithi188 ceph-mon[92425]: pgmap v11967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:05 smithi161 ceph-mon[91579]: pgmap v11968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:05 smithi188 ceph-mon[92425]: pgmap v11968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:07 smithi161 ceph-mon[91579]: pgmap v11969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:07 smithi188 ceph-mon[92425]: pgmap v11969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:09 smithi161 ceph-mon[91579]: pgmap v11970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:09 smithi188 ceph-mon[92425]: pgmap v11970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:11 smithi161 ceph-mon[91579]: pgmap v11971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:11 smithi188 ceph-mon[92425]: pgmap v11971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:13 smithi161 ceph-mon[91579]: pgmap v11972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:13 smithi188 ceph-mon[92425]: pgmap v11972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:15 smithi161 ceph-mon[91579]: pgmap v11973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:15 smithi188 ceph-mon[92425]: pgmap v11973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:17 smithi161 ceph-mon[91579]: pgmap v11974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:17 smithi188 ceph-mon[92425]: pgmap v11974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:19 smithi161 ceph-mon[91579]: pgmap v11975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:19 smithi188 ceph-mon[92425]: pgmap v11975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:21 smithi161 ceph-mon[91579]: pgmap v11976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:21 smithi188 ceph-mon[92425]: pgmap v11976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:23 smithi161 ceph-mon[91579]: pgmap v11977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:23 smithi188 ceph-mon[92425]: pgmap v11977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:25 smithi161 ceph-mon[91579]: pgmap v11978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:25 smithi188 ceph-mon[92425]: pgmap v11978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:27 smithi161 ceph-mon[91579]: pgmap v11979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:27 smithi188 ceph-mon[92425]: pgmap v11979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:29 smithi161 ceph-mon[91579]: pgmap v11980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:29 smithi188 ceph-mon[92425]: pgmap v11980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:31 smithi161 ceph-mon[91579]: pgmap v11981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:31 smithi188 ceph-mon[92425]: pgmap v11981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:33 smithi161 ceph-mon[91579]: pgmap v11982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:33 smithi188 ceph-mon[92425]: pgmap v11982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:35 smithi161 ceph-mon[91579]: pgmap v11983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:35 smithi188 ceph-mon[92425]: pgmap v11983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:37 smithi161 ceph-mon[91579]: pgmap v11984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:37 smithi188 ceph-mon[92425]: pgmap v11984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:39 smithi161 ceph-mon[91579]: pgmap v11985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:39 smithi188 ceph-mon[92425]: pgmap v11985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:41 smithi161 ceph-mon[91579]: pgmap v11986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:41 smithi188 ceph-mon[92425]: pgmap v11986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:43 smithi161 ceph-mon[91579]: pgmap v11987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:43 smithi188 ceph-mon[92425]: pgmap v11987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:45 smithi161 ceph-mon[91579]: pgmap v11988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:45 smithi188 ceph-mon[92425]: pgmap v11988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:47 smithi161 ceph-mon[91579]: pgmap v11989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:47 smithi188 ceph-mon[92425]: pgmap v11989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-mon[91579]: pgmap v11990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:18:49.606+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:18:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:18:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:18:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:18:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:18:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:18:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:18:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:18:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:18:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:18:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:18:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:18:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:18:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:18:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:18:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:18:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:18:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:18:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:18:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:18:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:18:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:18:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:18:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:18:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:18:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:18:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:18:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:18:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:18:49.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:18:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:18:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:18:49.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:18:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:18:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:18:49.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:18:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:18:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:18:49.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:18:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:18:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:18:49.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:18:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:18:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:18:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:18:49.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=695] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:18:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=695] Local address: 172.21.15.161, port 57712 2022-07-03T17:18:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=695] Peer address: 172.21.15.101, port 22 2022-07-03T17:18:49.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=695] Connection lost 2022-07-03T17:18:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=695] Aborting connection 2022-07-03T17:18:49.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:18:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:18:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:49 smithi188 ceph-mon[92425]: pgmap v11990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:18:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:18:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:18:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:18:50.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:18:50.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:18:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:18:50.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:18:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:51 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:18:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:18:51.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:18:51.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:18:51.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:18:51.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:18:51.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:18:51.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:18:51.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:18:51.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:18:51.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:18:51.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:18:51.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:18:51.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:18:51.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:18:51.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:18:51.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:18:51.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:18:51.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:18:51.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:18:51.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:18:51.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:18:51.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:18:51.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:18:51.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:18:51.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:18:51.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:18:51.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:18:51.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:18:51.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:18:51.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:18:51.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:18:51.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:18:51.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:18:51.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:18:51.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:18:51.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:18:51.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:18:51.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:18:51.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:18:51.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:18:51.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:18:51.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:18:51.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:18:51.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:18:51.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=695] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:18:51.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=695] Local address: 172.21.15.161, port 57712 2022-07-03T17:18:51.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=695] Peer address: 172.21.15.101, port 22 2022-07-03T17:18:51.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=695] Connection lost 2022-07-03T17:18:51.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=695] Aborting connection 2022-07-03T17:18:51.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:51 smithi161 ceph-mon[91579]: pgmap v11991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:51 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:18:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:18:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:18:51.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:18:51.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:18:51.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:18:51.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:18:51.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:18:51.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:18:51.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:18:51.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:18:51.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:18:51.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:18:51.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:18:51.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:18:51.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:18:51.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:18:51.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:18:51.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:18:51.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:18:51.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:18:51.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:18:51.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:18:51.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:18:51.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:18:51.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:18:51.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:18:51.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:18:51.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:18:51.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:18:51.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:18:51.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:18:51.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:18:51.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:18:51.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:18:51.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:18:51.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:18:51.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:18:51.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:18:51.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:18:51.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:18:51.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:18:51.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:18:51.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:18:51.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:18:51.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=695] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:18:51.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=695] Local address: 172.21.15.161, port 57712 2022-07-03T17:18:51.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=695] Peer address: 172.21.15.101, port 22 2022-07-03T17:18:51.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=695] Connection lost 2022-07-03T17:18:51.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=695] Aborting connection 2022-07-03T17:18:51.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:51 smithi188 ceph-mon[92425]: pgmap v11991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:18:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: pgmap v11992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: pgmap v11992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:53.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:18:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 186.2M 2022-07-03T17:18:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 195285590: error parsing value: Value '195285590' is below minimum 939524096 2022-07-03T17:18:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 76430k 2022-07-03T17:18:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 78264934: error parsing value: Value '78264934' is below minimum 939524096 2022-07-03T17:18:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 186.2M 2022-07-03T17:18:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 195285590: error parsing value: Value '195285590' is below minimum 939524096 2022-07-03T17:18:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 76430k 2022-07-03T17:18:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 78264934: error parsing value: Value '78264934' is below minimum 939524096 2022-07-03T17:18:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:55 smithi161 ceph-mon[91579]: pgmap v11993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:55 smithi188 ceph-mon[92425]: pgmap v11993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:57 smithi161 ceph-mon[91579]: pgmap v11994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:57 smithi188 ceph-mon[92425]: pgmap v11994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:18:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:18:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:18:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:18:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:18:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:18:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:18:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:18:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:18:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:18:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:18:59 smithi161 ceph-mon[91579]: pgmap v11995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:18:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:18:59 smithi188 ceph-mon[92425]: pgmap v11995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:01 smithi161 ceph-mon[91579]: pgmap v11996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:01 smithi188 ceph-mon[92425]: pgmap v11996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:03 smithi161 ceph-mon[91579]: pgmap v11997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:03 smithi188 ceph-mon[92425]: pgmap v11997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:05 smithi161 ceph-mon[91579]: pgmap v11998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:05 smithi188 ceph-mon[92425]: pgmap v11998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:07 smithi161 ceph-mon[91579]: pgmap v11999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:07 smithi188 ceph-mon[92425]: pgmap v11999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:08 smithi188 ceph-mon[92425]: pgmap v12000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:08 smithi161 ceph-mon[91579]: pgmap v12000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:11 smithi161 ceph-mon[91579]: pgmap v12001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:11 smithi188 ceph-mon[92425]: pgmap v12001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:13 smithi161 ceph-mon[91579]: pgmap v12002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:13 smithi188 ceph-mon[92425]: pgmap v12002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:15 smithi161 ceph-mon[91579]: pgmap v12003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:15 smithi188 ceph-mon[92425]: pgmap v12003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:17 smithi161 ceph-mon[91579]: pgmap v12004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:17 smithi188 ceph-mon[92425]: pgmap v12004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:19 smithi161 ceph-mon[91579]: pgmap v12005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:19 smithi188 ceph-mon[92425]: pgmap v12005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:21 smithi161 ceph-mon[91579]: pgmap v12006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:21 smithi188 ceph-mon[92425]: pgmap v12006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:23 smithi161 ceph-mon[91579]: pgmap v12007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:23 smithi188 ceph-mon[92425]: pgmap v12007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:25 smithi161 ceph-mon[91579]: pgmap v12008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:25 smithi188 ceph-mon[92425]: pgmap v12008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:27 smithi161 ceph-mon[91579]: pgmap v12009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:27 smithi188 ceph-mon[92425]: pgmap v12009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:29.393 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:29 smithi161 ceph-mon[91579]: pgmap v12010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:29 smithi188 ceph-mon[92425]: pgmap v12010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:31 smithi161 ceph-mon[91579]: pgmap v12011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:31 smithi188 ceph-mon[92425]: pgmap v12011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:33 smithi161 ceph-mon[91579]: pgmap v12012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:33 smithi188 ceph-mon[92425]: pgmap v12012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:35 smithi161 ceph-mon[91579]: pgmap v12013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:35 smithi188 ceph-mon[92425]: pgmap v12013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:37 smithi188 ceph-mon[92425]: pgmap v12014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:37 smithi161 ceph-mon[91579]: pgmap v12014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:39 smithi188 ceph-mon[92425]: pgmap v12015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:39 smithi161 ceph-mon[91579]: pgmap v12015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:41 smithi188 ceph-mon[92425]: pgmap v12016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:41 smithi161 ceph-mon[91579]: pgmap v12016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:43 smithi188 ceph-mon[92425]: pgmap v12017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:43 smithi161 ceph-mon[91579]: pgmap v12017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:45 smithi188 ceph-mon[92425]: pgmap v12018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:45 smithi161 ceph-mon[91579]: pgmap v12018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:47 smithi188 ceph-mon[92425]: pgmap v12019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:47 smithi161 ceph-mon[91579]: pgmap v12019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:49 smithi188 ceph-mon[92425]: pgmap v12020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:49 smithi161 ceph-mon[91579]: pgmap v12020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:51 smithi188 ceph-mon[92425]: pgmap v12021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:51 smithi161 ceph-mon[91579]: pgmap v12021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:19:52.856+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:19:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:19:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:19:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:19:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:19:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:19:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:19:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:19:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:19:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:19:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:19:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:19:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:19:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:19:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:19:53.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:19:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:19:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:19:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:19:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:19:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:19:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:19:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:19:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:19:53.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:19:53.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:19:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:19:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:19:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:19:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:19:53.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:19:53.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:19:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:19:53.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:19:53.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:19:53.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:19:53.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:19:53.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:19:53.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:19:53.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:19:53.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:19:53.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:19:53.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:19:53.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:19:53.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:19:53.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=697] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:19:53.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=697] Local address: 172.21.15.161, port 57742 2022-07-03T17:19:53.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=697] Peer address: 172.21.15.101, port 22 2022-07-03T17:19:53.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=697] Connection lost 2022-07-03T17:19:53.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=697] Aborting connection 2022-07-03T17:19:53.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:19:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:19:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:53 smithi161 ceph-mon[91579]: pgmap v12022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:19:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:19:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:19:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:53 smithi188 ceph-mon[92425]: pgmap v12022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:19:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:19:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:19:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:19:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:19:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:19:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:19:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:19:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:19:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:19:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:19:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:19:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:19:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:19:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:19:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:19:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:19:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:19:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:19:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:19:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:19:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:19:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:19:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:19:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:19:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:19:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:19:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:19:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:19:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:19:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:19:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:19:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:19:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:19:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:19:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:19:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:19:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:19:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:19:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:19:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:19:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:19:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:19:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:19:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:19:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:19:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:19:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:19:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:19:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=697] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:19:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=697] Local address: 172.21.15.161, port 57742 2022-07-03T17:19:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=697] Peer address: 172.21.15.101, port 22 2022-07-03T17:19:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=697] Connection lost 2022-07-03T17:19:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=697] Aborting connection 2022-07-03T17:19:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:19:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:19:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:19:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:19:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:19:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:19:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:19:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:19:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:19:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:19:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:19:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:19:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:19:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:19:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:19:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:19:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:19:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:19:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:19:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:19:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:19:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:19:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:19:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:19:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:19:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:19:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:19:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:19:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:19:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:19:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:19:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:19:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:19:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:19:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:19:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:19:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:19:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:19:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:19:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:19:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:19:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:19:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:19:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:19:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:19:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:19:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=697] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:19:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=697] Local address: 172.21.15.161, port 57742 2022-07-03T17:19:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=697] Peer address: 172.21.15.101, port 22 2022-07-03T17:19:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=697] Connection lost 2022-07-03T17:19:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=697] Aborting connection 2022-07-03T17:19:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:55 smithi161 ceph-mon[91579]: pgmap v12023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:55 smithi188 ceph-mon[92425]: pgmap v12023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:57 smithi161 ceph-mon[91579]: pgmap v12024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:57 smithi188 ceph-mon[92425]: pgmap v12024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:19:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:19:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:59 smithi161 ceph-mon[91579]: pgmap v12025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:19:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:19:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:19:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:19:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:59 smithi188 ceph-mon[92425]: pgmap v12025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:19:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:19:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:19:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:19:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:20:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:20:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:20:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:01 smithi161 ceph-mon[91579]: pgmap v12026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:01 smithi188 ceph-mon[92425]: pgmap v12026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:03 smithi161 ceph-mon[91579]: pgmap v12027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:03 smithi188 ceph-mon[92425]: pgmap v12027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:05 smithi161 ceph-mon[91579]: pgmap v12028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:05 smithi188 ceph-mon[92425]: pgmap v12028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:07 smithi161 ceph-mon[91579]: pgmap v12029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:07 smithi188 ceph-mon[92425]: pgmap v12029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:09 smithi161 ceph-mon[91579]: pgmap v12030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:09 smithi188 ceph-mon[92425]: pgmap v12030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:11 smithi161 ceph-mon[91579]: pgmap v12031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:11 smithi188 ceph-mon[92425]: pgmap v12031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:13 smithi161 ceph-mon[91579]: pgmap v12032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:13 smithi188 ceph-mon[92425]: pgmap v12032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:15 smithi161 ceph-mon[91579]: pgmap v12033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:15.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:15 smithi188 ceph-mon[92425]: pgmap v12033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:17 smithi161 ceph-mon[91579]: pgmap v12034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:17 smithi188 ceph-mon[92425]: pgmap v12034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:19 smithi161 ceph-mon[91579]: pgmap v12035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:19 smithi188 ceph-mon[92425]: pgmap v12035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:21 smithi161 ceph-mon[91579]: pgmap v12036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:21 smithi188 ceph-mon[92425]: pgmap v12036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:23 smithi161 ceph-mon[91579]: pgmap v12037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:23 smithi188 ceph-mon[92425]: pgmap v12037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:25 smithi161 ceph-mon[91579]: pgmap v12038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:25 smithi188 ceph-mon[92425]: pgmap v12038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:27 smithi161 ceph-mon[91579]: pgmap v12039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:27 smithi188 ceph-mon[92425]: pgmap v12039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:29 smithi161 ceph-mon[91579]: pgmap v12040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:29 smithi188 ceph-mon[92425]: pgmap v12040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:31 smithi161 ceph-mon[91579]: pgmap v12041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:31 smithi188 ceph-mon[92425]: pgmap v12041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:33 smithi161 ceph-mon[91579]: pgmap v12042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:33 smithi188 ceph-mon[92425]: pgmap v12042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:35 smithi161 ceph-mon[91579]: pgmap v12043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:35 smithi188 ceph-mon[92425]: pgmap v12043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:37 smithi161 ceph-mon[91579]: pgmap v12044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:37 smithi188 ceph-mon[92425]: pgmap v12044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:39 smithi161 ceph-mon[91579]: pgmap v12045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:39 smithi188 ceph-mon[92425]: pgmap v12045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:41 smithi161 ceph-mon[91579]: pgmap v12046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:41 smithi188 ceph-mon[92425]: pgmap v12046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:42.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:43 smithi161 ceph-mon[91579]: pgmap v12047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:43 smithi188 ceph-mon[92425]: pgmap v12047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:45 smithi161 ceph-mon[91579]: pgmap v12048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:45 smithi188 ceph-mon[92425]: pgmap v12048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:47 smithi161 ceph-mon[91579]: pgmap v12049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:47 smithi188 ceph-mon[92425]: pgmap v12049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:49 smithi161 ceph-mon[91579]: pgmap v12050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:49 smithi188 ceph-mon[92425]: pgmap v12050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:51 smithi161 ceph-mon[91579]: pgmap v12051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:51 smithi188 ceph-mon[92425]: pgmap v12051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-mon[91579]: pgmap v12052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:20:53.359+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:20:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:20:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:20:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:20:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:20:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:20:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:20:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:20:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:20:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:20:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:20:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:20:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:20:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:20:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:20:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:20:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:20:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:20:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:20:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:20:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:20:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:20:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:20:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:20:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:20:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:20:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:20:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:20:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:20:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:20:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:20:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:20:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:20:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:20:53.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:20:53.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:20:53.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:20:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:20:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:20:53.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:20:53.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:20:53.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:20:53.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:20:53.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:20:53.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:20:53.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=699] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:20:53.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=699] Local address: 172.21.15.161, port 57770 2022-07-03T17:20:53.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=699] Peer address: 172.21.15.101, port 22 2022-07-03T17:20:53.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=699] Connection lost 2022-07-03T17:20:53.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=699] Aborting connection 2022-07-03T17:20:53.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:20:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:20:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:53 smithi188 ceph-mon[92425]: pgmap v12052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:20:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:20:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:20:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:20:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:20:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:20:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:20:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:20:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:20:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:20:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:20:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:20:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:20:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:20:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:20:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:20:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:20:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:20:54.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:20:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:20:54.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:20:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:20:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:20:54.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:20:54.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=699] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:20:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=699] Local address: 172.21.15.161, port 57770 2022-07-03T17:20:54.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=699] Peer address: 172.21.15.101, port 22 2022-07-03T17:20:54.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=699] Connection lost 2022-07-03T17:20:54.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=699] Aborting connection 2022-07-03T17:20:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:20:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:20:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:20:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:20:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:20:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:20:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:20:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:20:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:20:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:20:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:20:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:20:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:20:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:20:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:20:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:20:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:20:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:20:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:20:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:20:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:20:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:20:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:20:54.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:20:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:20:54.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:20:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:20:54.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:20:54.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:20:54.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:20:54.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:20:54.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:20:54.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:20:54.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:20:54.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:20:54.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:20:54.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:20:54.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:20:54.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:20:54.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:20:54.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:20:54.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=699] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:20:54.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=699] Local address: 172.21.15.161, port 57770 2022-07-03T17:20:54.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=699] Peer address: 172.21.15.101, port 22 2022-07-03T17:20:54.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=699] Connection lost 2022-07-03T17:20:54.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=699] Aborting connection 2022-07-03T17:20:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:55 smithi161 ceph-mon[91579]: pgmap v12053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:55.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:55 smithi188 ceph-mon[92425]: pgmap v12053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:57 smithi161 ceph-mon[91579]: pgmap v12054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:57 smithi188 ceph-mon[92425]: pgmap v12054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:20:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:20:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:59 smithi161 ceph-mon[91579]: pgmap v12055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:20:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:20:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:20:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:20:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:59 smithi188 ceph-mon[92425]: pgmap v12055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:20:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:20:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:20:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:20:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:21:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:01 smithi161 ceph-mon[91579]: pgmap v12056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:01 smithi188 ceph-mon[92425]: pgmap v12056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:03 smithi161 ceph-mon[91579]: pgmap v12057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:03 smithi188 ceph-mon[92425]: pgmap v12057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:05 smithi161 ceph-mon[91579]: pgmap v12058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:05 smithi188 ceph-mon[92425]: pgmap v12058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:07 smithi161 ceph-mon[91579]: pgmap v12059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:07 smithi188 ceph-mon[92425]: pgmap v12059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:09 smithi161 ceph-mon[91579]: pgmap v12060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:09 smithi188 ceph-mon[92425]: pgmap v12060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:11 smithi161 ceph-mon[91579]: pgmap v12061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:11 smithi188 ceph-mon[92425]: pgmap v12061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:13 smithi161 ceph-mon[91579]: pgmap v12062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:13 smithi188 ceph-mon[92425]: pgmap v12062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:15 smithi161 ceph-mon[91579]: pgmap v12063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:15 smithi188 ceph-mon[92425]: pgmap v12063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:17 smithi161 ceph-mon[91579]: pgmap v12064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:17 smithi188 ceph-mon[92425]: pgmap v12064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:19 smithi161 ceph-mon[91579]: pgmap v12065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:19 smithi188 ceph-mon[92425]: pgmap v12065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:21 smithi161 ceph-mon[91579]: pgmap v12066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:21 smithi188 ceph-mon[92425]: pgmap v12066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:23 smithi161 ceph-mon[91579]: pgmap v12067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:23 smithi188 ceph-mon[92425]: pgmap v12067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:25 smithi161 ceph-mon[91579]: pgmap v12068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:25 smithi188 ceph-mon[92425]: pgmap v12068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:27 smithi161 ceph-mon[91579]: pgmap v12069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:27 smithi188 ceph-mon[92425]: pgmap v12069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:21:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:29 smithi161 ceph-mon[91579]: pgmap v12070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:29 smithi188 ceph-mon[92425]: pgmap v12070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:31 smithi188 ceph-mon[92425]: pgmap v12071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:31 smithi161 ceph-mon[91579]: pgmap v12071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:33 smithi188 ceph-mon[92425]: pgmap v12072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:33 smithi161 ceph-mon[91579]: pgmap v12072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:35 smithi188 ceph-mon[92425]: pgmap v12073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:35 smithi161 ceph-mon[91579]: pgmap v12073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:37 smithi188 ceph-mon[92425]: pgmap v12074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:37 smithi161 ceph-mon[91579]: pgmap v12074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:39 smithi188 ceph-mon[92425]: pgmap v12075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:39 smithi161 ceph-mon[91579]: pgmap v12075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:41 smithi188 ceph-mon[92425]: pgmap v12076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:41 smithi161 ceph-mon[91579]: pgmap v12076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:43 smithi188 ceph-mon[92425]: pgmap v12077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:43 smithi161 ceph-mon[91579]: pgmap v12077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:45 smithi188 ceph-mon[92425]: pgmap v12078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:45 smithi161 ceph-mon[91579]: pgmap v12078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:47 smithi161 ceph-mon[91579]: pgmap v12079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:47 smithi188 ceph-mon[92425]: pgmap v12079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:49 smithi161 ceph-mon[91579]: pgmap v12080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:49 smithi188 ceph-mon[92425]: pgmap v12080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:51 smithi161 ceph-mon[91579]: pgmap v12081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:51 smithi188 ceph-mon[92425]: pgmap v12081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:53.868 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-mon[91579]: pgmap v12082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:53.879 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:53 smithi188 ceph-mon[92425]: pgmap v12082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:54.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:21:53.865+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:21:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:21:54.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:21:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:21:54.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:21:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:21:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:21:54.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:21:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:21:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:21:54.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:21:54.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:21:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:21:54.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:21:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:21:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:21:54.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:21:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:21:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:21:54.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:21:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:21:54.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:21:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:21:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:21:54.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:21:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:21:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:21:54.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:21:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:21:54.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:21:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:21:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:21:54.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:21:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:21:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:21:54.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:21:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:21:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:21:54.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:21:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:21:54.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:21:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:21:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:21:54.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:21:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:21:54.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=701] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:21:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=701] Local address: 172.21.15.161, port 57798 2022-07-03T17:21:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=701] Peer address: 172.21.15.101, port 22 2022-07-03T17:21:54.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=701] Connection lost 2022-07-03T17:21:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=701] Aborting connection 2022-07-03T17:21:54.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:21:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:21:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:21:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:21:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:21:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:21:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:21:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:21:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:21:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:21:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:21:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:21:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:21:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:21:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:21:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:21:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:21:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:21:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:21:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:21:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:21:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:21:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:21:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:21:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:21:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:21:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:21:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:21:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:21:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:21:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:21:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:21:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:21:55.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:21:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:21:55.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:21:55.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:21:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:21:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:21:55.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:21:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:21:55.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:21:55.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:21:55.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:21:55.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:21:55.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:21:55.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:21:55.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:21:55.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:21:55.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=701] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:21:55.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=701] Local address: 172.21.15.161, port 57798 2022-07-03T17:21:55.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=701] Peer address: 172.21.15.101, port 22 2022-07-03T17:21:55.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=701] Connection lost 2022-07-03T17:21:55.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=701] Aborting connection 2022-07-03T17:21:55.909 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:55 smithi161 ceph-mon[91579]: pgmap v12083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:21:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:21:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:21:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:21:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:21:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:21:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:21:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:21:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:21:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:21:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:21:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:21:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:21:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:21:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:21:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:21:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:21:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:21:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:21:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:21:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:21:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:21:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:21:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:21:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:21:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:21:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:21:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:21:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:21:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:21:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:21:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:21:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:21:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:21:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:21:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:21:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:21:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:21:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:21:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:21:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:21:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:21:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:21:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:21:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:21:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:21:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=701] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:21:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=701] Local address: 172.21.15.161, port 57798 2022-07-03T17:21:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=701] Peer address: 172.21.15.101, port 22 2022-07-03T17:21:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=701] Connection lost 2022-07-03T17:21:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=701] Aborting connection 2022-07-03T17:21:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:55 smithi188 ceph-mon[92425]: pgmap v12083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:57 smithi161 ceph-mon[91579]: pgmap v12084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:57 smithi188 ceph-mon[92425]: pgmap v12084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:21:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:21:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:59 smithi161 ceph-mon[91579]: pgmap v12085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:21:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:21:59.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:21:59.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:21:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:59 smithi188 ceph-mon[92425]: pgmap v12085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:21:59.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:21:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:21:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:21:59.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:22:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:01 smithi161 ceph-mon[91579]: pgmap v12086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:01 smithi188 ceph-mon[92425]: pgmap v12086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:03 smithi161 ceph-mon[91579]: pgmap v12087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:03 smithi188 ceph-mon[92425]: pgmap v12087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:05 smithi161 ceph-mon[91579]: pgmap v12088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:05 smithi188 ceph-mon[92425]: pgmap v12088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:07 smithi161 ceph-mon[91579]: pgmap v12089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:07 smithi188 ceph-mon[92425]: pgmap v12089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:09 smithi161 ceph-mon[91579]: pgmap v12090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:09 smithi188 ceph-mon[92425]: pgmap v12090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:11 smithi161 ceph-mon[91579]: pgmap v12091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:11 smithi188 ceph-mon[92425]: pgmap v12091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:13 smithi161 ceph-mon[91579]: pgmap v12092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:13 smithi188 ceph-mon[92425]: pgmap v12092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:15 smithi161 ceph-mon[91579]: pgmap v12093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:15 smithi188 ceph-mon[92425]: pgmap v12093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:17 smithi161 ceph-mon[91579]: pgmap v12094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:17 smithi188 ceph-mon[92425]: pgmap v12094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:19 smithi161 ceph-mon[91579]: pgmap v12095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:19 smithi188 ceph-mon[92425]: pgmap v12095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:21 smithi161 ceph-mon[91579]: pgmap v12096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:21 smithi188 ceph-mon[92425]: pgmap v12096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:23 smithi161 ceph-mon[91579]: pgmap v12097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:23 smithi188 ceph-mon[92425]: pgmap v12097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:25 smithi161 ceph-mon[91579]: pgmap v12098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:25 smithi188 ceph-mon[92425]: pgmap v12098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:27 smithi161 ceph-mon[91579]: pgmap v12099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:27 smithi188 ceph-mon[92425]: pgmap v12099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:29 smithi161 ceph-mon[91579]: pgmap v12100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:29 smithi188 ceph-mon[92425]: pgmap v12100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:31 smithi161 ceph-mon[91579]: pgmap v12101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:31 smithi188 ceph-mon[92425]: pgmap v12101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:33 smithi161 ceph-mon[91579]: pgmap v12102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:33 smithi188 ceph-mon[92425]: pgmap v12102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:35 smithi161 ceph-mon[91579]: pgmap v12103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:35 smithi188 ceph-mon[92425]: pgmap v12103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:37 smithi161 ceph-mon[91579]: pgmap v12104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:37 smithi188 ceph-mon[92425]: pgmap v12104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:39 smithi161 ceph-mon[91579]: pgmap v12105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:39 smithi188 ceph-mon[92425]: pgmap v12105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:41 smithi161 ceph-mon[91579]: pgmap v12106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:41 smithi188 ceph-mon[92425]: pgmap v12106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:43.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:43 smithi161 ceph-mon[91579]: pgmap v12107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:43 smithi188 ceph-mon[92425]: pgmap v12107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:45 smithi161 ceph-mon[91579]: pgmap v12108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:45 smithi188 ceph-mon[92425]: pgmap v12108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:47 smithi161 ceph-mon[91579]: pgmap v12109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:47 smithi188 ceph-mon[92425]: pgmap v12109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:49 smithi161 ceph-mon[91579]: pgmap v12110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:49 smithi188 ceph-mon[92425]: pgmap v12110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:51 smithi161 ceph-mon[91579]: pgmap v12111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:51 smithi188 ceph-mon[92425]: pgmap v12111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:53 smithi161 ceph-mon[91579]: pgmap v12112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:53 smithi188 ceph-mon[92425]: pgmap v12112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:22:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:22:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:22:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:22:54.359+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:22:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:22:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:22:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:22:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:22:54.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:22:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:22:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:22:54.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:22:54.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:22:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:22:54.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:22:54.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:22:54.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:22:54.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:22:54.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:22:54.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:22:54.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:22:54.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:22:54.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:22:54.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:22:54.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:22:54.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:22:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:22:54.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:22:54.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:22:54.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:22:54.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:22:54.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:22:54.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:22:54.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:22:54.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:22:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:22:54.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:22:54.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:22:54.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:22:54.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:22:54.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:22:54.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:22:54.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:22:54.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:22:54.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:22:54.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:22:54.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:22:54.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:22:54.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=703] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:22:54.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=703] Local address: 172.21.15.161, port 57826 2022-07-03T17:22:54.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=703] Peer address: 172.21.15.101, port 22 2022-07-03T17:22:54.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=703] Connection lost 2022-07-03T17:22:54.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=703] Aborting connection 2022-07-03T17:22:54.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:22:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:22:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:22:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:22:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:55 smithi161 ceph-mon[91579]: pgmap v12113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:22:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:22:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:22:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:22:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:22:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:22:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:22:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=703] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:22:55.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=703] Local address: 172.21.15.161, port 57826 2022-07-03T17:22:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=703] Peer address: 172.21.15.101, port 22 2022-07-03T17:22:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=703] Connection lost 2022-07-03T17:22:55.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=703] Aborting connection 2022-07-03T17:22:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:55 smithi188 ceph-mon[92425]: pgmap v12113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:22:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:22:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:22:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:22:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:22:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:22:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:22:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:22:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:22:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:22:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:22:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:22:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:22:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:22:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:22:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:22:55.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:22:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:22:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:22:55.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=703] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:22:55.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=703] Local address: 172.21.15.161, port 57826 2022-07-03T17:22:55.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=703] Peer address: 172.21.15.101, port 22 2022-07-03T17:22:55.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=703] Connection lost 2022-07-03T17:22:55.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=703] Aborting connection 2022-07-03T17:22:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:57 smithi161 ceph-mon[91579]: pgmap v12114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:57 smithi188 ceph-mon[92425]: pgmap v12114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:22:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:22:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:22:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:22:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:22:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:22:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:22:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:22:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:22:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:22:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:22:59 smithi161 ceph-mon[91579]: pgmap v12115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:22:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:22:59 smithi188 ceph-mon[92425]: pgmap v12115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:01 smithi161 ceph-mon[91579]: pgmap v12116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:01 smithi188 ceph-mon[92425]: pgmap v12116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:03 smithi161 ceph-mon[91579]: pgmap v12117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:03 smithi188 ceph-mon[92425]: pgmap v12117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:05 smithi161 ceph-mon[91579]: pgmap v12118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:05 smithi188 ceph-mon[92425]: pgmap v12118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:07 smithi161 ceph-mon[91579]: pgmap v12119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:07 smithi188 ceph-mon[92425]: pgmap v12119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:09 smithi161 ceph-mon[91579]: pgmap v12120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:09 smithi188 ceph-mon[92425]: pgmap v12120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:11 smithi161 ceph-mon[91579]: pgmap v12121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:11 smithi188 ceph-mon[92425]: pgmap v12121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:13 smithi161 ceph-mon[91579]: pgmap v12122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:13 smithi188 ceph-mon[92425]: pgmap v12122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:15 smithi161 ceph-mon[91579]: pgmap v12123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:15 smithi188 ceph-mon[92425]: pgmap v12123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:17 smithi161 ceph-mon[91579]: pgmap v12124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:17 smithi188 ceph-mon[92425]: pgmap v12124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:17.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:19 smithi188 ceph-mon[92425]: pgmap v12125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:19 smithi161 ceph-mon[91579]: pgmap v12125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:21 smithi188 ceph-mon[92425]: pgmap v12126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:21 smithi161 ceph-mon[91579]: pgmap v12126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:23 smithi188 ceph-mon[92425]: pgmap v12127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:23 smithi161 ceph-mon[91579]: pgmap v12127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:24 smithi188 ceph-mon[92425]: pgmap v12128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:24 smithi161 ceph-mon[91579]: pgmap v12128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:27 smithi188 ceph-mon[92425]: pgmap v12129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:27 smithi161 ceph-mon[91579]: pgmap v12129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:29 smithi188 ceph-mon[92425]: pgmap v12130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:29 smithi161 ceph-mon[91579]: pgmap v12130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:31 smithi161 ceph-mon[91579]: pgmap v12131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:31 smithi188 ceph-mon[92425]: pgmap v12131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:33 smithi161 ceph-mon[91579]: pgmap v12132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:33 smithi188 ceph-mon[92425]: pgmap v12132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:35 smithi161 ceph-mon[91579]: pgmap v12133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:35 smithi188 ceph-mon[92425]: pgmap v12133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:37 smithi161 ceph-mon[91579]: pgmap v12134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:37 smithi188 ceph-mon[92425]: pgmap v12134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:39 smithi161 ceph-mon[91579]: pgmap v12135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:39 smithi188 ceph-mon[92425]: pgmap v12135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:41 smithi161 ceph-mon[91579]: pgmap v12136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:41 smithi188 ceph-mon[92425]: pgmap v12136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:43 smithi161 ceph-mon[91579]: pgmap v12137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:43 smithi188 ceph-mon[92425]: pgmap v12137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:45 smithi161 ceph-mon[91579]: pgmap v12138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:45 smithi188 ceph-mon[92425]: pgmap v12138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:47 smithi161 ceph-mon[91579]: pgmap v12139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:47 smithi188 ceph-mon[92425]: pgmap v12139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:49 smithi161 ceph-mon[91579]: pgmap v12140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:49 smithi188 ceph-mon[92425]: pgmap v12140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:51 smithi161 ceph-mon[91579]: pgmap v12141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:51 smithi188 ceph-mon[92425]: pgmap v12141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:53 smithi161 ceph-mon[91579]: pgmap v12142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:53 smithi188 ceph-mon[92425]: pgmap v12142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:55.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:23:54.846+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:23:55.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:23:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:23:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:23:55.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:23:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:23:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:23:55.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:23:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:23:55.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:23:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:23:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:23:55.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:23:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:23:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:23:55.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:23:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:23:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:23:55.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:23:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:23:55.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:23:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:23:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:23:55.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:23:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:23:55.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:23:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:23:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:23:55.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:23:55.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:23:55.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:23:55.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:23:55.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:23:55.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:23:55.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:23:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:23:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:23:55.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:23:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:23:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:23:55.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:23:55.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:23:55.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:23:55.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:23:55.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:23:55.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=705] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:23:55.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=705] Local address: 172.21.15.161, port 57854 2022-07-03T17:23:55.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=705] Peer address: 172.21.15.101, port 22 2022-07-03T17:23:55.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=705] Connection lost 2022-07-03T17:23:55.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=705] Aborting connection 2022-07-03T17:23:55.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:23:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:23:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:55 smithi161 ceph-mon[91579]: pgmap v12143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:23:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:23:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:23:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:55 smithi188 ceph-mon[92425]: pgmap v12143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:55.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:23:55.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:23:55.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:23:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:23:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:23:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:23:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:23:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:23:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:23:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:23:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:23:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:23:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:23:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:23:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:23:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:23:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:23:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:23:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:23:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:23:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:23:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:23:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:23:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:23:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:23:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:23:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:23:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:23:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:23:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:23:56.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:23:56.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:23:56.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:23:56.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:23:56.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:23:56.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:23:56.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:23:56.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:23:56.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:23:56.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:23:56.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:23:56.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:23:56.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:23:56.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:23:56.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:23:56.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:23:56.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:23:56.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:23:56.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:23:56.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=705] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:23:56.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=705] Local address: 172.21.15.161, port 57854 2022-07-03T17:23:56.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=705] Peer address: 172.21.15.101, port 22 2022-07-03T17:23:56.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=705] Connection lost 2022-07-03T17:23:56.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=705] Aborting connection 2022-07-03T17:23:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:23:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:23:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:23:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:23:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:23:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:23:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:23:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:23:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:23:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:23:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:23:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:23:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:23:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:23:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:23:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:23:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:23:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:23:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:23:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:23:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:23:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:23:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:23:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:23:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:23:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:23:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:23:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:23:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:23:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:23:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:23:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:23:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:23:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:23:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:23:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:23:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:23:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:23:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:23:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:23:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:23:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:23:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:23:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:23:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:23:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:23:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=705] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:23:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=705] Local address: 172.21.15.161, port 57854 2022-07-03T17:23:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=705] Peer address: 172.21.15.101, port 22 2022-07-03T17:23:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=705] Connection lost 2022-07-03T17:23:56.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=705] Aborting connection 2022-07-03T17:23:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:57 smithi161 ceph-mon[91579]: pgmap v12144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:57 smithi188 ceph-mon[92425]: pgmap v12144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:23:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:23:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:59 smithi161 ceph-mon[91579]: pgmap v12145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:23:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:23:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:23:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:23:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:59 smithi188 ceph-mon[92425]: pgmap v12145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:23:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:23:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:23:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:23:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:24:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:01 smithi161 ceph-mon[91579]: pgmap v12146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:01 smithi188 ceph-mon[92425]: pgmap v12146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:03 smithi161 ceph-mon[91579]: pgmap v12147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:03 smithi188 ceph-mon[92425]: pgmap v12147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:05 smithi161 ceph-mon[91579]: pgmap v12148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:05 smithi188 ceph-mon[92425]: pgmap v12148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:07 smithi161 ceph-mon[91579]: pgmap v12149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:07 smithi188 ceph-mon[92425]: pgmap v12149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:09 smithi161 ceph-mon[91579]: pgmap v12150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:09 smithi188 ceph-mon[92425]: pgmap v12150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:11 smithi161 ceph-mon[91579]: pgmap v12151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:11 smithi188 ceph-mon[92425]: pgmap v12151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:13 smithi161 ceph-mon[91579]: pgmap v12152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:13 smithi188 ceph-mon[92425]: pgmap v12152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:15 smithi161 ceph-mon[91579]: pgmap v12153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:15 smithi188 ceph-mon[92425]: pgmap v12153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:17 smithi161 ceph-mon[91579]: pgmap v12154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:17 smithi188 ceph-mon[92425]: pgmap v12154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:19.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:19 smithi161 ceph-mon[91579]: pgmap v12155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:19.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:19 smithi188 ceph-mon[92425]: pgmap v12155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:21 smithi161 ceph-mon[91579]: pgmap v12156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:21 smithi188 ceph-mon[92425]: pgmap v12156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:23 smithi161 ceph-mon[91579]: pgmap v12157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:23 smithi188 ceph-mon[92425]: pgmap v12157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:25 smithi161 ceph-mon[91579]: pgmap v12158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:25 smithi188 ceph-mon[92425]: pgmap v12158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:27 smithi161 ceph-mon[91579]: pgmap v12159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:27 smithi188 ceph-mon[92425]: pgmap v12159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:29 smithi161 ceph-mon[91579]: pgmap v12160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:29 smithi188 ceph-mon[92425]: pgmap v12160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:31 smithi161 ceph-mon[91579]: pgmap v12161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:31 smithi188 ceph-mon[92425]: pgmap v12161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:32.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:33 smithi161 ceph-mon[91579]: pgmap v12162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:33 smithi188 ceph-mon[92425]: pgmap v12162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:35 smithi161 ceph-mon[91579]: pgmap v12163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:35 smithi188 ceph-mon[92425]: pgmap v12163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:37 smithi161 ceph-mon[91579]: pgmap v12164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:37 smithi188 ceph-mon[92425]: pgmap v12164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:39 smithi161 ceph-mon[91579]: pgmap v12165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:39 smithi188 ceph-mon[92425]: pgmap v12165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:41 smithi161 ceph-mon[91579]: pgmap v12166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:41.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:41 smithi188 ceph-mon[92425]: pgmap v12166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:42.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:43 smithi161 ceph-mon[91579]: pgmap v12167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:43 smithi188 ceph-mon[92425]: pgmap v12167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:45 smithi161 ceph-mon[91579]: pgmap v12168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:45 smithi188 ceph-mon[92425]: pgmap v12168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:47 smithi161 ceph-mon[91579]: pgmap v12169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:47 smithi188 ceph-mon[92425]: pgmap v12169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:49 smithi161 ceph-mon[91579]: pgmap v12170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:49 smithi188 ceph-mon[92425]: pgmap v12170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:51 smithi161 ceph-mon[91579]: pgmap v12171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:51 smithi188 ceph-mon[92425]: pgmap v12171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:53 smithi161 ceph-mon[91579]: pgmap v12172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:53 smithi188 ceph-mon[92425]: pgmap v12172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-mon[91579]: pgmap v12173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:24:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:24:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:24:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:24:55.350+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:24:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:24:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:24:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:24:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:24:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:24:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:24:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:24:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:24:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:24:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:24:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:24:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:24:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:24:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:24:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:24:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:24:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:24:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:24:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:24:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:24:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:24:55.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:24:55.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:24:55.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:24:55.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:24:55.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:24:55.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:24:55.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:24:55.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:24:55.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:24:55.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:24:55.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:24:55.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:24:55.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:24:55.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:24:55.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:24:55.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:24:55.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:24:55.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:24:55.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:24:55.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:24:55.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:24:55.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:24:55.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:24:55.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=707] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:24:55.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=707] Local address: 172.21.15.161, port 57908 2022-07-03T17:24:55.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=707] Peer address: 172.21.15.101, port 22 2022-07-03T17:24:55.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=707] Connection lost 2022-07-03T17:24:55.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=707] Aborting connection 2022-07-03T17:24:55.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:24:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:24:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:55 smithi188 ceph-mon[92425]: pgmap v12173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:24:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:24:55.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:24:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:24:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:24:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:24:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:24:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:24:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:24:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:24:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:24:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:24:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:24:56.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:24:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:24:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:24:56.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:24:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:24:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:24:56.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:24:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:24:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:24:56.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:24:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:24:56.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:24:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:24:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:24:56.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:24:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:24:56.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:24:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:24:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:24:56.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:24:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:24:56.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:24:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:24:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:24:56.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:24:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:24:56.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:24:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:24:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:24:56.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:24:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:24:56.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:24:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:24:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:24:56.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:24:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:24:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=707] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:24:56.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=707] Local address: 172.21.15.161, port 57908 2022-07-03T17:24:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=707] Peer address: 172.21.15.101, port 22 2022-07-03T17:24:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=707] Connection lost 2022-07-03T17:24:56.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=707] Aborting connection 2022-07-03T17:24:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:24:56.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:24:56.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:24:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:24:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:24:56.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:24:56.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:24:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:24:56.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:24:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:24:56.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:24:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:24:56.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:24:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:24:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:24:56.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:24:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:24:56.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:24:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:24:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:24:56.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:24:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:24:56.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:24:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:24:56.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:24:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:24:56.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:24:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:24:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:24:56.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:24:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:24:56.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:24:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:24:56.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:24:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:24:56.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:24:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:24:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:24:56.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:24:56.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:24:56.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:24:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:24:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:24:56.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:24:56.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:24:56.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:24:56.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=707] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:24:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=707] Local address: 172.21.15.161, port 57908 2022-07-03T17:24:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=707] Peer address: 172.21.15.101, port 22 2022-07-03T17:24:56.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=707] Connection lost 2022-07-03T17:24:56.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=707] Aborting connection 2022-07-03T17:24:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:57 smithi161 ceph-mon[91579]: pgmap v12174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:57 smithi188 ceph-mon[92425]: pgmap v12174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:24:57.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:24:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:59 smithi161 ceph-mon[91579]: pgmap v12175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:24:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:24:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:24:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:24:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:59 smithi188 ceph-mon[92425]: pgmap v12175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:24:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:24:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:24:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:24:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:25:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:01 smithi161 ceph-mon[91579]: pgmap v12176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:01 smithi188 ceph-mon[92425]: pgmap v12176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:03 smithi161 ceph-mon[91579]: pgmap v12177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:03 smithi188 ceph-mon[92425]: pgmap v12177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:05 smithi161 ceph-mon[91579]: pgmap v12178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:05 smithi188 ceph-mon[92425]: pgmap v12178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:07 smithi161 ceph-mon[91579]: pgmap v12179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:07 smithi188 ceph-mon[92425]: pgmap v12179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:09 smithi161 ceph-mon[91579]: pgmap v12180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:09 smithi188 ceph-mon[92425]: pgmap v12180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:11 smithi161 ceph-mon[91579]: pgmap v12181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:11 smithi188 ceph-mon[92425]: pgmap v12181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:13 smithi161 ceph-mon[91579]: pgmap v12182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:13 smithi188 ceph-mon[92425]: pgmap v12182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:15 smithi161 ceph-mon[91579]: pgmap v12183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:15 smithi188 ceph-mon[92425]: pgmap v12183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:17 smithi161 ceph-mon[91579]: pgmap v12184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:17 smithi188 ceph-mon[92425]: pgmap v12184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:19 smithi161 ceph-mon[91579]: pgmap v12185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:19 smithi188 ceph-mon[92425]: pgmap v12185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:21 smithi161 ceph-mon[91579]: pgmap v12186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:21 smithi188 ceph-mon[92425]: pgmap v12186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:23 smithi161 ceph-mon[91579]: pgmap v12187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:23 smithi188 ceph-mon[92425]: pgmap v12187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:25 smithi188 ceph-mon[92425]: pgmap v12188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:25 smithi161 ceph-mon[91579]: pgmap v12188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:27 smithi188 ceph-mon[92425]: pgmap v12189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:27 smithi161 ceph-mon[91579]: pgmap v12189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:29 smithi188 ceph-mon[92425]: pgmap v12190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:29 smithi161 ceph-mon[91579]: pgmap v12190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:31 smithi188 ceph-mon[92425]: pgmap v12191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:31 smithi161 ceph-mon[91579]: pgmap v12191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:32.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:33 smithi188 ceph-mon[92425]: pgmap v12192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:33 smithi161 ceph-mon[91579]: pgmap v12192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:35 smithi188 ceph-mon[92425]: pgmap v12193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:35 smithi161 ceph-mon[91579]: pgmap v12193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:37 smithi188 ceph-mon[92425]: pgmap v12194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:37 smithi161 ceph-mon[91579]: pgmap v12194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:39 smithi188 ceph-mon[92425]: pgmap v12195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:39 smithi161 ceph-mon[91579]: pgmap v12195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:41 smithi188 ceph-mon[92425]: pgmap v12196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:41 smithi161 ceph-mon[91579]: pgmap v12196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:42.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:43 smithi188 ceph-mon[92425]: pgmap v12197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:43 smithi161 ceph-mon[91579]: pgmap v12197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:45 smithi188 ceph-mon[92425]: pgmap v12198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:45 smithi161 ceph-mon[91579]: pgmap v12198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:47 smithi188 ceph-mon[92425]: pgmap v12199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:47 smithi161 ceph-mon[91579]: pgmap v12199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:49 smithi161 ceph-mon[91579]: pgmap v12200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:49 smithi188 ceph-mon[92425]: pgmap v12200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:51 smithi161 ceph-mon[91579]: pgmap v12201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:51 smithi188 ceph-mon[92425]: pgmap v12201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:53 smithi161 ceph-mon[91579]: pgmap v12202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:53 smithi188 ceph-mon[92425]: pgmap v12202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:55.849 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-mon[91579]: pgmap v12203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:55.858 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:55 smithi188 ceph-mon[92425]: pgmap v12203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:25:55.845+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:25:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:25:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:25:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:25:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:25:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:25:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:25:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:25:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:25:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:25:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:25:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:25:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:25:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:25:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:25:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:25:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:25:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:25:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:25:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:25:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:25:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:25:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:25:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:25:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:25:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:25:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:25:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:25:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:25:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:25:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:25:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:25:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:25:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:25:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:25:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:25:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:25:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:25:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:25:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:25:56.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:25:56.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:25:56.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:25:56.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:25:56.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:25:56.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=709] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:25:56.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=709] Local address: 172.21.15.161, port 57936 2022-07-03T17:25:56.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=709] Peer address: 172.21.15.101, port 22 2022-07-03T17:25:56.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=709] Connection lost 2022-07-03T17:25:56.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=709] Aborting connection 2022-07-03T17:25:56.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:25:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:25:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:25:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:25:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:25:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:56 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:25:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:25:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:25:56.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:56.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:56 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:25:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:25:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:25:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:25:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:25:57.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:25:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:25:57.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:25:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:25:57.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:25:57.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:25:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:25:57.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:25:57.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:25:57.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:25:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:25:57.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:25:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:25:57.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:25:57.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:25:57.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:25:57.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:25:57.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:25:57.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:25:57.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:25:57.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:25:57.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:25:57.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:25:57.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:25:57.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:25:57.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:25:57.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:25:57.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:25:57.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:25:57.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:25:57.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:25:57.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:25:57.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:25:57.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:25:57.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:25:57.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:25:57.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:25:57.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:25:57.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:25:57.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:25:57.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:25:57.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:25:57.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=709] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:25:57.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=709] Local address: 172.21.15.161, port 57936 2022-07-03T17:25:57.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=709] Peer address: 172.21.15.101, port 22 2022-07-03T17:25:57.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=709] Connection lost 2022-07-03T17:25:57.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=709] Aborting connection 2022-07-03T17:25:57.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:57 smithi161 ceph-mon[91579]: pgmap v12204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:57.913 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:25:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:25:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:25:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:25:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:25:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:25:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:25:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:25:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:25:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:25:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:25:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:25:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:25:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:25:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:25:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:25:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:25:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:25:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:25:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:25:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:25:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:25:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:25:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:25:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:25:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:25:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:25:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:25:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:25:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:25:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:25:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:25:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:25:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:25:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:25:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:25:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:25:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:25:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:25:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:25:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:25:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:25:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:25:57.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:25:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:25:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:25:57.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=709] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:25:57.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=709] Local address: 172.21.15.161, port 57936 2022-07-03T17:25:57.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=709] Peer address: 172.21.15.101, port 22 2022-07-03T17:25:57.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=709] Connection lost 2022-07-03T17:25:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=709] Aborting connection 2022-07-03T17:25:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:57 smithi188 ceph-mon[92425]: pgmap v12204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:25:57.935 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:25:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:59 smithi161 ceph-mon[91579]: pgmap v12205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:25:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:25:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:25:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:25:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:59 smithi188 ceph-mon[92425]: pgmap v12205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:25:59.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:25:59.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:25:59.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:25:59.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:26:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:01 smithi161 ceph-mon[91579]: pgmap v12206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:01 smithi188 ceph-mon[92425]: pgmap v12206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:03 smithi161 ceph-mon[91579]: pgmap v12207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:03 smithi188 ceph-mon[92425]: pgmap v12207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:05 smithi161 ceph-mon[91579]: pgmap v12208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:05 smithi188 ceph-mon[92425]: pgmap v12208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:07 smithi161 ceph-mon[91579]: pgmap v12209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:07 smithi188 ceph-mon[92425]: pgmap v12209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:08.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:09 smithi161 ceph-mon[91579]: pgmap v12210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:09 smithi188 ceph-mon[92425]: pgmap v12210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:11 smithi161 ceph-mon[91579]: pgmap v12211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:11 smithi188 ceph-mon[92425]: pgmap v12211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:12.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:13 smithi161 ceph-mon[91579]: pgmap v12212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:13 smithi188 ceph-mon[92425]: pgmap v12212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:15 smithi161 ceph-mon[91579]: pgmap v12213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:15 smithi188 ceph-mon[92425]: pgmap v12213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:17 smithi161 ceph-mon[91579]: pgmap v12214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:17 smithi188 ceph-mon[92425]: pgmap v12214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:19 smithi161 ceph-mon[91579]: pgmap v12215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:19 smithi188 ceph-mon[92425]: pgmap v12215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:21 smithi161 ceph-mon[91579]: pgmap v12216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:21 smithi188 ceph-mon[92425]: pgmap v12216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:23 smithi161 ceph-mon[91579]: pgmap v12217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:23 smithi188 ceph-mon[92425]: pgmap v12217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:25 smithi161 ceph-mon[91579]: pgmap v12218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:25 smithi188 ceph-mon[92425]: pgmap v12218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:27 smithi161 ceph-mon[91579]: pgmap v12219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:27 smithi188 ceph-mon[92425]: pgmap v12219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:29 smithi161 ceph-mon[91579]: pgmap v12220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:29 smithi188 ceph-mon[92425]: pgmap v12220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:31 smithi161 ceph-mon[91579]: pgmap v12221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:31 smithi188 ceph-mon[92425]: pgmap v12221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:33 smithi161 ceph-mon[91579]: pgmap v12222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:33 smithi188 ceph-mon[92425]: pgmap v12222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:35 smithi161 ceph-mon[91579]: pgmap v12223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:35 smithi188 ceph-mon[92425]: pgmap v12223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:37 smithi161 ceph-mon[91579]: pgmap v12224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:37 smithi188 ceph-mon[92425]: pgmap v12224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:39 smithi161 ceph-mon[91579]: pgmap v12225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:39 smithi188 ceph-mon[92425]: pgmap v12225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:41 smithi161 ceph-mon[91579]: pgmap v12226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:41 smithi188 ceph-mon[92425]: pgmap v12226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:43 smithi161 ceph-mon[91579]: pgmap v12227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:43 smithi188 ceph-mon[92425]: pgmap v12227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:45 smithi161 ceph-mon[91579]: pgmap v12228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:45 smithi188 ceph-mon[92425]: pgmap v12228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:47 smithi161 ceph-mon[91579]: pgmap v12229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:47 smithi188 ceph-mon[92425]: pgmap v12229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:48.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:48.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:49 smithi161 ceph-mon[91579]: pgmap v12230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:49 smithi188 ceph-mon[92425]: pgmap v12230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:51 smithi161 ceph-mon[91579]: pgmap v12231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:51 smithi188 ceph-mon[92425]: pgmap v12231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:53 smithi161 ceph-mon[91579]: pgmap v12232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:53 smithi188 ceph-mon[92425]: pgmap v12232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:53.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:55 smithi161 ceph-mon[91579]: pgmap v12233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:55 smithi188 ceph-mon[92425]: pgmap v12233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:26:56.769+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:26:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:26:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:26:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:26:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:26:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:26:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:26:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:26:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:26:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:26:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:26:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:26:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:26:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:26:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:26:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:26:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:26:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:26:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:26:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:26:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:26:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:26:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:26:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:26:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:26:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:26:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:26:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:26:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:26:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:26:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:26:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:26:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:26:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:26:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:26:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:26:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:26:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:26:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:26:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:26:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:26:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:26:57.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:26:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:26:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:26:57.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=711] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:26:57.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=711] Local address: 172.21.15.161, port 57964 2022-07-03T17:26:57.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=711] Peer address: 172.21.15.101, port 22 2022-07-03T17:26:57.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=711] Connection lost 2022-07-03T17:26:57.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=711] Aborting connection 2022-07-03T17:26:57.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:26:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:26:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:57 smithi161 ceph-mon[91579]: pgmap v12234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:26:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:26:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:26:57.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:57 smithi188 ceph-mon[92425]: pgmap v12234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:26:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:26:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:26:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:26:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:26:58.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:26:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:26:58.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:26:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:26:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:26:58.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:26:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:26:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:26:58.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:26:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:26:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:26:58.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:26:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:26:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:26:58.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:26:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:26:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:26:58.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:26:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:26:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:26:58.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:26:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:26:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:26:58.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:26:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:26:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:26:58.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:26:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:26:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:26:58.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:26:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:26:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:26:58.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:26:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:26:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:26:58.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:26:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:26:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:26:58.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:26:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:26:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:26:58.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:26:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:26:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:26:58.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:26:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:26:58.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=711] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:26:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=711] Local address: 172.21.15.161, port 57964 2022-07-03T17:26:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=711] Peer address: 172.21.15.101, port 22 2022-07-03T17:26:58.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=711] Connection lost 2022-07-03T17:26:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=711] Aborting connection 2022-07-03T17:26:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:58.906 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:26:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:26:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:26:58.907 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:26:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:26:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:26:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:26:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:26:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:26:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:26:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:26:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:26:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:26:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:26:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:26:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:26:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:26:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:26:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:26:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:26:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:26:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:26:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:26:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:26:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:26:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:26:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:26:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:26:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:26:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:26:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:26:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:26:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:26:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:26:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:26:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:26:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:26:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:26:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:26:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:26:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:26:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:26:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:26:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:26:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:26:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:26:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:26:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:26:58.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:26:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:26:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=711] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:26:58.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=711] Local address: 172.21.15.161, port 57964 2022-07-03T17:26:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=711] Peer address: 172.21.15.101, port 22 2022-07-03T17:26:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=711] Connection lost 2022-07-03T17:26:58.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=711] Aborting connection 2022-07-03T17:26:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:26:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:26:58.936 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:26:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:26:58.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:26:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:26:59 smithi161 ceph-mon[91579]: pgmap v12235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:26:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:26:59 smithi188 ceph-mon[92425]: pgmap v12235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:01 smithi161 ceph-mon[91579]: pgmap v12236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:01 smithi188 ceph-mon[92425]: pgmap v12236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:03 smithi161 ceph-mon[91579]: pgmap v12237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:03 smithi188 ceph-mon[92425]: pgmap v12237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:05 smithi161 ceph-mon[91579]: pgmap v12238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:05 smithi188 ceph-mon[92425]: pgmap v12238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:07 smithi161 ceph-mon[91579]: pgmap v12239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:07 smithi188 ceph-mon[92425]: pgmap v12239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:08.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:09 smithi161 ceph-mon[91579]: pgmap v12240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:09 smithi188 ceph-mon[92425]: pgmap v12240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:11 smithi161 ceph-mon[91579]: pgmap v12241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:11 smithi188 ceph-mon[92425]: pgmap v12241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:13 smithi161 ceph-mon[91579]: pgmap v12242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:13 smithi188 ceph-mon[92425]: pgmap v12242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:15 smithi161 ceph-mon[91579]: pgmap v12243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:15 smithi188 ceph-mon[92425]: pgmap v12243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:17 smithi161 ceph-mon[91579]: pgmap v12244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:17 smithi188 ceph-mon[92425]: pgmap v12244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:18.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:19 smithi161 ceph-mon[91579]: pgmap v12245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:19 smithi188 ceph-mon[92425]: pgmap v12245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:21 smithi161 ceph-mon[91579]: pgmap v12246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:21.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:21 smithi188 ceph-mon[92425]: pgmap v12246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:23 smithi161 ceph-mon[91579]: pgmap v12247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:23 smithi188 ceph-mon[92425]: pgmap v12247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:25 smithi161 ceph-mon[91579]: pgmap v12248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:25 smithi188 ceph-mon[92425]: pgmap v12248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:27 smithi161 ceph-mon[91579]: pgmap v12249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:27 smithi188 ceph-mon[92425]: pgmap v12249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:29 smithi161 ceph-mon[91579]: pgmap v12250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:29 smithi188 ceph-mon[92425]: pgmap v12250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:31 smithi161 ceph-mon[91579]: pgmap v12251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:31 smithi188 ceph-mon[92425]: pgmap v12251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:33 smithi161 ceph-mon[91579]: pgmap v12252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:33 smithi188 ceph-mon[92425]: pgmap v12252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:35 smithi161 ceph-mon[91579]: pgmap v12253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:35 smithi188 ceph-mon[92425]: pgmap v12253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:37 smithi188 ceph-mon[92425]: pgmap v12254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:37 smithi161 ceph-mon[91579]: pgmap v12254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:39 smithi188 ceph-mon[92425]: pgmap v12255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:39 smithi161 ceph-mon[91579]: pgmap v12255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:41 smithi188 ceph-mon[92425]: pgmap v12256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:41 smithi161 ceph-mon[91579]: pgmap v12256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:43 smithi188 ceph-mon[92425]: pgmap v12257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:43 smithi161 ceph-mon[91579]: pgmap v12257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:45 smithi188 ceph-mon[92425]: pgmap v12258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:45 smithi161 ceph-mon[91579]: pgmap v12258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:47 smithi188 ceph-mon[92425]: pgmap v12259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:47 smithi161 ceph-mon[91579]: pgmap v12259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:49 smithi188 ceph-mon[92425]: pgmap v12260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:49 smithi161 ceph-mon[91579]: pgmap v12260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:51.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:51 smithi188 ceph-mon[92425]: pgmap v12261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:51 smithi161 ceph-mon[91579]: pgmap v12261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:53 smithi161 ceph-mon[91579]: pgmap v12262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:53 smithi188 ceph-mon[92425]: pgmap v12262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:27:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:55 smithi161 ceph-mon[91579]: pgmap v12263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:55 smithi188 ceph-mon[92425]: pgmap v12263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:27:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:27:57.271+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:27:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:27:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:27:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:27:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:27:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:27:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:27:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:27:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:27:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:27:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:27:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:27:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:27:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:27:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:27:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:27:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:27:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:27:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:27:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:27:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:27:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:27:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:27:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:27:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:27:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:27:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:27:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:27:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:27:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:27:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:27:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:27:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:27:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:27:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:27:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:27:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:27:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:27:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:27:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:27:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:27:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:27:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:27:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:27:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:27:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=713] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:27:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=713] Local address: 172.21.15.161, port 57994 2022-07-03T17:27:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=713] Peer address: 172.21.15.101, port 22 2022-07-03T17:27:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=713] Connection lost 2022-07-03T17:27:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=713] Aborting connection 2022-07-03T17:27:57.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:27:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-mon[91579]: pgmap v12264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:27:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:27:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:57 smithi188 ceph-mon[92425]: pgmap v12264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:27:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:27:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:27:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:27:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:27:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:27:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:27:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:27:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:27:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:27:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:27:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:27:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:27:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:27:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:27:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:27:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:27:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:27:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:27:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:27:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:27:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:27:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:27:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:27:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:27:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:27:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:27:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:27:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:27:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:27:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:27:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:27:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:27:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:27:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:27:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:27:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:27:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:27:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:27:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:27:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:27:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:27:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:27:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:27:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:27:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:27:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:27:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:27:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:27:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:27:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=713] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:27:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=713] Local address: 172.21.15.161, port 57994 2022-07-03T17:27:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=713] Peer address: 172.21.15.101, port 22 2022-07-03T17:27:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=713] Connection lost 2022-07-03T17:27:59.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=713] Aborting connection 2022-07-03T17:27:59.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:27:59.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:27:59.150 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:27:59.150 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:27:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:27:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:27:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:27:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:27:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:27:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:27:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:27:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:27:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:27:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:27:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:27:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:27:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:27:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:27:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:27:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:27:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:27:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:27:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:27:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:27:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:27:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:27:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:27:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:27:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:27:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:27:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:27:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:27:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:27:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:27:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:27:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:27:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:27:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:27:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:27:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:27:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:27:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:27:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:27:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:27:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:27:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:27:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:27:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:27:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:27:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:27:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=713] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:27:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=713] Local address: 172.21.15.161, port 57994 2022-07-03T17:27:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=713] Peer address: 172.21.15.101, port 22 2022-07-03T17:27:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=713] Connection lost 2022-07-03T17:27:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=713] Aborting connection 2022-07-03T17:27:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:27:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:27:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:27:59.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:27:59 smithi161 ceph-mon[91579]: pgmap v12265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:27:59 smithi188 ceph-mon[92425]: pgmap v12265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:00 smithi161 ceph-mon[91579]: pgmap v12266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:00 smithi188 ceph-mon[92425]: pgmap v12266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:03 smithi161 ceph-mon[91579]: pgmap v12267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:03 smithi188 ceph-mon[92425]: pgmap v12267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:05 smithi161 ceph-mon[91579]: pgmap v12268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:05 smithi188 ceph-mon[92425]: pgmap v12268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:07 smithi161 ceph-mon[91579]: pgmap v12269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:07 smithi188 ceph-mon[92425]: pgmap v12269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:09 smithi161 ceph-mon[91579]: pgmap v12270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:09 smithi188 ceph-mon[92425]: pgmap v12270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:11 smithi161 ceph-mon[91579]: pgmap v12271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:11 smithi188 ceph-mon[92425]: pgmap v12271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:13 smithi161 ceph-mon[91579]: pgmap v12272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:13 smithi188 ceph-mon[92425]: pgmap v12272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:15 smithi161 ceph-mon[91579]: pgmap v12273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:15 smithi188 ceph-mon[92425]: pgmap v12273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:17 smithi161 ceph-mon[91579]: pgmap v12274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:17 smithi188 ceph-mon[92425]: pgmap v12274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:19 smithi161 ceph-mon[91579]: pgmap v12275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:19 smithi188 ceph-mon[92425]: pgmap v12275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:21 smithi161 ceph-mon[91579]: pgmap v12276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:21 smithi188 ceph-mon[92425]: pgmap v12276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:23 smithi161 ceph-mon[91579]: pgmap v12277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:23 smithi188 ceph-mon[92425]: pgmap v12277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:25 smithi161 ceph-mon[91579]: pgmap v12278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:25 smithi188 ceph-mon[92425]: pgmap v12278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:27 smithi161 ceph-mon[91579]: pgmap v12279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:27 smithi188 ceph-mon[92425]: pgmap v12279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:29 smithi161 ceph-mon[91579]: pgmap v12280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:29 smithi188 ceph-mon[92425]: pgmap v12280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:31 smithi161 ceph-mon[91579]: pgmap v12281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:31 smithi188 ceph-mon[92425]: pgmap v12281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:32.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:33 smithi161 ceph-mon[91579]: pgmap v12282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:33 smithi188 ceph-mon[92425]: pgmap v12282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:35 smithi161 ceph-mon[91579]: pgmap v12283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:35 smithi188 ceph-mon[92425]: pgmap v12283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:37 smithi161 ceph-mon[91579]: pgmap v12284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:37 smithi188 ceph-mon[92425]: pgmap v12284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:39 smithi161 ceph-mon[91579]: pgmap v12285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:39 smithi188 ceph-mon[92425]: pgmap v12285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:41 smithi161 ceph-mon[91579]: pgmap v12286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:41 smithi188 ceph-mon[92425]: pgmap v12286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:43 smithi161 ceph-mon[91579]: pgmap v12287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:43 smithi188 ceph-mon[92425]: pgmap v12287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:45 smithi161 ceph-mon[91579]: pgmap v12288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:45 smithi188 ceph-mon[92425]: pgmap v12288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:47 smithi161 ceph-mon[91579]: pgmap v12289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:47 smithi188 ceph-mon[92425]: pgmap v12289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:47.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:49 smithi161 ceph-mon[91579]: pgmap v12290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:49 smithi188 ceph-mon[92425]: pgmap v12290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:51 smithi161 ceph-mon[91579]: pgmap v12291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:51 smithi188 ceph-mon[92425]: pgmap v12291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:53 smithi161 ceph-mon[91579]: pgmap v12292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:53 smithi188 ceph-mon[92425]: pgmap v12292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:55 smithi161 ceph-mon[91579]: pgmap v12293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:55 smithi188 ceph-mon[92425]: pgmap v12293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-mon[91579]: pgmap v12294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:57 smithi188 ceph-mon[92425]: pgmap v12294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:28:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:28:58.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:28:57.771+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:28:58.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:28:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:28:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:28:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:28:58.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:28:58.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:28:58.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:28:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:28:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:28:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:28:58.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:28:58.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:28:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:28:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:28:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:28:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:28:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:28:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:28:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:28:58.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:28:58.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:28:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:28:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:28:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:28:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:28:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:28:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:28:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:28:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:28:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:28:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:28:58.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:28:58.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:28:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:28:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:28:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:28:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:28:58.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:28:58.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:28:58.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:28:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:28:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:28:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:28:58.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:28:58.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=715] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:28:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=715] Local address: 172.21.15.161, port 58022 2022-07-03T17:28:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=715] Peer address: 172.21.15.101, port 22 2022-07-03T17:28:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=715] Connection lost 2022-07-03T17:28:58.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=715] Aborting connection 2022-07-03T17:28:58.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:28:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:28:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:28:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:28:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:28:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:28:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:28:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:28:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:28:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:28:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:28:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:28:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:28:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:28:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:28:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:28:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:28:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:28:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:28:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:28:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:28:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:28:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:28:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:28:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:28:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:28:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:28:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:28:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:28:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:28:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:28:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:28:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:28:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:28:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:28:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:28:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:28:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:28:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:28:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:28:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:28:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:28:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:28:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:28:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:28:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:28:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:28:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:28:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:28:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:28:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=715] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:28:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=715] Local address: 172.21.15.161, port 58022 2022-07-03T17:28:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=715] Peer address: 172.21.15.101, port 22 2022-07-03T17:28:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=715] Connection lost 2022-07-03T17:28:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=715] Aborting connection 2022-07-03T17:28:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: pgmap v12295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:28:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:28:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:28:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:28:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:28:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:28:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:28:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:28:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:28:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:28:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:28:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:28:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:28:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:28:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:28:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:28:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:28:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:28:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:28:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:28:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:28:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:28:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:28:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:28:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:28:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:28:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:28:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:28:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:28:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:28:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:28:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:28:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:28:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:28:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:28:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:28:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:28:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:28:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:28:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:28:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:28:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:28:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:28:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:28:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:28:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:28:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:28:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:28:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:28:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:28:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=715] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:28:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=715] Local address: 172.21.15.161, port 58022 2022-07-03T17:28:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=715] Peer address: 172.21.15.101, port 22 2022-07-03T17:28:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=715] Connection lost 2022-07-03T17:28:59.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=715] Aborting connection 2022-07-03T17:28:59.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: pgmap v12295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:28:59.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:28:59.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:28:59.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:28:59.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:29:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: pgmap v12296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: pgmap v12296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:29:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:01.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:29:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 67779k 2022-07-03T17:29:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 69406564: error parsing value: Value '69406564' is below minimum 939524096 2022-07-03T17:29:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:02 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 180.3M 2022-07-03T17:29:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:02 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 189111574: error parsing value: Value '189111574' is below minimum 939524096 2022-07-03T17:29:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 67779k 2022-07-03T17:29:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 69406564: error parsing value: Value '69406564' is below minimum 939524096 2022-07-03T17:29:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:02 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 180.3M 2022-07-03T17:29:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:02 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 189111574: error parsing value: Value '189111574' is below minimum 939524096 2022-07-03T17:29:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:03 smithi161 ceph-mon[91579]: pgmap v12297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:03 smithi188 ceph-mon[92425]: pgmap v12297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:05 smithi161 ceph-mon[91579]: pgmap v12298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:05 smithi188 ceph-mon[92425]: pgmap v12298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:07 smithi161 ceph-mon[91579]: pgmap v12299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:07.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:07 smithi188 ceph-mon[92425]: pgmap v12299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:09 smithi161 ceph-mon[91579]: pgmap v12300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:09 smithi188 ceph-mon[92425]: pgmap v12300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:11 smithi161 ceph-mon[91579]: pgmap v12301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:11 smithi188 ceph-mon[92425]: pgmap v12301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:12.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:13 smithi161 ceph-mon[91579]: pgmap v12302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:13 smithi188 ceph-mon[92425]: pgmap v12302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:15 smithi161 ceph-mon[91579]: pgmap v12303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:15 smithi188 ceph-mon[92425]: pgmap v12303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:17 smithi161 ceph-mon[91579]: pgmap v12304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:17 smithi188 ceph-mon[92425]: pgmap v12304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:19 smithi161 ceph-mon[91579]: pgmap v12305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:19 smithi188 ceph-mon[92425]: pgmap v12305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:21 smithi161 ceph-mon[91579]: pgmap v12306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:21 smithi188 ceph-mon[92425]: pgmap v12306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:23 smithi161 ceph-mon[91579]: pgmap v12307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:23 smithi188 ceph-mon[92425]: pgmap v12307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:25 smithi161 ceph-mon[91579]: pgmap v12308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:25 smithi188 ceph-mon[92425]: pgmap v12308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:27 smithi161 ceph-mon[91579]: pgmap v12309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:27 smithi188 ceph-mon[92425]: pgmap v12309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:29 smithi161 ceph-mon[91579]: pgmap v12310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:29 smithi188 ceph-mon[92425]: pgmap v12310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:31 smithi161 ceph-mon[91579]: pgmap v12311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:31 smithi188 ceph-mon[92425]: pgmap v12311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:33 smithi161 ceph-mon[91579]: pgmap v12312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:33 smithi188 ceph-mon[92425]: pgmap v12312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:35 smithi161 ceph-mon[91579]: pgmap v12313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:35 smithi188 ceph-mon[92425]: pgmap v12313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:37 smithi161 ceph-mon[91579]: pgmap v12314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:37 smithi188 ceph-mon[92425]: pgmap v12314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:39 smithi161 ceph-mon[91579]: pgmap v12315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:39 smithi188 ceph-mon[92425]: pgmap v12315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:41 smithi161 ceph-mon[91579]: pgmap v12316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:41 smithi188 ceph-mon[92425]: pgmap v12316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:43 smithi161 ceph-mon[91579]: pgmap v12317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:43 smithi188 ceph-mon[92425]: pgmap v12317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:45 smithi161 ceph-mon[91579]: pgmap v12318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:45 smithi188 ceph-mon[92425]: pgmap v12318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:47 smithi161 ceph-mon[91579]: pgmap v12319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:47 smithi188 ceph-mon[92425]: pgmap v12319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:49 smithi161 ceph-mon[91579]: pgmap v12320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:49 smithi188 ceph-mon[92425]: pgmap v12320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:51 smithi161 ceph-mon[91579]: pgmap v12321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:51 smithi188 ceph-mon[92425]: pgmap v12321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:53 smithi161 ceph-mon[91579]: pgmap v12322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:53 smithi188 ceph-mon[92425]: pgmap v12322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:55 smithi161 ceph-mon[91579]: pgmap v12323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:55 smithi188 ceph-mon[92425]: pgmap v12323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:57 smithi161 ceph-mon[91579]: pgmap v12324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:57 smithi188 ceph-mon[92425]: pgmap v12324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:29:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:29:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:29:58.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:29:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:29:58.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:29:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:29:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:29:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:29:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:29:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:29:59 smithi161 ceph-mon[91579]: pgmap v12325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:29:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:29:59 smithi188 ceph-mon[92425]: pgmap v12325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:30:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:30:01.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:30:01.038+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:30:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:30:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:30:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:30:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:30:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:30:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:30:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:30:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:30:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:30:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:30:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:30:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:30:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:30:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:30:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:30:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:30:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:30:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:30:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:30:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:30:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:30:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:30:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:30:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:30:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:30:01.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:30:01.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:30:01.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:30:01.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:30:01.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:30:01.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:30:01.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:30:01.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:30:01.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:30:01.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:30:01.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:30:01.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:30:01.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:30:01.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:30:01.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:30:01.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:30:01.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:30:01.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:30:01.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:30:01.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=717] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:30:01.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=717] Local address: 172.21.15.161, port 58050 2022-07-03T17:30:01.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=717] Peer address: 172.21.15.101, port 22 2022-07-03T17:30:01.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=717] Connection lost 2022-07-03T17:30:01.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=717] Aborting connection 2022-07-03T17:30:01.414 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:30:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-mon[91579]: pgmap v12326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:30:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:30:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:30:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:01 smithi188 ceph-mon[92425]: pgmap v12326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:30:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:30:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:30:02.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:30:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:30:01.971+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T17:30:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:30:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:30:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:30:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:30:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:30:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:30:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:30:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:30:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:30:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:30:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:30:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:30:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:30:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:30:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:30:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:30:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:30:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:30:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:30:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:30:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:30:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:30:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:30:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:30:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:30:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:30:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:30:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:30:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:30:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:30:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:30:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:30:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:30:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:30:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:30:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:30:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:30:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:30:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:30:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:30:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:30:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:30:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:30:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:30:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=717] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:30:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=717] Local address: 172.21.15.161, port 58050 2022-07-03T17:30:02.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=717] Peer address: 172.21.15.101, port 22 2022-07-03T17:30:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=717] Connection lost 2022-07-03T17:30:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=717] Aborting connection 2022-07-03T17:30:02.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:30:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:30:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:30:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:30:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:30:03.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:30:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:30:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:30:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:30:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:30:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:30:03.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:30:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:30:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:30:03.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:30:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:30:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:30:03.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:30:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:30:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:30:03.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:30:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:30:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:30:03.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:30:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:30:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:30:03.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:30:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:30:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:30:03.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:30:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:30:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:30:03.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:30:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:30:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:30:03.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:30:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:30:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:30:03.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:30:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:30:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:30:03.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:30:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:30:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:30:03.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:30:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:30:03.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:30:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=717] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:30:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=717] Local address: 172.21.15.161, port 58050 2022-07-03T17:30:03.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=717] Peer address: 172.21.15.101, port 22 2022-07-03T17:30:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=717] Connection lost 2022-07-03T17:30:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=717] Aborting connection 2022-07-03T17:30:03.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:30:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:03 smithi188 ceph-mon[92425]: pgmap v12327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:03 smithi161 ceph-mon[91579]: pgmap v12327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:05 smithi188 ceph-mon[92425]: pgmap v12328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:05 smithi161 ceph-mon[91579]: pgmap v12328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:07 smithi188 ceph-mon[92425]: pgmap v12329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:07 smithi161 ceph-mon[91579]: pgmap v12329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:09.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:09 smithi188 ceph-mon[92425]: pgmap v12330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:09 smithi161 ceph-mon[91579]: pgmap v12330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:11 smithi188 ceph-mon[92425]: pgmap v12331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:11 smithi161 ceph-mon[91579]: pgmap v12331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:13 smithi161 ceph-mon[91579]: pgmap v12332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:13 smithi188 ceph-mon[92425]: pgmap v12332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:15 smithi161 ceph-mon[91579]: pgmap v12333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:15 smithi188 ceph-mon[92425]: pgmap v12333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:17 smithi161 ceph-mon[91579]: pgmap v12334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:17 smithi188 ceph-mon[92425]: pgmap v12334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:19 smithi161 ceph-mon[91579]: pgmap v12335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:19 smithi188 ceph-mon[92425]: pgmap v12335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:21 smithi161 ceph-mon[91579]: pgmap v12336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:21 smithi188 ceph-mon[92425]: pgmap v12336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:23 smithi161 ceph-mon[91579]: pgmap v12337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:23 smithi188 ceph-mon[92425]: pgmap v12337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:25 smithi161 ceph-mon[91579]: pgmap v12338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:25 smithi188 ceph-mon[92425]: pgmap v12338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:27 smithi161 ceph-mon[91579]: pgmap v12339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:27 smithi188 ceph-mon[92425]: pgmap v12339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:29 smithi161 ceph-mon[91579]: pgmap v12340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:29 smithi188 ceph-mon[92425]: pgmap v12340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:31 smithi161 ceph-mon[91579]: pgmap v12341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:31 smithi188 ceph-mon[92425]: pgmap v12341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:33 smithi161 ceph-mon[91579]: pgmap v12342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:33 smithi188 ceph-mon[92425]: pgmap v12342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:35 smithi161 ceph-mon[91579]: pgmap v12343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:35 smithi188 ceph-mon[92425]: pgmap v12343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:37 smithi161 ceph-mon[91579]: pgmap v12344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:37 smithi188 ceph-mon[92425]: pgmap v12344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:39 smithi161 ceph-mon[91579]: pgmap v12345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:39 smithi188 ceph-mon[92425]: pgmap v12345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:41.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:41 smithi161 ceph-mon[91579]: pgmap v12346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:41 smithi188 ceph-mon[92425]: pgmap v12346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:43 smithi161 ceph-mon[91579]: pgmap v12347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:43 smithi188 ceph-mon[92425]: pgmap v12347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:45 smithi161 ceph-mon[91579]: pgmap v12348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:45 smithi188 ceph-mon[92425]: pgmap v12348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:47 smithi161 ceph-mon[91579]: pgmap v12349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:47 smithi188 ceph-mon[92425]: pgmap v12349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:49 smithi161 ceph-mon[91579]: pgmap v12350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:49 smithi188 ceph-mon[92425]: pgmap v12350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:51 smithi161 ceph-mon[91579]: pgmap v12351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:51 smithi188 ceph-mon[92425]: pgmap v12351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:53 smithi161 ceph-mon[91579]: pgmap v12352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:53 smithi188 ceph-mon[92425]: pgmap v12352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:55 smithi161 ceph-mon[91579]: pgmap v12353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:55 smithi188 ceph-mon[92425]: pgmap v12353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:57 smithi161 ceph-mon[91579]: pgmap v12354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:57 smithi188 ceph-mon[92425]: pgmap v12354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:30:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:59 smithi161 ceph-mon[91579]: pgmap v12355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:30:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:30:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:30:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:30:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:59 smithi188 ceph-mon[92425]: pgmap v12355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:30:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:30:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:30:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:30:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:31:01.557 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-mon[91579]: pgmap v12356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:01.558 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:31:01.544+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:31:01.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:31:01.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:31:01.560 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:31:01.560 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:31:01.561 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:31:01.562 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:31:01.562 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:31:01.563 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:31:01.563 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:31:01.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:31:01.565 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:31:01.565 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:31:01.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:31:01.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:31:01.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:31:01.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:31:01.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:31:01.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:31:01.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:31:01.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:31:01.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:31:01.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:31:01.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:31:01.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:31:01.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:31:01.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:31:01.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:31:01.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:31:01.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:31:01.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:31:01.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:31:01.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:31:01.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:31:01.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:31:01.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:31:01.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:31:01.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:31:01.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:31:01.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:31:01.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:31:01.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:31:01.577 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:31:01.577 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:31:01.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:31:01.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=719] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:31:01.578 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=719] Local address: 172.21.15.161, port 58056 2022-07-03T17:31:01.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=719] Peer address: 172.21.15.101, port 22 2022-07-03T17:31:01.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=719] Connection lost 2022-07-03T17:31:01.579 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=719] Aborting connection 2022-07-03T17:31:01.580 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:31:01.580 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:01 smithi188 ceph-mon[92425]: pgmap v12356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:31:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:31:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:31:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:31:02.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:31:02.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:31:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:31:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:31:02.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:31:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:31:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:31:02.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:31:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:31:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:31:02.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:31:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:31:02.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:31:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:31:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:31:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:31:02.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:31:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:31:02.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:31:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:31:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:31:02.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:31:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:31:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:31:02.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:31:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:31:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:31:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:31:02.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:31:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:31:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:31:02.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:31:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:31:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:31:02.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:31:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:31:02.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:31:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:31:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:31:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:31:02.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:31:02.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:31:02.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:31:02.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:31:02.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:31:02.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=719] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:31:02.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=719] Local address: 172.21.15.161, port 58056 2022-07-03T17:31:02.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=719] Peer address: 172.21.15.101, port 22 2022-07-03T17:31:02.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=719] Connection lost 2022-07-03T17:31:02.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=719] Aborting connection 2022-07-03T17:31:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:31:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:31:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:31:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:31:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:31:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:31:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:31:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:31:02.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:31:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:31:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:31:02.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:31:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:31:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:31:02.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:31:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:31:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:31:02.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:31:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:31:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:31:02.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:31:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:31:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:31:02.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:31:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:31:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:31:02.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:31:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:31:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:31:02.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:31:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:31:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:31:02.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:31:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:31:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:31:02.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:31:02.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:31:02.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:31:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:31:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:31:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:31:02.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:31:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:31:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:31:02.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:31:02.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:31:02.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:31:02.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:31:02.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:31:02.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=719] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:31:02.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=719] Local address: 172.21.15.161, port 58056 2022-07-03T17:31:02.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=719] Peer address: 172.21.15.101, port 22 2022-07-03T17:31:02.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=719] Connection lost 2022-07-03T17:31:02.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=719] Aborting connection 2022-07-03T17:31:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:03 smithi161 ceph-mon[91579]: pgmap v12357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:03 smithi188 ceph-mon[92425]: pgmap v12357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:05 smithi161 ceph-mon[91579]: pgmap v12358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:05 smithi188 ceph-mon[92425]: pgmap v12358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:07 smithi161 ceph-mon[91579]: pgmap v12359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:07 smithi188 ceph-mon[92425]: pgmap v12359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:09 smithi161 ceph-mon[91579]: pgmap v12360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:09 smithi188 ceph-mon[92425]: pgmap v12360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:11 smithi161 ceph-mon[91579]: pgmap v12361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:11 smithi188 ceph-mon[92425]: pgmap v12361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:13 smithi161 ceph-mon[91579]: pgmap v12362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:13 smithi188 ceph-mon[92425]: pgmap v12362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:31:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:15 smithi161 ceph-mon[91579]: pgmap v12363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:15 smithi188 ceph-mon[92425]: pgmap v12363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:17 smithi161 ceph-mon[91579]: pgmap v12364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:17 smithi188 ceph-mon[92425]: pgmap v12364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:19 smithi161 ceph-mon[91579]: pgmap v12365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:19 smithi188 ceph-mon[92425]: pgmap v12365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:21 smithi161 ceph-mon[91579]: pgmap v12366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:21 smithi188 ceph-mon[92425]: pgmap v12366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:23 smithi161 ceph-mon[91579]: pgmap v12367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:23.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:23 smithi188 ceph-mon[92425]: pgmap v12367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:25 smithi161 ceph-mon[91579]: pgmap v12368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:25 smithi188 ceph-mon[92425]: pgmap v12368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:27 smithi161 ceph-mon[91579]: pgmap v12369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:27 smithi188 ceph-mon[92425]: pgmap v12369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:29 smithi161 ceph-mon[91579]: pgmap v12370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:29 smithi188 ceph-mon[92425]: pgmap v12370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:31 smithi161 ceph-mon[91579]: pgmap v12371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:31 smithi188 ceph-mon[92425]: pgmap v12371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:33 smithi161 ceph-mon[91579]: pgmap v12372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:33 smithi188 ceph-mon[92425]: pgmap v12372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:35 smithi161 ceph-mon[91579]: pgmap v12373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:35 smithi188 ceph-mon[92425]: pgmap v12373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:37 smithi161 ceph-mon[91579]: pgmap v12374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:37 smithi188 ceph-mon[92425]: pgmap v12374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:39 smithi161 ceph-mon[91579]: pgmap v12375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:39.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:39 smithi188 ceph-mon[92425]: pgmap v12375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:41 smithi161 ceph-mon[91579]: pgmap v12376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:41 smithi188 ceph-mon[92425]: pgmap v12376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:43.638 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:43 smithi161 ceph-mon[91579]: pgmap v12377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:43.830 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:43 smithi188 ceph-mon[92425]: pgmap v12377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:45 smithi161 ceph-mon[91579]: pgmap v12378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:45 smithi188 ceph-mon[92425]: pgmap v12378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:47 smithi161 ceph-mon[91579]: pgmap v12379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:47 smithi188 ceph-mon[92425]: pgmap v12379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:49 smithi161 ceph-mon[91579]: pgmap v12380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:49 smithi188 ceph-mon[92425]: pgmap v12380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:51.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:51 smithi161 ceph-mon[91579]: pgmap v12381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:51 smithi188 ceph-mon[92425]: pgmap v12381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:53 smithi161 ceph-mon[91579]: pgmap v12382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:53 smithi188 ceph-mon[92425]: pgmap v12382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:55 smithi161 ceph-mon[91579]: pgmap v12383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:55 smithi188 ceph-mon[92425]: pgmap v12383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:57 smithi161 ceph-mon[91579]: pgmap v12384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:57 smithi188 ceph-mon[92425]: pgmap v12384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:31:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:59 smithi161 ceph-mon[91579]: pgmap v12385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:31:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:31:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:31:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:31:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:59 smithi188 ceph-mon[92425]: pgmap v12385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:31:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:31:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:31:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:31:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:32:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:01 smithi161 ceph-mon[91579]: pgmap v12386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:01 smithi188 ceph-mon[92425]: pgmap v12386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:02.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:32:02.045+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:32:02.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:32:02.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:32:02.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:32:02.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:32:02.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:32:02.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:32:02.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:32:02.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:32:02.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:32:02.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:32:02.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:32:02.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:32:02.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:32:02.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:32:02.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:32:02.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:32:02.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:32:02.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:32:02.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:32:02.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:32:02.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:32:02.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:32:02.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:32:02.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:32:02.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:32:02.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:32:02.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:32:02.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:32:02.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:32:02.334 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:32:02.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:32:02.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:32:02.335 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:32:02.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:32:02.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:32:02.336 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:32:02.337 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:32:02.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:32:02.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:32:02.338 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:32:02.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:32:02.339 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:32:02.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:32:02.340 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:32:02.341 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=721] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:32:02.342 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=721] Local address: 172.21.15.161, port 58060 2022-07-03T17:32:02.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=721] Peer address: 172.21.15.101, port 22 2022-07-03T17:32:02.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=721] Connection lost 2022-07-03T17:32:02.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=721] Aborting connection 2022-07-03T17:32:02.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:32:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:32:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:32:02.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:32:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:32:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:32:02.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:32:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:32:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:32:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:32:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:32:03.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:32:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:32:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:32:03.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:32:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:32:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:32:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:32:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:32:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:32:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:32:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:32:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:32:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:32:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:32:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:32:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:32:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:32:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:32:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:32:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:32:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:32:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:32:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:32:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:32:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:32:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:32:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:32:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:32:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:32:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:32:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:32:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:32:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:32:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:32:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:32:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:32:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:32:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:32:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:32:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:32:03.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:32:03.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:32:03.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=721] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:32:03.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=721] Local address: 172.21.15.161, port 58060 2022-07-03T17:32:03.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=721] Peer address: 172.21.15.101, port 22 2022-07-03T17:32:03.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=721] Connection lost 2022-07-03T17:32:03.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=721] Aborting connection 2022-07-03T17:32:03.655 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:03 smithi161 ceph-mon[91579]: pgmap v12387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:32:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:32:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:32:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:32:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:32:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:32:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:32:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:32:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:32:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:32:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:32:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:32:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:32:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:32:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:32:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:32:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:32:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:32:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:32:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:32:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:32:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:32:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:32:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:32:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:32:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:32:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:32:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:32:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:32:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:32:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:32:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:32:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:32:03.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:32:03.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:32:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:32:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:32:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:32:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:32:03.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:32:03.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:32:03.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:32:03.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:32:03.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:32:03.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:32:03.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:32:03.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=721] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:32:03.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=721] Local address: 172.21.15.161, port 58060 2022-07-03T17:32:03.696 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=721] Peer address: 172.21.15.101, port 22 2022-07-03T17:32:03.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=721] Connection lost 2022-07-03T17:32:03.697 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=721] Aborting connection 2022-07-03T17:32:03.698 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:03 smithi188 ceph-mon[92425]: pgmap v12387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:05 smithi161 ceph-mon[91579]: pgmap v12388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:05 smithi188 ceph-mon[92425]: pgmap v12388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:07 smithi161 ceph-mon[91579]: pgmap v12389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:07 smithi188 ceph-mon[92425]: pgmap v12389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:09 smithi161 ceph-mon[91579]: pgmap v12390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:09 smithi188 ceph-mon[92425]: pgmap v12390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:11 smithi161 ceph-mon[91579]: pgmap v12391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:11 smithi188 ceph-mon[92425]: pgmap v12391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:13 smithi161 ceph-mon[91579]: pgmap v12392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:13 smithi188 ceph-mon[92425]: pgmap v12392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:15.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:15 smithi161 ceph-mon[91579]: pgmap v12393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:15 smithi188 ceph-mon[92425]: pgmap v12393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:17 smithi161 ceph-mon[91579]: pgmap v12394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:17 smithi188 ceph-mon[92425]: pgmap v12394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:19 smithi161 ceph-mon[91579]: pgmap v12395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:19 smithi188 ceph-mon[92425]: pgmap v12395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:21 smithi161 ceph-mon[91579]: pgmap v12396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:21 smithi188 ceph-mon[92425]: pgmap v12396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:23 smithi161 ceph-mon[91579]: pgmap v12397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:23 smithi188 ceph-mon[92425]: pgmap v12397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:25 smithi161 ceph-mon[91579]: pgmap v12398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:25 smithi188 ceph-mon[92425]: pgmap v12398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:27 smithi161 ceph-mon[91579]: pgmap v12399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:27 smithi188 ceph-mon[92425]: pgmap v12399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:29 smithi161 ceph-mon[91579]: pgmap v12400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:29 smithi188 ceph-mon[92425]: pgmap v12400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:31 smithi161 ceph-mon[91579]: pgmap v12401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:31 smithi188 ceph-mon[92425]: pgmap v12401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:33 smithi161 ceph-mon[91579]: pgmap v12402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:33 smithi188 ceph-mon[92425]: pgmap v12402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:35 smithi161 ceph-mon[91579]: pgmap v12403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:35 smithi188 ceph-mon[92425]: pgmap v12403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:37 smithi161 ceph-mon[91579]: pgmap v12404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:37 smithi188 ceph-mon[92425]: pgmap v12404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:39 smithi161 ceph-mon[91579]: pgmap v12405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:39 smithi188 ceph-mon[92425]: pgmap v12405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:41 smithi161 ceph-mon[91579]: pgmap v12406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:41 smithi188 ceph-mon[92425]: pgmap v12406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:43 smithi161 ceph-mon[91579]: pgmap v12407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:43 smithi188 ceph-mon[92425]: pgmap v12407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:45 smithi161 ceph-mon[91579]: pgmap v12408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:45 smithi188 ceph-mon[92425]: pgmap v12408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:47 smithi161 ceph-mon[91579]: pgmap v12409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:47 smithi188 ceph-mon[92425]: pgmap v12409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:49 smithi161 ceph-mon[91579]: pgmap v12410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:49 smithi188 ceph-mon[92425]: pgmap v12410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:51 smithi161 ceph-mon[91579]: pgmap v12411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:51 smithi188 ceph-mon[92425]: pgmap v12411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:53 smithi161 ceph-mon[91579]: pgmap v12412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:53 smithi188 ceph-mon[92425]: pgmap v12412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:55 smithi161 ceph-mon[91579]: pgmap v12413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:55 smithi188 ceph-mon[92425]: pgmap v12413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:57 smithi161 ceph-mon[91579]: pgmap v12414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:57 smithi188 ceph-mon[92425]: pgmap v12414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:32:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:59 smithi161 ceph-mon[91579]: pgmap v12415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:32:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:32:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:32:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:32:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:59 smithi188 ceph-mon[92425]: pgmap v12415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:32:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:32:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:32:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:32:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:33:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:01 smithi161 ceph-mon[91579]: pgmap v12416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:01 smithi188 ceph-mon[92425]: pgmap v12416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:33:02.541+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:33:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:33:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:33:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:33:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:33:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:33:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:33:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:33:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:33:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:33:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:33:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:33:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:33:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:33:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:33:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:33:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:33:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:33:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:33:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:33:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:33:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:33:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:33:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:33:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:33:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:33:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:33:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:33:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:33:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:33:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:33:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:33:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:33:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:33:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:33:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:33:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:33:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:33:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:33:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:33:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:33:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:33:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:33:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:33:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:33:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=723] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:33:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=723] Local address: 172.21.15.161, port 58064 2022-07-03T17:33:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=723] Peer address: 172.21.15.101, port 22 2022-07-03T17:33:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=723] Connection lost 2022-07-03T17:33:02.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=723] Aborting connection 2022-07-03T17:33:02.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:33:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: pgmap v12417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:33:03.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:33:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:33:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:33:03.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:33:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:33:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:33:03.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:33:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:33:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:33:03.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:33:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:33:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:33:03.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:33:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:33:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:33:03.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:33:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:33:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:33:03.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:33:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:33:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:33:03.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:33:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:33:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:33:03.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:33:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:33:03.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:33:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:33:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:33:03.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:33:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:33:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:33:03.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:33:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:33:03.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:33:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:33:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:33:03.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:33:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:33:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:33:03.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:33:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:33:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:33:03.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:33:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:33:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:33:03.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:33:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:33:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=723] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:33:03.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=723] Local address: 172.21.15.161, port 58064 2022-07-03T17:33:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=723] Peer address: 172.21.15.101, port 22 2022-07-03T17:33:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=723] Connection lost 2022-07-03T17:33:03.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=723] Aborting connection 2022-07-03T17:33:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: pgmap v12417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:33:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:33:03.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:33:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:33:03.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:33:03.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:33:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:33:03.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:33:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:33:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:33:03.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:33:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:33:03.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:33:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:33:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:33:03.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:33:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:33:03.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:33:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:33:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:33:03.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:33:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:33:03.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:33:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:33:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:33:03.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:33:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:33:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:33:03.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:33:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:33:03.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:33:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:33:03.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:33:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:33:03.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:33:03.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:33:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:33:03.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:33:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:33:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:33:03.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:33:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:33:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:33:03.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:33:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:33:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:33:03.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:33:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:33:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:33:03.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=723] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:33:03.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=723] Local address: 172.21.15.161, port 58064 2022-07-03T17:33:03.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=723] Peer address: 172.21.15.101, port 22 2022-07-03T17:33:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=723] Connection lost 2022-07-03T17:33:03.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=723] Aborting connection 2022-07-03T17:33:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:05 smithi161 ceph-mon[91579]: pgmap v12418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:05 smithi188 ceph-mon[92425]: pgmap v12418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:07 smithi161 ceph-mon[91579]: pgmap v12419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:07 smithi188 ceph-mon[92425]: pgmap v12419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:09 smithi161 ceph-mon[91579]: pgmap v12420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:09 smithi188 ceph-mon[92425]: pgmap v12420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:11 smithi161 ceph-mon[91579]: pgmap v12421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:11 smithi188 ceph-mon[92425]: pgmap v12421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:13 smithi161 ceph-mon[91579]: pgmap v12422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:13 smithi188 ceph-mon[92425]: pgmap v12422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:15 smithi161 ceph-mon[91579]: pgmap v12423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:15 smithi188 ceph-mon[92425]: pgmap v12423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:17 smithi161 ceph-mon[91579]: pgmap v12424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:17 smithi188 ceph-mon[92425]: pgmap v12424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:19 smithi161 ceph-mon[91579]: pgmap v12425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:19 smithi188 ceph-mon[92425]: pgmap v12425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:21 smithi161 ceph-mon[91579]: pgmap v12426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:21 smithi188 ceph-mon[92425]: pgmap v12426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:23 smithi161 ceph-mon[91579]: pgmap v12427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:23 smithi188 ceph-mon[92425]: pgmap v12427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:25 smithi161 ceph-mon[91579]: pgmap v12428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:25 smithi188 ceph-mon[92425]: pgmap v12428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:27 smithi161 ceph-mon[91579]: pgmap v12429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:27 smithi188 ceph-mon[92425]: pgmap v12429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:29 smithi161 ceph-mon[91579]: pgmap v12430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:29 smithi188 ceph-mon[92425]: pgmap v12430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:31 smithi161 ceph-mon[91579]: pgmap v12431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:31 smithi188 ceph-mon[92425]: pgmap v12431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:33 smithi161 ceph-mon[91579]: pgmap v12432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:33 smithi188 ceph-mon[92425]: pgmap v12432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:35 smithi161 ceph-mon[91579]: pgmap v12433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:35 smithi188 ceph-mon[92425]: pgmap v12433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:37 smithi161 ceph-mon[91579]: pgmap v12434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:37 smithi188 ceph-mon[92425]: pgmap v12434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:39 smithi161 ceph-mon[91579]: pgmap v12435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:39 smithi188 ceph-mon[92425]: pgmap v12435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:41 smithi161 ceph-mon[91579]: pgmap v12436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:41 smithi188 ceph-mon[92425]: pgmap v12436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:43 smithi161 ceph-mon[91579]: pgmap v12437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:43 smithi188 ceph-mon[92425]: pgmap v12437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:45 smithi161 ceph-mon[91579]: pgmap v12438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:45 smithi188 ceph-mon[92425]: pgmap v12438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:47 smithi161 ceph-mon[91579]: pgmap v12439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:47 smithi188 ceph-mon[92425]: pgmap v12439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:49 smithi161 ceph-mon[91579]: pgmap v12440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:49 smithi188 ceph-mon[92425]: pgmap v12440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:51.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:51 smithi161 ceph-mon[91579]: pgmap v12441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:51 smithi188 ceph-mon[92425]: pgmap v12441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:53 smithi161 ceph-mon[91579]: pgmap v12442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:53 smithi188 ceph-mon[92425]: pgmap v12442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:55 smithi161 ceph-mon[91579]: pgmap v12443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:55 smithi188 ceph-mon[92425]: pgmap v12443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:57 smithi161 ceph-mon[91579]: pgmap v12444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:57 smithi188 ceph-mon[92425]: pgmap v12444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:33:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:59 smithi161 ceph-mon[91579]: pgmap v12445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:33:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:33:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:33:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:33:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:59 smithi188 ceph-mon[92425]: pgmap v12445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:33:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:33:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:33:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:33:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:34:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:01 smithi161 ceph-mon[91579]: pgmap v12446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:01 smithi188 ceph-mon[92425]: pgmap v12446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-mon[91579]: pgmap v12447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:34:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:34:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:34:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:34:03.043+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:34:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:34:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:34:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:34:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:34:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:34:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:34:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:34:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:34:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:34:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:34:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:34:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:34:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:34:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:34:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:34:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:34:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:34:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:34:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:34:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:34:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:34:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:34:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:34:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:34:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:34:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:34:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:34:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:34:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:34:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:34:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:34:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:34:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:34:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:34:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:34:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:34:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:34:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:34:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:34:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:34:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:34:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:34:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:34:03.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:34:03.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=725] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:34:03.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=725] Local address: 172.21.15.161, port 58068 2022-07-03T17:34:03.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=725] Peer address: 172.21.15.101, port 22 2022-07-03T17:34:03.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=725] Connection lost 2022-07-03T17:34:03.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=725] Aborting connection 2022-07-03T17:34:03.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:34:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:03 smithi188 ceph-mon[92425]: pgmap v12447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:34:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:34:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:34:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:34:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:34:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:34:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:34:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:34:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:34:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:34:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:34:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:34:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:34:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:34:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:34:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:34:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:34:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:34:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:34:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:34:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:34:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:34:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:34:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:34:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:34:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:34:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:34:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:34:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:34:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:34:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:34:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:34:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:34:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:34:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:34:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:34:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:34:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:34:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:34:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:34:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:34:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:34:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:34:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:34:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:34:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:34:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:34:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:34:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:34:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=725] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:34:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=725] Local address: 172.21.15.161, port 58068 2022-07-03T17:34:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=725] Peer address: 172.21.15.101, port 22 2022-07-03T17:34:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=725] Connection lost 2022-07-03T17:34:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=725] Aborting connection 2022-07-03T17:34:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:34:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:34:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:34:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:34:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:34:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:34:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:34:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:34:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:34:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:34:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:34:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:34:04.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:34:04.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:34:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:34:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:34:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:34:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:34:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:34:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:34:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:34:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:34:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:34:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:34:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:34:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:34:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:34:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:34:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:34:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:34:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:34:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:34:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:34:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:34:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:34:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:34:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:34:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:34:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:34:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:34:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:34:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:34:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:34:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:34:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:34:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:34:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=725] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:34:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=725] Local address: 172.21.15.161, port 58068 2022-07-03T17:34:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=725] Peer address: 172.21.15.101, port 22 2022-07-03T17:34:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=725] Connection lost 2022-07-03T17:34:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=725] Aborting connection 2022-07-03T17:34:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:05 smithi161 ceph-mon[91579]: pgmap v12448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:05 smithi188 ceph-mon[92425]: pgmap v12448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:07 smithi161 ceph-mon[91579]: pgmap v12449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:07 smithi188 ceph-mon[92425]: pgmap v12449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:09 smithi161 ceph-mon[91579]: pgmap v12450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:09 smithi188 ceph-mon[92425]: pgmap v12450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:11 smithi161 ceph-mon[91579]: pgmap v12451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:11 smithi188 ceph-mon[92425]: pgmap v12451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:13 smithi161 ceph-mon[91579]: pgmap v12452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:13 smithi188 ceph-mon[92425]: pgmap v12452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:15 smithi161 ceph-mon[91579]: pgmap v12453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:15 smithi188 ceph-mon[92425]: pgmap v12453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:17 smithi161 ceph-mon[91579]: pgmap v12454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:17 smithi188 ceph-mon[92425]: pgmap v12454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:19 smithi161 ceph-mon[91579]: pgmap v12455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:19 smithi188 ceph-mon[92425]: pgmap v12455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:21 smithi161 ceph-mon[91579]: pgmap v12456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:21 smithi188 ceph-mon[92425]: pgmap v12456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:23 smithi161 ceph-mon[91579]: pgmap v12457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:23 smithi188 ceph-mon[92425]: pgmap v12457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:25 smithi161 ceph-mon[91579]: pgmap v12458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:25 smithi188 ceph-mon[92425]: pgmap v12458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:27 smithi161 ceph-mon[91579]: pgmap v12459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:27 smithi188 ceph-mon[92425]: pgmap v12459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:29 smithi161 ceph-mon[91579]: pgmap v12460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:29 smithi188 ceph-mon[92425]: pgmap v12460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:31 smithi161 ceph-mon[91579]: pgmap v12461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:31 smithi188 ceph-mon[92425]: pgmap v12461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:33 smithi161 ceph-mon[91579]: pgmap v12462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:33 smithi188 ceph-mon[92425]: pgmap v12462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:35 smithi161 ceph-mon[91579]: pgmap v12463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:35 smithi188 ceph-mon[92425]: pgmap v12463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:37 smithi161 ceph-mon[91579]: pgmap v12464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:37 smithi188 ceph-mon[92425]: pgmap v12464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:39.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:39 smithi161 ceph-mon[91579]: pgmap v12465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:39 smithi188 ceph-mon[92425]: pgmap v12465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:41 smithi161 ceph-mon[91579]: pgmap v12466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:41 smithi188 ceph-mon[92425]: pgmap v12466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:43 smithi161 ceph-mon[91579]: pgmap v12467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:43 smithi188 ceph-mon[92425]: pgmap v12467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:45 smithi161 ceph-mon[91579]: pgmap v12468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:45 smithi188 ceph-mon[92425]: pgmap v12468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:47 smithi161 ceph-mon[91579]: pgmap v12469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:47 smithi188 ceph-mon[92425]: pgmap v12469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:49 smithi161 ceph-mon[91579]: pgmap v12470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:49 smithi188 ceph-mon[92425]: pgmap v12470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:51 smithi161 ceph-mon[91579]: pgmap v12471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:51 smithi188 ceph-mon[92425]: pgmap v12471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:53 smithi161 ceph-mon[91579]: pgmap v12472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:53 smithi188 ceph-mon[92425]: pgmap v12472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:55 smithi161 ceph-mon[91579]: pgmap v12473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:55 smithi188 ceph-mon[92425]: pgmap v12473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:57 smithi161 ceph-mon[91579]: pgmap v12474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:57 smithi188 ceph-mon[92425]: pgmap v12474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:34:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:59 smithi161 ceph-mon[91579]: pgmap v12475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:34:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:34:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:34:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:34:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:59 smithi188 ceph-mon[92425]: pgmap v12475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:34:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:34:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:34:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:34:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:35:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:01 smithi161 ceph-mon[91579]: pgmap v12476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:01 smithi188 ceph-mon[92425]: pgmap v12476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:03.552 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-mon[91579]: pgmap v12477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:03.561 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:03 smithi188 ceph-mon[92425]: pgmap v12477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:35:03.549+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:35:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:35:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:35:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:35:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:35:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:35:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:35:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:35:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:35:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:35:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:35:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:35:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:35:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:35:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:35:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:35:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:35:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:35:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:35:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:35:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:35:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:35:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:35:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:35:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:35:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:35:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:35:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:35:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:35:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:35:03.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:35:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:35:03.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:35:03.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:35:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:35:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:35:03.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:35:03.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:35:03.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:35:03.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:35:03.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:35:03.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:35:03.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:35:03.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:35:03.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:35:03.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=727] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:35:03.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=727] Local address: 172.21.15.161, port 58072 2022-07-03T17:35:03.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=727] Peer address: 172.21.15.101, port 22 2022-07-03T17:35:03.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=727] Connection lost 2022-07-03T17:35:03.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=727] Aborting connection 2022-07-03T17:35:03.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:35:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:35:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:35:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:35:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:35:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:35:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:35:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:35:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:35:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:35:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:35:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:35:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:35:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:35:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:35:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:35:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:35:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:35:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:35:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:35:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:35:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:35:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:35:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:35:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:35:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:35:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:35:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:35:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:35:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:35:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:35:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:35:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:35:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:35:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:35:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:35:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:35:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:35:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:35:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:35:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:35:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:35:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:35:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:35:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:35:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:35:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:35:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:35:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:35:04.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:35:04.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:35:04.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=727] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:35:04.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=727] Local address: 172.21.15.161, port 58072 2022-07-03T17:35:04.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=727] Peer address: 172.21.15.101, port 22 2022-07-03T17:35:04.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=727] Connection lost 2022-07-03T17:35:04.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=727] Aborting connection 2022-07-03T17:35:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:35:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:35:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:35:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:35:04.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:35:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:35:04.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:35:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:35:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:35:04.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:35:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:35:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:35:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:35:04.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:35:04.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:35:04.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:35:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:35:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:35:04.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:35:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:35:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:35:04.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:35:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:35:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:35:04.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:35:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:35:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:35:04.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:35:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:35:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:35:04.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:35:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:35:04.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:35:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:35:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:35:04.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:35:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:35:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:35:04.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:35:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:35:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:35:04.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:35:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:35:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:35:04.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:35:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:35:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:35:04.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:35:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:35:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=727] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:35:04.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=727] Local address: 172.21.15.161, port 58072 2022-07-03T17:35:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=727] Peer address: 172.21.15.101, port 22 2022-07-03T17:35:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=727] Connection lost 2022-07-03T17:35:04.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=727] Aborting connection 2022-07-03T17:35:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:05 smithi161 ceph-mon[91579]: pgmap v12478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:05 smithi188 ceph-mon[92425]: pgmap v12478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:07 smithi161 ceph-mon[91579]: pgmap v12479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:07 smithi188 ceph-mon[92425]: pgmap v12479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:09 smithi161 ceph-mon[91579]: pgmap v12480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:09 smithi188 ceph-mon[92425]: pgmap v12480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:11 smithi161 ceph-mon[91579]: pgmap v12481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:11 smithi188 ceph-mon[92425]: pgmap v12481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:13 smithi161 ceph-mon[91579]: pgmap v12482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:13 smithi188 ceph-mon[92425]: pgmap v12482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:15 smithi161 ceph-mon[91579]: pgmap v12483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:15 smithi188 ceph-mon[92425]: pgmap v12483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:17 smithi161 ceph-mon[91579]: pgmap v12484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:17 smithi188 ceph-mon[92425]: pgmap v12484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:19 smithi161 ceph-mon[91579]: pgmap v12485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:19 smithi188 ceph-mon[92425]: pgmap v12485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:21 smithi188 ceph-mon[92425]: pgmap v12486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:21 smithi161 ceph-mon[91579]: pgmap v12486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:23 smithi161 ceph-mon[91579]: pgmap v12487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:23 smithi188 ceph-mon[92425]: pgmap v12487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:25 smithi161 ceph-mon[91579]: pgmap v12488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:25 smithi188 ceph-mon[92425]: pgmap v12488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:27 smithi161 ceph-mon[91579]: pgmap v12489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:27 smithi188 ceph-mon[92425]: pgmap v12489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:29 smithi188 ceph-mon[92425]: pgmap v12490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:29 smithi161 ceph-mon[91579]: pgmap v12490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:31 smithi188 ceph-mon[92425]: pgmap v12491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:31 smithi161 ceph-mon[91579]: pgmap v12491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:33 smithi161 ceph-mon[91579]: pgmap v12492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:33 smithi188 ceph-mon[92425]: pgmap v12492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:35 smithi161 ceph-mon[91579]: pgmap v12493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:35 smithi188 ceph-mon[92425]: pgmap v12493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:37 smithi188 ceph-mon[92425]: pgmap v12494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:37 smithi161 ceph-mon[91579]: pgmap v12494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:39 smithi188 ceph-mon[92425]: pgmap v12495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:39 smithi161 ceph-mon[91579]: pgmap v12495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:41 smithi188 ceph-mon[92425]: pgmap v12496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:41 smithi161 ceph-mon[91579]: pgmap v12496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:43 smithi161 ceph-mon[91579]: pgmap v12497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:43 smithi188 ceph-mon[92425]: pgmap v12497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:45 smithi188 ceph-mon[92425]: pgmap v12498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:45 smithi161 ceph-mon[91579]: pgmap v12498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:47 smithi188 ceph-mon[92425]: pgmap v12499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:47 smithi161 ceph-mon[91579]: pgmap v12499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:49 smithi188 ceph-mon[92425]: pgmap v12500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:49 smithi161 ceph-mon[91579]: pgmap v12500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:51 smithi188 ceph-mon[92425]: pgmap v12501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:51 smithi161 ceph-mon[91579]: pgmap v12501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:53 smithi161 ceph-mon[91579]: pgmap v12502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:53 smithi188 ceph-mon[92425]: pgmap v12502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:55 smithi188 ceph-mon[92425]: pgmap v12503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:55 smithi161 ceph-mon[91579]: pgmap v12503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:57 smithi188 ceph-mon[92425]: pgmap v12504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:57 smithi161 ceph-mon[91579]: pgmap v12504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:35:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:59 smithi188 ceph-mon[92425]: pgmap v12505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:35:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:35:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:35:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:35:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:59 smithi161 ceph-mon[91579]: pgmap v12505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:35:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:35:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:35:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:35:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:36:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:01 smithi188 ceph-mon[92425]: pgmap v12506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:01 smithi161 ceph-mon[91579]: pgmap v12506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:03 smithi161 ceph-mon[91579]: pgmap v12507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:03 smithi188 ceph-mon[92425]: pgmap v12507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:36:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:36:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:36:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:36:04.051+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:36:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:36:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:36:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:36:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:36:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:36:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:36:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:36:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:36:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:36:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:36:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:36:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:36:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:36:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:36:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:36:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:36:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:36:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:36:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:36:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:36:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:36:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:36:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:36:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:36:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:36:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:36:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:36:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:36:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:36:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:36:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:36:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:36:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:36:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:36:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:36:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:36:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:36:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:36:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:36:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:36:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:36:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:36:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:36:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:36:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=729] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:36:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=729] Local address: 172.21.15.161, port 58076 2022-07-03T17:36:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=729] Peer address: 172.21.15.101, port 22 2022-07-03T17:36:04.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=729] Connection lost 2022-07-03T17:36:04.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=729] Aborting connection 2022-07-03T17:36:04.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:36:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:36:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:36:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:36:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:36:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:36:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:36:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:36:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:36:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:36:05.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:36:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:36:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:36:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:36:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:36:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:36:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:36:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:36:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:36:05.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:36:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:36:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:36:05.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:36:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:36:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:36:05.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:36:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:36:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:36:05.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:36:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:36:05.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:36:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:36:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:36:05.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:36:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:36:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:36:05.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:36:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:36:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:36:05.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:36:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:36:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:36:05.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:36:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:36:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:36:05.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:36:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:36:05.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:36:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:36:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:36:05.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=729] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:36:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=729] Local address: 172.21.15.161, port 58076 2022-07-03T17:36:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=729] Peer address: 172.21.15.101, port 22 2022-07-03T17:36:05.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=729] Connection lost 2022-07-03T17:36:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=729] Aborting connection 2022-07-03T17:36:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: pgmap v12508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:05.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:36:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:36:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:36:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:36:05.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:36:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:36:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:36:05.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:36:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:36:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:36:05.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:36:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:36:05.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:36:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:36:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:36:05.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:36:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:36:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:36:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:36:05.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:36:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:36:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:36:05.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:36:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:36:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:36:05.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:36:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:36:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:36:05.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:36:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:36:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:36:05.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:36:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:36:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:36:05.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:36:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:36:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:36:05.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:36:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:36:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:36:05.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:36:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:36:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:36:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:36:05.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:36:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:36:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=729] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:36:05.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=729] Local address: 172.21.15.161, port 58076 2022-07-03T17:36:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=729] Peer address: 172.21.15.101, port 22 2022-07-03T17:36:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=729] Connection lost 2022-07-03T17:36:05.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=729] Aborting connection 2022-07-03T17:36:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: pgmap v12508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:05.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:36:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:07 smithi161 ceph-mon[91579]: pgmap v12509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:07 smithi188 ceph-mon[92425]: pgmap v12509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:09 smithi161 ceph-mon[91579]: pgmap v12510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:09 smithi188 ceph-mon[92425]: pgmap v12510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:11 smithi161 ceph-mon[91579]: pgmap v12511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:11 smithi188 ceph-mon[92425]: pgmap v12511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:13 smithi161 ceph-mon[91579]: pgmap v12512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:13 smithi188 ceph-mon[92425]: pgmap v12512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:15 smithi188 ceph-mon[92425]: pgmap v12513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:15 smithi161 ceph-mon[91579]: pgmap v12513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:17 smithi188 ceph-mon[92425]: pgmap v12514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:17 smithi161 ceph-mon[91579]: pgmap v12514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:19 smithi188 ceph-mon[92425]: pgmap v12515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:19 smithi161 ceph-mon[91579]: pgmap v12515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:21 smithi188 ceph-mon[92425]: pgmap v12516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:21 smithi161 ceph-mon[91579]: pgmap v12516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:23 smithi161 ceph-mon[91579]: pgmap v12517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:23 smithi188 ceph-mon[92425]: pgmap v12517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:25 smithi188 ceph-mon[92425]: pgmap v12518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:25 smithi161 ceph-mon[91579]: pgmap v12518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:27 smithi188 ceph-mon[92425]: pgmap v12519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:27 smithi161 ceph-mon[91579]: pgmap v12519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:29 smithi188 ceph-mon[92425]: pgmap v12520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:29 smithi161 ceph-mon[91579]: pgmap v12520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:31 smithi188 ceph-mon[92425]: pgmap v12521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:31 smithi161 ceph-mon[91579]: pgmap v12521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:33 smithi161 ceph-mon[91579]: pgmap v12522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:33 smithi188 ceph-mon[92425]: pgmap v12522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:35 smithi188 ceph-mon[92425]: pgmap v12523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:35 smithi161 ceph-mon[91579]: pgmap v12523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:37 smithi188 ceph-mon[92425]: pgmap v12524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:37 smithi161 ceph-mon[91579]: pgmap v12524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:39 smithi188 ceph-mon[92425]: pgmap v12525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:39 smithi161 ceph-mon[91579]: pgmap v12525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:41 smithi188 ceph-mon[92425]: pgmap v12526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:41 smithi161 ceph-mon[91579]: pgmap v12526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:43 smithi161 ceph-mon[91579]: pgmap v12527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:43 smithi188 ceph-mon[92425]: pgmap v12527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:45 smithi188 ceph-mon[92425]: pgmap v12528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:45 smithi161 ceph-mon[91579]: pgmap v12528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:47.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:47 smithi188 ceph-mon[92425]: pgmap v12529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:47 smithi161 ceph-mon[91579]: pgmap v12529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:49 smithi188 ceph-mon[92425]: pgmap v12530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:49 smithi161 ceph-mon[91579]: pgmap v12530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:51 smithi188 ceph-mon[92425]: pgmap v12531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:51 smithi161 ceph-mon[91579]: pgmap v12531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:53 smithi161 ceph-mon[91579]: pgmap v12532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:53 smithi188 ceph-mon[92425]: pgmap v12532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:55 smithi188 ceph-mon[92425]: pgmap v12533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:55 smithi161 ceph-mon[91579]: pgmap v12533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:57 smithi188 ceph-mon[92425]: pgmap v12534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:57 smithi161 ceph-mon[91579]: pgmap v12534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:36:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:59 smithi188 ceph-mon[92425]: pgmap v12535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:36:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:36:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:36:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:36:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:59 smithi161 ceph-mon[91579]: pgmap v12535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:36:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:36:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:36:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:36:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:37:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:01 smithi188 ceph-mon[92425]: pgmap v12536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:01 smithi161 ceph-mon[91579]: pgmap v12536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:03 smithi161 ceph-mon[91579]: pgmap v12537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:03 smithi188 ceph-mon[92425]: pgmap v12537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:37:04.976+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:37:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:37:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:37:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:37:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:37:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:37:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:37:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:37:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:37:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:37:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:37:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:37:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:37:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:37:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:37:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:37:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:37:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:37:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:37:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:37:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:37:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:37:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:37:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:37:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:37:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:37:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:37:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:37:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:37:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:37:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:37:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:37:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:37:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:37:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:37:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:37:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:37:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:37:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:37:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:37:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:37:05.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:37:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:37:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:37:05.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:37:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=731] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:37:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=731] Local address: 172.21.15.161, port 58080 2022-07-03T17:37:05.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=731] Peer address: 172.21.15.101, port 22 2022-07-03T17:37:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=731] Connection lost 2022-07-03T17:37:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=731] Aborting connection 2022-07-03T17:37:05.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:37:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:37:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:05 smithi188 ceph-mon[92425]: pgmap v12538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:37:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:37:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:37:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:05 smithi161 ceph-mon[91579]: pgmap v12538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:37:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:37:05.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:37:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:37:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:37:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:37:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:37:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:37:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:37:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:37:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:37:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:37:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:37:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:37:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:37:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:37:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:37:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:37:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:37:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:37:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:37:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:37:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:37:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:37:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:37:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:37:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:37:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:37:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:37:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:37:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:37:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:37:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:37:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:37:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:37:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:37:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:37:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:37:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:37:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:37:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:37:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:37:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:37:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:37:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:37:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:37:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:37:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:37:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:37:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=731] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:37:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=731] Local address: 172.21.15.161, port 58080 2022-07-03T17:37:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=731] Peer address: 172.21.15.101, port 22 2022-07-03T17:37:06.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=731] Connection lost 2022-07-03T17:37:06.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=731] Aborting connection 2022-07-03T17:37:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:37:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:37:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:37:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:37:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:37:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:37:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:37:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:37:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:37:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:37:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:37:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:37:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:37:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:37:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:37:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:37:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:37:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:37:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:37:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:37:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:37:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:37:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:37:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:37:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:37:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:37:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:37:06.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:37:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:37:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:37:06.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:37:06.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:37:06.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:37:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:37:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:37:06.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:37:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:37:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:37:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:37:06.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:37:06.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:37:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:37:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:37:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:37:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:37:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:37:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=731] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:37:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=731] Local address: 172.21.15.161, port 58080 2022-07-03T17:37:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=731] Peer address: 172.21.15.101, port 22 2022-07-03T17:37:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=731] Connection lost 2022-07-03T17:37:06.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=731] Aborting connection 2022-07-03T17:37:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:07 smithi161 ceph-mon[91579]: pgmap v12539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:07 smithi188 ceph-mon[92425]: pgmap v12539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:09 smithi161 ceph-mon[91579]: pgmap v12540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:09 smithi188 ceph-mon[92425]: pgmap v12540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:11 smithi161 ceph-mon[91579]: pgmap v12541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:11 smithi188 ceph-mon[92425]: pgmap v12541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:13 smithi161 ceph-mon[91579]: pgmap v12542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:13 smithi188 ceph-mon[92425]: pgmap v12542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:15 smithi188 ceph-mon[92425]: pgmap v12543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:15 smithi161 ceph-mon[91579]: pgmap v12543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:17 smithi188 ceph-mon[92425]: pgmap v12544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:17 smithi161 ceph-mon[91579]: pgmap v12544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:19 smithi188 ceph-mon[92425]: pgmap v12545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:19 smithi161 ceph-mon[91579]: pgmap v12545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:21 smithi161 ceph-mon[91579]: pgmap v12546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:21 smithi188 ceph-mon[92425]: pgmap v12546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:23 smithi161 ceph-mon[91579]: pgmap v12547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:23 smithi188 ceph-mon[92425]: pgmap v12547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:25 smithi188 ceph-mon[92425]: pgmap v12548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:25 smithi161 ceph-mon[91579]: pgmap v12548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:27 smithi188 ceph-mon[92425]: pgmap v12549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:27 smithi161 ceph-mon[91579]: pgmap v12549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:29 smithi161 ceph-mon[91579]: pgmap v12550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:29 smithi188 ceph-mon[92425]: pgmap v12550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:31 smithi161 ceph-mon[91579]: pgmap v12551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:31 smithi188 ceph-mon[92425]: pgmap v12551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:33 smithi161 ceph-mon[91579]: pgmap v12552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:33 smithi188 ceph-mon[92425]: pgmap v12552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:35 smithi188 ceph-mon[92425]: pgmap v12553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:35 smithi161 ceph-mon[91579]: pgmap v12553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:37 smithi161 ceph-mon[91579]: pgmap v12554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:37 smithi188 ceph-mon[92425]: pgmap v12554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:39 smithi161 ceph-mon[91579]: pgmap v12555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:39 smithi188 ceph-mon[92425]: pgmap v12555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:41 smithi161 ceph-mon[91579]: pgmap v12556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:41 smithi188 ceph-mon[92425]: pgmap v12556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:43 smithi161 ceph-mon[91579]: pgmap v12557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:43 smithi188 ceph-mon[92425]: pgmap v12557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:45 smithi161 ceph-mon[91579]: pgmap v12558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:45 smithi188 ceph-mon[92425]: pgmap v12558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:47 smithi161 ceph-mon[91579]: pgmap v12559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:47 smithi188 ceph-mon[92425]: pgmap v12559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:49 smithi161 ceph-mon[91579]: pgmap v12560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:49 smithi188 ceph-mon[92425]: pgmap v12560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:51 smithi161 ceph-mon[91579]: pgmap v12561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:51 smithi188 ceph-mon[92425]: pgmap v12561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:53 smithi161 ceph-mon[91579]: pgmap v12562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:53 smithi188 ceph-mon[92425]: pgmap v12562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:55 smithi161 ceph-mon[91579]: pgmap v12563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:55 smithi188 ceph-mon[92425]: pgmap v12563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:57 smithi161 ceph-mon[91579]: pgmap v12564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:57 smithi188 ceph-mon[92425]: pgmap v12564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:37:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:59 smithi161 ceph-mon[91579]: pgmap v12565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:37:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:37:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:37:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:37:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:59 smithi188 ceph-mon[92425]: pgmap v12565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:37:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:37:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:37:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:37:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:38:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:01 smithi161 ceph-mon[91579]: pgmap v12566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:01 smithi188 ceph-mon[92425]: pgmap v12566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:03 smithi161 ceph-mon[91579]: pgmap v12567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:03 smithi188 ceph-mon[92425]: pgmap v12567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-mon[91579]: pgmap v12568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:38:05.475+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:38:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:38:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:38:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:38:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:38:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:38:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:38:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:38:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:38:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:38:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:38:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:38:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:38:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:38:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:38:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:38:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:38:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:38:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:38:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:38:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:38:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:38:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:38:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:38:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:38:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:38:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:38:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:38:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:38:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:38:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:38:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:38:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:38:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:38:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:38:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:38:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:38:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:38:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:38:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:38:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:38:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:38:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:38:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:38:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:38:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=733] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:38:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=733] Local address: 172.21.15.161, port 58084 2022-07-03T17:38:05.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=733] Peer address: 172.21.15.101, port 22 2022-07-03T17:38:05.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=733] Connection lost 2022-07-03T17:38:05.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=733] Aborting connection 2022-07-03T17:38:05.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:38:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:38:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:05 smithi188 ceph-mon[92425]: pgmap v12568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:38:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:38:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:38:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:38:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:38:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:38:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:38:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:38:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:38:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:38:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:38:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:38:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:38:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:38:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:38:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:38:06.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:38:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:38:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:38:06.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:38:06.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:38:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:38:06.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:38:06.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:38:06.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:38:06.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:38:06.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:38:06.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:38:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:38:06.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:38:06.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:38:06.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:38:06.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:38:06.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:38:06.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:38:06.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:38:06.909 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:38:06.910 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:38:06.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:38:06.911 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:38:06.912 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:38:06.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:38:06.913 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:38:06.914 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:38:06.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:38:06.915 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:38:06.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:38:06.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:38:06.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:38:06.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=733] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:38:06.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=733] Local address: 172.21.15.161, port 58084 2022-07-03T17:38:06.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=733] Peer address: 172.21.15.101, port 22 2022-07-03T17:38:06.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=733] Connection lost 2022-07-03T17:38:06.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=733] Aborting connection 2022-07-03T17:38:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:38:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:38:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:38:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:38:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:38:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:38:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:38:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:38:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:38:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:38:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:38:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:38:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:38:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:38:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:38:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:38:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:38:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:38:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:38:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:38:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:38:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:38:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:38:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:38:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:38:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:38:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:38:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:38:06.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:38:06.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:38:06.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:38:06.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:38:06.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:38:06.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:38:06.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:38:06.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:38:06.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:38:06.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:38:06.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:38:06.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:38:06.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:38:06.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:38:06.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:38:06.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:38:06.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:38:06.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:38:06.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:38:06.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:38:06.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:38:06.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=733] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:38:06.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=733] Local address: 172.21.15.161, port 58084 2022-07-03T17:38:06.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=733] Peer address: 172.21.15.101, port 22 2022-07-03T17:38:06.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=733] Connection lost 2022-07-03T17:38:06.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=733] Aborting connection 2022-07-03T17:38:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:07 smithi161 ceph-mon[91579]: pgmap v12569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:07 smithi188 ceph-mon[92425]: pgmap v12569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:09 smithi161 ceph-mon[91579]: pgmap v12570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:09 smithi188 ceph-mon[92425]: pgmap v12570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:11 smithi161 ceph-mon[91579]: pgmap v12571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:11 smithi188 ceph-mon[92425]: pgmap v12571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:13 smithi161 ceph-mon[91579]: pgmap v12572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:13 smithi188 ceph-mon[92425]: pgmap v12572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:15 smithi161 ceph-mon[91579]: pgmap v12573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:15 smithi188 ceph-mon[92425]: pgmap v12573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:17 smithi161 ceph-mon[91579]: pgmap v12574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:17 smithi188 ceph-mon[92425]: pgmap v12574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:19 smithi161 ceph-mon[91579]: pgmap v12575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:19 smithi188 ceph-mon[92425]: pgmap v12575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:21 smithi161 ceph-mon[91579]: pgmap v12576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:21 smithi188 ceph-mon[92425]: pgmap v12576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:23 smithi161 ceph-mon[91579]: pgmap v12577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:23 smithi188 ceph-mon[92425]: pgmap v12577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:25 smithi161 ceph-mon[91579]: pgmap v12578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:25 smithi188 ceph-mon[92425]: pgmap v12578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:27 smithi161 ceph-mon[91579]: pgmap v12579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:27 smithi188 ceph-mon[92425]: pgmap v12579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:29 smithi161 ceph-mon[91579]: pgmap v12580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:29 smithi188 ceph-mon[92425]: pgmap v12580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:31 smithi161 ceph-mon[91579]: pgmap v12581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:31 smithi188 ceph-mon[92425]: pgmap v12581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:33 smithi161 ceph-mon[91579]: pgmap v12582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:33 smithi188 ceph-mon[92425]: pgmap v12582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:35 smithi161 ceph-mon[91579]: pgmap v12583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:35 smithi188 ceph-mon[92425]: pgmap v12583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:37 smithi161 ceph-mon[91579]: pgmap v12584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:37 smithi188 ceph-mon[92425]: pgmap v12584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:39 smithi161 ceph-mon[91579]: pgmap v12585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:39 smithi188 ceph-mon[92425]: pgmap v12585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:41 smithi161 ceph-mon[91579]: pgmap v12586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:41 smithi188 ceph-mon[92425]: pgmap v12586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:43 smithi161 ceph-mon[91579]: pgmap v12587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:43 smithi188 ceph-mon[92425]: pgmap v12587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:45 smithi161 ceph-mon[91579]: pgmap v12588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:45 smithi188 ceph-mon[92425]: pgmap v12588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:47 smithi161 ceph-mon[91579]: pgmap v12589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:47 smithi188 ceph-mon[92425]: pgmap v12589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:49 smithi161 ceph-mon[91579]: pgmap v12590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:49 smithi188 ceph-mon[92425]: pgmap v12590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:51 smithi161 ceph-mon[91579]: pgmap v12591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:51 smithi188 ceph-mon[92425]: pgmap v12591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:53 smithi161 ceph-mon[91579]: pgmap v12592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:53 smithi188 ceph-mon[92425]: pgmap v12592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:55 smithi161 ceph-mon[91579]: pgmap v12593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:55 smithi188 ceph-mon[92425]: pgmap v12593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:57 smithi161 ceph-mon[91579]: pgmap v12594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:57 smithi188 ceph-mon[92425]: pgmap v12594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:38:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:59 smithi161 ceph-mon[91579]: pgmap v12595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:38:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:38:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:38:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:38:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:59 smithi188 ceph-mon[92425]: pgmap v12595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:38:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:38:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:38:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:38:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:39:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:01 smithi161 ceph-mon[91579]: pgmap v12596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:01 smithi188 ceph-mon[92425]: pgmap v12596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:03 smithi161 ceph-mon[91579]: pgmap v12597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:03 smithi188 ceph-mon[92425]: pgmap v12597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-mon[91579]: pgmap v12598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:05 smithi188 ceph-mon[92425]: pgmap v12598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:39:05.976+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:39:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:39:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:39:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:39:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:39:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:39:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:39:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:39:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:39:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:39:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:39:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:39:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:39:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:39:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:39:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:39:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:39:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:39:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:39:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:39:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:39:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:39:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:39:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:39:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:39:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:39:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:39:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:39:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:39:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:39:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:39:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:39:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:39:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:39:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:39:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:39:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:39:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:39:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:39:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:39:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:39:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:39:06.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:39:06.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:39:06.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:39:06.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=735] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:39:06.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=735] Local address: 172.21.15.161, port 58116 2022-07-03T17:39:06.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=735] Peer address: 172.21.15.101, port 22 2022-07-03T17:39:06.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=735] Connection lost 2022-07-03T17:39:06.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=735] Aborting connection 2022-07-03T17:39:06.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:39:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:39:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:39:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:39:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:39:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:39:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:39:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:39:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:39:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:39:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:39:07.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:39:07.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:39:07.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:39:07.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:39:07.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:39:07.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:39:07.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:39:07.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:39:07.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:39:07.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:39:07.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:39:07.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:39:07.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:39:07.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:39:07.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:39:07.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:39:07.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:39:07.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:39:07.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:39:07.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:39:07.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:39:07.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:39:07.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:39:07.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:39:07.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:39:07.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:39:07.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:39:07.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:39:07.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:39:07.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:39:07.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:39:07.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:39:07.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:39:07.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:39:07.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:39:07.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:39:07.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:39:07.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:39:07.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:39:07.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:39:07.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:39:07.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:39:07.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:39:07.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=735] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:39:07.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=735] Local address: 172.21.15.161, port 58116 2022-07-03T17:39:07.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=735] Peer address: 172.21.15.101, port 22 2022-07-03T17:39:07.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=735] Connection lost 2022-07-03T17:39:07.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=735] Aborting connection 2022-07-03T17:39:07.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:07 smithi161 ceph-mon[91579]: pgmap v12599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:39:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:39:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:39:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:39:07.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:39:07.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:39:07.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:39:07.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:39:07.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:39:07.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:39:07.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:39:07.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:39:07.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:39:07.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:39:07.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:39:07.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:39:07.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:39:07.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:39:07.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:39:07.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:39:07.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:39:07.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:39:07.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:39:07.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:39:07.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:39:07.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:39:07.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:39:07.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:39:07.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:39:07.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:39:07.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:39:07.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:39:07.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:39:07.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:39:07.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:39:07.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:39:07.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:39:07.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:39:07.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:39:07.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:39:07.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:39:07.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:39:07.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:39:07.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:39:07.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:39:07.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:39:07.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=735] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:39:07.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=735] Local address: 172.21.15.161, port 58116 2022-07-03T17:39:07.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=735] Peer address: 172.21.15.101, port 22 2022-07-03T17:39:07.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=735] Connection lost 2022-07-03T17:39:07.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=735] Aborting connection 2022-07-03T17:39:07.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:07 smithi188 ceph-mon[92425]: pgmap v12599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: pgmap v12600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: pgmap v12600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:39:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:39:11.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 59915k 2022-07-03T17:39:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 61353500: error parsing value: Value '61353500' is below minimum 939524096 2022-07-03T17:39:11.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:10 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 175.2M 2022-07-03T17:39:11.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:10 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 183742865: error parsing value: Value '183742865' is below minimum 939524096 2022-07-03T17:39:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 59915k 2022-07-03T17:39:11.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 61353500: error parsing value: Value '61353500' is below minimum 939524096 2022-07-03T17:39:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:10 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 175.2M 2022-07-03T17:39:11.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:10 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 183742865: error parsing value: Value '183742865' is below minimum 939524096 2022-07-03T17:39:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:11 smithi161 ceph-mon[91579]: pgmap v12601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:11 smithi188 ceph-mon[92425]: pgmap v12601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:13 smithi161 ceph-mon[91579]: pgmap v12602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:13 smithi188 ceph-mon[92425]: pgmap v12602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:15 smithi161 ceph-mon[91579]: pgmap v12603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:15 smithi188 ceph-mon[92425]: pgmap v12603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:17 smithi161 ceph-mon[91579]: pgmap v12604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:17 smithi188 ceph-mon[92425]: pgmap v12604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:19 smithi161 ceph-mon[91579]: pgmap v12605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:19 smithi188 ceph-mon[92425]: pgmap v12605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:21 smithi161 ceph-mon[91579]: pgmap v12606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:21 smithi188 ceph-mon[92425]: pgmap v12606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:23 smithi161 ceph-mon[91579]: pgmap v12607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:23 smithi188 ceph-mon[92425]: pgmap v12607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:25 smithi161 ceph-mon[91579]: pgmap v12608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:25.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:25 smithi188 ceph-mon[92425]: pgmap v12608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:27 smithi161 ceph-mon[91579]: pgmap v12609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:27 smithi188 ceph-mon[92425]: pgmap v12609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:29 smithi161 ceph-mon[91579]: pgmap v12610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:29 smithi188 ceph-mon[92425]: pgmap v12610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:31 smithi161 ceph-mon[91579]: pgmap v12611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:31 smithi188 ceph-mon[92425]: pgmap v12611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:33 smithi161 ceph-mon[91579]: pgmap v12612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:33 smithi188 ceph-mon[92425]: pgmap v12612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:35 smithi161 ceph-mon[91579]: pgmap v12613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:35 smithi188 ceph-mon[92425]: pgmap v12613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:37 smithi161 ceph-mon[91579]: pgmap v12614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:37 smithi188 ceph-mon[92425]: pgmap v12614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:39 smithi161 ceph-mon[91579]: pgmap v12615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:39 smithi188 ceph-mon[92425]: pgmap v12615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:41 smithi161 ceph-mon[91579]: pgmap v12616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:41 smithi188 ceph-mon[92425]: pgmap v12616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:43 smithi161 ceph-mon[91579]: pgmap v12617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:43 smithi188 ceph-mon[92425]: pgmap v12617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:45 smithi161 ceph-mon[91579]: pgmap v12618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:45 smithi188 ceph-mon[92425]: pgmap v12618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:47 smithi161 ceph-mon[91579]: pgmap v12619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:47 smithi188 ceph-mon[92425]: pgmap v12619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:49 smithi161 ceph-mon[91579]: pgmap v12620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:49 smithi188 ceph-mon[92425]: pgmap v12620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:51 smithi161 ceph-mon[91579]: pgmap v12621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:51 smithi188 ceph-mon[92425]: pgmap v12621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:53 smithi161 ceph-mon[91579]: pgmap v12622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:53 smithi188 ceph-mon[92425]: pgmap v12622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:55 smithi161 ceph-mon[91579]: pgmap v12623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:55 smithi188 ceph-mon[92425]: pgmap v12623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:57 smithi161 ceph-mon[91579]: pgmap v12624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:57 smithi188 ceph-mon[92425]: pgmap v12624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:39:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:59 smithi161 ceph-mon[91579]: pgmap v12625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:39:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:39:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:39:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:39:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:59 smithi188 ceph-mon[92425]: pgmap v12625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:39:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:39:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:39:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:39:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:40:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:40:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:40:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:01 smithi161 ceph-mon[91579]: pgmap v12626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:01 smithi188 ceph-mon[92425]: pgmap v12626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:03 smithi161 ceph-mon[91579]: pgmap v12627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:03 smithi188 ceph-mon[92425]: pgmap v12627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:05 smithi161 ceph-mon[91579]: pgmap v12628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:05 smithi188 ceph-mon[92425]: pgmap v12628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:07 smithi161 ceph-mon[91579]: pgmap v12629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:07 smithi188 ceph-mon[92425]: pgmap v12629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:09.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-mon[91579]: pgmap v12630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:40:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:40:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:40:09.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:40:09.259+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:40:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:40:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:40:09.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:40:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:40:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:40:09.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:40:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:40:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:40:09.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:40:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:40:09.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:40:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:40:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:40:09.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:40:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:40:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:40:09.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:40:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:40:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:40:09.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:40:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:40:09.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:40:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:40:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:40:09.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:40:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:40:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:40:09.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:40:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:40:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:40:09.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:40:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:40:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:40:09.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:40:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:40:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:40:09.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:40:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:40:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:40:09.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:40:09.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:40:09.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:40:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:40:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:40:09.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=737] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:40:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=737] Local address: 172.21.15.161, port 58134 2022-07-03T17:40:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=737] Peer address: 172.21.15.101, port 22 2022-07-03T17:40:09.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=737] Connection lost 2022-07-03T17:40:09.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=737] Aborting connection 2022-07-03T17:40:09.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:40:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:40:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:09 smithi188 ceph-mon[92425]: pgmap v12630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:40:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:40:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:40:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:40:10.981 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:40:10.981 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:40:10.982 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:40:10.982 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:40:10.982 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:40:10.983 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:40:10.983 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:40:10.983 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:40:10.984 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:40:10.984 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:40:10.984 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:40:10.985 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:40:10.985 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:40:10.985 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:40:10.986 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:40:10.986 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:40:10.987 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:40:10.987 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:40:10.987 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:40:10.988 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:40:10.988 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:40:10.988 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:40:10.989 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:40:10.989 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:40:10.990 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:40:10.990 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:40:10.990 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:40:10.991 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:40:10.991 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:40:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:40:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:40:10.992 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:40:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:40:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:40:10.993 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:40:10.994 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:40:10.994 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:40:10.995 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:40:10.995 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:40:10.995 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:40:10.996 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:40:10.996 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:40:10.996 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:40:10.997 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:40:10.997 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:40:10.997 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=737] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:40:10.998 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=737] Local address: 172.21.15.161, port 58134 2022-07-03T17:40:10.998 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=737] Peer address: 172.21.15.101, port 22 2022-07-03T17:40:10.999 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=737] Connection lost 2022-07-03T17:40:10.999 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=737] Aborting connection 2022-07-03T17:40:11.000 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:40:11.000 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:40:11.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:40:11.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:40:11.001 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:40:11.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:40:11.002 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:40:11.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:40:11.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:40:11.003 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:40:11.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:40:11.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:40:11.004 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:40:11.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:40:11.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:40:11.005 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:40:11.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:40:11.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:40:11.006 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:40:11.007 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:40:11.007 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:40:11.007 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:40:11.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:40:11.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:40:11.008 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:40:11.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:40:11.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:40:11.009 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:40:11.010 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:40:11.010 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:40:11.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:40:11.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:40:11.011 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:40:11.012 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:40:11.012 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:40:11.012 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:40:11.013 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:40:11.013 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:40:11.014 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:40:11.014 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:40:11.014 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:40:11.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:40:11.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:40:11.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:40:11.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:40:11.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:40:11.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=737] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:40:11.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=737] Local address: 172.21.15.161, port 58134 2022-07-03T17:40:11.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=737] Peer address: 172.21.15.101, port 22 2022-07-03T17:40:11.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=737] Connection lost 2022-07-03T17:40:11.018 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=737] Aborting connection 2022-07-03T17:40:11.859 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:11 smithi161 ceph-mon[91579]: pgmap v12631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:11 smithi188 ceph-mon[92425]: pgmap v12631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:13 smithi161 ceph-mon[91579]: pgmap v12632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:13 smithi188 ceph-mon[92425]: pgmap v12632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:15 smithi161 ceph-mon[91579]: pgmap v12633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:15 smithi188 ceph-mon[92425]: pgmap v12633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:17 smithi161 ceph-mon[91579]: pgmap v12634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:17 smithi188 ceph-mon[92425]: pgmap v12634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:19 smithi161 ceph-mon[91579]: pgmap v12635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:19 smithi188 ceph-mon[92425]: pgmap v12635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:21.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:21 smithi161 ceph-mon[91579]: pgmap v12636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:21 smithi188 ceph-mon[92425]: pgmap v12636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:23 smithi161 ceph-mon[91579]: pgmap v12637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:23 smithi188 ceph-mon[92425]: pgmap v12637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:25.851 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:25 smithi161 ceph-mon[91579]: pgmap v12638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:25 smithi188 ceph-mon[92425]: pgmap v12638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:27 smithi161 ceph-mon[91579]: pgmap v12639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:27 smithi188 ceph-mon[92425]: pgmap v12639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:29 smithi161 ceph-mon[91579]: pgmap v12640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:29 smithi188 ceph-mon[92425]: pgmap v12640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:31 smithi161 ceph-mon[91579]: pgmap v12641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:31 smithi188 ceph-mon[92425]: pgmap v12641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:33 smithi161 ceph-mon[91579]: pgmap v12642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:33 smithi188 ceph-mon[92425]: pgmap v12642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:35 smithi161 ceph-mon[91579]: pgmap v12643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:35 smithi188 ceph-mon[92425]: pgmap v12643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:37 smithi161 ceph-mon[91579]: pgmap v12644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:37 smithi188 ceph-mon[92425]: pgmap v12644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:39 smithi161 ceph-mon[91579]: pgmap v12645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:39 smithi188 ceph-mon[92425]: pgmap v12645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:41 smithi161 ceph-mon[91579]: pgmap v12646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:41.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:41 smithi188 ceph-mon[92425]: pgmap v12646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:43 smithi161 ceph-mon[91579]: pgmap v12647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:43 smithi188 ceph-mon[92425]: pgmap v12647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:45 smithi161 ceph-mon[91579]: pgmap v12648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:45 smithi188 ceph-mon[92425]: pgmap v12648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:47 smithi161 ceph-mon[91579]: pgmap v12649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:47 smithi188 ceph-mon[92425]: pgmap v12649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:49 smithi161 ceph-mon[91579]: pgmap v12650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:49 smithi188 ceph-mon[92425]: pgmap v12650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:51 smithi161 ceph-mon[91579]: pgmap v12651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:51 smithi188 ceph-mon[92425]: pgmap v12651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:53 smithi161 ceph-mon[91579]: pgmap v12652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:53 smithi188 ceph-mon[92425]: pgmap v12652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:55 smithi161 ceph-mon[91579]: pgmap v12653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:55 smithi188 ceph-mon[92425]: pgmap v12653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:57 smithi161 ceph-mon[91579]: pgmap v12654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:57 smithi188 ceph-mon[92425]: pgmap v12654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:40:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:59 smithi161 ceph-mon[91579]: pgmap v12655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:40:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:40:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:40:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:59 smithi188 ceph-mon[92425]: pgmap v12655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:40:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:40:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:40:59.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:40:59.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:41:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:01 smithi161 ceph-mon[91579]: pgmap v12656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:01 smithi188 ceph-mon[92425]: pgmap v12656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:03 smithi161 ceph-mon[91579]: pgmap v12657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:03 smithi188 ceph-mon[92425]: pgmap v12657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:05 smithi161 ceph-mon[91579]: pgmap v12658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:05 smithi188 ceph-mon[92425]: pgmap v12658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:07.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:07 smithi161 ceph-mon[91579]: pgmap v12659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:07 smithi188 ceph-mon[92425]: pgmap v12659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:09.771 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-mon[91579]: pgmap v12660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:09.783 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:09 smithi188 ceph-mon[92425]: pgmap v12660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:10.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:41:09.768+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:41:10.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:41:10.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:41:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:41:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:41:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:41:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:41:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:41:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:41:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:41:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:41:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:41:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:41:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:41:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:41:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:41:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:41:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:41:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:41:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:41:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:41:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:41:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:41:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:41:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:41:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:41:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:41:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:41:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:41:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:41:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:41:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:41:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:41:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:41:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:41:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:41:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:41:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:41:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:41:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:41:10.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:41:10.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:41:10.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:41:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:41:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:41:10.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=739] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:41:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=739] Local address: 172.21.15.161, port 58140 2022-07-03T17:41:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=739] Peer address: 172.21.15.101, port 22 2022-07-03T17:41:10.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=739] Connection lost 2022-07-03T17:41:10.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=739] Aborting connection 2022-07-03T17:41:10.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:41:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:41:10.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:41:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:41:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:41:10.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:41:10.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:41:10.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:41:10.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:41:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:41:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:41:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:41:10.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:41:10.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:41:10.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:41:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:41:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:41:10.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:41:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:41:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:41:10.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:41:10.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:41:10.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:41:10.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:41:10.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:41:10.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:41:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:41:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:41:10.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:41:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:41:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:41:10.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:41:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:41:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:41:10.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:41:10.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:41:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:41:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:41:10.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:41:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:41:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:41:10.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:41:10.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:41:10.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:41:10.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:41:10.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:41:10.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:41:10.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:41:10.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:41:10.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:41:10.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=739] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:41:10.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=739] Local address: 172.21.15.161, port 58140 2022-07-03T17:41:10.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=739] Peer address: 172.21.15.101, port 22 2022-07-03T17:41:10.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=739] Connection lost 2022-07-03T17:41:10.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=739] Aborting connection 2022-07-03T17:41:10.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:41:10.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:41:10.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:41:10.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:41:10.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:41:10.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:41:10.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:41:10.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:41:10.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:41:10.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:41:10.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:41:10.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:41:10.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:41:10.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:41:10.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:41:10.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:41:10.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:41:10.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:41:10.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:41:10.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:41:10.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:41:10.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:41:10.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:41:10.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:41:10.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:41:10.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:41:10.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:41:10.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:41:10.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:41:10.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:41:10.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:41:10.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:41:10.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:41:10.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:41:10.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:41:10.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:41:10.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:41:10.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:41:10.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:41:10.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:41:10.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:41:10.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:41:10.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:41:10.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:41:10.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:41:10.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:41:10.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:41:10.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=739] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:41:10.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=739] Local address: 172.21.15.161, port 58140 2022-07-03T17:41:10.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=739] Peer address: 172.21.15.101, port 22 2022-07-03T17:41:10.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=739] Connection lost 2022-07-03T17:41:10.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=739] Aborting connection 2022-07-03T17:41:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:11 smithi161 ceph-mon[91579]: pgmap v12661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:11 smithi188 ceph-mon[92425]: pgmap v12661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:13 smithi161 ceph-mon[91579]: pgmap v12662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:13 smithi188 ceph-mon[92425]: pgmap v12662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:15 smithi161 ceph-mon[91579]: pgmap v12663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:15 smithi188 ceph-mon[92425]: pgmap v12663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:17 smithi161 ceph-mon[91579]: pgmap v12664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:17 smithi188 ceph-mon[92425]: pgmap v12664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:19 smithi161 ceph-mon[91579]: pgmap v12665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:19 smithi188 ceph-mon[92425]: pgmap v12665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:21 smithi161 ceph-mon[91579]: pgmap v12666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:21 smithi188 ceph-mon[92425]: pgmap v12666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:23 smithi161 ceph-mon[91579]: pgmap v12667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:23 smithi188 ceph-mon[92425]: pgmap v12667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:25 smithi161 ceph-mon[91579]: pgmap v12668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:25 smithi188 ceph-mon[92425]: pgmap v12668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:27 smithi161 ceph-mon[91579]: pgmap v12669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:27 smithi188 ceph-mon[92425]: pgmap v12669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:29 smithi161 ceph-mon[91579]: pgmap v12670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:29 smithi188 ceph-mon[92425]: pgmap v12670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:31 smithi161 ceph-mon[91579]: pgmap v12671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:31 smithi188 ceph-mon[92425]: pgmap v12671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:33 smithi161 ceph-mon[91579]: pgmap v12672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:33 smithi188 ceph-mon[92425]: pgmap v12672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:35 smithi161 ceph-mon[91579]: pgmap v12673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:35 smithi188 ceph-mon[92425]: pgmap v12673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:37 smithi161 ceph-mon[91579]: pgmap v12674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:37 smithi188 ceph-mon[92425]: pgmap v12674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:39 smithi161 ceph-mon[91579]: pgmap v12675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:39 smithi188 ceph-mon[92425]: pgmap v12675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:41 smithi161 ceph-mon[91579]: pgmap v12676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:41 smithi188 ceph-mon[92425]: pgmap v12676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:43 smithi161 ceph-mon[91579]: pgmap v12677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:43 smithi188 ceph-mon[92425]: pgmap v12677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:45 smithi161 ceph-mon[91579]: pgmap v12678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:45 smithi188 ceph-mon[92425]: pgmap v12678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:47 smithi161 ceph-mon[91579]: pgmap v12679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:47 smithi188 ceph-mon[92425]: pgmap v12679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:49 smithi161 ceph-mon[91579]: pgmap v12680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:49 smithi188 ceph-mon[92425]: pgmap v12680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:51 smithi161 ceph-mon[91579]: pgmap v12681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:51 smithi188 ceph-mon[92425]: pgmap v12681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:53 smithi161 ceph-mon[91579]: pgmap v12682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:53 smithi188 ceph-mon[92425]: pgmap v12682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:41:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:55 smithi161 ceph-mon[91579]: pgmap v12683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:55 smithi188 ceph-mon[92425]: pgmap v12683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:57 smithi161 ceph-mon[91579]: pgmap v12684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:41:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:57 smithi188 ceph-mon[92425]: pgmap v12684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:41:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:59 smithi161 ceph-mon[91579]: pgmap v12685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:41:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:41:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:41:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:41:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:59 smithi188 ceph-mon[92425]: pgmap v12685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:41:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:41:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:41:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:41:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:42:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:01 smithi161 ceph-mon[91579]: pgmap v12686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:42:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:01 smithi188 ceph-mon[92425]: pgmap v12686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:42:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:03 smithi161 ceph-mon[91579]: pgmap v12687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:42:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:03 smithi188 ceph-mon[92425]: pgmap v12687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:42:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:05 smithi161 ceph-mon[91579]: pgmap v12688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:05 smithi188 ceph-mon[92425]: pgmap v12688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:07 smithi161 ceph-mon[91579]: pgmap v12689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:07 smithi188 ceph-mon[92425]: pgmap v12689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:09 smithi161 ceph-mon[91579]: pgmap v12690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:09 smithi188 ceph-mon[92425]: pgmap v12690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:42:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:42:10.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:42:10.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:42:10.274+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:42:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:42:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:42:10.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:42:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:42:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:42:10.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:42:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:42:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:42:10.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:42:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:42:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:42:10.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:42:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:42:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:42:10.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:42:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:42:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:42:10.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:42:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:42:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:42:10.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:42:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:42:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:42:10.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:42:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:42:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:42:10.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:42:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:42:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:42:10.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:42:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:42:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:42:10.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:42:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:42:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:42:10.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:42:10.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:42:10.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:42:10.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:42:10.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:42:10.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:42:10.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:42:10.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:42:10.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:42:10.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=741] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:42:10.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=741] Local address: 172.21.15.161, port 58144 2022-07-03T17:42:10.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=741] Peer address: 172.21.15.101, port 22 2022-07-03T17:42:10.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=741] Connection lost 2022-07-03T17:42:10.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=741] Aborting connection 2022-07-03T17:42:10.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:42:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:42:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:42:10.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:42:11.784 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:42:11.785 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:42:11.785 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:42:11.786 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:42:11.786 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:42:11.786 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:42:11.787 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:42:11.787 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:42:11.788 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:42:11.788 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:42:11.789 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:42:11.789 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:42:11.789 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:42:11.790 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:42:11.790 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:42:11.790 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:42:11.791 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:42:11.791 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:42:11.791 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:42:11.792 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:42:11.792 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:42:11.792 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:42:11.793 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:42:11.793 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:42:11.793 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:42:11.794 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:42:11.794 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:42:11.794 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:42:11.795 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:42:11.795 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:42:11.795 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:42:11.796 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:42:11.796 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:42:11.796 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:42:11.797 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:42:11.797 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:42:11.797 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:42:11.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:42:11.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:42:11.798 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:42:11.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:42:11.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:42:11.799 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:42:11.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:42:11.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:42:11.800 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:42:11.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=741] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:42:11.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=741] Local address: 172.21.15.161, port 58144 2022-07-03T17:42:11.801 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=741] Peer address: 172.21.15.101, port 22 2022-07-03T17:42:11.802 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=741] Connection lost 2022-07-03T17:42:11.802 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=741] Aborting connection 2022-07-03T17:42:11.802 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:11 smithi161 ceph-mon[91579]: pgmap v12691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:42:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:42:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:42:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:42:11.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:42:11.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:42:11.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:42:11.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:42:11.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:42:11.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:42:11.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:42:11.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:42:11.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:42:11.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:42:11.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:42:11.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:42:11.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:42:11.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:42:11.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:42:11.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:42:11.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:42:11.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:42:11.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:42:11.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:42:11.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:42:11.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:42:11.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:42:11.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:42:11.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:42:11.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:42:11.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:42:11.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:42:11.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:42:11.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:42:11.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:42:11.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:42:11.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:42:11.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:42:11.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:42:11.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:42:11.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:42:11.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:42:11.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:42:11.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:42:11.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:42:11.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:42:11.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=741] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:42:11.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=741] Local address: 172.21.15.161, port 58144 2022-07-03T17:42:11.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=741] Peer address: 172.21.15.101, port 22 2022-07-03T17:42:11.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=741] Connection lost 2022-07-03T17:42:11.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=741] Aborting connection 2022-07-03T17:42:11.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:11 smithi188 ceph-mon[92425]: pgmap v12691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:13.530 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:13 smithi161 ceph-mon[91579]: pgmap v12692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:13.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:13 smithi188 ceph-mon[92425]: pgmap v12692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:14 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T17:42:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:42:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:42:14.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:14 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T17:42:14.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:42:14.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:42:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:15 smithi161 ceph-mon[91579]: pgmap v12693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:15 smithi188 ceph-mon[92425]: pgmap v12693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:17 smithi161 ceph-mon[91579]: pgmap v12694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:17 smithi188 ceph-mon[92425]: pgmap v12694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:19 smithi161 ceph-mon[91579]: pgmap v12695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:19 smithi188 ceph-mon[92425]: pgmap v12695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:21 smithi161 ceph-mon[91579]: pgmap v12696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:21.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:21 smithi188 ceph-mon[92425]: pgmap v12696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:23 smithi161 ceph-mon[91579]: pgmap v12697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:23 smithi188 ceph-mon[92425]: pgmap v12697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:25 smithi161 ceph-mon[91579]: pgmap v12698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:25 smithi188 ceph-mon[92425]: pgmap v12698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:27 smithi161 ceph-mon[91579]: pgmap v12699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:27 smithi188 ceph-mon[92425]: pgmap v12699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:29 smithi161 ceph-mon[91579]: pgmap v12700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:29 smithi188 ceph-mon[92425]: pgmap v12700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:31 smithi161 ceph-mon[91579]: pgmap v12701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:31 smithi188 ceph-mon[92425]: pgmap v12701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:33 smithi161 ceph-mon[91579]: pgmap v12702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:33 smithi188 ceph-mon[92425]: pgmap v12702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:35 smithi161 ceph-mon[91579]: pgmap v12703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:35 smithi188 ceph-mon[92425]: pgmap v12703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:37 smithi161 ceph-mon[91579]: pgmap v12704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:37 smithi188 ceph-mon[92425]: pgmap v12704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:39 smithi161 ceph-mon[91579]: pgmap v12705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:39 smithi188 ceph-mon[92425]: pgmap v12705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:41 smithi161 ceph-mon[91579]: pgmap v12706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:41.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:41 smithi188 ceph-mon[92425]: pgmap v12706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:43 smithi161 ceph-mon[91579]: pgmap v12707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:43 smithi188 ceph-mon[92425]: pgmap v12707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:45 smithi161 ceph-mon[91579]: pgmap v12708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:45 smithi188 ceph-mon[92425]: pgmap v12708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:47 smithi161 ceph-mon[91579]: pgmap v12709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:47 smithi188 ceph-mon[92425]: pgmap v12709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:49 smithi161 ceph-mon[91579]: pgmap v12710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:49 smithi188 ceph-mon[92425]: pgmap v12710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:51 smithi161 ceph-mon[91579]: pgmap v12711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:51 smithi188 ceph-mon[92425]: pgmap v12711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:53 smithi161 ceph-mon[91579]: pgmap v12712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:53 smithi188 ceph-mon[92425]: pgmap v12712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:55 smithi161 ceph-mon[91579]: pgmap v12713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:55 smithi188 ceph-mon[92425]: pgmap v12713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:57 smithi161 ceph-mon[91579]: pgmap v12714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:57 smithi188 ceph-mon[92425]: pgmap v12714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:42:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:59 smithi161 ceph-mon[91579]: pgmap v12715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:42:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:42:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:42:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:42:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:59 smithi188 ceph-mon[92425]: pgmap v12715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:42:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:42:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:42:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:42:59.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:43:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:01 smithi161 ceph-mon[91579]: pgmap v12716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:01 smithi188 ceph-mon[92425]: pgmap v12716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:03 smithi161 ceph-mon[91579]: pgmap v12717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:03 smithi188 ceph-mon[92425]: pgmap v12717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:05 smithi161 ceph-mon[91579]: pgmap v12718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:05 smithi188 ceph-mon[92425]: pgmap v12718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:07 smithi161 ceph-mon[91579]: pgmap v12719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:07 smithi188 ceph-mon[92425]: pgmap v12719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:09 smithi161 ceph-mon[91579]: pgmap v12720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:09 smithi188 ceph-mon[92425]: pgmap v12720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:11 smithi161 ceph-mon[91579]: pgmap v12721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:11 smithi188 ceph-mon[92425]: pgmap v12721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-mon[91579]: pgmap v12722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:13.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:43:13.555+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:43:13.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:43:13.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:43:13.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:43:13.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:43:13.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:43:13.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:43:13.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:43:13.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:43:13.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:43:13.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:43:13.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:43:13.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:43:13.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:43:13.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:43:13.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:43:13.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:43:13.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:43:13.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:43:13.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:43:13.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:43:13.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:43:13.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:43:13.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:43:13.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:43:13.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:43:13.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:43:13.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:43:13.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:43:13.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:43:13.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:43:13.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:43:13.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:43:13.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:43:13.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:43:13.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:43:13.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:43:13.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:43:13.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:43:13.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:43:13.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:43:13.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:43:13.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:43:13.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:43:13.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:43:13.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=743] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:43:13.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=743] Local address: 172.21.15.161, port 58148 2022-07-03T17:43:13.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=743] Peer address: 172.21.15.101, port 22 2022-07-03T17:43:13.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=743] Connection lost 2022-07-03T17:43:13.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=743] Aborting connection 2022-07-03T17:43:13.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:43:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:43:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:13 smithi188 ceph-mon[92425]: pgmap v12722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:43:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:43:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:43:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:43:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:43:14.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:43:14.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:43:14.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:43:14.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:43:14.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:43:14.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:43:14.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:43:14.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:43:14.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:43:14.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:43:14.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:43:14.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:43:14.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:43:14.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:43:14.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:43:14.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:43:14.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:43:14.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:43:14.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:43:14.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:43:14.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:43:14.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:43:14.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:43:14.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:43:14.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:43:14.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:43:14.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:43:14.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:43:14.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:43:14.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:43:14.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:43:14.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:43:14.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:43:14.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:43:14.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:43:14.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:43:14.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:43:14.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:43:14.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=743] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:43:14.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=743] Local address: 172.21.15.161, port 58148 2022-07-03T17:43:14.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=743] Peer address: 172.21.15.101, port 22 2022-07-03T17:43:14.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=743] Connection lost 2022-07-03T17:43:14.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=743] Aborting connection 2022-07-03T17:43:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:43:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:43:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:43:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:43:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:43:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:43:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:43:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:43:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:43:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:43:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:43:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:43:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:43:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:43:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:43:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:43:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:43:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:43:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:43:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:43:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:43:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:43:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:43:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:43:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:43:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:43:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:43:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:43:14.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:43:14.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:43:14.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:43:14.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:43:14.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:43:14.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:43:14.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:43:14.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:43:14.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:43:14.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:43:14.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:43:14.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=743] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:43:14.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=743] Local address: 172.21.15.161, port 58148 2022-07-03T17:43:14.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=743] Peer address: 172.21.15.101, port 22 2022-07-03T17:43:14.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=743] Connection lost 2022-07-03T17:43:14.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=743] Aborting connection 2022-07-03T17:43:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:15 smithi188 ceph-mon[92425]: pgmap v12723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:15 smithi161 ceph-mon[91579]: pgmap v12723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:17 smithi161 ceph-mon[91579]: pgmap v12724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:17 smithi188 ceph-mon[92425]: pgmap v12724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:19 smithi161 ceph-mon[91579]: pgmap v12725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:19 smithi188 ceph-mon[92425]: pgmap v12725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:21 smithi161 ceph-mon[91579]: pgmap v12726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:21 smithi188 ceph-mon[92425]: pgmap v12726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:23 smithi188 ceph-mon[92425]: pgmap v12727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:23 smithi161 ceph-mon[91579]: pgmap v12727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:25 smithi161 ceph-mon[91579]: pgmap v12728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:25 smithi188 ceph-mon[92425]: pgmap v12728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:27 smithi161 ceph-mon[91579]: pgmap v12729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:27 smithi188 ceph-mon[92425]: pgmap v12729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:29 smithi161 ceph-mon[91579]: pgmap v12730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:29 smithi188 ceph-mon[92425]: pgmap v12730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:31 smithi161 ceph-mon[91579]: pgmap v12731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:31 smithi188 ceph-mon[92425]: pgmap v12731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:33 smithi188 ceph-mon[92425]: pgmap v12732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:33 smithi161 ceph-mon[91579]: pgmap v12732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:35 smithi161 ceph-mon[91579]: pgmap v12733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:35 smithi188 ceph-mon[92425]: pgmap v12733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:37 smithi161 ceph-mon[91579]: pgmap v12734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:37 smithi188 ceph-mon[92425]: pgmap v12734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:39 smithi161 ceph-mon[91579]: pgmap v12735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:39 smithi188 ceph-mon[92425]: pgmap v12735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:41 smithi161 ceph-mon[91579]: pgmap v12736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:41 smithi188 ceph-mon[92425]: pgmap v12736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:43 smithi188 ceph-mon[92425]: pgmap v12737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:43 smithi161 ceph-mon[91579]: pgmap v12737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:45 smithi161 ceph-mon[91579]: pgmap v12738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:45 smithi188 ceph-mon[92425]: pgmap v12738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:47 smithi161 ceph-mon[91579]: pgmap v12739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:47 smithi188 ceph-mon[92425]: pgmap v12739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:49 smithi161 ceph-mon[91579]: pgmap v12740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:49 smithi188 ceph-mon[92425]: pgmap v12740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:51 smithi161 ceph-mon[91579]: pgmap v12741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:51 smithi188 ceph-mon[92425]: pgmap v12741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:53 smithi188 ceph-mon[92425]: pgmap v12742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:53 smithi161 ceph-mon[91579]: pgmap v12742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:55 smithi161 ceph-mon[91579]: pgmap v12743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:55 smithi188 ceph-mon[92425]: pgmap v12743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:57 smithi161 ceph-mon[91579]: pgmap v12744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:57 smithi188 ceph-mon[92425]: pgmap v12744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:43:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:59 smithi161 ceph-mon[91579]: pgmap v12745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:43:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:43:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:43:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:43:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:59 smithi188 ceph-mon[92425]: pgmap v12745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:43:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:43:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:43:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:43:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:44:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:01 smithi161 ceph-mon[91579]: pgmap v12746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:01 smithi188 ceph-mon[92425]: pgmap v12746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:03 smithi188 ceph-mon[92425]: pgmap v12747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:03 smithi161 ceph-mon[91579]: pgmap v12747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:05 smithi161 ceph-mon[91579]: pgmap v12748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:05 smithi188 ceph-mon[92425]: pgmap v12748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:07 smithi161 ceph-mon[91579]: pgmap v12749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:07 smithi188 ceph-mon[92425]: pgmap v12749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:09 smithi161 ceph-mon[91579]: pgmap v12750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:09 smithi188 ceph-mon[92425]: pgmap v12750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:11 smithi161 ceph-mon[91579]: pgmap v12751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:11 smithi188 ceph-mon[92425]: pgmap v12751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:13 smithi188 ceph-mon[92425]: pgmap v12752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:13 smithi161 ceph-mon[91579]: pgmap v12752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:44:14.065+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:44:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:44:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:44:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:44:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:44:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:44:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:44:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:44:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:44:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:44:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:44:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:44:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:44:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:44:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:44:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:44:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:44:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:44:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:44:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:44:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:44:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:44:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:44:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:44:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:44:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:44:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:44:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:44:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:44:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:44:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:44:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:44:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:44:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:44:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:44:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:44:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:44:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:44:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:44:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:44:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:44:14.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:44:14.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:44:14.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:44:14.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:44:14.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=745] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:44:14.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=745] Local address: 172.21.15.161, port 58152 2022-07-03T17:44:14.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=745] Peer address: 172.21.15.101, port 22 2022-07-03T17:44:14.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=745] Connection lost 2022-07-03T17:44:14.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=745] Aborting connection 2022-07-03T17:44:14.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:44:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:44:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:44:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:44:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:44:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:44:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:44:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:44:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:44:15.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:44:15.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:44:15.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:44:15.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:44:15.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:44:15.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:44:15.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:44:15.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:44:15.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:44:15.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:44:15.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:44:15.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:44:15.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:44:15.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:44:15.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:44:15.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:44:15.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:44:15.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:44:15.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:44:15.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:44:15.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:44:15.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:44:15.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:44:15.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:44:15.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:44:15.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:44:15.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:44:15.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:44:15.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:44:15.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:44:15.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:44:15.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:44:15.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:44:15.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:44:15.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:44:15.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:44:15.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:44:15.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:44:15.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:44:15.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:44:15.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:44:15.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:44:15.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:44:15.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:44:15.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=745] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:44:15.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=745] Local address: 172.21.15.161, port 58152 2022-07-03T17:44:15.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=745] Peer address: 172.21.15.101, port 22 2022-07-03T17:44:15.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=745] Connection lost 2022-07-03T17:44:15.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=745] Aborting connection 2022-07-03T17:44:15.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:15 smithi188 ceph-mon[92425]: pgmap v12753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:44:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:44:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:44:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:44:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:44:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:44:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:44:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:44:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:44:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:44:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:44:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:44:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:44:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:44:15.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:44:15.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:44:15.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:44:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:44:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:44:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:44:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:44:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:44:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:44:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:44:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:44:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:44:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:44:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:44:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:44:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:44:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:44:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:44:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:44:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:44:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:44:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:44:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:44:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:44:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:44:15.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:44:15.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:44:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:44:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:44:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:44:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:44:15.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:44:15.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=745] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:44:15.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=745] Local address: 172.21.15.161, port 58152 2022-07-03T17:44:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=745] Peer address: 172.21.15.101, port 22 2022-07-03T17:44:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=745] Connection lost 2022-07-03T17:44:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=745] Aborting connection 2022-07-03T17:44:15.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:15 smithi161 ceph-mon[91579]: pgmap v12753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:17 smithi161 ceph-mon[91579]: pgmap v12754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:17 smithi188 ceph-mon[92425]: pgmap v12754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:19 smithi161 ceph-mon[91579]: pgmap v12755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:19 smithi188 ceph-mon[92425]: pgmap v12755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:21 smithi161 ceph-mon[91579]: pgmap v12756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:21 smithi188 ceph-mon[92425]: pgmap v12756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:23 smithi188 ceph-mon[92425]: pgmap v12757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:23 smithi161 ceph-mon[91579]: pgmap v12757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:25 smithi161 ceph-mon[91579]: pgmap v12758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:25 smithi188 ceph-mon[92425]: pgmap v12758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:27 smithi161 ceph-mon[91579]: pgmap v12759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:27 smithi188 ceph-mon[92425]: pgmap v12759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:29 smithi161 ceph-mon[91579]: pgmap v12760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:29 smithi188 ceph-mon[92425]: pgmap v12760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:31 smithi161 ceph-mon[91579]: pgmap v12761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:31 smithi188 ceph-mon[92425]: pgmap v12761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:33 smithi188 ceph-mon[92425]: pgmap v12762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:33 smithi161 ceph-mon[91579]: pgmap v12762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:35 smithi161 ceph-mon[91579]: pgmap v12763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:35 smithi188 ceph-mon[92425]: pgmap v12763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:37 smithi161 ceph-mon[91579]: pgmap v12764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:37 smithi188 ceph-mon[92425]: pgmap v12764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:39 smithi161 ceph-mon[91579]: pgmap v12765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:39 smithi188 ceph-mon[92425]: pgmap v12765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:41 smithi161 ceph-mon[91579]: pgmap v12766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:41 smithi188 ceph-mon[92425]: pgmap v12766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:43 smithi188 ceph-mon[92425]: pgmap v12767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:43 smithi161 ceph-mon[91579]: pgmap v12767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:45 smithi161 ceph-mon[91579]: pgmap v12768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:45 smithi188 ceph-mon[92425]: pgmap v12768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:47 smithi161 ceph-mon[91579]: pgmap v12769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:47 smithi188 ceph-mon[92425]: pgmap v12769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:49 smithi161 ceph-mon[91579]: pgmap v12770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:49 smithi188 ceph-mon[92425]: pgmap v12770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:51 smithi161 ceph-mon[91579]: pgmap v12771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:51 smithi188 ceph-mon[92425]: pgmap v12771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:53 smithi188 ceph-mon[92425]: pgmap v12772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:53.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:53 smithi161 ceph-mon[91579]: pgmap v12772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:55 smithi161 ceph-mon[91579]: pgmap v12773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:55 smithi188 ceph-mon[92425]: pgmap v12773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:57.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:57 smithi161 ceph-mon[91579]: pgmap v12774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:57 smithi188 ceph-mon[92425]: pgmap v12774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:44:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:59 smithi161 ceph-mon[91579]: pgmap v12775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:44:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:44:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:44:59.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:44:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:59 smithi188 ceph-mon[92425]: pgmap v12775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:44:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:44:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:44:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:44:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:45:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:01 smithi161 ceph-mon[91579]: pgmap v12776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:01 smithi188 ceph-mon[92425]: pgmap v12776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:03 smithi188 ceph-mon[92425]: pgmap v12777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:03 smithi161 ceph-mon[91579]: pgmap v12777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:05.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:05 smithi161 ceph-mon[91579]: pgmap v12778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:05 smithi188 ceph-mon[92425]: pgmap v12778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:07 smithi161 ceph-mon[91579]: pgmap v12779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:07 smithi188 ceph-mon[92425]: pgmap v12779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:09 smithi161 ceph-mon[91579]: pgmap v12780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:09.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:09 smithi188 ceph-mon[92425]: pgmap v12780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:11 smithi161 ceph-mon[91579]: pgmap v12781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:11 smithi188 ceph-mon[92425]: pgmap v12781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:12.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:13.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:13 smithi161 ceph-mon[91579]: pgmap v12782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:13 smithi188 ceph-mon[92425]: pgmap v12782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:45:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:45:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:45:14.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:45:14.568+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:45:14.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:45:14.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:45:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:45:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:45:14.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:45:14.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:45:14.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:45:14.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:45:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:45:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:45:14.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:45:14.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:45:14.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:45:14.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:45:14.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:45:14.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:45:14.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:45:14.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:45:14.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:45:14.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:45:14.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:45:14.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:45:14.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:45:14.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:45:14.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:45:14.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:45:14.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:45:14.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:45:14.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:45:14.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:45:14.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:45:14.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:45:14.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:45:14.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:45:14.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:45:14.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:45:14.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:45:14.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:45:14.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:45:14.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:45:14.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:45:14.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:45:14.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:45:14.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:45:14.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=747] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:45:14.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=747] Local address: 172.21.15.161, port 58162 2022-07-03T17:45:14.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=747] Peer address: 172.21.15.101, port 22 2022-07-03T17:45:14.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=747] Connection lost 2022-07-03T17:45:14.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=747] Aborting connection 2022-07-03T17:45:14.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:45:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:45:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:45:14.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:45:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:15 smithi161 ceph-mon[91579]: pgmap v12783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:45:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:45:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:45:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:45:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:45:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:45:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:45:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:45:15.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:45:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:45:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:45:15.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:45:15.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:45:15.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:45:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:45:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:45:15.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:45:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:45:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:45:15.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:45:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:45:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:45:15.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:45:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:45:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:45:15.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:45:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:45:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:45:15.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:45:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:45:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:45:15.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:45:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:45:15.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:45:15.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:45:15.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:45:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:45:15.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:45:15.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:45:15.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:45:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:45:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:45:15.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:45:15.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:45:15.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:45:15.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:45:15.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=747] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:45:15.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=747] Local address: 172.21.15.161, port 58162 2022-07-03T17:45:15.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=747] Peer address: 172.21.15.101, port 22 2022-07-03T17:45:15.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=747] Connection lost 2022-07-03T17:45:15.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=747] Aborting connection 2022-07-03T17:45:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:15 smithi188 ceph-mon[92425]: pgmap v12783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:45:15.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:45:15.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:45:15.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:45:15.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:45:15.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:45:15.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:45:15.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:45:15.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:45:15.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:45:15.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:45:15.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:45:15.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:45:15.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:45:15.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:45:15.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:45:15.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:45:15.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:45:15.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:45:15.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:45:15.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:45:15.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:45:15.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:45:15.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:45:15.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:45:15.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:45:15.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:45:15.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:45:15.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:45:15.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:45:15.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:45:15.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:45:15.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:45:15.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:45:15.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:45:15.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:45:15.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:45:15.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:45:15.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:45:15.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:45:15.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:45:15.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:45:15.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:45:15.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:45:15.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:45:15.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:45:15.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=747] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:45:15.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=747] Local address: 172.21.15.161, port 58162 2022-07-03T17:45:15.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=747] Peer address: 172.21.15.101, port 22 2022-07-03T17:45:15.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=747] Connection lost 2022-07-03T17:45:15.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=747] Aborting connection 2022-07-03T17:45:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:17 smithi161 ceph-mon[91579]: pgmap v12784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:17 smithi188 ceph-mon[92425]: pgmap v12784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:19 smithi161 ceph-mon[91579]: pgmap v12785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:19 smithi188 ceph-mon[92425]: pgmap v12785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:21.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:21 smithi161 ceph-mon[91579]: pgmap v12786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:21 smithi188 ceph-mon[92425]: pgmap v12786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:23 smithi161 ceph-mon[91579]: pgmap v12787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:23 smithi188 ceph-mon[92425]: pgmap v12787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:25.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:25 smithi161 ceph-mon[91579]: pgmap v12788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:25 smithi188 ceph-mon[92425]: pgmap v12788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:27 smithi161 ceph-mon[91579]: pgmap v12789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:27 smithi188 ceph-mon[92425]: pgmap v12789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:29 smithi161 ceph-mon[91579]: pgmap v12790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:29 smithi188 ceph-mon[92425]: pgmap v12790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:31 smithi161 ceph-mon[91579]: pgmap v12791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:31 smithi188 ceph-mon[92425]: pgmap v12791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:33 smithi161 ceph-mon[91579]: pgmap v12792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:33 smithi188 ceph-mon[92425]: pgmap v12792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:35 smithi161 ceph-mon[91579]: pgmap v12793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:35 smithi188 ceph-mon[92425]: pgmap v12793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:37 smithi161 ceph-mon[91579]: pgmap v12794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:37 smithi188 ceph-mon[92425]: pgmap v12794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:39 smithi161 ceph-mon[91579]: pgmap v12795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:39 smithi188 ceph-mon[92425]: pgmap v12795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:41 smithi161 ceph-mon[91579]: pgmap v12796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:41 smithi188 ceph-mon[92425]: pgmap v12796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:43.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:43 smithi161 ceph-mon[91579]: pgmap v12797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:43 smithi188 ceph-mon[92425]: pgmap v12797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:45 smithi161 ceph-mon[91579]: pgmap v12798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:45 smithi188 ceph-mon[92425]: pgmap v12798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:47 smithi161 ceph-mon[91579]: pgmap v12799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:47.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:47 smithi188 ceph-mon[92425]: pgmap v12799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:49 smithi161 ceph-mon[91579]: pgmap v12800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:49 smithi188 ceph-mon[92425]: pgmap v12800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:51.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:51 smithi161 ceph-mon[91579]: pgmap v12801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:51 smithi188 ceph-mon[92425]: pgmap v12801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:53 smithi161 ceph-mon[91579]: pgmap v12802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:53 smithi188 ceph-mon[92425]: pgmap v12802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:55 smithi188 ceph-mon[92425]: pgmap v12803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:55 smithi161 ceph-mon[91579]: pgmap v12803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:45:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:57 smithi188 ceph-mon[92425]: pgmap v12804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:57 smithi161 ceph-mon[91579]: pgmap v12804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:45:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:45:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:45:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:45:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:45:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:45:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:45:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:45:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:45:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:45:59 smithi188 ceph-mon[92425]: pgmap v12805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:45:59 smithi161 ceph-mon[91579]: pgmap v12805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:01 smithi188 ceph-mon[92425]: pgmap v12806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:01 smithi161 ceph-mon[91579]: pgmap v12806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:03 smithi188 ceph-mon[92425]: pgmap v12807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:03 smithi161 ceph-mon[91579]: pgmap v12807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:05 smithi188 ceph-mon[92425]: pgmap v12808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:06.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:05 smithi161 ceph-mon[91579]: pgmap v12808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:07 smithi188 ceph-mon[92425]: pgmap v12809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:07.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:07 smithi161 ceph-mon[91579]: pgmap v12809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:09.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:09 smithi188 ceph-mon[92425]: pgmap v12810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:09 smithi161 ceph-mon[91579]: pgmap v12810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:11 smithi188 ceph-mon[92425]: pgmap v12811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:11 smithi161 ceph-mon[91579]: pgmap v12811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:13 smithi188 ceph-mon[92425]: pgmap v12812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:13 smithi161 ceph-mon[91579]: pgmap v12812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:15.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:46:15.068+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:46:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:46:15.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:46:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:46:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:46:15.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:46:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:46:15.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:46:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:46:15.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:46:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:46:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:46:15.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:46:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:46:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:46:15.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:46:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:46:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:46:15.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:46:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:46:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:46:15.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:46:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:46:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:46:15.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:46:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:46:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:46:15.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:46:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:46:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:46:15.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:46:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:46:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:46:15.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:46:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:46:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:46:15.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:46:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:46:15.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:46:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:46:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:46:15.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:46:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:46:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:46:15.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:46:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=749] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:46:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=749] Local address: 172.21.15.161, port 58190 2022-07-03T17:46:15.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=749] Peer address: 172.21.15.101, port 22 2022-07-03T17:46:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=749] Connection lost 2022-07-03T17:46:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=749] Aborting connection 2022-07-03T17:46:15.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:46:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: pgmap v12813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:46:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:46:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:46:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: pgmap v12813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:46:15.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:46:15.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:46:15.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:15.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:17.014 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:46:17.015 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:46:17.015 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:46:17.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:46:17.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:46:17.016 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:46:17.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:46:17.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:46:17.017 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:46:17.018 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:46:17.018 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:46:17.019 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:46:17.019 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:46:17.020 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:46:17.020 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:46:17.021 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:46:17.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:46:17.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:46:17.022 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:46:17.023 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:46:17.024 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:46:17.024 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:46:17.024 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:46:17.025 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:46:17.025 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:46:17.025 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:46:17.026 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:46:17.026 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:46:17.026 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:46:17.027 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:46:17.027 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:46:17.028 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:46:17.028 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:46:17.028 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:46:17.029 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:46:17.029 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:46:17.029 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:46:17.030 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:46:17.030 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:46:17.030 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:46:17.031 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:46:17.031 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:46:17.031 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:46:17.032 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:46:17.032 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:46:17.032 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:46:17.033 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=749] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:46:17.033 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=749] Local address: 172.21.15.161, port 58190 2022-07-03T17:46:17.033 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=749] Peer address: 172.21.15.101, port 22 2022-07-03T17:46:17.034 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=749] Connection lost 2022-07-03T17:46:17.034 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=749] Aborting connection 2022-07-03T17:46:17.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:46:17.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:46:17.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:46:17.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:46:17.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:46:17.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:46:17.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:46:17.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:46:17.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:46:17.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:46:17.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:46:17.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:46:17.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:46:17.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:46:17.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:46:17.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:46:17.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:46:17.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:46:17.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:46:17.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:46:17.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:46:17.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:46:17.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:46:17.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:46:17.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:46:17.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:46:17.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:46:17.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:46:17.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:46:17.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:46:17.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:46:17.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:46:17.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:46:17.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:46:17.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:46:17.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:46:17.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:46:17.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:46:17.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:46:17.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:46:17.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:46:17.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:46:17.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:46:17.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:46:17.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:46:17.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:46:17.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=749] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:46:17.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=749] Local address: 172.21.15.161, port 58190 2022-07-03T17:46:17.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=749] Peer address: 172.21.15.101, port 22 2022-07-03T17:46:17.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=749] Connection lost 2022-07-03T17:46:17.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=749] Aborting connection 2022-07-03T17:46:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:17 smithi161 ceph-mon[91579]: pgmap v12814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:18.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:17 smithi188 ceph-mon[92425]: pgmap v12814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:18.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:19 smithi161 ceph-mon[91579]: pgmap v12815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:19 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T17:46:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:19 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:19 smithi188 ceph-mon[92425]: pgmap v12815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:19 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T17:46:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:20.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:46:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:21 smithi161 ceph-mon[91579]: pgmap v12816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:21 smithi188 ceph-mon[92425]: pgmap v12816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:24.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:23 smithi161 ceph-mon[91579]: pgmap v12817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:23 smithi188 ceph-mon[92425]: pgmap v12817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:24 smithi161 ceph-mon[91579]: pgmap v12818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:24 smithi188 ceph-mon[92425]: pgmap v12818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:27 smithi161 ceph-mon[91579]: pgmap v12819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:27 smithi188 ceph-mon[92425]: pgmap v12819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:29 smithi161 ceph-mon[91579]: pgmap v12820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:29 smithi188 ceph-mon[92425]: pgmap v12820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:31 smithi161 ceph-mon[91579]: pgmap v12821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:31 smithi188 ceph-mon[92425]: pgmap v12821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:33 smithi161 ceph-mon[91579]: pgmap v12822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:33 smithi188 ceph-mon[92425]: pgmap v12822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:35 smithi161 ceph-mon[91579]: pgmap v12823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:35 smithi188 ceph-mon[92425]: pgmap v12823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:37 smithi161 ceph-mon[91579]: pgmap v12824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:37 smithi188 ceph-mon[92425]: pgmap v12824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:39 smithi161 ceph-mon[91579]: pgmap v12825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:39 smithi188 ceph-mon[92425]: pgmap v12825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:41 smithi161 ceph-mon[91579]: pgmap v12826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:41 smithi188 ceph-mon[92425]: pgmap v12826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:43 smithi161 ceph-mon[91579]: pgmap v12827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:43 smithi188 ceph-mon[92425]: pgmap v12827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:45 smithi161 ceph-mon[91579]: pgmap v12828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:45 smithi188 ceph-mon[92425]: pgmap v12828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:47 smithi161 ceph-mon[91579]: pgmap v12829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:47.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:47.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:47 smithi188 ceph-mon[92425]: pgmap v12829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:47.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:49 smithi161 ceph-mon[91579]: pgmap v12830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:49 smithi188 ceph-mon[92425]: pgmap v12830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:51 smithi161 ceph-mon[91579]: pgmap v12831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:51.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:51 smithi188 ceph-mon[92425]: pgmap v12831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:53 smithi188 ceph-mon[92425]: pgmap v12832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:53 smithi161 ceph-mon[91579]: pgmap v12832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:55 smithi188 ceph-mon[92425]: pgmap v12833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:55 smithi161 ceph-mon[91579]: pgmap v12833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:46:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:57 smithi188 ceph-mon[92425]: pgmap v12834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:57 smithi161 ceph-mon[91579]: pgmap v12834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:46:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:46:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:46:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:46:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:46:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:46:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:46:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:46:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:46:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:46:59 smithi188 ceph-mon[92425]: pgmap v12835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:46:59 smithi161 ceph-mon[91579]: pgmap v12835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:01 smithi188 ceph-mon[92425]: pgmap v12836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:01 smithi161 ceph-mon[91579]: pgmap v12836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:03 smithi188 ceph-mon[92425]: pgmap v12837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:03 smithi161 ceph-mon[91579]: pgmap v12837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:05 smithi188 ceph-mon[92425]: pgmap v12838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:05 smithi161 ceph-mon[91579]: pgmap v12838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:07 smithi188 ceph-mon[92425]: pgmap v12839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:07.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:07 smithi161 ceph-mon[91579]: pgmap v12839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:09 smithi161 ceph-mon[91579]: pgmap v12840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:09 smithi188 ceph-mon[92425]: pgmap v12840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:11 smithi161 ceph-mon[91579]: pgmap v12841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:11 smithi188 ceph-mon[92425]: pgmap v12841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:13 smithi161 ceph-mon[91579]: pgmap v12842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:13 smithi188 ceph-mon[92425]: pgmap v12842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:15 smithi161 ceph-mon[91579]: pgmap v12843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:15 smithi188 ceph-mon[92425]: pgmap v12843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:17 smithi161 ceph-mon[91579]: pgmap v12844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:17 smithi188 ceph-mon[92425]: pgmap v12844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:18.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:19.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:47:18.882+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:47:19.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:47:19.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:47:19.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:47:19.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:47:19.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:47:19.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:47:19.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:47:19.374 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:47:19.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:47:19.375 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:47:19.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:47:19.376 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:47:19.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:47:19.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:47:19.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:47:19.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:47:19.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:47:19.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:47:19.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:47:19.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:47:19.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:47:19.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:47:19.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:47:19.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:47:19.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:47:19.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:47:19.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:47:19.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:47:19.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:47:19.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:47:19.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:47:19.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:47:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:47:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:47:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:47:19.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:47:19.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:47:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:47:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:47:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:47:19.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:47:19.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:47:19.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:47:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:47:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=751] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:47:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=751] Local address: 172.21.15.161, port 58220 2022-07-03T17:47:19.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=751] Peer address: 172.21.15.101, port 22 2022-07-03T17:47:19.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=751] Connection lost 2022-07-03T17:47:19.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=751] Aborting connection 2022-07-03T17:47:19.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:47:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:47:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:19 smithi161 ceph-mon[91579]: pgmap v12845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:47:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:47:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:47:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:19 smithi188 ceph-mon[92425]: pgmap v12845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:47:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:47:20.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:47:21.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:47:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:47:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:47:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:47:21.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:47:21.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:47:21.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:47:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:47:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:47:21.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:47:21.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:47:21.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:47:21.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:47:21.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:47:21.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:47:21.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:47:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:47:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:47:21.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:47:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:47:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:47:21.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:47:21.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:47:21.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:47:21.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:47:21.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:47:21.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:47:21.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:47:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:47:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:47:21.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:47:21.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:47:21.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:47:21.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:47:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:47:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:47:21.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:47:21.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:47:21.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:47:21.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:47:21.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:47:21.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:47:21.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:47:21.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:47:21.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:47:21.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:47:21.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=751] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:47:21.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=751] Local address: 172.21.15.161, port 58220 2022-07-03T17:47:21.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=751] Peer address: 172.21.15.101, port 22 2022-07-03T17:47:21.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=751] Connection lost 2022-07-03T17:47:21.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=751] Aborting connection 2022-07-03T17:47:21.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.157 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:47:21.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:47:21.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:47:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:47:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:47:21.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:47:21.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:47:21.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:47:21.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:47:21.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:47:21.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:47:21.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:47:21.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:47:21.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:47:21.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:47:21.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:47:21.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:47:21.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:47:21.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:47:21.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:47:21.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:47:21.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:47:21.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:47:21.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:47:21.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:47:21.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:47:21.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:47:21.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:47:21.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:47:21.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:47:21.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:47:21.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:47:21.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:47:21.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:47:21.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:47:21.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:47:21.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:47:21.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:47:21.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:47:21.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:47:21.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:47:21.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:47:21.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:47:21.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:47:21.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:47:21.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:47:21.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=751] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:47:21.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=751] Local address: 172.21.15.161, port 58220 2022-07-03T17:47:21.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=751] Peer address: 172.21.15.101, port 22 2022-07-03T17:47:21.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=751] Connection lost 2022-07-03T17:47:21.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=751] Aborting connection 2022-07-03T17:47:21.192 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:21.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:47:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:21 smithi161 ceph-mon[91579]: pgmap v12846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:21 smithi188 ceph-mon[92425]: pgmap v12846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:23 smithi161 ceph-mon[91579]: pgmap v12847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:23 smithi188 ceph-mon[92425]: pgmap v12847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:25 smithi161 ceph-mon[91579]: pgmap v12848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:25 smithi188 ceph-mon[92425]: pgmap v12848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:27 smithi161 ceph-mon[91579]: pgmap v12849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:27 smithi188 ceph-mon[92425]: pgmap v12849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:29 smithi161 ceph-mon[91579]: pgmap v12850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:29 smithi188 ceph-mon[92425]: pgmap v12850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:31 smithi161 ceph-mon[91579]: pgmap v12851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:31 smithi188 ceph-mon[92425]: pgmap v12851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:33 smithi161 ceph-mon[91579]: pgmap v12852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:33 smithi188 ceph-mon[92425]: pgmap v12852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:35 smithi161 ceph-mon[91579]: pgmap v12853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:35 smithi188 ceph-mon[92425]: pgmap v12853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:37 smithi161 ceph-mon[91579]: pgmap v12854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:37 smithi188 ceph-mon[92425]: pgmap v12854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:39 smithi161 ceph-mon[91579]: pgmap v12855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:40.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:39 smithi188 ceph-mon[92425]: pgmap v12855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:41 smithi161 ceph-mon[91579]: pgmap v12856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:41 smithi188 ceph-mon[92425]: pgmap v12856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:43 smithi161 ceph-mon[91579]: pgmap v12857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:43 smithi188 ceph-mon[92425]: pgmap v12857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:45 smithi161 ceph-mon[91579]: pgmap v12858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:45 smithi188 ceph-mon[92425]: pgmap v12858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:47 smithi161 ceph-mon[91579]: pgmap v12859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:47 smithi188 ceph-mon[92425]: pgmap v12859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:49 smithi161 ceph-mon[91579]: pgmap v12860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:49 smithi188 ceph-mon[92425]: pgmap v12860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:51 smithi161 ceph-mon[91579]: pgmap v12861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:51 smithi188 ceph-mon[92425]: pgmap v12861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:53 smithi161 ceph-mon[91579]: pgmap v12862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:53 smithi188 ceph-mon[92425]: pgmap v12862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:55 smithi161 ceph-mon[91579]: pgmap v12863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:55 smithi188 ceph-mon[92425]: pgmap v12863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:47:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:57 smithi161 ceph-mon[91579]: pgmap v12864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:57 smithi188 ceph-mon[92425]: pgmap v12864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:47:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:47:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:47:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:47:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:47:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:47:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:47:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:47:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:47:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:48:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:47:59 smithi161 ceph-mon[91579]: pgmap v12865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:47:59 smithi188 ceph-mon[92425]: pgmap v12865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:01 smithi161 ceph-mon[91579]: pgmap v12866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:01 smithi188 ceph-mon[92425]: pgmap v12866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:03 smithi161 ceph-mon[91579]: pgmap v12867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:03 smithi188 ceph-mon[92425]: pgmap v12867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:05 smithi161 ceph-mon[91579]: pgmap v12868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:05 smithi188 ceph-mon[92425]: pgmap v12868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:07 smithi161 ceph-mon[91579]: pgmap v12869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:07 smithi188 ceph-mon[92425]: pgmap v12869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:08 smithi161 ceph-mon[91579]: pgmap v12870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:08 smithi188 ceph-mon[92425]: pgmap v12870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:11.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:11 smithi161 ceph-mon[91579]: pgmap v12871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:11.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:11 smithi188 ceph-mon[92425]: pgmap v12871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:12.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:12.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:13 smithi161 ceph-mon[91579]: pgmap v12872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:13.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:13 smithi188 ceph-mon[92425]: pgmap v12872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:15 smithi161 ceph-mon[91579]: pgmap v12873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:15.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:15 smithi188 ceph-mon[92425]: pgmap v12873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:17 smithi161 ceph-mon[91579]: pgmap v12874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:17.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:17 smithi188 ceph-mon[92425]: pgmap v12874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:17.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:19.875 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-mon[91579]: pgmap v12875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:19.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:48:19.809+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:48:19.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:48:19.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:48:19.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:48:19.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:48:19.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:48:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:48:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:48:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:48:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:48:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:48:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:48:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:48:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:48:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:48:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:48:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:48:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:48:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:48:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:48:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:48:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:48:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:48:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:48:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:48:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:48:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:48:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:48:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:48:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:48:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:48:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:48:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:48:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:48:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:48:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:48:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:48:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:48:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:48:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:48:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:48:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:48:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:48:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:48:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:48:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=753] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:48:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=753] Local address: 172.21.15.161, port 58248 2022-07-03T17:48:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=753] Peer address: 172.21.15.101, port 22 2022-07-03T17:48:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=753] Connection lost 2022-07-03T17:48:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=753] Aborting connection 2022-07-03T17:48:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:48:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:48:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:19 smithi188 ceph-mon[92425]: pgmap v12875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:48:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:48:20.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:48:20.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:48:20.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:48:20.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:48:20.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:48:20.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:48:20.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:48:20.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:48:20.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:48:20.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:48:20.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:48:20.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:48:20.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:48:20.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:48:20.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:48:20.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:48:20.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:48:20.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:48:20.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:48:20.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:48:20.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:48:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:48:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:48:20.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:48:20.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:48:20.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:48:20.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:48:20.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:48:20.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:48:20.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:48:20.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:48:20.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:48:20.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:48:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:48:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:48:20.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:48:20.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:48:20.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:48:20.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:48:20.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:48:20.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:48:20.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:48:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:48:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:48:20.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:48:20.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:48:20.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:48:20.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=753] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:48:20.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=753] Local address: 172.21.15.161, port 58248 2022-07-03T17:48:20.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=753] Peer address: 172.21.15.101, port 22 2022-07-03T17:48:20.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=753] Connection lost 2022-07-03T17:48:20.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=753] Aborting connection 2022-07-03T17:48:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:48:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:48:20.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:48:20.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:48:20.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:48:20.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:48:20.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:48:20.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:48:20.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:48:20.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:48:20.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:48:20.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:48:20.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:48:20.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:48:20.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:48:20.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:48:20.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:48:20.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:48:20.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:48:20.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:48:20.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:48:20.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:48:20.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:48:20.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:48:20.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:48:20.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:48:20.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:48:20.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:48:20.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:48:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:48:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:48:20.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:48:20.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:48:20.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:48:20.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:48:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:48:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:48:20.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:48:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:48:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:48:20.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:48:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:48:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:48:20.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:48:20.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:48:20.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:48:20.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:48:20.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:48:20.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:48:20.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=753] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:48:20.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=753] Local address: 172.21.15.161, port 58248 2022-07-03T17:48:20.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=753] Peer address: 172.21.15.101, port 22 2022-07-03T17:48:20.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=753] Connection lost 2022-07-03T17:48:20.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=753] Aborting connection 2022-07-03T17:48:21.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:21 smithi161 ceph-mon[91579]: pgmap v12876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:21.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:21 smithi188 ceph-mon[92425]: pgmap v12876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:23.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:23 smithi188 ceph-mon[92425]: pgmap v12877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:23 smithi161 ceph-mon[91579]: pgmap v12877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:25.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:25 smithi188 ceph-mon[92425]: pgmap v12878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:25 smithi161 ceph-mon[91579]: pgmap v12878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:27 smithi188 ceph-mon[92425]: pgmap v12879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:27 smithi161 ceph-mon[91579]: pgmap v12879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:29 smithi188 ceph-mon[92425]: pgmap v12880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:29 smithi161 ceph-mon[91579]: pgmap v12880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:31 smithi188 ceph-mon[92425]: pgmap v12881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:31 smithi161 ceph-mon[91579]: pgmap v12881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:33 smithi188 ceph-mon[92425]: pgmap v12882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:33 smithi161 ceph-mon[91579]: pgmap v12882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:35 smithi188 ceph-mon[92425]: pgmap v12883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:35 smithi161 ceph-mon[91579]: pgmap v12883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:37 smithi188 ceph-mon[92425]: pgmap v12884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:37 smithi161 ceph-mon[91579]: pgmap v12884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:39 smithi188 ceph-mon[92425]: pgmap v12885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:39 smithi161 ceph-mon[91579]: pgmap v12885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:41 smithi188 ceph-mon[92425]: pgmap v12886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:41 smithi161 ceph-mon[91579]: pgmap v12886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:43 smithi188 ceph-mon[92425]: pgmap v12887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:43 smithi161 ceph-mon[91579]: pgmap v12887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:45 smithi161 ceph-mon[91579]: pgmap v12888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:45 smithi188 ceph-mon[92425]: pgmap v12888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:47 smithi161 ceph-mon[91579]: pgmap v12889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:47 smithi188 ceph-mon[92425]: pgmap v12889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:49 smithi161 ceph-mon[91579]: pgmap v12890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:49 smithi188 ceph-mon[92425]: pgmap v12890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:51 smithi161 ceph-mon[91579]: pgmap v12891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:51 smithi188 ceph-mon[92425]: pgmap v12891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:53 smithi161 ceph-mon[91579]: pgmap v12892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:53 smithi188 ceph-mon[92425]: pgmap v12892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:55 smithi161 ceph-mon[91579]: pgmap v12893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:55 smithi188 ceph-mon[92425]: pgmap v12893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:48:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:57 smithi161 ceph-mon[91579]: pgmap v12894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:57 smithi188 ceph-mon[92425]: pgmap v12894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:48:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:48:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:48:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:48:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:48:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:48:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:48:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:48:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:48:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:48:59.701 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:48:59 smithi161 ceph-mon[91579]: pgmap v12895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:48:59 smithi188 ceph-mon[92425]: pgmap v12895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:01 smithi161 ceph-mon[91579]: pgmap v12896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:01 smithi188 ceph-mon[92425]: pgmap v12896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:03 smithi161 ceph-mon[91579]: pgmap v12897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:03 smithi188 ceph-mon[92425]: pgmap v12897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:05 smithi161 ceph-mon[91579]: pgmap v12898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:05 smithi188 ceph-mon[92425]: pgmap v12898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:07 smithi161 ceph-mon[91579]: pgmap v12899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:07 smithi188 ceph-mon[92425]: pgmap v12899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:09 smithi161 ceph-mon[91579]: pgmap v12900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:09 smithi188 ceph-mon[92425]: pgmap v12900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:11 smithi161 ceph-mon[91579]: pgmap v12901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:11 smithi188 ceph-mon[92425]: pgmap v12901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:13 smithi161 ceph-mon[91579]: pgmap v12902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:13 smithi188 ceph-mon[92425]: pgmap v12902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:15 smithi161 ceph-mon[91579]: pgmap v12903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:15 smithi188 ceph-mon[92425]: pgmap v12903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:17 smithi161 ceph-mon[91579]: pgmap v12904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:17 smithi188 ceph-mon[92425]: pgmap v12904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:19 smithi161 ceph-mon[91579]: pgmap v12905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:19 smithi188 ceph-mon[92425]: pgmap v12905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:20.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:49:20.301+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:49:20.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:49:20.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:49:20.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:49:20.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:49:20.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:49:20.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:49:20.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:49:20.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:49:20.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:49:20.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:49:20.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:49:20.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:49:20.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:49:20.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:49:20.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:49:20.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:49:20.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:49:20.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:49:20.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:49:20.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:49:20.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:49:20.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:49:20.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:49:20.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:49:20.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:49:20.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:49:20.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:49:20.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:49:20.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:49:20.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:49:20.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:49:20.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:49:20.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:49:20.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:49:20.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:49:20.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:49:20.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:49:20.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:49:20.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:49:20.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:49:20.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:49:20.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:49:20.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:49:20.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:49:20.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=755] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:49:20.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=755] Local address: 172.21.15.161, port 58276 2022-07-03T17:49:20.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=755] Peer address: 172.21.15.101, port 22 2022-07-03T17:49:20.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=755] Connection lost 2022-07-03T17:49:20.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=755] Aborting connection 2022-07-03T17:49:20.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:49:21.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:49:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:49:21.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:49:21.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:49:21.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:49:21.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:49:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:49:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:49:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:49:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:49:22.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:49:22.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:49:22.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:49:22.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:49:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:49:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:49:22.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:49:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:49:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:49:22.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:49:22.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:49:22.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:49:22.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:49:22.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:49:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:49:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:49:22.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:49:22.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:49:22.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:49:22.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:49:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:49:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:49:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:49:22.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:49:22.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:49:22.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:49:22.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:49:22.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:49:22.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:49:22.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:49:22.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:49:22.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:49:22.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:49:22.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:49:22.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:49:22.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:49:22.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:49:22.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:49:22.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:49:22.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:49:22.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=755] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:49:22.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=755] Local address: 172.21.15.161, port 58276 2022-07-03T17:49:22.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=755] Peer address: 172.21.15.101, port 22 2022-07-03T17:49:22.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=755] Connection lost 2022-07-03T17:49:22.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=755] Aborting connection 2022-07-03T17:49:22.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:21 smithi161 ceph-mon[91579]: pgmap v12906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:49:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:49:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:49:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:49:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:49:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:49:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:49:22.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:49:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:49:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:49:22.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:49:22.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:49:22.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:49:22.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:49:22.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:49:22.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:49:22.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:49:22.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:49:22.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:49:22.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:49:22.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:49:22.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:49:22.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:49:22.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:49:22.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:49:22.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:49:22.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:49:22.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:49:22.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:49:22.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:49:22.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:49:22.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:49:22.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:49:22.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:49:22.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:49:22.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:49:22.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:49:22.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:49:22.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:49:22.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:49:22.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:49:22.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:49:22.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:49:22.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:49:22.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:49:22.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:49:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=755] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:49:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=755] Local address: 172.21.15.161, port 58276 2022-07-03T17:49:22.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=755] Peer address: 172.21.15.101, port 22 2022-07-03T17:49:22.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=755] Connection lost 2022-07-03T17:49:22.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=755] Aborting connection 2022-07-03T17:49:22.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:21 smithi188 ceph-mon[92425]: pgmap v12906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:23.012 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:23.088 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: pgmap v12907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: pgmap v12907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:49:24.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:24.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:49:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:24 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 51264k 2022-07-03T17:49:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:24 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 52495130: error parsing value: Value '52495130' is below minimum 939524096 2022-07-03T17:49:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:24 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 169.0M 2022-07-03T17:49:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:24 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 177300414: error parsing value: Value '177300414' is below minimum 939524096 2022-07-03T17:49:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:24 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 51264k 2022-07-03T17:49:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:24 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 52495130: error parsing value: Value '52495130' is below minimum 939524096 2022-07-03T17:49:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:24 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 169.0M 2022-07-03T17:49:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:24 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 177300414: error parsing value: Value '177300414' is below minimum 939524096 2022-07-03T17:49:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:25 smithi161 ceph-mon[91579]: pgmap v12908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:25 smithi188 ceph-mon[92425]: pgmap v12908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:27 smithi161 ceph-mon[91579]: pgmap v12909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:27 smithi188 ceph-mon[92425]: pgmap v12909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:29 smithi161 ceph-mon[91579]: pgmap v12910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:29 smithi188 ceph-mon[92425]: pgmap v12910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:31 smithi161 ceph-mon[91579]: pgmap v12911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:31 smithi188 ceph-mon[92425]: pgmap v12911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:33 smithi161 ceph-mon[91579]: pgmap v12912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:33 smithi188 ceph-mon[92425]: pgmap v12912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:35 smithi161 ceph-mon[91579]: pgmap v12913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:35 smithi188 ceph-mon[92425]: pgmap v12913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:37 smithi161 ceph-mon[91579]: pgmap v12914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:37 smithi188 ceph-mon[92425]: pgmap v12914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:39 smithi161 ceph-mon[91579]: pgmap v12915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:39 smithi188 ceph-mon[92425]: pgmap v12915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:41 smithi161 ceph-mon[91579]: pgmap v12916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:41 smithi188 ceph-mon[92425]: pgmap v12916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:43 smithi161 ceph-mon[91579]: pgmap v12917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:43 smithi188 ceph-mon[92425]: pgmap v12917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:45 smithi161 ceph-mon[91579]: pgmap v12918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:45 smithi188 ceph-mon[92425]: pgmap v12918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:47 smithi161 ceph-mon[91579]: pgmap v12919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:47 smithi188 ceph-mon[92425]: pgmap v12919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:49 smithi161 ceph-mon[91579]: pgmap v12920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:49 smithi188 ceph-mon[92425]: pgmap v12920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:51 smithi161 ceph-mon[91579]: pgmap v12921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:51 smithi188 ceph-mon[92425]: pgmap v12921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:53 smithi161 ceph-mon[91579]: pgmap v12922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:53 smithi188 ceph-mon[92425]: pgmap v12922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:55 smithi161 ceph-mon[91579]: pgmap v12923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:55 smithi188 ceph-mon[92425]: pgmap v12923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:49:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:57 smithi161 ceph-mon[91579]: pgmap v12924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:57 smithi188 ceph-mon[92425]: pgmap v12924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:49:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:49:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:49:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:49:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:49:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:49:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:49:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:49:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:49:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:50:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:49:59 smithi161 ceph-mon[91579]: pgmap v12925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:49:59 smithi188 ceph-mon[92425]: pgmap v12925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:50:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T17:50:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:01 smithi161 ceph-mon[91579]: pgmap v12926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:01 smithi188 ceph-mon[92425]: pgmap v12926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:03 smithi161 ceph-mon[91579]: pgmap v12927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:03 smithi188 ceph-mon[92425]: pgmap v12927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:06.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:05 smithi161 ceph-mon[91579]: pgmap v12928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:05 smithi188 ceph-mon[92425]: pgmap v12928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:07 smithi161 ceph-mon[91579]: pgmap v12929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:07 smithi188 ceph-mon[92425]: pgmap v12929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:09 smithi161 ceph-mon[91579]: pgmap v12930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:09 smithi188 ceph-mon[92425]: pgmap v12930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:11 smithi161 ceph-mon[91579]: pgmap v12931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:11 smithi188 ceph-mon[92425]: pgmap v12931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:13 smithi161 ceph-mon[91579]: pgmap v12932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:13 smithi188 ceph-mon[92425]: pgmap v12932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:15 smithi161 ceph-mon[91579]: pgmap v12933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:15 smithi188 ceph-mon[92425]: pgmap v12933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:17 smithi161 ceph-mon[91579]: pgmap v12934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:17 smithi188 ceph-mon[92425]: pgmap v12934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:18.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:19 smithi161 ceph-mon[91579]: pgmap v12935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:19 smithi188 ceph-mon[92425]: pgmap v12935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:21 smithi161 ceph-mon[91579]: pgmap v12936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:21 smithi188 ceph-mon[92425]: pgmap v12936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:23.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:50:23.565+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:50:23.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:50:23.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:50:23.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:50:23.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:50:23.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:50:23.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:50:23.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:50:23.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:50:23.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:50:23.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:50:23.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:50:23.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:50:23.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:50:23.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:50:23.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:50:23.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:50:23.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:50:23.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:50:23.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:50:23.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:50:23.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:50:23.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:50:23.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:50:23.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:50:23.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:50:23.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:50:23.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:50:23.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:50:23.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:50:23.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:50:23.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:50:23.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:50:23.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:50:23.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:50:23.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:50:23.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:50:23.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:50:23.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:50:23.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:50:23.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:50:23.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:50:23.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:50:23.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:50:23.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:50:23.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=757] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:50:23.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=757] Local address: 172.21.15.161, port 58306 2022-07-03T17:50:23.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=757] Peer address: 172.21.15.101, port 22 2022-07-03T17:50:23.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=757] Connection lost 2022-07-03T17:50:23.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=757] Aborting connection 2022-07-03T17:50:23.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:50:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-mon[91579]: pgmap v12937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:50:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:50:24.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:50:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:23 smithi188 ceph-mon[92425]: pgmap v12937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:50:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:50:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:50:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:50:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:50:25.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:50:25.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:50:25.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:50:25.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:50:25.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:50:25.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:50:25.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:50:25.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:50:25.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:50:25.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:50:25.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:50:25.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:50:25.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:50:25.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:50:25.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:50:25.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:50:25.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:50:25.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:50:25.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:50:25.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:50:25.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:50:25.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:50:25.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:50:25.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:50:25.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:50:25.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:50:25.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:50:25.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:50:25.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:50:25.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:50:25.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:50:25.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:50:25.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:50:25.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:50:25.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:50:25.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:50:25.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:50:25.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:50:25.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:50:25.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:50:25.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:50:25.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:50:25.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:50:25.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:50:25.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=757] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:50:25.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=757] Local address: 172.21.15.161, port 58306 2022-07-03T17:50:25.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=757] Peer address: 172.21.15.101, port 22 2022-07-03T17:50:25.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=757] Connection lost 2022-07-03T17:50:25.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=757] Aborting connection 2022-07-03T17:50:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:50:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:50:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:50:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:50:25.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:50:25.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:50:25.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:50:25.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:50:25.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:50:25.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:50:25.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:50:25.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:50:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:50:25.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:50:25.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:50:25.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:50:25.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:50:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:50:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:50:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:50:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:50:25.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:50:25.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:50:25.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:50:25.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:50:25.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:50:25.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:50:25.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:50:25.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:50:25.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:50:25.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:50:25.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:50:25.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:50:25.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:50:25.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:50:25.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:50:25.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:50:25.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:50:25.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:50:25.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:50:25.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:50:25.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:50:25.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:50:25.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:50:25.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:50:25.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:50:25.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=757] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:50:25.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=757] Local address: 172.21.15.161, port 58306 2022-07-03T17:50:25.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=757] Peer address: 172.21.15.101, port 22 2022-07-03T17:50:25.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=757] Connection lost 2022-07-03T17:50:25.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=757] Aborting connection 2022-07-03T17:50:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:25 smithi161 ceph-mon[91579]: pgmap v12938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:25 smithi188 ceph-mon[92425]: pgmap v12938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:27 smithi161 ceph-mon[91579]: pgmap v12939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:27 smithi188 ceph-mon[92425]: pgmap v12939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:29 smithi161 ceph-mon[91579]: pgmap v12940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:29 smithi188 ceph-mon[92425]: pgmap v12940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:31 smithi161 ceph-mon[91579]: pgmap v12941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:31 smithi188 ceph-mon[92425]: pgmap v12941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:33 smithi188 ceph-mon[92425]: pgmap v12942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:33 smithi161 ceph-mon[91579]: pgmap v12942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:34 smithi161 ceph-mon[91579]: pgmap v12943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:34 smithi188 ceph-mon[92425]: pgmap v12943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:37 smithi188 ceph-mon[92425]: pgmap v12944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:37 smithi161 ceph-mon[91579]: pgmap v12944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:39 smithi188 ceph-mon[92425]: pgmap v12945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:39 smithi161 ceph-mon[91579]: pgmap v12945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:41 smithi188 ceph-mon[92425]: pgmap v12946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:41 smithi161 ceph-mon[91579]: pgmap v12946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:43.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:43 smithi188 ceph-mon[92425]: pgmap v12947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:43 smithi161 ceph-mon[91579]: pgmap v12947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:45 smithi188 ceph-mon[92425]: pgmap v12948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:45 smithi161 ceph-mon[91579]: pgmap v12948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:47 smithi161 ceph-mon[91579]: pgmap v12949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:48.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:47 smithi188 ceph-mon[92425]: pgmap v12949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:49 smithi161 ceph-mon[91579]: pgmap v12950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:49 smithi188 ceph-mon[92425]: pgmap v12950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:51 smithi161 ceph-mon[91579]: pgmap v12951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:51 smithi188 ceph-mon[92425]: pgmap v12951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:53 smithi161 ceph-mon[91579]: pgmap v12952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:53 smithi188 ceph-mon[92425]: pgmap v12952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:55 smithi161 ceph-mon[91579]: pgmap v12953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:55 smithi188 ceph-mon[92425]: pgmap v12953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:50:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:57 smithi161 ceph-mon[91579]: pgmap v12954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:57 smithi188 ceph-mon[92425]: pgmap v12954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:50:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:50:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:50:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:50:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:50:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:50:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:50:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:51:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:50:59 smithi161 ceph-mon[91579]: pgmap v12955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:50:59 smithi188 ceph-mon[92425]: pgmap v12955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:01 smithi161 ceph-mon[91579]: pgmap v12956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:01 smithi188 ceph-mon[92425]: pgmap v12956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:03 smithi161 ceph-mon[91579]: pgmap v12957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:03 smithi188 ceph-mon[92425]: pgmap v12957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:05 smithi161 ceph-mon[91579]: pgmap v12958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:05 smithi188 ceph-mon[92425]: pgmap v12958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:07 smithi161 ceph-mon[91579]: pgmap v12959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:07 smithi188 ceph-mon[92425]: pgmap v12959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:09 smithi161 ceph-mon[91579]: pgmap v12960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:09 smithi188 ceph-mon[92425]: pgmap v12960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:12.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:11 smithi161 ceph-mon[91579]: pgmap v12961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:11 smithi188 ceph-mon[92425]: pgmap v12961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:13 smithi161 ceph-mon[91579]: pgmap v12962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:13 smithi188 ceph-mon[92425]: pgmap v12962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:15 smithi161 ceph-mon[91579]: pgmap v12963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:15 smithi188 ceph-mon[92425]: pgmap v12963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:18.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:17 smithi161 ceph-mon[91579]: pgmap v12964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:18.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:17 smithi188 ceph-mon[92425]: pgmap v12964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:19 smithi161 ceph-mon[91579]: pgmap v12965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:19 smithi188 ceph-mon[92425]: pgmap v12965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:21 smithi161 ceph-mon[91579]: pgmap v12966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:22.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:21 smithi188 ceph-mon[92425]: pgmap v12966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:23.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:24.076 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:23 smithi161 ceph-mon[91579]: pgmap v12967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:24.086 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:23 smithi188 ceph-mon[92425]: pgmap v12967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:24.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:51:24.073+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:51:24.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:51:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:51:24.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:51:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:51:24.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:51:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:51:24.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:51:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:51:24.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:51:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:51:24.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:51:24.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:51:24.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:51:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:51:24.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:51:24.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:51:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:51:24.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:51:24.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:51:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:51:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:51:24.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:51:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:51:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:51:24.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:51:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:51:24.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:51:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:51:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:51:24.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:51:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:51:24.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:51:24.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:51:24.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:51:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:51:24.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:51:24.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:51:24.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:51:24.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:51:24.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:51:24.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=759] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:51:24.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=759] Local address: 172.21.15.161, port 58334 2022-07-03T17:51:24.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=759] Peer address: 172.21.15.101, port 22 2022-07-03T17:51:24.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=759] Connection lost 2022-07-03T17:51:24.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=759] Aborting connection 2022-07-03T17:51:24.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:51:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:51:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:51:25.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:51:25.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:51:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:51:25.177 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:51:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:51:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:51:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:51:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:51:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:51:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:51:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:51:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:51:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:51:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:51:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:51:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:51:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:51:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:51:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:51:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:51:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:51:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:51:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:51:26.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:51:26.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:51:26.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:51:26.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:51:26.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:51:26.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:51:26.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:51:26.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:51:26.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:51:26.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:51:26.165 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:51:26.166 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:51:26.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:51:26.167 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:51:26.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:51:26.168 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:51:26.171 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:51:26.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:51:26.172 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:51:26.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:51:26.173 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:51:26.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:51:26.174 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:51:26.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:51:26.175 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:51:26.178 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:51:26.179 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:51:26.189 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=759] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:51:26.190 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=759] Local address: 172.21.15.161, port 58334 2022-07-03T17:51:26.190 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=759] Peer address: 172.21.15.101, port 22 2022-07-03T17:51:26.191 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=759] Connection lost 2022-07-03T17:51:26.192 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=759] Aborting connection 2022-07-03T17:51:26.192 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:25 smithi161 ceph-mon[91579]: pgmap v12968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:26.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:51:26.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:51:26.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:51:26.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:51:26.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:51:26.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:51:26.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:51:26.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:51:26.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:51:26.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:51:26.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:51:26.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:51:26.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:51:26.200 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:51:26.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:51:26.201 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:51:26.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:51:26.203 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:51:26.204 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:51:26.205 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:51:26.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:51:26.206 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:51:26.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:51:26.207 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:51:26.214 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:51:26.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:51:26.216 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:51:26.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:51:26.217 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:51:26.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:51:26.218 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:51:26.219 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:51:26.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:51:26.220 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:51:26.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:51:26.221 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:51:26.222 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:51:26.223 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:51:26.224 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:51:26.225 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:51:26.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:51:26.226 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:51:26.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:51:26.227 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:51:26.228 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:51:26.229 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:51:26.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=759] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:51:26.230 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=759] Local address: 172.21.15.161, port 58334 2022-07-03T17:51:26.231 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=759] Peer address: 172.21.15.101, port 22 2022-07-03T17:51:26.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=759] Connection lost 2022-07-03T17:51:26.232 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=759] Aborting connection 2022-07-03T17:51:26.233 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:25 smithi188 ceph-mon[92425]: pgmap v12968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:27 smithi161 ceph-mon[91579]: pgmap v12969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:27 smithi188 ceph-mon[92425]: pgmap v12969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:29 smithi161 ceph-mon[91579]: pgmap v12970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:29 smithi188 ceph-mon[92425]: pgmap v12970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:31 smithi161 ceph-mon[91579]: pgmap v12971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:31 smithi188 ceph-mon[92425]: pgmap v12971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:33 smithi161 ceph-mon[91579]: pgmap v12972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:33 smithi188 ceph-mon[92425]: pgmap v12972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:35 smithi161 ceph-mon[91579]: pgmap v12973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:35 smithi188 ceph-mon[92425]: pgmap v12973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:37 smithi161 ceph-mon[91579]: pgmap v12974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:38.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:37 smithi188 ceph-mon[92425]: pgmap v12974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:38.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:39 smithi161 ceph-mon[91579]: pgmap v12975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:39 smithi188 ceph-mon[92425]: pgmap v12975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:41 smithi161 ceph-mon[91579]: pgmap v12976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:41 smithi188 ceph-mon[92425]: pgmap v12976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:43 smithi161 ceph-mon[91579]: pgmap v12977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:43 smithi188 ceph-mon[92425]: pgmap v12977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:45 smithi161 ceph-mon[91579]: pgmap v12978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:45 smithi188 ceph-mon[92425]: pgmap v12978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:47 smithi161 ceph-mon[91579]: pgmap v12979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:47 smithi188 ceph-mon[92425]: pgmap v12979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:50.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:49 smithi161 ceph-mon[91579]: pgmap v12980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:49 smithi188 ceph-mon[92425]: pgmap v12980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:51 smithi161 ceph-mon[91579]: pgmap v12981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:52.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:51 smithi188 ceph-mon[92425]: pgmap v12981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:53 smithi161 ceph-mon[91579]: pgmap v12982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:53 smithi188 ceph-mon[92425]: pgmap v12982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:55 smithi161 ceph-mon[91579]: pgmap v12983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:55 smithi188 ceph-mon[92425]: pgmap v12983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:51:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:57 smithi161 ceph-mon[91579]: pgmap v12984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:57 smithi188 ceph-mon[92425]: pgmap v12984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:51:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:51:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:51:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:51:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:51:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:51:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:51:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:51:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:51:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:52:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:51:59 smithi161 ceph-mon[91579]: pgmap v12985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:51:59 smithi188 ceph-mon[92425]: pgmap v12985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:01 smithi161 ceph-mon[91579]: pgmap v12986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:01 smithi188 ceph-mon[92425]: pgmap v12986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:03.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:03.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:03 smithi161 ceph-mon[91579]: pgmap v12987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:03 smithi188 ceph-mon[92425]: pgmap v12987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:05 smithi161 ceph-mon[91579]: pgmap v12988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:05 smithi188 ceph-mon[92425]: pgmap v12988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:07 smithi161 ceph-mon[91579]: pgmap v12989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:07 smithi188 ceph-mon[92425]: pgmap v12989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:09 smithi161 ceph-mon[91579]: pgmap v12990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:09 smithi188 ceph-mon[92425]: pgmap v12990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:11 smithi161 ceph-mon[91579]: pgmap v12991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:11 smithi188 ceph-mon[92425]: pgmap v12991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:13 smithi161 ceph-mon[91579]: pgmap v12992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:13 smithi188 ceph-mon[92425]: pgmap v12992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:15 smithi161 ceph-mon[91579]: pgmap v12993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:15 smithi188 ceph-mon[92425]: pgmap v12993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:17 smithi161 ceph-mon[91579]: pgmap v12994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:18.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:17 smithi188 ceph-mon[92425]: pgmap v12994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:19 smithi161 ceph-mon[91579]: pgmap v12995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:19 smithi188 ceph-mon[92425]: pgmap v12995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:22.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:21 smithi161 ceph-mon[91579]: pgmap v12996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:22.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:21 smithi188 ceph-mon[92425]: pgmap v12996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:23 smithi161 ceph-mon[91579]: pgmap v12997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:23 smithi188 ceph-mon[92425]: pgmap v12997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:24.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:52:24.573+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:52:24.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:52:24.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:52:24.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:52:24.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:52:24.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:52:24.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:52:24.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:52:24.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:52:24.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:52:24.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:52:24.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:52:24.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:52:24.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:52:24.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:52:24.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:52:24.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:52:24.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:52:24.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:52:24.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:52:24.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:52:24.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:52:24.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:52:24.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:52:24.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:52:24.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:52:24.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:52:24.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:52:24.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:52:24.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:52:24.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:52:24.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:52:24.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:52:24.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:52:24.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:52:24.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:52:24.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:52:24.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:52:24.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:52:24.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:52:24.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:52:24.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:52:24.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:52:24.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:52:24.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:52:24.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=761] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:52:24.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=761] Local address: 172.21.15.161, port 58362 2022-07-03T17:52:24.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=761] Peer address: 172.21.15.101, port 22 2022-07-03T17:52:24.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=761] Connection lost 2022-07-03T17:52:24.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=761] Aborting connection 2022-07-03T17:52:24.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:52:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:52:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:52:25.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:52:25.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:52:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:52:25.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:52:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:52:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:52:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:52:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:52:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:52:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:52:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:52:26.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:52:26.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:52:26.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:52:26.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:52:26.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:52:26.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:52:26.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:52:26.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:52:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:52:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:52:26.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:52:26.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:52:26.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:52:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:52:26.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:52:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:52:26.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:52:26.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=761] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:52:26.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=761] Local address: 172.21.15.161, port 58362 2022-07-03T17:52:26.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=761] Peer address: 172.21.15.101, port 22 2022-07-03T17:52:26.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=761] Connection lost 2022-07-03T17:52:26.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=761] Aborting connection 2022-07-03T17:52:26.151 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:25 smithi161 ceph-mon[91579]: pgmap v12998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:52:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:52:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:52:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:52:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:52:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:52:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:52:26.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:52:26.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:52:26.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:52:26.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:52:26.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:52:26.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:52:26.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:52:26.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:52:26.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:52:26.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:52:26.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:52:26.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:52:26.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:52:26.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:52:26.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:52:26.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:52:26.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:52:26.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:52:26.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:52:26.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:52:26.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:52:26.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:52:26.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:52:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:52:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:52:26.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:52:26.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:52:26.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:52:26.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:52:26.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:52:26.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:52:26.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:52:26.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:52:26.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:52:26.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:52:26.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:52:26.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:52:26.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:52:26.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:52:26.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=761] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:52:26.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=761] Local address: 172.21.15.161, port 58362 2022-07-03T17:52:26.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=761] Peer address: 172.21.15.101, port 22 2022-07-03T17:52:26.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=761] Connection lost 2022-07-03T17:52:26.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=761] Aborting connection 2022-07-03T17:52:26.193 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:25 smithi188 ceph-mon[92425]: pgmap v12998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:27 smithi161 ceph-mon[91579]: pgmap v12999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:27 smithi188 ceph-mon[92425]: pgmap v12999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:29 smithi161 ceph-mon[91579]: pgmap v13000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:29 smithi188 ceph-mon[92425]: pgmap v13000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:31 smithi161 ceph-mon[91579]: pgmap v13001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:31 smithi188 ceph-mon[92425]: pgmap v13001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:34.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:33 smithi161 ceph-mon[91579]: pgmap v13002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:33 smithi188 ceph-mon[92425]: pgmap v13002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:35 smithi161 ceph-mon[91579]: pgmap v13003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:35 smithi188 ceph-mon[92425]: pgmap v13003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:37 smithi161 ceph-mon[91579]: pgmap v13004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:37 smithi188 ceph-mon[92425]: pgmap v13004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T17:52:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:40.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:39 smithi161 ceph-mon[91579]: pgmap v13005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:39 smithi188 ceph-mon[92425]: pgmap v13005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:41 smithi161 ceph-mon[91579]: pgmap v13006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:41 smithi188 ceph-mon[92425]: pgmap v13006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:43 smithi188 ceph-mon[92425]: pgmap v13007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:43 smithi161 ceph-mon[91579]: pgmap v13007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:45 smithi188 ceph-mon[92425]: pgmap v13008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:45 smithi161 ceph-mon[91579]: pgmap v13008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:47 smithi188 ceph-mon[92425]: pgmap v13009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:48.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:48.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:47 smithi161 ceph-mon[91579]: pgmap v13009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:48.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:50.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:49 smithi188 ceph-mon[92425]: pgmap v13010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:49 smithi161 ceph-mon[91579]: pgmap v13010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:51 smithi188 ceph-mon[92425]: pgmap v13011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:51 smithi161 ceph-mon[91579]: pgmap v13011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:53.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:53 smithi188 ceph-mon[92425]: pgmap v13012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:53 smithi161 ceph-mon[91579]: pgmap v13012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:55 smithi188 ceph-mon[92425]: pgmap v13013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:55 smithi161 ceph-mon[91579]: pgmap v13013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:52:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:57 smithi188 ceph-mon[92425]: pgmap v13014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:57 smithi161 ceph-mon[91579]: pgmap v13014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:52:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:52:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:52:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:52:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:52:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:52:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:52:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:52:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:52:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:53:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:52:59 smithi188 ceph-mon[92425]: pgmap v13015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:52:59 smithi161 ceph-mon[91579]: pgmap v13015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:00 smithi161 ceph-mon[91579]: pgmap v13016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:00 smithi188 ceph-mon[92425]: pgmap v13016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:02.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:03.889 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:03 smithi161 ceph-mon[91579]: pgmap v13017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:03 smithi188 ceph-mon[92425]: pgmap v13017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:05 smithi161 ceph-mon[91579]: pgmap v13018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:05 smithi188 ceph-mon[92425]: pgmap v13018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:07 smithi161 ceph-mon[91579]: pgmap v13019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:07 smithi188 ceph-mon[92425]: pgmap v13019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:09 smithi161 ceph-mon[91579]: pgmap v13020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:09 smithi188 ceph-mon[92425]: pgmap v13020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:11 smithi161 ceph-mon[91579]: pgmap v13021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:11 smithi188 ceph-mon[92425]: pgmap v13021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:13 smithi161 ceph-mon[91579]: pgmap v13022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:13 smithi188 ceph-mon[92425]: pgmap v13022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:15 smithi161 ceph-mon[91579]: pgmap v13023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:15 smithi188 ceph-mon[92425]: pgmap v13023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:17 smithi161 ceph-mon[91579]: pgmap v13024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:18.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:17 smithi188 ceph-mon[92425]: pgmap v13024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:20.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:19 smithi161 ceph-mon[91579]: pgmap v13025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:19 smithi188 ceph-mon[92425]: pgmap v13025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:21 smithi161 ceph-mon[91579]: pgmap v13026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:21 smithi188 ceph-mon[92425]: pgmap v13026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:24.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:23 smithi161 ceph-mon[91579]: pgmap v13027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:23 smithi188 ceph-mon[92425]: pgmap v13027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:25.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:53:25.077+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:53:25.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:53:25.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:53:25.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:53:25.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:53:25.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:53:25.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:53:25.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:53:25.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:53:25.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:53:25.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:53:25.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:53:25.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:53:25.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:53:25.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:53:25.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:53:25.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:53:25.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:53:25.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:53:25.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:53:25.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:53:25.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:53:25.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:53:25.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:53:25.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:53:25.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:53:25.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:53:25.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:53:25.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:53:25.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:53:25.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:53:25.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:53:25.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:53:25.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:53:25.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:53:25.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:53:25.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:53:25.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:53:25.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:53:25.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:53:25.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:53:25.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:53:25.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:53:25.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:53:25.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:53:25.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=763] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:53:25.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=763] Local address: 172.21.15.161, port 58390 2022-07-03T17:53:25.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=763] Peer address: 172.21.15.101, port 22 2022-07-03T17:53:25.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=763] Connection lost 2022-07-03T17:53:25.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=763] Aborting connection 2022-07-03T17:53:25.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:53:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-mon[91579]: pgmap v13028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:53:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:53:26.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:53:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:25 smithi188 ceph-mon[92425]: pgmap v13028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:53:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:53:26.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:53:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:53:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:53:27.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:53:27.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:53:27.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:53:27.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:53:27.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:53:27.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:53:27.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:53:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:53:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:53:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:53:27.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:53:27.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:53:27.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:53:27.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:53:27.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:53:27.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:53:27.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:53:27.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:53:27.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:53:27.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:53:27.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:53:27.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:53:27.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:53:27.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:53:27.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:53:27.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:53:27.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:53:27.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:53:27.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:53:27.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:53:27.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:53:27.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:53:27.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:53:27.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:53:27.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:53:27.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:53:27.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:53:27.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:53:27.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:53:27.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:53:27.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:53:27.158 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:53:27.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:53:27.159 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:53:27.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=763] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:53:27.160 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=763] Local address: 172.21.15.161, port 58390 2022-07-03T17:53:27.161 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=763] Peer address: 172.21.15.101, port 22 2022-07-03T17:53:27.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=763] Connection lost 2022-07-03T17:53:27.162 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=763] Aborting connection 2022-07-03T17:53:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:53:27.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:53:27.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:53:27.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:53:27.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:53:27.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:53:27.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:53:27.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:53:27.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:53:27.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:53:27.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:53:27.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:53:27.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:53:27.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:53:27.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:53:27.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:53:27.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:53:27.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:53:27.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:53:27.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:53:27.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:53:27.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:53:27.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:53:27.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:53:27.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:53:27.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:53:27.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:53:27.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:53:27.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:53:27.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:53:27.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:53:27.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:53:27.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:53:27.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:53:27.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:53:27.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:53:27.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:53:27.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:53:27.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:53:27.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:53:27.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:53:27.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:53:27.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:53:27.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:53:27.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:53:27.195 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:53:27.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=763] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:53:27.196 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=763] Local address: 172.21.15.161, port 58390 2022-07-03T17:53:27.197 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=763] Peer address: 172.21.15.101, port 22 2022-07-03T17:53:27.198 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=763] Connection lost 2022-07-03T17:53:27.199 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=763] Aborting connection 2022-07-03T17:53:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:27 smithi161 ceph-mon[91579]: pgmap v13029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:27 smithi188 ceph-mon[92425]: pgmap v13029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:29 smithi161 ceph-mon[91579]: pgmap v13030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:29 smithi188 ceph-mon[92425]: pgmap v13030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:31 smithi161 ceph-mon[91579]: pgmap v13031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:32.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:31 smithi188 ceph-mon[92425]: pgmap v13031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:33 smithi161 ceph-mon[91579]: pgmap v13032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:33 smithi188 ceph-mon[92425]: pgmap v13032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:35 smithi161 ceph-mon[91579]: pgmap v13033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:35 smithi188 ceph-mon[92425]: pgmap v13033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:37 smithi161 ceph-mon[91579]: pgmap v13034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:37 smithi188 ceph-mon[92425]: pgmap v13034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:38.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:39 smithi161 ceph-mon[91579]: pgmap v13035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:39 smithi188 ceph-mon[92425]: pgmap v13035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:41 smithi161 ceph-mon[91579]: pgmap v13036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:41 smithi188 ceph-mon[92425]: pgmap v13036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:43 smithi161 ceph-mon[91579]: pgmap v13037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:43 smithi188 ceph-mon[92425]: pgmap v13037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:45 smithi161 ceph-mon[91579]: pgmap v13038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:45 smithi188 ceph-mon[92425]: pgmap v13038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:47 smithi161 ceph-mon[91579]: pgmap v13039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:47 smithi188 ceph-mon[92425]: pgmap v13039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:49 smithi161 ceph-mon[91579]: pgmap v13040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:49 smithi188 ceph-mon[92425]: pgmap v13040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:51 smithi161 ceph-mon[91579]: pgmap v13041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:51 smithi188 ceph-mon[92425]: pgmap v13041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:53 smithi161 ceph-mon[91579]: pgmap v13042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:53 smithi188 ceph-mon[92425]: pgmap v13042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:55 smithi161 ceph-mon[91579]: pgmap v13043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:55 smithi188 ceph-mon[92425]: pgmap v13043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:53:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:57 smithi161 ceph-mon[91579]: pgmap v13044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:57 smithi188 ceph-mon[92425]: pgmap v13044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:53:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:53:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:53:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:53:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:53:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:53:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:53:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:53:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:53:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:54:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:53:59 smithi161 ceph-mon[91579]: pgmap v13045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:53:59 smithi188 ceph-mon[92425]: pgmap v13045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:01 smithi161 ceph-mon[91579]: pgmap v13046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:01 smithi188 ceph-mon[92425]: pgmap v13046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:03.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:03 smithi161 ceph-mon[91579]: pgmap v13047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:03 smithi188 ceph-mon[92425]: pgmap v13047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:05 smithi161 ceph-mon[91579]: pgmap v13048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:05 smithi188 ceph-mon[92425]: pgmap v13048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:07 smithi161 ceph-mon[91579]: pgmap v13049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:07 smithi188 ceph-mon[92425]: pgmap v13049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:09 smithi161 ceph-mon[91579]: pgmap v13050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:09 smithi188 ceph-mon[92425]: pgmap v13050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:11 smithi161 ceph-mon[91579]: pgmap v13051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:11 smithi188 ceph-mon[92425]: pgmap v13051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:13 smithi161 ceph-mon[91579]: pgmap v13052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:13 smithi188 ceph-mon[92425]: pgmap v13052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:15 smithi161 ceph-mon[91579]: pgmap v13053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:15 smithi188 ceph-mon[92425]: pgmap v13053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:17 smithi161 ceph-mon[91579]: pgmap v13054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:17 smithi188 ceph-mon[92425]: pgmap v13054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:19 smithi161 ceph-mon[91579]: pgmap v13055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:19 smithi188 ceph-mon[92425]: pgmap v13055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:21 smithi161 ceph-mon[91579]: pgmap v13056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:21 smithi188 ceph-mon[92425]: pgmap v13056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:23 smithi161 ceph-mon[91579]: pgmap v13057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:23 smithi188 ceph-mon[92425]: pgmap v13057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:25.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:54:25.574+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:54:25.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:54:25.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:54:25.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:54:25.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:54:25.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:54:25.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:54:25.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:54:25.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:54:25.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:54:25.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:54:25.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:54:25.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:54:25.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:54:25.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:54:25.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:54:25.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:54:25.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:54:25.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:54:25.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:54:25.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:54:25.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:54:25.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:54:25.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:54:25.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:54:25.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:54:25.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:54:25.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:54:25.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:54:25.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:54:25.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:54:25.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:54:25.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:54:25.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:54:25.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:54:25.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:54:25.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:54:25.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:54:25.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:54:25.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:54:25.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:54:25.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:54:25.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:54:25.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:54:25.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:54:25.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=765] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:54:25.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=765] Local address: 172.21.15.161, port 58442 2022-07-03T17:54:25.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=765] Peer address: 172.21.15.101, port 22 2022-07-03T17:54:25.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=765] Connection lost 2022-07-03T17:54:25.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=765] Aborting connection 2022-07-03T17:54:25.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:54:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-mon[91579]: pgmap v13058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:54:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:54:26.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:54:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:25 smithi188 ceph-mon[92425]: pgmap v13058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:54:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:54:26.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:54:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:54:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:54:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:54:27.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:54:27.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:54:27.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:54:27.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:54:27.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:54:27.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:54:27.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:54:27.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:54:27.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:54:27.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:54:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:54:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:54:27.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:54:27.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:54:27.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:54:27.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:54:27.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:54:27.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:54:27.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:54:27.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:54:27.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:54:27.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:54:27.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:54:27.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:54:27.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:54:27.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:54:27.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:54:27.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:54:27.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:54:27.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:54:27.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:54:27.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:54:27.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:54:27.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:54:27.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:54:27.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:54:27.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:54:27.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:54:27.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:54:27.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:54:27.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:54:27.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:54:27.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:54:27.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=765] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:54:27.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=765] Local address: 172.21.15.161, port 58442 2022-07-03T17:54:27.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=765] Peer address: 172.21.15.101, port 22 2022-07-03T17:54:27.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=765] Connection lost 2022-07-03T17:54:27.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=765] Aborting connection 2022-07-03T17:54:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:54:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:54:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:54:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:54:27.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:54:27.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:54:27.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:54:27.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:54:27.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:54:27.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:54:27.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:54:27.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:54:27.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:54:27.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:54:27.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:54:27.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:54:27.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:54:27.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:54:27.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:54:27.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:54:27.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:54:27.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:54:27.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:54:27.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:54:27.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:54:27.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:54:27.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:54:27.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:54:27.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:54:27.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:54:27.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:54:27.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:54:27.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:54:27.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:54:27.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:54:27.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:54:27.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:54:27.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:54:27.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:54:27.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:54:27.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:54:27.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:54:27.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:54:27.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:54:27.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:54:27.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:54:27.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=765] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:54:27.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=765] Local address: 172.21.15.161, port 58442 2022-07-03T17:54:27.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=765] Peer address: 172.21.15.101, port 22 2022-07-03T17:54:27.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=765] Connection lost 2022-07-03T17:54:27.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=765] Aborting connection 2022-07-03T17:54:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:27 smithi161 ceph-mon[91579]: pgmap v13059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:27 smithi188 ceph-mon[92425]: pgmap v13059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:29 smithi161 ceph-mon[91579]: pgmap v13060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:29 smithi188 ceph-mon[92425]: pgmap v13060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:31 smithi161 ceph-mon[91579]: pgmap v13061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:31 smithi188 ceph-mon[92425]: pgmap v13061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:33 smithi161 ceph-mon[91579]: pgmap v13062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:33 smithi188 ceph-mon[92425]: pgmap v13062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:35 smithi161 ceph-mon[91579]: pgmap v13063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:35 smithi188 ceph-mon[92425]: pgmap v13063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:37 smithi161 ceph-mon[91579]: pgmap v13064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:37 smithi188 ceph-mon[92425]: pgmap v13064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:39 smithi188 ceph-mon[92425]: pgmap v13065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:39 smithi161 ceph-mon[91579]: pgmap v13065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:41 smithi188 ceph-mon[92425]: pgmap v13066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:41 smithi161 ceph-mon[91579]: pgmap v13066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:43 smithi188 ceph-mon[92425]: pgmap v13067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:43 smithi161 ceph-mon[91579]: pgmap v13067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:45 smithi188 ceph-mon[92425]: pgmap v13068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:45 smithi161 ceph-mon[91579]: pgmap v13068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:47 smithi188 ceph-mon[92425]: pgmap v13069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:47 smithi161 ceph-mon[91579]: pgmap v13069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:48.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:49 smithi188 ceph-mon[92425]: pgmap v13070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:49 smithi161 ceph-mon[91579]: pgmap v13070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:51 smithi188 ceph-mon[92425]: pgmap v13071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:51 smithi161 ceph-mon[91579]: pgmap v13071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:53 smithi188 ceph-mon[92425]: pgmap v13072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:53 smithi161 ceph-mon[91579]: pgmap v13072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:54:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:55 smithi188 ceph-mon[92425]: pgmap v13073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:55 smithi161 ceph-mon[91579]: pgmap v13073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:57 smithi188 ceph-mon[92425]: pgmap v13074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:57 smithi161 ceph-mon[91579]: pgmap v13074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:54:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:54:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:54:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:54:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:54:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:54:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:54:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:54:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:54:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:55:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:54:59 smithi161 ceph-mon[91579]: pgmap v13075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:54:59 smithi188 ceph-mon[92425]: pgmap v13075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:01 smithi161 ceph-mon[91579]: pgmap v13076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:01 smithi188 ceph-mon[92425]: pgmap v13076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:03 smithi161 ceph-mon[91579]: pgmap v13077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:03 smithi188 ceph-mon[92425]: pgmap v13077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:05 smithi161 ceph-mon[91579]: pgmap v13078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:05 smithi188 ceph-mon[92425]: pgmap v13078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:06 smithi161 ceph-mon[91579]: pgmap v13079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:06 smithi188 ceph-mon[92425]: pgmap v13079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:08 smithi161 ceph-mon[91579]: pgmap v13080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:08 smithi188 ceph-mon[92425]: pgmap v13080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:11 smithi161 ceph-mon[91579]: pgmap v13081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:11 smithi188 ceph-mon[92425]: pgmap v13081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:13 smithi161 ceph-mon[91579]: pgmap v13082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:13 smithi188 ceph-mon[92425]: pgmap v13082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:15 smithi161 ceph-mon[91579]: pgmap v13083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:15 smithi188 ceph-mon[92425]: pgmap v13083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:17 smithi161 ceph-mon[91579]: pgmap v13084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:17 smithi188 ceph-mon[92425]: pgmap v13084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:19 smithi161 ceph-mon[91579]: pgmap v13085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:19 smithi188 ceph-mon[92425]: pgmap v13085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:21 smithi161 ceph-mon[91579]: pgmap v13086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:21 smithi188 ceph-mon[92425]: pgmap v13086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:23 smithi161 ceph-mon[91579]: pgmap v13087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:23 smithi188 ceph-mon[92425]: pgmap v13087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:26.068 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:25 smithi161 ceph-mon[91579]: pgmap v13088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:26.077 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:25 smithi188 ceph-mon[92425]: pgmap v13088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:26.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:55:26.065+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:55:26.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:55:26.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:55:26.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:55:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:55:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:55:26.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:55:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:55:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:55:26.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:55:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:55:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:55:26.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:55:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:55:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:55:26.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:55:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:55:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:55:26.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:55:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:55:26.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:55:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:55:26.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:55:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:55:26.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:55:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:55:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:55:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:55:26.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:55:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:55:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:55:26.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:55:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:55:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:55:26.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:55:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:55:26.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:55:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:55:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:55:26.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:55:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:55:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:55:26.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:55:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:55:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:55:26.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=767] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:55:26.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=767] Local address: 172.21.15.161, port 58470 2022-07-03T17:55:26.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=767] Peer address: 172.21.15.101, port 22 2022-07-03T17:55:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=767] Connection lost 2022-07-03T17:55:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=767] Aborting connection 2022-07-03T17:55:26.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:55:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:55:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:55:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:55:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:55:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:55:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:55:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:55:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:55:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:55:28.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:55:28.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:55:28.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:55:28.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:55:28.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:55:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:55:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:55:28.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:55:28.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:55:28.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:55:28.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:55:28.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:55:28.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:55:28.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:55:28.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:55:28.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:55:28.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:55:28.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:55:28.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:55:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:55:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:55:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:55:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:55:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:55:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:55:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:55:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:55:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:55:28.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:55:28.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:55:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:55:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:55:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:55:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:55:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:55:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:55:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:55:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:55:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:55:28.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:55:28.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:55:28.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:55:28.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:55:28.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=767] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:55:28.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=767] Local address: 172.21.15.161, port 58470 2022-07-03T17:55:28.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=767] Peer address: 172.21.15.101, port 22 2022-07-03T17:55:28.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=767] Connection lost 2022-07-03T17:55:28.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=767] Aborting connection 2022-07-03T17:55:28.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:27 smithi161 ceph-mon[91579]: pgmap v13089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:28.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:55:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:55:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:55:28.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:55:28.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:55:28.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:55:28.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:55:28.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:55:28.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:55:28.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:55:28.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:55:28.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:55:28.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:55:28.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:55:28.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:55:28.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:55:28.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:55:28.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:55:28.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:55:28.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:55:28.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:55:28.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:55:28.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:55:28.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:55:28.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:55:28.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:55:28.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:55:28.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:55:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:55:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:55:28.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:55:28.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:55:28.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:55:28.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:55:28.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:55:28.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:55:28.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:55:28.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:55:28.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:55:28.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:55:28.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:55:28.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:55:28.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:55:28.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:55:28.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:55:28.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=767] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:55:28.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=767] Local address: 172.21.15.161, port 58470 2022-07-03T17:55:28.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=767] Peer address: 172.21.15.101, port 22 2022-07-03T17:55:28.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=767] Connection lost 2022-07-03T17:55:28.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=767] Aborting connection 2022-07-03T17:55:28.194 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:27 smithi188 ceph-mon[92425]: pgmap v13089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:28.195 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:29 smithi161 ceph-mon[91579]: pgmap v13090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:29 smithi188 ceph-mon[92425]: pgmap v13090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:31 smithi161 ceph-mon[91579]: pgmap v13091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:31 smithi188 ceph-mon[92425]: pgmap v13091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:33 smithi161 ceph-mon[91579]: pgmap v13092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:33 smithi188 ceph-mon[92425]: pgmap v13092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:35 smithi161 ceph-mon[91579]: pgmap v13093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:35 smithi188 ceph-mon[92425]: pgmap v13093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:37 smithi161 ceph-mon[91579]: pgmap v13094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:37 smithi188 ceph-mon[92425]: pgmap v13094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:39 smithi161 ceph-mon[91579]: pgmap v13095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:39 smithi188 ceph-mon[92425]: pgmap v13095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:41 smithi161 ceph-mon[91579]: pgmap v13096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:41 smithi188 ceph-mon[92425]: pgmap v13096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:43 smithi161 ceph-mon[91579]: pgmap v13097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:43 smithi188 ceph-mon[92425]: pgmap v13097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:45 smithi161 ceph-mon[91579]: pgmap v13098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:45 smithi188 ceph-mon[92425]: pgmap v13098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:47 smithi161 ceph-mon[91579]: pgmap v13099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:47 smithi188 ceph-mon[92425]: pgmap v13099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:49 smithi161 ceph-mon[91579]: pgmap v13100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:49 smithi188 ceph-mon[92425]: pgmap v13100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:51 smithi161 ceph-mon[91579]: pgmap v13101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:51 smithi188 ceph-mon[92425]: pgmap v13101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:53.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:53 smithi161 ceph-mon[91579]: pgmap v13102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:53 smithi188 ceph-mon[92425]: pgmap v13102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:55:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:55 smithi161 ceph-mon[91579]: pgmap v13103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:55 smithi188 ceph-mon[92425]: pgmap v13103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:57 smithi161 ceph-mon[91579]: pgmap v13104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:57 smithi188 ceph-mon[92425]: pgmap v13104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:55:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:55:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:55:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:55:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:55:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:55:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:55:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:56:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:55:59 smithi161 ceph-mon[91579]: pgmap v13105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:55:59 smithi188 ceph-mon[92425]: pgmap v13105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:01 smithi161 ceph-mon[91579]: pgmap v13106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:01 smithi188 ceph-mon[92425]: pgmap v13106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:03 smithi161 ceph-mon[91579]: pgmap v13107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:03 smithi188 ceph-mon[92425]: pgmap v13107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:06.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:05 smithi161 ceph-mon[91579]: pgmap v13108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:05 smithi188 ceph-mon[92425]: pgmap v13108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:07 smithi161 ceph-mon[91579]: pgmap v13109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:07 smithi188 ceph-mon[92425]: pgmap v13109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:09 smithi161 ceph-mon[91579]: pgmap v13110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:09 smithi188 ceph-mon[92425]: pgmap v13110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:11 smithi161 ceph-mon[91579]: pgmap v13111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:11 smithi188 ceph-mon[92425]: pgmap v13111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:13 smithi161 ceph-mon[91579]: pgmap v13112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:13 smithi188 ceph-mon[92425]: pgmap v13112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:15 smithi161 ceph-mon[91579]: pgmap v13113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:15 smithi188 ceph-mon[92425]: pgmap v13113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:17 smithi161 ceph-mon[91579]: pgmap v13114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:18.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:17 smithi188 ceph-mon[92425]: pgmap v13114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:19 smithi161 ceph-mon[91579]: pgmap v13115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:19 smithi188 ceph-mon[92425]: pgmap v13115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:21 smithi161 ceph-mon[91579]: pgmap v13116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:21 smithi188 ceph-mon[92425]: pgmap v13116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:23.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:23 smithi161 ceph-mon[91579]: pgmap v13117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:23 smithi188 ceph-mon[92425]: pgmap v13117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:25 smithi161 ceph-mon[91579]: pgmap v13118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:25 smithi188 ceph-mon[92425]: pgmap v13118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:26.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:56:26.564+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:56:26.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:56:26.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:56:26.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:56:26.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:56:26.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:56:26.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:56:26.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:56:26.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:56:26.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:56:26.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:56:26.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:56:26.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:56:26.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:56:26.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:56:26.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:56:26.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:56:26.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:56:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:56:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:56:26.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:56:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:56:26.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:56:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:56:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:56:26.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:56:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:56:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:56:26.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:56:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:56:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:56:26.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:56:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:56:26.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:56:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:56:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:56:26.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:56:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:56:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:56:26.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:56:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:56:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:56:26.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:56:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:56:26.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:56:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=769] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:56:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=769] Local address: 172.21.15.161, port 58498 2022-07-03T17:56:26.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=769] Peer address: 172.21.15.101, port 22 2022-07-03T17:56:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=769] Connection lost 2022-07-03T17:56:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=769] Aborting connection 2022-07-03T17:56:26.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:56:27.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:56:27.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:56:27.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:56:27.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:56:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:56:27.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:56:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:56:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:56:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:56:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:56:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:56:28.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:56:28.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:56:28.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:56:28.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:56:28.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:56:28.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:56:28.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:56:28.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:56:28.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:56:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:56:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:56:28.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:56:28.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:56:28.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:56:28.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:56:28.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:56:28.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:56:28.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:56:28.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:56:28.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:56:28.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:56:28.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:56:28.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:56:28.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:56:28.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:56:28.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:56:28.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:56:28.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:56:28.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:56:28.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:56:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:56:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:56:28.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:56:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:56:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:56:28.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:56:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:56:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:56:28.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:56:28.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:56:28.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:56:28.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=769] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:56:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=769] Local address: 172.21.15.161, port 58498 2022-07-03T17:56:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=769] Peer address: 172.21.15.101, port 22 2022-07-03T17:56:28.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=769] Connection lost 2022-07-03T17:56:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=769] Aborting connection 2022-07-03T17:56:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: pgmap v13119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:56:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:56:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:56:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:56:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:56:28.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:56:28.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:56:28.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:56:28.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:56:28.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:56:28.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:56:28.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:56:28.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:56:28.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:56:28.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:56:28.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:56:28.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:56:28.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:56:28.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:56:28.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:56:28.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:56:28.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:56:28.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:56:28.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:56:28.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:56:28.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:56:28.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:56:28.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:56:28.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:56:28.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:56:28.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:56:28.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:56:28.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:56:28.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:56:28.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:56:28.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:56:28.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:56:28.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:56:28.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:56:28.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:56:28.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:56:28.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:56:28.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:56:28.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:56:28.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:56:28.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:56:28.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=769] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:56:28.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=769] Local address: 172.21.15.161, port 58498 2022-07-03T17:56:28.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=769] Peer address: 172.21.15.101, port 22 2022-07-03T17:56:28.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=769] Connection lost 2022-07-03T17:56:28.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=769] Aborting connection 2022-07-03T17:56:28.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: pgmap v13119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:28.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:56:28.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:29 smithi161 ceph-mon[91579]: pgmap v13120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:29 smithi188 ceph-mon[92425]: pgmap v13120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:31 smithi161 ceph-mon[91579]: pgmap v13121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:31 smithi188 ceph-mon[92425]: pgmap v13121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:33.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:33 smithi188 ceph-mon[92425]: pgmap v13122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:33 smithi161 ceph-mon[91579]: pgmap v13122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:35 smithi188 ceph-mon[92425]: pgmap v13123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:35 smithi161 ceph-mon[91579]: pgmap v13123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:37 smithi188 ceph-mon[92425]: pgmap v13124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:37 smithi161 ceph-mon[91579]: pgmap v13124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:39 smithi188 ceph-mon[92425]: pgmap v13125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:39 smithi161 ceph-mon[91579]: pgmap v13125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:41 smithi188 ceph-mon[92425]: pgmap v13126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:41 smithi161 ceph-mon[91579]: pgmap v13126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:43 smithi188 ceph-mon[92425]: pgmap v13127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:43 smithi161 ceph-mon[91579]: pgmap v13127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:45 smithi188 ceph-mon[92425]: pgmap v13128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:45 smithi161 ceph-mon[91579]: pgmap v13128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:47 smithi188 ceph-mon[92425]: pgmap v13129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:47 smithi161 ceph-mon[91579]: pgmap v13129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:49 smithi188 ceph-mon[92425]: pgmap v13130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:49 smithi161 ceph-mon[91579]: pgmap v13130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:51 smithi188 ceph-mon[92425]: pgmap v13131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:51 smithi161 ceph-mon[91579]: pgmap v13131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:53 smithi188 ceph-mon[92425]: pgmap v13132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:53 smithi161 ceph-mon[91579]: pgmap v13132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:56:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:55 smithi188 ceph-mon[92425]: pgmap v13133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:55 smithi161 ceph-mon[91579]: pgmap v13133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:57 smithi161 ceph-mon[91579]: pgmap v13134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:57 smithi188 ceph-mon[92425]: pgmap v13134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:56:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:56:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:56:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:56:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:56:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:56:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:57:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:56:59 smithi161 ceph-mon[91579]: pgmap v13135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:56:59 smithi188 ceph-mon[92425]: pgmap v13135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:01 smithi161 ceph-mon[91579]: pgmap v13136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:01 smithi188 ceph-mon[92425]: pgmap v13136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:03 smithi161 ceph-mon[91579]: pgmap v13137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:03 smithi188 ceph-mon[92425]: pgmap v13137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:05 smithi161 ceph-mon[91579]: pgmap v13138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:05 smithi188 ceph-mon[92425]: pgmap v13138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:07 smithi161 ceph-mon[91579]: pgmap v13139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:07 smithi188 ceph-mon[92425]: pgmap v13139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:09 smithi161 ceph-mon[91579]: pgmap v13140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:09 smithi188 ceph-mon[92425]: pgmap v13140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:11 smithi161 ceph-mon[91579]: pgmap v13141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:11 smithi188 ceph-mon[92425]: pgmap v13141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:13 smithi161 ceph-mon[91579]: pgmap v13142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:13 smithi188 ceph-mon[92425]: pgmap v13142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:15 smithi161 ceph-mon[91579]: pgmap v13143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:15 smithi188 ceph-mon[92425]: pgmap v13143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:16 smithi161 ceph-mon[91579]: pgmap v13144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:16 smithi188 ceph-mon[92425]: pgmap v13144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:18 smithi161 ceph-mon[91579]: pgmap v13145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:18 smithi188 ceph-mon[92425]: pgmap v13145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:21 smithi161 ceph-mon[91579]: pgmap v13146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:21 smithi188 ceph-mon[92425]: pgmap v13146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:23 smithi161 ceph-mon[91579]: pgmap v13147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:23 smithi188 ceph-mon[92425]: pgmap v13147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:26.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:25 smithi161 ceph-mon[91579]: pgmap v13148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:25 smithi188 ceph-mon[92425]: pgmap v13148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:27.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:57:27.492+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:57:27.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:57:27.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:57:27.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:57:27.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:57:27.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:57:27.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:57:27.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:57:27.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:57:27.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:57:27.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:57:27.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:57:27.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:57:27.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:57:27.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:57:27.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:57:27.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:57:27.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:57:27.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:57:27.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:57:27.792 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:57:27.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:57:27.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:57:27.793 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:57:27.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:57:27.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:57:27.794 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:57:27.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:57:27.795 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:57:27.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:57:27.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:57:27.796 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:57:27.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:57:27.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:57:27.797 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:57:27.798 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:57:27.798 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:57:27.799 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:57:27.799 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:57:27.799 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:57:27.800 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:57:27.800 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:57:27.800 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:57:27.801 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:57:27.801 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:57:27.801 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=771] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:57:27.802 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=771] Local address: 172.21.15.161, port 58528 2022-07-03T17:57:27.802 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=771] Peer address: 172.21.15.101, port 22 2022-07-03T17:57:27.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=771] Connection lost 2022-07-03T17:57:27.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=771] Aborting connection 2022-07-03T17:57:27.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:57:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-mon[91579]: pgmap v13149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:57:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:57:28.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:57:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:27 smithi188 ceph-mon[92425]: pgmap v13149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:57:28.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:57:28.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:57:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:57:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:57:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:57:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:57:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:57:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:57:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:57:29.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:57:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:57:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:57:29.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:57:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:57:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:57:29.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:57:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:57:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:57:29.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:57:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:57:29.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:57:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:57:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:57:29.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:57:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:57:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:57:29.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:57:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:57:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:57:29.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:57:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:57:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:57:29.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:57:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:57:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:57:29.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:57:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:57:29.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:57:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:57:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:57:29.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:57:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:57:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:57:29.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:57:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:57:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:57:29.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:57:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:57:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=771] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:57:29.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=771] Local address: 172.21.15.161, port 58528 2022-07-03T17:57:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=771] Peer address: 172.21.15.101, port 22 2022-07-03T17:57:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=771] Connection lost 2022-07-03T17:57:29.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=771] Aborting connection 2022-07-03T17:57:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:57:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:57:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:57:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:57:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:57:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:57:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:57:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:57:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:57:29.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:57:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:57:29.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:57:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:57:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:57:29.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:57:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:57:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:57:29.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:57:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:57:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:57:29.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:57:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:57:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:57:29.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:57:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:57:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:57:29.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:57:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:57:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:57:29.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:57:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:57:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:57:29.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:57:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:57:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:57:29.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:57:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:57:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:57:29.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:57:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:57:29.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:57:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:57:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:57:29.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:57:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:57:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:57:29.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=771] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:57:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=771] Local address: 172.21.15.161, port 58528 2022-07-03T17:57:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=771] Peer address: 172.21.15.101, port 22 2022-07-03T17:57:29.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=771] Connection lost 2022-07-03T17:57:29.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=771] Aborting connection 2022-07-03T17:57:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:29 smithi161 ceph-mon[91579]: pgmap v13150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:29 smithi188 ceph-mon[92425]: pgmap v13150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:31 smithi161 ceph-mon[91579]: pgmap v13151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:31 smithi188 ceph-mon[92425]: pgmap v13151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:33 smithi161 ceph-mon[91579]: pgmap v13152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:33 smithi188 ceph-mon[92425]: pgmap v13152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:35 smithi161 ceph-mon[91579]: pgmap v13153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:35 smithi188 ceph-mon[92425]: pgmap v13153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:37 smithi161 ceph-mon[91579]: pgmap v13154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:37 smithi188 ceph-mon[92425]: pgmap v13154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:39 smithi161 ceph-mon[91579]: pgmap v13155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:39 smithi188 ceph-mon[92425]: pgmap v13155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:41 smithi161 ceph-mon[91579]: pgmap v13156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:41 smithi188 ceph-mon[92425]: pgmap v13156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:44.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:43 smithi161 ceph-mon[91579]: pgmap v13157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:43 smithi188 ceph-mon[92425]: pgmap v13157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:45 smithi161 ceph-mon[91579]: pgmap v13158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:45 smithi188 ceph-mon[92425]: pgmap v13158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:47 smithi161 ceph-mon[91579]: pgmap v13159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:47 smithi188 ceph-mon[92425]: pgmap v13159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:49 smithi161 ceph-mon[91579]: pgmap v13160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:49 smithi188 ceph-mon[92425]: pgmap v13160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:51 smithi161 ceph-mon[91579]: pgmap v13161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:51 smithi188 ceph-mon[92425]: pgmap v13161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:53.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:53 smithi161 ceph-mon[91579]: pgmap v13162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:53 smithi188 ceph-mon[92425]: pgmap v13162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:57:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:55 smithi161 ceph-mon[91579]: pgmap v13163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:55 smithi188 ceph-mon[92425]: pgmap v13163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:57 smithi161 ceph-mon[91579]: pgmap v13164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:57 smithi188 ceph-mon[92425]: pgmap v13164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:57:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:57:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:57:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:57:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:57:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:57:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:57:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:57:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:57:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:57:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:57:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:58:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:57:59 smithi161 ceph-mon[91579]: pgmap v13165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:57:59 smithi188 ceph-mon[92425]: pgmap v13165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:01 smithi161 ceph-mon[91579]: pgmap v13166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:01 smithi188 ceph-mon[92425]: pgmap v13166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:04.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:03 smithi161 ceph-mon[91579]: pgmap v13167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:04.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:03 smithi188 ceph-mon[92425]: pgmap v13167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:04.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:05 smithi161 ceph-mon[91579]: pgmap v13168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:05 smithi188 ceph-mon[92425]: pgmap v13168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:07 smithi161 ceph-mon[91579]: pgmap v13169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:07 smithi188 ceph-mon[92425]: pgmap v13169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:09.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:09 smithi161 ceph-mon[91579]: pgmap v13170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:09 smithi188 ceph-mon[92425]: pgmap v13170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:11 smithi161 ceph-mon[91579]: pgmap v13171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:11 smithi188 ceph-mon[92425]: pgmap v13171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:13 smithi161 ceph-mon[91579]: pgmap v13172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:13 smithi188 ceph-mon[92425]: pgmap v13172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:15 smithi161 ceph-mon[91579]: pgmap v13173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:15 smithi188 ceph-mon[92425]: pgmap v13173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:17 smithi161 ceph-mon[91579]: pgmap v13174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:17 smithi188 ceph-mon[92425]: pgmap v13174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:19 smithi161 ceph-mon[91579]: pgmap v13175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:19 smithi188 ceph-mon[92425]: pgmap v13175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:21 smithi188 ceph-mon[92425]: pgmap v13176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:21 smithi161 ceph-mon[91579]: pgmap v13176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:23 smithi188 ceph-mon[92425]: pgmap v13177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:23 smithi161 ceph-mon[91579]: pgmap v13177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:25 smithi188 ceph-mon[92425]: pgmap v13178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:25 smithi161 ceph-mon[91579]: pgmap v13178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:27 smithi188 ceph-mon[92425]: pgmap v13179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-mon[91579]: pgmap v13179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:58:27.993+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:58:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:58:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:58:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:58:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:58:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:58:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:58:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:58:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:58:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:58:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:58:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:58:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:58:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:58:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:58:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:58:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:58:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:58:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:58:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:58:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:58:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:58:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:58:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:58:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:58:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:58:28.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:58:28.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:58:28.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:58:28.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:58:28.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:58:28.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:58:28.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:58:28.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:58:28.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:58:28.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:58:28.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:58:28.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:58:28.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:58:28.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:58:28.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:58:28.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:58:28.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:58:28.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:58:28.418 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:58:28.419 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=773] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:58:28.420 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=773] Local address: 172.21.15.161, port 58556 2022-07-03T17:58:28.420 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=773] Peer address: 172.21.15.101, port 22 2022-07-03T17:58:28.421 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=773] Connection lost 2022-07-03T17:58:28.422 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=773] Aborting connection 2022-07-03T17:58:28.422 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:58:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:58:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:58:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:58:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:58:29.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:58:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:58:29.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:58:29.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:58:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:58:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:58:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:58:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:58:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:58:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:58:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:58:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:58:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:58:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:58:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:58:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:58:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:58:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:58:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:58:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:58:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:58:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:58:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:58:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:58:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:58:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:58:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:58:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:58:30.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:58:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:58:30.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:58:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=773] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:58:30.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=773] Local address: 172.21.15.161, port 58556 2022-07-03T17:58:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=773] Peer address: 172.21.15.101, port 22 2022-07-03T17:58:30.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=773] Connection lost 2022-07-03T17:58:30.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=773] Aborting connection 2022-07-03T17:58:30.188 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:29 smithi188 ceph-mon[92425]: pgmap v13180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:58:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:58:30.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:58:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:58:30.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:58:30.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:58:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:58:30.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:58:30.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:58:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:58:30.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:58:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:58:30.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:58:30.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:58:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:58:30.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:58:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:58:30.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:58:30.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:58:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:58:30.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:58:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:58:30.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:58:30.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:58:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:58:30.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:58:30.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:58:30.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:58:30.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:58:30.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:58:30.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:58:30.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:58:30.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:58:30.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:58:30.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:58:30.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:58:30.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:58:30.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:58:30.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:58:30.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:58:30.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:58:30.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:58:30.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:58:30.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:58:30.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:58:30.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:58:30.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=773] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:58:30.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=773] Local address: 172.21.15.161, port 58556 2022-07-03T17:58:30.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=773] Peer address: 172.21.15.101, port 22 2022-07-03T17:58:30.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=773] Connection lost 2022-07-03T17:58:30.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=773] Aborting connection 2022-07-03T17:58:30.413 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:29 smithi161 ceph-mon[91579]: pgmap v13180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:31 smithi188 ceph-mon[92425]: pgmap v13181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:31 smithi161 ceph-mon[91579]: pgmap v13181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:33 smithi188 ceph-mon[92425]: pgmap v13182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:33 smithi161 ceph-mon[91579]: pgmap v13182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:35 smithi188 ceph-mon[92425]: pgmap v13183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:35 smithi161 ceph-mon[91579]: pgmap v13183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:37 smithi188 ceph-mon[92425]: pgmap v13184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:37 smithi161 ceph-mon[91579]: pgmap v13184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:39 smithi161 ceph-mon[91579]: pgmap v13185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:39 smithi188 ceph-mon[92425]: pgmap v13185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:41 smithi161 ceph-mon[91579]: pgmap v13186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:41 smithi188 ceph-mon[92425]: pgmap v13186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:43 smithi161 ceph-mon[91579]: pgmap v13187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:43 smithi188 ceph-mon[92425]: pgmap v13187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:45 smithi161 ceph-mon[91579]: pgmap v13188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:45 smithi188 ceph-mon[92425]: pgmap v13188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:47 smithi161 ceph-mon[91579]: pgmap v13189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:47 smithi188 ceph-mon[92425]: pgmap v13189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:49 smithi161 ceph-mon[91579]: pgmap v13190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:49 smithi188 ceph-mon[92425]: pgmap v13190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:51 smithi161 ceph-mon[91579]: pgmap v13191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:51 smithi188 ceph-mon[92425]: pgmap v13191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:53 smithi161 ceph-mon[91579]: pgmap v13192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:53 smithi188 ceph-mon[92425]: pgmap v13192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:58:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:55 smithi161 ceph-mon[91579]: pgmap v13193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:55 smithi188 ceph-mon[92425]: pgmap v13193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:57 smithi161 ceph-mon[91579]: pgmap v13194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:57 smithi188 ceph-mon[92425]: pgmap v13194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:58:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:58:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:58:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:58:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:58:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:58:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:58:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:58:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:58:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:58:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:59:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:58:59 smithi161 ceph-mon[91579]: pgmap v13195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:58:59 smithi188 ceph-mon[92425]: pgmap v13195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:01 smithi161 ceph-mon[91579]: pgmap v13196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:01 smithi188 ceph-mon[92425]: pgmap v13196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:02.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:03 smithi161 ceph-mon[91579]: pgmap v13197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:03 smithi188 ceph-mon[92425]: pgmap v13197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:06.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:05 smithi161 ceph-mon[91579]: pgmap v13198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:06.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:05 smithi188 ceph-mon[92425]: pgmap v13198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:07 smithi161 ceph-mon[91579]: pgmap v13199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:07 smithi188 ceph-mon[92425]: pgmap v13199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:09 smithi161 ceph-mon[91579]: pgmap v13200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:09 smithi188 ceph-mon[92425]: pgmap v13200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:11 smithi161 ceph-mon[91579]: pgmap v13201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:11 smithi188 ceph-mon[92425]: pgmap v13201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:13 smithi161 ceph-mon[91579]: pgmap v13202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:13 smithi188 ceph-mon[92425]: pgmap v13202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:15 smithi161 ceph-mon[91579]: pgmap v13203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:15 smithi188 ceph-mon[92425]: pgmap v13203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:18.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:17 smithi161 ceph-mon[91579]: pgmap v13204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:18.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:17 smithi188 ceph-mon[92425]: pgmap v13204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:19 smithi161 ceph-mon[91579]: pgmap v13205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:19 smithi188 ceph-mon[92425]: pgmap v13205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:22.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:21 smithi161 ceph-mon[91579]: pgmap v13206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:21 smithi188 ceph-mon[92425]: pgmap v13206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:23.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:23 smithi161 ceph-mon[91579]: pgmap v13207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:23 smithi188 ceph-mon[92425]: pgmap v13207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:26.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:25 smithi161 ceph-mon[91579]: pgmap v13208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:25 smithi188 ceph-mon[92425]: pgmap v13208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:27 smithi161 ceph-mon[91579]: pgmap v13209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:28.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:27 smithi188 ceph-mon[92425]: pgmap v13209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:28.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:59:28.493+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:59:28.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:59:28.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:59:28.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T17:59:28.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:59:28.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:59:28.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:59:28.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T17:59:28.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:59:28.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T17:59:28.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:59:28.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T17:59:28.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:59:28.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:59:28.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T17:59:28.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:59:28.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T17:59:28.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:59:28.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T17:59:28.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:59:28.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T17:59:28.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:59:28.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:59:28.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:59:28.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:59:28.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:59:28.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T17:59:28.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:59:28.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:59:28.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:59:28.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T17:59:28.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:59:28.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T17:59:28.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:59:28.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:59:28.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:59:28.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T17:59:28.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:59:28.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T17:59:28.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:59:28.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T17:59:28.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:59:28.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T17:59:28.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:59:28.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:59:28.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=775] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:59:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=775] Local address: 172.21.15.161, port 58584 2022-07-03T17:59:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=775] Peer address: 172.21.15.101, port 22 2022-07-03T17:59:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=775] Connection lost 2022-07-03T17:59:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=775] Aborting connection 2022-07-03T17:59:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T17:59:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:29.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:59:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:59:29.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:59:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T17:59:29.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T17:59:29.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T17:59:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:59:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:59:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:59:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:59:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T17:59:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:59:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:59:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:59:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T17:59:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:59:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T17:59:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:59:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T17:59:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:59:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:59:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:59:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T17:59:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T17:59:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:59:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T17:59:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:59:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T17:59:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:59:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:59:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:59:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:59:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:59:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:59:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:59:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:59:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:59:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T17:59:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:59:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T17:59:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:59:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:59:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:59:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:59:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:59:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T17:59:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:59:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:59:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:59:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T17:59:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:59:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:59:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=775] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:59:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=775] Local address: 172.21.15.161, port 58584 2022-07-03T17:59:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=775] Peer address: 172.21.15.101, port 22 2022-07-03T17:59:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=775] Connection lost 2022-07-03T17:59:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=775] Aborting connection 2022-07-03T17:59:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:29 smithi161 ceph-mon[91579]: pgmap v13210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T17:59:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T17:59:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:59:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T17:59:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T17:59:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T17:59:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T17:59:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T17:59:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T17:59:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T17:59:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T17:59:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T17:59:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T17:59:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T17:59:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:59:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T17:59:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T17:59:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T17:59:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T17:59:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T17:59:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T17:59:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T17:59:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T17:59:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T17:59:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T17:59:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T17:59:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T17:59:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T17:59:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T17:59:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T17:59:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T17:59:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T17:59:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T17:59:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T17:59:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T17:59:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T17:59:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T17:59:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T17:59:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T17:59:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T17:59:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T17:59:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T17:59:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T17:59:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T17:59:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T17:59:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T17:59:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=775] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T17:59:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=775] Local address: 172.21.15.161, port 58584 2022-07-03T17:59:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=775] Peer address: 172.21.15.101, port 22 2022-07-03T17:59:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=775] Connection lost 2022-07-03T17:59:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=775] Aborting connection 2022-07-03T17:59:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:29 smithi188 ceph-mon[92425]: pgmap v13210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: pgmap v13211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.140 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.141 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.142 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.143 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: pgmap v13211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:32.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T17:59:32.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.174 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:32.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T17:59:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 42351k 2022-07-03T17:59:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:32 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 162.9M 2022-07-03T17:59:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 43368325: error parsing value: Value '43368325' is below minimum 939524096 2022-07-03T17:59:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:32 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 170857963: error parsing value: Value '170857963' is below minimum 939524096 2022-07-03T17:59:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 42351k 2022-07-03T17:59:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:32 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 162.9M 2022-07-03T17:59:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 43368325: error parsing value: Value '43368325' is below minimum 939524096 2022-07-03T17:59:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:32 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 170857963: error parsing value: Value '170857963' is below minimum 939524096 2022-07-03T17:59:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:33 smithi161 ceph-mon[91579]: pgmap v13212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:33 smithi188 ceph-mon[92425]: pgmap v13212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:35 smithi161 ceph-mon[91579]: pgmap v13213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:35 smithi188 ceph-mon[92425]: pgmap v13213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:37 smithi161 ceph-mon[91579]: pgmap v13214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:37 smithi188 ceph-mon[92425]: pgmap v13214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:39 smithi188 ceph-mon[92425]: pgmap v13215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:39 smithi161 ceph-mon[91579]: pgmap v13215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:41 smithi188 ceph-mon[92425]: pgmap v13216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:41 smithi161 ceph-mon[91579]: pgmap v13216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:43 smithi188 ceph-mon[92425]: pgmap v13217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:43 smithi161 ceph-mon[91579]: pgmap v13217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:45 smithi188 ceph-mon[92425]: pgmap v13218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:45 smithi161 ceph-mon[91579]: pgmap v13218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:47 smithi188 ceph-mon[92425]: pgmap v13219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:48.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:48.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:47 smithi161 ceph-mon[91579]: pgmap v13219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:48.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:49.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:49 smithi188 ceph-mon[92425]: pgmap v13220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:49 smithi161 ceph-mon[91579]: pgmap v13220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:51 smithi188 ceph-mon[92425]: pgmap v13221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:51 smithi161 ceph-mon[91579]: pgmap v13221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:53 smithi188 ceph-mon[92425]: pgmap v13222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:53 smithi161 ceph-mon[91579]: pgmap v13222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T17:59:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:55 smithi188 ceph-mon[92425]: pgmap v13223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:55 smithi161 ceph-mon[91579]: pgmap v13223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:57 smithi161 ceph-mon[91579]: pgmap v13224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:57 smithi188 ceph-mon[92425]: pgmap v13224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T17:59:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T17:59:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:59:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:59:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:59:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:59:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T17:59:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:59:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T17:59:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T17:59:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:00:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 17:59:59 smithi161 ceph-mon[91579]: pgmap v13225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 17:59:59 smithi188 ceph-mon[92425]: pgmap v13225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:00:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:00:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:01 smithi161 ceph-mon[91579]: pgmap v13226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:01 smithi188 ceph-mon[92425]: pgmap v13226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:03 smithi161 ceph-mon[91579]: pgmap v13227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:03 smithi188 ceph-mon[92425]: pgmap v13227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:05 smithi161 ceph-mon[91579]: pgmap v13228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:05 smithi188 ceph-mon[92425]: pgmap v13228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:07 smithi161 ceph-mon[91579]: pgmap v13229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:07 smithi188 ceph-mon[92425]: pgmap v13229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:09 smithi161 ceph-mon[91579]: pgmap v13230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:09 smithi188 ceph-mon[92425]: pgmap v13230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:11 smithi161 ceph-mon[91579]: pgmap v13231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:11 smithi188 ceph-mon[92425]: pgmap v13231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:13 smithi161 ceph-mon[91579]: pgmap v13232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:13 smithi188 ceph-mon[92425]: pgmap v13232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:15 smithi161 ceph-mon[91579]: pgmap v13233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:15 smithi188 ceph-mon[92425]: pgmap v13233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:17 smithi161 ceph-mon[91579]: pgmap v13234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:17 smithi188 ceph-mon[92425]: pgmap v13234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:19 smithi161 ceph-mon[91579]: pgmap v13235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:19 smithi188 ceph-mon[92425]: pgmap v13235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:21 smithi161 ceph-mon[91579]: pgmap v13236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:21 smithi188 ceph-mon[92425]: pgmap v13236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:23 smithi161 ceph-mon[91579]: pgmap v13237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:23 smithi188 ceph-mon[92425]: pgmap v13237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:25 smithi161 ceph-mon[91579]: pgmap v13238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:25 smithi188 ceph-mon[92425]: pgmap v13238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:27 smithi161 ceph-mon[91579]: pgmap v13239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:27 smithi188 ceph-mon[92425]: pgmap v13239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:29 smithi161 ceph-mon[91579]: pgmap v13240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:29 smithi188 ceph-mon[92425]: pgmap v13240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:30 smithi161 ceph-mon[91579]: pgmap v13241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:30 smithi188 ceph-mon[92425]: pgmap v13241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:31.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:00:31.660+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:00:31.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:00:31.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:00:31.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:00:31.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:00:31.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:00:31.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:00:31.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:00:31.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:00:31.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:00:31.994 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:00:31.994 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:00:32.006 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:00:32.007 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:00:32.007 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:00:32.007 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:00:32.008 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:00:32.008 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:00:32.009 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:00:32.009 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:00:32.009 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:00:32.010 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:00:32.010 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:00:32.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:00:32.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:00:32.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:00:32.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:00:32.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:00:32.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:00:32.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:00:32.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:00:32.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:00:32.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:00:32.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:00:32.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:00:32.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:00:32.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:00:32.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:00:32.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:00:32.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:00:32.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:00:32.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:00:32.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:00:32.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:00:32.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:00:32.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=777] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:00:32.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=777] Local address: 172.21.15.161, port 58612 2022-07-03T18:00:32.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=777] Peer address: 172.21.15.101, port 22 2022-07-03T18:00:32.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=777] Connection lost 2022-07-03T18:00:32.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=777] Aborting connection 2022-07-03T18:00:32.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:00:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:00:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:00:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:00:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:00:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:00:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:00:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:00:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:00:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:00:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:00:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:00:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:00:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:00:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:00:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:00:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:00:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:00:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:00:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:00:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:00:32.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:00:32.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:00:32.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=777] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:00:32.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=777] Local address: 172.21.15.161, port 58612 2022-07-03T18:00:32.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=777] Peer address: 172.21.15.101, port 22 2022-07-03T18:00:32.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=777] Connection lost 2022-07-03T18:00:32.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=777] Aborting connection 2022-07-03T18:00:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:00:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:00:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:00:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:00:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:00:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:00:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:00:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:00:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:00:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:00:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:00:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:00:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:00:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:00:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:00:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:00:32.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:00:32.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:00:32.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:00:32.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:00:32.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:00:32.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:00:32.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:00:32.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:00:32.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:00:32.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:00:32.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:00:32.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:00:32.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=777] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:00:32.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=777] Local address: 172.21.15.161, port 58612 2022-07-03T18:00:32.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=777] Peer address: 172.21.15.101, port 22 2022-07-03T18:00:32.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=777] Connection lost 2022-07-03T18:00:32.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=777] Aborting connection 2022-07-03T18:00:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:32 smithi161 ceph-mon[91579]: pgmap v13242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:32 smithi188 ceph-mon[92425]: pgmap v13242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:35 smithi161 ceph-mon[91579]: pgmap v13243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:35 smithi188 ceph-mon[92425]: pgmap v13243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:37 smithi161 ceph-mon[91579]: pgmap v13244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:37 smithi188 ceph-mon[92425]: pgmap v13244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:38.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:39.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:39 smithi161 ceph-mon[91579]: pgmap v13245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:39 smithi188 ceph-mon[92425]: pgmap v13245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:41 smithi161 ceph-mon[91579]: pgmap v13246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:42.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:41 smithi188 ceph-mon[92425]: pgmap v13246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:43 smithi161 ceph-mon[91579]: pgmap v13247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:44.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:43 smithi188 ceph-mon[92425]: pgmap v13247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:45 smithi161 ceph-mon[91579]: pgmap v13248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:46.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:45 smithi188 ceph-mon[92425]: pgmap v13248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:47 smithi161 ceph-mon[91579]: pgmap v13249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:48.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:47 smithi188 ceph-mon[92425]: pgmap v13249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:48.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:49 smithi161 ceph-mon[91579]: pgmap v13250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:49 smithi188 ceph-mon[92425]: pgmap v13250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:51 smithi161 ceph-mon[91579]: pgmap v13251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:51 smithi188 ceph-mon[92425]: pgmap v13251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:53.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:53.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:53 smithi161 ceph-mon[91579]: pgmap v13252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:53 smithi188 ceph-mon[92425]: pgmap v13252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:00:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:55 smithi161 ceph-mon[91579]: pgmap v13253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:55 smithi188 ceph-mon[92425]: pgmap v13253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:57 smithi161 ceph-mon[91579]: pgmap v13254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:57 smithi188 ceph-mon[92425]: pgmap v13254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:00:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:00:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:00:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:00:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:00:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:00:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:00:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:00:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:00:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:00:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:01:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:00:59 smithi188 ceph-mon[92425]: pgmap v13255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:00:59 smithi161 ceph-mon[91579]: pgmap v13255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:01 smithi188 ceph-mon[92425]: pgmap v13256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:01 smithi161 ceph-mon[91579]: pgmap v13256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:04.172 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:03 smithi188 ceph-mon[92425]: pgmap v13257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:03 smithi161 ceph-mon[91579]: pgmap v13257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:05 smithi188 ceph-mon[92425]: pgmap v13258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:05 smithi161 ceph-mon[91579]: pgmap v13258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:07 smithi188 ceph-mon[92425]: pgmap v13259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:07 smithi161 ceph-mon[91579]: pgmap v13259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:09 smithi188 ceph-mon[92425]: pgmap v13260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:09 smithi161 ceph-mon[91579]: pgmap v13260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:11 smithi188 ceph-mon[92425]: pgmap v13261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:11 smithi161 ceph-mon[91579]: pgmap v13261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:13 smithi188 ceph-mon[92425]: pgmap v13262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:13 smithi161 ceph-mon[91579]: pgmap v13262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:15 smithi161 ceph-mon[91579]: pgmap v13263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:15 smithi188 ceph-mon[92425]: pgmap v13263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:17 smithi161 ceph-mon[91579]: pgmap v13264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:17 smithi188 ceph-mon[92425]: pgmap v13264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:19 smithi161 ceph-mon[91579]: pgmap v13265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:19 smithi188 ceph-mon[92425]: pgmap v13265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:21 smithi161 ceph-mon[91579]: pgmap v13266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:21 smithi188 ceph-mon[92425]: pgmap v13266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:01:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:23 smithi161 ceph-mon[91579]: pgmap v13267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:23 smithi188 ceph-mon[92425]: pgmap v13267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:25 smithi161 ceph-mon[91579]: pgmap v13268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:25 smithi188 ceph-mon[92425]: pgmap v13268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:27 smithi161 ceph-mon[91579]: pgmap v13269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:27 smithi188 ceph-mon[92425]: pgmap v13269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:29 smithi161 ceph-mon[91579]: pgmap v13270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:29 smithi188 ceph-mon[92425]: pgmap v13270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:32.233 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:31 smithi161 ceph-mon[91579]: pgmap v13271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:32.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:01:32.167+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:01:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:01:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:01:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:01:32.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:01:32.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:01:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:01:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:01:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:01:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:01:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:01:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:01:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:01:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:01:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:01:32.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:01:32.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:01:32.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:01:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:01:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:01:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:01:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:01:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:01:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:01:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:01:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:01:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:01:32.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:01:32.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:01:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:01:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:01:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:01:32.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:01:32.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:01:32.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:01:32.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:01:32.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:01:32.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:01:32.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:01:32.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:01:32.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:01:32.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:01:32.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:01:32.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:01:32.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:01:32.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=779] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:01:32.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=779] Local address: 172.21.15.161, port 58640 2022-07-03T18:01:32.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=779] Peer address: 172.21.15.101, port 22 2022-07-03T18:01:32.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=779] Connection lost 2022-07-03T18:01:32.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=779] Aborting connection 2022-07-03T18:01:32.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:01:32.253 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:31 smithi188 ceph-mon[92425]: pgmap v13271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:01:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:01:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:01:33.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:01:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:01:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:01:33.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:01:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:01:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:01:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:01:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:01:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:01:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:01:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:01:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:01:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:01:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:01:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:01:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:01:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:01:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:01:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:01:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:01:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:01:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:01:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:01:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:01:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:01:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:01:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:01:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:01:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:01:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:01:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:01:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:01:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:01:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:01:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:01:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:01:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:01:34.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:01:34.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:01:34.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:01:34.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:01:34.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:01:34.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:01:34.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:01:34.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:01:34.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:01:34.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:01:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:01:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:01:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=779] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:01:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=779] Local address: 172.21.15.161, port 58640 2022-07-03T18:01:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=779] Peer address: 172.21.15.101, port 22 2022-07-03T18:01:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=779] Connection lost 2022-07-03T18:01:34.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=779] Aborting connection 2022-07-03T18:01:34.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:33 smithi161 ceph-mon[91579]: pgmap v13272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:34.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:01:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:01:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:01:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:01:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:01:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:01:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:01:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:01:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:01:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:01:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:01:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:01:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:01:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:01:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:01:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:01:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:01:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:01:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:01:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:01:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:01:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:01:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:01:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:01:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:01:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:01:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:01:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:01:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:01:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:01:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:01:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:01:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:01:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:01:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:01:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:01:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:01:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:01:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:01:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:01:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:01:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:01:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:01:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:01:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:01:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:01:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=779] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:01:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=779] Local address: 172.21.15.161, port 58640 2022-07-03T18:01:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=779] Peer address: 172.21.15.101, port 22 2022-07-03T18:01:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=779] Connection lost 2022-07-03T18:01:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=779] Aborting connection 2022-07-03T18:01:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:33 smithi188 ceph-mon[92425]: pgmap v13272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:34.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:35 smithi161 ceph-mon[91579]: pgmap v13273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:35 smithi188 ceph-mon[92425]: pgmap v13273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:37 smithi161 ceph-mon[91579]: pgmap v13274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:37 smithi188 ceph-mon[92425]: pgmap v13274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:39 smithi161 ceph-mon[91579]: pgmap v13275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:39 smithi188 ceph-mon[92425]: pgmap v13275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:41 smithi161 ceph-mon[91579]: pgmap v13276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:41 smithi188 ceph-mon[92425]: pgmap v13276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:43 smithi161 ceph-mon[91579]: pgmap v13277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:43 smithi188 ceph-mon[92425]: pgmap v13277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:45 smithi161 ceph-mon[91579]: pgmap v13278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:45 smithi188 ceph-mon[92425]: pgmap v13278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:47 smithi161 ceph-mon[91579]: pgmap v13279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:47 smithi188 ceph-mon[92425]: pgmap v13279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:49 smithi161 ceph-mon[91579]: pgmap v13280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:49 smithi188 ceph-mon[92425]: pgmap v13280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:51 smithi161 ceph-mon[91579]: pgmap v13281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:51 smithi188 ceph-mon[92425]: pgmap v13281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:53 smithi161 ceph-mon[91579]: pgmap v13282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:53 smithi188 ceph-mon[92425]: pgmap v13282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:01:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:54 smithi161 ceph-mon[91579]: pgmap v13283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:54 smithi188 ceph-mon[92425]: pgmap v13283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:57 smithi161 ceph-mon[91579]: pgmap v13284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:57 smithi188 ceph-mon[92425]: pgmap v13284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:01:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:01:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:01:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:01:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:01:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:01:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:01:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:01:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:01:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:01:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:02:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:01:59 smithi161 ceph-mon[91579]: pgmap v13285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:01:59 smithi188 ceph-mon[92425]: pgmap v13285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:01 smithi161 ceph-mon[91579]: pgmap v13286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:01 smithi188 ceph-mon[92425]: pgmap v13286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:03.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:03.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:03 smithi161 ceph-mon[91579]: pgmap v13287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:03 smithi188 ceph-mon[92425]: pgmap v13287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:06.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:05 smithi188 ceph-mon[92425]: pgmap v13288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:05 smithi161 ceph-mon[91579]: pgmap v13288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:07 smithi188 ceph-mon[92425]: pgmap v13289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:07 smithi161 ceph-mon[91579]: pgmap v13289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:09 smithi188 ceph-mon[92425]: pgmap v13290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:09 smithi161 ceph-mon[91579]: pgmap v13290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:11 smithi188 ceph-mon[92425]: pgmap v13291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:11 smithi161 ceph-mon[91579]: pgmap v13291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:13 smithi188 ceph-mon[92425]: pgmap v13292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:13 smithi161 ceph-mon[91579]: pgmap v13292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:15 smithi188 ceph-mon[92425]: pgmap v13293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:15 smithi161 ceph-mon[91579]: pgmap v13293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:17 smithi188 ceph-mon[92425]: pgmap v13294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:17 smithi161 ceph-mon[91579]: pgmap v13294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:19 smithi161 ceph-mon[91579]: pgmap v13295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:19 smithi188 ceph-mon[92425]: pgmap v13295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:21 smithi161 ceph-mon[91579]: pgmap v13296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:21 smithi188 ceph-mon[92425]: pgmap v13296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:23 smithi161 ceph-mon[91579]: pgmap v13297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:23 smithi188 ceph-mon[92425]: pgmap v13297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:25 smithi161 ceph-mon[91579]: pgmap v13298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:25 smithi188 ceph-mon[92425]: pgmap v13298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:27 smithi161 ceph-mon[91579]: pgmap v13299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:27 smithi188 ceph-mon[92425]: pgmap v13299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:29 smithi161 ceph-mon[91579]: pgmap v13300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:29 smithi188 ceph-mon[92425]: pgmap v13300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:31 smithi161 ceph-mon[91579]: pgmap v13301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:31 smithi188 ceph-mon[92425]: pgmap v13301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:32.939 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:02:32.680+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:02:32.940 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:02:32.940 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:02:32.941 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:02:32.941 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:02:32.942 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:02:32.942 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:02:32.943 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:02:32.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:02:32.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:02:32.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:02:32.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:02:32.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:02:32.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:02:32.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:02:32.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:02:32.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:02:32.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:02:32.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:02:32.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:02:32.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:02:32.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:02:32.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:02:32.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:02:32.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:02:32.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:02:32.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:02:32.952 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:02:32.952 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:02:32.952 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:02:32.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:02:32.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:02:32.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:02:32.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:02:32.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:02:32.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:02:32.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:02:32.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:02:32.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:02:32.956 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:02:32.956 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:02:32.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:02:32.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:02:32.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:02:32.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:02:32.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=781] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:02:32.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=781] Local address: 172.21.15.161, port 58670 2022-07-03T18:02:32.959 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=781] Peer address: 172.21.15.101, port 22 2022-07-03T18:02:32.959 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=781] Connection lost 2022-07-03T18:02:32.960 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=781] Aborting connection 2022-07-03T18:02:32.960 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:02:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:02:33.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:02:33.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:02:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:02:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:02:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:02:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:02:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:02:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:02:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:02:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:02:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:02:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:02:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:02:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:02:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:02:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:02:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:02:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:02:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:02:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:02:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:02:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:02:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:02:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:02:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:02:34.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:02:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:02:34.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:02:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:02:34.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:02:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:02:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:02:34.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:02:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:02:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:02:34.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:02:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:02:34.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:02:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:02:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:02:34.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:02:34.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:02:34.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:02:34.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:02:34.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:02:34.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:02:34.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:02:34.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:02:34.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:02:34.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:02:34.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:02:34.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=781] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:02:34.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=781] Local address: 172.21.15.161, port 58670 2022-07-03T18:02:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=781] Peer address: 172.21.15.101, port 22 2022-07-03T18:02:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=781] Connection lost 2022-07-03T18:02:34.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=781] Aborting connection 2022-07-03T18:02:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:33 smithi161 ceph-mon[91579]: pgmap v13302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:34.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:02:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:02:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:02:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:02:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:02:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:02:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:02:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:02:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:02:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:02:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:02:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:02:34.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:02:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:02:34.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:02:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:02:34.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:02:34.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:02:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:02:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:02:34.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:02:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:02:34.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:02:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:02:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:02:34.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:02:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:02:34.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:02:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:02:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:02:34.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:02:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:02:34.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:02:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:02:34.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:02:34.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:02:34.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:02:34.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:02:34.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:02:34.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:02:34.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:02:34.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:02:34.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:02:34.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:02:34.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:02:34.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:02:34.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=781] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:02:34.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=781] Local address: 172.21.15.161, port 58670 2022-07-03T18:02:34.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=781] Peer address: 172.21.15.101, port 22 2022-07-03T18:02:34.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=781] Connection lost 2022-07-03T18:02:34.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=781] Aborting connection 2022-07-03T18:02:34.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:33 smithi188 ceph-mon[92425]: pgmap v13302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:34.439 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:35 smithi161 ceph-mon[91579]: pgmap v13303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:35 smithi188 ceph-mon[92425]: pgmap v13303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:37 smithi161 ceph-mon[91579]: pgmap v13304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:37 smithi188 ceph-mon[92425]: pgmap v13304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:39 smithi161 ceph-mon[91579]: pgmap v13305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:39 smithi188 ceph-mon[92425]: pgmap v13305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:41 smithi161 ceph-mon[91579]: pgmap v13306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:41 smithi188 ceph-mon[92425]: pgmap v13306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:43 smithi161 ceph-mon[91579]: pgmap v13307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:43 smithi188 ceph-mon[92425]: pgmap v13307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:45 smithi161 ceph-mon[91579]: pgmap v13308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:45 smithi188 ceph-mon[92425]: pgmap v13308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:47 smithi161 ceph-mon[91579]: pgmap v13309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:47 smithi188 ceph-mon[92425]: pgmap v13309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:49 smithi161 ceph-mon[91579]: pgmap v13310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:49 smithi188 ceph-mon[92425]: pgmap v13310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:51 smithi161 ceph-mon[91579]: pgmap v13311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:51 smithi188 ceph-mon[92425]: pgmap v13311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:53 smithi161 ceph-mon[91579]: pgmap v13312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:53 smithi188 ceph-mon[92425]: pgmap v13312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:02:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:55 smithi161 ceph-mon[91579]: pgmap v13313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:55 smithi188 ceph-mon[92425]: pgmap v13313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:57 smithi161 ceph-mon[91579]: pgmap v13314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:57 smithi188 ceph-mon[92425]: pgmap v13314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:02:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:02:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:02:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:02:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:02:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:02:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:02:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:02:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:03:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:02:59 smithi161 ceph-mon[91579]: pgmap v13315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:02:59 smithi188 ceph-mon[92425]: pgmap v13315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:01 smithi161 ceph-mon[91579]: pgmap v13316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:01 smithi188 ceph-mon[92425]: pgmap v13316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:03 smithi161 ceph-mon[91579]: pgmap v13317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:03 smithi188 ceph-mon[92425]: pgmap v13317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:05 smithi161 ceph-mon[91579]: pgmap v13318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:06 smithi188 ceph-mon[92425]: pgmap v13318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:07 smithi161 ceph-mon[91579]: pgmap v13319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:07 smithi188 ceph-mon[92425]: pgmap v13319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:09 smithi161 ceph-mon[91579]: pgmap v13320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:09 smithi188 ceph-mon[92425]: pgmap v13320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:11 smithi188 ceph-mon[92425]: pgmap v13321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:11 smithi161 ceph-mon[91579]: pgmap v13321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:13 smithi188 ceph-mon[92425]: pgmap v13322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:13 smithi161 ceph-mon[91579]: pgmap v13322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:15 smithi188 ceph-mon[92425]: pgmap v13323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:15 smithi161 ceph-mon[91579]: pgmap v13323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:17 smithi188 ceph-mon[92425]: pgmap v13324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:18.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:17 smithi161 ceph-mon[91579]: pgmap v13324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:19 smithi188 ceph-mon[92425]: pgmap v13325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:19 smithi161 ceph-mon[91579]: pgmap v13325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:22.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:21 smithi188 ceph-mon[92425]: pgmap v13326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:21 smithi161 ceph-mon[91579]: pgmap v13326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:23 smithi188 ceph-mon[92425]: pgmap v13327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:23 smithi161 ceph-mon[91579]: pgmap v13327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:26.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:25 smithi188 ceph-mon[92425]: pgmap v13328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:25 smithi161 ceph-mon[91579]: pgmap v13328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:27 smithi188 ceph-mon[92425]: pgmap v13329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:27 smithi161 ceph-mon[91579]: pgmap v13329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:29 smithi161 ceph-mon[91579]: pgmap v13330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:29 smithi188 ceph-mon[92425]: pgmap v13330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:31 smithi161 ceph-mon[91579]: pgmap v13331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:31 smithi188 ceph-mon[92425]: pgmap v13331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:33.180 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:33.191 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:33.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:03:33.177+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:03:33.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:03:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:03:33.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:03:33.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:03:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:03:33.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:03:33.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:03:33.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:03:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:03:33.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:03:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:03:33.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:03:33.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:03:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:03:33.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:03:33.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:03:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:03:33.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:03:33.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:03:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:03:33.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:03:33.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:03:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:03:33.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:03:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:03:33.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:03:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:03:33.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:03:33.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:03:33.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:03:33.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:03:33.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:03:33.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:03:33.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:03:33.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:03:33.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:03:33.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:03:33.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:03:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:03:33.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:03:33.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:03:33.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:03:33.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:03:33.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:03:33.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=783] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:03:33.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=783] Local address: 172.21.15.161, port 58698 2022-07-03T18:03:33.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=783] Peer address: 172.21.15.101, port 22 2022-07-03T18:03:33.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=783] Connection lost 2022-07-03T18:03:33.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=783] Aborting connection 2022-07-03T18:03:33.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:03:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-mon[91579]: pgmap v13332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:03:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:03:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:03:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:33 smithi188 ceph-mon[92425]: pgmap v13332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:03:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:03:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:03:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:03:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:03:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:03:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:03:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:03:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:03:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:03:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:03:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:03:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:03:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:03:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:03:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:03:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:03:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:03:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:03:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:03:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:03:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:03:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:03:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:03:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:03:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:03:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:03:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:03:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:03:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:03:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:03:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:03:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:03:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:03:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:03:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:03:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:03:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:03:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:03:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:03:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:03:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:03:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:03:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:03:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:03:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:03:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:03:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:03:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:03:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=783] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:03:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=783] Local address: 172.21.15.161, port 58698 2022-07-03T18:03:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=783] Peer address: 172.21.15.101, port 22 2022-07-03T18:03:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=783] Connection lost 2022-07-03T18:03:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=783] Aborting connection 2022-07-03T18:03:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:03:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:03:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:03:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:03:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:03:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:03:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:03:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:03:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:03:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:03:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:03:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:03:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:03:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:03:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:03:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:03:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:03:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:03:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:03:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:03:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:03:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:03:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:03:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:03:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:03:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:03:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:03:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:03:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:03:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:03:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:03:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:03:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:03:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:03:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:03:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:03:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:03:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:03:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:03:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:03:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:03:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:03:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:03:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:03:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:03:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:03:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=783] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:03:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=783] Local address: 172.21.15.161, port 58698 2022-07-03T18:03:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=783] Peer address: 172.21.15.101, port 22 2022-07-03T18:03:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=783] Connection lost 2022-07-03T18:03:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=783] Aborting connection 2022-07-03T18:03:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:35 smithi161 ceph-mon[91579]: pgmap v13333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:35 smithi188 ceph-mon[92425]: pgmap v13333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:37 smithi161 ceph-mon[91579]: pgmap v13334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:37 smithi188 ceph-mon[92425]: pgmap v13334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:39 smithi161 ceph-mon[91579]: pgmap v13335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:39 smithi188 ceph-mon[92425]: pgmap v13335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:41 smithi161 ceph-mon[91579]: pgmap v13336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:41 smithi188 ceph-mon[92425]: pgmap v13336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:43 smithi161 ceph-mon[91579]: pgmap v13337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:43 smithi188 ceph-mon[92425]: pgmap v13337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:45 smithi161 ceph-mon[91579]: pgmap v13338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:45 smithi188 ceph-mon[92425]: pgmap v13338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:47 smithi161 ceph-mon[91579]: pgmap v13339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:47 smithi188 ceph-mon[92425]: pgmap v13339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:48.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:49 smithi161 ceph-mon[91579]: pgmap v13340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:49 smithi188 ceph-mon[92425]: pgmap v13340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:51 smithi161 ceph-mon[91579]: pgmap v13341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:51 smithi188 ceph-mon[92425]: pgmap v13341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:53 smithi161 ceph-mon[91579]: pgmap v13342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:53 smithi188 ceph-mon[92425]: pgmap v13342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:03:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:55 smithi161 ceph-mon[91579]: pgmap v13343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:55 smithi188 ceph-mon[92425]: pgmap v13343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:57 smithi161 ceph-mon[91579]: pgmap v13344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:57 smithi188 ceph-mon[92425]: pgmap v13344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:03:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:03:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:03:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:03:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:03:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:03:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:03:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:03:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:03:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:03:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:04:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:03:59 smithi161 ceph-mon[91579]: pgmap v13345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:03:59 smithi188 ceph-mon[92425]: pgmap v13345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:01 smithi161 ceph-mon[91579]: pgmap v13346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:01 smithi188 ceph-mon[92425]: pgmap v13346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:03 smithi161 ceph-mon[91579]: pgmap v13347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:03 smithi188 ceph-mon[92425]: pgmap v13347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:05 smithi161 ceph-mon[91579]: pgmap v13348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:05 smithi188 ceph-mon[92425]: pgmap v13348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:07 smithi161 ceph-mon[91579]: pgmap v13349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:07 smithi188 ceph-mon[92425]: pgmap v13349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:09 smithi161 ceph-mon[91579]: pgmap v13350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:09 smithi188 ceph-mon[92425]: pgmap v13350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:12 smithi161 ceph-mon[91579]: pgmap v13351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:12 smithi188 ceph-mon[92425]: pgmap v13351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:13.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:14 smithi161 ceph-mon[91579]: pgmap v13352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:14 smithi188 ceph-mon[92425]: pgmap v13352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:16 smithi161 ceph-mon[91579]: pgmap v13353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:16 smithi188 ceph-mon[92425]: pgmap v13353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:18 smithi161 ceph-mon[91579]: pgmap v13354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:18 smithi188 ceph-mon[92425]: pgmap v13354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:20 smithi161 ceph-mon[91579]: pgmap v13355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:20 smithi188 ceph-mon[92425]: pgmap v13355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:21 smithi161 ceph-mon[91579]: pgmap v13356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:21 smithi188 ceph-mon[92425]: pgmap v13356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:22.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:23 smithi161 ceph-mon[91579]: pgmap v13357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:23 smithi188 ceph-mon[92425]: pgmap v13357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:26.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:25 smithi188 ceph-mon[92425]: pgmap v13358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:25 smithi161 ceph-mon[91579]: pgmap v13358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:27 smithi188 ceph-mon[92425]: pgmap v13359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:28.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:27 smithi161 ceph-mon[91579]: pgmap v13359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:29 smithi161 ceph-mon[91579]: pgmap v13360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:29 smithi188 ceph-mon[92425]: pgmap v13360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:31 smithi161 ceph-mon[91579]: pgmap v13361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:31 smithi188 ceph-mon[92425]: pgmap v13361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-mon[91579]: pgmap v13362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:34.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:04:34.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:04:34.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:04:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:04:33.685+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:04:34.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:04:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:04:34.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:04:34.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:04:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:04:34.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:04:34.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:04:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:04:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:04:34.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:04:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:04:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:04:34.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:04:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:04:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:04:34.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:04:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:04:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:04:34.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:04:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:04:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:04:34.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:04:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:04:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:04:34.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:04:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:04:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:04:34.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:04:34.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:04:34.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:04:34.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:04:34.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:04:34.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:04:34.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:04:34.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:04:34.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:04:34.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:04:34.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:04:34.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:04:34.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:04:34.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:04:34.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:04:34.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:04:34.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:04:34.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=785] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:04:34.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=785] Local address: 172.21.15.161, port 58726 2022-07-03T18:04:34.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=785] Peer address: 172.21.15.101, port 22 2022-07-03T18:04:34.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=785] Connection lost 2022-07-03T18:04:34.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=785] Aborting connection 2022-07-03T18:04:34.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:04:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:04:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:33 smithi188 ceph-mon[92425]: pgmap v13362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:04:34.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:04:34.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:04:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:04:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:04:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:04:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:04:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:04:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:04:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:04:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:04:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:04:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:04:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:04:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:04:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:04:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:04:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:04:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:04:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:04:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:04:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:04:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:04:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:04:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:04:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:04:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:04:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:04:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:04:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:04:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:04:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:04:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:04:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:04:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:04:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:04:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:04:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:04:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:04:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:04:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:04:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:04:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:04:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:04:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:04:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:04:35.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:04:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:04:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:04:35.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=785] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:04:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=785] Local address: 172.21.15.161, port 58726 2022-07-03T18:04:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=785] Peer address: 172.21.15.101, port 22 2022-07-03T18:04:35.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=785] Connection lost 2022-07-03T18:04:35.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=785] Aborting connection 2022-07-03T18:04:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:04:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:04:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:04:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:04:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:04:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:04:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:04:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:04:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:04:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:04:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:04:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:04:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:04:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:04:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:04:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:04:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:04:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:04:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:04:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:04:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:04:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:04:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:04:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:04:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:04:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:04:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:04:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:04:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:04:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:04:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:04:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:04:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:04:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:04:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:04:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:04:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:04:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:04:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:04:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:04:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:04:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:04:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:04:35.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:04:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:04:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:04:35.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=785] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:04:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=785] Local address: 172.21.15.161, port 58726 2022-07-03T18:04:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=785] Peer address: 172.21.15.101, port 22 2022-07-03T18:04:35.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=785] Connection lost 2022-07-03T18:04:35.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=785] Aborting connection 2022-07-03T18:04:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:35 smithi161 ceph-mon[91579]: pgmap v13363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:35 smithi188 ceph-mon[92425]: pgmap v13363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:37 smithi161 ceph-mon[91579]: pgmap v13364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:37 smithi188 ceph-mon[92425]: pgmap v13364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:39 smithi161 ceph-mon[91579]: pgmap v13365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:39 smithi188 ceph-mon[92425]: pgmap v13365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:42.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:41 smithi188 ceph-mon[92425]: pgmap v13366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:41 smithi161 ceph-mon[91579]: pgmap v13366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:44.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:43 smithi188 ceph-mon[92425]: pgmap v13367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:44.227 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:43 smithi161 ceph-mon[91579]: pgmap v13367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:45 smithi161 ceph-mon[91579]: pgmap v13368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:45 smithi188 ceph-mon[92425]: pgmap v13368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:47 smithi161 ceph-mon[91579]: pgmap v13369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:47 smithi188 ceph-mon[92425]: pgmap v13369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:49 smithi161 ceph-mon[91579]: pgmap v13370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:49 smithi188 ceph-mon[92425]: pgmap v13370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:51 smithi161 ceph-mon[91579]: pgmap v13371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:51 smithi188 ceph-mon[92425]: pgmap v13371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:53 smithi161 ceph-mon[91579]: pgmap v13372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:53 smithi188 ceph-mon[92425]: pgmap v13372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:04:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:55 smithi188 ceph-mon[92425]: pgmap v13373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:55 smithi161 ceph-mon[91579]: pgmap v13373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:57 smithi161 ceph-mon[91579]: pgmap v13374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:57 smithi188 ceph-mon[92425]: pgmap v13374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:04:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:04:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:04:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:04:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:04:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:04:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:04:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:04:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:04:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:04:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:05:00.347 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:04:59 smithi188 ceph-mon[92425]: pgmap v13375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:04:59 smithi161 ceph-mon[91579]: pgmap v13375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:01 smithi161 ceph-mon[91579]: pgmap v13376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:01 smithi188 ceph-mon[92425]: pgmap v13376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:03 smithi161 ceph-mon[91579]: pgmap v13377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:03 smithi188 ceph-mon[92425]: pgmap v13377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:05 smithi161 ceph-mon[91579]: pgmap v13378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:05 smithi188 ceph-mon[92425]: pgmap v13378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:07 smithi161 ceph-mon[91579]: pgmap v13379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:07 smithi188 ceph-mon[92425]: pgmap v13379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:09 smithi161 ceph-mon[91579]: pgmap v13380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:10.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:09 smithi188 ceph-mon[92425]: pgmap v13380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:12.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:12 smithi161 ceph-mon[91579]: pgmap v13381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:12 smithi188 ceph-mon[92425]: pgmap v13381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:14 smithi161 ceph-mon[91579]: pgmap v13382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:14 smithi188 ceph-mon[92425]: pgmap v13382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:16 smithi161 ceph-mon[91579]: pgmap v13383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:16 smithi188 ceph-mon[92425]: pgmap v13383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:18 smithi161 ceph-mon[91579]: pgmap v13384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:18 smithi188 ceph-mon[92425]: pgmap v13384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:20 smithi161 ceph-mon[91579]: pgmap v13385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:20 smithi188 ceph-mon[92425]: pgmap v13385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:22 smithi161 ceph-mon[91579]: pgmap v13386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:22 smithi188 ceph-mon[92425]: pgmap v13386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:23 smithi161 ceph-mon[91579]: pgmap v13387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:23 smithi188 ceph-mon[92425]: pgmap v13387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:25 smithi161 ceph-mon[91579]: pgmap v13388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:25 smithi188 ceph-mon[92425]: pgmap v13388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:27 smithi161 ceph-mon[91579]: pgmap v13389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:27 smithi188 ceph-mon[92425]: pgmap v13389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:29 smithi161 ceph-mon[91579]: pgmap v13390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:29 smithi188 ceph-mon[92425]: pgmap v13390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:31 smithi161 ceph-mon[91579]: pgmap v13391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:31 smithi188 ceph-mon[92425]: pgmap v13391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:34.195 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:33 smithi161 ceph-mon[91579]: pgmap v13392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:34.195 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:34.196 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:05:34.185+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:05:34.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:05:34.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:05:34.197 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:05:34.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:05:34.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:05:34.198 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:05:34.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:05:34.199 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:05:34.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:05:34.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:05:34.200 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:05:34.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:05:34.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:05:34.201 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:05:34.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:05:34.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:05:34.202 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:05:34.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:05:34.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:05:34.203 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:05:34.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:05:34.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:05:34.204 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:05:34.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:05:34.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:05:34.205 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:05:34.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:05:34.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:05:34.206 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:05:34.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:05:34.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:05:34.207 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:05:34.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:05:34.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:05:34.208 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:05:34.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:05:34.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:05:34.209 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:05:34.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:05:34.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:05:34.210 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:05:34.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:05:34.211 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:05:34.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:05:34.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=787] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:05:34.212 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=787] Local address: 172.21.15.161, port 58754 2022-07-03T18:05:34.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=787] Peer address: 172.21.15.101, port 22 2022-07-03T18:05:34.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=787] Connection lost 2022-07-03T18:05:34.213 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=787] Aborting connection 2022-07-03T18:05:34.214 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:05:34.214 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:33 smithi188 ceph-mon[92425]: pgmap v13392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:34.215 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:05:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:05:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:05:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:05:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:05:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:05:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:05:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:05:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:05:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:05:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:05:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:05:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:05:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:05:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:05:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:05:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:05:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:05:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:05:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:05:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:05:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:05:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:05:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:05:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:05:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:05:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:05:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:05:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:05:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:05:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:05:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:05:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:05:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:05:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:05:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:05:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:05:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:05:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:05:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:05:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:05:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:05:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:05:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:05:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:05:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:05:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:05:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:05:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:05:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:05:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:05:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:05:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=787] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:05:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=787] Local address: 172.21.15.161, port 58754 2022-07-03T18:05:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=787] Peer address: 172.21.15.101, port 22 2022-07-03T18:05:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=787] Connection lost 2022-07-03T18:05:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=787] Aborting connection 2022-07-03T18:05:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:35 smithi161 ceph-mon[91579]: pgmap v13393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:05:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:05:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:05:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:05:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:05:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:05:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:05:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:05:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:05:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:05:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:05:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:05:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:05:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:05:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:05:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:05:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:05:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:05:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:05:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:05:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:05:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:05:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:05:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:05:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:05:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:05:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:05:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:05:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:05:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:05:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:05:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:05:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:05:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:05:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:05:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:05:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:05:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:05:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:05:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:05:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:05:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:05:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:05:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:05:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:05:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:05:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=787] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:05:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=787] Local address: 172.21.15.161, port 58754 2022-07-03T18:05:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=787] Peer address: 172.21.15.101, port 22 2022-07-03T18:05:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=787] Connection lost 2022-07-03T18:05:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=787] Aborting connection 2022-07-03T18:05:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:35 smithi188 ceph-mon[92425]: pgmap v13393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:37 smithi161 ceph-mon[91579]: pgmap v13394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:37 smithi188 ceph-mon[92425]: pgmap v13394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:39 smithi161 ceph-mon[91579]: pgmap v13395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:39 smithi188 ceph-mon[92425]: pgmap v13395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:41 smithi161 ceph-mon[91579]: pgmap v13396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:41 smithi188 ceph-mon[92425]: pgmap v13396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:43.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:43 smithi161 ceph-mon[91579]: pgmap v13397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:43 smithi188 ceph-mon[92425]: pgmap v13397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:45 smithi161 ceph-mon[91579]: pgmap v13398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:45 smithi188 ceph-mon[92425]: pgmap v13398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:47 smithi161 ceph-mon[91579]: pgmap v13399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:47 smithi188 ceph-mon[92425]: pgmap v13399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:49 smithi161 ceph-mon[91579]: pgmap v13400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:49 smithi188 ceph-mon[92425]: pgmap v13400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:51 smithi161 ceph-mon[91579]: pgmap v13401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:51 smithi188 ceph-mon[92425]: pgmap v13401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:53 smithi161 ceph-mon[91579]: pgmap v13402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:53 smithi188 ceph-mon[92425]: pgmap v13402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:05:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:55 smithi161 ceph-mon[91579]: pgmap v13403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:55 smithi188 ceph-mon[92425]: pgmap v13403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:57 smithi161 ceph-mon[91579]: pgmap v13404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:57 smithi188 ceph-mon[92425]: pgmap v13404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:05:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:05:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:05:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:05:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:05:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:05:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:05:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:05:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:05:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:05:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:06:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:05:59 smithi161 ceph-mon[91579]: pgmap v13405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:05:59 smithi188 ceph-mon[92425]: pgmap v13405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:01 smithi161 ceph-mon[91579]: pgmap v13406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:01 smithi188 ceph-mon[92425]: pgmap v13406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:04 smithi161 ceph-mon[91579]: pgmap v13407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:04 smithi188 ceph-mon[92425]: pgmap v13407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:06 smithi161 ceph-mon[91579]: pgmap v13408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:06 smithi188 ceph-mon[92425]: pgmap v13408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:08 smithi161 ceph-mon[91579]: pgmap v13409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:08 smithi188 ceph-mon[92425]: pgmap v13409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:10 smithi161 ceph-mon[91579]: pgmap v13410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:10 smithi188 ceph-mon[92425]: pgmap v13410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:12 smithi161 ceph-mon[91579]: pgmap v13411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:12 smithi188 ceph-mon[92425]: pgmap v13411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:14 smithi161 ceph-mon[91579]: pgmap v13412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:14 smithi188 ceph-mon[92425]: pgmap v13412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:16 smithi161 ceph-mon[91579]: pgmap v13413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:16 smithi188 ceph-mon[92425]: pgmap v13413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:18 smithi161 ceph-mon[91579]: pgmap v13414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:18 smithi188 ceph-mon[92425]: pgmap v13414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:20 smithi161 ceph-mon[91579]: pgmap v13415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:20 smithi188 ceph-mon[92425]: pgmap v13415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:21 smithi161 ceph-mon[91579]: pgmap v13416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:21 smithi188 ceph-mon[92425]: pgmap v13416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:22.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:22.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:23 smithi161 ceph-mon[91579]: pgmap v13417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:24.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:23 smithi188 ceph-mon[92425]: pgmap v13417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:25 smithi161 ceph-mon[91579]: pgmap v13418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:26.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:25 smithi188 ceph-mon[92425]: pgmap v13418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:27 smithi161 ceph-mon[91579]: pgmap v13419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:27 smithi188 ceph-mon[92425]: pgmap v13419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:29 smithi161 ceph-mon[91579]: pgmap v13420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:29 smithi188 ceph-mon[92425]: pgmap v13420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:31 smithi161 ceph-mon[91579]: pgmap v13421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:31 smithi188 ceph-mon[92425]: pgmap v13421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:33 smithi161 ceph-mon[91579]: pgmap v13422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:33 smithi188 ceph-mon[92425]: pgmap v13422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:34.966 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:06:34.695+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:06:34.967 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:06:34.967 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:06:34.968 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:06:34.968 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:06:34.969 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:06:34.970 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:06:34.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:06:34.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:06:34.971 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:06:34.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:06:34.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:06:34.972 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:06:34.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:06:34.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:06:34.973 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:06:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:06:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:06:34.974 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:06:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:06:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:06:34.975 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:06:34.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:06:34.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:06:34.976 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:06:34.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:06:34.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:06:34.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:06:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:06:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:06:34.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:06:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:06:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:06:34.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:06:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:06:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:06:34.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:06:34.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:06:34.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:06:34.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:06:34.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:06:34.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:06:34.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:06:34.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:06:34.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:06:34.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=789] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:06:34.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=789] Local address: 172.21.15.161, port 58782 2022-07-03T18:06:34.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=789] Peer address: 172.21.15.101, port 22 2022-07-03T18:06:34.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=789] Connection lost 2022-07-03T18:06:34.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=789] Aborting connection 2022-07-03T18:06:34.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:06:34.989 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:06:34.990 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:06:34.990 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:06:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:06:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:06:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:06:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:06:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:06:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:06:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:06:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:06:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:06:36.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:06:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:06:36.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:06:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:06:36.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:06:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:06:36.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:06:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:06:36.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:06:36.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:06:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:06:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:06:36.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:06:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:06:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:06:36.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:06:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:06:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:06:36.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:06:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:06:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:06:36.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:06:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:06:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:06:36.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:06:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:06:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:06:36.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:06:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:06:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:06:36.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:06:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:06:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:06:36.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:06:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:06:36.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:06:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:06:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:06:36.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:06:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:06:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=789] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:06:36.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=789] Local address: 172.21.15.161, port 58782 2022-07-03T18:06:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=789] Peer address: 172.21.15.101, port 22 2022-07-03T18:06:36.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=789] Connection lost 2022-07-03T18:06:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=789] Aborting connection 2022-07-03T18:06:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: pgmap v13423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:36.402 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.403 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:06:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:06:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:06:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:06:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:06:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:06:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:06:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:06:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:06:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:06:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:06:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:06:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:06:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:06:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:06:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:06:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:06:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:06:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:06:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:06:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:06:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:06:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:06:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:06:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:06:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:06:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:06:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:06:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:06:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:06:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:06:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:06:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:06:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:06:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:06:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:06:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:06:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:06:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:06:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:06:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:06:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:06:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:06:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:06:36.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:06:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:06:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=789] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:06:36.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=789] Local address: 172.21.15.161, port 58782 2022-07-03T18:06:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=789] Peer address: 172.21.15.101, port 22 2022-07-03T18:06:36.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=789] Connection lost 2022-07-03T18:06:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=789] Aborting connection 2022-07-03T18:06:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: pgmap v13423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:36.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:36.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:06:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:38 smithi161 ceph-mon[91579]: pgmap v13424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:38 smithi188 ceph-mon[92425]: pgmap v13424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:40 smithi161 ceph-mon[91579]: pgmap v13425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:40 smithi188 ceph-mon[92425]: pgmap v13425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:42 smithi161 ceph-mon[91579]: pgmap v13426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:42 smithi188 ceph-mon[92425]: pgmap v13426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:44 smithi161 ceph-mon[91579]: pgmap v13427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:44 smithi188 ceph-mon[92425]: pgmap v13427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:46 smithi161 ceph-mon[91579]: pgmap v13428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:46 smithi188 ceph-mon[92425]: pgmap v13428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:47 smithi161 ceph-mon[91579]: pgmap v13429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:47 smithi188 ceph-mon[92425]: pgmap v13429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:49 smithi161 ceph-mon[91579]: pgmap v13430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:49 smithi188 ceph-mon[92425]: pgmap v13430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:51 smithi161 ceph-mon[91579]: pgmap v13431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:51 smithi188 ceph-mon[92425]: pgmap v13431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:53 smithi161 ceph-mon[91579]: pgmap v13432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:53 smithi188 ceph-mon[92425]: pgmap v13432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:06:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:55 smithi161 ceph-mon[91579]: pgmap v13433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:55 smithi188 ceph-mon[92425]: pgmap v13433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:57 smithi161 ceph-mon[91579]: pgmap v13434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:57 smithi188 ceph-mon[92425]: pgmap v13434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:06:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:06:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:06:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:06:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:06:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:06:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:06:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:06:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:06:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:06:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:07:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:06:59 smithi161 ceph-mon[91579]: pgmap v13435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:06:59 smithi188 ceph-mon[92425]: pgmap v13435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:01 smithi161 ceph-mon[91579]: pgmap v13436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:01 smithi188 ceph-mon[92425]: pgmap v13436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:03 smithi161 ceph-mon[91579]: pgmap v13437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:03 smithi188 ceph-mon[92425]: pgmap v13437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:05 smithi161 ceph-mon[91579]: pgmap v13438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:05 smithi188 ceph-mon[92425]: pgmap v13438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:07 smithi161 ceph-mon[91579]: pgmap v13439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:07 smithi188 ceph-mon[92425]: pgmap v13439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:09 smithi161 ceph-mon[91579]: pgmap v13440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:09 smithi188 ceph-mon[92425]: pgmap v13440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:11 smithi161 ceph-mon[91579]: pgmap v13441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:11 smithi188 ceph-mon[92425]: pgmap v13441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:13 smithi161 ceph-mon[91579]: pgmap v13442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:13 smithi188 ceph-mon[92425]: pgmap v13442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:15 smithi161 ceph-mon[91579]: pgmap v13443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:15 smithi188 ceph-mon[92425]: pgmap v13443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:17 smithi161 ceph-mon[91579]: pgmap v13444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:18.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:18 smithi188 ceph-mon[92425]: pgmap v13444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:20 smithi161 ceph-mon[91579]: pgmap v13445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:20 smithi188 ceph-mon[92425]: pgmap v13445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:22 smithi161 ceph-mon[91579]: pgmap v13446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:22 smithi188 ceph-mon[92425]: pgmap v13446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:24 smithi161 ceph-mon[91579]: pgmap v13447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:24 smithi188 ceph-mon[92425]: pgmap v13447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:26 smithi161 ceph-mon[91579]: pgmap v13448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:26 smithi188 ceph-mon[92425]: pgmap v13448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:28 smithi161 ceph-mon[91579]: pgmap v13449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:28.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:28 smithi188 ceph-mon[92425]: pgmap v13449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:29.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:30 smithi161 ceph-mon[91579]: pgmap v13450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:30.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:30 smithi188 ceph-mon[92425]: pgmap v13450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:32 smithi161 ceph-mon[91579]: pgmap v13451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:32 smithi188 ceph-mon[92425]: pgmap v13451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:34 smithi161 ceph-mon[91579]: pgmap v13452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:34 smithi188 ceph-mon[92425]: pgmap v13452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:07:35.621+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:07:35.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:07:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:07:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:07:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:07:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:07:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:07:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:07:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:07:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:07:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:07:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:07:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:07:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:07:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:07:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:07:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:07:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:07:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:07:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:07:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:07:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:07:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:07:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:07:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:07:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:07:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:07:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:07:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:07:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:07:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:07:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:07:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:07:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:07:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:07:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:07:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:07:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:07:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:07:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:07:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:07:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:07:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:07:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:07:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:07:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=791] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:07:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=791] Local address: 172.21.15.161, port 58810 2022-07-03T18:07:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=791] Peer address: 172.21.15.101, port 22 2022-07-03T18:07:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=791] Connection lost 2022-07-03T18:07:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=791] Aborting connection 2022-07-03T18:07:35.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:07:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:36 smithi161 ceph-mon[91579]: pgmap v13453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:07:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:07:36.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:07:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:36 smithi188 ceph-mon[92425]: pgmap v13453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:07:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:07:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:07:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:07:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:07:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:07:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:07:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:07:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:07:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:07:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:07:37.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:07:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:07:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:07:37.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:07:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:07:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:07:37.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:07:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:07:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:07:37.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:07:37.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:07:37.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:07:37.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:07:37.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:07:37.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:07:37.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:07:37.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:07:37.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:07:37.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:07:37.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:07:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:07:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:07:37.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:07:37.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:07:37.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:07:37.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:07:37.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:07:37.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:07:37.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:07:37.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:07:37.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:07:37.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:07:37.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:07:37.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:07:37.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:07:37.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:07:37.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:07:37.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:07:37.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=791] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:07:37.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=791] Local address: 172.21.15.161, port 58810 2022-07-03T18:07:37.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=791] Peer address: 172.21.15.101, port 22 2022-07-03T18:07:37.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=791] Connection lost 2022-07-03T18:07:37.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=791] Aborting connection 2022-07-03T18:07:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:07:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:07:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:07:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:07:37.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:07:37.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:07:37.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:07:37.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:07:37.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:07:37.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:07:37.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:07:37.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:07:37.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:07:37.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:07:37.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:07:37.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:07:37.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:07:37.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:07:37.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:07:37.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:07:37.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:07:37.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:07:37.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:07:37.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:07:37.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:07:37.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:07:37.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:07:37.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:07:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:07:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:07:37.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:07:37.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:07:37.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:07:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:07:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:07:37.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:07:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:07:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:07:37.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:07:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:07:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:07:37.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:07:37.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:07:37.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:07:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:07:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:07:37.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=791] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:07:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=791] Local address: 172.21.15.161, port 58810 2022-07-03T18:07:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=791] Peer address: 172.21.15.101, port 22 2022-07-03T18:07:37.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=791] Connection lost 2022-07-03T18:07:37.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=791] Aborting connection 2022-07-03T18:07:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:38 smithi161 ceph-mon[91579]: pgmap v13454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:38 smithi188 ceph-mon[92425]: pgmap v13454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:39 smithi161 ceph-mon[91579]: pgmap v13455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:39 smithi188 ceph-mon[92425]: pgmap v13455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:41 smithi161 ceph-mon[91579]: pgmap v13456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:41 smithi188 ceph-mon[92425]: pgmap v13456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:43 smithi161 ceph-mon[91579]: pgmap v13457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:43 smithi188 ceph-mon[92425]: pgmap v13457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:45 smithi161 ceph-mon[91579]: pgmap v13458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:45 smithi188 ceph-mon[92425]: pgmap v13458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:47 smithi161 ceph-mon[91579]: pgmap v13459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:47 smithi188 ceph-mon[92425]: pgmap v13459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:49 smithi161 ceph-mon[91579]: pgmap v13460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:49 smithi188 ceph-mon[92425]: pgmap v13460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:51 smithi161 ceph-mon[91579]: pgmap v13461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:51 smithi188 ceph-mon[92425]: pgmap v13461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:53 smithi161 ceph-mon[91579]: pgmap v13462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:54.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:53 smithi188 ceph-mon[92425]: pgmap v13462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:07:54.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:55 smithi161 ceph-mon[91579]: pgmap v13463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:55 smithi188 ceph-mon[92425]: pgmap v13463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:57 smithi161 ceph-mon[91579]: pgmap v13464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:57 smithi188 ceph-mon[92425]: pgmap v13464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:07:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:07:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:07:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:07:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:07:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:07:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:07:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:07:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:07:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:07:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:08:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:07:59 smithi161 ceph-mon[91579]: pgmap v13465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:07:59 smithi188 ceph-mon[92425]: pgmap v13465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:02 smithi161 ceph-mon[91579]: pgmap v13466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:02 smithi188 ceph-mon[92425]: pgmap v13466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:04 smithi161 ceph-mon[91579]: pgmap v13467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:04 smithi188 ceph-mon[92425]: pgmap v13467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:06 smithi161 ceph-mon[91579]: pgmap v13468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:06 smithi188 ceph-mon[92425]: pgmap v13468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:08 smithi161 ceph-mon[91579]: pgmap v13469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:08 smithi188 ceph-mon[92425]: pgmap v13469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:10 smithi161 ceph-mon[91579]: pgmap v13470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:10 smithi188 ceph-mon[92425]: pgmap v13470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:12 smithi161 ceph-mon[91579]: pgmap v13471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:12 smithi188 ceph-mon[92425]: pgmap v13471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:14 smithi161 ceph-mon[91579]: pgmap v13472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:14 smithi188 ceph-mon[92425]: pgmap v13472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:16 smithi161 ceph-mon[91579]: pgmap v13473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:16.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:16 smithi188 ceph-mon[92425]: pgmap v13473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:18 smithi161 ceph-mon[91579]: pgmap v13474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:18 smithi188 ceph-mon[92425]: pgmap v13474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:18.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:20 smithi161 ceph-mon[91579]: pgmap v13475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:20 smithi188 ceph-mon[92425]: pgmap v13475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:22 smithi161 ceph-mon[91579]: pgmap v13476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:22 smithi188 ceph-mon[92425]: pgmap v13476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:24 smithi161 ceph-mon[91579]: pgmap v13477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:24 smithi188 ceph-mon[92425]: pgmap v13477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:25 smithi161 ceph-mon[91579]: pgmap v13478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:25 smithi188 ceph-mon[92425]: pgmap v13478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:27 smithi161 ceph-mon[91579]: pgmap v13479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:27 smithi188 ceph-mon[92425]: pgmap v13479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:29 smithi161 ceph-mon[91579]: pgmap v13480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:29 smithi188 ceph-mon[92425]: pgmap v13480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:31 smithi161 ceph-mon[91579]: pgmap v13481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:31 smithi188 ceph-mon[92425]: pgmap v13481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:33.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:33 smithi161 ceph-mon[91579]: pgmap v13482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:33 smithi188 ceph-mon[92425]: pgmap v13482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-mon[91579]: pgmap v13483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:08:36.118+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:08:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:08:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:08:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:08:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:08:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:08:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:08:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:08:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:08:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:08:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:08:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:08:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:08:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:08:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:08:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:08:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:08:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:08:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:08:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:08:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:08:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:08:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:08:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:08:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:08:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:08:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:08:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:08:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:08:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:08:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:08:36.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:08:36.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:08:36.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:08:36.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:08:36.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:08:36.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:08:36.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:08:36.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:08:36.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:08:36.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:08:36.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:08:36.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:08:36.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:08:36.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:08:36.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=793] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:08:36.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=793] Local address: 172.21.15.161, port 58838 2022-07-03T18:08:36.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=793] Peer address: 172.21.15.101, port 22 2022-07-03T18:08:36.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=793] Connection lost 2022-07-03T18:08:36.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=793] Aborting connection 2022-07-03T18:08:36.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:08:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:36 smithi188 ceph-mon[92425]: pgmap v13483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:08:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:08:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:08:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:08:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:08:37.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:08:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:08:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:08:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:08:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:08:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:08:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:08:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:08:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:08:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:08:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:08:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:08:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:08:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:08:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:08:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:08:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:08:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:08:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:08:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:08:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:08:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:08:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:08:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:08:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:08:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:08:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:08:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:08:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:08:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:08:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:08:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:08:38.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:08:38.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:08:38.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:08:38.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:08:38.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:08:38.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:08:38.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:08:38.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:08:38.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:08:38.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=793] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:08:38.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=793] Local address: 172.21.15.161, port 58838 2022-07-03T18:08:38.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=793] Peer address: 172.21.15.101, port 22 2022-07-03T18:08:38.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=793] Connection lost 2022-07-03T18:08:38.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=793] Aborting connection 2022-07-03T18:08:38.407 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:38 smithi161 ceph-mon[91579]: pgmap v13484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:38.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:08:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:08:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:08:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:08:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:08:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:08:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:08:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:08:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:08:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:08:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:08:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:08:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:08:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:08:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:08:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:08:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:08:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:08:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:08:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:08:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:08:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:08:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:08:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:08:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:08:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:08:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:08:38.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:08:38.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:08:38.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:08:38.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:08:38.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:08:38.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:08:38.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:08:38.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:08:38.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:08:38.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:08:38.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:08:38.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:08:38.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:08:38.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=793] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:08:38.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=793] Local address: 172.21.15.161, port 58838 2022-07-03T18:08:38.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=793] Peer address: 172.21.15.101, port 22 2022-07-03T18:08:38.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=793] Connection lost 2022-07-03T18:08:38.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=793] Aborting connection 2022-07-03T18:08:38.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:38 smithi188 ceph-mon[92425]: pgmap v13484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:38.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:40 smithi161 ceph-mon[91579]: pgmap v13485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:40 smithi188 ceph-mon[92425]: pgmap v13485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:42 smithi161 ceph-mon[91579]: pgmap v13486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:42.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:42 smithi188 ceph-mon[92425]: pgmap v13486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:43.427 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:44 smithi161 ceph-mon[91579]: pgmap v13487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:44 smithi188 ceph-mon[92425]: pgmap v13487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:46 smithi161 ceph-mon[91579]: pgmap v13488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:46 smithi188 ceph-mon[92425]: pgmap v13488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:08:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:08:47.472+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T18:08:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:48 smithi161 ceph-mon[91579]: pgmap v13489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:48 smithi188 ceph-mon[92425]: pgmap v13489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:08:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:50 smithi161 ceph-mon[91579]: pgmap v13490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:50 smithi188 ceph-mon[92425]: pgmap v13490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:52 smithi161 ceph-mon[91579]: pgmap v13491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:52 smithi188 ceph-mon[92425]: pgmap v13491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:53 smithi161 ceph-mon[91579]: pgmap v13492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:53 smithi188 ceph-mon[92425]: pgmap v13492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:08:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:55 smithi161 ceph-mon[91579]: pgmap v13493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:55 smithi188 ceph-mon[92425]: pgmap v13493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:57 smithi161 ceph-mon[91579]: pgmap v13494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:57 smithi188 ceph-mon[92425]: pgmap v13494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:08:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:08:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:08:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:08:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:08:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:08:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:08:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:08:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:08:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:09:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:08:59 smithi161 ceph-mon[91579]: pgmap v13495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:08:59 smithi188 ceph-mon[92425]: pgmap v13495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:02 smithi161 ceph-mon[91579]: pgmap v13496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:02 smithi188 ceph-mon[92425]: pgmap v13496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:03 smithi161 ceph-mon[91579]: pgmap v13497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:03 smithi188 ceph-mon[92425]: pgmap v13497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:05 smithi161 ceph-mon[91579]: pgmap v13498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:05 smithi188 ceph-mon[92425]: pgmap v13498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:07 smithi161 ceph-mon[91579]: pgmap v13499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:07 smithi188 ceph-mon[92425]: pgmap v13499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:10 smithi161 ceph-mon[91579]: pgmap v13500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:10 smithi188 ceph-mon[92425]: pgmap v13500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:12 smithi161 ceph-mon[91579]: pgmap v13501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:12 smithi188 ceph-mon[92425]: pgmap v13501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:13 smithi161 ceph-mon[91579]: pgmap v13502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:13 smithi188 ceph-mon[92425]: pgmap v13502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:15 smithi161 ceph-mon[91579]: pgmap v13503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:15 smithi188 ceph-mon[92425]: pgmap v13503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:17 smithi161 ceph-mon[91579]: pgmap v13504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:18 smithi188 ceph-mon[92425]: pgmap v13504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:20 smithi161 ceph-mon[91579]: pgmap v13505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:20 smithi188 ceph-mon[92425]: pgmap v13505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:22 smithi161 ceph-mon[91579]: pgmap v13506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:22 smithi188 ceph-mon[92425]: pgmap v13506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:23 smithi161 ceph-mon[91579]: pgmap v13507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:23 smithi188 ceph-mon[92425]: pgmap v13507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:25 smithi161 ceph-mon[91579]: pgmap v13508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:25 smithi188 ceph-mon[92425]: pgmap v13508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:28 smithi161 ceph-mon[91579]: pgmap v13509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:28 smithi188 ceph-mon[92425]: pgmap v13509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:29.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:30 smithi161 ceph-mon[91579]: pgmap v13510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:30 smithi188 ceph-mon[92425]: pgmap v13510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:32 smithi161 ceph-mon[91579]: pgmap v13511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:32 smithi188 ceph-mon[92425]: pgmap v13511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:33 smithi161 ceph-mon[91579]: pgmap v13512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:33 smithi188 ceph-mon[92425]: pgmap v13512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-mon[91579]: pgmap v13513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:36 smithi188 ceph-mon[92425]: pgmap v13513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:09:36.619+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:09:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:09:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:09:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:09:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:09:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:09:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:09:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:09:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:09:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:09:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:09:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:09:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:09:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:09:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:09:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:09:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:09:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:09:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:09:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:09:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:09:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:09:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:09:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:09:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:09:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:09:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:09:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:09:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:09:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:09:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:09:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:09:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:09:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:09:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:09:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:09:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:09:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:09:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:09:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:09:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:09:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:09:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:09:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:09:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:09:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=795] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:09:36.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=795] Local address: 172.21.15.161, port 58876 2022-07-03T18:09:36.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=795] Peer address: 172.21.15.101, port 22 2022-07-03T18:09:36.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=795] Connection lost 2022-07-03T18:09:36.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=795] Aborting connection 2022-07-03T18:09:36.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:09:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:09:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:09:37.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:09:37.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:09:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:09:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:09:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:09:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:09:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:09:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:09:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:09:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:09:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:09:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:09:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:09:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:09:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:09:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:09:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:09:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:09:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:09:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:09:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:09:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:09:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:09:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:09:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:09:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:09:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:09:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:09:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:09:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:09:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:09:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:09:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:09:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:09:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:09:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:09:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:09:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:09:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:09:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:09:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:09:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:09:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:09:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:09:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:09:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:09:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:09:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:09:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:09:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:09:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:09:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=795] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:09:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=795] Local address: 172.21.15.161, port 58876 2022-07-03T18:09:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=795] Peer address: 172.21.15.101, port 22 2022-07-03T18:09:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=795] Connection lost 2022-07-03T18:09:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=795] Aborting connection 2022-07-03T18:09:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:38 smithi161 ceph-mon[91579]: pgmap v13514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:09:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:09:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:09:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:09:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:09:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:09:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:09:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:09:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:09:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:09:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:09:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:09:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:09:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:09:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:09:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:09:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:09:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:09:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:09:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:09:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:09:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:09:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:09:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:09:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:09:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:09:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:09:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:09:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:09:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:09:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:09:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:09:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:09:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:09:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:09:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:09:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:09:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:09:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:09:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:09:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:09:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:09:38.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:09:38.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:09:38.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:09:38.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:09:38.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=795] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:09:38.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=795] Local address: 172.21.15.161, port 58876 2022-07-03T18:09:38.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=795] Peer address: 172.21.15.101, port 22 2022-07-03T18:09:38.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=795] Connection lost 2022-07-03T18:09:38.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=795] Aborting connection 2022-07-03T18:09:38.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:38 smithi188 ceph-mon[92425]: pgmap v13514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:39.398 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: pgmap v13515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: pgmap v13515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:09:40.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:40.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:09:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:41 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 33438k 2022-07-03T18:09:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:41 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 34241519: error parsing value: Value '34241519' is below minimum 939524096 2022-07-03T18:09:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:41 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 156.7M 2022-07-03T18:09:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:41 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 164415512: error parsing value: Value '164415512' is below minimum 939524096 2022-07-03T18:09:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:41 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 33438k 2022-07-03T18:09:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:41 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 34241519: error parsing value: Value '34241519' is below minimum 939524096 2022-07-03T18:09:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:41 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 156.7M 2022-07-03T18:09:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:41 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 164415512: error parsing value: Value '164415512' is below minimum 939524096 2022-07-03T18:09:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:42 smithi161 ceph-mon[91579]: pgmap v13516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:42 smithi188 ceph-mon[92425]: pgmap v13516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:43 smithi161 ceph-mon[91579]: pgmap v13517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:43 smithi188 ceph-mon[92425]: pgmap v13517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:46 smithi161 ceph-mon[91579]: pgmap v13518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:46.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:46 smithi188 ceph-mon[92425]: pgmap v13518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:48 smithi161 ceph-mon[91579]: pgmap v13519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:48 smithi188 ceph-mon[92425]: pgmap v13519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:50 smithi161 ceph-mon[91579]: pgmap v13520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:50 smithi188 ceph-mon[92425]: pgmap v13520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:52 smithi161 ceph-mon[91579]: pgmap v13521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:52 smithi188 ceph-mon[92425]: pgmap v13521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:53 smithi161 ceph-mon[91579]: pgmap v13522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:53 smithi188 ceph-mon[92425]: pgmap v13522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:09:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:56 smithi161 ceph-mon[91579]: pgmap v13523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:56 smithi188 ceph-mon[92425]: pgmap v13523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:58 smithi161 ceph-mon[91579]: pgmap v13524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:58 smithi188 ceph-mon[92425]: pgmap v13524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:09:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:09:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:09:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:09:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:09:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:09:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:09:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:10:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:00 smithi161 ceph-mon[91579]: pgmap v13525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:10:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:00 smithi188 ceph-mon[92425]: pgmap v13525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:10:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:02 smithi161 ceph-mon[91579]: pgmap v13526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:02 smithi188 ceph-mon[92425]: pgmap v13526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:03 smithi161 ceph-mon[91579]: pgmap v13527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:03 smithi188 ceph-mon[92425]: pgmap v13527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:06 smithi161 ceph-mon[91579]: pgmap v13528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:06 smithi188 ceph-mon[92425]: pgmap v13528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:08 smithi161 ceph-mon[91579]: pgmap v13529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:08 smithi188 ceph-mon[92425]: pgmap v13529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:10 smithi161 ceph-mon[91579]: pgmap v13530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:10 smithi188 ceph-mon[92425]: pgmap v13530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:12 smithi161 ceph-mon[91579]: pgmap v13531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:12 smithi188 ceph-mon[92425]: pgmap v13531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:13 smithi161 ceph-mon[91579]: pgmap v13532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:13 smithi188 ceph-mon[92425]: pgmap v13532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:16 smithi161 ceph-mon[91579]: pgmap v13533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:16 smithi188 ceph-mon[92425]: pgmap v13533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:18 smithi161 ceph-mon[91579]: pgmap v13534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:18 smithi188 ceph-mon[92425]: pgmap v13534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:20 smithi161 ceph-mon[91579]: pgmap v13535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:20 smithi188 ceph-mon[92425]: pgmap v13535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:22 smithi161 ceph-mon[91579]: pgmap v13536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:22.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:22 smithi188 ceph-mon[92425]: pgmap v13536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:23 smithi161 ceph-mon[91579]: pgmap v13537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:23 smithi188 ceph-mon[92425]: pgmap v13537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:26 smithi161 ceph-mon[91579]: pgmap v13538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:26 smithi188 ceph-mon[92425]: pgmap v13538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:28 smithi161 ceph-mon[91579]: pgmap v13539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:28 smithi188 ceph-mon[92425]: pgmap v13539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:30 smithi161 ceph-mon[91579]: pgmap v13540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:30 smithi188 ceph-mon[92425]: pgmap v13540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:32 smithi161 ceph-mon[91579]: pgmap v13541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:32 smithi188 ceph-mon[92425]: pgmap v13541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:33 smithi161 ceph-mon[91579]: pgmap v13542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:33 smithi188 ceph-mon[92425]: pgmap v13542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:36 smithi161 ceph-mon[91579]: pgmap v13543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:36 smithi188 ceph-mon[92425]: pgmap v13543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:38 smithi161 ceph-mon[91579]: pgmap v13544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:38 smithi188 ceph-mon[92425]: pgmap v13544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:40 smithi188 ceph-mon[92425]: pgmap v13545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:10:40.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:10:40.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:10:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:40 smithi161 ceph-mon[91579]: pgmap v13545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:10:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:10:40.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:10:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:10:39.877+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:10:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:10:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:10:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:10:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:10:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:10:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:10:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:10:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:10:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:10:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:10:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:10:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:10:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:10:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:10:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:10:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:10:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:10:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:10:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:10:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:10:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:10:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:10:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:10:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:10:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:10:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:10:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:10:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:10:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:10:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:10:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:10:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:10:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:10:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:10:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:10:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:10:40.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:10:40.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:10:40.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:10:40.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:10:40.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:10:40.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:10:40.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:10:40.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:10:40.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=797] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:10:40.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=797] Local address: 172.21.15.161, port 58880 2022-07-03T18:10:40.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=797] Peer address: 172.21.15.101, port 22 2022-07-03T18:10:40.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=797] Connection lost 2022-07-03T18:10:40.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=797] Aborting connection 2022-07-03T18:10:40.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:10:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:10:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:10:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:10:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:10:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:10:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:10:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:10:41.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:10:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:10:41.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:10:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:10:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:10:41.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:10:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:10:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:10:41.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:10:41.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:10:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:10:41.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:10:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:10:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:10:41.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:10:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:10:41.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:10:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:10:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:10:41.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:10:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:10:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:10:41.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:10:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:10:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:10:41.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:10:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:10:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:10:41.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:10:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:10:41.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:10:41.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:10:41.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:10:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:10:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:10:41.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:10:41.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:10:41.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:10:41.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:10:41.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=797] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:10:41.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=797] Local address: 172.21.15.161, port 58880 2022-07-03T18:10:41.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=797] Peer address: 172.21.15.101, port 22 2022-07-03T18:10:41.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=797] Connection lost 2022-07-03T18:10:41.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=797] Aborting connection 2022-07-03T18:10:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:10:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:10:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:10:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:10:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:10:41.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:10:41.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:10:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:10:41.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:10:41.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:10:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:10:41.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:10:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:10:41.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:10:41.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:10:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:10:41.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:10:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:10:41.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:10:41.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:10:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:10:41.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:10:41.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:10:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:10:41.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:10:41.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:10:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:10:41.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:10:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:10:41.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:10:41.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:10:41.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:10:41.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:10:41.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:10:41.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:10:41.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:10:41.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:10:41.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:10:41.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:10:41.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:10:41.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:10:41.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:10:41.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:10:41.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:10:41.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:10:41.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:10:41.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=797] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:10:41.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=797] Local address: 172.21.15.161, port 58880 2022-07-03T18:10:41.444 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=797] Peer address: 172.21.15.101, port 22 2022-07-03T18:10:41.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=797] Connection lost 2022-07-03T18:10:41.445 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=797] Aborting connection 2022-07-03T18:10:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:42 smithi161 ceph-mon[91579]: pgmap v13546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:42 smithi188 ceph-mon[92425]: pgmap v13546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:43 smithi161 ceph-mon[91579]: pgmap v13547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:43 smithi188 ceph-mon[92425]: pgmap v13547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:46 smithi161 ceph-mon[91579]: pgmap v13548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:46 smithi188 ceph-mon[92425]: pgmap v13548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:48 smithi161 ceph-mon[91579]: pgmap v13549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:48 smithi188 ceph-mon[92425]: pgmap v13549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:50 smithi161 ceph-mon[91579]: pgmap v13550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:50 smithi188 ceph-mon[92425]: pgmap v13550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:52 smithi161 ceph-mon[91579]: pgmap v13551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:52 smithi188 ceph-mon[92425]: pgmap v13551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:53 smithi161 ceph-mon[91579]: pgmap v13552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:53 smithi188 ceph-mon[92425]: pgmap v13552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:10:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:56 smithi161 ceph-mon[91579]: pgmap v13553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:56 smithi188 ceph-mon[92425]: pgmap v13553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:58 smithi161 ceph-mon[91579]: pgmap v13554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:58 smithi188 ceph-mon[92425]: pgmap v13554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:10:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:10:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:10:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:10:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:10:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:10:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:10:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:11:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:00 smithi161 ceph-mon[91579]: pgmap v13555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:00 smithi188 ceph-mon[92425]: pgmap v13555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:02 smithi161 ceph-mon[91579]: pgmap v13556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:02.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:02 smithi188 ceph-mon[92425]: pgmap v13556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:03 smithi161 ceph-mon[91579]: pgmap v13557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:04.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:04.173 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:03 smithi188 ceph-mon[92425]: pgmap v13557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:04.175 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:06 smithi161 ceph-mon[91579]: pgmap v13558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:06 smithi188 ceph-mon[92425]: pgmap v13558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:08 smithi161 ceph-mon[91579]: pgmap v13559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:08 smithi188 ceph-mon[92425]: pgmap v13559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:10 smithi161 ceph-mon[91579]: pgmap v13560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:10 smithi188 ceph-mon[92425]: pgmap v13560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:12 smithi161 ceph-mon[91579]: pgmap v13561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:12 smithi188 ceph-mon[92425]: pgmap v13561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:13 smithi161 ceph-mon[91579]: pgmap v13562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:13 smithi188 ceph-mon[92425]: pgmap v13562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:16 smithi161 ceph-mon[91579]: pgmap v13563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:16 smithi188 ceph-mon[92425]: pgmap v13563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:18 smithi161 ceph-mon[91579]: pgmap v13564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:18.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:18 smithi188 ceph-mon[92425]: pgmap v13564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:20 smithi161 ceph-mon[91579]: pgmap v13565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:20 smithi188 ceph-mon[92425]: pgmap v13565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:22 smithi161 ceph-mon[91579]: pgmap v13566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:22.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:22 smithi188 ceph-mon[92425]: pgmap v13566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:24.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:23 smithi161 ceph-mon[91579]: pgmap v13567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:24.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:23 smithi188 ceph-mon[92425]: pgmap v13567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:26.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:26 smithi161 ceph-mon[91579]: pgmap v13568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:26 smithi188 ceph-mon[92425]: pgmap v13568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:28 smithi161 ceph-mon[91579]: pgmap v13569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:28 smithi188 ceph-mon[92425]: pgmap v13569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:30 smithi161 ceph-mon[91579]: pgmap v13570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:30.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:30 smithi188 ceph-mon[92425]: pgmap v13570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:32 smithi161 ceph-mon[91579]: pgmap v13571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:32 smithi188 ceph-mon[92425]: pgmap v13571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:33 smithi161 ceph-mon[91579]: pgmap v13572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:33 smithi188 ceph-mon[92425]: pgmap v13572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:36.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:36 smithi161 ceph-mon[91579]: pgmap v13573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:36 smithi188 ceph-mon[92425]: pgmap v13573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:38 smithi161 ceph-mon[91579]: pgmap v13574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:38 smithi188 ceph-mon[92425]: pgmap v13574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:40.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-mon[91579]: pgmap v13575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:40.397 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:40 smithi188 ceph-mon[92425]: pgmap v13575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:40.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:11:40.383+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:11:40.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:11:40.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:11:40.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:11:40.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:11:40.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:11:40.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:11:40.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:11:40.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:11:40.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:11:40.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:11:40.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:11:40.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:11:40.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:11:40.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:11:40.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:11:40.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:11:40.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:11:40.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:11:40.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:11:40.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:11:40.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:11:40.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:11:40.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:11:40.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:11:40.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:11:40.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:11:40.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:11:40.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:11:40.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:11:40.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:11:40.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:11:40.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:11:40.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:11:40.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:11:40.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:11:40.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:11:40.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:11:40.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:11:40.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:11:40.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:11:40.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:11:40.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:11:40.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:11:40.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:11:40.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=799] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:11:40.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=799] Local address: 172.21.15.161, port 58884 2022-07-03T18:11:40.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=799] Peer address: 172.21.15.101, port 22 2022-07-03T18:11:40.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=799] Connection lost 2022-07-03T18:11:40.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=799] Aborting connection 2022-07-03T18:11:40.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:11:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:11:41.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:11:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:11:41.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:11:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:11:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:11:41.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:11:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:42 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:11:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:11:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:11:42.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:11:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:11:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:11:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:11:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:11:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:11:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:11:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:11:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:11:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:11:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:11:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:11:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:11:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:11:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:11:42.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:11:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:11:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:11:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:11:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:11:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:11:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:11:42.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:11:42.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:11:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:11:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:11:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:11:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:11:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:11:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:11:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:11:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:11:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:11:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:11:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:11:42.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:11:42.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:11:42.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:11:42.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:11:42.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:11:42.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:11:42.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:11:42.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:11:42.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=799] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:11:42.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=799] Local address: 172.21.15.161, port 58884 2022-07-03T18:11:42.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=799] Peer address: 172.21.15.101, port 22 2022-07-03T18:11:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=799] Connection lost 2022-07-03T18:11:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=799] Aborting connection 2022-07-03T18:11:42.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:42 smithi161 ceph-mon[91579]: pgmap v13576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:42 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:11:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:11:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:11:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:11:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:11:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:11:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:11:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:11:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:11:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:11:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:11:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:11:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:11:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:11:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:11:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:11:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:11:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:11:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:11:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:11:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:11:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:11:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:11:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:11:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:11:42.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:11:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:11:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:11:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:11:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:11:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:11:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:11:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:11:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:11:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:11:42.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:11:42.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:11:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:11:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:11:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:11:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:11:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:11:42.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:11:42.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:11:42.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:11:42.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:11:42.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=799] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:11:42.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=799] Local address: 172.21.15.161, port 58884 2022-07-03T18:11:42.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=799] Peer address: 172.21.15.101, port 22 2022-07-03T18:11:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=799] Connection lost 2022-07-03T18:11:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=799] Aborting connection 2022-07-03T18:11:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:42 smithi188 ceph-mon[92425]: pgmap v13576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:43 smithi161 ceph-mon[91579]: pgmap v13577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:44.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:43 smithi188 ceph-mon[92425]: pgmap v13577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:46 smithi161 ceph-mon[91579]: pgmap v13578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:46 smithi188 ceph-mon[92425]: pgmap v13578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:48 smithi161 ceph-mon[91579]: pgmap v13579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:48 smithi188 ceph-mon[92425]: pgmap v13579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:50 smithi161 ceph-mon[91579]: pgmap v13580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:50 smithi188 ceph-mon[92425]: pgmap v13580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:52 smithi161 ceph-mon[91579]: pgmap v13581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:52.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:52 smithi188 ceph-mon[92425]: pgmap v13581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:53 smithi161 ceph-mon[91579]: pgmap v13582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:53 smithi188 ceph-mon[92425]: pgmap v13582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:11:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:56 smithi161 ceph-mon[91579]: pgmap v13583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:56 smithi188 ceph-mon[92425]: pgmap v13583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:58 smithi161 ceph-mon[91579]: pgmap v13584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:58 smithi188 ceph-mon[92425]: pgmap v13584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:11:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:11:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:11:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:11:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:11:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:11:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:11:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:11:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:11:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:12:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:00 smithi161 ceph-mon[91579]: pgmap v13585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:00 smithi188 ceph-mon[92425]: pgmap v13585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:02 smithi161 ceph-mon[91579]: pgmap v13586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:02 smithi188 ceph-mon[92425]: pgmap v13586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:03 smithi161 ceph-mon[91579]: pgmap v13587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:03 smithi188 ceph-mon[92425]: pgmap v13587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:06 smithi161 ceph-mon[91579]: pgmap v13588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:06 smithi188 ceph-mon[92425]: pgmap v13588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:08 smithi161 ceph-mon[91579]: pgmap v13589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:08 smithi188 ceph-mon[92425]: pgmap v13589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:12:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:10 smithi161 ceph-mon[91579]: pgmap v13590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:10 smithi188 ceph-mon[92425]: pgmap v13590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:12 smithi161 ceph-mon[91579]: pgmap v13591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:12 smithi188 ceph-mon[92425]: pgmap v13591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:13 smithi161 ceph-mon[91579]: pgmap v13592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:13 smithi188 ceph-mon[92425]: pgmap v13592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:16 smithi161 ceph-mon[91579]: pgmap v13593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:16 smithi188 ceph-mon[92425]: pgmap v13593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:18 smithi161 ceph-mon[91579]: pgmap v13594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:18 smithi188 ceph-mon[92425]: pgmap v13594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:20 smithi161 ceph-mon[91579]: pgmap v13595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:20 smithi188 ceph-mon[92425]: pgmap v13595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:22 smithi161 ceph-mon[91579]: pgmap v13596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:22 smithi188 ceph-mon[92425]: pgmap v13596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:23 smithi161 ceph-mon[91579]: pgmap v13597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:23 smithi188 ceph-mon[92425]: pgmap v13597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:26 smithi161 ceph-mon[91579]: pgmap v13598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:26 smithi188 ceph-mon[92425]: pgmap v13598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:28 smithi161 ceph-mon[91579]: pgmap v13599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:28 smithi188 ceph-mon[92425]: pgmap v13599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:30 smithi161 ceph-mon[91579]: pgmap v13600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:30.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:30 smithi188 ceph-mon[92425]: pgmap v13600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:32 smithi161 ceph-mon[91579]: pgmap v13601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:32 smithi188 ceph-mon[92425]: pgmap v13601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:33 smithi161 ceph-mon[91579]: pgmap v13602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:33 smithi188 ceph-mon[92425]: pgmap v13602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:36 smithi161 ceph-mon[91579]: pgmap v13603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:36 smithi188 ceph-mon[92425]: pgmap v13603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:38 smithi161 ceph-mon[91579]: pgmap v13604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:38 smithi188 ceph-mon[92425]: pgmap v13604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-mon[91579]: pgmap v13605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:40 smithi188 ceph-mon[92425]: pgmap v13605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:41.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:12:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:12:41.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:12:41.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:12:40.874+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:12:41.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:12:41.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:12:41.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:12:41.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:12:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:12:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:12:41.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:12:41.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:12:41.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:12:41.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:12:41.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:12:41.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:12:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:12:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:12:41.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:12:41.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:12:41.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:12:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:12:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:12:41.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:12:41.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:12:41.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:12:41.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:12:41.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:12:41.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:12:41.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:12:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:12:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:12:41.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:12:41.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:12:41.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:12:41.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:12:41.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:12:41.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:12:41.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:12:41.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:12:41.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:12:41.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:12:41.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:12:41.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:12:41.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:12:41.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:12:41.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:12:41.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:12:41.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=801] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:12:41.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=801] Local address: 172.21.15.161, port 58888 2022-07-03T18:12:41.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=801] Peer address: 172.21.15.101, port 22 2022-07-03T18:12:41.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=801] Connection lost 2022-07-03T18:12:41.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=801] Aborting connection 2022-07-03T18:12:41.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:12:41.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:12:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:12:41.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:12:42.376 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:42 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:12:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:12:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:12:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:12:42.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:12:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:12:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:12:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:12:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:12:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:12:42.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:12:42.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:12:42.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:12:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:12:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:12:42.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:12:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:12:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:12:42.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:12:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:12:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:12:42.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:12:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:12:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:12:42.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:12:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:12:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:12:42.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:12:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:12:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:12:42.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:12:42.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:12:42.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:12:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:12:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:12:42.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:12:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:12:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:12:42.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:12:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:12:42.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:12:42.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:12:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:12:42.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:12:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:12:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:12:42.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=801] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:12:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=801] Local address: 172.21.15.161, port 58888 2022-07-03T18:12:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=801] Peer address: 172.21.15.101, port 22 2022-07-03T18:12:42.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=801] Connection lost 2022-07-03T18:12:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=801] Aborting connection 2022-07-03T18:12:42.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:42 smithi161 ceph-mon[91579]: pgmap v13606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:42 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:12:42.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:12:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:12:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:12:42.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:12:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:12:42.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:12:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:12:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:12:42.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:12:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:12:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:12:42.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:12:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:12:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:12:42.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:12:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:12:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:12:42.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:12:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:12:42.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:12:42.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:12:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:12:42.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:12:42.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:12:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:12:42.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:12:42.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:12:42.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:12:42.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:12:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:12:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:12:42.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:12:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:12:42.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:12:42.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:12:42.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:12:42.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:12:42.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:12:42.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:12:42.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:12:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:12:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:12:42.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:12:42.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:12:42.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:12:42.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=801] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:12:42.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=801] Local address: 172.21.15.161, port 58888 2022-07-03T18:12:42.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=801] Peer address: 172.21.15.101, port 22 2022-07-03T18:12:42.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=801] Connection lost 2022-07-03T18:12:42.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=801] Aborting connection 2022-07-03T18:12:42.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:42 smithi188 ceph-mon[92425]: pgmap v13606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:44.099 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:43 smithi161 ceph-mon[91579]: pgmap v13607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:44.100 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:43 smithi188 ceph-mon[92425]: pgmap v13607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:45 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T18:12:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:12:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:12:45.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:45 smithi161 ceph-mon[91579]: pgmap v13608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:45 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T18:12:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:12:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:12:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:45 smithi188 ceph-mon[92425]: pgmap v13608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:48 smithi161 ceph-mon[91579]: pgmap v13609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:48 smithi188 ceph-mon[92425]: pgmap v13609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:50 smithi161 ceph-mon[91579]: pgmap v13610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:50 smithi188 ceph-mon[92425]: pgmap v13610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:52 smithi161 ceph-mon[91579]: pgmap v13611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:52 smithi188 ceph-mon[92425]: pgmap v13611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:53 smithi161 ceph-mon[91579]: pgmap v13612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:53 smithi188 ceph-mon[92425]: pgmap v13612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:12:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:56 smithi161 ceph-mon[91579]: pgmap v13613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:56 smithi188 ceph-mon[92425]: pgmap v13613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:58 smithi161 ceph-mon[91579]: pgmap v13614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:58 smithi188 ceph-mon[92425]: pgmap v13614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:12:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:12:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:12:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:12:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:12:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:12:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:12:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:12:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:12:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:13:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:00 smithi161 ceph-mon[91579]: pgmap v13615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:00 smithi188 ceph-mon[92425]: pgmap v13615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:02 smithi161 ceph-mon[91579]: pgmap v13616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:02 smithi188 ceph-mon[92425]: pgmap v13616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:03 smithi161 ceph-mon[91579]: pgmap v13617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:03 smithi188 ceph-mon[92425]: pgmap v13617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:06 smithi161 ceph-mon[91579]: pgmap v13618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:06 smithi188 ceph-mon[92425]: pgmap v13618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:08 smithi161 ceph-mon[91579]: pgmap v13619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:08 smithi188 ceph-mon[92425]: pgmap v13619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:10 smithi161 ceph-mon[91579]: pgmap v13620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:10 smithi188 ceph-mon[92425]: pgmap v13620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:12 smithi161 ceph-mon[91579]: pgmap v13621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:12 smithi188 ceph-mon[92425]: pgmap v13621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:13 smithi161 ceph-mon[91579]: pgmap v13622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:13 smithi188 ceph-mon[92425]: pgmap v13622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:16 smithi161 ceph-mon[91579]: pgmap v13623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:16 smithi188 ceph-mon[92425]: pgmap v13623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:18 smithi161 ceph-mon[91579]: pgmap v13624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:18.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:18 smithi188 ceph-mon[92425]: pgmap v13624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:20 smithi161 ceph-mon[91579]: pgmap v13625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:20 smithi188 ceph-mon[92425]: pgmap v13625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:22.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:22 smithi161 ceph-mon[91579]: pgmap v13626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:22 smithi188 ceph-mon[92425]: pgmap v13626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:23 smithi161 ceph-mon[91579]: pgmap v13627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:23 smithi188 ceph-mon[92425]: pgmap v13627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:26.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:26 smithi161 ceph-mon[91579]: pgmap v13628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:26 smithi188 ceph-mon[92425]: pgmap v13628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:28 smithi161 ceph-mon[91579]: pgmap v13629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:28 smithi188 ceph-mon[92425]: pgmap v13629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:30 smithi161 ceph-mon[91579]: pgmap v13630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:30 smithi188 ceph-mon[92425]: pgmap v13630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:32 smithi161 ceph-mon[91579]: pgmap v13631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:32 smithi188 ceph-mon[92425]: pgmap v13631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:33 smithi161 ceph-mon[91579]: pgmap v13632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:33 smithi188 ceph-mon[92425]: pgmap v13632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:36 smithi161 ceph-mon[91579]: pgmap v13633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:36 smithi188 ceph-mon[92425]: pgmap v13633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:38 smithi161 ceph-mon[91579]: pgmap v13634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:38 smithi188 ceph-mon[92425]: pgmap v13634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:40 smithi161 ceph-mon[91579]: pgmap v13635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:40 smithi188 ceph-mon[92425]: pgmap v13635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:42 smithi161 ceph-mon[91579]: pgmap v13636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:42 smithi188 ceph-mon[92425]: pgmap v13636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:43 smithi161 ceph-mon[91579]: pgmap v13637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:43 smithi188 ceph-mon[92425]: pgmap v13637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:13:44.242+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:13:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:13:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:13:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:13:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:13:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:13:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:13:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:13:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:13:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:13:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:13:44.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:13:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:13:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:13:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:13:44.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:13:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:13:44.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:13:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:13:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:13:44.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:13:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:13:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:13:44.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:13:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:13:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:13:44.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:13:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:13:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:13:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:13:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:13:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:13:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:13:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:13:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:13:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:13:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:13:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:13:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:13:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:13:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:13:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:13:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:13:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:13:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:13:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=803] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:13:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=803] Local address: 172.21.15.161, port 58892 2022-07-03T18:13:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=803] Peer address: 172.21.15.101, port 22 2022-07-03T18:13:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=803] Connection lost 2022-07-03T18:13:44.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=803] Aborting connection 2022-07-03T18:13:44.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:13:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:13:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:13:45.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:13:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:13:45.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:13:45.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:13:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:13:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:13:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:13:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:13:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:13:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:13:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:13:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:13:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:13:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:13:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:13:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:13:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:13:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:13:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:13:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:13:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:13:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:13:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:13:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:13:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:13:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:13:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:13:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:13:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:13:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:13:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:13:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:13:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:13:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:13:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:13:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:13:46.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:13:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:13:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:13:46.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:13:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:13:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:13:46.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:13:46.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:13:46.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:13:46.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:13:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:13:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:13:46.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:13:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:13:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=803] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:13:46.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=803] Local address: 172.21.15.161, port 58892 2022-07-03T18:13:46.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=803] Peer address: 172.21.15.101, port 22 2022-07-03T18:13:46.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=803] Connection lost 2022-07-03T18:13:46.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=803] Aborting connection 2022-07-03T18:13:46.155 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:45 smithi161 ceph-mon[91579]: pgmap v13638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:13:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:13:46.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:13:46.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:13:46.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:13:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:13:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:13:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:13:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:13:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:13:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:13:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:13:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:13:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:13:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:13:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:13:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:13:46.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:13:46.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:13:46.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:13:46.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:13:46.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:13:46.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:13:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:13:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:13:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:13:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:13:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:13:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:13:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:13:46.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:13:46.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:13:46.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:13:46.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:13:46.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:13:46.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:13:46.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:13:46.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:13:46.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:13:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:13:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:13:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:13:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:13:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:13:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:13:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:13:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=803] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:13:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=803] Local address: 172.21.15.161, port 58892 2022-07-03T18:13:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=803] Peer address: 172.21.15.101, port 22 2022-07-03T18:13:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=803] Connection lost 2022-07-03T18:13:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=803] Aborting connection 2022-07-03T18:13:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:45 smithi188 ceph-mon[92425]: pgmap v13638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:48 smithi161 ceph-mon[91579]: pgmap v13639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:48 smithi188 ceph-mon[92425]: pgmap v13639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:50.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:50 smithi161 ceph-mon[91579]: pgmap v13640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:50 smithi188 ceph-mon[92425]: pgmap v13640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:52 smithi161 ceph-mon[91579]: pgmap v13641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:52 smithi188 ceph-mon[92425]: pgmap v13641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:53 smithi161 ceph-mon[91579]: pgmap v13642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:53 smithi188 ceph-mon[92425]: pgmap v13642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:13:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:56 smithi161 ceph-mon[91579]: pgmap v13643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:56 smithi188 ceph-mon[92425]: pgmap v13643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:58 smithi161 ceph-mon[91579]: pgmap v13644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:58 smithi188 ceph-mon[92425]: pgmap v13644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:13:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:13:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:13:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:13:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:13:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:13:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:13:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:13:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:13:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:14:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:00 smithi161 ceph-mon[91579]: pgmap v13645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:00 smithi188 ceph-mon[92425]: pgmap v13645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:02 smithi161 ceph-mon[91579]: pgmap v13646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:02.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:02 smithi188 ceph-mon[92425]: pgmap v13646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:03 smithi161 ceph-mon[91579]: pgmap v13647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:03 smithi188 ceph-mon[92425]: pgmap v13647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:06 smithi161 ceph-mon[91579]: pgmap v13648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:06 smithi188 ceph-mon[92425]: pgmap v13648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:08 smithi161 ceph-mon[91579]: pgmap v13649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:08 smithi188 ceph-mon[92425]: pgmap v13649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:10 smithi161 ceph-mon[91579]: pgmap v13650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:10 smithi188 ceph-mon[92425]: pgmap v13650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:12 smithi161 ceph-mon[91579]: pgmap v13651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:12 smithi188 ceph-mon[92425]: pgmap v13651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:13 smithi161 ceph-mon[91579]: pgmap v13652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:13 smithi188 ceph-mon[92425]: pgmap v13652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:16 smithi161 ceph-mon[91579]: pgmap v13653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:16 smithi188 ceph-mon[92425]: pgmap v13653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:18.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:18 smithi161 ceph-mon[91579]: pgmap v13654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:18 smithi188 ceph-mon[92425]: pgmap v13654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:20 smithi161 ceph-mon[91579]: pgmap v13655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:20 smithi188 ceph-mon[92425]: pgmap v13655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:22.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:22 smithi161 ceph-mon[91579]: pgmap v13656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:22.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:22 smithi188 ceph-mon[92425]: pgmap v13656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:23 smithi161 ceph-mon[91579]: pgmap v13657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:23 smithi188 ceph-mon[92425]: pgmap v13657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:26 smithi161 ceph-mon[91579]: pgmap v13658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:26.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:26 smithi188 ceph-mon[92425]: pgmap v13658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:28 smithi161 ceph-mon[91579]: pgmap v13659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:28.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:28 smithi188 ceph-mon[92425]: pgmap v13659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:30 smithi161 ceph-mon[91579]: pgmap v13660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:30 smithi188 ceph-mon[92425]: pgmap v13660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:32 smithi161 ceph-mon[91579]: pgmap v13661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:32 smithi188 ceph-mon[92425]: pgmap v13661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:33 smithi161 ceph-mon[91579]: pgmap v13662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:33 smithi188 ceph-mon[92425]: pgmap v13662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:36.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:36 smithi161 ceph-mon[91579]: pgmap v13663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:36 smithi188 ceph-mon[92425]: pgmap v13663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:38 smithi161 ceph-mon[91579]: pgmap v13664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:38 smithi188 ceph-mon[92425]: pgmap v13664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:40.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:40 smithi161 ceph-mon[91579]: pgmap v13665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:40 smithi188 ceph-mon[92425]: pgmap v13665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:42 smithi161 ceph-mon[91579]: pgmap v13666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:42 smithi188 ceph-mon[92425]: pgmap v13666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:43 smithi161 ceph-mon[91579]: pgmap v13667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:43 smithi188 ceph-mon[92425]: pgmap v13667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:14:45.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:14:45.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:14:45.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:14:44.757+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:14:45.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:14:45.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:14:45.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:14:45.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:14:45.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:14:45.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:14:45.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:14:45.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:14:45.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:14:45.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:14:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:14:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:14:45.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:14:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:14:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:14:45.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:14:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:14:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:14:45.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:14:45.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:14:45.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:14:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:14:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:14:45.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:14:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:14:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:14:45.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:14:45.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:14:45.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:14:45.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:14:45.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:14:45.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:14:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:14:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:14:45.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:14:45.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:14:45.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:14:45.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:14:45.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:14:45.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:14:45.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:14:45.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:14:45.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:14:45.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:14:45.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=805] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:14:45.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=805] Local address: 172.21.15.161, port 58896 2022-07-03T18:14:45.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=805] Peer address: 172.21.15.101, port 22 2022-07-03T18:14:45.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=805] Connection lost 2022-07-03T18:14:45.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=805] Aborting connection 2022-07-03T18:14:45.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:14:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:14:45.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:14:45.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:14:45.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:14:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:14:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:14:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:14:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:14:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:14:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=805] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=805] Local address: 172.21.15.161, port 58896 2022-07-03T18:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=805] Peer address: 172.21.15.101, port 22 2022-07-03T18:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=805] Connection lost 2022-07-03T18:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=805] Aborting connection 2022-07-03T18:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:45 smithi161 ceph-mon[91579]: pgmap v13668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:14:46.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:14:46.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:14:46.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:14:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:14:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:14:46.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:14:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:14:46.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:14:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:14:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:14:46.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:14:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:14:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:14:46.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:14:46.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:14:46.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:14:46.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:14:46.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:14:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:14:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:14:46.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:14:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:14:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:14:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:14:46.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:14:46.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:14:46.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:14:46.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:14:46.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:14:46.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:14:46.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:14:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:14:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:14:46.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:14:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:14:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:14:46.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:14:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:14:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:14:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:14:46.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:14:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:14:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:14:46.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:14:46.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:14:46.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=805] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:14:46.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=805] Local address: 172.21.15.161, port 58896 2022-07-03T18:14:46.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=805] Peer address: 172.21.15.101, port 22 2022-07-03T18:14:46.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=805] Connection lost 2022-07-03T18:14:46.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=805] Aborting connection 2022-07-03T18:14:46.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:45 smithi188 ceph-mon[92425]: pgmap v13668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:48.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:48 smithi161 ceph-mon[91579]: pgmap v13669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:48.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:48 smithi188 ceph-mon[92425]: pgmap v13669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:50 smithi161 ceph-mon[91579]: pgmap v13670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:50.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:50 smithi188 ceph-mon[92425]: pgmap v13670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:51 smithi161 ceph-mon[91579]: pgmap v13671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:51 smithi188 ceph-mon[92425]: pgmap v13671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:53 smithi161 ceph-mon[91579]: pgmap v13672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:53 smithi188 ceph-mon[92425]: pgmap v13672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:14:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:56 smithi161 ceph-mon[91579]: pgmap v13673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:56 smithi188 ceph-mon[92425]: pgmap v13673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:58 smithi161 ceph-mon[91579]: pgmap v13674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:58 smithi188 ceph-mon[92425]: pgmap v13674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:14:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:14:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:14:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:14:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:14:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:14:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:14:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:14:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:14:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:15:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:00 smithi161 ceph-mon[91579]: pgmap v13675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:00 smithi188 ceph-mon[92425]: pgmap v13675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:01.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:01 smithi161 ceph-mon[91579]: pgmap v13676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:01 smithi188 ceph-mon[92425]: pgmap v13676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:03 smithi161 ceph-mon[91579]: pgmap v13677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:03 smithi188 ceph-mon[92425]: pgmap v13677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:06 smithi161 ceph-mon[91579]: pgmap v13678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:06 smithi188 ceph-mon[92425]: pgmap v13678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:08 smithi161 ceph-mon[91579]: pgmap v13679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:08 smithi188 ceph-mon[92425]: pgmap v13679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:10 smithi161 ceph-mon[91579]: pgmap v13680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:10 smithi188 ceph-mon[92425]: pgmap v13680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:11 smithi161 ceph-mon[91579]: pgmap v13681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:11 smithi188 ceph-mon[92425]: pgmap v13681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:13 smithi161 ceph-mon[91579]: pgmap v13682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:13 smithi188 ceph-mon[92425]: pgmap v13682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:16 smithi161 ceph-mon[91579]: pgmap v13683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:16 smithi188 ceph-mon[92425]: pgmap v13683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:18.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:18 smithi161 ceph-mon[91579]: pgmap v13684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:18 smithi188 ceph-mon[92425]: pgmap v13684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:20 smithi161 ceph-mon[91579]: pgmap v13685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:20 smithi188 ceph-mon[92425]: pgmap v13685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:21 smithi161 ceph-mon[91579]: pgmap v13686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:21 smithi188 ceph-mon[92425]: pgmap v13686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:23 smithi161 ceph-mon[91579]: pgmap v13687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:23 smithi188 ceph-mon[92425]: pgmap v13687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:26.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:26 smithi161 ceph-mon[91579]: pgmap v13688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:26 smithi188 ceph-mon[92425]: pgmap v13688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:28 smithi161 ceph-mon[91579]: pgmap v13689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:28.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:28 smithi188 ceph-mon[92425]: pgmap v13689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:29.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:30.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:30 smithi161 ceph-mon[91579]: pgmap v13690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:30 smithi188 ceph-mon[92425]: pgmap v13690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:31 smithi161 ceph-mon[91579]: pgmap v13691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:31 smithi188 ceph-mon[92425]: pgmap v13691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:33 smithi161 ceph-mon[91579]: pgmap v13692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:34.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:33 smithi188 ceph-mon[92425]: pgmap v13692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:36.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:36 smithi161 ceph-mon[91579]: pgmap v13693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:36 smithi188 ceph-mon[92425]: pgmap v13693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:38 smithi161 ceph-mon[91579]: pgmap v13694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:38 smithi188 ceph-mon[92425]: pgmap v13694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:39 smithi161 ceph-mon[91579]: pgmap v13695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:39 smithi188 ceph-mon[92425]: pgmap v13695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:42 smithi161 ceph-mon[91579]: pgmap v13696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:42.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:42 smithi188 ceph-mon[92425]: pgmap v13696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:43 smithi161 ceph-mon[91579]: pgmap v13697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:43 smithi188 ceph-mon[92425]: pgmap v13697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:45.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:15:45.259+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:15:45.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:15:45.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:15:45.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:15:45.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:15:45.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:15:45.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:15:45.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:15:45.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:15:45.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:15:45.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:15:45.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:15:45.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:15:45.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:15:45.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:15:45.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:15:45.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:15:45.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:15:45.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:15:45.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:15:45.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:15:45.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:15:45.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:15:45.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:15:45.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:15:45.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:15:45.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:15:45.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:15:45.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:15:45.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:15:45.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:15:45.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:15:45.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:15:45.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:15:45.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:15:45.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:15:45.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:15:45.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:15:45.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:15:45.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:15:45.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:15:45.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:15:45.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:15:45.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:15:45.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:15:45.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=807] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:15:45.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=807] Local address: 172.21.15.161, port 58900 2022-07-03T18:15:45.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=807] Peer address: 172.21.15.101, port 22 2022-07-03T18:15:45.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=807] Connection lost 2022-07-03T18:15:45.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=807] Aborting connection 2022-07-03T18:15:45.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:15:46.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:46 smithi161 ceph-mon[91579]: pgmap v13698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:15:46.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:15:46.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:15:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:46 smithi188 ceph-mon[92425]: pgmap v13698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:46.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:15:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:15:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:15:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:15:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:15:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:15:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:15:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:15:47.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:15:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:15:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:15:47.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:15:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:15:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:15:47.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:15:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:15:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:15:47.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:15:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:15:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:15:47.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:15:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:15:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:15:47.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:15:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:15:47.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:15:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:15:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:15:47.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:15:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:15:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:15:47.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:15:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:15:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:15:47.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:15:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:15:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:15:47.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:15:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:15:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:15:47.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:15:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:15:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:15:47.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:15:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:15:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:15:47.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:15:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:15:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:15:47.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=807] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:15:47.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=807] Local address: 172.21.15.161, port 58900 2022-07-03T18:15:47.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=807] Peer address: 172.21.15.101, port 22 2022-07-03T18:15:47.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=807] Connection lost 2022-07-03T18:15:47.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=807] Aborting connection 2022-07-03T18:15:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:15:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:15:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:15:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:15:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:15:47.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:15:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:15:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:15:47.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:15:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:15:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:15:47.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:15:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:15:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:15:47.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:15:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:15:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:15:47.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:15:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:15:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:15:47.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:15:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:15:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:15:47.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:15:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:15:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:15:47.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:15:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:15:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:15:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:15:47.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:15:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:15:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:15:47.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:15:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:15:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:15:47.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:15:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:15:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:15:47.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:15:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:15:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:15:47.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:15:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:15:47.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:15:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:15:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=807] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:15:47.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=807] Local address: 172.21.15.161, port 58900 2022-07-03T18:15:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=807] Peer address: 172.21.15.101, port 22 2022-07-03T18:15:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=807] Connection lost 2022-07-03T18:15:47.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=807] Aborting connection 2022-07-03T18:15:48.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:48 smithi161 ceph-mon[91579]: pgmap v13699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:48.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:48 smithi188 ceph-mon[92425]: pgmap v13699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:49 smithi188 ceph-mon[92425]: pgmap v13700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:49.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:49 smithi161 ceph-mon[91579]: pgmap v13700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:51 smithi161 ceph-mon[91579]: pgmap v13701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:51 smithi188 ceph-mon[92425]: pgmap v13701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:53 smithi161 ceph-mon[91579]: pgmap v13702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:53 smithi188 ceph-mon[92425]: pgmap v13702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:15:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:55 smithi161 ceph-mon[91579]: pgmap v13703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:55 smithi188 ceph-mon[92425]: pgmap v13703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:57 smithi161 ceph-mon[91579]: pgmap v13704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:57 smithi188 ceph-mon[92425]: pgmap v13704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:15:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:15:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:15:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:15:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:15:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:15:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:15:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:15:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:15:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:15:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:15:59 smithi161 ceph-mon[91579]: pgmap v13705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:15:59 smithi188 ceph-mon[92425]: pgmap v13705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:01 smithi161 ceph-mon[91579]: pgmap v13706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:01 smithi188 ceph-mon[92425]: pgmap v13706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:03 smithi161 ceph-mon[91579]: pgmap v13707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:03 smithi188 ceph-mon[92425]: pgmap v13707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:05 smithi161 ceph-mon[91579]: pgmap v13708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:05 smithi188 ceph-mon[92425]: pgmap v13708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:07 smithi161 ceph-mon[91579]: pgmap v13709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:07 smithi188 ceph-mon[92425]: pgmap v13709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:09 smithi161 ceph-mon[91579]: pgmap v13710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:09 smithi188 ceph-mon[92425]: pgmap v13710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:11 smithi161 ceph-mon[91579]: pgmap v13711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:11 smithi188 ceph-mon[92425]: pgmap v13711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:13 smithi161 ceph-mon[91579]: pgmap v13712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:13 smithi188 ceph-mon[92425]: pgmap v13712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:15 smithi161 ceph-mon[91579]: pgmap v13713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:15 smithi188 ceph-mon[92425]: pgmap v13713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:17 smithi161 ceph-mon[91579]: pgmap v13714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:17 smithi188 ceph-mon[92425]: pgmap v13714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:20.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:19 smithi161 ceph-mon[91579]: pgmap v13715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:19 smithi188 ceph-mon[92425]: pgmap v13715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:21 smithi161 ceph-mon[91579]: pgmap v13716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:21 smithi188 ceph-mon[92425]: pgmap v13716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:23 smithi161 ceph-mon[91579]: pgmap v13717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:23 smithi188 ceph-mon[92425]: pgmap v13717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:24.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:25 smithi161 ceph-mon[91579]: pgmap v13718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:25 smithi188 ceph-mon[92425]: pgmap v13718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:27 smithi161 ceph-mon[91579]: pgmap v13719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:27 smithi188 ceph-mon[92425]: pgmap v13719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:29 smithi161 ceph-mon[91579]: pgmap v13720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:29 smithi188 ceph-mon[92425]: pgmap v13720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:31 smithi161 ceph-mon[91579]: pgmap v13721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:31 smithi188 ceph-mon[92425]: pgmap v13721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:33 smithi161 ceph-mon[91579]: pgmap v13722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:33 smithi188 ceph-mon[92425]: pgmap v13722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:35 smithi161 ceph-mon[91579]: pgmap v13723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:35 smithi188 ceph-mon[92425]: pgmap v13723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:37 smithi161 ceph-mon[91579]: pgmap v13724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:37 smithi188 ceph-mon[92425]: pgmap v13724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:39.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:40.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:39 smithi161 ceph-mon[91579]: pgmap v13725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:39 smithi188 ceph-mon[92425]: pgmap v13725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:41 smithi161 ceph-mon[91579]: pgmap v13726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:41 smithi188 ceph-mon[92425]: pgmap v13726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:43 smithi161 ceph-mon[91579]: pgmap v13727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:43 smithi188 ceph-mon[92425]: pgmap v13727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-mon[91579]: pgmap v13728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:45 smithi188 ceph-mon[92425]: pgmap v13728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:46.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:16:45.756+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:16:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:16:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:16:46.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:16:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:16:46.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:16:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:16:46.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:16:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:16:46.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:16:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:16:46.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:16:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:16:46.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:16:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:16:46.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:16:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:16:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:16:46.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:16:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:16:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:16:46.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:16:46.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:16:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:16:46.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:16:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:16:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:16:46.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:16:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:16:46.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:16:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:16:46.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:16:46.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:16:46.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:16:46.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:16:46.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:16:46.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:16:46.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:16:46.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:16:46.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:16:46.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:16:46.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:16:46.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:16:46.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:16:46.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:16:46.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=809] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:16:46.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=809] Local address: 172.21.15.161, port 58904 2022-07-03T18:16:46.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=809] Peer address: 172.21.15.101, port 22 2022-07-03T18:16:46.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=809] Connection lost 2022-07-03T18:16:46.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=809] Aborting connection 2022-07-03T18:16:46.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:16:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:16:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:16:46.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:16:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:16:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:16:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:16:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:16:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:16:46.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:16:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:16:46.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:16:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:16:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:16:46.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:16:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:16:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:16:46.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:16:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:16:46.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:16:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:16:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:16:46.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:16:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:16:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:16:46.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:16:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:16:46.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:16:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:16:46.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:16:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:16:46.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:16:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:16:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:16:46.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:16:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:16:46.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:16:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:16:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:16:46.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:16:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:16:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:16:46.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:16:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:16:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:16:46.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:16:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:16:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:16:46.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:16:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:16:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:16:46.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=809] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:16:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=809] Local address: 172.21.15.161, port 58904 2022-07-03T18:16:46.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=809] Peer address: 172.21.15.101, port 22 2022-07-03T18:16:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=809] Connection lost 2022-07-03T18:16:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=809] Aborting connection 2022-07-03T18:16:46.651 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:16:46.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:16:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:16:46.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:16:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:16:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:16:46.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:16:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:16:46.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:16:46.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:16:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:16:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:16:46.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:16:46.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:16:46.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:16:46.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:16:46.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:16:46.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:16:46.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:16:46.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:16:46.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:16:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:16:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:16:46.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:16:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:16:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:16:46.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:16:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:16:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:16:46.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:16:46.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:16:46.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:16:46.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:16:46.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:16:46.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:16:46.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:16:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:16:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:16:46.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:16:46.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:16:46.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:16:46.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:16:46.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:16:46.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:16:46.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:16:46.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:16:46.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:16:46.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:16:46.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:16:46.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=809] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:16:46.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=809] Local address: 172.21.15.161, port 58904 2022-07-03T18:16:46.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=809] Peer address: 172.21.15.101, port 22 2022-07-03T18:16:46.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=809] Connection lost 2022-07-03T18:16:46.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=809] Aborting connection 2022-07-03T18:16:46.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.504 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.505 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.506 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:47 smithi188 ceph-mon[92425]: pgmap v13729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:47.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:47 smithi161 ceph-mon[91579]: pgmap v13729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:49.130 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:49 smithi161 ceph-mon[91579]: pgmap v13730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:49 smithi188 ceph-mon[92425]: pgmap v13730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:50.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:16:51.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:50 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T18:16:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:50 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T18:16:52.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:51 smithi161 ceph-mon[91579]: pgmap v13731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:51 smithi188 ceph-mon[92425]: pgmap v13731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:53 smithi161 ceph-mon[91579]: pgmap v13732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:54.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:53 smithi188 ceph-mon[92425]: pgmap v13732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:16:54.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:55 smithi161 ceph-mon[91579]: pgmap v13733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:55 smithi188 ceph-mon[92425]: pgmap v13733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:57 smithi161 ceph-mon[91579]: pgmap v13734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:57 smithi188 ceph-mon[92425]: pgmap v13734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:16:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:16:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:16:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:16:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:16:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:16:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:16:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:16:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:16:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:16:59 smithi161 ceph-mon[91579]: pgmap v13735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:16:59 smithi188 ceph-mon[92425]: pgmap v13735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:01 smithi161 ceph-mon[91579]: pgmap v13736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:01 smithi188 ceph-mon[92425]: pgmap v13736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:03 smithi161 ceph-mon[91579]: pgmap v13737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:03 smithi188 ceph-mon[92425]: pgmap v13737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:05 smithi161 ceph-mon[91579]: pgmap v13738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:05 smithi188 ceph-mon[92425]: pgmap v13738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:07 smithi161 ceph-mon[91579]: pgmap v13739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:07 smithi188 ceph-mon[92425]: pgmap v13739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:09.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:09 smithi161 ceph-mon[91579]: pgmap v13740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:09 smithi188 ceph-mon[92425]: pgmap v13740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:11 smithi161 ceph-mon[91579]: pgmap v13741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:11 smithi188 ceph-mon[92425]: pgmap v13741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:13 smithi161 ceph-mon[91579]: pgmap v13742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:13 smithi188 ceph-mon[92425]: pgmap v13742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:15 smithi161 ceph-mon[91579]: pgmap v13743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:15 smithi188 ceph-mon[92425]: pgmap v13743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:17 smithi161 ceph-mon[91579]: pgmap v13744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:17 smithi188 ceph-mon[92425]: pgmap v13744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:19 smithi161 ceph-mon[91579]: pgmap v13745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:19 smithi188 ceph-mon[92425]: pgmap v13745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:21 smithi161 ceph-mon[91579]: pgmap v13746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:21 smithi188 ceph-mon[92425]: pgmap v13746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:23 smithi161 ceph-mon[91579]: pgmap v13747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:23 smithi188 ceph-mon[92425]: pgmap v13747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:25 smithi161 ceph-mon[91579]: pgmap v13748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:25 smithi188 ceph-mon[92425]: pgmap v13748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:27 smithi161 ceph-mon[91579]: pgmap v13749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:27 smithi188 ceph-mon[92425]: pgmap v13749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:29 smithi161 ceph-mon[91579]: pgmap v13750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:29 smithi188 ceph-mon[92425]: pgmap v13750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:31 smithi161 ceph-mon[91579]: pgmap v13751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:31 smithi188 ceph-mon[92425]: pgmap v13751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:33 smithi161 ceph-mon[91579]: pgmap v13752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:33 smithi188 ceph-mon[92425]: pgmap v13752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:35 smithi161 ceph-mon[91579]: pgmap v13753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:35 smithi188 ceph-mon[92425]: pgmap v13753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:37 smithi161 ceph-mon[91579]: pgmap v13754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:37 smithi188 ceph-mon[92425]: pgmap v13754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:39.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:39 smithi161 ceph-mon[91579]: pgmap v13755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:39 smithi188 ceph-mon[92425]: pgmap v13755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:41 smithi161 ceph-mon[91579]: pgmap v13756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:41 smithi188 ceph-mon[92425]: pgmap v13756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:43 smithi161 ceph-mon[91579]: pgmap v13757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:43 smithi188 ceph-mon[92425]: pgmap v13757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:45 smithi161 ceph-mon[91579]: pgmap v13758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:45 smithi188 ceph-mon[92425]: pgmap v13758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:47 smithi161 ceph-mon[91579]: pgmap v13759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:47 smithi188 ceph-mon[92425]: pgmap v13759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:49.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:17:49.566+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:17:49.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:17:49.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:17:49.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:17:49.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:17:49.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:17:49.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:17:49.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:17:49.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:17:49.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:17:49.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:17:49.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:17:49.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:17:49.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:17:49.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:17:49.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:17:49.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:17:49.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:17:49.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:17:49.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:17:49.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:17:49.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:17:49.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:17:49.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:17:49.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:17:49.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:17:49.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:17:49.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:17:49.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:17:49.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:17:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:17:49.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:17:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:17:49.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:17:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:17:49.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:17:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:17:49.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:17:49.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:17:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:17:49.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:17:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:17:49.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:17:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:17:49.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:17:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=811] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:17:49.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=811] Local address: 172.21.15.161, port 58908 2022-07-03T18:17:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=811] Peer address: 172.21.15.101, port 22 2022-07-03T18:17:49.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=811] Connection lost 2022-07-03T18:17:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=811] Aborting connection 2022-07-03T18:17:49.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:17:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-mon[91579]: pgmap v13760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:50.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:17:50.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:17:50.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:17:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:49 smithi188 ceph-mon[92425]: pgmap v13760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:17:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:17:50.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:17:51.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:17:51.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:17:51.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:17:51.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:17:51.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:17:51.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:17:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:17:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:17:51.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:17:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:17:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:17:51.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:17:51.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:17:51.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:17:51.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:17:51.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:17:51.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:17:51.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:17:51.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:17:51.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:17:51.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:17:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:17:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:17:51.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:17:51.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:17:51.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:17:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:17:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:17:51.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:17:51.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:17:51.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:17:51.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:17:51.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:17:51.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:17:51.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:17:51.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:17:51.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:17:51.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:17:51.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:17:51.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:17:51.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:17:51.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:17:51.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:17:51.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:17:51.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:17:51.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:17:51.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=811] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:17:51.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=811] Local address: 172.21.15.161, port 58908 2022-07-03T18:17:51.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=811] Peer address: 172.21.15.101, port 22 2022-07-03T18:17:51.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=811] Connection lost 2022-07-03T18:17:51.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=811] Aborting connection 2022-07-03T18:17:51.152 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.153 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.154 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:50 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:17:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:17:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:17:51.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:17:51.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:17:51.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:17:51.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:17:51.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:17:51.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:17:51.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:17:51.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:17:51.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:17:51.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:17:51.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:17:51.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:17:51.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:17:51.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:17:51.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:17:51.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:17:51.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:17:51.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:17:51.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:17:51.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:17:51.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:17:51.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:17:51.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:17:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:17:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:17:51.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:17:51.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:17:51.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:17:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:17:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:17:51.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:17:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:17:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:17:51.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:17:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:17:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:17:51.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:17:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:17:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:17:51.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:17:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:17:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:17:51.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:17:51.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=811] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:17:51.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=811] Local address: 172.21.15.161, port 58908 2022-07-03T18:17:51.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=811] Peer address: 172.21.15.101, port 22 2022-07-03T18:17:51.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=811] Connection lost 2022-07-03T18:17:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=811] Aborting connection 2022-07-03T18:17:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:51.187 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:50 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:17:52.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:51 smithi161 ceph-mon[91579]: pgmap v13761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:52.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:51 smithi188 ceph-mon[92425]: pgmap v13761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:53 smithi161 ceph-mon[91579]: pgmap v13762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:53 smithi188 ceph-mon[92425]: pgmap v13762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:17:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:55 smithi161 ceph-mon[91579]: pgmap v13763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:55 smithi188 ceph-mon[92425]: pgmap v13763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:57 smithi161 ceph-mon[91579]: pgmap v13764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:57 smithi188 ceph-mon[92425]: pgmap v13764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:17:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:17:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:17:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:17:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:17:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:17:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:17:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:17:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:17:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:17:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:17:59 smithi161 ceph-mon[91579]: pgmap v13765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:17:59 smithi188 ceph-mon[92425]: pgmap v13765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:01 smithi161 ceph-mon[91579]: pgmap v13766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:01 smithi188 ceph-mon[92425]: pgmap v13766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:04.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:03 smithi161 ceph-mon[91579]: pgmap v13767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:04.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:03 smithi188 ceph-mon[92425]: pgmap v13767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:05 smithi161 ceph-mon[91579]: pgmap v13768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:05 smithi188 ceph-mon[92425]: pgmap v13768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:07 smithi161 ceph-mon[91579]: pgmap v13769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:07 smithi188 ceph-mon[92425]: pgmap v13769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:09 smithi161 ceph-mon[91579]: pgmap v13770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:09 smithi188 ceph-mon[92425]: pgmap v13770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:11 smithi161 ceph-mon[91579]: pgmap v13771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:11 smithi188 ceph-mon[92425]: pgmap v13771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:13 smithi161 ceph-mon[91579]: pgmap v13772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:13 smithi188 ceph-mon[92425]: pgmap v13772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:15 smithi161 ceph-mon[91579]: pgmap v13773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:15 smithi188 ceph-mon[92425]: pgmap v13773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:17 smithi161 ceph-mon[91579]: pgmap v13774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:17 smithi188 ceph-mon[92425]: pgmap v13774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:19.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:20.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:19 smithi161 ceph-mon[91579]: pgmap v13775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:19 smithi188 ceph-mon[92425]: pgmap v13775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:21 smithi161 ceph-mon[91579]: pgmap v13776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:21 smithi188 ceph-mon[92425]: pgmap v13776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:24.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:23 smithi161 ceph-mon[91579]: pgmap v13777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:24.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:23 smithi188 ceph-mon[92425]: pgmap v13777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:25 smithi161 ceph-mon[91579]: pgmap v13778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:25 smithi188 ceph-mon[92425]: pgmap v13778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:27 smithi161 ceph-mon[91579]: pgmap v13779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:27 smithi188 ceph-mon[92425]: pgmap v13779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:29.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:29 smithi161 ceph-mon[91579]: pgmap v13780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:29 smithi188 ceph-mon[92425]: pgmap v13780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:31 smithi161 ceph-mon[91579]: pgmap v13781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:31 smithi188 ceph-mon[92425]: pgmap v13781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:34.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:33 smithi161 ceph-mon[91579]: pgmap v13782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:34.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:33 smithi188 ceph-mon[92425]: pgmap v13782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:35 smithi161 ceph-mon[91579]: pgmap v13783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:35 smithi188 ceph-mon[92425]: pgmap v13783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:37 smithi161 ceph-mon[91579]: pgmap v13784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:37 smithi188 ceph-mon[92425]: pgmap v13784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:39.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:39.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:40.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:39 smithi161 ceph-mon[91579]: pgmap v13785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:40.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:39 smithi188 ceph-mon[92425]: pgmap v13785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:41 smithi161 ceph-mon[91579]: pgmap v13786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:41 smithi188 ceph-mon[92425]: pgmap v13786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:43 smithi161 ceph-mon[91579]: pgmap v13787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:43 smithi188 ceph-mon[92425]: pgmap v13787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:45 smithi161 ceph-mon[91579]: pgmap v13788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:45 smithi188 ceph-mon[92425]: pgmap v13788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:47 smithi161 ceph-mon[91579]: pgmap v13789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:47 smithi188 ceph-mon[92425]: pgmap v13789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:50.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:49 smithi161 ceph-mon[91579]: pgmap v13790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:50.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:49 smithi188 ceph-mon[92425]: pgmap v13790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:50.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:18:50.484+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:18:50.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:18:50.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:18:50.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:18:50.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:18:50.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:18:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:18:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:18:50.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:18:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:18:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:18:50.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:18:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:18:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:18:50.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:18:50.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:18:50.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:18:50.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:18:50.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:18:50.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:18:50.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:18:50.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:18:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:18:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:18:50.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:18:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:18:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:18:50.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:18:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:18:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:18:50.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:18:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:18:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:18:50.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:18:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:18:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:18:50.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:18:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:18:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:18:50.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:18:50.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:18:50.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:18:50.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:18:50.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:18:50.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:18:50.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=813] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:18:50.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=813] Local address: 172.21.15.161, port 58912 2022-07-03T18:18:50.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=813] Peer address: 172.21.15.101, port 22 2022-07-03T18:18:50.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=813] Connection lost 2022-07-03T18:18:50.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=813] Aborting connection 2022-07-03T18:18:50.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:18:51.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:18:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:18:51.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:18:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:18:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:18:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:18:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:51 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:18:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:51 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:18:52.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:18:52.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:18:52.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:18:52.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:18:52.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:18:52.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:18:52.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:18:52.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:18:52.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:18:52.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:18:52.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:18:52.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:18:52.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:18:52.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:18:52.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:18:52.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:18:52.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:18:52.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:18:52.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:18:52.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:18:52.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:18:52.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:18:52.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:18:52.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:18:52.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:18:52.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:18:52.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:18:52.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:18:52.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:18:52.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:18:52.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:18:52.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:18:52.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:18:52.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:18:52.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:18:52.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:18:52.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:18:52.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:18:52.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:18:52.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:18:52.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:18:52.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:18:52.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:18:52.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:18:52.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=813] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:18:52.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=813] Local address: 172.21.15.161, port 58912 2022-07-03T18:18:52.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=813] Peer address: 172.21.15.101, port 22 2022-07-03T18:18:52.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=813] Connection lost 2022-07-03T18:18:52.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=813] Aborting connection 2022-07-03T18:18:52.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:51 smithi188 ceph-mon[92425]: pgmap v13791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:52.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:51 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:18:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:51 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:18:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:18:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:18:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:18:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:18:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:18:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:18:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:18:52.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:18:52.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:18:52.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:18:52.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:18:52.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:18:52.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:18:52.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:18:52.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:18:52.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:18:52.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:18:52.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:18:52.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:18:52.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:18:52.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:18:52.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:18:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:18:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:18:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:18:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:18:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:18:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:18:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:18:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:18:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:18:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:18:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:18:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:18:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:18:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:18:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:18:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:18:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:18:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:18:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:18:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:18:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:18:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:18:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=813] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:18:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=813] Local address: 172.21.15.161, port 58912 2022-07-03T18:18:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=813] Peer address: 172.21.15.101, port 22 2022-07-03T18:18:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=813] Connection lost 2022-07-03T18:18:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=813] Aborting connection 2022-07-03T18:18:52.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:51 smithi161 ceph-mon[91579]: pgmap v13791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:53 smithi161 ceph-mon[91579]: pgmap v13792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:53 smithi188 ceph-mon[92425]: pgmap v13792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:18:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:55 smithi161 ceph-mon[91579]: pgmap v13793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:55 smithi188 ceph-mon[92425]: pgmap v13793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:57 smithi161 ceph-mon[91579]: pgmap v13794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:57 smithi188 ceph-mon[92425]: pgmap v13794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:18:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:18:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:18:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:18:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:18:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:18:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:18:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:18:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:18:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:18:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:18:59 smithi188 ceph-mon[92425]: pgmap v13795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:18:59 smithi161 ceph-mon[91579]: pgmap v13795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:01 smithi161 ceph-mon[91579]: pgmap v13796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:01 smithi188 ceph-mon[92425]: pgmap v13796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:04.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:03 smithi188 ceph-mon[92425]: pgmap v13797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:03 smithi161 ceph-mon[91579]: pgmap v13797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:05 smithi161 ceph-mon[91579]: pgmap v13798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:05 smithi188 ceph-mon[92425]: pgmap v13798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:07 smithi161 ceph-mon[91579]: pgmap v13799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:07 smithi188 ceph-mon[92425]: pgmap v13799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:09 smithi188 ceph-mon[92425]: pgmap v13800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:09 smithi161 ceph-mon[91579]: pgmap v13800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:11 smithi161 ceph-mon[91579]: pgmap v13801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:11 smithi188 ceph-mon[92425]: pgmap v13801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:13 smithi188 ceph-mon[92425]: pgmap v13802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:13 smithi161 ceph-mon[91579]: pgmap v13802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:15 smithi161 ceph-mon[91579]: pgmap v13803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:15 smithi188 ceph-mon[92425]: pgmap v13803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:17 smithi161 ceph-mon[91579]: pgmap v13804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:17 smithi188 ceph-mon[92425]: pgmap v13804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:19 smithi188 ceph-mon[92425]: pgmap v13805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:19 smithi161 ceph-mon[91579]: pgmap v13805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:21 smithi161 ceph-mon[91579]: pgmap v13806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:21 smithi188 ceph-mon[92425]: pgmap v13806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:24.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:23 smithi188 ceph-mon[92425]: pgmap v13807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:23 smithi161 ceph-mon[91579]: pgmap v13807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:25 smithi161 ceph-mon[91579]: pgmap v13808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:25 smithi188 ceph-mon[92425]: pgmap v13808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:27 smithi161 ceph-mon[91579]: pgmap v13809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:27 smithi188 ceph-mon[92425]: pgmap v13809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:29 smithi188 ceph-mon[92425]: pgmap v13810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:29 smithi161 ceph-mon[91579]: pgmap v13810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:31 smithi161 ceph-mon[91579]: pgmap v13811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:31 smithi188 ceph-mon[92425]: pgmap v13811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:33 smithi188 ceph-mon[92425]: pgmap v13812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:33 smithi161 ceph-mon[91579]: pgmap v13812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:35 smithi161 ceph-mon[91579]: pgmap v13813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:35 smithi188 ceph-mon[92425]: pgmap v13813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:37 smithi161 ceph-mon[91579]: pgmap v13814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:37 smithi188 ceph-mon[92425]: pgmap v13814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:39 smithi188 ceph-mon[92425]: pgmap v13815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:39 smithi161 ceph-mon[91579]: pgmap v13815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:41.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:41 smithi161 ceph-mon[91579]: pgmap v13816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:41 smithi188 ceph-mon[92425]: pgmap v13816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:43 smithi188 ceph-mon[92425]: pgmap v13817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:43 smithi161 ceph-mon[91579]: pgmap v13817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:45 smithi161 ceph-mon[91579]: pgmap v13818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:45.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:45 smithi188 ceph-mon[92425]: pgmap v13818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:47 smithi161 ceph-mon[91579]: pgmap v13819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:47 smithi188 ceph-mon[92425]: pgmap v13819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:49.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:49 smithi188 ceph-mon[92425]: pgmap v13820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:49 smithi161 ceph-mon[91579]: pgmap v13820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:19:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:19:51.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:19:51.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:19:50.987+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:19:51.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:19:51.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:19:51.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:19:51.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:19:51.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:19:51.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:19:51.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:19:51.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:19:51.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:19:51.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:19:51.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:19:51.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:19:51.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:19:51.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:19:51.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:19:51.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:19:51.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:19:51.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:19:51.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:19:51.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:19:51.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:19:51.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:19:51.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:19:51.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:19:51.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:19:51.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:19:51.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:19:51.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:19:51.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:19:51.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:19:51.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:19:51.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:19:51.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:19:51.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:19:51.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:19:51.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:19:51.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:19:51.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:19:51.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:19:51.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:19:51.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:19:51.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:19:51.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:19:51.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:19:51.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=815] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:19:51.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=815] Local address: 172.21.15.161, port 58916 2022-07-03T18:19:51.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=815] Peer address: 172.21.15.101, port 22 2022-07-03T18:19:51.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=815] Connection lost 2022-07-03T18:19:51.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=815] Aborting connection 2022-07-03T18:19:51.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:19:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:19:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:19:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:19:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:19:52.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:52 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:19:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:52 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:19:52.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:19:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:19:52.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:19:52.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:19:52.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:19:52.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:19:52.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:19:52.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:19:52.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:19:52.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:19:52.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:19:52.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:19:52.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:19:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:19:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:19:52.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:19:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:19:52.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:19:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:19:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:19:52.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:19:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:19:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:19:52.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:19:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:19:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:19:52.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:19:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:19:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:19:52.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:19:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:19:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:19:52.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:19:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:19:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:19:52.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:19:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:19:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:19:52.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:19:52.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:19:52.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:19:52.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:19:52.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:19:52.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:19:52.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=815] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:19:52.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=815] Local address: 172.21.15.161, port 58916 2022-07-03T18:19:52.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=815] Peer address: 172.21.15.101, port 22 2022-07-03T18:19:52.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=815] Connection lost 2022-07-03T18:19:52.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=815] Aborting connection 2022-07-03T18:19:52.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:52 smithi161 ceph-mon[91579]: pgmap v13821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:52 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:19:52.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:52 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:19:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:19:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:19:52.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:19:52.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:19:52.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:19:52.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:19:52.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:19:52.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:19:52.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:19:52.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:19:52.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:19:52.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:19:52.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:19:52.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:19:52.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:19:52.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:19:52.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:19:52.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:19:52.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:19:52.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:19:52.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:19:52.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:19:52.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:19:52.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:19:52.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:19:52.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:19:52.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:19:52.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:19:52.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:19:52.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:19:52.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:19:52.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:19:52.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:19:52.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:19:52.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:19:52.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:19:52.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:19:52.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:19:52.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:19:52.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:19:52.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:19:52.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:19:52.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:19:52.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:19:52.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=815] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:19:52.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=815] Local address: 172.21.15.161, port 58916 2022-07-03T18:19:52.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=815] Peer address: 172.21.15.101, port 22 2022-07-03T18:19:52.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=815] Connection lost 2022-07-03T18:19:52.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=815] Aborting connection 2022-07-03T18:19:52.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:52 smithi188 ceph-mon[92425]: pgmap v13821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: pgmap v13822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:54.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: pgmap v13822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:19:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:19:54.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 149.6M 2022-07-03T18:19:55.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 156899320: error parsing value: Value '156899320' is below minimum 939524096 2022-07-03T18:19:55.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 149.6M 2022-07-03T18:19:55.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 156899320: error parsing value: Value '156899320' is below minimum 939524096 2022-07-03T18:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:55.389 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:19:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:55 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 24263k 2022-07-03T18:19:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:55 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 24846278: error parsing value: Value '24846278' is below minimum 939524096 2022-07-03T18:19:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:55 smithi188 ceph-mon[92425]: pgmap v13823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:55 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 24263k 2022-07-03T18:19:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:55 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 24846278: error parsing value: Value '24846278' is below minimum 939524096 2022-07-03T18:19:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:55 smithi161 ceph-mon[91579]: pgmap v13823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:57 smithi161 ceph-mon[91579]: pgmap v13824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:57.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:57 smithi188 ceph-mon[92425]: pgmap v13824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:19:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:19:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:19:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:19:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:19:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:19:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:19:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:19:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:19:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:19:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:19:59 smithi188 ceph-mon[92425]: pgmap v13825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:19:59 smithi161 ceph-mon[91579]: pgmap v13825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:20:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:20:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:01 smithi188 ceph-mon[92425]: pgmap v13826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:01 smithi161 ceph-mon[91579]: pgmap v13826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:03 smithi188 ceph-mon[92425]: pgmap v13827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:04.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:03 smithi161 ceph-mon[91579]: pgmap v13827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:05 smithi161 ceph-mon[91579]: pgmap v13828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:05 smithi188 ceph-mon[92425]: pgmap v13828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:07 smithi161 ceph-mon[91579]: pgmap v13829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:07 smithi188 ceph-mon[92425]: pgmap v13829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:09 smithi188 ceph-mon[92425]: pgmap v13830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:09 smithi161 ceph-mon[91579]: pgmap v13830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:11 smithi161 ceph-mon[91579]: pgmap v13831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:11 smithi188 ceph-mon[92425]: pgmap v13831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:13 smithi188 ceph-mon[92425]: pgmap v13832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:13 smithi161 ceph-mon[91579]: pgmap v13832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:15 smithi161 ceph-mon[91579]: pgmap v13833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:15 smithi188 ceph-mon[92425]: pgmap v13833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:17 smithi161 ceph-mon[91579]: pgmap v13834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:17 smithi188 ceph-mon[92425]: pgmap v13834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:19.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:20.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:19 smithi188 ceph-mon[92425]: pgmap v13835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:19 smithi161 ceph-mon[91579]: pgmap v13835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:21 smithi161 ceph-mon[91579]: pgmap v13836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:21 smithi188 ceph-mon[92425]: pgmap v13836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:23 smithi188 ceph-mon[92425]: pgmap v13837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:24.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:23 smithi161 ceph-mon[91579]: pgmap v13837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:25 smithi161 ceph-mon[91579]: pgmap v13838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:25 smithi188 ceph-mon[92425]: pgmap v13838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:27 smithi161 ceph-mon[91579]: pgmap v13839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:27 smithi188 ceph-mon[92425]: pgmap v13839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:29.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:29 smithi188 ceph-mon[92425]: pgmap v13840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:29 smithi161 ceph-mon[91579]: pgmap v13840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:31 smithi161 ceph-mon[91579]: pgmap v13841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:31 smithi188 ceph-mon[92425]: pgmap v13841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:34.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:33 smithi188 ceph-mon[92425]: pgmap v13842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:34.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:33 smithi161 ceph-mon[91579]: pgmap v13842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:35 smithi161 ceph-mon[91579]: pgmap v13843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:35 smithi188 ceph-mon[92425]: pgmap v13843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:37 smithi161 ceph-mon[91579]: pgmap v13844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:37 smithi188 ceph-mon[92425]: pgmap v13844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:39.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:40.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:39 smithi188 ceph-mon[92425]: pgmap v13845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:39 smithi161 ceph-mon[91579]: pgmap v13845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:41 smithi161 ceph-mon[91579]: pgmap v13846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:41 smithi188 ceph-mon[92425]: pgmap v13846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:43 smithi188 ceph-mon[92425]: pgmap v13847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:43 smithi161 ceph-mon[91579]: pgmap v13847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:45 smithi161 ceph-mon[91579]: pgmap v13848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:45 smithi188 ceph-mon[92425]: pgmap v13848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:47 smithi161 ceph-mon[91579]: pgmap v13849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:47 smithi188 ceph-mon[92425]: pgmap v13849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:50.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:49 smithi188 ceph-mon[92425]: pgmap v13850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:49 smithi161 ceph-mon[91579]: pgmap v13850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:51 smithi161 ceph-mon[91579]: pgmap v13851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:51 smithi188 ceph-mon[92425]: pgmap v13851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:53 smithi188 ceph-mon[92425]: pgmap v13852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:54.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:54.251 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:53 smithi161 ceph-mon[91579]: pgmap v13852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:20:54.251 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:54.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:20:54.246+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:20:54.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:20:54.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:20:54.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:20:54.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:20:54.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:20:54.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:20:54.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:20:54.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:20:54.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:20:54.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:20:54.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:20:54.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:20:54.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:20:54.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:20:54.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:20:54.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:20:54.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:20:54.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:20:54.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:20:54.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:20:54.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:20:54.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:20:54.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:20:54.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:20:54.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:20:54.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:20:54.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:20:54.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:20:54.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:20:54.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:20:54.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:20:54.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:20:54.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:20:54.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:20:54.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:20:54.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:20:54.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:20:54.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:20:54.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:20:54.271 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:20:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:20:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:20:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:20:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:20:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=817] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:20:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=817] Local address: 172.21.15.161, port 58920 2022-07-03T18:20:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=817] Peer address: 172.21.15.101, port 22 2022-07-03T18:20:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=817] Connection lost 2022-07-03T18:20:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=817] Aborting connection 2022-07-03T18:20:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:20:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:20:55.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:20:55.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:20:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:20:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:20:55.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:20:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:20:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:20:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:20:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:20:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:20:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:20:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:20:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:20:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:20:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:20:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:20:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:20:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:20:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:20:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:20:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:20:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:20:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:20:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:20:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:20:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:20:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:20:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:20:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:20:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:20:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:20:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:20:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:20:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:20:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:20:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:20:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:20:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:20:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:20:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:20:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:20:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:20:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:20:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:20:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:20:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:20:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:20:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:20:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:20:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:20:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:20:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=817] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:20:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=817] Local address: 172.21.15.161, port 58920 2022-07-03T18:20:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=817] Peer address: 172.21.15.101, port 22 2022-07-03T18:20:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=817] Connection lost 2022-07-03T18:20:56.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=817] Aborting connection 2022-07-03T18:20:56.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:55 smithi161 ceph-mon[91579]: pgmap v13853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:20:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:20:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:20:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:20:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:20:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:20:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:20:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:20:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:20:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:20:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:20:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:20:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:20:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:20:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:20:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:20:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:20:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:20:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:20:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:20:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:20:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:20:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:20:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:20:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:20:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:20:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:20:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:20:56.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:20:56.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:20:56.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:20:56.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:20:56.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:20:56.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:20:56.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:20:56.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:20:56.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:20:56.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:20:56.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:20:56.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:20:56.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:20:56.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:20:56.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:20:56.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:20:56.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:20:56.441 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:20:56.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=817] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:20:56.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=817] Local address: 172.21.15.161, port 58920 2022-07-03T18:20:56.442 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=817] Peer address: 172.21.15.101, port 22 2022-07-03T18:20:56.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=817] Connection lost 2022-07-03T18:20:56.443 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=817] Aborting connection 2022-07-03T18:20:56.443 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:55 smithi188 ceph-mon[92425]: pgmap v13853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: pgmap v13854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:20:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:20:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:20:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:20:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:20:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: pgmap v13854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:20:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:20:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:20:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:20:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:20:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:00 smithi188 ceph-mon[92425]: pgmap v13855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:00 smithi161 ceph-mon[91579]: pgmap v13855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:03.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:02 smithi188 ceph-mon[92425]: pgmap v13856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:02 smithi161 ceph-mon[91579]: pgmap v13856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:04 smithi161 ceph-mon[91579]: pgmap v13857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:04 smithi188 ceph-mon[92425]: pgmap v13857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:06 smithi161 ceph-mon[91579]: pgmap v13858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:06 smithi188 ceph-mon[92425]: pgmap v13858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:08 smithi161 ceph-mon[91579]: pgmap v13859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:08 smithi188 ceph-mon[92425]: pgmap v13859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:10 smithi161 ceph-mon[91579]: pgmap v13860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:10 smithi188 ceph-mon[92425]: pgmap v13860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:12 smithi161 ceph-mon[91579]: pgmap v13861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:13.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:12 smithi188 ceph-mon[92425]: pgmap v13861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:14 smithi161 ceph-mon[91579]: pgmap v13862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:14 smithi188 ceph-mon[92425]: pgmap v13862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:16 smithi161 ceph-mon[91579]: pgmap v13863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:17.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:16 smithi188 ceph-mon[92425]: pgmap v13863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:18 smithi161 ceph-mon[91579]: pgmap v13864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:18 smithi188 ceph-mon[92425]: pgmap v13864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:20 smithi161 ceph-mon[91579]: pgmap v13865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:20 smithi188 ceph-mon[92425]: pgmap v13865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:22 smithi161 ceph-mon[91579]: pgmap v13866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:22 smithi188 ceph-mon[92425]: pgmap v13866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:24 smithi161 ceph-mon[91579]: pgmap v13867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:24 smithi188 ceph-mon[92425]: pgmap v13867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:26 smithi161 ceph-mon[91579]: pgmap v13868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:26 smithi188 ceph-mon[92425]: pgmap v13868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:28 smithi161 ceph-mon[91579]: pgmap v13869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:28 smithi188 ceph-mon[92425]: pgmap v13869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:30 smithi161 ceph-mon[91579]: pgmap v13870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:30 smithi188 ceph-mon[92425]: pgmap v13870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:32 smithi161 ceph-mon[91579]: pgmap v13871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:32 smithi188 ceph-mon[92425]: pgmap v13871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:34 smithi161 ceph-mon[91579]: pgmap v13872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:34 smithi188 ceph-mon[92425]: pgmap v13872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:36 smithi161 ceph-mon[91579]: pgmap v13873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:36 smithi188 ceph-mon[92425]: pgmap v13873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:38 smithi161 ceph-mon[91579]: pgmap v13874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:38 smithi188 ceph-mon[92425]: pgmap v13874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:40 smithi161 ceph-mon[91579]: pgmap v13875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:40 smithi188 ceph-mon[92425]: pgmap v13875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:42 smithi161 ceph-mon[91579]: pgmap v13876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:42 smithi188 ceph-mon[92425]: pgmap v13876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:44 smithi161 ceph-mon[91579]: pgmap v13877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:44 smithi188 ceph-mon[92425]: pgmap v13877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:47 smithi161 ceph-mon[91579]: pgmap v13878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:47 smithi188 ceph-mon[92425]: pgmap v13878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:49 smithi161 ceph-mon[91579]: pgmap v13879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:49 smithi188 ceph-mon[92425]: pgmap v13879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:51 smithi161 ceph-mon[91579]: pgmap v13880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:51 smithi188 ceph-mon[92425]: pgmap v13880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:53 smithi161 ceph-mon[91579]: pgmap v13881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:53 smithi188 ceph-mon[92425]: pgmap v13881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:55.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:21:54.755+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:21:55.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:21:55.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:21:55.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:21:55.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:21:55.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:21:55.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:21:55.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:21:55.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:21:55.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:21:55.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:21:55.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:21:55.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:21:55.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:21:55.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:21:55.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:21:55.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:21:55.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:21:55.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:21:55.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:21:55.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:21:55.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:21:55.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:21:55.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:21:55.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:21:55.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:21:55.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:21:55.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:21:55.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:21:55.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:21:55.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:21:55.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:21:55.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:21:55.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:21:55.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:21:55.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:21:55.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:21:55.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:21:55.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:21:55.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:21:55.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:21:55.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:21:55.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:21:55.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:21:55.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:21:55.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=819] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:21:55.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=819] Local address: 172.21.15.161, port 58924 2022-07-03T18:21:55.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=819] Peer address: 172.21.15.101, port 22 2022-07-03T18:21:55.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=819] Connection lost 2022-07-03T18:21:55.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=819] Aborting connection 2022-07-03T18:21:55.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:21:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:21:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:55 smithi161 ceph-mon[91579]: pgmap v13882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:21:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:21:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:21:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:55 smithi188 ceph-mon[92425]: pgmap v13882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:21:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:21:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:21:55.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:21:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:21:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:21:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:21:56.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:21:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:21:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:21:56.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:21:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:21:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:21:56.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:21:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:21:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:21:56.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:21:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:21:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:21:56.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:21:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:21:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:21:56.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:21:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:21:56.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:21:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:21:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:21:56.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:21:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:21:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:21:56.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:21:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:21:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:21:56.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:21:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:21:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:21:56.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:21:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:21:56.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:21:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:21:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:21:56.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:21:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:21:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:21:56.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:21:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:21:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:21:56.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:21:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:21:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:21:56.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=819] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:21:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=819] Local address: 172.21.15.161, port 58924 2022-07-03T18:21:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=819] Peer address: 172.21.15.101, port 22 2022-07-03T18:21:56.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=819] Connection lost 2022-07-03T18:21:56.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=819] Aborting connection 2022-07-03T18:21:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:21:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:21:56.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:21:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:21:56.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:21:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:21:56.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:21:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:21:56.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:21:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:21:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:21:56.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:21:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:21:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:21:56.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:21:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:21:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:21:56.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:21:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:21:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:21:56.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:21:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:21:56.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:21:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:21:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:21:56.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:21:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:21:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:21:56.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:21:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:21:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:21:56.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:21:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:21:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:21:56.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:21:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:21:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:21:56.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:21:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:21:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:21:56.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:21:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:21:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:21:56.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:21:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:21:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:21:56.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=819] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:21:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=819] Local address: 172.21.15.161, port 58924 2022-07-03T18:21:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=819] Peer address: 172.21.15.101, port 22 2022-07-03T18:21:56.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=819] Connection lost 2022-07-03T18:21:56.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=819] Aborting connection 2022-07-03T18:21:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:57 smithi161 ceph-mon[91579]: pgmap v13883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:57 smithi188 ceph-mon[92425]: pgmap v13883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: pgmap v13884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:21:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:21:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:21:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:21:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:21:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: pgmap v13884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:21:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:21:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:21:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:21:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:21:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:01 smithi161 ceph-mon[91579]: pgmap v13885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:01 smithi188 ceph-mon[92425]: pgmap v13885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:03 smithi161 ceph-mon[91579]: pgmap v13886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:03 smithi188 ceph-mon[92425]: pgmap v13886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:05 smithi161 ceph-mon[91579]: pgmap v13887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:05 smithi188 ceph-mon[92425]: pgmap v13887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:07 smithi161 ceph-mon[91579]: pgmap v13888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:07 smithi188 ceph-mon[92425]: pgmap v13888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:09 smithi161 ceph-mon[91579]: pgmap v13889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:09 smithi188 ceph-mon[92425]: pgmap v13889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:11 smithi161 ceph-mon[91579]: pgmap v13890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:11 smithi188 ceph-mon[92425]: pgmap v13890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:13 smithi161 ceph-mon[91579]: pgmap v13891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:13 smithi188 ceph-mon[92425]: pgmap v13891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:15 smithi161 ceph-mon[91579]: pgmap v13892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:15 smithi188 ceph-mon[92425]: pgmap v13892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:17 smithi161 ceph-mon[91579]: pgmap v13893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:17 smithi188 ceph-mon[92425]: pgmap v13893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:19 smithi161 ceph-mon[91579]: pgmap v13894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:19 smithi188 ceph-mon[92425]: pgmap v13894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:21 smithi161 ceph-mon[91579]: pgmap v13895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:21 smithi188 ceph-mon[92425]: pgmap v13895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:23 smithi161 ceph-mon[91579]: pgmap v13896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:23 smithi188 ceph-mon[92425]: pgmap v13896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:25 smithi161 ceph-mon[91579]: pgmap v13897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:25 smithi188 ceph-mon[92425]: pgmap v13897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:27.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:27 smithi161 ceph-mon[91579]: pgmap v13898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:27 smithi188 ceph-mon[92425]: pgmap v13898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:29 smithi161 ceph-mon[91579]: pgmap v13899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:29 smithi188 ceph-mon[92425]: pgmap v13899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:31 smithi161 ceph-mon[91579]: pgmap v13900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:31 smithi188 ceph-mon[92425]: pgmap v13900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:33 smithi161 ceph-mon[91579]: pgmap v13901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:33 smithi188 ceph-mon[92425]: pgmap v13901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:35 smithi161 ceph-mon[91579]: pgmap v13902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:35 smithi188 ceph-mon[92425]: pgmap v13902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:37 smithi161 ceph-mon[91579]: pgmap v13903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:37 smithi188 ceph-mon[92425]: pgmap v13903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:39 smithi161 ceph-mon[91579]: pgmap v13904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:39 smithi188 ceph-mon[92425]: pgmap v13904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:41 smithi161 ceph-mon[91579]: pgmap v13905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:41 smithi188 ceph-mon[92425]: pgmap v13905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:43 smithi161 ceph-mon[91579]: pgmap v13906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:43.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:43 smithi188 ceph-mon[92425]: pgmap v13906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:45 smithi161 ceph-mon[91579]: pgmap v13907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:45 smithi188 ceph-mon[92425]: pgmap v13907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:47 smithi188 ceph-mon[92425]: pgmap v13908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:47 smithi161 ceph-mon[91579]: pgmap v13908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:49 smithi188 ceph-mon[92425]: pgmap v13909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:49 smithi161 ceph-mon[91579]: pgmap v13909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:22:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:51 smithi188 ceph-mon[92425]: pgmap v13910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:51 smithi161 ceph-mon[91579]: pgmap v13910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:53 smithi188 ceph-mon[92425]: pgmap v13911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:53 smithi161 ceph-mon[91579]: pgmap v13911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:55 smithi188 ceph-mon[92425]: pgmap v13912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-mon[91579]: pgmap v13912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:22:55.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:22:55.258+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:22:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:22:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:22:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:22:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:22:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:22:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:22:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:22:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:22:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:22:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:22:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:22:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:22:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:22:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:22:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:22:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:22:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:22:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:22:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:22:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:22:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:22:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:22:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:22:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:22:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:22:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:22:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:22:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:22:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:22:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:22:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:22:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:22:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:22:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:22:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:22:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:22:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:22:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:22:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:22:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:22:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:22:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:22:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:22:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:22:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=821] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:22:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=821] Local address: 172.21.15.161, port 58928 2022-07-03T18:22:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=821] Peer address: 172.21.15.101, port 22 2022-07-03T18:22:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=821] Connection lost 2022-07-03T18:22:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=821] Aborting connection 2022-07-03T18:22:55.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:22:56.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:22:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:22:56.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:22:56.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:22:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:22:56.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:22:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:57 smithi188 ceph-mon[92425]: pgmap v13913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:22:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:22:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:22:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:22:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:22:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:22:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:22:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:22:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:22:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:22:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:22:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:22:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:22:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:22:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:22:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:22:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:22:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:22:57.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:22:57.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:22:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:22:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:22:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:22:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:22:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:22:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:22:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:22:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:22:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:22:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:22:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:22:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:22:57.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:22:57.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:22:57.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:22:57.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:22:57.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:22:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:22:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:22:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:22:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:22:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:22:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:22:57.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:22:57.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:22:57.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:22:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=821] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:22:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=821] Local address: 172.21.15.161, port 58928 2022-07-03T18:22:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=821] Peer address: 172.21.15.101, port 22 2022-07-03T18:22:57.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=821] Connection lost 2022-07-03T18:22:57.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=821] Aborting connection 2022-07-03T18:22:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:57 smithi161 ceph-mon[91579]: pgmap v13913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:22:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:22:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:22:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:22:57.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:22:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:22:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:22:57.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:22:57.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:22:57.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:22:57.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:22:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:22:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:22:57.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:22:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:22:57.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:22:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:22:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:22:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:22:57.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:22:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:22:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:22:57.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:22:57.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:22:57.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:22:57.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:22:57.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:22:57.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:22:57.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:22:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:22:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:22:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:22:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:22:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:22:57.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:22:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:22:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:22:57.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:22:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:22:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:22:57.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:22:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:22:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:22:57.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:22:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:22:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:22:57.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=821] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:22:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=821] Local address: 172.21.15.161, port 58928 2022-07-03T18:22:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=821] Peer address: 172.21.15.101, port 22 2022-07-03T18:22:57.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=821] Connection lost 2022-07-03T18:22:57.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=821] Aborting connection 2022-07-03T18:22:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: pgmap v13914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:22:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:22:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:22:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:22:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:22:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: pgmap v13914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:22:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:22:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:22:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:22:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:22:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:01 smithi188 ceph-mon[92425]: pgmap v13915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:01 smithi161 ceph-mon[91579]: pgmap v13915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:03 smithi161 ceph-mon[91579]: pgmap v13916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:03 smithi188 ceph-mon[92425]: pgmap v13916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:05 smithi161 ceph-mon[91579]: pgmap v13917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:05 smithi188 ceph-mon[92425]: pgmap v13917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:07 smithi161 ceph-mon[91579]: pgmap v13918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:07 smithi188 ceph-mon[92425]: pgmap v13918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:09 smithi161 ceph-mon[91579]: pgmap v13919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:09 smithi188 ceph-mon[92425]: pgmap v13919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:11 smithi161 ceph-mon[91579]: pgmap v13920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:11 smithi188 ceph-mon[92425]: pgmap v13920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:13 smithi161 ceph-mon[91579]: pgmap v13921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:13 smithi188 ceph-mon[92425]: pgmap v13921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:14 smithi161 ceph-mon[91579]: pgmap v13922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:14.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:14 smithi188 ceph-mon[92425]: pgmap v13922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:16 smithi161 ceph-mon[91579]: pgmap v13923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:16 smithi188 ceph-mon[92425]: pgmap v13923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:18 smithi161 ceph-mon[91579]: pgmap v13924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:18 smithi188 ceph-mon[92425]: pgmap v13924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:20 smithi161 ceph-mon[91579]: pgmap v13925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:20 smithi188 ceph-mon[92425]: pgmap v13925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:22 smithi161 ceph-mon[91579]: pgmap v13926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:22 smithi188 ceph-mon[92425]: pgmap v13926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:24 smithi161 ceph-mon[91579]: pgmap v13927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:24 smithi188 ceph-mon[92425]: pgmap v13927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:26 smithi161 ceph-mon[91579]: pgmap v13928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:26 smithi188 ceph-mon[92425]: pgmap v13928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:28 smithi161 ceph-mon[91579]: pgmap v13929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:28 smithi188 ceph-mon[92425]: pgmap v13929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:30 smithi161 ceph-mon[91579]: pgmap v13930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:30 smithi188 ceph-mon[92425]: pgmap v13930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:32 smithi161 ceph-mon[91579]: pgmap v13931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:32 smithi188 ceph-mon[92425]: pgmap v13931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:34 smithi161 ceph-mon[91579]: pgmap v13932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:34 smithi188 ceph-mon[92425]: pgmap v13932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:36 smithi161 ceph-mon[91579]: pgmap v13933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:36 smithi188 ceph-mon[92425]: pgmap v13933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:38 smithi161 ceph-mon[91579]: pgmap v13934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:38 smithi188 ceph-mon[92425]: pgmap v13934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:40 smithi161 ceph-mon[91579]: pgmap v13935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:40 smithi188 ceph-mon[92425]: pgmap v13935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:42 smithi161 ceph-mon[91579]: pgmap v13936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:42 smithi188 ceph-mon[92425]: pgmap v13936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:44 smithi161 ceph-mon[91579]: pgmap v13937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:44 smithi188 ceph-mon[92425]: pgmap v13937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:46 smithi161 ceph-mon[91579]: pgmap v13938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:46 smithi188 ceph-mon[92425]: pgmap v13938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:49 smithi161 ceph-mon[91579]: pgmap v13939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:49 smithi188 ceph-mon[92425]: pgmap v13939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:51 smithi161 ceph-mon[91579]: pgmap v13940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:51 smithi188 ceph-mon[92425]: pgmap v13940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:53 smithi161 ceph-mon[91579]: pgmap v13941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:53 smithi188 ceph-mon[92425]: pgmap v13941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-mon[91579]: pgmap v13942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:55 smithi188 ceph-mon[92425]: pgmap v13942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:23:56.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:23:55.757+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:23:56.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:23:56.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:23:56.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:23:56.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:23:56.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:23:56.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:23:56.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:23:56.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:23:56.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:23:56.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:23:56.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:23:56.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:23:56.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:23:56.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:23:56.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:23:56.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:23:56.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:23:56.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:23:56.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:23:56.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:23:56.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:23:56.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:23:56.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:23:56.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:23:56.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:23:56.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:23:56.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:23:56.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:23:56.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:23:56.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:23:56.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:23:56.028 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:23:56.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:23:56.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:23:56.029 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:23:56.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:23:56.030 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:23:56.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:23:56.031 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:23:56.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:23:56.032 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:23:56.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:23:56.033 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:23:56.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:23:56.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=823] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:23:56.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=823] Local address: 172.21.15.161, port 58932 2022-07-03T18:23:56.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=823] Peer address: 172.21.15.101, port 22 2022-07-03T18:23:56.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=823] Connection lost 2022-07-03T18:23:56.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=823] Aborting connection 2022-07-03T18:23:56.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:23:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:23:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:23:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:23:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:23:56.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:23:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:23:56.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:23:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:57 smithi161 ceph-mon[91579]: pgmap v13943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:23:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:23:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:23:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:23:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:23:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:23:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:23:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:23:57.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:23:57.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:23:57.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:23:57.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:23:57.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:23:57.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:23:57.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:23:57.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:23:57.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:23:57.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:23:57.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:23:57.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:23:57.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:23:57.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:23:57.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:23:57.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:23:57.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:23:57.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:23:57.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:23:57.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:23:57.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:23:57.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:23:57.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:23:57.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:23:57.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:23:57.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:23:57.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:23:57.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:23:57.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:23:57.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:23:57.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:23:57.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:23:57.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:23:57.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:23:57.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:23:57.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:23:57.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:23:57.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:23:57.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=823] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:23:57.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=823] Local address: 172.21.15.161, port 58932 2022-07-03T18:23:57.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=823] Peer address: 172.21.15.101, port 22 2022-07-03T18:23:57.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=823] Connection lost 2022-07-03T18:23:57.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=823] Aborting connection 2022-07-03T18:23:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:57 smithi188 ceph-mon[92425]: pgmap v13943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:23:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:23:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:23:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:23:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:23:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:23:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:23:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:23:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:23:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:23:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:23:57.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:23:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:23:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:23:57.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:23:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:23:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:23:57.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:23:57.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:23:57.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:23:57.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:23:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:23:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:23:57.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:23:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:23:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:23:57.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:23:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:23:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:23:57.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:23:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:23:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:23:57.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:23:57.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:23:57.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:23:57.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:23:57.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:23:57.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:23:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:23:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:23:57.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:23:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:23:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:23:57.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:23:57.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:23:57.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:23:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=823] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:23:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=823] Local address: 172.21.15.161, port 58932 2022-07-03T18:23:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=823] Peer address: 172.21.15.101, port 22 2022-07-03T18:23:57.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=823] Connection lost 2022-07-03T18:23:57.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=823] Aborting connection 2022-07-03T18:23:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: pgmap v13944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:23:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:23:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:23:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:23:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:23:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: pgmap v13944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:23:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:23:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:23:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:23:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:23:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:01 smithi161 ceph-mon[91579]: pgmap v13945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:01 smithi188 ceph-mon[92425]: pgmap v13945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:03 smithi161 ceph-mon[91579]: pgmap v13946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:03 smithi188 ceph-mon[92425]: pgmap v13946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:05 smithi161 ceph-mon[91579]: pgmap v13947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:05 smithi188 ceph-mon[92425]: pgmap v13947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:07 smithi161 ceph-mon[91579]: pgmap v13948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:07 smithi188 ceph-mon[92425]: pgmap v13948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:09 smithi161 ceph-mon[91579]: pgmap v13949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:09 smithi188 ceph-mon[92425]: pgmap v13949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:11 smithi161 ceph-mon[91579]: pgmap v13950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:11 smithi188 ceph-mon[92425]: pgmap v13950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:13 smithi161 ceph-mon[91579]: pgmap v13951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:13 smithi188 ceph-mon[92425]: pgmap v13951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:15 smithi161 ceph-mon[91579]: pgmap v13952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:15 smithi188 ceph-mon[92425]: pgmap v13952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:17 smithi161 ceph-mon[91579]: pgmap v13953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:17 smithi188 ceph-mon[92425]: pgmap v13953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:19 smithi161 ceph-mon[91579]: pgmap v13954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:19 smithi188 ceph-mon[92425]: pgmap v13954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:21 smithi161 ceph-mon[91579]: pgmap v13955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:21 smithi188 ceph-mon[92425]: pgmap v13955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:23 smithi161 ceph-mon[91579]: pgmap v13956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:23 smithi188 ceph-mon[92425]: pgmap v13956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:25 smithi161 ceph-mon[91579]: pgmap v13957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:25 smithi188 ceph-mon[92425]: pgmap v13957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:27 smithi161 ceph-mon[91579]: pgmap v13958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:27 smithi188 ceph-mon[92425]: pgmap v13958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:29 smithi161 ceph-mon[91579]: pgmap v13959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:29 smithi188 ceph-mon[92425]: pgmap v13959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:31 smithi161 ceph-mon[91579]: pgmap v13960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:31 smithi188 ceph-mon[92425]: pgmap v13960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:33 smithi161 ceph-mon[91579]: pgmap v13961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:33 smithi188 ceph-mon[92425]: pgmap v13961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:35 smithi161 ceph-mon[91579]: pgmap v13962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:35 smithi188 ceph-mon[92425]: pgmap v13962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:37 smithi161 ceph-mon[91579]: pgmap v13963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:37 smithi188 ceph-mon[92425]: pgmap v13963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:39 smithi161 ceph-mon[91579]: pgmap v13964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:39 smithi188 ceph-mon[92425]: pgmap v13964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:41 smithi161 ceph-mon[91579]: pgmap v13965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:41 smithi188 ceph-mon[92425]: pgmap v13965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:43 smithi161 ceph-mon[91579]: pgmap v13966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:43 smithi188 ceph-mon[92425]: pgmap v13966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:45 smithi161 ceph-mon[91579]: pgmap v13967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:45 smithi188 ceph-mon[92425]: pgmap v13967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:47 smithi161 ceph-mon[91579]: pgmap v13968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:47 smithi188 ceph-mon[92425]: pgmap v13968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:49 smithi161 ceph-mon[91579]: pgmap v13969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:49 smithi188 ceph-mon[92425]: pgmap v13969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:51 smithi188 ceph-mon[92425]: pgmap v13970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:51 smithi161 ceph-mon[91579]: pgmap v13970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:53 smithi188 ceph-mon[92425]: pgmap v13971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:53 smithi161 ceph-mon[91579]: pgmap v13971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:55 smithi188 ceph-mon[92425]: pgmap v13972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:55 smithi161 ceph-mon[91579]: pgmap v13972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:24:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:24:56.258+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:24:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:24:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:24:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:24:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:24:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:24:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:24:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:24:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:24:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:24:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:24:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:24:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:24:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:24:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:24:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:24:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:24:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:24:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:24:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:24:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:24:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:24:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:24:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:24:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:24:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:24:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:24:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:24:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:24:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:24:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:24:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:24:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:24:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:24:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:24:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:24:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:24:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:24:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:24:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:24:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:24:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:24:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:24:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:24:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:24:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=825] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:24:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=825] Local address: 172.21.15.161, port 58962 2022-07-03T18:24:56.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=825] Peer address: 172.21.15.101, port 22 2022-07-03T18:24:56.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=825] Connection lost 2022-07-03T18:24:56.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=825] Aborting connection 2022-07-03T18:24:56.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:24:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:57 smithi188 ceph-mon[92425]: pgmap v13973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:24:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:24:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:24:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:57 smithi161 ceph-mon[91579]: pgmap v13973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:24:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:24:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:24:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:24:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:24:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:24:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:24:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:24:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:24:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:24:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:24:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:24:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:24:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:24:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:24:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:24:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:24:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:24:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:24:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:24:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:24:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:24:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:24:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:24:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:24:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:24:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:24:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:24:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:24:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:24:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:24:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:24:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:24:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:24:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:24:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:24:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:24:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:24:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:24:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:24:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:24:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:24:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:24:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:24:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:24:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:24:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:24:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:24:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:24:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=825] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:24:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=825] Local address: 172.21.15.161, port 58962 2022-07-03T18:24:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=825] Peer address: 172.21.15.101, port 22 2022-07-03T18:24:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=825] Connection lost 2022-07-03T18:24:58.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=825] Aborting connection 2022-07-03T18:24:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:24:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:24:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:24:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:24:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:24:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:24:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:24:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:24:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:24:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:24:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:24:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:24:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:24:58.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:24:58.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:24:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:24:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:24:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:24:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:24:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:24:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:24:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:24:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:24:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:24:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:24:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:24:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:24:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:24:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:24:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:24:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:24:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:24:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:24:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:24:58.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:24:58.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:24:58.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:24:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:24:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:24:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:24:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:24:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:24:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:24:58.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:24:58.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:24:58.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:24:58.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=825] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:24:58.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=825] Local address: 172.21.15.161, port 58962 2022-07-03T18:24:58.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=825] Peer address: 172.21.15.101, port 22 2022-07-03T18:24:58.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=825] Connection lost 2022-07-03T18:24:58.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=825] Aborting connection 2022-07-03T18:24:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: pgmap v13974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:24:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:24:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:24:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:24:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:24:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: pgmap v13974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:24:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:24:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:24:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:24:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:24:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:01 smithi188 ceph-mon[92425]: pgmap v13975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:01 smithi161 ceph-mon[91579]: pgmap v13975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:03 smithi188 ceph-mon[92425]: pgmap v13976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:03 smithi161 ceph-mon[91579]: pgmap v13976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:05 smithi161 ceph-mon[91579]: pgmap v13977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:05 smithi188 ceph-mon[92425]: pgmap v13977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:07 smithi161 ceph-mon[91579]: pgmap v13978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:07 smithi188 ceph-mon[92425]: pgmap v13978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:09 smithi161 ceph-mon[91579]: pgmap v13979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:09 smithi188 ceph-mon[92425]: pgmap v13979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:11 smithi161 ceph-mon[91579]: pgmap v13980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:11 smithi188 ceph-mon[92425]: pgmap v13980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:13 smithi161 ceph-mon[91579]: pgmap v13981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:13 smithi188 ceph-mon[92425]: pgmap v13981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:15 smithi161 ceph-mon[91579]: pgmap v13982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:15 smithi188 ceph-mon[92425]: pgmap v13982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:17 smithi161 ceph-mon[91579]: pgmap v13983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:17 smithi188 ceph-mon[92425]: pgmap v13983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:19 smithi161 ceph-mon[91579]: pgmap v13984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:19 smithi188 ceph-mon[92425]: pgmap v13984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:21 smithi161 ceph-mon[91579]: pgmap v13985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:21 smithi188 ceph-mon[92425]: pgmap v13985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:23 smithi161 ceph-mon[91579]: pgmap v13986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:23 smithi188 ceph-mon[92425]: pgmap v13986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:25 smithi161 ceph-mon[91579]: pgmap v13987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:25 smithi188 ceph-mon[92425]: pgmap v13987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:26 smithi161 ceph-mon[91579]: pgmap v13988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:26 smithi188 ceph-mon[92425]: pgmap v13988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:28 smithi161 ceph-mon[91579]: pgmap v13989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:28 smithi188 ceph-mon[92425]: pgmap v13989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:30 smithi161 ceph-mon[91579]: pgmap v13990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:30 smithi188 ceph-mon[92425]: pgmap v13990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:32 smithi161 ceph-mon[91579]: pgmap v13991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:32 smithi188 ceph-mon[92425]: pgmap v13991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:34 smithi161 ceph-mon[91579]: pgmap v13992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:34 smithi188 ceph-mon[92425]: pgmap v13992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:36 smithi161 ceph-mon[91579]: pgmap v13993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:36 smithi188 ceph-mon[92425]: pgmap v13993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:38 smithi161 ceph-mon[91579]: pgmap v13994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:38 smithi188 ceph-mon[92425]: pgmap v13994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:40 smithi161 ceph-mon[91579]: pgmap v13995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:40 smithi188 ceph-mon[92425]: pgmap v13995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:42 smithi161 ceph-mon[91579]: pgmap v13996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:42 smithi188 ceph-mon[92425]: pgmap v13996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:45 smithi161 ceph-mon[91579]: pgmap v13997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:45 smithi188 ceph-mon[92425]: pgmap v13997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:47.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:47 smithi161 ceph-mon[91579]: pgmap v13998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:47 smithi188 ceph-mon[92425]: pgmap v13998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:49 smithi161 ceph-mon[91579]: pgmap v13999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:49 smithi188 ceph-mon[92425]: pgmap v13999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:51 smithi161 ceph-mon[91579]: pgmap v14000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:51 smithi188 ceph-mon[92425]: pgmap v14000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:53 smithi161 ceph-mon[91579]: pgmap v14001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:53 smithi188 ceph-mon[92425]: pgmap v14001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:55 smithi161 ceph-mon[91579]: pgmap v14002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:55 smithi188 ceph-mon[92425]: pgmap v14002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:25:57.034 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:25:56.761+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:25:57.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:25:57.035 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:25:57.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:25:57.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:25:57.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:25:57.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:25:57.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:25:57.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:25:57.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:25:57.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:25:57.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:25:57.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:25:57.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:25:57.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:25:57.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:25:57.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:25:57.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:25:57.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:25:57.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:25:57.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:25:57.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:25:57.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:25:57.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:25:57.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:25:57.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:25:57.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:25:57.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:25:57.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:25:57.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:25:57.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:25:57.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:25:57.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:25:57.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:25:57.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:25:57.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:25:57.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:25:57.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:25:57.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:25:57.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:25:57.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:25:57.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:25:57.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:25:57.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:25:57.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:25:57.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=827] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:25:57.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=827] Local address: 172.21.15.161, port 58966 2022-07-03T18:25:57.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=827] Peer address: 172.21.15.101, port 22 2022-07-03T18:25:57.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=827] Connection lost 2022-07-03T18:25:57.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=827] Aborting connection 2022-07-03T18:25:57.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:25:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:25:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:57 smithi161 ceph-mon[91579]: pgmap v14003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:25:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:25:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:25:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:57 smithi188 ceph-mon[92425]: pgmap v14003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:25:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:25:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:25:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:25:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:25:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:25:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:25:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:25:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:25:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:25:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:25:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:25:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:25:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:25:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:25:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:25:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:25:58.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:25:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:25:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:25:58.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:25:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:25:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:25:58.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:25:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:25:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:25:58.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:25:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:25:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:25:58.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:25:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:25:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:25:58.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:25:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:25:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:25:58.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:25:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:25:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:25:58.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:25:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:25:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:25:58.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:25:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:25:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:25:58.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:25:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:25:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:25:58.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:25:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:25:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=827] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:25:58.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=827] Local address: 172.21.15.161, port 58966 2022-07-03T18:25:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=827] Peer address: 172.21.15.101, port 22 2022-07-03T18:25:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=827] Connection lost 2022-07-03T18:25:58.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=827] Aborting connection 2022-07-03T18:25:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:25:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:25:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:25:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:25:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:25:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:25:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:25:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:25:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:25:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:25:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:25:58.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:25:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:25:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:25:58.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:25:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:25:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:25:58.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:25:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:25:58.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:25:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:25:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:25:58.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:25:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:25:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:25:58.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:25:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:25:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:25:58.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:25:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:25:58.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:25:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:25:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:25:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:25:58.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:25:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:25:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:25:58.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:25:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:25:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:25:58.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:25:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:25:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:25:58.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:25:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:25:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:25:58.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=827] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:25:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=827] Local address: 172.21.15.161, port 58966 2022-07-03T18:25:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=827] Peer address: 172.21.15.101, port 22 2022-07-03T18:25:58.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=827] Connection lost 2022-07-03T18:25:58.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=827] Aborting connection 2022-07-03T18:25:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: pgmap v14004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:25:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:25:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: pgmap v14004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:25:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:25:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:25:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:01 smithi161 ceph-mon[91579]: pgmap v14005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:01 smithi188 ceph-mon[92425]: pgmap v14005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:03 smithi161 ceph-mon[91579]: pgmap v14006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:03 smithi188 ceph-mon[92425]: pgmap v14006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:05 smithi161 ceph-mon[91579]: pgmap v14007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:05 smithi188 ceph-mon[92425]: pgmap v14007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:07 smithi161 ceph-mon[91579]: pgmap v14008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:07 smithi188 ceph-mon[92425]: pgmap v14008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:09 smithi161 ceph-mon[91579]: pgmap v14009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:09 smithi188 ceph-mon[92425]: pgmap v14009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:11 smithi161 ceph-mon[91579]: pgmap v14010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:11 smithi188 ceph-mon[92425]: pgmap v14010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:13 smithi161 ceph-mon[91579]: pgmap v14011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:13 smithi188 ceph-mon[92425]: pgmap v14011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:15 smithi161 ceph-mon[91579]: pgmap v14012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:15 smithi188 ceph-mon[92425]: pgmap v14012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:17 smithi161 ceph-mon[91579]: pgmap v14013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:17 smithi188 ceph-mon[92425]: pgmap v14013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:19 smithi161 ceph-mon[91579]: pgmap v14014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:19 smithi188 ceph-mon[92425]: pgmap v14014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:21 smithi161 ceph-mon[91579]: pgmap v14015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:21 smithi188 ceph-mon[92425]: pgmap v14015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:23 smithi161 ceph-mon[91579]: pgmap v14016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:23 smithi188 ceph-mon[92425]: pgmap v14016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:25 smithi161 ceph-mon[91579]: pgmap v14017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:25 smithi188 ceph-mon[92425]: pgmap v14017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:27 smithi161 ceph-mon[91579]: pgmap v14018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:27 smithi188 ceph-mon[92425]: pgmap v14018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:29 smithi161 ceph-mon[91579]: pgmap v14019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:29 smithi188 ceph-mon[92425]: pgmap v14019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:31 smithi161 ceph-mon[91579]: pgmap v14020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:31 smithi188 ceph-mon[92425]: pgmap v14020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:33.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:33 smithi161 ceph-mon[91579]: pgmap v14021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:33 smithi188 ceph-mon[92425]: pgmap v14021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:35 smithi161 ceph-mon[91579]: pgmap v14022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:35 smithi188 ceph-mon[92425]: pgmap v14022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:37 smithi161 ceph-mon[91579]: pgmap v14023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:37 smithi188 ceph-mon[92425]: pgmap v14023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:39 smithi161 ceph-mon[91579]: pgmap v14024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:39 smithi188 ceph-mon[92425]: pgmap v14024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:41 smithi188 ceph-mon[92425]: pgmap v14025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:41 smithi161 ceph-mon[91579]: pgmap v14025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:43 smithi188 ceph-mon[92425]: pgmap v14026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:43 smithi161 ceph-mon[91579]: pgmap v14026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:45 smithi188 ceph-mon[92425]: pgmap v14027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:45 smithi161 ceph-mon[91579]: pgmap v14027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:47 smithi188 ceph-mon[92425]: pgmap v14028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:47 smithi161 ceph-mon[91579]: pgmap v14028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:49 smithi188 ceph-mon[92425]: pgmap v14029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:49 smithi161 ceph-mon[91579]: pgmap v14029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:51 smithi188 ceph-mon[92425]: pgmap v14030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:51 smithi161 ceph-mon[91579]: pgmap v14030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:53 smithi188 ceph-mon[92425]: pgmap v14031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:53 smithi161 ceph-mon[91579]: pgmap v14031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:55 smithi161 ceph-mon[91579]: pgmap v14032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:55 smithi188 ceph-mon[92425]: pgmap v14032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:26:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-mon[91579]: pgmap v14033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:26:57.258+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:26:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:26:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:26:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:26:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:26:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:26:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:26:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:26:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:26:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:26:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:26:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:26:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:26:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:26:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:26:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:26:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:26:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:26:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:26:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:26:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:26:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:26:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:26:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:26:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:26:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:26:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:26:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:26:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:26:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:26:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:26:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:26:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:26:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:26:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:26:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:26:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:26:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:26:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:26:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:26:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:26:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:26:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:26:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:26:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:26:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=829] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:26:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=829] Local address: 172.21.15.161, port 58970 2022-07-03T18:26:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=829] Peer address: 172.21.15.101, port 22 2022-07-03T18:26:57.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=829] Connection lost 2022-07-03T18:26:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=829] Aborting connection 2022-07-03T18:26:57.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:26:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:57 smithi188 ceph-mon[92425]: pgmap v14033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:26:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:26:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:26:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:26:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:26:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:26:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:26:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:26:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:26:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:26:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:26:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:26:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:26:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:26:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:26:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:26:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:26:59.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:26:59.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:26:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:26:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:26:59.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:26:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:26:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:26:59.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:26:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:26:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:26:59.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:26:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:26:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:26:59.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:26:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:26:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:26:59.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:26:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:26:59.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:26:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:26:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:26:59.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:26:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:26:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:26:59.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:26:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:26:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:26:59.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:26:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:26:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:26:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:26:59.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:26:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:26:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:26:59.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:26:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:26:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=829] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:26:59.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=829] Local address: 172.21.15.161, port 58970 2022-07-03T18:26:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=829] Peer address: 172.21.15.101, port 22 2022-07-03T18:26:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=829] Connection lost 2022-07-03T18:26:59.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=829] Aborting connection 2022-07-03T18:26:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: pgmap v14034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:26:59.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:26:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:26:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:26:59.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:26:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:26:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:26:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:26:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:26:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:26:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:26:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:26:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:26:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:26:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:26:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:26:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:26:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:26:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:26:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:26:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:26:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:26:59.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:26:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:26:59.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:26:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:26:59.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:26:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:26:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:26:59.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:26:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:26:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:26:59.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:26:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:26:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:26:59.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:26:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:26:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:26:59.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:26:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:26:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:26:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:26:59.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:26:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:26:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:26:59.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:26:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:26:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:26:59.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:26:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:26:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:26:59.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=829] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:26:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=829] Local address: 172.21.15.161, port 58970 2022-07-03T18:26:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=829] Peer address: 172.21.15.101, port 22 2022-07-03T18:26:59.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=829] Connection lost 2022-07-03T18:26:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=829] Aborting connection 2022-07-03T18:26:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: pgmap v14034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:26:59.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:26:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:26:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:26:59.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:26:59.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:01 smithi161 ceph-mon[91579]: pgmap v14035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:01 smithi188 ceph-mon[92425]: pgmap v14035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:03 smithi161 ceph-mon[91579]: pgmap v14036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:03 smithi188 ceph-mon[92425]: pgmap v14036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:05 smithi161 ceph-mon[91579]: pgmap v14037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:05 smithi188 ceph-mon[92425]: pgmap v14037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:07 smithi161 ceph-mon[91579]: pgmap v14038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:07 smithi188 ceph-mon[92425]: pgmap v14038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:09 smithi161 ceph-mon[91579]: pgmap v14039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:09 smithi188 ceph-mon[92425]: pgmap v14039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:11 smithi161 ceph-mon[91579]: pgmap v14040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:11 smithi188 ceph-mon[92425]: pgmap v14040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:13 smithi161 ceph-mon[91579]: pgmap v14041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:13 smithi188 ceph-mon[92425]: pgmap v14041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:15 smithi161 ceph-mon[91579]: pgmap v14042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:15 smithi188 ceph-mon[92425]: pgmap v14042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:17 smithi161 ceph-mon[91579]: pgmap v14043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:17 smithi188 ceph-mon[92425]: pgmap v14043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:19 smithi161 ceph-mon[91579]: pgmap v14044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:19 smithi188 ceph-mon[92425]: pgmap v14044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:21 smithi161 ceph-mon[91579]: pgmap v14045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:21 smithi188 ceph-mon[92425]: pgmap v14045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:22 smithi161 ceph-mon[91579]: pgmap v14046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:22 smithi188 ceph-mon[92425]: pgmap v14046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:24 smithi161 ceph-mon[91579]: pgmap v14047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:24 smithi188 ceph-mon[92425]: pgmap v14047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:27 smithi161 ceph-mon[91579]: pgmap v14048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:27.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:27 smithi188 ceph-mon[92425]: pgmap v14048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:29 smithi161 ceph-mon[91579]: pgmap v14049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:29 smithi188 ceph-mon[92425]: pgmap v14049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:31 smithi161 ceph-mon[91579]: pgmap v14050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:31 smithi188 ceph-mon[92425]: pgmap v14050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:33 smithi161 ceph-mon[91579]: pgmap v14051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:33 smithi188 ceph-mon[92425]: pgmap v14051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:35 smithi161 ceph-mon[91579]: pgmap v14052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:35 smithi188 ceph-mon[92425]: pgmap v14052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:37 smithi161 ceph-mon[91579]: pgmap v14053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:37 smithi188 ceph-mon[92425]: pgmap v14053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:39 smithi161 ceph-mon[91579]: pgmap v14054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:39 smithi188 ceph-mon[92425]: pgmap v14054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:41 smithi161 ceph-mon[91579]: pgmap v14055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:41 smithi188 ceph-mon[92425]: pgmap v14055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:43 smithi161 ceph-mon[91579]: pgmap v14056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:43 smithi188 ceph-mon[92425]: pgmap v14056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:45 smithi161 ceph-mon[91579]: pgmap v14057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:45 smithi188 ceph-mon[92425]: pgmap v14057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:47 smithi161 ceph-mon[91579]: pgmap v14058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:47.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:47 smithi188 ceph-mon[92425]: pgmap v14058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:49 smithi161 ceph-mon[91579]: pgmap v14059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:49 smithi188 ceph-mon[92425]: pgmap v14059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:51 smithi161 ceph-mon[91579]: pgmap v14060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:51.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:51 smithi188 ceph-mon[92425]: pgmap v14060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:53 smithi161 ceph-mon[91579]: pgmap v14061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:53 smithi188 ceph-mon[92425]: pgmap v14061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:55 smithi161 ceph-mon[91579]: pgmap v14062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:55 smithi188 ceph-mon[92425]: pgmap v14062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:27:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:57 smithi161 ceph-mon[91579]: pgmap v14063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:57 smithi188 ceph-mon[92425]: pgmap v14063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:27:58.184+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:27:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:27:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:27:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:27:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:27:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:27:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:27:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:27:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:27:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=831] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:27:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=831] Local address: 172.21.15.161, port 58974 2022-07-03T18:27:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=831] Peer address: 172.21.15.101, port 22 2022-07-03T18:27:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=831] Connection lost 2022-07-03T18:27:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=831] Aborting connection 2022-07-03T18:27:58.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:27:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: pgmap v14064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:27:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:27:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:27:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:27:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:27:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:27:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:27:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:27:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:27:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:27:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:27:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:27:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:27:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:27:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:27:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:27:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:27:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:27:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:27:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:27:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:27:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:27:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:27:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:27:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:27:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:27:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:27:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:27:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:27:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:27:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:27:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:27:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:27:59.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:27:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:27:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:27:59.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:27:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:27:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:27:59.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:27:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:27:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:27:59.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:27:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:27:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:27:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:27:59.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:27:59.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:27:59.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:27:59.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=831] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:27:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=831] Local address: 172.21.15.161, port 58974 2022-07-03T18:27:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=831] Peer address: 172.21.15.101, port 22 2022-07-03T18:27:59.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=831] Connection lost 2022-07-03T18:27:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=831] Aborting connection 2022-07-03T18:27:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:27:59.404 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:27:59.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:27:59.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:27:59.405 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:27:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: pgmap v14064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:27:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:27:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:27:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:27:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:27:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:27:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:27:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:27:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:27:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:27:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:27:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:27:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:27:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:27:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:27:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:27:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:27:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:27:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:27:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:27:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:27:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:27:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:27:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:27:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:27:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:27:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:27:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:27:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:27:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:27:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:27:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:27:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:27:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:27:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:27:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:27:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:27:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:27:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:27:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:27:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:27:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:27:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:27:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:27:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:27:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:27:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:27:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:27:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:27:59.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:27:59.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=831] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:27:59.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=831] Local address: 172.21.15.161, port 58974 2022-07-03T18:27:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=831] Peer address: 172.21.15.101, port 22 2022-07-03T18:27:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=831] Connection lost 2022-07-03T18:27:59.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=831] Aborting connection 2022-07-03T18:27:59.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:27:59.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:27:59.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:27:59.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:27:59.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:01 smithi161 ceph-mon[91579]: pgmap v14065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:01.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:01 smithi188 ceph-mon[92425]: pgmap v14065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:03 smithi161 ceph-mon[91579]: pgmap v14066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:03 smithi188 ceph-mon[92425]: pgmap v14066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:05 smithi161 ceph-mon[91579]: pgmap v14067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:05 smithi188 ceph-mon[92425]: pgmap v14067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:07 smithi161 ceph-mon[91579]: pgmap v14068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:07 smithi188 ceph-mon[92425]: pgmap v14068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:09 smithi161 ceph-mon[91579]: pgmap v14069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:09 smithi188 ceph-mon[92425]: pgmap v14069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:11 smithi161 ceph-mon[91579]: pgmap v14070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:11 smithi188 ceph-mon[92425]: pgmap v14070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:13 smithi161 ceph-mon[91579]: pgmap v14071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:13 smithi188 ceph-mon[92425]: pgmap v14071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:15 smithi161 ceph-mon[91579]: pgmap v14072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:15 smithi188 ceph-mon[92425]: pgmap v14072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:17 smithi161 ceph-mon[91579]: pgmap v14073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:17 smithi188 ceph-mon[92425]: pgmap v14073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:19 smithi161 ceph-mon[91579]: pgmap v14074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:19 smithi188 ceph-mon[92425]: pgmap v14074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:21 smithi161 ceph-mon[91579]: pgmap v14075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:21 smithi188 ceph-mon[92425]: pgmap v14075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:23 smithi161 ceph-mon[91579]: pgmap v14076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:23 smithi188 ceph-mon[92425]: pgmap v14076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:25 smithi161 ceph-mon[91579]: pgmap v14077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:25 smithi188 ceph-mon[92425]: pgmap v14077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:27 smithi161 ceph-mon[91579]: pgmap v14078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:27 smithi188 ceph-mon[92425]: pgmap v14078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:29 smithi161 ceph-mon[91579]: pgmap v14079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:29.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:29 smithi188 ceph-mon[92425]: pgmap v14079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:29.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:31 smithi161 ceph-mon[91579]: pgmap v14080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:31 smithi188 ceph-mon[92425]: pgmap v14080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:33 smithi161 ceph-mon[91579]: pgmap v14081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:33 smithi188 ceph-mon[92425]: pgmap v14081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:35 smithi188 ceph-mon[92425]: pgmap v14082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:35 smithi161 ceph-mon[91579]: pgmap v14082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:37.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:37 smithi188 ceph-mon[92425]: pgmap v14083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:37 smithi161 ceph-mon[91579]: pgmap v14083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:39 smithi188 ceph-mon[92425]: pgmap v14084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:39 smithi161 ceph-mon[91579]: pgmap v14084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:41.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:41 smithi188 ceph-mon[92425]: pgmap v14085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:41 smithi161 ceph-mon[91579]: pgmap v14085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:43 smithi188 ceph-mon[92425]: pgmap v14086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:43 smithi161 ceph-mon[91579]: pgmap v14086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:45 smithi188 ceph-mon[92425]: pgmap v14087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:45 smithi161 ceph-mon[91579]: pgmap v14087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:47 smithi188 ceph-mon[92425]: pgmap v14088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:47 smithi161 ceph-mon[91579]: pgmap v14088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:49 smithi188 ceph-mon[92425]: pgmap v14089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:49 smithi161 ceph-mon[91579]: pgmap v14089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:51 smithi161 ceph-mon[91579]: pgmap v14090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:51.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:51 smithi188 ceph-mon[92425]: pgmap v14090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:53 smithi161 ceph-mon[91579]: pgmap v14091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:53 smithi188 ceph-mon[92425]: pgmap v14091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:55 smithi161 ceph-mon[91579]: pgmap v14092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:55 smithi188 ceph-mon[92425]: pgmap v14092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:28:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:57 smithi161 ceph-mon[91579]: pgmap v14093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:57 smithi188 ceph-mon[92425]: pgmap v14093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:59.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:28:58.679+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:28:59.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:28:59.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:28:59.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:28:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:28:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:28:59.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:28:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:28:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:28:59.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:28:59.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:28:59.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:28:59.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:28:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:28:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:28:59.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:28:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:28:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:28:59.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:28:59.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:28:59.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:28:59.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:28:59.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:28:59.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:28:59.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:28:59.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:28:59.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:28:59.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:28:59.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:28:59.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:28:59.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:28:59.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:28:59.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:28:59.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:28:59.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:28:59.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:28:59.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:28:59.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:28:59.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:28:59.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:28:59.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:28:59.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:28:59.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:28:59.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:28:59.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:28:59.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=833] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:28:59.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=833] Local address: 172.21.15.161, port 58978 2022-07-03T18:28:59.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=833] Peer address: 172.21.15.101, port 22 2022-07-03T18:28:59.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=833] Connection lost 2022-07-03T18:28:59.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=833] Aborting connection 2022-07-03T18:28:59.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:28:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: pgmap v14094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:28:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:28:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:28:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:28:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:28:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:28:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: pgmap v14094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:28:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:28:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:28:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:28:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:28:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:28:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:28:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:28:59.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:29:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:29:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:29:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:29:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:29:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:29:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:29:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:29:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:29:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:29:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:29:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:29:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:29:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:29:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:29:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:29:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:29:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:29:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:29:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:29:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:29:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:29:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:29:00.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:29:00.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:29:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:29:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:29:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:29:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:29:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:29:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:29:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:29:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:29:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:29:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:29:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:29:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:29:00.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:29:00.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:29:00.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:29:00.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:29:00.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:29:00.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:29:00.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:29:00.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:29:00.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:29:00.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=833] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:29:00.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=833] Local address: 172.21.15.161, port 58978 2022-07-03T18:29:00.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=833] Peer address: 172.21.15.101, port 22 2022-07-03T18:29:00.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=833] Connection lost 2022-07-03T18:29:00.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=833] Aborting connection 2022-07-03T18:29:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:29:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:29:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:29:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:29:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:29:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:29:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:29:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:29:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:29:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:29:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:29:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:29:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:29:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:29:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:29:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:29:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:29:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:29:00.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:29:00.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:29:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:29:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:29:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:29:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:29:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:29:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:29:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:29:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:29:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:29:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:29:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:29:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:29:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:29:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:29:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:29:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:29:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:29:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:29:00.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:29:00.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:29:00.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:29:00.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:29:00.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:29:00.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:29:00.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:29:00.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:29:00.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=833] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:29:00.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=833] Local address: 172.21.15.161, port 58978 2022-07-03T18:29:00.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=833] Peer address: 172.21.15.101, port 22 2022-07-03T18:29:00.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=833] Connection lost 2022-07-03T18:29:00.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=833] Aborting connection 2022-07-03T18:29:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:01 smithi161 ceph-mon[91579]: pgmap v14095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:01 smithi188 ceph-mon[92425]: pgmap v14095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:03 smithi161 ceph-mon[91579]: pgmap v14096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:03 smithi188 ceph-mon[92425]: pgmap v14096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:05 smithi161 ceph-mon[91579]: pgmap v14097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:05 smithi188 ceph-mon[92425]: pgmap v14097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:07 smithi161 ceph-mon[91579]: pgmap v14098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:07 smithi188 ceph-mon[92425]: pgmap v14098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:09 smithi161 ceph-mon[91579]: pgmap v14099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:09 smithi188 ceph-mon[92425]: pgmap v14099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:11 smithi161 ceph-mon[91579]: pgmap v14100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:11 smithi188 ceph-mon[92425]: pgmap v14100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:13 smithi161 ceph-mon[91579]: pgmap v14101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:13 smithi188 ceph-mon[92425]: pgmap v14101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:15 smithi161 ceph-mon[91579]: pgmap v14102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:15 smithi188 ceph-mon[92425]: pgmap v14102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:17 smithi161 ceph-mon[91579]: pgmap v14103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:17 smithi188 ceph-mon[92425]: pgmap v14103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:19 smithi161 ceph-mon[91579]: pgmap v14104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:19 smithi188 ceph-mon[92425]: pgmap v14104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:21 smithi161 ceph-mon[91579]: pgmap v14105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:21 smithi188 ceph-mon[92425]: pgmap v14105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:22 smithi161 ceph-mon[91579]: pgmap v14106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:22 smithi188 ceph-mon[92425]: pgmap v14106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:25 smithi161 ceph-mon[91579]: pgmap v14107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:25 smithi188 ceph-mon[92425]: pgmap v14107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:27 smithi161 ceph-mon[91579]: pgmap v14108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:27 smithi188 ceph-mon[92425]: pgmap v14108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:29 smithi161 ceph-mon[91579]: pgmap v14109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:29 smithi188 ceph-mon[92425]: pgmap v14109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:31 smithi161 ceph-mon[91579]: pgmap v14110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:31 smithi188 ceph-mon[92425]: pgmap v14110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:33 smithi161 ceph-mon[91579]: pgmap v14111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:33 smithi188 ceph-mon[92425]: pgmap v14111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:35 smithi161 ceph-mon[91579]: pgmap v14112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:35 smithi188 ceph-mon[92425]: pgmap v14112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:37 smithi161 ceph-mon[91579]: pgmap v14113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:37 smithi188 ceph-mon[92425]: pgmap v14113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:39 smithi161 ceph-mon[91579]: pgmap v14114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:39 smithi188 ceph-mon[92425]: pgmap v14114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:40.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:40.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:41 smithi161 ceph-mon[91579]: pgmap v14115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:41 smithi188 ceph-mon[92425]: pgmap v14115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:43 smithi161 ceph-mon[91579]: pgmap v14116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:43 smithi188 ceph-mon[92425]: pgmap v14116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:45 smithi161 ceph-mon[91579]: pgmap v14117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:45 smithi188 ceph-mon[92425]: pgmap v14117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:47 smithi161 ceph-mon[91579]: pgmap v14118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:47 smithi188 ceph-mon[92425]: pgmap v14118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:49 smithi161 ceph-mon[91579]: pgmap v14119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:49 smithi188 ceph-mon[92425]: pgmap v14119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:51.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:51 smithi161 ceph-mon[91579]: pgmap v14120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:51 smithi188 ceph-mon[92425]: pgmap v14120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:53 smithi161 ceph-mon[91579]: pgmap v14121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:53 smithi188 ceph-mon[92425]: pgmap v14121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:54.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:55 smithi161 ceph-mon[91579]: pgmap v14122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:55 smithi188 ceph-mon[92425]: pgmap v14122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:29:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:57 smithi161 ceph-mon[91579]: pgmap v14123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:57 smithi188 ceph-mon[92425]: pgmap v14123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: pgmap v14124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:29:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:29:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:29:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:29:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:29:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:29:59.185+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:29:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:29:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:29:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:29:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:29:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:29:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:29:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:29:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:29:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:29:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:29:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:29:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:29:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:29:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:29:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:29:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:29:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:29:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:29:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:29:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:29:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:29:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:29:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:29:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:29:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:29:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:29:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:29:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:29:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:29:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:29:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:29:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:29:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:29:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:29:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:29:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:29:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:29:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:29:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:29:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:29:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:29:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:29:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:29:59.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:29:59.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=835] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:29:59.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=835] Local address: 172.21.15.161, port 58982 2022-07-03T18:29:59.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=835] Peer address: 172.21.15.101, port 22 2022-07-03T18:29:59.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=835] Connection lost 2022-07-03T18:29:59.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=835] Aborting connection 2022-07-03T18:29:59.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:29:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: pgmap v14124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:29:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:29:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:29:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:29:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:29:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:30:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:30:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:30:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:30:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:30:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:30:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:30:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:30:00.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:30:00.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:30:00.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:30:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:30:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:30:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:30:00.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:30:00.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:30:00.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:30:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:30:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:30:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:30:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:30:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:30:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:30:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:30:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:30:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:30:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:30:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:30:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:30:00.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:30:00.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:30:00.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:30:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:30:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:30:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:30:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:30:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:30:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:30:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:30:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:30:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:30:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:30:00.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:30:00.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:30:00.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:30:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:30:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:30:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:30:00.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:30:00.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=835] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:30:00.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=835] Local address: 172.21.15.161, port 58982 2022-07-03T18:30:00.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=835] Peer address: 172.21.15.101, port 22 2022-07-03T18:30:00.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=835] Connection lost 2022-07-03T18:30:00.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=835] Aborting connection 2022-07-03T18:30:00.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:30:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:30:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:30:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:30:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:30:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:30:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:30:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:30:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:30:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:30:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:30:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:30:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:30:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:30:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:30:00.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:30:00.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:30:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:30:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:30:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:30:00.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:30:00.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:30:00.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:30:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:30:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:30:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:30:00.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:30:00.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:30:00.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:30:00.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:30:00.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:30:00.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:30:00.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:30:00.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:30:00.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:30:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:30:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:30:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:30:00.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:30:00.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:30:00.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:30:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:30:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:30:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:30:00.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:30:00.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:30:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:30:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:30:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:30:00.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:30:00.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=835] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:30:00.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=835] Local address: 172.21.15.161, port 58982 2022-07-03T18:30:00.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=835] Peer address: 172.21.15.101, port 22 2022-07-03T18:30:00.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=835] Connection lost 2022-07-03T18:30:00.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=835] Aborting connection 2022-07-03T18:30:00.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:30:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:01 smithi161 ceph-mon[91579]: pgmap v14125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:01 smithi188 ceph-mon[92425]: pgmap v14125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: pgmap v14126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: pgmap v14126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:30:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:30:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 15875k 2022-07-03T18:30:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 143.9M 2022-07-03T18:30:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 16256344: error parsing value: Value '16256344' is below minimum 939524096 2022-07-03T18:30:04.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 150993739: error parsing value: Value '150993739' is below minimum 939524096 2022-07-03T18:30:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 15875k 2022-07-03T18:30:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 143.9M 2022-07-03T18:30:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 16256344: error parsing value: Value '16256344' is below minimum 939524096 2022-07-03T18:30:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 150993739: error parsing value: Value '150993739' is below minimum 939524096 2022-07-03T18:30:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:05 smithi161 ceph-mon[91579]: pgmap v14127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:05 smithi188 ceph-mon[92425]: pgmap v14127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:07 smithi161 ceph-mon[91579]: pgmap v14128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:07 smithi188 ceph-mon[92425]: pgmap v14128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:09 smithi161 ceph-mon[91579]: pgmap v14129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:09 smithi188 ceph-mon[92425]: pgmap v14129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:11 smithi161 ceph-mon[91579]: pgmap v14130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:11 smithi188 ceph-mon[92425]: pgmap v14130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:13 smithi161 ceph-mon[91579]: pgmap v14131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:13 smithi188 ceph-mon[92425]: pgmap v14131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:15 smithi161 ceph-mon[91579]: pgmap v14132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:15 smithi188 ceph-mon[92425]: pgmap v14132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:17.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:17 smithi161 ceph-mon[91579]: pgmap v14133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:17 smithi188 ceph-mon[92425]: pgmap v14133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:19 smithi161 ceph-mon[91579]: pgmap v14134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:19 smithi188 ceph-mon[92425]: pgmap v14134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:21 smithi161 ceph-mon[91579]: pgmap v14135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:21 smithi188 ceph-mon[92425]: pgmap v14135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:23 smithi161 ceph-mon[91579]: pgmap v14136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:23 smithi188 ceph-mon[92425]: pgmap v14136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:25 smithi161 ceph-mon[91579]: pgmap v14137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:25 smithi188 ceph-mon[92425]: pgmap v14137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:27 smithi161 ceph-mon[91579]: pgmap v14138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:27 smithi188 ceph-mon[92425]: pgmap v14138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:29 smithi161 ceph-mon[91579]: pgmap v14139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:29 smithi188 ceph-mon[92425]: pgmap v14139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:31 smithi161 ceph-mon[91579]: pgmap v14140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:31 smithi188 ceph-mon[92425]: pgmap v14140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:33.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:33 smithi161 ceph-mon[91579]: pgmap v14141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:33 smithi188 ceph-mon[92425]: pgmap v14141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:35 smithi161 ceph-mon[91579]: pgmap v14142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:35 smithi188 ceph-mon[92425]: pgmap v14142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:37.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:37 smithi161 ceph-mon[91579]: pgmap v14143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:37 smithi188 ceph-mon[92425]: pgmap v14143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:39 smithi161 ceph-mon[91579]: pgmap v14144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:39 smithi188 ceph-mon[92425]: pgmap v14144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:41 smithi161 ceph-mon[91579]: pgmap v14145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:41 smithi188 ceph-mon[92425]: pgmap v14145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:43 smithi161 ceph-mon[91579]: pgmap v14146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:43 smithi188 ceph-mon[92425]: pgmap v14146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:45 smithi161 ceph-mon[91579]: pgmap v14147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:45 smithi188 ceph-mon[92425]: pgmap v14147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:47 smithi161 ceph-mon[91579]: pgmap v14148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:47 smithi188 ceph-mon[92425]: pgmap v14148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:49 smithi161 ceph-mon[91579]: pgmap v14149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:49 smithi188 ceph-mon[92425]: pgmap v14149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:51 smithi161 ceph-mon[91579]: pgmap v14150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:51 smithi188 ceph-mon[92425]: pgmap v14150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:53 smithi161 ceph-mon[91579]: pgmap v14151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:53 smithi188 ceph-mon[92425]: pgmap v14151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:55 smithi188 ceph-mon[92425]: pgmap v14152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:55 smithi161 ceph-mon[91579]: pgmap v14152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:30:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:57 smithi188 ceph-mon[92425]: pgmap v14153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:57 smithi161 ceph-mon[91579]: pgmap v14153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: pgmap v14154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:30:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:30:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:30:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:30:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:30:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: pgmap v14154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:30:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:30:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:30:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:30:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:30:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:01 smithi188 ceph-mon[92425]: pgmap v14155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:01 smithi161 ceph-mon[91579]: pgmap v14155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:31:02.447+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:31:02.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:31:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:31:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:31:02.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:31:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:31:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:31:02.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:31:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:31:02.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:31:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:31:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:31:02.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:31:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:31:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:31:02.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:31:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:31:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:31:02.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:31:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:31:02.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:31:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:31:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:31:02.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:31:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:31:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:31:02.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:31:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:31:02.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:31:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:31:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:31:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:31:02.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:31:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:31:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:31:02.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:31:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:31:02.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:31:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:31:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:31:02.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:31:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:31:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:31:02.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:31:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:31:02.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=837] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:31:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=837] Local address: 172.21.15.161, port 58986 2022-07-03T18:31:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=837] Peer address: 172.21.15.101, port 22 2022-07-03T18:31:02.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=837] Connection lost 2022-07-03T18:31:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=837] Aborting connection 2022-07-03T18:31:02.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:31:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:31:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:03 smithi188 ceph-mon[92425]: pgmap v14156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:31:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:31:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:31:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:03 smithi161 ceph-mon[91579]: pgmap v14156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:31:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:31:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:31:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:31:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:31:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:31:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:31:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:31:04.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:31:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:31:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:31:04.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:31:04.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:31:04.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:31:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:31:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:31:04.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:31:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:31:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:31:04.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:31:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:31:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:31:04.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:31:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:31:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:31:04.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:31:04.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:31:04.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:31:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:31:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:31:04.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:31:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:31:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:31:04.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:31:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:31:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:31:04.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:31:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:31:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:31:04.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:31:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:31:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:31:04.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:31:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:31:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:31:04.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:31:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:31:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:31:04.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:31:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=837] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:31:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=837] Local address: 172.21.15.161, port 58986 2022-07-03T18:31:04.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=837] Peer address: 172.21.15.101, port 22 2022-07-03T18:31:04.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=837] Connection lost 2022-07-03T18:31:04.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=837] Aborting connection 2022-07-03T18:31:04.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:31:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:31:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:31:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:31:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:31:04.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:31:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:31:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:31:04.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:31:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:31:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:31:04.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:31:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:31:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:31:04.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:31:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:31:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:31:04.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:31:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:31:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:31:04.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:31:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:31:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:31:04.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:31:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:31:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:31:04.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:31:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:31:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:31:04.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:31:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:31:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:31:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:31:04.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:31:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:31:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:31:04.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:31:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:31:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:31:04.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:31:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:31:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:31:04.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:31:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:31:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:31:04.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:31:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=837] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:31:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=837] Local address: 172.21.15.161, port 58986 2022-07-03T18:31:04.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=837] Peer address: 172.21.15.101, port 22 2022-07-03T18:31:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=837] Connection lost 2022-07-03T18:31:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=837] Aborting connection 2022-07-03T18:31:04.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:05 smithi188 ceph-mon[92425]: pgmap v14157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:05 smithi161 ceph-mon[91579]: pgmap v14157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:07 smithi188 ceph-mon[92425]: pgmap v14158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:07 smithi161 ceph-mon[91579]: pgmap v14158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:09 smithi188 ceph-mon[92425]: pgmap v14159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:09 smithi161 ceph-mon[91579]: pgmap v14159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:11 smithi188 ceph-mon[92425]: pgmap v14160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:11 smithi161 ceph-mon[91579]: pgmap v14160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:13 smithi161 ceph-mon[91579]: pgmap v14161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:13 smithi188 ceph-mon[92425]: pgmap v14161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:15 smithi161 ceph-mon[91579]: pgmap v14162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:15 smithi188 ceph-mon[92425]: pgmap v14162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:17 smithi161 ceph-mon[91579]: pgmap v14163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:17 smithi188 ceph-mon[92425]: pgmap v14163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:19 smithi161 ceph-mon[91579]: pgmap v14164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:19 smithi188 ceph-mon[92425]: pgmap v14164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:21 smithi161 ceph-mon[91579]: pgmap v14165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:21 smithi188 ceph-mon[92425]: pgmap v14165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:23 smithi161 ceph-mon[91579]: pgmap v14166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:23 smithi188 ceph-mon[92425]: pgmap v14166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:25 smithi161 ceph-mon[91579]: pgmap v14167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:25 smithi188 ceph-mon[92425]: pgmap v14167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:27 smithi161 ceph-mon[91579]: pgmap v14168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:27 smithi188 ceph-mon[92425]: pgmap v14168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:29 smithi161 ceph-mon[91579]: pgmap v14169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:29 smithi188 ceph-mon[92425]: pgmap v14169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:31 smithi161 ceph-mon[91579]: pgmap v14170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:31 smithi188 ceph-mon[92425]: pgmap v14170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:33 smithi161 ceph-mon[91579]: pgmap v14171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:33 smithi188 ceph-mon[92425]: pgmap v14171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:35 smithi161 ceph-mon[91579]: pgmap v14172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:35 smithi188 ceph-mon[92425]: pgmap v14172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:37 smithi161 ceph-mon[91579]: pgmap v14173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:37 smithi188 ceph-mon[92425]: pgmap v14173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:39 smithi161 ceph-mon[91579]: pgmap v14174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:39 smithi188 ceph-mon[92425]: pgmap v14174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:31:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:41 smithi161 ceph-mon[91579]: pgmap v14175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:41 smithi188 ceph-mon[92425]: pgmap v14175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:43 smithi161 ceph-mon[91579]: pgmap v14176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:43 smithi188 ceph-mon[92425]: pgmap v14176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:45 smithi161 ceph-mon[91579]: pgmap v14177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:45 smithi188 ceph-mon[92425]: pgmap v14177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:47 smithi161 ceph-mon[91579]: pgmap v14178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:47 smithi188 ceph-mon[92425]: pgmap v14178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:49 smithi161 ceph-mon[91579]: pgmap v14179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:49 smithi188 ceph-mon[92425]: pgmap v14179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:51 smithi161 ceph-mon[91579]: pgmap v14180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:51 smithi188 ceph-mon[92425]: pgmap v14180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:53 smithi161 ceph-mon[91579]: pgmap v14181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:53 smithi188 ceph-mon[92425]: pgmap v14181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:55 smithi161 ceph-mon[91579]: pgmap v14182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:55 smithi188 ceph-mon[92425]: pgmap v14182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:31:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:57 smithi161 ceph-mon[91579]: pgmap v14183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:57 smithi188 ceph-mon[92425]: pgmap v14183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: pgmap v14184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:31:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:31:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:31:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:31:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:31:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: pgmap v14184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:31:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:31:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:31:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:31:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:31:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:00 smithi161 ceph-mon[91579]: pgmap v14185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:00 smithi188 ceph-mon[92425]: pgmap v14185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-mon[91579]: pgmap v14186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:02 smithi188 ceph-mon[92425]: pgmap v14186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:03.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:32:02.950+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:32:03.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:32:03.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:32:03.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:32:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:32:03.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:32:03.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:32:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:32:03.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:32:03.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:32:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:32:03.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:32:03.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:32:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:32:03.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:32:03.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:32:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:32:03.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:32:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:32:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:32:03.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:32:03.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:32:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:32:03.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:32:03.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:32:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:32:03.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:32:03.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:32:03.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:32:03.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:32:03.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:32:03.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:32:03.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:32:03.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:32:03.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:32:03.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:32:03.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:32:03.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:32:03.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:32:03.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:32:03.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:32:03.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:32:03.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:32:03.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:32:03.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:32:03.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=839] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:32:03.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=839] Local address: 172.21.15.161, port 58990 2022-07-03T18:32:03.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=839] Peer address: 172.21.15.101, port 22 2022-07-03T18:32:03.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=839] Connection lost 2022-07-03T18:32:03.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=839] Aborting connection 2022-07-03T18:32:03.413 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:32:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:32:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:32:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:32:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:32:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:32:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:32:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:32:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:32:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:32:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:32:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:32:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:32:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:32:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:32:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:32:03.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:32:03.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:32:03.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:32:03.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:32:03.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:32:03.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:32:03.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:32:03.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:32:03.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:32:03.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:32:03.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:32:03.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:32:03.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:32:03.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:32:03.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:32:03.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:32:03.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:32:03.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:32:03.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:32:03.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:32:03.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:32:03.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:32:03.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:32:03.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:32:03.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:32:03.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:32:03.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:32:03.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:32:03.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:32:03.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:32:03.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:32:03.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:32:03.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:32:03.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:32:03.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:32:03.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=839] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:32:03.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=839] Local address: 172.21.15.161, port 58990 2022-07-03T18:32:03.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=839] Peer address: 172.21.15.101, port 22 2022-07-03T18:32:03.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=839] Connection lost 2022-07-03T18:32:03.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=839] Aborting connection 2022-07-03T18:32:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:32:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:32:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:32:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:32:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:32:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:32:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:32:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:32:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:32:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:32:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:32:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:32:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:32:03.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:32:03.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:32:03.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:32:03.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:32:03.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:32:03.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:32:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:32:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:32:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:32:03.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:32:03.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:32:03.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:32:03.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:32:03.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:32:03.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:32:03.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:32:03.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:32:03.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:32:03.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:32:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:32:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:32:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:32:03.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:32:03.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:32:03.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:32:03.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:32:03.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:32:03.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:32:03.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:32:03.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:32:03.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:32:03.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:32:03.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:32:03.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:32:03.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:32:03.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:32:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=839] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:32:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=839] Local address: 172.21.15.161, port 58990 2022-07-03T18:32:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=839] Peer address: 172.21.15.101, port 22 2022-07-03T18:32:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=839] Connection lost 2022-07-03T18:32:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=839] Aborting connection 2022-07-03T18:32:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:05 smithi161 ceph-mon[91579]: pgmap v14187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:05 smithi188 ceph-mon[92425]: pgmap v14187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:07 smithi161 ceph-mon[91579]: pgmap v14188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:07 smithi188 ceph-mon[92425]: pgmap v14188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:09 smithi161 ceph-mon[91579]: pgmap v14189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:09 smithi188 ceph-mon[92425]: pgmap v14189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:11 smithi161 ceph-mon[91579]: pgmap v14190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:11 smithi188 ceph-mon[92425]: pgmap v14190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:13 smithi161 ceph-mon[91579]: pgmap v14191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:13 smithi188 ceph-mon[92425]: pgmap v14191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:15 smithi161 ceph-mon[91579]: pgmap v14192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:15 smithi188 ceph-mon[92425]: pgmap v14192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:17 smithi161 ceph-mon[91579]: pgmap v14193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:17 smithi188 ceph-mon[92425]: pgmap v14193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:19 smithi161 ceph-mon[91579]: pgmap v14194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:19 smithi188 ceph-mon[92425]: pgmap v14194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:21 smithi161 ceph-mon[91579]: pgmap v14195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:21 smithi188 ceph-mon[92425]: pgmap v14195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:23 smithi161 ceph-mon[91579]: pgmap v14196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:23 smithi188 ceph-mon[92425]: pgmap v14196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:25 smithi161 ceph-mon[91579]: pgmap v14197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:25 smithi188 ceph-mon[92425]: pgmap v14197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:27.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:27 smithi161 ceph-mon[91579]: pgmap v14198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:27 smithi188 ceph-mon[92425]: pgmap v14198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:29.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:29 smithi161 ceph-mon[91579]: pgmap v14199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:29 smithi188 ceph-mon[92425]: pgmap v14199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:31 smithi161 ceph-mon[91579]: pgmap v14200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:31 smithi188 ceph-mon[92425]: pgmap v14200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:33 smithi188 ceph-mon[92425]: pgmap v14201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:33 smithi161 ceph-mon[91579]: pgmap v14201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:35 smithi188 ceph-mon[92425]: pgmap v14202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:35 smithi161 ceph-mon[91579]: pgmap v14202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:37 smithi188 ceph-mon[92425]: pgmap v14203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:37 smithi161 ceph-mon[91579]: pgmap v14203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:39 smithi188 ceph-mon[92425]: pgmap v14204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:39 smithi161 ceph-mon[91579]: pgmap v14204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:40.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:41 smithi188 ceph-mon[92425]: pgmap v14205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:41 smithi161 ceph-mon[91579]: pgmap v14205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:43 smithi188 ceph-mon[92425]: pgmap v14206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:43 smithi161 ceph-mon[91579]: pgmap v14206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:45 smithi188 ceph-mon[92425]: pgmap v14207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:45 smithi161 ceph-mon[91579]: pgmap v14207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:47 smithi161 ceph-mon[91579]: pgmap v14208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:47 smithi188 ceph-mon[92425]: pgmap v14208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:49 smithi161 ceph-mon[91579]: pgmap v14209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:49 smithi188 ceph-mon[92425]: pgmap v14209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:51 smithi161 ceph-mon[91579]: pgmap v14210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:51 smithi188 ceph-mon[92425]: pgmap v14210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:53 smithi161 ceph-mon[91579]: pgmap v14211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:53 smithi188 ceph-mon[92425]: pgmap v14211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:55 smithi161 ceph-mon[91579]: pgmap v14212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:55 smithi188 ceph-mon[92425]: pgmap v14212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:32:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:57 smithi161 ceph-mon[91579]: pgmap v14213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:57 smithi188 ceph-mon[92425]: pgmap v14213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: pgmap v14214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:32:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:32:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:32:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:32:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:32:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: pgmap v14214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:32:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:32:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:32:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:32:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:32:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:01 smithi161 ceph-mon[91579]: pgmap v14215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:01 smithi188 ceph-mon[92425]: pgmap v14215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:03.457 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-mon[91579]: pgmap v14216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:03.466 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:03 smithi188 ceph-mon[92425]: pgmap v14216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:33:03.454+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:33:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:33:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:33:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:33:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:33:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:33:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:33:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:33:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:33:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:33:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:33:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:33:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:33:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:33:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:33:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:33:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:33:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:33:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:33:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:33:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:33:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:33:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:33:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:33:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:33:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:33:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:33:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:33:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:33:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:33:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:33:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:33:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:33:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:33:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:33:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:33:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:33:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:33:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:33:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:33:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:33:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:33:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:33:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:33:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:33:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=841] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:33:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=841] Local address: 172.21.15.161, port 58994 2022-07-03T18:33:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=841] Peer address: 172.21.15.101, port 22 2022-07-03T18:33:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=841] Connection lost 2022-07-03T18:33:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=841] Aborting connection 2022-07-03T18:33:03.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:33:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:33:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:33:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:33:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:33:04.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:33:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:33:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:33:04.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:05 smithi161 ceph-mon[91579]: pgmap v14217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:33:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:33:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:33:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:33:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:33:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:33:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:33:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:33:05.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:33:05.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:33:05.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:33:05.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:33:05.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:33:05.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:33:05.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:33:05.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:33:05.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:33:05.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:33:05.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:33:05.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:33:05.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:33:05.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:33:05.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:33:05.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:33:05.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:33:05.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:33:05.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:33:05.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:33:05.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:33:05.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:33:05.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:33:05.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:33:05.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:33:05.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:33:05.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:33:05.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:33:05.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:33:05.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:33:05.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:33:05.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:33:05.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:33:05.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:33:05.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:33:05.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:33:05.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:33:05.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:33:05.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=841] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:33:05.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=841] Local address: 172.21.15.161, port 58994 2022-07-03T18:33:05.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=841] Peer address: 172.21.15.101, port 22 2022-07-03T18:33:05.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=841] Connection lost 2022-07-03T18:33:05.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=841] Aborting connection 2022-07-03T18:33:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:05 smithi188 ceph-mon[92425]: pgmap v14217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:33:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:33:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:33:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:33:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:33:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:33:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:33:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:33:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:33:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:33:05.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:33:05.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:33:05.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:33:05.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:33:05.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:33:05.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:33:05.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:33:05.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:33:05.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:33:05.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:33:05.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:33:05.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:33:05.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:33:05.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:33:05.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:33:05.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:33:05.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:33:05.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:33:05.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:33:05.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:33:05.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:33:05.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:33:05.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:33:05.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:33:05.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:33:05.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:33:05.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:33:05.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:33:05.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:33:05.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:33:05.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:33:05.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:33:05.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:33:05.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:33:05.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:33:05.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:33:05.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=841] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:33:05.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=841] Local address: 172.21.15.161, port 58994 2022-07-03T18:33:05.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=841] Peer address: 172.21.15.101, port 22 2022-07-03T18:33:05.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=841] Connection lost 2022-07-03T18:33:05.695 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=841] Aborting connection 2022-07-03T18:33:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:07 smithi161 ceph-mon[91579]: pgmap v14218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:07 smithi188 ceph-mon[92425]: pgmap v14218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:09 smithi161 ceph-mon[91579]: pgmap v14219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:09 smithi188 ceph-mon[92425]: pgmap v14219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:11 smithi161 ceph-mon[91579]: pgmap v14220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:11 smithi188 ceph-mon[92425]: pgmap v14220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:13 smithi161 ceph-mon[91579]: pgmap v14221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:13 smithi188 ceph-mon[92425]: pgmap v14221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:15 smithi161 ceph-mon[91579]: pgmap v14222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:15 smithi188 ceph-mon[92425]: pgmap v14222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:17 smithi161 ceph-mon[91579]: pgmap v14223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:17 smithi188 ceph-mon[92425]: pgmap v14223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:19 smithi161 ceph-mon[91579]: pgmap v14224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:19 smithi188 ceph-mon[92425]: pgmap v14224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:21 smithi161 ceph-mon[91579]: pgmap v14225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:21 smithi188 ceph-mon[92425]: pgmap v14225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:23 smithi161 ceph-mon[91579]: pgmap v14226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:23 smithi188 ceph-mon[92425]: pgmap v14226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:25 smithi161 ceph-mon[91579]: pgmap v14227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:25 smithi188 ceph-mon[92425]: pgmap v14227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:27 smithi161 ceph-mon[91579]: pgmap v14228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:27 smithi188 ceph-mon[92425]: pgmap v14228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:29 smithi161 ceph-mon[91579]: pgmap v14229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:29 smithi188 ceph-mon[92425]: pgmap v14229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:31 smithi161 ceph-mon[91579]: pgmap v14230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:31 smithi188 ceph-mon[92425]: pgmap v14230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:33 smithi161 ceph-mon[91579]: pgmap v14231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:33 smithi188 ceph-mon[92425]: pgmap v14231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:35 smithi161 ceph-mon[91579]: pgmap v14232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:35 smithi188 ceph-mon[92425]: pgmap v14232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:37 smithi161 ceph-mon[91579]: pgmap v14233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:37 smithi188 ceph-mon[92425]: pgmap v14233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:39 smithi161 ceph-mon[91579]: pgmap v14234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:39 smithi188 ceph-mon[92425]: pgmap v14234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:41 smithi161 ceph-mon[91579]: pgmap v14235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:41 smithi188 ceph-mon[92425]: pgmap v14235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:42 smithi161 ceph-mon[91579]: pgmap v14236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:42 smithi188 ceph-mon[92425]: pgmap v14236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:45.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:45 smithi161 ceph-mon[91579]: pgmap v14237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:45.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:45.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:45 smithi188 ceph-mon[92425]: pgmap v14237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:45.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:47.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:47 smithi161 ceph-mon[91579]: pgmap v14238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:47 smithi188 ceph-mon[92425]: pgmap v14238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:49 smithi161 ceph-mon[91579]: pgmap v14239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:49 smithi188 ceph-mon[92425]: pgmap v14239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:50.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:51.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:51 smithi161 ceph-mon[91579]: pgmap v14240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:51 smithi188 ceph-mon[92425]: pgmap v14240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:53 smithi161 ceph-mon[91579]: pgmap v14241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:53 smithi188 ceph-mon[92425]: pgmap v14241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:55.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:55 smithi161 ceph-mon[91579]: pgmap v14242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:55.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:55 smithi188 ceph-mon[92425]: pgmap v14242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:33:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:33:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:57 smithi161 ceph-mon[91579]: pgmap v14243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:57 smithi188 ceph-mon[92425]: pgmap v14243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:59 smithi161 ceph-mon[91579]: pgmap v14244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:33:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:33:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:33:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:33:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:59 smithi188 ceph-mon[92425]: pgmap v14244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:33:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:33:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:33:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:33:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:34:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:01.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:01 smithi161 ceph-mon[91579]: pgmap v14245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:01 smithi188 ceph-mon[92425]: pgmap v14245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-mon[91579]: pgmap v14246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:03 smithi188 ceph-mon[92425]: pgmap v14246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:04.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:34:04.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:34:04.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:34:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:34:03.953+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:34:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:34:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:34:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:34:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:34:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:34:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:34:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:34:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:34:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:34:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:34:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:34:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:34:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:34:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:34:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:34:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:34:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:34:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:34:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:34:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:34:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:34:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:34:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:34:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:34:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:34:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:34:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:34:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:34:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:34:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:34:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:34:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:34:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:34:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:34:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:34:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:34:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:34:04.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:34:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:34:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:34:04.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:34:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:34:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:34:04.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:34:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=843] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:34:04.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=843] Local address: 172.21.15.161, port 58998 2022-07-03T18:34:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=843] Peer address: 172.21.15.101, port 22 2022-07-03T18:34:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=843] Connection lost 2022-07-03T18:34:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=843] Aborting connection 2022-07-03T18:34:04.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:34:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:34:04.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:34:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:34:04.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:34:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:05 smithi161 ceph-mon[91579]: pgmap v14247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:34:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:34:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:34:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:34:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:34:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:34:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:34:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:34:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:34:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:34:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:34:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:34:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:34:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:34:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:34:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:34:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:34:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:34:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:34:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:34:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:34:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:34:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:34:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:34:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:34:05.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:34:05.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:34:05.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:34:05.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:34:05.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:34:05.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:34:05.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:34:05.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:34:05.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:34:05.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:34:05.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:34:05.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:34:05.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:34:05.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:34:05.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:34:05.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:34:05.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:34:05.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:34:05.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=843] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:34:05.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=843] Local address: 172.21.15.161, port 58998 2022-07-03T18:34:05.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=843] Peer address: 172.21.15.101, port 22 2022-07-03T18:34:05.411 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=843] Connection lost 2022-07-03T18:34:05.412 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=843] Aborting connection 2022-07-03T18:34:05.412 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:05 smithi188 ceph-mon[92425]: pgmap v14247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:34:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:34:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:34:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:34:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:34:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:34:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:34:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:34:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:34:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:34:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:34:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:34:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:34:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:34:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:34:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:34:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:34:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:34:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:34:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:34:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:34:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:34:05.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=843] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:34:05.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=843] Local address: 172.21.15.161, port 58998 2022-07-03T18:34:05.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=843] Peer address: 172.21.15.101, port 22 2022-07-03T18:34:05.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=843] Connection lost 2022-07-03T18:34:05.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=843] Aborting connection 2022-07-03T18:34:05.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:07 smithi188 ceph-mon[92425]: pgmap v14248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:07 smithi161 ceph-mon[91579]: pgmap v14248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:09 smithi188 ceph-mon[92425]: pgmap v14249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:09 smithi161 ceph-mon[91579]: pgmap v14249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:11 smithi188 ceph-mon[92425]: pgmap v14250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:11 smithi161 ceph-mon[91579]: pgmap v14250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:13.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:13 smithi188 ceph-mon[92425]: pgmap v14251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:13 smithi161 ceph-mon[91579]: pgmap v14251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:15 smithi188 ceph-mon[92425]: pgmap v14252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:15 smithi161 ceph-mon[91579]: pgmap v14252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:17.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:17 smithi188 ceph-mon[92425]: pgmap v14253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:17 smithi161 ceph-mon[91579]: pgmap v14253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:19 smithi188 ceph-mon[92425]: pgmap v14254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:19 smithi161 ceph-mon[91579]: pgmap v14254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:21 smithi161 ceph-mon[91579]: pgmap v14255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:21 smithi188 ceph-mon[92425]: pgmap v14255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:23 smithi161 ceph-mon[91579]: pgmap v14256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:23 smithi188 ceph-mon[92425]: pgmap v14256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:25 smithi161 ceph-mon[91579]: pgmap v14257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:25 smithi188 ceph-mon[92425]: pgmap v14257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:27 smithi161 ceph-mon[91579]: pgmap v14258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:27 smithi188 ceph-mon[92425]: pgmap v14258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:29 smithi161 ceph-mon[91579]: pgmap v14259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:29 smithi188 ceph-mon[92425]: pgmap v14259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:31 smithi161 ceph-mon[91579]: pgmap v14260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:31 smithi188 ceph-mon[92425]: pgmap v14260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:33 smithi161 ceph-mon[91579]: pgmap v14261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:33 smithi188 ceph-mon[92425]: pgmap v14261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:35 smithi161 ceph-mon[91579]: pgmap v14262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:35 smithi188 ceph-mon[92425]: pgmap v14262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:37 smithi161 ceph-mon[91579]: pgmap v14263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:37 smithi188 ceph-mon[92425]: pgmap v14263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:39 smithi161 ceph-mon[91579]: pgmap v14264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:39 smithi188 ceph-mon[92425]: pgmap v14264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:41 smithi161 ceph-mon[91579]: pgmap v14265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:41 smithi188 ceph-mon[92425]: pgmap v14265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:43 smithi161 ceph-mon[91579]: pgmap v14266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:43 smithi188 ceph-mon[92425]: pgmap v14266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:45 smithi161 ceph-mon[91579]: pgmap v14267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:45 smithi188 ceph-mon[92425]: pgmap v14267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:47 smithi161 ceph-mon[91579]: pgmap v14268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:47 smithi188 ceph-mon[92425]: pgmap v14268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:49 smithi161 ceph-mon[91579]: pgmap v14269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:49.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:49 smithi188 ceph-mon[92425]: pgmap v14269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:49.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:51 smithi161 ceph-mon[91579]: pgmap v14270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:51 smithi188 ceph-mon[92425]: pgmap v14270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:53 smithi161 ceph-mon[91579]: pgmap v14271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:53 smithi188 ceph-mon[92425]: pgmap v14271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:55 smithi161 ceph-mon[91579]: pgmap v14272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:55 smithi188 ceph-mon[92425]: pgmap v14272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:34:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:57 smithi161 ceph-mon[91579]: pgmap v14273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:57 smithi188 ceph-mon[92425]: pgmap v14273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: pgmap v14274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:34:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:34:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:34:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:34:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:34:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: pgmap v14274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:34:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:34:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:34:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:34:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:34:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:01 smithi161 ceph-mon[91579]: pgmap v14275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:01 smithi188 ceph-mon[92425]: pgmap v14275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:03 smithi161 ceph-mon[91579]: pgmap v14276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:03 smithi188 ceph-mon[92425]: pgmap v14276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:04.520 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:04.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:35:04.462+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:35:04.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:35:04.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:35:04.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:35:04.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:35:04.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:35:04.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:35:04.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:35:04.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:35:04.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:35:04.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:35:04.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:35:04.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:35:04.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:35:04.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:35:04.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:35:04.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:35:04.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:35:04.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:35:04.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:35:04.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:35:04.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:35:04.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:35:04.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:35:04.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:35:04.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:35:04.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:35:04.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:35:04.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:35:04.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:35:04.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:35:04.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:35:04.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:35:04.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:35:04.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:35:04.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:35:04.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:35:04.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:35:04.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:35:04.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:35:04.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:35:04.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:35:04.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:35:04.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:35:04.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:35:04.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=845] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:35:04.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=845] Local address: 172.21.15.161, port 59002 2022-07-03T18:35:04.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=845] Peer address: 172.21.15.101, port 22 2022-07-03T18:35:04.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=845] Connection lost 2022-07-03T18:35:04.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=845] Aborting connection 2022-07-03T18:35:04.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:35:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:35:04.538 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:05 smithi161 ceph-mon[91579]: pgmap v14277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:35:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:35:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:35:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:05 smithi188 ceph-mon[92425]: pgmap v14277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:35:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:35:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:35:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:35:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:35:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:35:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:35:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:35:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:35:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:35:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:35:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:35:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:35:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:35:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:35:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:35:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:35:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:35:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:35:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:35:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:35:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:35:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:35:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:35:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:35:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:35:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:35:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:35:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:35:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:35:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:35:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:35:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:35:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:35:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:35:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:35:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:35:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:35:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:35:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:35:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:35:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:35:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:35:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:35:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:35:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:35:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:35:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:35:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:35:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=845] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:35:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=845] Local address: 172.21.15.161, port 59002 2022-07-03T18:35:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=845] Peer address: 172.21.15.101, port 22 2022-07-03T18:35:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=845] Connection lost 2022-07-03T18:35:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=845] Aborting connection 2022-07-03T18:35:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:35:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:35:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:35:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:35:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:35:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:35:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:35:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:35:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:35:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:35:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:35:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:35:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:35:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:35:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:35:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:35:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:35:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:35:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:35:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:35:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:35:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:35:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:35:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:35:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:35:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:35:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:35:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:35:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:35:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:35:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:35:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:35:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:35:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:35:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:35:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:35:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:35:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:35:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:35:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:35:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:35:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:35:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:35:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:35:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:35:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:35:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=845] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:35:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=845] Local address: 172.21.15.161, port 59002 2022-07-03T18:35:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=845] Peer address: 172.21.15.101, port 22 2022-07-03T18:35:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=845] Connection lost 2022-07-03T18:35:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=845] Aborting connection 2022-07-03T18:35:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:07 smithi161 ceph-mon[91579]: pgmap v14278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:07 smithi188 ceph-mon[92425]: pgmap v14278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:09 smithi161 ceph-mon[91579]: pgmap v14279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:09 smithi188 ceph-mon[92425]: pgmap v14279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:11 smithi161 ceph-mon[91579]: pgmap v14280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:11 smithi188 ceph-mon[92425]: pgmap v14280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:13 smithi161 ceph-mon[91579]: pgmap v14281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:13 smithi188 ceph-mon[92425]: pgmap v14281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:15 smithi161 ceph-mon[91579]: pgmap v14282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:15 smithi188 ceph-mon[92425]: pgmap v14282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:16 smithi161 ceph-mon[91579]: pgmap v14283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:16 smithi188 ceph-mon[92425]: pgmap v14283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:19 smithi161 ceph-mon[91579]: pgmap v14284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:19 smithi188 ceph-mon[92425]: pgmap v14284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:21 smithi161 ceph-mon[91579]: pgmap v14285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:21 smithi188 ceph-mon[92425]: pgmap v14285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:23 smithi161 ceph-mon[91579]: pgmap v14286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:23 smithi188 ceph-mon[92425]: pgmap v14286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:25 smithi188 ceph-mon[92425]: pgmap v14287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:25 smithi161 ceph-mon[91579]: pgmap v14287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:27.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:27 smithi188 ceph-mon[92425]: pgmap v14288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:27 smithi161 ceph-mon[91579]: pgmap v14288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:29 smithi188 ceph-mon[92425]: pgmap v14289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:29 smithi161 ceph-mon[91579]: pgmap v14289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:31 smithi188 ceph-mon[92425]: pgmap v14290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:31 smithi161 ceph-mon[91579]: pgmap v14290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:33.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:33 smithi188 ceph-mon[92425]: pgmap v14291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:33 smithi161 ceph-mon[91579]: pgmap v14291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:35 smithi188 ceph-mon[92425]: pgmap v14292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:35 smithi161 ceph-mon[91579]: pgmap v14292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:37.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:37 smithi188 ceph-mon[92425]: pgmap v14293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:37 smithi161 ceph-mon[91579]: pgmap v14293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:39 smithi161 ceph-mon[91579]: pgmap v14294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:39 smithi188 ceph-mon[92425]: pgmap v14294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:41 smithi161 ceph-mon[91579]: pgmap v14295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:41 smithi188 ceph-mon[92425]: pgmap v14295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:43 smithi161 ceph-mon[91579]: pgmap v14296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:43 smithi188 ceph-mon[92425]: pgmap v14296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:45 smithi161 ceph-mon[91579]: pgmap v14297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:45 smithi188 ceph-mon[92425]: pgmap v14297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:47 smithi161 ceph-mon[91579]: pgmap v14298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:47 smithi188 ceph-mon[92425]: pgmap v14298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:49 smithi161 ceph-mon[91579]: pgmap v14299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:49 smithi188 ceph-mon[92425]: pgmap v14299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:51 smithi161 ceph-mon[91579]: pgmap v14300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:51 smithi188 ceph-mon[92425]: pgmap v14300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:53 smithi161 ceph-mon[91579]: pgmap v14301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:53 smithi188 ceph-mon[92425]: pgmap v14301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:55 smithi161 ceph-mon[91579]: pgmap v14302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:55 smithi188 ceph-mon[92425]: pgmap v14302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:35:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:35:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:57 smithi161 ceph-mon[91579]: pgmap v14303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:57 smithi188 ceph-mon[92425]: pgmap v14303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:59 smithi161 ceph-mon[91579]: pgmap v14304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:35:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:35:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:35:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:35:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:59 smithi188 ceph-mon[92425]: pgmap v14304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:35:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:35:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:35:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:35:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:36:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:01 smithi161 ceph-mon[91579]: pgmap v14305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:01 smithi188 ceph-mon[92425]: pgmap v14305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:03 smithi161 ceph-mon[91579]: pgmap v14306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:03 smithi188 ceph-mon[92425]: pgmap v14306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:05.222 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:05 smithi161 ceph-mon[91579]: pgmap v14307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:05.222 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:05.223 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:36:04.963+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:36:05.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:36:05.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:36:05.224 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:36:05.225 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:36:05.225 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:36:05.225 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:36:05.226 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:36:05.226 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:36:05.226 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:36:05.227 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:36:05.227 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:36:05.227 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:36:05.228 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:36:05.228 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:36:05.228 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:36:05.229 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:36:05.229 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:36:05.229 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:36:05.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:36:05.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:36:05.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:36:05.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:36:05.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:36:05.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:36:05.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:36:05.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:36:05.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:36:05.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:36:05.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:36:05.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:36:05.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:36:05.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:36:05.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:36:05.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:36:05.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:36:05.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:36:05.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:36:05.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:36:05.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:36:05.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:36:05.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:36:05.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:36:05.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:36:05.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:36:05.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=847] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:36:05.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=847] Local address: 172.21.15.161, port 59006 2022-07-03T18:36:05.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=847] Peer address: 172.21.15.101, port 22 2022-07-03T18:36:05.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=847] Connection lost 2022-07-03T18:36:05.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=847] Aborting connection 2022-07-03T18:36:05.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:36:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:36:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:05 smithi188 ceph-mon[92425]: pgmap v14307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:36:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:36:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:36:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:36:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:36:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:36:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:36:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:36:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:36:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:36:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:36:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:36:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:36:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:36:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:36:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:36:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:36:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:36:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:36:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:36:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:36:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:36:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:36:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:36:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:36:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:36:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:36:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:36:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:36:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:36:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:36:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:36:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:36:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:36:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:36:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:36:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:36:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:36:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:36:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:36:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:36:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:36:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:36:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:36:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:36:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:36:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:36:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:36:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:36:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:36:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:36:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:36:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=847] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:36:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=847] Local address: 172.21.15.161, port 59006 2022-07-03T18:36:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=847] Peer address: 172.21.15.101, port 22 2022-07-03T18:36:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=847] Connection lost 2022-07-03T18:36:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=847] Aborting connection 2022-07-03T18:36:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:36:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:36:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:36:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:36:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:36:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:36:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:36:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:36:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:36:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:36:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:36:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:36:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:36:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:36:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:36:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:36:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:36:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:36:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:36:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:36:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:36:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:36:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:36:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:36:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:36:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:36:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:36:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:36:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:36:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:36:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:36:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:36:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:36:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:36:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:36:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:36:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:36:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:36:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:36:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:36:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:36:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:36:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:36:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:36:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:36:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:36:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=847] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:36:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=847] Local address: 172.21.15.161, port 59006 2022-07-03T18:36:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=847] Peer address: 172.21.15.101, port 22 2022-07-03T18:36:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=847] Connection lost 2022-07-03T18:36:06.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=847] Aborting connection 2022-07-03T18:36:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:07 smithi161 ceph-mon[91579]: pgmap v14308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:07 smithi188 ceph-mon[92425]: pgmap v14308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:09 smithi161 ceph-mon[91579]: pgmap v14309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:09 smithi188 ceph-mon[92425]: pgmap v14309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:11 smithi161 ceph-mon[91579]: pgmap v14310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:11 smithi188 ceph-mon[92425]: pgmap v14310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:13 smithi161 ceph-mon[91579]: pgmap v14311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:13 smithi188 ceph-mon[92425]: pgmap v14311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:15 smithi161 ceph-mon[91579]: pgmap v14312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:15 smithi188 ceph-mon[92425]: pgmap v14312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:17 smithi161 ceph-mon[91579]: pgmap v14313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:17 smithi188 ceph-mon[92425]: pgmap v14313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:19 smithi161 ceph-mon[91579]: pgmap v14314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:19 smithi188 ceph-mon[92425]: pgmap v14314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:19.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:21 smithi161 ceph-mon[91579]: pgmap v14315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:21 smithi188 ceph-mon[92425]: pgmap v14315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:23 smithi161 ceph-mon[91579]: pgmap v14316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:23 smithi188 ceph-mon[92425]: pgmap v14316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:25 smithi161 ceph-mon[91579]: pgmap v14317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:25 smithi188 ceph-mon[92425]: pgmap v14317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:27 smithi161 ceph-mon[91579]: pgmap v14318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:27 smithi188 ceph-mon[92425]: pgmap v14318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:29 smithi161 ceph-mon[91579]: pgmap v14319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:29 smithi188 ceph-mon[92425]: pgmap v14319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:31 smithi161 ceph-mon[91579]: pgmap v14320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:31 smithi188 ceph-mon[92425]: pgmap v14320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:33 smithi161 ceph-mon[91579]: pgmap v14321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:33 smithi188 ceph-mon[92425]: pgmap v14321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:34.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:35 smithi161 ceph-mon[91579]: pgmap v14322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:35 smithi188 ceph-mon[92425]: pgmap v14322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:37 smithi161 ceph-mon[91579]: pgmap v14323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:37 smithi188 ceph-mon[92425]: pgmap v14323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:39 smithi161 ceph-mon[91579]: pgmap v14324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:39.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:39 smithi188 ceph-mon[92425]: pgmap v14324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:39.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:41 smithi161 ceph-mon[91579]: pgmap v14325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:41 smithi188 ceph-mon[92425]: pgmap v14325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:43 smithi161 ceph-mon[91579]: pgmap v14326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:43 smithi188 ceph-mon[92425]: pgmap v14326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:45 smithi161 ceph-mon[91579]: pgmap v14327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:45 smithi188 ceph-mon[92425]: pgmap v14327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:46 smithi161 ceph-mon[91579]: pgmap v14328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:46 smithi188 ceph-mon[92425]: pgmap v14328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:49 smithi188 ceph-mon[92425]: pgmap v14329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:49 smithi161 ceph-mon[91579]: pgmap v14329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:50.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:51 smithi188 ceph-mon[92425]: pgmap v14330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:51 smithi161 ceph-mon[91579]: pgmap v14330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:53 smithi188 ceph-mon[92425]: pgmap v14331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:53 smithi161 ceph-mon[91579]: pgmap v14331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:55.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:55 smithi188 ceph-mon[92425]: pgmap v14332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:55.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:55 smithi161 ceph-mon[91579]: pgmap v14332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:36:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:36:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:57 smithi188 ceph-mon[92425]: pgmap v14333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:57 smithi161 ceph-mon[91579]: pgmap v14333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:59 smithi188 ceph-mon[92425]: pgmap v14334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:36:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:36:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:36:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:36:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:59 smithi161 ceph-mon[91579]: pgmap v14334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:36:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:36:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:36:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:36:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:37:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:01.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:01 smithi188 ceph-mon[92425]: pgmap v14335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:01 smithi161 ceph-mon[91579]: pgmap v14335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:03 smithi161 ceph-mon[91579]: pgmap v14336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:03 smithi188 ceph-mon[92425]: pgmap v14336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:05.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-mon[91579]: pgmap v14337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:05.469 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:05.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:05 smithi188 ceph-mon[92425]: pgmap v14337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:05.478 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:37:05.466+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:37:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:37:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:37:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:37:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:37:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:37:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:37:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:37:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:37:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:37:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:37:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:37:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:37:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:37:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:37:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:37:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:37:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:37:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:37:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:37:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:37:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:37:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:37:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:37:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:37:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:37:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:37:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:37:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:37:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:37:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:37:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:37:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:37:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:37:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:37:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:37:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:37:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:37:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:37:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:37:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:37:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:37:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:37:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:37:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:37:05.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=849] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:37:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=849] Local address: 172.21.15.161, port 59010 2022-07-03T18:37:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=849] Peer address: 172.21.15.101, port 22 2022-07-03T18:37:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=849] Connection lost 2022-07-03T18:37:05.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=849] Aborting connection 2022-07-03T18:37:05.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:37:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:37:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:37:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:37:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:37:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:37:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:37:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:37:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:37:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:37:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:07 smithi161 ceph-mon[91579]: pgmap v14338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:37:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:37:07.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:37:07.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:37:07.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:37:07.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:37:07.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:37:07.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:37:07.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:37:07.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:37:07.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:37:07.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:37:07.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:37:07.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:37:07.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:37:07.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:37:07.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:37:07.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:37:07.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:37:07.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:37:07.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:37:07.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:37:07.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:37:07.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:37:07.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:37:07.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:37:07.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:37:07.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:37:07.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:37:07.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:37:07.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:37:07.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:37:07.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:37:07.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:37:07.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:37:07.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:37:07.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:37:07.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:37:07.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:37:07.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:37:07.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:37:07.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:37:07.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:37:07.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:37:07.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:37:07.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=849] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:37:07.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=849] Local address: 172.21.15.161, port 59010 2022-07-03T18:37:07.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=849] Peer address: 172.21.15.101, port 22 2022-07-03T18:37:07.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=849] Connection lost 2022-07-03T18:37:07.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=849] Aborting connection 2022-07-03T18:37:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:37:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:07 smithi188 ceph-mon[92425]: pgmap v14338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:37:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:37:07.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:37:07.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:37:07.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:37:07.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:37:07.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:37:07.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:37:07.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:37:07.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:37:07.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:37:07.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:37:07.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:37:07.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:37:07.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:37:07.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:37:07.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:37:07.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:37:07.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:37:07.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:37:07.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:37:07.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:37:07.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:37:07.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:37:07.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:37:07.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:37:07.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:37:07.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:37:07.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:37:07.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:37:07.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:37:07.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:37:07.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:37:07.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:37:07.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:37:07.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:37:07.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:37:07.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:37:07.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:37:07.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:37:07.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:37:07.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:37:07.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:37:07.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:37:07.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:37:07.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=849] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:37:07.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=849] Local address: 172.21.15.161, port 59010 2022-07-03T18:37:07.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=849] Peer address: 172.21.15.101, port 22 2022-07-03T18:37:07.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=849] Connection lost 2022-07-03T18:37:07.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=849] Aborting connection 2022-07-03T18:37:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:09 smithi161 ceph-mon[91579]: pgmap v14339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:09 smithi188 ceph-mon[92425]: pgmap v14339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:11 smithi161 ceph-mon[91579]: pgmap v14340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:11 smithi188 ceph-mon[92425]: pgmap v14340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:13 smithi161 ceph-mon[91579]: pgmap v14341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:13 smithi188 ceph-mon[92425]: pgmap v14341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:15 smithi161 ceph-mon[91579]: pgmap v14342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:15 smithi188 ceph-mon[92425]: pgmap v14342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:17 smithi161 ceph-mon[91579]: pgmap v14343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:17 smithi188 ceph-mon[92425]: pgmap v14343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:19 smithi161 ceph-mon[91579]: pgmap v14344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:19 smithi188 ceph-mon[92425]: pgmap v14344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:21 smithi161 ceph-mon[91579]: pgmap v14345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:21 smithi188 ceph-mon[92425]: pgmap v14345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:23 smithi161 ceph-mon[91579]: pgmap v14346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:23 smithi188 ceph-mon[92425]: pgmap v14346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:25 smithi161 ceph-mon[91579]: pgmap v14347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:25 smithi188 ceph-mon[92425]: pgmap v14347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:27 smithi161 ceph-mon[91579]: pgmap v14348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:27 smithi188 ceph-mon[92425]: pgmap v14348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:29 smithi161 ceph-mon[91579]: pgmap v14349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:29 smithi188 ceph-mon[92425]: pgmap v14349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:31 smithi161 ceph-mon[91579]: pgmap v14350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:31 smithi188 ceph-mon[92425]: pgmap v14350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:33 smithi161 ceph-mon[91579]: pgmap v14351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:33 smithi188 ceph-mon[92425]: pgmap v14351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:35 smithi161 ceph-mon[91579]: pgmap v14352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:35 smithi188 ceph-mon[92425]: pgmap v14352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:37 smithi161 ceph-mon[91579]: pgmap v14353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:37 smithi188 ceph-mon[92425]: pgmap v14353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:39 smithi161 ceph-mon[91579]: pgmap v14354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:39 smithi188 ceph-mon[92425]: pgmap v14354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:41 smithi161 ceph-mon[91579]: pgmap v14355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:41 smithi188 ceph-mon[92425]: pgmap v14355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:43 smithi161 ceph-mon[91579]: pgmap v14356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:43 smithi188 ceph-mon[92425]: pgmap v14356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:45 smithi161 ceph-mon[91579]: pgmap v14357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:45 smithi188 ceph-mon[92425]: pgmap v14357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:47 smithi161 ceph-mon[91579]: pgmap v14358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:47 smithi188 ceph-mon[92425]: pgmap v14358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:49 smithi161 ceph-mon[91579]: pgmap v14359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:49 smithi188 ceph-mon[92425]: pgmap v14359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:51 smithi161 ceph-mon[91579]: pgmap v14360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:51 smithi188 ceph-mon[92425]: pgmap v14360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:53 smithi161 ceph-mon[91579]: pgmap v14361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:53 smithi188 ceph-mon[92425]: pgmap v14361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:55 smithi161 ceph-mon[91579]: pgmap v14362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:55 smithi188 ceph-mon[92425]: pgmap v14362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:37:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:37:57.641 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:57 smithi161 ceph-mon[91579]: pgmap v14363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:57 smithi188 ceph-mon[92425]: pgmap v14363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:59 smithi161 ceph-mon[91579]: pgmap v14364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:37:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:37:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:37:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:59 smithi188 ceph-mon[92425]: pgmap v14364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:37:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:37:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:37:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:37:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:38:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:01 smithi161 ceph-mon[91579]: pgmap v14365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:01 smithi188 ceph-mon[92425]: pgmap v14365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:03 smithi161 ceph-mon[91579]: pgmap v14366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:03 smithi188 ceph-mon[92425]: pgmap v14366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:05 smithi161 ceph-mon[91579]: pgmap v14367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:05 smithi188 ceph-mon[92425]: pgmap v14367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:38:06.391+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:38:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:38:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:38:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:38:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:38:06.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:38:06.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:38:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:38:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:38:06.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:38:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:38:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:38:06.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:38:06.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:38:06.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:38:06.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:38:06.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:38:06.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:38:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:38:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:38:06.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:38:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:38:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:38:06.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:38:06.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:38:06.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:38:06.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:38:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:38:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:38:06.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:38:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:38:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:38:06.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:38:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:38:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:38:06.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:38:06.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:38:06.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:38:06.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:38:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:38:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:38:06.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:38:06.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=851] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:38:06.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=851] Local address: 172.21.15.161, port 59014 2022-07-03T18:38:06.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=851] Peer address: 172.21.15.101, port 22 2022-07-03T18:38:06.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=851] Connection lost 2022-07-03T18:38:06.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=851] Aborting connection 2022-07-03T18:38:06.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:38:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:07 smithi161 ceph-mon[91579]: pgmap v14368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:38:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:38:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:38:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:07 smithi188 ceph-mon[92425]: pgmap v14368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:38:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:38:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:38:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:38:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:38:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:38:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:38:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:38:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:38:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:38:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:38:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:38:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:38:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=851] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:38:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=851] Local address: 172.21.15.161, port 59014 2022-07-03T18:38:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=851] Peer address: 172.21.15.101, port 22 2022-07-03T18:38:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=851] Connection lost 2022-07-03T18:38:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=851] Aborting connection 2022-07-03T18:38:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:38:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:38:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:38:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:38:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:38:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:38:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:38:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:38:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:38:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:38:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:38:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:38:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:38:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:38:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:38:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:38:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:38:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:38:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:38:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:38:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:38:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:38:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:38:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:38:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:38:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=851] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:38:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=851] Local address: 172.21.15.161, port 59014 2022-07-03T18:38:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=851] Peer address: 172.21.15.101, port 22 2022-07-03T18:38:08.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=851] Connection lost 2022-07-03T18:38:08.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=851] Aborting connection 2022-07-03T18:38:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:09 smithi161 ceph-mon[91579]: pgmap v14369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:09 smithi188 ceph-mon[92425]: pgmap v14369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:11 smithi161 ceph-mon[91579]: pgmap v14370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:11 smithi188 ceph-mon[92425]: pgmap v14370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:13 smithi161 ceph-mon[91579]: pgmap v14371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:13 smithi188 ceph-mon[92425]: pgmap v14371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:15 smithi161 ceph-mon[91579]: pgmap v14372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:15 smithi188 ceph-mon[92425]: pgmap v14372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:17 smithi161 ceph-mon[91579]: pgmap v14373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:17 smithi188 ceph-mon[92425]: pgmap v14373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:19 smithi161 ceph-mon[91579]: pgmap v14374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:19 smithi188 ceph-mon[92425]: pgmap v14374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:21 smithi161 ceph-mon[91579]: pgmap v14375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:21 smithi188 ceph-mon[92425]: pgmap v14375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:23 smithi161 ceph-mon[91579]: pgmap v14376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:23 smithi188 ceph-mon[92425]: pgmap v14376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:24.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:25 smithi161 ceph-mon[91579]: pgmap v14377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:25 smithi188 ceph-mon[92425]: pgmap v14377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:26 smithi161 ceph-mon[91579]: pgmap v14378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:26 smithi188 ceph-mon[92425]: pgmap v14378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:29.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:29 smithi188 ceph-mon[92425]: pgmap v14379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:29 smithi161 ceph-mon[91579]: pgmap v14379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:30.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:31 smithi188 ceph-mon[92425]: pgmap v14380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:31 smithi161 ceph-mon[91579]: pgmap v14380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:33 smithi161 ceph-mon[91579]: pgmap v14381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:33 smithi188 ceph-mon[92425]: pgmap v14381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:35 smithi161 ceph-mon[91579]: pgmap v14382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:35 smithi188 ceph-mon[92425]: pgmap v14382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:37 smithi161 ceph-mon[91579]: pgmap v14383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:37 smithi188 ceph-mon[92425]: pgmap v14383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:39 smithi161 ceph-mon[91579]: pgmap v14384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:39 smithi188 ceph-mon[92425]: pgmap v14384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:41 smithi161 ceph-mon[91579]: pgmap v14385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:41 smithi188 ceph-mon[92425]: pgmap v14385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:43 smithi161 ceph-mon[91579]: pgmap v14386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:43 smithi188 ceph-mon[92425]: pgmap v14386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:45 smithi161 ceph-mon[91579]: pgmap v14387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:45 smithi188 ceph-mon[92425]: pgmap v14387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:47 smithi161 ceph-mon[91579]: pgmap v14388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:47 smithi188 ceph-mon[92425]: pgmap v14388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:49 smithi161 ceph-mon[91579]: pgmap v14389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:49 smithi188 ceph-mon[92425]: pgmap v14389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:51 smithi161 ceph-mon[91579]: pgmap v14390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:51 smithi188 ceph-mon[92425]: pgmap v14390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:53 smithi161 ceph-mon[91579]: pgmap v14391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:38:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:53 smithi188 ceph-mon[92425]: pgmap v14391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:38:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:55 smithi161 ceph-mon[91579]: pgmap v14392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:55.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:55 smithi188 ceph-mon[92425]: pgmap v14392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:38:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:38:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:57 smithi161 ceph-mon[91579]: pgmap v14393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:57 smithi188 ceph-mon[92425]: pgmap v14393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:38:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:38:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:59 smithi161 ceph-mon[91579]: pgmap v14394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:38:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:38:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:38:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:38:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:59 smithi188 ceph-mon[92425]: pgmap v14394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:38:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:38:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:38:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:38:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:39:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:01 smithi161 ceph-mon[91579]: pgmap v14395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:01 smithi188 ceph-mon[92425]: pgmap v14395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:03 smithi161 ceph-mon[91579]: pgmap v14396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:03 smithi188 ceph-mon[92425]: pgmap v14396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:03.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:05 smithi161 ceph-mon[91579]: pgmap v14397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:05 smithi188 ceph-mon[92425]: pgmap v14397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:07.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:39:06.896+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:39:07.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:39:07.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:39:07.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:39:07.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:39:07.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:39:07.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:39:07.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:39:07.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:39:07.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:39:07.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:39:07.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:39:07.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:39:07.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:39:07.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:39:07.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:39:07.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:39:07.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:39:07.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:39:07.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:39:07.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:39:07.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:39:07.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:39:07.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:39:07.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:39:07.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:39:07.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:39:07.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:39:07.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:39:07.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:39:07.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:39:07.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:39:07.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:39:07.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:39:07.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:39:07.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:39:07.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:39:07.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:39:07.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:39:07.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:39:07.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:39:07.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:39:07.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:39:07.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:39:07.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:39:07.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=853] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:39:07.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=853] Local address: 172.21.15.161, port 59056 2022-07-03T18:39:07.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=853] Peer address: 172.21.15.101, port 22 2022-07-03T18:39:07.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=853] Connection lost 2022-07-03T18:39:07.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=853] Aborting connection 2022-07-03T18:39:07.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:39:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:39:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:07 smithi161 ceph-mon[91579]: pgmap v14398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:39:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:39:07.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:39:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:07 smithi188 ceph-mon[92425]: pgmap v14398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:39:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:39:07.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:39:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:39:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:39:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:39:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:39:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:39:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:39:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:39:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:39:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:39:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:39:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:39:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:39:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:39:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:39:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:39:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:39:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:39:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:39:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:39:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:39:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:39:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:39:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:39:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:39:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:39:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:39:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:39:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:39:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:39:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:39:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:39:08.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:39:08.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:39:08.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:39:08.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:39:08.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:39:08.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:39:08.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:39:08.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:39:08.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:39:08.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:39:08.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:39:08.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:39:08.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:39:08.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:39:08.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:39:08.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=853] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:39:08.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=853] Local address: 172.21.15.161, port 59056 2022-07-03T18:39:08.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=853] Peer address: 172.21.15.101, port 22 2022-07-03T18:39:08.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=853] Connection lost 2022-07-03T18:39:08.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=853] Aborting connection 2022-07-03T18:39:08.660 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:39:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:39:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:39:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:39:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:39:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:39:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:39:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:39:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:39:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:39:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:39:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:39:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:39:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:39:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:39:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:39:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:39:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:39:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:39:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:39:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:39:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:39:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:39:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:39:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:39:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:39:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:39:08.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:39:08.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:39:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:39:08.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:39:08.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:39:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:39:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:39:08.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:39:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:39:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:39:08.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:39:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:39:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:39:08.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:39:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:39:08.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:39:08.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:39:08.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:39:08.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:39:08.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=853] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:39:08.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=853] Local address: 172.21.15.161, port 59056 2022-07-03T18:39:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=853] Peer address: 172.21.15.101, port 22 2022-07-03T18:39:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=853] Connection lost 2022-07-03T18:39:08.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=853] Aborting connection 2022-07-03T18:39:08.692 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:09 smithi161 ceph-mon[91579]: pgmap v14399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:09 smithi188 ceph-mon[92425]: pgmap v14399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:11 smithi161 ceph-mon[91579]: pgmap v14400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:11 smithi188 ceph-mon[92425]: pgmap v14400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:13 smithi161 ceph-mon[91579]: pgmap v14401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:13 smithi188 ceph-mon[92425]: pgmap v14401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:15 smithi161 ceph-mon[91579]: pgmap v14402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:15 smithi188 ceph-mon[92425]: pgmap v14402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:17 smithi161 ceph-mon[91579]: pgmap v14403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:17 smithi188 ceph-mon[92425]: pgmap v14403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:19 smithi161 ceph-mon[91579]: pgmap v14404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:19 smithi188 ceph-mon[92425]: pgmap v14404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:21 smithi161 ceph-mon[91579]: pgmap v14405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:21 smithi188 ceph-mon[92425]: pgmap v14405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:23 smithi161 ceph-mon[91579]: pgmap v14406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:23 smithi188 ceph-mon[92425]: pgmap v14406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:25 smithi161 ceph-mon[91579]: pgmap v14407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:25 smithi188 ceph-mon[92425]: pgmap v14407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:27 smithi161 ceph-mon[91579]: pgmap v14408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:27 smithi188 ceph-mon[92425]: pgmap v14408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:29 smithi161 ceph-mon[91579]: pgmap v14409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:29 smithi188 ceph-mon[92425]: pgmap v14409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:31 smithi161 ceph-mon[91579]: pgmap v14410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:31 smithi188 ceph-mon[92425]: pgmap v14410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:33 smithi161 ceph-mon[91579]: pgmap v14411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:33 smithi188 ceph-mon[92425]: pgmap v14411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:35 smithi161 ceph-mon[91579]: pgmap v14412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:35 smithi188 ceph-mon[92425]: pgmap v14412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:37 smithi161 ceph-mon[91579]: pgmap v14413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:37 smithi188 ceph-mon[92425]: pgmap v14413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:39 smithi161 ceph-mon[91579]: pgmap v14414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:39 smithi188 ceph-mon[92425]: pgmap v14414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:41 smithi161 ceph-mon[91579]: pgmap v14415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:41 smithi188 ceph-mon[92425]: pgmap v14415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:43 smithi161 ceph-mon[91579]: pgmap v14416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:43 smithi188 ceph-mon[92425]: pgmap v14416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:44 smithi161 ceph-mon[91579]: pgmap v14417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:44.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:44 smithi188 ceph-mon[92425]: pgmap v14417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:46.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:46 smithi161 ceph-mon[91579]: pgmap v14418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:46.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:46 smithi188 ceph-mon[92425]: pgmap v14418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:49 smithi161 ceph-mon[91579]: pgmap v14419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:49 smithi188 ceph-mon[92425]: pgmap v14419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:51 smithi161 ceph-mon[91579]: pgmap v14420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:51 smithi188 ceph-mon[92425]: pgmap v14420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:53 smithi161 ceph-mon[91579]: pgmap v14421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:53 smithi188 ceph-mon[92425]: pgmap v14421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:55 smithi161 ceph-mon[91579]: pgmap v14422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:55 smithi188 ceph-mon[92425]: pgmap v14422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:39:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:39:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:57 smithi161 ceph-mon[91579]: pgmap v14423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:57 smithi188 ceph-mon[92425]: pgmap v14423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:39:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:59 smithi161 ceph-mon[91579]: pgmap v14424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:39:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:39:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:39:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:39:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:59 smithi188 ceph-mon[92425]: pgmap v14424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:39:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:39:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:39:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:39:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:40:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:40:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:40:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:01 smithi161 ceph-mon[91579]: pgmap v14425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:01 smithi188 ceph-mon[92425]: pgmap v14425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:03 smithi161 ceph-mon[91579]: pgmap v14426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:03 smithi188 ceph-mon[92425]: pgmap v14426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:05 smithi161 ceph-mon[91579]: pgmap v14427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:05 smithi188 ceph-mon[92425]: pgmap v14427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:07.469 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:07 smithi188 ceph-mon[92425]: pgmap v14428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-mon[91579]: pgmap v14428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:07.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:40:07.396+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:40:07.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:40:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:40:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:40:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:40:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:40:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:40:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:40:07.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:40:07.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:40:07.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:40:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:40:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:40:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:40:07.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:40:07.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:40:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:40:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:40:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:40:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:40:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:40:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:40:07.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:40:07.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:40:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:40:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:40:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:40:07.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:40:07.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:40:07.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:40:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:40:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:40:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:40:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:40:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:40:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:40:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:40:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:40:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:40:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:40:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:40:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:40:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:40:07.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:40:07.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:40:07.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=855] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:40:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=855] Local address: 172.21.15.161, port 59084 2022-07-03T18:40:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=855] Peer address: 172.21.15.101, port 22 2022-07-03T18:40:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=855] Connection lost 2022-07-03T18:40:07.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=855] Aborting connection 2022-07-03T18:40:07.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:40:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:40:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:40:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:40:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:40:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:40:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:40:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:40:09.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:40:09.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:40:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:40:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:40:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:40:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:40:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:40:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:40:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:40:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:40:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:40:09.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:40:09.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:40:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:40:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:40:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:40:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:40:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:40:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:40:09.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:40:09.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:40:09.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:40:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:40:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:40:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:40:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:40:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:40:09.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:40:09.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:40:09.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:40:09.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:40:09.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:40:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:40:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:40:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:40:09.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:40:09.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:40:09.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:40:09.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:40:09.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:40:09.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:40:09.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:40:09.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:40:09.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:40:09.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:40:09.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=855] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:40:09.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=855] Local address: 172.21.15.161, port 59084 2022-07-03T18:40:09.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=855] Peer address: 172.21.15.101, port 22 2022-07-03T18:40:09.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=855] Connection lost 2022-07-03T18:40:09.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=855] Aborting connection 2022-07-03T18:40:09.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:09 smithi161 ceph-mon[91579]: pgmap v14429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:40:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:40:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:40:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:40:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:40:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:40:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:40:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:40:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:40:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:40:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:40:09.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:40:09.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:40:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:40:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:40:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:40:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:40:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:40:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:40:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:40:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:40:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:40:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:40:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:40:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:40:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:40:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:40:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:40:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:40:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:40:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:40:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:40:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:40:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:40:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:40:09.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:40:09.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:40:09.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:40:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:40:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:40:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:40:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:40:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:40:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:40:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:40:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:40:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=855] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:40:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=855] Local address: 172.21.15.161, port 59084 2022-07-03T18:40:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=855] Peer address: 172.21.15.101, port 22 2022-07-03T18:40:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=855] Connection lost 2022-07-03T18:40:09.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=855] Aborting connection 2022-07-03T18:40:09.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:09 smithi188 ceph-mon[92425]: pgmap v14429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:10 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:10 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:40:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: pgmap v14430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: pgmap v14430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:11.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:40:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:12 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi161 to 7486k 2022-07-03T18:40:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:12 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi161 to 7666409: error parsing value: Value '7666409' is below minimum 939524096 2022-07-03T18:40:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:12 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 137.5M 2022-07-03T18:40:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:12 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 144282853: error parsing value: Value '144282853' is below minimum 939524096 2022-07-03T18:40:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:12 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi161 to 7486k 2022-07-03T18:40:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:12 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi161 to 7666409: error parsing value: Value '7666409' is below minimum 939524096 2022-07-03T18:40:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:12 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 137.5M 2022-07-03T18:40:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:12 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 144282853: error parsing value: Value '144282853' is below minimum 939524096 2022-07-03T18:40:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:13 smithi161 ceph-mon[91579]: pgmap v14431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:13.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:13 smithi188 ceph-mon[92425]: pgmap v14431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:13.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:15 smithi161 ceph-mon[91579]: pgmap v14432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:15 smithi188 ceph-mon[92425]: pgmap v14432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:17 smithi161 ceph-mon[91579]: pgmap v14433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:17 smithi188 ceph-mon[92425]: pgmap v14433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:19 smithi161 ceph-mon[91579]: pgmap v14434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:19 smithi188 ceph-mon[92425]: pgmap v14434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:21 smithi161 ceph-mon[91579]: pgmap v14435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:21 smithi188 ceph-mon[92425]: pgmap v14435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:23 smithi161 ceph-mon[91579]: pgmap v14436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:23 smithi188 ceph-mon[92425]: pgmap v14436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:25 smithi161 ceph-mon[91579]: pgmap v14437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:25 smithi188 ceph-mon[92425]: pgmap v14437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:27 smithi161 ceph-mon[91579]: pgmap v14438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:27 smithi188 ceph-mon[92425]: pgmap v14438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:29 smithi161 ceph-mon[91579]: pgmap v14439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:29 smithi188 ceph-mon[92425]: pgmap v14439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:31 smithi161 ceph-mon[91579]: pgmap v14440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:31 smithi188 ceph-mon[92425]: pgmap v14440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:33 smithi161 ceph-mon[91579]: pgmap v14441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:33 smithi188 ceph-mon[92425]: pgmap v14441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:35 smithi161 ceph-mon[91579]: pgmap v14442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:35 smithi188 ceph-mon[92425]: pgmap v14442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:37 smithi161 ceph-mon[91579]: pgmap v14443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:37 smithi188 ceph-mon[92425]: pgmap v14443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:39 smithi161 ceph-mon[91579]: pgmap v14444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:39 smithi188 ceph-mon[92425]: pgmap v14444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:41 smithi161 ceph-mon[91579]: pgmap v14445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:41 smithi188 ceph-mon[92425]: pgmap v14445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:43 smithi161 ceph-mon[91579]: pgmap v14446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:43 smithi188 ceph-mon[92425]: pgmap v14446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:45 smithi161 ceph-mon[91579]: pgmap v14447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:45 smithi188 ceph-mon[92425]: pgmap v14447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:47 smithi161 ceph-mon[91579]: pgmap v14448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:47 smithi188 ceph-mon[92425]: pgmap v14448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:49 smithi161 ceph-mon[91579]: pgmap v14449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:49 smithi188 ceph-mon[92425]: pgmap v14449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:51 smithi161 ceph-mon[91579]: pgmap v14450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:51 smithi188 ceph-mon[92425]: pgmap v14450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:52 smithi188 ceph-mon[92425]: pgmap v14451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:52 smithi161 ceph-mon[91579]: pgmap v14451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:54 smithi188 ceph-mon[92425]: pgmap v14452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:54 smithi161 ceph-mon[91579]: pgmap v14452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:40:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:40:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:56 smithi161 ceph-mon[91579]: pgmap v14453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:56 smithi188 ceph-mon[92425]: pgmap v14453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:40:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:59 smithi161 ceph-mon[91579]: pgmap v14454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:40:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:40:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:40:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:40:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:59 smithi188 ceph-mon[92425]: pgmap v14454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:40:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:40:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:40:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:40:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:41:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:01 smithi161 ceph-mon[91579]: pgmap v14455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:01 smithi188 ceph-mon[92425]: pgmap v14455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:03 smithi161 ceph-mon[91579]: pgmap v14456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:03 smithi188 ceph-mon[92425]: pgmap v14456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:03.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:05 smithi161 ceph-mon[91579]: pgmap v14457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:05 smithi188 ceph-mon[92425]: pgmap v14457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:07 smithi161 ceph-mon[91579]: pgmap v14458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:07 smithi188 ceph-mon[92425]: pgmap v14458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:09 smithi161 ceph-mon[91579]: pgmap v14459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:09 smithi188 ceph-mon[92425]: pgmap v14459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:10.572 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:10.583 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:10.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:41:10.569+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:41:10.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:41:10.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:41:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:41:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:41:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:41:10.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:41:10.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:41:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:41:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:41:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:41:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:41:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:41:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:41:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:41:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:41:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:41:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:41:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:41:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:41:10.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:41:10.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:41:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:41:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:41:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:41:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:41:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:41:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:41:10.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:41:10.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:41:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:41:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:41:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:41:10.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:41:10.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:41:10.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:41:10.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:41:10.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:41:10.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:41:10.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:41:10.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:41:10.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:41:10.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:41:10.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:41:10.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:41:10.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=857] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:41:10.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=857] Local address: 172.21.15.161, port 59114 2022-07-03T18:41:10.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=857] Peer address: 172.21.15.101, port 22 2022-07-03T18:41:10.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=857] Connection lost 2022-07-03T18:41:10.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=857] Aborting connection 2022-07-03T18:41:10.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:41:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:41:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:11 smithi161 ceph-mon[91579]: pgmap v14460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:41:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:41:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:41:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:11 smithi188 ceph-mon[92425]: pgmap v14460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:41:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:41:11.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:41:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:41:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:41:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:41:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:41:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:41:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:41:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:41:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:41:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:41:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:41:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:41:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:41:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:41:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:41:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:41:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:41:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:41:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:41:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:41:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:41:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:41:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:41:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:41:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:41:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:41:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:41:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:41:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:41:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:41:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:41:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:41:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:41:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:41:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:41:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:41:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:41:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:41:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:41:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:41:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:41:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:41:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:41:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:41:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:41:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:41:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:41:12.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=857] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:41:12.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=857] Local address: 172.21.15.161, port 59114 2022-07-03T18:41:12.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=857] Peer address: 172.21.15.101, port 22 2022-07-03T18:41:12.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=857] Connection lost 2022-07-03T18:41:12.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=857] Aborting connection 2022-07-03T18:41:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:41:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:41:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:41:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:41:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:41:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:41:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:41:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:41:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:41:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:41:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:41:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:41:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:41:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:41:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:41:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:41:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:41:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:41:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:41:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:41:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:41:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:41:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:41:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:41:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:41:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:41:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:41:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:41:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:41:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:41:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:41:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:41:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:41:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:41:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:41:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:41:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:41:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:41:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:41:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:41:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:41:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:41:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:41:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:41:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:41:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:41:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=857] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:41:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=857] Local address: 172.21.15.161, port 59114 2022-07-03T18:41:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=857] Peer address: 172.21.15.101, port 22 2022-07-03T18:41:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=857] Connection lost 2022-07-03T18:41:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=857] Aborting connection 2022-07-03T18:41:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:13 smithi161 ceph-mon[91579]: pgmap v14461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:13 smithi188 ceph-mon[92425]: pgmap v14461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:41:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:15 smithi161 ceph-mon[91579]: pgmap v14462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:15 smithi188 ceph-mon[92425]: pgmap v14462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:17 smithi161 ceph-mon[91579]: pgmap v14463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:17 smithi188 ceph-mon[92425]: pgmap v14463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:19 smithi161 ceph-mon[91579]: pgmap v14464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:19 smithi188 ceph-mon[92425]: pgmap v14464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:21 smithi161 ceph-mon[91579]: pgmap v14465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:21 smithi188 ceph-mon[92425]: pgmap v14465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:23 smithi161 ceph-mon[91579]: pgmap v14466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:23 smithi188 ceph-mon[92425]: pgmap v14466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:25 smithi161 ceph-mon[91579]: pgmap v14467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:25 smithi188 ceph-mon[92425]: pgmap v14467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:27 smithi161 ceph-mon[91579]: pgmap v14468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:27 smithi188 ceph-mon[92425]: pgmap v14468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:29 smithi161 ceph-mon[91579]: pgmap v14469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:29 smithi188 ceph-mon[92425]: pgmap v14469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:31 smithi161 ceph-mon[91579]: pgmap v14470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:31 smithi188 ceph-mon[92425]: pgmap v14470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:33 smithi161 ceph-mon[91579]: pgmap v14471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:33 smithi188 ceph-mon[92425]: pgmap v14471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:35 smithi161 ceph-mon[91579]: pgmap v14472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:35 smithi188 ceph-mon[92425]: pgmap v14472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:37 smithi161 ceph-mon[91579]: pgmap v14473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:37 smithi188 ceph-mon[92425]: pgmap v14473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:39 smithi161 ceph-mon[91579]: pgmap v14474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:39 smithi188 ceph-mon[92425]: pgmap v14474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:41 smithi161 ceph-mon[91579]: pgmap v14475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:41 smithi188 ceph-mon[92425]: pgmap v14475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:43 smithi161 ceph-mon[91579]: pgmap v14476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:43 smithi188 ceph-mon[92425]: pgmap v14476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:45 smithi161 ceph-mon[91579]: pgmap v14477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:45 smithi188 ceph-mon[92425]: pgmap v14477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:47 smithi161 ceph-mon[91579]: pgmap v14478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:47 smithi188 ceph-mon[92425]: pgmap v14478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:49 smithi161 ceph-mon[91579]: pgmap v14479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:49 smithi188 ceph-mon[92425]: pgmap v14479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:51 smithi161 ceph-mon[91579]: pgmap v14480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:51 smithi188 ceph-mon[92425]: pgmap v14480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:53 smithi161 ceph-mon[91579]: pgmap v14481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:53 smithi188 ceph-mon[92425]: pgmap v14481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:55 smithi161 ceph-mon[91579]: pgmap v14482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:55 smithi188 ceph-mon[92425]: pgmap v14482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:41:57.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:57 smithi161 ceph-mon[91579]: pgmap v14483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:57.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:57 smithi188 ceph-mon[92425]: pgmap v14483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:41:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:41:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:59 smithi161 ceph-mon[91579]: pgmap v14484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:41:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:41:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:41:59.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:41:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:59 smithi188 ceph-mon[92425]: pgmap v14484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:41:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:41:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:41:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:41:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:42:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:01 smithi161 ceph-mon[91579]: pgmap v14485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:01 smithi188 ceph-mon[92425]: pgmap v14485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:02 smithi161 ceph-mon[91579]: pgmap v14486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:02.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:02 smithi188 ceph-mon[92425]: pgmap v14486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:04.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:04 smithi161 ceph-mon[91579]: pgmap v14487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:04.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:04 smithi188 ceph-mon[92425]: pgmap v14487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:06 smithi161 ceph-mon[91579]: pgmap v14488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:06 smithi188 ceph-mon[92425]: pgmap v14488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:07.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:07.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:09 smithi161 ceph-mon[91579]: pgmap v14489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:09 smithi188 ceph-mon[92425]: pgmap v14489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-mon[91579]: pgmap v14490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:42:11.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:42:11.057+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:42:11.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:42:11.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:42:11.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:42:11.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:42:11.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:42:11.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:42:11.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:42:11.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:42:11.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:42:11.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:42:11.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:42:11.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:42:11.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:42:11.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:42:11.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:42:11.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:42:11.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:42:11.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:42:11.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:42:11.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:42:11.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:42:11.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:42:11.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:42:11.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:42:11.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:42:11.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:42:11.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:42:11.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:42:11.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:42:11.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:42:11.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:42:11.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:42:11.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:42:11.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:42:11.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:42:11.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:42:11.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:42:11.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:42:11.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:42:11.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:42:11.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:42:11.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:42:11.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:42:11.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:42:11.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=859] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:42:11.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=859] Local address: 172.21.15.161, port 59142 2022-07-03T18:42:11.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=859] Peer address: 172.21.15.101, port 22 2022-07-03T18:42:11.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=859] Connection lost 2022-07-03T18:42:11.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=859] Aborting connection 2022-07-03T18:42:11.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:11 smithi188 ceph-mon[92425]: pgmap v14490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:42:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:42:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:42:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:42:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:42:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:42:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:42:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:42:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:42:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:42:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:42:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:42:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:42:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:42:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:42:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:42:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:42:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:42:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:42:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:42:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:42:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:42:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:42:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:42:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:42:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:42:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:42:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:42:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:42:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:42:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:42:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:42:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:42:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:42:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:42:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:42:12.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:42:12.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:42:12.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:42:12.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:42:12.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:42:12.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:42:12.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:42:12.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:42:12.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:42:12.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:42:12.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:42:12.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=859] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:42:12.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=859] Local address: 172.21.15.161, port 59142 2022-07-03T18:42:12.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=859] Peer address: 172.21.15.101, port 22 2022-07-03T18:42:12.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=859] Connection lost 2022-07-03T18:42:12.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=859] Aborting connection 2022-07-03T18:42:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:42:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:42:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:42:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:42:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:42:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:42:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:42:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:42:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:42:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:42:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:42:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:42:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:42:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:42:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:42:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:42:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:42:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:42:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:42:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:42:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:42:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:42:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:42:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:42:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:42:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:42:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:42:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:42:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:42:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:42:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:42:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:42:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:42:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:42:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:42:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:42:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:42:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:42:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:42:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:42:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:42:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:42:12.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:42:12.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:42:12.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:42:12.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:42:12.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=859] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:42:12.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=859] Local address: 172.21.15.161, port 59142 2022-07-03T18:42:12.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=859] Peer address: 172.21.15.101, port 22 2022-07-03T18:42:12.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=859] Connection lost 2022-07-03T18:42:12.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=859] Aborting connection 2022-07-03T18:42:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:13 smithi161 ceph-mon[91579]: pgmap v14491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:13 smithi188 ceph-mon[92425]: pgmap v14491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:15 smithi161 ceph-mon[91579]: pgmap v14492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:15 smithi188 ceph-mon[92425]: pgmap v14492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:17 smithi161 ceph-mon[91579]: pgmap v14493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:17 smithi188 ceph-mon[92425]: pgmap v14493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:19 smithi161 ceph-mon[91579]: pgmap v14494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:19 smithi188 ceph-mon[92425]: pgmap v14494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:20.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:21 smithi161 ceph-mon[91579]: pgmap v14495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:21 smithi188 ceph-mon[92425]: pgmap v14495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:23 smithi161 ceph-mon[91579]: pgmap v14496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:23 smithi188 ceph-mon[92425]: pgmap v14496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:25 smithi161 ceph-mon[91579]: pgmap v14497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:25 smithi188 ceph-mon[92425]: pgmap v14497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:27 smithi161 ceph-mon[91579]: pgmap v14498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:27 smithi188 ceph-mon[92425]: pgmap v14498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:29 smithi161 ceph-mon[91579]: pgmap v14499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:29 smithi188 ceph-mon[92425]: pgmap v14499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:31 smithi161 ceph-mon[91579]: pgmap v14500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:31 smithi188 ceph-mon[92425]: pgmap v14500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:33 smithi161 ceph-mon[91579]: pgmap v14501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:33 smithi188 ceph-mon[92425]: pgmap v14501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:35 smithi161 ceph-mon[91579]: pgmap v14502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:35 smithi188 ceph-mon[92425]: pgmap v14502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:37 smithi161 ceph-mon[91579]: pgmap v14503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:37 smithi188 ceph-mon[92425]: pgmap v14503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:39 smithi161 ceph-mon[91579]: pgmap v14504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:39 smithi188 ceph-mon[92425]: pgmap v14504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:41 smithi161 ceph-mon[91579]: pgmap v14505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:41 smithi188 ceph-mon[92425]: pgmap v14505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:43.548 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:43 smithi161 ceph-mon[91579]: pgmap v14506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:43.549 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:43 smithi188 ceph-mon[92425]: pgmap v14506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:45.591 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:45 smithi161 ceph-mon[91579]: pgmap v14507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:45.592 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:45 smithi188 ceph-mon[92425]: pgmap v14507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:47.564 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:47 smithi161 ceph-mon[91579]: pgmap v14508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:47 smithi188 ceph-mon[92425]: pgmap v14508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:49 smithi161 ceph-mon[91579]: pgmap v14509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:49 smithi188 ceph-mon[92425]: pgmap v14509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:51 smithi161 ceph-mon[91579]: pgmap v14510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:51 smithi188 ceph-mon[92425]: pgmap v14510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:53 smithi161 ceph-mon[91579]: pgmap v14511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:53 smithi188 ceph-mon[92425]: pgmap v14511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:55 smithi161 ceph-mon[91579]: pgmap v14512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:55 smithi188 ceph-mon[92425]: pgmap v14512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:42:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:57 smithi161 ceph-mon[91579]: pgmap v14513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:57 smithi188 ceph-mon[92425]: pgmap v14513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:42:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:42:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:59 smithi161 ceph-mon[91579]: pgmap v14514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:42:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:42:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:42:59.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:42:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:59 smithi188 ceph-mon[92425]: pgmap v14514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:42:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:42:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:42:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:42:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:43:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:01 smithi161 ceph-mon[91579]: pgmap v14515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:01 smithi188 ceph-mon[92425]: pgmap v14515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:03 smithi161 ceph-mon[91579]: pgmap v14516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:03 smithi188 ceph-mon[92425]: pgmap v14516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:05 smithi161 ceph-mon[91579]: pgmap v14517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:05 smithi188 ceph-mon[92425]: pgmap v14517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:07 smithi161 ceph-mon[91579]: pgmap v14518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:07 smithi188 ceph-mon[92425]: pgmap v14518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:09 smithi161 ceph-mon[91579]: pgmap v14519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:09 smithi188 ceph-mon[92425]: pgmap v14519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:11.619 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:11 smithi188 ceph-mon[92425]: pgmap v14520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:11.622 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-mon[91579]: pgmap v14520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:11.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:43:11.558+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:43:11.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:43:11.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:43:11.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:43:11.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:43:11.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:43:11.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:43:11.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:43:11.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:43:11.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:43:11.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:43:11.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:43:11.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:43:11.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:43:11.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:43:11.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:43:11.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:43:11.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:43:11.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:43:11.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:43:11.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:43:11.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:43:11.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:43:11.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:43:11.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:43:11.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:43:11.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:43:11.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:43:11.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:43:11.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:43:11.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:43:11.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:43:11.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:43:11.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:43:11.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:43:11.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:43:11.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:43:11.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:43:11.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:43:11.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:43:11.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:43:11.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:43:11.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:43:11.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:43:11.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:43:11.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=861] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:43:11.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=861] Local address: 172.21.15.161, port 59170 2022-07-03T18:43:11.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=861] Peer address: 172.21.15.101, port 22 2022-07-03T18:43:11.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=861] Connection lost 2022-07-03T18:43:11.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=861] Aborting connection 2022-07-03T18:43:11.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:43:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:43:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:43:12.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:43:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:43:12.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:43:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:43:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:43:12.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:43:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:43:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:43:12.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:43:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:43:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:43:12.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:43:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:43:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:43:12.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:43:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:43:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:43:12.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:43:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:43:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:43:12.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:43:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:43:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:43:12.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:43:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:43:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:43:12.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:43:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:43:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:43:12.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:43:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:43:12.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:43:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:43:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:43:12.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:43:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:43:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:43:12.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:43:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:43:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:43:12.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:43:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:43:12.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:43:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:43:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:43:12.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:43:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:43:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:43:12.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=861] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:43:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=861] Local address: 172.21.15.161, port 59170 2022-07-03T18:43:12.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=861] Peer address: 172.21.15.101, port 22 2022-07-03T18:43:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=861] Connection lost 2022-07-03T18:43:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=861] Aborting connection 2022-07-03T18:43:12.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:12 smithi161 ceph-mon[91579]: pgmap v14521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:12.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:43:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:43:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:43:12.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:43:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:43:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:43:12.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:43:12.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:43:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:43:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:43:12.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:43:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:43:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:43:12.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:43:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:43:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:43:12.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:43:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:43:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:43:12.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:43:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:43:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:43:12.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:43:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:43:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:43:12.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:43:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:43:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:43:12.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:43:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:43:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:43:12.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:43:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:43:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:43:12.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:43:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:43:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:43:12.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:43:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:43:12.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:43:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:43:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:43:12.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:43:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:43:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:43:12.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:43:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:43:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:43:12.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:43:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=861] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:43:12.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=861] Local address: 172.21.15.161, port 59170 2022-07-03T18:43:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=861] Peer address: 172.21.15.101, port 22 2022-07-03T18:43:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=861] Connection lost 2022-07-03T18:43:12.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=861] Aborting connection 2022-07-03T18:43:12.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:12 smithi188 ceph-mon[92425]: pgmap v14521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:14.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:14 smithi161 ceph-mon[91579]: pgmap v14522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:14 smithi188 ceph-mon[92425]: pgmap v14522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:15 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T18:43:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:43:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:15 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:43:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:15 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T18:43:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:43:15.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:15 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:43:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:16 smithi188 ceph-mon[92425]: pgmap v14523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:16.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:16 smithi161 ceph-mon[91579]: pgmap v14523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:17.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:17.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:19 smithi161 ceph-mon[91579]: pgmap v14524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:19 smithi188 ceph-mon[92425]: pgmap v14524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:20.515 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:21 smithi161 ceph-mon[91579]: pgmap v14525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:21 smithi188 ceph-mon[92425]: pgmap v14525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:23 smithi161 ceph-mon[91579]: pgmap v14526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:23 smithi188 ceph-mon[92425]: pgmap v14526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:25 smithi161 ceph-mon[91579]: pgmap v14527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:25 smithi188 ceph-mon[92425]: pgmap v14527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:27 smithi161 ceph-mon[91579]: pgmap v14528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:27 smithi188 ceph-mon[92425]: pgmap v14528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:29 smithi161 ceph-mon[91579]: pgmap v14529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:29 smithi188 ceph-mon[92425]: pgmap v14529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:31 smithi161 ceph-mon[91579]: pgmap v14530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:31 smithi188 ceph-mon[92425]: pgmap v14530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:33 smithi161 ceph-mon[91579]: pgmap v14531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:33 smithi188 ceph-mon[92425]: pgmap v14531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:35 smithi161 ceph-mon[91579]: pgmap v14532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:35 smithi188 ceph-mon[92425]: pgmap v14532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:37 smithi161 ceph-mon[91579]: pgmap v14533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:37 smithi188 ceph-mon[92425]: pgmap v14533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:39 smithi161 ceph-mon[91579]: pgmap v14534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:39 smithi188 ceph-mon[92425]: pgmap v14534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:41 smithi161 ceph-mon[91579]: pgmap v14535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:41 smithi188 ceph-mon[92425]: pgmap v14535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:43 smithi161 ceph-mon[91579]: pgmap v14536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:43 smithi188 ceph-mon[92425]: pgmap v14536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:45 smithi161 ceph-mon[91579]: pgmap v14537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:45 smithi188 ceph-mon[92425]: pgmap v14537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:47 smithi161 ceph-mon[91579]: pgmap v14538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:47 smithi188 ceph-mon[92425]: pgmap v14538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:49 smithi161 ceph-mon[91579]: pgmap v14539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:49 smithi188 ceph-mon[92425]: pgmap v14539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:51 smithi161 ceph-mon[91579]: pgmap v14540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:51 smithi188 ceph-mon[92425]: pgmap v14540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:53 smithi161 ceph-mon[91579]: pgmap v14541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:53 smithi188 ceph-mon[92425]: pgmap v14541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:55 smithi161 ceph-mon[91579]: pgmap v14542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:55 smithi188 ceph-mon[92425]: pgmap v14542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:43:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:57 smithi161 ceph-mon[91579]: pgmap v14543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:57 smithi188 ceph-mon[92425]: pgmap v14543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:43:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:43:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:59 smithi161 ceph-mon[91579]: pgmap v14544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:43:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:43:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:43:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:43:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:59 smithi188 ceph-mon[92425]: pgmap v14544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:43:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:43:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:43:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:43:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:44:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:01 smithi161 ceph-mon[91579]: pgmap v14545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:01 smithi188 ceph-mon[92425]: pgmap v14545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:03 smithi161 ceph-mon[91579]: pgmap v14546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:03 smithi188 ceph-mon[92425]: pgmap v14546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:05 smithi161 ceph-mon[91579]: pgmap v14547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:05 smithi188 ceph-mon[92425]: pgmap v14547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:07 smithi161 ceph-mon[91579]: pgmap v14548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:07 smithi188 ceph-mon[92425]: pgmap v14548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:09 smithi161 ceph-mon[91579]: pgmap v14549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:09 smithi188 ceph-mon[92425]: pgmap v14549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:11 smithi161 ceph-mon[91579]: pgmap v14550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:11 smithi188 ceph-mon[92425]: pgmap v14550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:13 smithi161 ceph-mon[91579]: pgmap v14551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:13 smithi188 ceph-mon[92425]: pgmap v14551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:15.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:44:14.761+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:44:15.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:44:15.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:44:15.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:44:15.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:44:15.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:44:15.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:44:15.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:44:15.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:44:15.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:44:15.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:44:15.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:44:15.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:44:15.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:44:15.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:44:15.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:44:15.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:44:15.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:44:15.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:44:15.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:44:15.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:44:15.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:44:15.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:44:15.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:44:15.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:44:15.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:44:15.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:44:15.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:44:15.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:44:15.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:44:15.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:44:15.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:44:15.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:44:15.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:44:15.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:44:15.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:44:15.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:44:15.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:44:15.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:44:15.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:44:15.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:44:15.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:44:15.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:44:15.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:44:15.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:44:15.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=863] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:44:15.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=863] Local address: 172.21.15.161, port 59200 2022-07-03T18:44:15.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=863] Peer address: 172.21.15.101, port 22 2022-07-03T18:44:15.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=863] Connection lost 2022-07-03T18:44:15.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=863] Aborting connection 2022-07-03T18:44:15.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:44:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:44:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:15 smithi161 ceph-mon[91579]: pgmap v14552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:44:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:44:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:44:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:15 smithi188 ceph-mon[92425]: pgmap v14552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:44:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:44:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:44:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:44:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:44:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:44:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:44:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:44:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:44:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:44:16.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:44:16.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:44:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:44:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:44:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:44:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:44:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:44:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:44:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:44:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:44:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:44:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:44:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:44:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:44:16.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:44:16.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:44:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:44:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:44:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:44:16.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:44:16.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:44:16.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:44:16.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:44:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:44:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:44:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:44:16.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:44:16.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:44:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:44:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:44:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:44:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:44:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:44:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:44:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:44:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:44:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:44:16.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:44:16.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:44:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=863] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:44:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=863] Local address: 172.21.15.161, port 59200 2022-07-03T18:44:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=863] Peer address: 172.21.15.101, port 22 2022-07-03T18:44:16.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=863] Connection lost 2022-07-03T18:44:16.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=863] Aborting connection 2022-07-03T18:44:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:44:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:44:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:44:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:44:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:44:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:44:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:44:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:44:16.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:44:16.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:44:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:44:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:44:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:44:16.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:44:16.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:44:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:44:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:44:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:44:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:44:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:44:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:44:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:44:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:44:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:44:16.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:44:16.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:44:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:44:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:44:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:44:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:44:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:44:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:44:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:44:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:44:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:44:16.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:44:16.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:44:16.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:44:16.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:44:16.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:44:16.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:44:16.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:44:16.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:44:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:44:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:44:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:44:16.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=863] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:44:16.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=863] Local address: 172.21.15.161, port 59200 2022-07-03T18:44:16.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=863] Peer address: 172.21.15.101, port 22 2022-07-03T18:44:16.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=863] Connection lost 2022-07-03T18:44:16.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=863] Aborting connection 2022-07-03T18:44:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:17 smithi161 ceph-mon[91579]: pgmap v14553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:17 smithi188 ceph-mon[92425]: pgmap v14553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:19 smithi161 ceph-mon[91579]: pgmap v14554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:19 smithi188 ceph-mon[92425]: pgmap v14554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:21 smithi161 ceph-mon[91579]: pgmap v14555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:21 smithi188 ceph-mon[92425]: pgmap v14555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:22 smithi161 ceph-mon[91579]: pgmap v14556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:22 smithi188 ceph-mon[92425]: pgmap v14556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:24.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:24 smithi188 ceph-mon[92425]: pgmap v14557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:24 smithi161 ceph-mon[91579]: pgmap v14557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:26 smithi188 ceph-mon[92425]: pgmap v14558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:26 smithi161 ceph-mon[91579]: pgmap v14558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:28.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:29 smithi161 ceph-mon[91579]: pgmap v14559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:29 smithi188 ceph-mon[92425]: pgmap v14559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:31 smithi161 ceph-mon[91579]: pgmap v14560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:31 smithi188 ceph-mon[92425]: pgmap v14560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:33 smithi161 ceph-mon[91579]: pgmap v14561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:33 smithi188 ceph-mon[92425]: pgmap v14561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:35 smithi161 ceph-mon[91579]: pgmap v14562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:35 smithi188 ceph-mon[92425]: pgmap v14562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:37 smithi161 ceph-mon[91579]: pgmap v14563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:37 smithi188 ceph-mon[92425]: pgmap v14563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:39 smithi161 ceph-mon[91579]: pgmap v14564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:39 smithi188 ceph-mon[92425]: pgmap v14564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:41 smithi161 ceph-mon[91579]: pgmap v14565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:41 smithi188 ceph-mon[92425]: pgmap v14565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:43 smithi161 ceph-mon[91579]: pgmap v14566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:43 smithi188 ceph-mon[92425]: pgmap v14566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:45 smithi161 ceph-mon[91579]: pgmap v14567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:45.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:45 smithi188 ceph-mon[92425]: pgmap v14567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:47 smithi161 ceph-mon[91579]: pgmap v14568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:47 smithi188 ceph-mon[92425]: pgmap v14568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:49 smithi161 ceph-mon[91579]: pgmap v14569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:49 smithi188 ceph-mon[92425]: pgmap v14569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:51 smithi161 ceph-mon[91579]: pgmap v14570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:51 smithi188 ceph-mon[92425]: pgmap v14570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:53 smithi161 ceph-mon[91579]: pgmap v14571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:53 smithi188 ceph-mon[92425]: pgmap v14571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:55 smithi161 ceph-mon[91579]: pgmap v14572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:55.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:55 smithi188 ceph-mon[92425]: pgmap v14572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:44:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:57 smithi161 ceph-mon[91579]: pgmap v14573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:57 smithi188 ceph-mon[92425]: pgmap v14573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:44:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:44:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:59 smithi161 ceph-mon[91579]: pgmap v14574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:44:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:44:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:44:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:44:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:59 smithi188 ceph-mon[92425]: pgmap v14574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:44:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:44:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:44:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:44:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:45:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:01 smithi161 ceph-mon[91579]: pgmap v14575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:01 smithi188 ceph-mon[92425]: pgmap v14575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:03 smithi161 ceph-mon[91579]: pgmap v14576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:03 smithi188 ceph-mon[92425]: pgmap v14576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:05 smithi161 ceph-mon[91579]: pgmap v14577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:05 smithi188 ceph-mon[92425]: pgmap v14577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:07 smithi161 ceph-mon[91579]: pgmap v14578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:07 smithi188 ceph-mon[92425]: pgmap v14578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:09 smithi161 ceph-mon[91579]: pgmap v14579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:09 smithi188 ceph-mon[92425]: pgmap v14579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:10.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:11 smithi161 ceph-mon[91579]: pgmap v14580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:11 smithi188 ceph-mon[92425]: pgmap v14580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:13 smithi161 ceph-mon[91579]: pgmap v14581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:13 smithi188 ceph-mon[92425]: pgmap v14581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-mon[91579]: pgmap v14582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:45:15.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:45:15.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:45:15.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:45:15.276+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:45:15.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:45:15.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:45:15.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:45:15.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:45:15.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:45:15.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:45:15.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:45:15.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:45:15.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:45:15.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:45:15.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:45:15.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:45:15.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:45:15.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:45:15.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:45:15.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:45:15.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:45:15.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:45:15.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:45:15.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:45:15.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:45:15.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:45:15.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:45:15.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:45:15.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:45:15.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:45:15.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:45:15.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:45:15.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:45:15.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:45:15.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:45:15.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:45:15.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:45:15.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:45:15.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:45:15.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:45:15.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:45:15.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:45:15.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:45:15.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:45:15.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:45:15.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:45:15.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:45:15.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:45:15.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=865] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:45:15.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=865] Local address: 172.21.15.161, port 59228 2022-07-03T18:45:15.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=865] Peer address: 172.21.15.101, port 22 2022-07-03T18:45:15.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=865] Connection lost 2022-07-03T18:45:15.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=865] Aborting connection 2022-07-03T18:45:15.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:45:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:15 smithi188 ceph-mon[92425]: pgmap v14582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:45:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:45:15.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:45:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:45:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:45:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:45:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:45:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:45:16.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:45:16.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:45:16.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:45:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:45:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:45:16.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:45:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:45:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:45:16.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:45:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:45:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:45:16.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:45:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:45:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:45:16.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:45:16.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:45:16.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:45:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:45:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:45:16.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:45:16.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:45:16.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:45:16.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:45:16.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:45:16.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:45:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:45:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:45:16.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:45:16.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:45:16.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:45:16.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:45:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:45:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:45:16.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:45:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:45:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:45:16.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:45:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:45:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:45:16.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:45:16.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:45:16.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=865] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:45:16.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=865] Local address: 172.21.15.161, port 59228 2022-07-03T18:45:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=865] Peer address: 172.21.15.101, port 22 2022-07-03T18:45:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=865] Connection lost 2022-07-03T18:45:16.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=865] Aborting connection 2022-07-03T18:45:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:45:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:45:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:45:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:45:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:45:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:45:16.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:45:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:45:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:45:16.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:45:16.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:45:16.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:45:16.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:45:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:45:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:45:16.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:45:16.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:45:16.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:45:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:45:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:45:16.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:45:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:45:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:45:16.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:45:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:45:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:45:16.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:45:16.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:45:16.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:45:16.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:45:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:45:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:45:16.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:45:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:45:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:45:16.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:45:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:45:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:45:16.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:45:16.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:45:16.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:45:16.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:45:16.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:45:16.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:45:16.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:45:16.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:45:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=865] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:45:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=865] Local address: 172.21.15.161, port 59228 2022-07-03T18:45:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=865] Peer address: 172.21.15.101, port 22 2022-07-03T18:45:16.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=865] Connection lost 2022-07-03T18:45:16.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=865] Aborting connection 2022-07-03T18:45:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:17 smithi161 ceph-mon[91579]: pgmap v14583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:17 smithi188 ceph-mon[92425]: pgmap v14583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:19 smithi161 ceph-mon[91579]: pgmap v14584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:19 smithi188 ceph-mon[92425]: pgmap v14584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:21 smithi161 ceph-mon[91579]: pgmap v14585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:21 smithi188 ceph-mon[92425]: pgmap v14585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:23 smithi161 ceph-mon[91579]: pgmap v14586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:23 smithi188 ceph-mon[92425]: pgmap v14586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:25 smithi161 ceph-mon[91579]: pgmap v14587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:25 smithi188 ceph-mon[92425]: pgmap v14587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:27 smithi161 ceph-mon[91579]: pgmap v14588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:27 smithi188 ceph-mon[92425]: pgmap v14588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:29 smithi161 ceph-mon[91579]: pgmap v14589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:29 smithi188 ceph-mon[92425]: pgmap v14589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:31 smithi188 ceph-mon[92425]: pgmap v14590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:31 smithi161 ceph-mon[91579]: pgmap v14590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:32.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:32 smithi188 ceph-mon[92425]: pgmap v14591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:32 smithi161 ceph-mon[91579]: pgmap v14591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:34.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:34 smithi188 ceph-mon[92425]: pgmap v14592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:34 smithi161 ceph-mon[91579]: pgmap v14592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:36 smithi188 ceph-mon[92425]: pgmap v14593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:36 smithi161 ceph-mon[91579]: pgmap v14593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:38.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:39 smithi161 ceph-mon[91579]: pgmap v14594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:39 smithi188 ceph-mon[92425]: pgmap v14594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:40.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:41 smithi161 ceph-mon[91579]: pgmap v14595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:41 smithi188 ceph-mon[92425]: pgmap v14595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:43 smithi161 ceph-mon[91579]: pgmap v14596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:43 smithi188 ceph-mon[92425]: pgmap v14596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:45 smithi161 ceph-mon[91579]: pgmap v14597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:45 smithi188 ceph-mon[92425]: pgmap v14597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:47.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:47 smithi161 ceph-mon[91579]: pgmap v14598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:47 smithi188 ceph-mon[92425]: pgmap v14598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:49 smithi161 ceph-mon[91579]: pgmap v14599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:49 smithi188 ceph-mon[92425]: pgmap v14599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:51 smithi161 ceph-mon[91579]: pgmap v14600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:51 smithi188 ceph-mon[92425]: pgmap v14600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:53 smithi161 ceph-mon[91579]: pgmap v14601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:53 smithi188 ceph-mon[92425]: pgmap v14601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:55 smithi161 ceph-mon[91579]: pgmap v14602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:55 smithi188 ceph-mon[92425]: pgmap v14602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:45:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:57 smithi161 ceph-mon[91579]: pgmap v14603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:57.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:57 smithi188 ceph-mon[92425]: pgmap v14603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:45:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:45:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:59 smithi161 ceph-mon[91579]: pgmap v14604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:45:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:45:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:45:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:45:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:59 smithi188 ceph-mon[92425]: pgmap v14604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:45:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:45:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:45:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:45:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:46:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:01 smithi161 ceph-mon[91579]: pgmap v14605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:01 smithi188 ceph-mon[92425]: pgmap v14605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:03 smithi161 ceph-mon[91579]: pgmap v14606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:03 smithi188 ceph-mon[92425]: pgmap v14606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:05 smithi161 ceph-mon[91579]: pgmap v14607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:05 smithi188 ceph-mon[92425]: pgmap v14607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:07 smithi161 ceph-mon[91579]: pgmap v14608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:07 smithi188 ceph-mon[92425]: pgmap v14608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:09 smithi161 ceph-mon[91579]: pgmap v14609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:09 smithi188 ceph-mon[92425]: pgmap v14609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:10.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:11 smithi161 ceph-mon[91579]: pgmap v14610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:11 smithi188 ceph-mon[92425]: pgmap v14610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:13 smithi161 ceph-mon[91579]: pgmap v14611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:13 smithi188 ceph-mon[92425]: pgmap v14611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-mon[91579]: pgmap v14612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:15 smithi188 ceph-mon[92425]: pgmap v14612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:16.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:46:15.782+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:46:16.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:46:16.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:46:16.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:46:16.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:46:16.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:46:16.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:46:16.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:46:16.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:46:16.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:46:16.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:46:16.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:46:16.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:46:16.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:46:16.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:46:16.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:46:16.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:46:16.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:46:16.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:46:16.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:46:16.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:46:16.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:46:16.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:46:16.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:46:16.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:46:16.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:46:16.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:46:16.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:46:16.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:46:16.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:46:16.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:46:16.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:46:16.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:46:16.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:46:16.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:46:16.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:46:16.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:46:16.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:46:16.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:46:16.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:46:16.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:46:16.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:46:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:46:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:46:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:46:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=867] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:46:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=867] Local address: 172.21.15.161, port 59256 2022-07-03T18:46:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=867] Peer address: 172.21.15.101, port 22 2022-07-03T18:46:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=867] Connection lost 2022-07-03T18:46:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=867] Aborting connection 2022-07-03T18:46:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:46:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:46:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:46:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:46:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:46:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:46:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:46:16.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:46:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:17 smithi161 ceph-mon[91579]: pgmap v14613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:46:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:46:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:46:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:46:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:46:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:46:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:46:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:46:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:46:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:46:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:46:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:46:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:46:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:46:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:46:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:46:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=867] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:46:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=867] Local address: 172.21.15.161, port 59256 2022-07-03T18:46:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=867] Peer address: 172.21.15.101, port 22 2022-07-03T18:46:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=867] Connection lost 2022-07-03T18:46:17.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=867] Aborting connection 2022-07-03T18:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:17 smithi188 ceph-mon[92425]: pgmap v14613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:46:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:46:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:46:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:46:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:46:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:46:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:46:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:46:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:46:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:46:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:46:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:46:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:46:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:46:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:46:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:46:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:46:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:46:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:46:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:46:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:46:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:46:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:46:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:46:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:46:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:46:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:46:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:46:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=867] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:46:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=867] Local address: 172.21.15.161, port 59256 2022-07-03T18:46:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=867] Peer address: 172.21.15.101, port 22 2022-07-03T18:46:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=867] Connection lost 2022-07-03T18:46:17.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=867] Aborting connection 2022-07-03T18:46:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:19 smithi161 ceph-mon[91579]: pgmap v14614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:19 smithi188 ceph-mon[92425]: pgmap v14614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:21 smithi161 ceph-mon[91579]: pgmap v14615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:21 smithi188 ceph-mon[92425]: pgmap v14615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:23 smithi161 ceph-mon[91579]: pgmap v14616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:23 smithi188 ceph-mon[92425]: pgmap v14616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:25 smithi161 ceph-mon[91579]: pgmap v14617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:25 smithi188 ceph-mon[92425]: pgmap v14617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:27 smithi161 ceph-mon[91579]: pgmap v14618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:27 smithi188 ceph-mon[92425]: pgmap v14618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:29 smithi161 ceph-mon[91579]: pgmap v14619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:29 smithi188 ceph-mon[92425]: pgmap v14619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:31 smithi161 ceph-mon[91579]: pgmap v14620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:31 smithi188 ceph-mon[92425]: pgmap v14620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:33 smithi188 ceph-mon[92425]: pgmap v14621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:33 smithi161 ceph-mon[91579]: pgmap v14621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:35 smithi188 ceph-mon[92425]: pgmap v14622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:35 smithi161 ceph-mon[91579]: pgmap v14622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:37 smithi188 ceph-mon[92425]: pgmap v14623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:37 smithi161 ceph-mon[91579]: pgmap v14623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:39 smithi188 ceph-mon[92425]: pgmap v14624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:39 smithi161 ceph-mon[91579]: pgmap v14624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:41 smithi188 ceph-mon[92425]: pgmap v14625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:41 smithi161 ceph-mon[91579]: pgmap v14625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:42 smithi188 ceph-mon[92425]: pgmap v14626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:42 smithi161 ceph-mon[91579]: pgmap v14626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:44 smithi161 ceph-mon[91579]: pgmap v14627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:44 smithi188 ceph-mon[92425]: pgmap v14627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:46 smithi161 ceph-mon[91579]: pgmap v14628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:46 smithi188 ceph-mon[92425]: pgmap v14628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:49.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:49 smithi161 ceph-mon[91579]: pgmap v14629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:49 smithi188 ceph-mon[92425]: pgmap v14629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:51 smithi161 ceph-mon[91579]: pgmap v14630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:51 smithi188 ceph-mon[92425]: pgmap v14630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:53 smithi161 ceph-mon[91579]: pgmap v14631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:53 smithi188 ceph-mon[92425]: pgmap v14631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:55 smithi161 ceph-mon[91579]: pgmap v14632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:55.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:55 smithi188 ceph-mon[92425]: pgmap v14632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:46:57.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:57 smithi161 ceph-mon[91579]: pgmap v14633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:57 smithi188 ceph-mon[92425]: pgmap v14633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:46:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:46:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:59 smithi161 ceph-mon[91579]: pgmap v14634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:46:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:46:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:46:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:46:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:59 smithi188 ceph-mon[92425]: pgmap v14634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:46:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:46:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:46:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:46:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:47:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:01 smithi161 ceph-mon[91579]: pgmap v14635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:01 smithi188 ceph-mon[92425]: pgmap v14635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:03 smithi161 ceph-mon[91579]: pgmap v14636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:03.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:03 smithi188 ceph-mon[92425]: pgmap v14636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:05 smithi161 ceph-mon[91579]: pgmap v14637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:05 smithi188 ceph-mon[92425]: pgmap v14637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:07 smithi161 ceph-mon[91579]: pgmap v14638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:07 smithi188 ceph-mon[92425]: pgmap v14638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:09 smithi161 ceph-mon[91579]: pgmap v14639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:09 smithi188 ceph-mon[92425]: pgmap v14639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:11 smithi161 ceph-mon[91579]: pgmap v14640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:11 smithi188 ceph-mon[92425]: pgmap v14640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:13 smithi161 ceph-mon[91579]: pgmap v14641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:13 smithi188 ceph-mon[92425]: pgmap v14641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:15 smithi161 ceph-mon[91579]: pgmap v14642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:15 smithi188 ceph-mon[92425]: pgmap v14642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:47:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:47:16.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:47:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:47:16.283+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:47:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:47:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:47:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:47:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:47:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:47:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:47:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:47:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:47:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:47:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:47:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:47:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:47:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:47:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:47:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:47:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:47:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:47:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:47:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:47:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:47:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:47:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:47:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:47:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:47:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:47:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:47:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:47:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:47:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:47:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:47:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:47:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:47:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:47:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:47:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:47:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:47:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:47:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:47:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:47:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:47:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:47:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:47:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:47:16.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:47:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=869] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:47:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=869] Local address: 172.21.15.161, port 59284 2022-07-03T18:47:16.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=869] Peer address: 172.21.15.101, port 22 2022-07-03T18:47:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=869] Connection lost 2022-07-03T18:47:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=869] Aborting connection 2022-07-03T18:47:16.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:47:16.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:47:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:47:16.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:47:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: pgmap v14643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:47:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:47:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:47:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:47:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:47:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:47:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:47:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:47:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:47:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:47:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=869] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:47:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=869] Local address: 172.21.15.161, port 59284 2022-07-03T18:47:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=869] Peer address: 172.21.15.101, port 22 2022-07-03T18:47:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=869] Connection lost 2022-07-03T18:47:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=869] Aborting connection 2022-07-03T18:47:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: pgmap v14643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:47:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:47:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:47:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:47:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:47:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:47:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:47:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:47:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:47:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:47:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:47:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:47:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:47:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:47:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:47:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:47:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:47:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:47:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:47:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:47:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:47:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:47:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:47:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:47:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:47:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:47:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:47:17.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:47:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:47:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:47:17.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:47:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:47:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:47:17.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:47:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=869] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:47:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=869] Local address: 172.21.15.161, port 59284 2022-07-03T18:47:17.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=869] Peer address: 172.21.15.101, port 22 2022-07-03T18:47:17.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=869] Connection lost 2022-07-03T18:47:17.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=869] Aborting connection 2022-07-03T18:47:17.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:17.688 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:18.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:47:17.646+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T18:47:18.567 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T18:47:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:19 smithi161 ceph-mon[91579]: pgmap v14644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:19 smithi188 ceph-mon[92425]: pgmap v14644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:20.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:20.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:47:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:21 smithi161 ceph-mon[91579]: pgmap v14645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:21 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T18:47:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:21 smithi188 ceph-mon[92425]: pgmap v14645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:21 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T18:47:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:23 smithi161 ceph-mon[91579]: pgmap v14646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:23 smithi188 ceph-mon[92425]: pgmap v14646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:25 smithi161 ceph-mon[91579]: pgmap v14647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:25 smithi188 ceph-mon[92425]: pgmap v14647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:27 smithi161 ceph-mon[91579]: pgmap v14648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:27 smithi188 ceph-mon[92425]: pgmap v14648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:29 smithi161 ceph-mon[91579]: pgmap v14649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:29 smithi188 ceph-mon[92425]: pgmap v14649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:31 smithi161 ceph-mon[91579]: pgmap v14650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:31 smithi188 ceph-mon[92425]: pgmap v14650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:33 smithi161 ceph-mon[91579]: pgmap v14651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:33 smithi188 ceph-mon[92425]: pgmap v14651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:35 smithi161 ceph-mon[91579]: pgmap v14652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:35 smithi188 ceph-mon[92425]: pgmap v14652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:37 smithi161 ceph-mon[91579]: pgmap v14653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:37 smithi188 ceph-mon[92425]: pgmap v14653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:39 smithi161 ceph-mon[91579]: pgmap v14654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:39 smithi188 ceph-mon[92425]: pgmap v14654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:41 smithi188 ceph-mon[92425]: pgmap v14655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:41 smithi161 ceph-mon[91579]: pgmap v14655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:43 smithi188 ceph-mon[92425]: pgmap v14656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:43 smithi161 ceph-mon[91579]: pgmap v14656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:45 smithi188 ceph-mon[92425]: pgmap v14657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:45 smithi161 ceph-mon[91579]: pgmap v14657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:47 smithi188 ceph-mon[92425]: pgmap v14658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:47 smithi161 ceph-mon[91579]: pgmap v14658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:49 smithi188 ceph-mon[92425]: pgmap v14659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:49 smithi161 ceph-mon[91579]: pgmap v14659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:51 smithi188 ceph-mon[92425]: pgmap v14660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:51 smithi161 ceph-mon[91579]: pgmap v14660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:53 smithi188 ceph-mon[92425]: pgmap v14661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:53 smithi161 ceph-mon[91579]: pgmap v14661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:54 smithi188 ceph-mon[92425]: pgmap v14662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:54 smithi161 ceph-mon[91579]: pgmap v14662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:47:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:56 smithi161 ceph-mon[91579]: pgmap v14663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:56 smithi188 ceph-mon[92425]: pgmap v14663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:47:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:59 smithi161 ceph-mon[91579]: pgmap v14664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:47:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:47:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:47:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:47:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:59 smithi188 ceph-mon[92425]: pgmap v14664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:47:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:47:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:47:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:47:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:48:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:01 smithi161 ceph-mon[91579]: pgmap v14665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:01 smithi188 ceph-mon[92425]: pgmap v14665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:03 smithi161 ceph-mon[91579]: pgmap v14666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:03 smithi188 ceph-mon[92425]: pgmap v14666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:05 smithi161 ceph-mon[91579]: pgmap v14667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:05 smithi188 ceph-mon[92425]: pgmap v14667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:07 smithi161 ceph-mon[91579]: pgmap v14668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:07 smithi188 ceph-mon[92425]: pgmap v14668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:09 smithi161 ceph-mon[91579]: pgmap v14669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:09 smithi188 ceph-mon[92425]: pgmap v14669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:10.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:11 smithi161 ceph-mon[91579]: pgmap v14670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:11.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:11 smithi188 ceph-mon[92425]: pgmap v14670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:13 smithi161 ceph-mon[91579]: pgmap v14671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:13 smithi188 ceph-mon[92425]: pgmap v14671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:15 smithi161 ceph-mon[91579]: pgmap v14672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:15 smithi188 ceph-mon[92425]: pgmap v14672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:15.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:17 smithi161 ceph-mon[91579]: pgmap v14673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:17 smithi188 ceph-mon[92425]: pgmap v14673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:19 smithi161 ceph-mon[91579]: pgmap v14674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:19 smithi188 ceph-mon[92425]: pgmap v14674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:48:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:48:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:48:20.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:48:20.099+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:48:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:48:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:48:20.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:48:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:48:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:48:20.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:48:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:48:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:48:20.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:48:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:48:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:48:20.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:48:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:48:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:48:20.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:48:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:48:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:48:20.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:48:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:48:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:48:20.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:48:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:48:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:48:20.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:48:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:48:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:48:20.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:48:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:48:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:48:20.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:48:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:48:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:48:20.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:48:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:48:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:48:20.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:48:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:48:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:48:20.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:48:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:48:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:48:20.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:48:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:48:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:48:20.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=871] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:48:20.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=871] Local address: 172.21.15.161, port 59290 2022-07-03T18:48:20.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=871] Peer address: 172.21.15.101, port 22 2022-07-03T18:48:20.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=871] Connection lost 2022-07-03T18:48:20.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=871] Aborting connection 2022-07-03T18:48:20.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:48:20 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:48:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:48:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:48:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:48:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: pgmap v14675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:48:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:48:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:48:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:48:21.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:48:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:48:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:48:21.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:48:21.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:48:21.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:48:21.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:48:21.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:48:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:48:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:48:21.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:48:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:48:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:48:21.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:48:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:48:21.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:48:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:48:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:48:21.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:48:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:48:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:48:21.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:48:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:48:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:48:21.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:48:21.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:48:21.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:48:21.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:48:21.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:48:21.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:48:21.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:48:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:48:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:48:21.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:48:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:48:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:48:21.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:48:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:48:21.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:48:21.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:48:21.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:48:21.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:48:21.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=871] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:48:21.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=871] Local address: 172.21.15.161, port 59290 2022-07-03T18:48:21.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=871] Peer address: 172.21.15.101, port 22 2022-07-03T18:48:21.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=871] Connection lost 2022-07-03T18:48:21.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=871] Aborting connection 2022-07-03T18:48:21.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.650 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: pgmap v14675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:48:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:48:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:48:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:48:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:48:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:48:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:48:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:48:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:48:21.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:48:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:48:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:48:21.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:48:21.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:48:21.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:48:21.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:48:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:48:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:48:21.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:48:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:48:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:48:21.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:48:21.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:48:21.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:48:21.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:48:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:48:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:48:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:48:21.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:48:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:48:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:48:21.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:48:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:48:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:48:21.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:48:21.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:48:21.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:48:21.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:48:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:48:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:48:21.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:48:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:48:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:48:21.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:48:21.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:48:21.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:48:21.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=871] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:48:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=871] Local address: 172.21.15.161, port 59290 2022-07-03T18:48:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=871] Peer address: 172.21.15.101, port 22 2022-07-03T18:48:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=871] Connection lost 2022-07-03T18:48:21.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=871] Aborting connection 2022-07-03T18:48:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:21.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:48:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:23 smithi161 ceph-mon[91579]: pgmap v14676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:23 smithi188 ceph-mon[92425]: pgmap v14676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:25 smithi161 ceph-mon[91579]: pgmap v14677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:25 smithi188 ceph-mon[92425]: pgmap v14677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:27 smithi161 ceph-mon[91579]: pgmap v14678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:27 smithi188 ceph-mon[92425]: pgmap v14678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:29 smithi161 ceph-mon[91579]: pgmap v14679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:29 smithi188 ceph-mon[92425]: pgmap v14679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:31 smithi161 ceph-mon[91579]: pgmap v14680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:31 smithi188 ceph-mon[92425]: pgmap v14680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:33 smithi161 ceph-mon[91579]: pgmap v14681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:33 smithi188 ceph-mon[92425]: pgmap v14681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:35 smithi161 ceph-mon[91579]: pgmap v14682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:35 smithi188 ceph-mon[92425]: pgmap v14682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:37 smithi161 ceph-mon[91579]: pgmap v14683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:37 smithi188 ceph-mon[92425]: pgmap v14683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:39.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:39 smithi161 ceph-mon[91579]: pgmap v14684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:39 smithi188 ceph-mon[92425]: pgmap v14684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:41 smithi161 ceph-mon[91579]: pgmap v14685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:41 smithi188 ceph-mon[92425]: pgmap v14685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:43 smithi188 ceph-mon[92425]: pgmap v14686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:43.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:43 smithi161 ceph-mon[91579]: pgmap v14686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:45 smithi188 ceph-mon[92425]: pgmap v14687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:45 smithi161 ceph-mon[91579]: pgmap v14687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:47 smithi188 ceph-mon[92425]: pgmap v14688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:47.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:47 smithi161 ceph-mon[91579]: pgmap v14688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:49 smithi188 ceph-mon[92425]: pgmap v14689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:49 smithi161 ceph-mon[91579]: pgmap v14689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:51 smithi188 ceph-mon[92425]: pgmap v14690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:51 smithi161 ceph-mon[91579]: pgmap v14690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:53 smithi188 ceph-mon[92425]: pgmap v14691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:53 smithi161 ceph-mon[91579]: pgmap v14691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:55 smithi188 ceph-mon[92425]: pgmap v14692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:55.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:55 smithi161 ceph-mon[91579]: pgmap v14692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:48:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:57 smithi188 ceph-mon[92425]: pgmap v14693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:57 smithi161 ceph-mon[91579]: pgmap v14693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:48:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:59 smithi161 ceph-mon[91579]: pgmap v14694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:48:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:48:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:48:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:48:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:59 smithi188 ceph-mon[92425]: pgmap v14694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:48:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:48:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:48:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:48:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:49:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:01 smithi161 ceph-mon[91579]: pgmap v14695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:01 smithi188 ceph-mon[92425]: pgmap v14695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:02 smithi161 ceph-mon[91579]: pgmap v14696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:02 smithi188 ceph-mon[92425]: pgmap v14696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:05 smithi161 ceph-mon[91579]: pgmap v14697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:05 smithi188 ceph-mon[92425]: pgmap v14697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:07 smithi161 ceph-mon[91579]: pgmap v14698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:07 smithi188 ceph-mon[92425]: pgmap v14698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:09 smithi161 ceph-mon[91579]: pgmap v14699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:09 smithi188 ceph-mon[92425]: pgmap v14699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:11 smithi161 ceph-mon[91579]: pgmap v14700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:11 smithi188 ceph-mon[92425]: pgmap v14700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:13 smithi161 ceph-mon[91579]: pgmap v14701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:13 smithi188 ceph-mon[92425]: pgmap v14701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:15 smithi161 ceph-mon[91579]: pgmap v14702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:15 smithi188 ceph-mon[92425]: pgmap v14702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:17 smithi161 ceph-mon[91579]: pgmap v14703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:17 smithi188 ceph-mon[92425]: pgmap v14703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:19 smithi161 ceph-mon[91579]: pgmap v14704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:19 smithi188 ceph-mon[92425]: pgmap v14704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:21.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:49:21.016+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:49:21.343 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:49:21.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:49:21.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:49:21.344 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:49:21.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:49:21.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:49:21.345 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:49:21.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:49:21.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:49:21.346 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:49:21.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:49:21.347 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:49:21.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:49:21.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:49:21.348 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:49:21.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:49:21.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:49:21.349 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:49:21.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:49:21.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:49:21.350 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:49:21.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:49:21.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:49:21.351 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:49:21.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:49:21.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:49:21.352 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:49:21.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:49:21.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:49:21.353 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:49:21.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:49:21.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:49:21.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:49:21.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:49:21.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:49:21.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:49:21.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:49:21.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:49:21.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:49:21.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:49:21.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:49:21.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:49:21.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:49:21.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:49:21.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=873] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:49:21.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=873] Local address: 172.21.15.161, port 59294 2022-07-03T18:49:21.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=873] Peer address: 172.21.15.101, port 22 2022-07-03T18:49:21.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=873] Connection lost 2022-07-03T18:49:21.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=873] Aborting connection 2022-07-03T18:49:21.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:49:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-mon[91579]: pgmap v14705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:49:21.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:49:21.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:49:21.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:21 smithi188 ceph-mon[92425]: pgmap v14705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:49:21.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:49:21.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:49:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:22 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:49:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:22 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:49:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:49:22.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:49:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:49:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:49:22.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:49:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:49:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:49:22.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:49:22.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:49:22.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:49:22.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:49:22.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:49:22.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:49:22.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:49:22.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:49:22.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:49:22.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:49:22.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:49:22.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:49:22.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:49:22.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:49:22.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:49:22.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:49:22.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:49:22.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:49:22.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:49:22.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:49:22.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:49:22.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:49:22.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:49:22.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:49:22.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:49:22.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:49:22.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:49:22.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:49:22.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:49:22.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:49:22.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:49:22.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:49:22.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:49:22.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:49:22.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:49:22.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:49:22.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:49:22.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=873] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:49:22.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=873] Local address: 172.21.15.161, port 59294 2022-07-03T18:49:22.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=873] Peer address: 172.21.15.101, port 22 2022-07-03T18:49:22.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=873] Connection lost 2022-07-03T18:49:22.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=873] Aborting connection 2022-07-03T18:49:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:22 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:49:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:22 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:49:22.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:49:22.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:49:22.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:49:22.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:49:22.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:49:22.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:49:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:49:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:49:22.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:49:22.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:49:22.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:49:22.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:49:22.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:49:22.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:49:22.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:49:22.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:49:22.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:49:22.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:49:22.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:49:22.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:49:22.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:49:22.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:49:22.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:49:22.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:49:22.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:49:22.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:49:22.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:49:22.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:49:22.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:49:22.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:49:22.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:49:22.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:49:22.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:49:22.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:49:22.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:49:22.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:49:22.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:49:22.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:49:22.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:49:22.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:49:22.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:49:22.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:49:22.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:49:22.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:49:22.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=873] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:49:22.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=873] Local address: 172.21.15.161, port 59294 2022-07-03T18:49:22.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=873] Peer address: 172.21.15.101, port 22 2022-07-03T18:49:22.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=873] Connection lost 2022-07-03T18:49:22.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=873] Aborting connection 2022-07-03T18:49:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:23 smithi161 ceph-mon[91579]: pgmap v14706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:23 smithi188 ceph-mon[92425]: pgmap v14706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:25 smithi161 ceph-mon[91579]: pgmap v14707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:25 smithi188 ceph-mon[92425]: pgmap v14707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:27 smithi161 ceph-mon[91579]: pgmap v14708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:27 smithi188 ceph-mon[92425]: pgmap v14708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:29 smithi161 ceph-mon[91579]: pgmap v14709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:29 smithi188 ceph-mon[92425]: pgmap v14709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:31 smithi161 ceph-mon[91579]: pgmap v14710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:31 smithi188 ceph-mon[92425]: pgmap v14710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:33 smithi161 ceph-mon[91579]: pgmap v14711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:33 smithi188 ceph-mon[92425]: pgmap v14711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:35 smithi161 ceph-mon[91579]: pgmap v14712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:35 smithi188 ceph-mon[92425]: pgmap v14712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:37 smithi188 ceph-mon[92425]: pgmap v14713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:37 smithi161 ceph-mon[91579]: pgmap v14713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:39 smithi188 ceph-mon[92425]: pgmap v14714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:39 smithi161 ceph-mon[91579]: pgmap v14714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:41 smithi188 ceph-mon[92425]: pgmap v14715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:41 smithi161 ceph-mon[91579]: pgmap v14715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:43 smithi188 ceph-mon[92425]: pgmap v14716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:43 smithi161 ceph-mon[91579]: pgmap v14716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:45 smithi188 ceph-mon[92425]: pgmap v14717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:45 smithi161 ceph-mon[91579]: pgmap v14717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:47 smithi188 ceph-mon[92425]: pgmap v14718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:47 smithi161 ceph-mon[91579]: pgmap v14718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:49 smithi188 ceph-mon[92425]: pgmap v14719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:49 smithi161 ceph-mon[91579]: pgmap v14719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:50.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:51 smithi188 ceph-mon[92425]: pgmap v14720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:51 smithi161 ceph-mon[91579]: pgmap v14720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:53 smithi161 ceph-mon[91579]: pgmap v14721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:53 smithi188 ceph-mon[92425]: pgmap v14721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:55 smithi161 ceph-mon[91579]: pgmap v14722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:55 smithi188 ceph-mon[92425]: pgmap v14722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:49:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:57 smithi161 ceph-mon[91579]: pgmap v14723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:57 smithi188 ceph-mon[92425]: pgmap v14723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:49:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:59 smithi161 ceph-mon[91579]: pgmap v14724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:49:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:49:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:49:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:49:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:59 smithi188 ceph-mon[92425]: pgmap v14724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:49:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:49:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:49:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:49:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:50:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:50:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T18:50:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:01 smithi161 ceph-mon[91579]: pgmap v14725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:01 smithi188 ceph-mon[92425]: pgmap v14725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:02 smithi161 ceph-mon[91579]: pgmap v14726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:02 smithi188 ceph-mon[92425]: pgmap v14726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:05 smithi161 ceph-mon[91579]: pgmap v14727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:05 smithi188 ceph-mon[92425]: pgmap v14727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:07.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:07 smithi161 ceph-mon[91579]: pgmap v14728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:07.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:07 smithi188 ceph-mon[92425]: pgmap v14728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:09 smithi161 ceph-mon[91579]: pgmap v14729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:09 smithi188 ceph-mon[92425]: pgmap v14729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:11 smithi161 ceph-mon[91579]: pgmap v14730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:11 smithi188 ceph-mon[92425]: pgmap v14730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:13 smithi161 ceph-mon[91579]: pgmap v14731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:13 smithi188 ceph-mon[92425]: pgmap v14731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:15 smithi161 ceph-mon[91579]: pgmap v14732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:15 smithi188 ceph-mon[92425]: pgmap v14732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:17 smithi161 ceph-mon[91579]: pgmap v14733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:17 smithi188 ceph-mon[92425]: pgmap v14733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:19 smithi161 ceph-mon[91579]: pgmap v14734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:19 smithi188 ceph-mon[92425]: pgmap v14734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-mon[91579]: pgmap v14735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:21.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:50:21.514+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:50:21.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:50:21.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:50:21.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:50:21.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:50:21.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:50:21.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:50:21.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:50:21.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:50:21.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:50:21.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:50:21.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:50:21.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:50:21.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:50:21.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:50:21.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:50:21.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:50:21.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:50:21.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:50:21.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:50:21.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:50:21.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:50:21.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:50:21.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:50:21.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:50:21.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:50:21.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:50:21.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:50:21.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:50:21.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:50:21.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:50:21.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:50:21.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:50:21.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:50:21.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:50:21.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:50:21.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:50:21.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:50:21.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:50:21.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:50:21.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:50:21.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:50:21.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:50:21.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:50:21.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:50:21.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=875] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:50:21.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=875] Local address: 172.21.15.161, port 59298 2022-07-03T18:50:21.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=875] Peer address: 172.21.15.101, port 22 2022-07-03T18:50:21.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=875] Connection lost 2022-07-03T18:50:21.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=875] Aborting connection 2022-07-03T18:50:21.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:50:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:21 smithi188 ceph-mon[92425]: pgmap v14735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:50:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:50:22.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:50:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:50:22.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:50:22.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:50:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:23 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:50:23.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:23 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:50:23.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:50:23.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:50:23.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:50:23.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:50:23.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:50:23.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:50:23.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:50:23.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:50:23.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:50:23.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:50:23.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:50:23.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:50:23.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:50:23.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:50:23.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:50:23.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:50:23.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:50:23.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:50:23.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:50:23.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:50:23.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:50:23.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:50:23.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:50:23.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:50:23.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:50:23.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:50:23.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:50:23.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:50:23.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:50:23.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:50:23.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:50:23.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:50:23.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:50:23.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:50:23.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:50:23.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:50:23.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:50:23.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:50:23.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:50:23.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:50:23.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:50:23.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:50:23.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:50:23.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:50:23.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=875] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:50:23.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=875] Local address: 172.21.15.161, port 59298 2022-07-03T18:50:23.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=875] Peer address: 172.21.15.101, port 22 2022-07-03T18:50:23.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=875] Connection lost 2022-07-03T18:50:23.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=875] Aborting connection 2022-07-03T18:50:23.652 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:23 smithi161 ceph-mon[91579]: pgmap v14736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:23 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:50:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:23 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:50:23.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:50:23.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:50:23.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:50:23.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:50:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:50:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:50:23.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:50:23.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:50:23.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:50:23.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:50:23.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:50:23.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:50:23.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:50:23.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:50:23.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:50:23.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=875] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:50:23.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=875] Local address: 172.21.15.161, port 59298 2022-07-03T18:50:23.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=875] Peer address: 172.21.15.101, port 22 2022-07-03T18:50:23.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=875] Connection lost 2022-07-03T18:50:23.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=875] Aborting connection 2022-07-03T18:50:23.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:23 smithi188 ceph-mon[92425]: pgmap v14736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: pgmap v14737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]': finished 2022-07-03T18:50:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: pgmap v14737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:50:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]': finished 2022-07-03T18:50:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T18:50:26.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:26 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 131.1M 2022-07-03T18:50:26.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:26 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 137571967: error parsing value: Value '137571967' is below minimum 939524096 2022-07-03T18:50:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:26 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 131.1M 2022-07-03T18:50:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:26 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 137571967: error parsing value: Value '137571967' is below minimum 939524096 2022-07-03T18:50:27.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:27 smithi188 ceph-mon[92425]: pgmap v14738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:27 smithi161 ceph-mon[91579]: pgmap v14738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:29 smithi161 ceph-mon[91579]: pgmap v14739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:29 smithi188 ceph-mon[92425]: pgmap v14739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:31 smithi161 ceph-mon[91579]: pgmap v14740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:31 smithi188 ceph-mon[92425]: pgmap v14740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:33 smithi161 ceph-mon[91579]: pgmap v14741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:33 smithi188 ceph-mon[92425]: pgmap v14741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:35 smithi161 ceph-mon[91579]: pgmap v14742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:35 smithi188 ceph-mon[92425]: pgmap v14742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:37 smithi161 ceph-mon[91579]: pgmap v14743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:37 smithi188 ceph-mon[92425]: pgmap v14743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:39 smithi161 ceph-mon[91579]: pgmap v14744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:39 smithi188 ceph-mon[92425]: pgmap v14744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:41 smithi161 ceph-mon[91579]: pgmap v14745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:41 smithi188 ceph-mon[92425]: pgmap v14745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:43 smithi161 ceph-mon[91579]: pgmap v14746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:43 smithi188 ceph-mon[92425]: pgmap v14746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:45 smithi161 ceph-mon[91579]: pgmap v14747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:45.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:45.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:45 smithi188 ceph-mon[92425]: pgmap v14747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:45.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:47 smithi161 ceph-mon[91579]: pgmap v14748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:47.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:47 smithi188 ceph-mon[92425]: pgmap v14748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:49 smithi161 ceph-mon[91579]: pgmap v14749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:49 smithi188 ceph-mon[92425]: pgmap v14749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:50.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:51 smithi161 ceph-mon[91579]: pgmap v14750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:51 smithi188 ceph-mon[92425]: pgmap v14750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:53 smithi161 ceph-mon[91579]: pgmap v14751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:53 smithi188 ceph-mon[92425]: pgmap v14751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:55 smithi188 ceph-mon[92425]: pgmap v14752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:55 smithi161 ceph-mon[91579]: pgmap v14752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:50:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:57 smithi188 ceph-mon[92425]: pgmap v14753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:57 smithi161 ceph-mon[91579]: pgmap v14753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:50:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:59 smithi188 ceph-mon[92425]: pgmap v14754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:50:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:50:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:50:59.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:50:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:59 smithi161 ceph-mon[91579]: pgmap v14754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:50:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:50:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:50:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:50:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:51:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:01 smithi188 ceph-mon[92425]: pgmap v14755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:01 smithi161 ceph-mon[91579]: pgmap v14755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:03.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:03 smithi188 ceph-mon[92425]: pgmap v14756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:03 smithi161 ceph-mon[91579]: pgmap v14756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:05 smithi188 ceph-mon[92425]: pgmap v14757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:05 smithi161 ceph-mon[91579]: pgmap v14757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:07 smithi188 ceph-mon[92425]: pgmap v14758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:07 smithi161 ceph-mon[91579]: pgmap v14758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:09 smithi188 ceph-mon[92425]: pgmap v14759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:09 smithi161 ceph-mon[91579]: pgmap v14759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:11.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:11 smithi161 ceph-mon[91579]: pgmap v14760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:11.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:11 smithi188 ceph-mon[92425]: pgmap v14760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:13 smithi161 ceph-mon[91579]: pgmap v14761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:13 smithi188 ceph-mon[92425]: pgmap v14761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:15 smithi161 ceph-mon[91579]: pgmap v14762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:15.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:15 smithi188 ceph-mon[92425]: pgmap v14762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:17 smithi161 ceph-mon[91579]: pgmap v14763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:17 smithi188 ceph-mon[92425]: pgmap v14763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:19 smithi161 ceph-mon[91579]: pgmap v14764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:19 smithi188 ceph-mon[92425]: pgmap v14764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:21 smithi161 ceph-mon[91579]: pgmap v14765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:21 smithi188 ceph-mon[92425]: pgmap v14765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:22 smithi161 ceph-mon[91579]: pgmap v14766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:22.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:22 smithi188 ceph-mon[92425]: pgmap v14766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:25.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:51:24.764+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:51:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:51:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:51:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:51:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:51:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:51:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:51:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:51:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:51:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:51:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:51:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:51:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:51:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:51:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:51:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:51:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:51:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:51:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:51:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:51:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:51:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:51:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:51:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:51:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:51:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:51:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:51:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:51:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:51:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:51:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:51:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:51:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:51:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:51:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:51:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:51:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:51:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:51:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:51:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:51:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:51:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:51:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:51:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:51:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:51:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=877] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:51:25.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=877] Local address: 172.21.15.161, port 59302 2022-07-03T18:51:25.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=877] Peer address: 172.21.15.101, port 22 2022-07-03T18:51:25.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=877] Connection lost 2022-07-03T18:51:25.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=877] Aborting connection 2022-07-03T18:51:25.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:51:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:25 smithi161 ceph-mon[91579]: pgmap v14767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:51:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:51:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:51:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:25 smithi188 ceph-mon[92425]: pgmap v14767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:51:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:51:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:51:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:51:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:51:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:51:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:51:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:51:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:51:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:51:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:51:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:51:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:51:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:51:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:51:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:51:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:51:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:51:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:51:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:51:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:51:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:51:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:51:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:51:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:51:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:51:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:51:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:51:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:51:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:51:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:51:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:51:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:51:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:51:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:51:26.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:51:26.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:51:26.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:51:26.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:51:26.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:51:26.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:51:26.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:51:26.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:51:26.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:51:26.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:51:26.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:51:26.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:51:26.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:51:26.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:51:26.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=877] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:51:26.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=877] Local address: 172.21.15.161, port 59302 2022-07-03T18:51:26.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=877] Peer address: 172.21.15.101, port 22 2022-07-03T18:51:26.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=877] Connection lost 2022-07-03T18:51:26.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=877] Aborting connection 2022-07-03T18:51:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:51:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:51:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:51:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:51:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:51:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:51:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:51:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:51:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:51:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:51:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:51:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:51:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:51:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:51:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:51:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:51:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:51:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:51:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:51:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:51:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:51:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:51:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:51:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:51:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:51:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:51:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:51:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:51:26.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:51:26.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:51:26.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:51:26.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:51:26.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:51:26.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:51:26.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:51:26.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:51:26.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:51:26.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:51:26.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:51:26.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:51:26.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:51:26.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:51:26.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:51:26.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:51:26.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:51:26.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:51:26.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=877] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:51:26.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=877] Local address: 172.21.15.161, port 59302 2022-07-03T18:51:26.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=877] Peer address: 172.21.15.101, port 22 2022-07-03T18:51:26.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=877] Connection lost 2022-07-03T18:51:26.694 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=877] Aborting connection 2022-07-03T18:51:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:27 smithi161 ceph-mon[91579]: pgmap v14768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:27 smithi188 ceph-mon[92425]: pgmap v14768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:29 smithi161 ceph-mon[91579]: pgmap v14769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:29 smithi188 ceph-mon[92425]: pgmap v14769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:31 smithi161 ceph-mon[91579]: pgmap v14770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:31 smithi188 ceph-mon[92425]: pgmap v14770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:33 smithi161 ceph-mon[91579]: pgmap v14771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:33 smithi188 ceph-mon[92425]: pgmap v14771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:35 smithi161 ceph-mon[91579]: pgmap v14772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:35 smithi188 ceph-mon[92425]: pgmap v14772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:37 smithi161 ceph-mon[91579]: pgmap v14773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:37 smithi188 ceph-mon[92425]: pgmap v14773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:39 smithi161 ceph-mon[91579]: pgmap v14774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:39 smithi188 ceph-mon[92425]: pgmap v14774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:40.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:41 smithi161 ceph-mon[91579]: pgmap v14775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:41 smithi188 ceph-mon[92425]: pgmap v14775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:43 smithi188 ceph-mon[92425]: pgmap v14776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:43 smithi161 ceph-mon[91579]: pgmap v14776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T18:51:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:45 smithi188 ceph-mon[92425]: pgmap v14777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:45 smithi161 ceph-mon[91579]: pgmap v14777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:47 smithi188 ceph-mon[92425]: pgmap v14778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:47 smithi161 ceph-mon[91579]: pgmap v14778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:49.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:49 smithi188 ceph-mon[92425]: pgmap v14779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:49 smithi161 ceph-mon[91579]: pgmap v14779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:51 smithi188 ceph-mon[92425]: pgmap v14780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:51 smithi161 ceph-mon[91579]: pgmap v14780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:53 smithi188 ceph-mon[92425]: pgmap v14781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:53 smithi161 ceph-mon[91579]: pgmap v14781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:55 smithi188 ceph-mon[92425]: pgmap v14782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:55 smithi161 ceph-mon[91579]: pgmap v14782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:51:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:57 smithi188 ceph-mon[92425]: pgmap v14783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:57 smithi161 ceph-mon[91579]: pgmap v14783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:51:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:59 smithi188 ceph-mon[92425]: pgmap v14784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:51:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:51:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:51:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:51:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:59 smithi161 ceph-mon[91579]: pgmap v14784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:51:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:51:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:51:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:51:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:52:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:01 smithi161 ceph-mon[91579]: pgmap v14785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:01 smithi188 ceph-mon[92425]: pgmap v14785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:03 smithi161 ceph-mon[91579]: pgmap v14786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:03 smithi188 ceph-mon[92425]: pgmap v14786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:05.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:05 smithi161 ceph-mon[91579]: pgmap v14787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:05.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:05 smithi188 ceph-mon[92425]: pgmap v14787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:05.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:07 smithi161 ceph-mon[91579]: pgmap v14788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:07 smithi188 ceph-mon[92425]: pgmap v14788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:09 smithi161 ceph-mon[91579]: pgmap v14789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:09 smithi188 ceph-mon[92425]: pgmap v14789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:11 smithi161 ceph-mon[91579]: pgmap v14790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:11 smithi188 ceph-mon[92425]: pgmap v14790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:13 smithi161 ceph-mon[91579]: pgmap v14791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:13 smithi188 ceph-mon[92425]: pgmap v14791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:14 smithi161 ceph-mon[91579]: pgmap v14792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:14 smithi188 ceph-mon[92425]: pgmap v14792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:16 smithi161 ceph-mon[91579]: pgmap v14793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:16 smithi188 ceph-mon[92425]: pgmap v14793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:19 smithi161 ceph-mon[91579]: pgmap v14794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:19 smithi188 ceph-mon[92425]: pgmap v14794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:20.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:21 smithi161 ceph-mon[91579]: pgmap v14795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:21 smithi188 ceph-mon[92425]: pgmap v14795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:23 smithi161 ceph-mon[91579]: pgmap v14796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:23 smithi188 ceph-mon[92425]: pgmap v14796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-mon[91579]: pgmap v14797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:52:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:52:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:52:25.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:52:25.262+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:52:25.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:52:25.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:52:25.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:52:25.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:52:25.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:52:25.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:52:25.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:52:25.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:52:25.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:52:25.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:52:25.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:52:25.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:52:25.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:52:25.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:52:25.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:52:25.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:52:25.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:52:25.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:52:25.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:52:25.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:52:25.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:52:25.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:52:25.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:52:25.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:52:25.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:52:25.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:52:25.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:52:25.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:52:25.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:52:25.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:52:25.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:52:25.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:52:25.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:52:25.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:52:25.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:52:25.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:52:25.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:52:25.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:52:25.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:52:25.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:52:25.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:52:25.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:52:25.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:52:25.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:52:25.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=879] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:52:25.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=879] Local address: 172.21.15.161, port 59306 2022-07-03T18:52:25.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=879] Peer address: 172.21.15.101, port 22 2022-07-03T18:52:25.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=879] Connection lost 2022-07-03T18:52:25.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=879] Aborting connection 2022-07-03T18:52:25.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:52:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:25 smithi188 ceph-mon[92425]: pgmap v14797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:52:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:52:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:52:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:52:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:52:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:52:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:52:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:52:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:52:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:52:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:52:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:52:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:52:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:52:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:52:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:52:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=879] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:52:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=879] Local address: 172.21.15.161, port 59306 2022-07-03T18:52:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=879] Peer address: 172.21.15.101, port 22 2022-07-03T18:52:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=879] Connection lost 2022-07-03T18:52:26.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=879] Aborting connection 2022-07-03T18:52:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:52:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:52:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:52:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:52:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:52:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:52:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:52:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:52:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:52:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:52:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:52:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:52:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:52:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:52:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:52:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:52:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:52:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:52:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:52:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:52:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:52:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:52:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:52:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:52:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:52:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:52:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:52:26.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:52:26.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:52:26.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:52:26.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:52:26.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:52:26.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:52:26.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:52:26.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:52:26.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:52:26.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:52:26.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:52:26.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:52:26.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:52:26.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:52:26.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:52:26.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:52:26.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=879] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:52:26.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=879] Local address: 172.21.15.161, port 59306 2022-07-03T18:52:26.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=879] Peer address: 172.21.15.101, port 22 2022-07-03T18:52:26.692 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=879] Connection lost 2022-07-03T18:52:26.693 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=879] Aborting connection 2022-07-03T18:52:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:27 smithi161 ceph-mon[91579]: pgmap v14798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:27 smithi188 ceph-mon[92425]: pgmap v14798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:29 smithi161 ceph-mon[91579]: pgmap v14799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:29 smithi188 ceph-mon[92425]: pgmap v14799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:31 smithi188 ceph-mon[92425]: pgmap v14800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:31 smithi161 ceph-mon[91579]: pgmap v14800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:33 smithi188 ceph-mon[92425]: pgmap v14801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:33 smithi161 ceph-mon[91579]: pgmap v14801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:35 smithi188 ceph-mon[92425]: pgmap v14802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:35 smithi161 ceph-mon[91579]: pgmap v14802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:37 smithi188 ceph-mon[92425]: pgmap v14803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:37 smithi161 ceph-mon[91579]: pgmap v14803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:39 smithi188 ceph-mon[92425]: pgmap v14804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:39 smithi161 ceph-mon[91579]: pgmap v14804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:41 smithi188 ceph-mon[92425]: pgmap v14805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:41 smithi161 ceph-mon[91579]: pgmap v14805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:43 smithi188 ceph-mon[92425]: pgmap v14806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:43 smithi161 ceph-mon[91579]: pgmap v14806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:45 smithi188 ceph-mon[92425]: pgmap v14807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:45 smithi161 ceph-mon[91579]: pgmap v14807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:47 smithi188 ceph-mon[92425]: pgmap v14808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:47 smithi161 ceph-mon[91579]: pgmap v14808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:49 smithi161 ceph-mon[91579]: pgmap v14809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:49 smithi188 ceph-mon[92425]: pgmap v14809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:51 smithi161 ceph-mon[91579]: pgmap v14810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:51 smithi188 ceph-mon[92425]: pgmap v14810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:53 smithi161 ceph-mon[91579]: pgmap v14811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:53 smithi188 ceph-mon[92425]: pgmap v14811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:55 smithi161 ceph-mon[91579]: pgmap v14812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:55 smithi188 ceph-mon[92425]: pgmap v14812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:52:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:57 smithi161 ceph-mon[91579]: pgmap v14813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:57 smithi188 ceph-mon[92425]: pgmap v14813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:59 smithi161 ceph-mon[91579]: pgmap v14814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:52:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:52:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:52:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:52:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:59 smithi188 ceph-mon[92425]: pgmap v14814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:52:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:52:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:52:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:52:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:53:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:01 smithi161 ceph-mon[91579]: pgmap v14815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:01 smithi188 ceph-mon[92425]: pgmap v14815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:03 smithi161 ceph-mon[91579]: pgmap v14816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:03 smithi188 ceph-mon[92425]: pgmap v14816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:04 smithi161 ceph-mon[91579]: pgmap v14817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:04 smithi188 ceph-mon[92425]: pgmap v14817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:06 smithi161 ceph-mon[91579]: pgmap v14818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:06 smithi188 ceph-mon[92425]: pgmap v14818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:09 smithi161 ceph-mon[91579]: pgmap v14819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:09 smithi188 ceph-mon[92425]: pgmap v14819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:11 smithi161 ceph-mon[91579]: pgmap v14820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:11 smithi188 ceph-mon[92425]: pgmap v14820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:13 smithi161 ceph-mon[91579]: pgmap v14821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:13 smithi188 ceph-mon[92425]: pgmap v14821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:15 smithi161 ceph-mon[91579]: pgmap v14822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:15.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:15 smithi188 ceph-mon[92425]: pgmap v14822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:17 smithi161 ceph-mon[91579]: pgmap v14823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:17 smithi188 ceph-mon[92425]: pgmap v14823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:19 smithi188 ceph-mon[92425]: pgmap v14824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:19 smithi161 ceph-mon[91579]: pgmap v14824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:20.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:21 smithi188 ceph-mon[92425]: pgmap v14825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:21 smithi161 ceph-mon[91579]: pgmap v14825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:23 smithi188 ceph-mon[92425]: pgmap v14826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:23 smithi161 ceph-mon[91579]: pgmap v14826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:25 smithi188 ceph-mon[92425]: pgmap v14827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:25.768 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-mon[91579]: pgmap v14827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:25.768 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:26.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:53:25.766+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:53:26.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:53:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:53:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:53:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:53:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:53:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:53:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:53:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:53:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:53:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:53:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:53:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:53:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:53:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:53:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:53:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:53:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:53:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:53:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:53:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:53:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:53:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:53:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:53:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:53:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:53:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:53:26.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:53:26.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:53:26.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:53:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:53:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:53:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:53:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:53:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:53:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:53:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:53:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:53:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:53:26.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:53:26.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:53:26.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:53:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:53:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:53:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:53:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=881] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:53:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=881] Local address: 172.21.15.161, port 59310 2022-07-03T18:53:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=881] Peer address: 172.21.15.101, port 22 2022-07-03T18:53:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=881] Connection lost 2022-07-03T18:53:26.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=881] Aborting connection 2022-07-03T18:53:26.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:53:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:53:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:53:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:53:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:53:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:53:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:53:26.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:53:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:53:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:53:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:53:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:53:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:53:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:53:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:53:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:53:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:53:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:53:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:53:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:53:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=881] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=881] Local address: 172.21.15.161, port 59310 2022-07-03T18:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=881] Peer address: 172.21.15.101, port 22 2022-07-03T18:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=881] Connection lost 2022-07-03T18:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=881] Aborting connection 2022-07-03T18:53:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:27 smithi188 ceph-mon[92425]: pgmap v14828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:53:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:53:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:53:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:53:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:53:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:53:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:53:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:53:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:53:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:53:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:53:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:53:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:53:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:53:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:53:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:53:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:53:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:53:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:53:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:53:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:53:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:53:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:53:27.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:53:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:53:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:53:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:53:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:53:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:53:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:53:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:53:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:53:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:53:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:53:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:53:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:53:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:53:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:53:27.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:53:27.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:53:27.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:53:27.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:53:27.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:53:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:53:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:53:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:53:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=881] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:53:27.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=881] Local address: 172.21.15.161, port 59310 2022-07-03T18:53:27.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=881] Peer address: 172.21.15.101, port 22 2022-07-03T18:53:27.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=881] Connection lost 2022-07-03T18:53:27.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=881] Aborting connection 2022-07-03T18:53:27.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:27 smithi161 ceph-mon[91579]: pgmap v14828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:29 smithi188 ceph-mon[92425]: pgmap v14829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:29 smithi161 ceph-mon[91579]: pgmap v14829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:31 smithi188 ceph-mon[92425]: pgmap v14830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:31 smithi161 ceph-mon[91579]: pgmap v14830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:33 smithi161 ceph-mon[91579]: pgmap v14831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:33 smithi188 ceph-mon[92425]: pgmap v14831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:35 smithi161 ceph-mon[91579]: pgmap v14832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:35 smithi188 ceph-mon[92425]: pgmap v14832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:37 smithi161 ceph-mon[91579]: pgmap v14833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:37 smithi188 ceph-mon[92425]: pgmap v14833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:39 smithi161 ceph-mon[91579]: pgmap v14834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:39 smithi188 ceph-mon[92425]: pgmap v14834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:41 smithi161 ceph-mon[91579]: pgmap v14835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:41 smithi188 ceph-mon[92425]: pgmap v14835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:43 smithi161 ceph-mon[91579]: pgmap v14836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:43 smithi188 ceph-mon[92425]: pgmap v14836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:45 smithi161 ceph-mon[91579]: pgmap v14837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:45 smithi188 ceph-mon[92425]: pgmap v14837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:47 smithi161 ceph-mon[91579]: pgmap v14838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:47 smithi188 ceph-mon[92425]: pgmap v14838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:49 smithi161 ceph-mon[91579]: pgmap v14839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:49 smithi188 ceph-mon[92425]: pgmap v14839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:51.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:51 smithi161 ceph-mon[91579]: pgmap v14840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:51 smithi188 ceph-mon[92425]: pgmap v14840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:53 smithi161 ceph-mon[91579]: pgmap v14841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:53 smithi188 ceph-mon[92425]: pgmap v14841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:55 smithi161 ceph-mon[91579]: pgmap v14842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:55 smithi188 ceph-mon[92425]: pgmap v14842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:53:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:57 smithi161 ceph-mon[91579]: pgmap v14843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:57 smithi188 ceph-mon[92425]: pgmap v14843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:59 smithi161 ceph-mon[91579]: pgmap v14844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:53:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:53:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:53:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:59 smithi188 ceph-mon[92425]: pgmap v14844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:53:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:53:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:53:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:53:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:54:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:01 smithi161 ceph-mon[91579]: pgmap v14845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:01 smithi188 ceph-mon[92425]: pgmap v14845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:02 smithi161 ceph-mon[91579]: pgmap v14846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:02 smithi188 ceph-mon[92425]: pgmap v14846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:05 smithi161 ceph-mon[91579]: pgmap v14847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:05 smithi188 ceph-mon[92425]: pgmap v14847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:07 smithi188 ceph-mon[92425]: pgmap v14848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:07 smithi161 ceph-mon[91579]: pgmap v14848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:09 smithi188 ceph-mon[92425]: pgmap v14849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:09 smithi161 ceph-mon[91579]: pgmap v14849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:10.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:11 smithi188 ceph-mon[92425]: pgmap v14850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:11 smithi161 ceph-mon[91579]: pgmap v14850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:13 smithi188 ceph-mon[92425]: pgmap v14851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:13 smithi161 ceph-mon[91579]: pgmap v14851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:15 smithi188 ceph-mon[92425]: pgmap v14852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:15.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:15 smithi161 ceph-mon[91579]: pgmap v14852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:17 smithi188 ceph-mon[92425]: pgmap v14853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:17 smithi161 ceph-mon[91579]: pgmap v14853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:19 smithi188 ceph-mon[92425]: pgmap v14854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:19 smithi161 ceph-mon[91579]: pgmap v14854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:21 smithi188 ceph-mon[92425]: pgmap v14855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:21 smithi161 ceph-mon[91579]: pgmap v14855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:23 smithi161 ceph-mon[91579]: pgmap v14856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:23 smithi188 ceph-mon[92425]: pgmap v14856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:25 smithi161 ceph-mon[91579]: pgmap v14857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:25 smithi188 ceph-mon[92425]: pgmap v14857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:54:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:54:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:54:26.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:54:26.264+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:54:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:54:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:54:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:54:26.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:54:26.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:54:26.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:54:26.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:54:26.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:54:26.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:54:26.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:54:26.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:54:26.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:54:26.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:54:26.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:54:26.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:54:26.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:54:26.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:54:26.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:54:26.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:54:26.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:54:26.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:54:26.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:54:26.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:54:26.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:54:26.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:54:26.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:54:26.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:54:26.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:54:26.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:54:26.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:54:26.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:54:26.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:54:26.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:54:26.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:54:26.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:54:26.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:54:26.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:54:26.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:54:26.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:54:26.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:54:26.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:54:26.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:54:26.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:54:26.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:54:26.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=883] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:54:26.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=883] Local address: 172.21.15.161, port 59338 2022-07-03T18:54:26.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=883] Peer address: 172.21.15.101, port 22 2022-07-03T18:54:26.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=883] Connection lost 2022-07-03T18:54:26.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=883] Aborting connection 2022-07-03T18:54:26.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:54:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:54:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:54:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:54:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:27 smithi161 ceph-mon[91579]: pgmap v14858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:54:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:54:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:54:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:54:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:54:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:54:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:54:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:54:27.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:54:27.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:54:27.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:54:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:54:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:54:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:54:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:54:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:54:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:54:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:54:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:54:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=883] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=883] Local address: 172.21.15.161, port 59338 2022-07-03T18:54:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=883] Peer address: 172.21.15.101, port 22 2022-07-03T18:54:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=883] Connection lost 2022-07-03T18:54:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=883] Aborting connection 2022-07-03T18:54:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:27 smithi188 ceph-mon[92425]: pgmap v14858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:54:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:54:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:54:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:54:27.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:54:27.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:54:27.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:54:27.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:54:27.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:54:27.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:54:27.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:54:27.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:54:27.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:54:27.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=883] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:54:27.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=883] Local address: 172.21.15.161, port 59338 2022-07-03T18:54:27.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=883] Peer address: 172.21.15.101, port 22 2022-07-03T18:54:27.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=883] Connection lost 2022-07-03T18:54:27.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=883] Aborting connection 2022-07-03T18:54:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:29 smithi161 ceph-mon[91579]: pgmap v14859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:29 smithi188 ceph-mon[92425]: pgmap v14859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:31 smithi161 ceph-mon[91579]: pgmap v14860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:31 smithi188 ceph-mon[92425]: pgmap v14860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:33 smithi161 ceph-mon[91579]: pgmap v14861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:33 smithi188 ceph-mon[92425]: pgmap v14861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:35 smithi161 ceph-mon[91579]: pgmap v14862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:35 smithi188 ceph-mon[92425]: pgmap v14862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:37 smithi161 ceph-mon[91579]: pgmap v14863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:37 smithi188 ceph-mon[92425]: pgmap v14863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:39 smithi161 ceph-mon[91579]: pgmap v14864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:39 smithi188 ceph-mon[92425]: pgmap v14864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:41 smithi161 ceph-mon[91579]: pgmap v14865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:41 smithi188 ceph-mon[92425]: pgmap v14865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:43 smithi161 ceph-mon[91579]: pgmap v14866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:43 smithi188 ceph-mon[92425]: pgmap v14866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:45 smithi161 ceph-mon[91579]: pgmap v14867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:45 smithi188 ceph-mon[92425]: pgmap v14867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:47 smithi161 ceph-mon[91579]: pgmap v14868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:47 smithi188 ceph-mon[92425]: pgmap v14868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:49 smithi161 ceph-mon[91579]: pgmap v14869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:49 smithi188 ceph-mon[92425]: pgmap v14869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:51 smithi161 ceph-mon[91579]: pgmap v14870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:51 smithi188 ceph-mon[92425]: pgmap v14870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:52 smithi161 ceph-mon[91579]: pgmap v14871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:52 smithi188 ceph-mon[92425]: pgmap v14871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:55 smithi188 ceph-mon[92425]: pgmap v14872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:55 smithi161 ceph-mon[91579]: pgmap v14872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:54:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:57 smithi188 ceph-mon[92425]: pgmap v14873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:57 smithi161 ceph-mon[91579]: pgmap v14873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:59 smithi188 ceph-mon[92425]: pgmap v14874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:54:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:54:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:54:59.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:54:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:59 smithi161 ceph-mon[91579]: pgmap v14874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:54:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:54:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:54:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:54:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:55:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:01 smithi188 ceph-mon[92425]: pgmap v14875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:01 smithi161 ceph-mon[91579]: pgmap v14875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:03 smithi188 ceph-mon[92425]: pgmap v14876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:03 smithi161 ceph-mon[91579]: pgmap v14876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:05 smithi188 ceph-mon[92425]: pgmap v14877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:05 smithi161 ceph-mon[91579]: pgmap v14877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:07 smithi188 ceph-mon[92425]: pgmap v14878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:07 smithi161 ceph-mon[91579]: pgmap v14878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:09 smithi161 ceph-mon[91579]: pgmap v14879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:09 smithi188 ceph-mon[92425]: pgmap v14879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:11 smithi161 ceph-mon[91579]: pgmap v14880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:11 smithi188 ceph-mon[92425]: pgmap v14880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:13 smithi161 ceph-mon[91579]: pgmap v14881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:13 smithi188 ceph-mon[92425]: pgmap v14881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:15 smithi161 ceph-mon[91579]: pgmap v14882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:15 smithi188 ceph-mon[92425]: pgmap v14882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:17 smithi161 ceph-mon[91579]: pgmap v14883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:17 smithi188 ceph-mon[92425]: pgmap v14883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:19 smithi161 ceph-mon[91579]: pgmap v14884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:19 smithi188 ceph-mon[92425]: pgmap v14884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:21 smithi161 ceph-mon[91579]: pgmap v14885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:21 smithi188 ceph-mon[92425]: pgmap v14885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:23 smithi161 ceph-mon[91579]: pgmap v14886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:23 smithi188 ceph-mon[92425]: pgmap v14886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:25 smithi161 ceph-mon[91579]: pgmap v14887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:25 smithi188 ceph-mon[92425]: pgmap v14887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:27.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:55:26.765+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:55:27.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:55:27.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:55:27.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:55:27.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:55:27.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:55:27.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:55:27.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:55:27.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:55:27.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:55:27.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:55:27.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:55:27.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:55:27.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:55:27.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:55:27.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:55:27.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:55:27.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:55:27.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:55:27.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:55:27.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:55:27.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:55:27.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:55:27.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:55:27.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:55:27.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:55:27.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:55:27.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:55:27.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:55:27.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:55:27.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:55:27.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:55:27.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:55:27.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:55:27.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:55:27.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:55:27.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:55:27.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:55:27.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:55:27.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:55:27.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:55:27.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:55:27.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:55:27.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:55:27.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:55:27.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=885] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:55:27.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=885] Local address: 172.21.15.161, port 59342 2022-07-03T18:55:27.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=885] Peer address: 172.21.15.101, port 22 2022-07-03T18:55:27.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=885] Connection lost 2022-07-03T18:55:27.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=885] Aborting connection 2022-07-03T18:55:27.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:55:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:55:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:27 smithi161 ceph-mon[91579]: pgmap v14888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:55:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:55:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:55:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:27 smithi188 ceph-mon[92425]: pgmap v14888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:55:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:55:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:55:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:55:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:55:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:55:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:55:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:55:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:55:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:55:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:55:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:55:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:55:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:55:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:55:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:55:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:55:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:55:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:55:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:55:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:55:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:55:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:55:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:55:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:55:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:55:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:55:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:55:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:55:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:55:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:55:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=885] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:55:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=885] Local address: 172.21.15.161, port 59342 2022-07-03T18:55:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=885] Peer address: 172.21.15.101, port 22 2022-07-03T18:55:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=885] Connection lost 2022-07-03T18:55:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=885] Aborting connection 2022-07-03T18:55:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:55:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:55:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:55:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:55:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:55:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:55:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:55:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:55:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:55:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:55:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:55:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:55:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:55:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:55:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:55:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:55:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=885] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:55:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=885] Local address: 172.21.15.161, port 59342 2022-07-03T18:55:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=885] Peer address: 172.21.15.101, port 22 2022-07-03T18:55:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=885] Connection lost 2022-07-03T18:55:28.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=885] Aborting connection 2022-07-03T18:55:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:29 smithi161 ceph-mon[91579]: pgmap v14889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:29 smithi188 ceph-mon[92425]: pgmap v14889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:31 smithi161 ceph-mon[91579]: pgmap v14890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:31 smithi188 ceph-mon[92425]: pgmap v14890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:33 smithi161 ceph-mon[91579]: pgmap v14891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:33 smithi188 ceph-mon[92425]: pgmap v14891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:35 smithi161 ceph-mon[91579]: pgmap v14892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:35 smithi188 ceph-mon[92425]: pgmap v14892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:37 smithi161 ceph-mon[91579]: pgmap v14893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:37 smithi188 ceph-mon[92425]: pgmap v14893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:39 smithi161 ceph-mon[91579]: pgmap v14894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:39 smithi188 ceph-mon[92425]: pgmap v14894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:41 smithi161 ceph-mon[91579]: pgmap v14895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:41 smithi188 ceph-mon[92425]: pgmap v14895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:42 smithi161 ceph-mon[91579]: pgmap v14896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:42 smithi188 ceph-mon[92425]: pgmap v14896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:45 smithi188 ceph-mon[92425]: pgmap v14897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:45.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:45 smithi161 ceph-mon[91579]: pgmap v14897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:47 smithi188 ceph-mon[92425]: pgmap v14898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:47 smithi161 ceph-mon[91579]: pgmap v14898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:49 smithi188 ceph-mon[92425]: pgmap v14899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:49 smithi161 ceph-mon[91579]: pgmap v14899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:51 smithi161 ceph-mon[91579]: pgmap v14900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:51 smithi188 ceph-mon[92425]: pgmap v14900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:53 smithi161 ceph-mon[91579]: pgmap v14901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:53 smithi188 ceph-mon[92425]: pgmap v14901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:55 smithi161 ceph-mon[91579]: pgmap v14902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:55 smithi188 ceph-mon[92425]: pgmap v14902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:55:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:57 smithi161 ceph-mon[91579]: pgmap v14903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:57 smithi188 ceph-mon[92425]: pgmap v14903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:55:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:59 smithi161 ceph-mon[91579]: pgmap v14904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:55:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:55:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:55:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:55:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:59 smithi188 ceph-mon[92425]: pgmap v14904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:55:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:55:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:55:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:55:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:56:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:01 smithi161 ceph-mon[91579]: pgmap v14905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:01 smithi188 ceph-mon[92425]: pgmap v14905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:03 smithi161 ceph-mon[91579]: pgmap v14906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:03 smithi188 ceph-mon[92425]: pgmap v14906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:05 smithi161 ceph-mon[91579]: pgmap v14907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:05 smithi188 ceph-mon[92425]: pgmap v14907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:07 smithi161 ceph-mon[91579]: pgmap v14908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:07 smithi188 ceph-mon[92425]: pgmap v14908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:09 smithi161 ceph-mon[91579]: pgmap v14909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:09 smithi188 ceph-mon[92425]: pgmap v14909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:11 smithi161 ceph-mon[91579]: pgmap v14910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:11 smithi188 ceph-mon[92425]: pgmap v14910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:13 smithi161 ceph-mon[91579]: pgmap v14911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:13 smithi188 ceph-mon[92425]: pgmap v14911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:15 smithi161 ceph-mon[91579]: pgmap v14912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:15 smithi188 ceph-mon[92425]: pgmap v14912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:17 smithi161 ceph-mon[91579]: pgmap v14913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:17 smithi188 ceph-mon[92425]: pgmap v14913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:19 smithi161 ceph-mon[91579]: pgmap v14914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:19 smithi188 ceph-mon[92425]: pgmap v14914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:21 smithi161 ceph-mon[91579]: pgmap v14915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:21 smithi188 ceph-mon[92425]: pgmap v14915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:23 smithi161 ceph-mon[91579]: pgmap v14916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:23 smithi188 ceph-mon[92425]: pgmap v14916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:25 smithi161 ceph-mon[91579]: pgmap v14917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:25 smithi188 ceph-mon[92425]: pgmap v14917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:26 smithi161 ceph-mon[91579]: pgmap v14918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:26 smithi188 ceph-mon[92425]: pgmap v14918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:56:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:56:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:56:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:56:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:56:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:56:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:56:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:56:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:56:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:56:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:56:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:56:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:56:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:56:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:56:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:56:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:56:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:56:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:56:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:56:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:56:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:56:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:56:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:56:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:56:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:56:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:56:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:56:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:56:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:56:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:56:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:56:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:56:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:56:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:56:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:56:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:56:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:56:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:56:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:56:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:56:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:56:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:56:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:56:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:56:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:56:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:56:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:56:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:56:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=887] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:56:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=887] Local address: 172.21.15.161, port 59346 2022-07-03T18:56:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=887] Peer address: 172.21.15.101, port 22 2022-07-03T18:56:27.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=887] Connection lost 2022-07-03T18:56:27.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=887] Aborting connection 2022-07-03T18:56:27.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:56:27.265+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:56:27.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:56:27.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:56:27.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:56:27.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:56:27.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:56:27.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:56:27.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:56:27.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:56:27.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:56:27.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:56:27.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:56:27.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:56:27.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:56:27.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:56:27.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:56:27.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:56:27.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:56:27.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:56:27.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:56:27.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:56:27.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:56:27.656 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:56:27.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:56:27.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:56:27.657 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:56:27.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:56:27.658 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:56:27.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:56:27.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:56:27.659 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:56:27.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:56:27.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:56:27.660 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:56:27.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:56:27.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:56:27.661 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:56:27.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:56:27.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:56:27.662 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:56:27.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:56:27.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:56:27.663 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:56:27.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:56:27.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:56:27.664 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=887] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:56:27.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=887] Local address: 172.21.15.161, port 59346 2022-07-03T18:56:27.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=887] Peer address: 172.21.15.101, port 22 2022-07-03T18:56:27.665 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=887] Connection lost 2022-07-03T18:56:27.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=887] Aborting connection 2022-07-03T18:56:27.666 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:56:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:56:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:56:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:56:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:56:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:56:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:56:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:56:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:56:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:56:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:56:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:56:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:56:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:56:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:56:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:56:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:56:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:56:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:56:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:56:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:56:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:56:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:56:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:56:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:56:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:56:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:56:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:56:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:56:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:56:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:56:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:56:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:56:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:56:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:56:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:56:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:56:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:56:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:56:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:56:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:56:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:56:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:56:27.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:56:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:56:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:56:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:56:27.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:56:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:56:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:56:27.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=887] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:56:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=887] Local address: 172.21.15.161, port 59346 2022-07-03T18:56:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=887] Peer address: 172.21.15.101, port 22 2022-07-03T18:56:27.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=887] Connection lost 2022-07-03T18:56:27.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=887] Aborting connection 2022-07-03T18:56:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:29 smithi188 ceph-mon[92425]: pgmap v14919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:29 smithi161 ceph-mon[91579]: pgmap v14919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:31 smithi161 ceph-mon[91579]: pgmap v14920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:31 smithi188 ceph-mon[92425]: pgmap v14920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:33 smithi161 ceph-mon[91579]: pgmap v14921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:33 smithi188 ceph-mon[92425]: pgmap v14921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:35 smithi161 ceph-mon[91579]: pgmap v14922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:35 smithi188 ceph-mon[92425]: pgmap v14922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:37 smithi161 ceph-mon[91579]: pgmap v14923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:37 smithi188 ceph-mon[92425]: pgmap v14923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:39 smithi161 ceph-mon[91579]: pgmap v14924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:39 smithi188 ceph-mon[92425]: pgmap v14924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:41 smithi161 ceph-mon[91579]: pgmap v14925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:41 smithi188 ceph-mon[92425]: pgmap v14925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:43 smithi161 ceph-mon[91579]: pgmap v14926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:43 smithi188 ceph-mon[92425]: pgmap v14926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:45 smithi161 ceph-mon[91579]: pgmap v14927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:45 smithi188 ceph-mon[92425]: pgmap v14927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:47 smithi161 ceph-mon[91579]: pgmap v14928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:47 smithi188 ceph-mon[92425]: pgmap v14928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:49 smithi161 ceph-mon[91579]: pgmap v14929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:49 smithi188 ceph-mon[92425]: pgmap v14929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:51 smithi161 ceph-mon[91579]: pgmap v14930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:51 smithi188 ceph-mon[92425]: pgmap v14930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:53 smithi161 ceph-mon[91579]: pgmap v14931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:53 smithi188 ceph-mon[92425]: pgmap v14931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:55 smithi161 ceph-mon[91579]: pgmap v14932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:55 smithi188 ceph-mon[92425]: pgmap v14932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:56:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:57 smithi161 ceph-mon[91579]: pgmap v14933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:57 smithi188 ceph-mon[92425]: pgmap v14933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:56:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:59 smithi161 ceph-mon[91579]: pgmap v14934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:56:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:56:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:56:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:56:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:59 smithi188 ceph-mon[92425]: pgmap v14934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:56:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:56:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:56:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:56:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:57:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:01 smithi161 ceph-mon[91579]: pgmap v14935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:01 smithi188 ceph-mon[92425]: pgmap v14935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:03 smithi161 ceph-mon[91579]: pgmap v14936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:03 smithi188 ceph-mon[92425]: pgmap v14936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:05 smithi161 ceph-mon[91579]: pgmap v14937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:05 smithi188 ceph-mon[92425]: pgmap v14937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:07 smithi161 ceph-mon[91579]: pgmap v14938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:07 smithi188 ceph-mon[92425]: pgmap v14938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:09 smithi161 ceph-mon[91579]: pgmap v14939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:09 smithi188 ceph-mon[92425]: pgmap v14939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:10 smithi161 ceph-mon[91579]: pgmap v14940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:10.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:10 smithi188 ceph-mon[92425]: pgmap v14940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:12 smithi188 ceph-mon[92425]: pgmap v14941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:12 smithi161 ceph-mon[91579]: pgmap v14941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:15 smithi161 ceph-mon[91579]: pgmap v14942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:15 smithi188 ceph-mon[92425]: pgmap v14942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:17 smithi161 ceph-mon[91579]: pgmap v14943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:17 smithi188 ceph-mon[92425]: pgmap v14943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:19 smithi161 ceph-mon[91579]: pgmap v14944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:19 smithi188 ceph-mon[92425]: pgmap v14944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:21 smithi161 ceph-mon[91579]: pgmap v14945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:21 smithi188 ceph-mon[92425]: pgmap v14945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:23 smithi161 ceph-mon[91579]: pgmap v14946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:23 smithi188 ceph-mon[92425]: pgmap v14946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:25 smithi161 ceph-mon[91579]: pgmap v14947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:25 smithi188 ceph-mon[92425]: pgmap v14947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:27.768 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-mon[91579]: pgmap v14948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:27.776 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:27 smithi188 ceph-mon[92425]: pgmap v14948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:28.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:57:27.767+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:57:28.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:57:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:57:28.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:57:28.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:57:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:57:28.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:57:28.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:57:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:57:28.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:57:28.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:57:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:57:28.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:57:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:57:28.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:57:28.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:57:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:57:28.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:57:28.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:57:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:57:28.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:57:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:57:28.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:57:28.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:57:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:57:28.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:57:28.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:57:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:57:28.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:57:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:57:28.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:57:28.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:57:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:57:28.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:57:28.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:57:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:57:28.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:57:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:57:28.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:57:28.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:57:28.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:57:28.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:57:28.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:57:28.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:57:28.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:57:28.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=889] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:57:28.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=889] Local address: 172.21.15.161, port 59350 2022-07-03T18:57:28.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=889] Peer address: 172.21.15.101, port 22 2022-07-03T18:57:28.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=889] Connection lost 2022-07-03T18:57:28.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=889] Aborting connection 2022-07-03T18:57:28.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:57:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:57:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:57:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:57:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:57:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:57:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:57:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:57:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T18:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:57:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:57:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:57:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:57:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=889] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:57:29.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=889] Local address: 172.21.15.161, port 59350 2022-07-03T18:57:29.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=889] Peer address: 172.21.15.101, port 22 2022-07-03T18:57:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=889] Connection lost 2022-07-03T18:57:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=889] Aborting connection 2022-07-03T18:57:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:29 smithi161 ceph-mon[91579]: pgmap v14949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:57:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:57:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:57:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=889] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:57:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=889] Local address: 172.21.15.161, port 59350 2022-07-03T18:57:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=889] Peer address: 172.21.15.101, port 22 2022-07-03T18:57:29.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=889] Connection lost 2022-07-03T18:57:29.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=889] Aborting connection 2022-07-03T18:57:29.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:29 smithi188 ceph-mon[92425]: pgmap v14949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:31 smithi161 ceph-mon[91579]: pgmap v14950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:31 smithi188 ceph-mon[92425]: pgmap v14950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:33 smithi161 ceph-mon[91579]: pgmap v14951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:33 smithi188 ceph-mon[92425]: pgmap v14951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:35 smithi161 ceph-mon[91579]: pgmap v14952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:35 smithi188 ceph-mon[92425]: pgmap v14952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:37 smithi161 ceph-mon[91579]: pgmap v14953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:37 smithi188 ceph-mon[92425]: pgmap v14953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:39 smithi161 ceph-mon[91579]: pgmap v14954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:39 smithi188 ceph-mon[92425]: pgmap v14954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:41 smithi161 ceph-mon[91579]: pgmap v14955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:41 smithi188 ceph-mon[92425]: pgmap v14955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:43 smithi161 ceph-mon[91579]: pgmap v14956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:43 smithi188 ceph-mon[92425]: pgmap v14956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:45 smithi161 ceph-mon[91579]: pgmap v14957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:45 smithi188 ceph-mon[92425]: pgmap v14957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:47 smithi161 ceph-mon[91579]: pgmap v14958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:47 smithi188 ceph-mon[92425]: pgmap v14958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:49 smithi161 ceph-mon[91579]: pgmap v14959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:49 smithi188 ceph-mon[92425]: pgmap v14959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:51 smithi161 ceph-mon[91579]: pgmap v14960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:51 smithi188 ceph-mon[92425]: pgmap v14960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:53 smithi161 ceph-mon[91579]: pgmap v14961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:53 smithi188 ceph-mon[92425]: pgmap v14961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:54 smithi161 ceph-mon[91579]: pgmap v14962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:54 smithi188 ceph-mon[92425]: pgmap v14962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:57:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:56 smithi161 ceph-mon[91579]: pgmap v14963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:56.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:56 smithi188 ceph-mon[92425]: pgmap v14963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:57:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:59 smithi161 ceph-mon[91579]: pgmap v14964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:57:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:57:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:57:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:57:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:59 smithi188 ceph-mon[92425]: pgmap v14964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:57:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:57:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:57:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:57:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:58:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:01 smithi161 ceph-mon[91579]: pgmap v14965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:01 smithi188 ceph-mon[92425]: pgmap v14965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:03 smithi161 ceph-mon[91579]: pgmap v14966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:03 smithi188 ceph-mon[92425]: pgmap v14966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:05 smithi161 ceph-mon[91579]: pgmap v14967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:05 smithi188 ceph-mon[92425]: pgmap v14967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:07 smithi161 ceph-mon[91579]: pgmap v14968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:07 smithi188 ceph-mon[92425]: pgmap v14968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:09 smithi161 ceph-mon[91579]: pgmap v14969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:09 smithi188 ceph-mon[92425]: pgmap v14969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:11 smithi161 ceph-mon[91579]: pgmap v14970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:11 smithi188 ceph-mon[92425]: pgmap v14970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:13 smithi161 ceph-mon[91579]: pgmap v14971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:13 smithi188 ceph-mon[92425]: pgmap v14971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:15 smithi161 ceph-mon[91579]: pgmap v14972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:15 smithi188 ceph-mon[92425]: pgmap v14972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:17 smithi161 ceph-mon[91579]: pgmap v14973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:17 smithi188 ceph-mon[92425]: pgmap v14973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:19 smithi161 ceph-mon[91579]: pgmap v14974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:19 smithi188 ceph-mon[92425]: pgmap v14974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:21 smithi161 ceph-mon[91579]: pgmap v14975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:21 smithi188 ceph-mon[92425]: pgmap v14975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:23 smithi161 ceph-mon[91579]: pgmap v14976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:23 smithi188 ceph-mon[92425]: pgmap v14976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:25 smithi161 ceph-mon[91579]: pgmap v14977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:25 smithi188 ceph-mon[92425]: pgmap v14977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:27 smithi161 ceph-mon[91579]: pgmap v14978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:27 smithi188 ceph-mon[92425]: pgmap v14978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:29.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:58:28.695+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:58:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:58:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:58:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:58:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:58:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:58:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:58:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:58:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:58:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=891] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:58:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=891] Local address: 172.21.15.161, port 59354 2022-07-03T18:58:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=891] Peer address: 172.21.15.101, port 22 2022-07-03T18:58:29.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=891] Connection lost 2022-07-03T18:58:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=891] Aborting connection 2022-07-03T18:58:29.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:58:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:29 smithi161 ceph-mon[91579]: pgmap v14979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:58:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:58:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:58:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:29 smithi188 ceph-mon[92425]: pgmap v14979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:58:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:58:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:58:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:58:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:58:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:58:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:58:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:58:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:58:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:58:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:58:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:58:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:58:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:58:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:58:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:58:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:58:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:58:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:58:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:58:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:58:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:58:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:58:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:58:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:58:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:58:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:58:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:58:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:58:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:58:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:58:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:58:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:58:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:58:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:58:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:58:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:58:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:58:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:58:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:58:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:58:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:58:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:58:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:58:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:58:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:58:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:58:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:58:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:58:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=891] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:58:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=891] Local address: 172.21.15.161, port 59354 2022-07-03T18:58:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=891] Peer address: 172.21.15.101, port 22 2022-07-03T18:58:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=891] Connection lost 2022-07-03T18:58:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=891] Aborting connection 2022-07-03T18:58:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=891] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=891] Local address: 172.21.15.161, port 59354 2022-07-03T18:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=891] Peer address: 172.21.15.101, port 22 2022-07-03T18:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=891] Connection lost 2022-07-03T18:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=891] Aborting connection 2022-07-03T18:58:30.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:31 smithi161 ceph-mon[91579]: pgmap v14980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:31 smithi188 ceph-mon[92425]: pgmap v14980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:33 smithi161 ceph-mon[91579]: pgmap v14981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:33 smithi188 ceph-mon[92425]: pgmap v14981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:35 smithi161 ceph-mon[91579]: pgmap v14982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:35 smithi188 ceph-mon[92425]: pgmap v14982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:37 smithi161 ceph-mon[91579]: pgmap v14983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:37 smithi188 ceph-mon[92425]: pgmap v14983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:39 smithi161 ceph-mon[91579]: pgmap v14984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:39 smithi188 ceph-mon[92425]: pgmap v14984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:40 smithi161 ceph-mon[91579]: pgmap v14985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:40 smithi188 ceph-mon[92425]: pgmap v14985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:42 smithi161 ceph-mon[91579]: pgmap v14986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:42 smithi188 ceph-mon[92425]: pgmap v14986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:45 smithi161 ceph-mon[91579]: pgmap v14987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:45 smithi188 ceph-mon[92425]: pgmap v14987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:47 smithi161 ceph-mon[91579]: pgmap v14988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:47 smithi188 ceph-mon[92425]: pgmap v14988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:49 smithi161 ceph-mon[91579]: pgmap v14989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:49 smithi188 ceph-mon[92425]: pgmap v14989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:51 smithi161 ceph-mon[91579]: pgmap v14990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:51 smithi188 ceph-mon[92425]: pgmap v14990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:53 smithi161 ceph-mon[91579]: pgmap v14991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:53 smithi188 ceph-mon[92425]: pgmap v14991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:55 smithi161 ceph-mon[91579]: pgmap v14992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:55 smithi188 ceph-mon[92425]: pgmap v14992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:58:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:57 smithi161 ceph-mon[91579]: pgmap v14993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:57 smithi188 ceph-mon[92425]: pgmap v14993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:58:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:59 smithi161 ceph-mon[91579]: pgmap v14994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:58:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:58:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:58:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:58:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:59 smithi188 ceph-mon[92425]: pgmap v14994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:58:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:58:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:58:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:58:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:59:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:01 smithi161 ceph-mon[91579]: pgmap v14995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:01 smithi188 ceph-mon[92425]: pgmap v14995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:03 smithi161 ceph-mon[91579]: pgmap v14996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:03 smithi188 ceph-mon[92425]: pgmap v14996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:05 smithi161 ceph-mon[91579]: pgmap v14997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:05 smithi188 ceph-mon[92425]: pgmap v14997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:07 smithi161 ceph-mon[91579]: pgmap v14998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:07 smithi188 ceph-mon[92425]: pgmap v14998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:09 smithi161 ceph-mon[91579]: pgmap v14999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:09 smithi188 ceph-mon[92425]: pgmap v14999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:11 smithi161 ceph-mon[91579]: pgmap v15000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:11 smithi188 ceph-mon[92425]: pgmap v15000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:13 smithi161 ceph-mon[91579]: pgmap v15001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:13 smithi188 ceph-mon[92425]: pgmap v15001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:15 smithi161 ceph-mon[91579]: pgmap v15002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:15 smithi188 ceph-mon[92425]: pgmap v15002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:17 smithi161 ceph-mon[91579]: pgmap v15003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:17 smithi188 ceph-mon[92425]: pgmap v15003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:19 smithi161 ceph-mon[91579]: pgmap v15004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:19 smithi188 ceph-mon[92425]: pgmap v15004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:20 smithi161 ceph-mon[91579]: pgmap v15005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:20.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:20 smithi188 ceph-mon[92425]: pgmap v15005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:22 smithi161 ceph-mon[91579]: pgmap v15006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:22 smithi188 ceph-mon[92425]: pgmap v15006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:25 smithi161 ceph-mon[91579]: pgmap v15007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:25 smithi188 ceph-mon[92425]: pgmap v15007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:27 smithi161 ceph-mon[91579]: pgmap v15008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:27 smithi188 ceph-mon[92425]: pgmap v15008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:29.464 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:59:29.182+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:59:29.464 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:59:29.465 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:59:29.465 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T18:59:29.465 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:59:29.466 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:59:29.466 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:59:29.467 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T18:59:29.467 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:59:29.467 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T18:59:29.468 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:59:29.468 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T18:59:29.468 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:59:29.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:59:29.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T18:59:29.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:59:29.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T18:59:29.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:59:29.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T18:59:29.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:59:29.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T18:59:29.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:59:29.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:59:29.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:59:29.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:59:29.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:59:29.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T18:59:29.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:59:29.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:59:29.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:59:29.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T18:59:29.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:59:29.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T18:59:29.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:59:29.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:59:29.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:59:29.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T18:59:29.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:59:29.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T18:59:29.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:59:29.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T18:59:29.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:59:29.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T18:59:29.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:59:29.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:59:29.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=893] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:59:29.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=893] Local address: 172.21.15.161, port 59358 2022-07-03T18:59:29.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=893] Peer address: 172.21.15.101, port 22 2022-07-03T18:59:29.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=893] Connection lost 2022-07-03T18:59:29.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=893] Aborting connection 2022-07-03T18:59:29.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T18:59:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-mon[91579]: pgmap v15009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:59:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:59:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:59:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:29 smithi188 ceph-mon[92425]: pgmap v15009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T18:59:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T18:59:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T18:59:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:59:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:59:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:59:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:59:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T18:59:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:59:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:59:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:59:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T18:59:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:59:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T18:59:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:59:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T18:59:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:59:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:59:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:59:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T18:59:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T18:59:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:59:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T18:59:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:59:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T18:59:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:59:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:59:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:59:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:59:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:59:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:59:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:59:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:59:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:59:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T18:59:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:59:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T18:59:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:59:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:59:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:59:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:59:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:59:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T18:59:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:59:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:59:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:59:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T18:59:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:59:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:59:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=893] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:59:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=893] Local address: 172.21.15.161, port 59358 2022-07-03T18:59:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=893] Peer address: 172.21.15.101, port 22 2022-07-03T18:59:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=893] Connection lost 2022-07-03T18:59:30.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=893] Aborting connection 2022-07-03T18:59:30.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T18:59:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T18:59:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:59:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T18:59:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T18:59:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T18:59:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T18:59:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T18:59:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T18:59:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T18:59:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T18:59:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T18:59:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T18:59:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T18:59:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:59:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T18:59:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T18:59:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T18:59:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T18:59:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T18:59:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T18:59:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T18:59:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T18:59:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T18:59:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T18:59:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T18:59:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T18:59:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T18:59:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T18:59:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T18:59:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T18:59:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T18:59:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T18:59:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T18:59:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T18:59:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T18:59:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T18:59:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T18:59:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T18:59:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T18:59:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T18:59:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T18:59:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T18:59:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T18:59:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T18:59:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T18:59:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=893] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T18:59:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=893] Local address: 172.21.15.161, port 59358 2022-07-03T18:59:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=893] Peer address: 172.21.15.101, port 22 2022-07-03T18:59:30.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=893] Connection lost 2022-07-03T18:59:30.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=893] Aborting connection 2022-07-03T18:59:30.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:31 smithi161 ceph-mon[91579]: pgmap v15010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:31 smithi188 ceph-mon[92425]: pgmap v15010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:33 smithi161 ceph-mon[91579]: pgmap v15011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:33 smithi188 ceph-mon[92425]: pgmap v15011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:35 smithi161 ceph-mon[91579]: pgmap v15012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:35 smithi188 ceph-mon[92425]: pgmap v15012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:37 smithi161 ceph-mon[91579]: pgmap v15013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:37 smithi188 ceph-mon[92425]: pgmap v15013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:39 smithi161 ceph-mon[91579]: pgmap v15014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:39 smithi188 ceph-mon[92425]: pgmap v15014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:41 smithi161 ceph-mon[91579]: pgmap v15015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:41 smithi188 ceph-mon[92425]: pgmap v15015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:43 smithi161 ceph-mon[91579]: pgmap v15016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:43 smithi188 ceph-mon[92425]: pgmap v15016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:45 smithi161 ceph-mon[91579]: pgmap v15017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:45 smithi188 ceph-mon[92425]: pgmap v15017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:47 smithi161 ceph-mon[91579]: pgmap v15018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:47 smithi188 ceph-mon[92425]: pgmap v15018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:49 smithi161 ceph-mon[91579]: pgmap v15019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:49 smithi188 ceph-mon[92425]: pgmap v15019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:51 smithi161 ceph-mon[91579]: pgmap v15020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:51 smithi188 ceph-mon[92425]: pgmap v15020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:53 smithi161 ceph-mon[91579]: pgmap v15021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:53 smithi188 ceph-mon[92425]: pgmap v15021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:55 smithi161 ceph-mon[91579]: pgmap v15022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:55 smithi188 ceph-mon[92425]: pgmap v15022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T18:59:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:56 smithi161 ceph-mon[91579]: pgmap v15023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:56 smithi188 ceph-mon[92425]: pgmap v15023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T18:59:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:59 smithi161 ceph-mon[91579]: pgmap v15024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:59:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:59:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:59:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 18:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:59:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:59 smithi188 ceph-mon[92425]: pgmap v15024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T18:59:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:59:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T18:59:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T18:59:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 18:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:00:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:00:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:00:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:01 smithi161 ceph-mon[91579]: pgmap v15025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:01 smithi188 ceph-mon[92425]: pgmap v15025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:03 smithi161 ceph-mon[91579]: pgmap v15026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:03 smithi188 ceph-mon[92425]: pgmap v15026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:05 smithi161 ceph-mon[91579]: pgmap v15027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:05 smithi188 ceph-mon[92425]: pgmap v15027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:07 smithi161 ceph-mon[91579]: pgmap v15028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:07 smithi188 ceph-mon[92425]: pgmap v15028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:09 smithi161 ceph-mon[91579]: pgmap v15029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:09.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:09 smithi188 ceph-mon[92425]: pgmap v15029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:11 smithi161 ceph-mon[91579]: pgmap v15030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:11 smithi188 ceph-mon[92425]: pgmap v15030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:13 smithi161 ceph-mon[91579]: pgmap v15031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:13 smithi188 ceph-mon[92425]: pgmap v15031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:15 smithi161 ceph-mon[91579]: pgmap v15032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:15 smithi188 ceph-mon[92425]: pgmap v15032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:17 smithi161 ceph-mon[91579]: pgmap v15033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:17 smithi188 ceph-mon[92425]: pgmap v15033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:19 smithi161 ceph-mon[91579]: pgmap v15034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:19 smithi188 ceph-mon[92425]: pgmap v15034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:21 smithi161 ceph-mon[91579]: pgmap v15035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:21 smithi188 ceph-mon[92425]: pgmap v15035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:23 smithi161 ceph-mon[91579]: pgmap v15036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:23 smithi188 ceph-mon[92425]: pgmap v15036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:25 smithi161 ceph-mon[91579]: pgmap v15037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:25.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:25 smithi188 ceph-mon[92425]: pgmap v15037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:27 smithi161 ceph-mon[91579]: pgmap v15038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:27 smithi188 ceph-mon[92425]: pgmap v15038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-mon[91579]: pgmap v15039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:00:29.691+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:00:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:00:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:00:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:00:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:00:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:00:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:00:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:00:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:00:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:00:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:00:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:00:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:00:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:00:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:00:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:00:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:00:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:00:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:00:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:00:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:00:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:00:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:00:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:00:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:00:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:00:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:00:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:00:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:00:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:00:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:00:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:00:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:00:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:00:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:00:29.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:00:29.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:00:29.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:00:29.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:00:29.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:00:29.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:00:29.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:00:29.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:00:29.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:00:29.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:00:29.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=895] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:00:29.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=895] Local address: 172.21.15.161, port 59362 2022-07-03T19:00:29.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=895] Peer address: 172.21.15.101, port 22 2022-07-03T19:00:29.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=895] Connection lost 2022-07-03T19:00:29.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=895] Aborting connection 2022-07-03T19:00:29.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:00:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:00:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:29 smithi188 ceph-mon[92425]: pgmap v15039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:00:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:00:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:00:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:00:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:00:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:00:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:00:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:00:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:00:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:00:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:00:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:00:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:00:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:00:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:00:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:00:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:00:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:00:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:00:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:00:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:00:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:00:31.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:00:31.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:00:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:00:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:00:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:00:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:00:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:00:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:00:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:00:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:00:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:00:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:00:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:00:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:00:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:00:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:00:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:00:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:00:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:00:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:00:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:00:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:00:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:00:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:00:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:00:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:00:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:00:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:00:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:00:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:00:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=895] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:00:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=895] Local address: 172.21.15.161, port 59362 2022-07-03T19:00:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=895] Peer address: 172.21.15.101, port 22 2022-07-03T19:00:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=895] Connection lost 2022-07-03T19:00:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=895] Aborting connection 2022-07-03T19:00:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:31 smithi161 ceph-mon[91579]: pgmap v15040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:00:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:00:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:00:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:00:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:00:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:00:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:00:31.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:00:31.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:00:31.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:00:31.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:00:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:00:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:00:31.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:00:31.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:00:31.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:00:31.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:00:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:00:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:00:31.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:00:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:00:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:00:31.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:00:31.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:00:31.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:00:31.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:00:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:00:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:00:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:00:31.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:00:31.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:00:31.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:00:31.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:00:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:00:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:00:31.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:00:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:00:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:00:31.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:00:31.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:00:31.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:00:31.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:00:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:00:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:00:31.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=895] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:00:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=895] Local address: 172.21.15.161, port 59362 2022-07-03T19:00:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=895] Peer address: 172.21.15.101, port 22 2022-07-03T19:00:31.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=895] Connection lost 2022-07-03T19:00:31.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=895] Aborting connection 2022-07-03T19:00:31.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:31 smithi188 ceph-mon[92425]: pgmap v15040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: pgmap v15041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: pgmap v15041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:00:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:00:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:34 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 125.3M 2022-07-03T19:00:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:34 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 131397951: error parsing value: Value '131397951' is below minimum 939524096 2022-07-03T19:00:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:34 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 125.3M 2022-07-03T19:00:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:34 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 131397951: error parsing value: Value '131397951' is below minimum 939524096 2022-07-03T19:00:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:35 smithi161 ceph-mon[91579]: pgmap v15042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:35 smithi188 ceph-mon[92425]: pgmap v15042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:37 smithi161 ceph-mon[91579]: pgmap v15043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:37 smithi188 ceph-mon[92425]: pgmap v15043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:39 smithi161 ceph-mon[91579]: pgmap v15044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:39 smithi188 ceph-mon[92425]: pgmap v15044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:41 smithi161 ceph-mon[91579]: pgmap v15045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:41 smithi188 ceph-mon[92425]: pgmap v15045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:43 smithi161 ceph-mon[91579]: pgmap v15046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:43 smithi188 ceph-mon[92425]: pgmap v15046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:45 smithi161 ceph-mon[91579]: pgmap v15047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:45 smithi188 ceph-mon[92425]: pgmap v15047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:47 smithi161 ceph-mon[91579]: pgmap v15048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:47 smithi188 ceph-mon[92425]: pgmap v15048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:49 smithi161 ceph-mon[91579]: pgmap v15049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:49 smithi188 ceph-mon[92425]: pgmap v15049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:51 smithi161 ceph-mon[91579]: pgmap v15050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:51 smithi188 ceph-mon[92425]: pgmap v15050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:53 smithi161 ceph-mon[91579]: pgmap v15051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:53 smithi188 ceph-mon[92425]: pgmap v15051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:55 smithi161 ceph-mon[91579]: pgmap v15052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:55 smithi188 ceph-mon[92425]: pgmap v15052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:00:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:57 smithi161 ceph-mon[91579]: pgmap v15053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:57 smithi188 ceph-mon[92425]: pgmap v15053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:00:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:59 smithi161 ceph-mon[91579]: pgmap v15054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:00:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:00:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:00:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:00:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:59 smithi188 ceph-mon[92425]: pgmap v15054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:00:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:00:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:00:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:00:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:01:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:01 smithi161 ceph-mon[91579]: pgmap v15055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:01 smithi188 ceph-mon[92425]: pgmap v15055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:02 smithi161 ceph-mon[91579]: pgmap v15056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:02 smithi188 ceph-mon[92425]: pgmap v15056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:05 smithi161 ceph-mon[91579]: pgmap v15057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:05 smithi188 ceph-mon[92425]: pgmap v15057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:07 smithi161 ceph-mon[91579]: pgmap v15058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:07 smithi188 ceph-mon[92425]: pgmap v15058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:09 smithi161 ceph-mon[91579]: pgmap v15059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:09 smithi188 ceph-mon[92425]: pgmap v15059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:11 smithi161 ceph-mon[91579]: pgmap v15060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:11 smithi188 ceph-mon[92425]: pgmap v15060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:13 smithi161 ceph-mon[91579]: pgmap v15061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:13 smithi188 ceph-mon[92425]: pgmap v15061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:15 smithi161 ceph-mon[91579]: pgmap v15062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:15 smithi188 ceph-mon[92425]: pgmap v15062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:17 smithi161 ceph-mon[91579]: pgmap v15063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:17 smithi188 ceph-mon[92425]: pgmap v15063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:19 smithi161 ceph-mon[91579]: pgmap v15064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:19 smithi188 ceph-mon[92425]: pgmap v15064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:21 smithi161 ceph-mon[91579]: pgmap v15065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:21 smithi188 ceph-mon[92425]: pgmap v15065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:23 smithi161 ceph-mon[91579]: pgmap v15066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:23 smithi188 ceph-mon[92425]: pgmap v15066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:01:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:25 smithi161 ceph-mon[91579]: pgmap v15067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:25 smithi188 ceph-mon[92425]: pgmap v15067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:25.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:27 smithi161 ceph-mon[91579]: pgmap v15068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:27 smithi188 ceph-mon[92425]: pgmap v15068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:29 smithi161 ceph-mon[91579]: pgmap v15069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:29 smithi188 ceph-mon[92425]: pgmap v15069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:31 smithi161 ceph-mon[91579]: pgmap v15070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:31 smithi188 ceph-mon[92425]: pgmap v15070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-mon[91579]: pgmap v15071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:32 smithi188 ceph-mon[92425]: pgmap v15071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:33.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:01:32.958+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:01:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:01:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:01:33.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:01:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:01:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:01:33.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:01:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:01:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:01:33.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:01:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:01:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:01:33.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:01:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:01:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:01:33.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:01:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:01:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:01:33.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:01:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:01:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:01:33.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:01:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:01:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:01:33.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:01:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:01:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:01:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:01:33.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:01:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:01:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:01:33.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:01:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:01:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:01:33.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:01:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:01:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:01:33.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:01:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:01:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:01:33.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:01:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:01:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:01:33.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:01:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:01:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=897] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:01:33.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=897] Local address: 172.21.15.161, port 59366 2022-07-03T19:01:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=897] Peer address: 172.21.15.101, port 22 2022-07-03T19:01:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=897] Connection lost 2022-07-03T19:01:33.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=897] Aborting connection 2022-07-03T19:01:33.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:01:32 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:01:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:01:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:01:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:01:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:01:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:01:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:01:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:01:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:01:33.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:01:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:01:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:01:33.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:01:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:01:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:01:33.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:01:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:01:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:01:33.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:01:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:01:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:01:33.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:01:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:01:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:01:33.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:01:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:01:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:01:33.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:01:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:01:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:01:33.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:01:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:01:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:01:33.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:01:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:01:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:01:33.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:01:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:01:33.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:01:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:01:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:01:33.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:01:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:01:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:01:33.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:01:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:01:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:01:33.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:01:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:01:33.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:01:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=897] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:01:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=897] Local address: 172.21.15.161, port 59366 2022-07-03T19:01:33.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=897] Peer address: 172.21.15.101, port 22 2022-07-03T19:01:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=897] Connection lost 2022-07-03T19:01:33.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=897] Aborting connection 2022-07-03T19:01:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:01:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:01:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:01:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:01:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:01:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:01:33.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:01:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:01:33.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:01:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:01:33.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:01:33.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:01:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:01:33.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:01:33.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:01:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:01:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:01:33.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:01:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:01:33.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:01:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:01:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:01:33.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:01:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:01:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:01:33.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:01:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:01:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:01:33.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:01:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:01:33.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:01:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:01:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:01:33.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:01:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:01:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:01:33.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:01:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:01:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:01:33.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:01:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:01:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:01:33.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:01:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:01:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:01:33.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:01:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:01:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:01:33.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:01:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=897] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:01:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=897] Local address: 172.21.15.161, port 59366 2022-07-03T19:01:33.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=897] Peer address: 172.21.15.101, port 22 2022-07-03T19:01:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=897] Connection lost 2022-07-03T19:01:33.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=897] Aborting connection 2022-07-03T19:01:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:34 smithi161 ceph-mon[91579]: pgmap v15072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:34 smithi188 ceph-mon[92425]: pgmap v15072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:36 smithi161 ceph-mon[91579]: pgmap v15073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:36 smithi188 ceph-mon[92425]: pgmap v15073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:39 smithi161 ceph-mon[91579]: pgmap v15074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:39 smithi188 ceph-mon[92425]: pgmap v15074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:41 smithi161 ceph-mon[91579]: pgmap v15075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:41 smithi188 ceph-mon[92425]: pgmap v15075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:43 smithi161 ceph-mon[91579]: pgmap v15076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:43 smithi188 ceph-mon[92425]: pgmap v15076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:45 smithi161 ceph-mon[91579]: pgmap v15077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:45 smithi188 ceph-mon[92425]: pgmap v15077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:47 smithi161 ceph-mon[91579]: pgmap v15078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:47 smithi188 ceph-mon[92425]: pgmap v15078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:49 smithi161 ceph-mon[91579]: pgmap v15079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:49 smithi188 ceph-mon[92425]: pgmap v15079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:51 smithi161 ceph-mon[91579]: pgmap v15080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:51 smithi188 ceph-mon[92425]: pgmap v15080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:53 smithi161 ceph-mon[91579]: pgmap v15081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:53 smithi188 ceph-mon[92425]: pgmap v15081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:55 smithi161 ceph-mon[91579]: pgmap v15082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:55 smithi188 ceph-mon[92425]: pgmap v15082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:01:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:57 smithi161 ceph-mon[91579]: pgmap v15083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:57 smithi188 ceph-mon[92425]: pgmap v15083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:01:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:59 smithi161 ceph-mon[91579]: pgmap v15084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:01:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:01:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:01:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:01:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:59 smithi188 ceph-mon[92425]: pgmap v15084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:01:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:01:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:01:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:01:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:02:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:01 smithi161 ceph-mon[91579]: pgmap v15085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:01 smithi188 ceph-mon[92425]: pgmap v15085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:02 smithi161 ceph-mon[91579]: pgmap v15086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:02 smithi188 ceph-mon[92425]: pgmap v15086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:05 smithi161 ceph-mon[91579]: pgmap v15087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:05 smithi188 ceph-mon[92425]: pgmap v15087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:07 smithi161 ceph-mon[91579]: pgmap v15088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:07 smithi188 ceph-mon[92425]: pgmap v15088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:09 smithi161 ceph-mon[91579]: pgmap v15089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:09 smithi188 ceph-mon[92425]: pgmap v15089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:11 smithi161 ceph-mon[91579]: pgmap v15090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:11 smithi188 ceph-mon[92425]: pgmap v15090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:13 smithi161 ceph-mon[91579]: pgmap v15091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:13 smithi188 ceph-mon[92425]: pgmap v15091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:15 smithi161 ceph-mon[91579]: pgmap v15092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:15 smithi188 ceph-mon[92425]: pgmap v15092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:15.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:17 smithi161 ceph-mon[91579]: pgmap v15093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:17 smithi188 ceph-mon[92425]: pgmap v15093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:19 smithi161 ceph-mon[91579]: pgmap v15094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:19 smithi188 ceph-mon[92425]: pgmap v15094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:21 smithi161 ceph-mon[91579]: pgmap v15095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:21 smithi188 ceph-mon[92425]: pgmap v15095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:22 smithi161 ceph-mon[91579]: pgmap v15096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:22 smithi188 ceph-mon[92425]: pgmap v15096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:24 smithi161 ceph-mon[91579]: pgmap v15097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:24 smithi188 ceph-mon[92425]: pgmap v15097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:26 smithi161 ceph-mon[91579]: pgmap v15098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:26 smithi188 ceph-mon[92425]: pgmap v15098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:29 smithi161 ceph-mon[91579]: pgmap v15099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:29 smithi188 ceph-mon[92425]: pgmap v15099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:31 smithi161 ceph-mon[91579]: pgmap v15100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:31 smithi188 ceph-mon[92425]: pgmap v15100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-mon[91579]: pgmap v15101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:02:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:02:33.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:02:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:02:33.462+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:02:33.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:02:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:02:33.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:02:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:02:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:02:33.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:02:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:02:33.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:02:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:02:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:02:33.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:02:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:02:33.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:02:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:02:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:02:33.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:02:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:02:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:02:33.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:02:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:02:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:02:33.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:02:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:02:33.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:02:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:02:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:02:33.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:02:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:02:33.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:02:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:02:33.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:02:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:02:33.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:02:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:02:33.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:02:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:02:33.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:02:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:02:33.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:02:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:02:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:02:33.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:02:33.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:02:33.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:02:33.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=899] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:02:33.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=899] Local address: 172.21.15.161, port 59378 2022-07-03T19:02:33.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=899] Peer address: 172.21.15.101, port 22 2022-07-03T19:02:33.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=899] Connection lost 2022-07-03T19:02:33.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=899] Aborting connection 2022-07-03T19:02:33.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:02:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:33 smithi188 ceph-mon[92425]: pgmap v15101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:02:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:02:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:02:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:34 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:02:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:34 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:02:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:02:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:02:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:02:34.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:02:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:02:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:02:34.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:02:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:02:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:02:34.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:02:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:02:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:02:34.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:02:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:02:34.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:02:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:02:34.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:02:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:02:34.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:02:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:02:34.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:02:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:02:34.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:02:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:02:34.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:02:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:02:34.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:02:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:02:34.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:02:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:02:34.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:02:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:02:34.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:02:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:02:34.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:02:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:02:34.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:02:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:02:34.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:02:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:02:34.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:02:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:02:34.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:02:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:02:34.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=899] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:02:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=899] Local address: 172.21.15.161, port 59378 2022-07-03T19:02:34.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=899] Peer address: 172.21.15.101, port 22 2022-07-03T19:02:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=899] Connection lost 2022-07-03T19:02:34.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=899] Aborting connection 2022-07-03T19:02:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:34 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:02:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:34 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:02:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:02:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:02:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:02:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:02:34.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:02:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:02:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:02:34.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:02:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:02:34.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:02:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:02:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:02:34.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:02:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:02:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:02:34.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:02:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:02:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:02:34.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:02:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:02:34.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:02:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:02:34.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:02:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:02:34.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:02:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:02:34.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:02:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:02:34.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:02:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:02:34.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:02:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:02:34.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:02:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:02:34.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:02:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:02:34.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:02:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:02:34.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:02:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:02:34.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:02:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:02:34.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:02:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:02:34.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=899] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:02:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=899] Local address: 172.21.15.161, port 59378 2022-07-03T19:02:34.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=899] Peer address: 172.21.15.101, port 22 2022-07-03T19:02:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=899] Connection lost 2022-07-03T19:02:34.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=899] Aborting connection 2022-07-03T19:02:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:35 smithi161 ceph-mon[91579]: pgmap v15102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:35 smithi188 ceph-mon[92425]: pgmap v15102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:37 smithi161 ceph-mon[91579]: pgmap v15103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:37 smithi188 ceph-mon[92425]: pgmap v15103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:39 smithi161 ceph-mon[91579]: pgmap v15104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:39 smithi188 ceph-mon[92425]: pgmap v15104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:41 smithi161 ceph-mon[91579]: pgmap v15105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:41 smithi188 ceph-mon[92425]: pgmap v15105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:43 smithi161 ceph-mon[91579]: pgmap v15106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:43 smithi188 ceph-mon[92425]: pgmap v15106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:43.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:45 smithi161 ceph-mon[91579]: pgmap v15107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:45 smithi188 ceph-mon[92425]: pgmap v15107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:47 smithi161 ceph-mon[91579]: pgmap v15108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:47 smithi188 ceph-mon[92425]: pgmap v15108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:49 smithi161 ceph-mon[91579]: pgmap v15109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:49 smithi188 ceph-mon[92425]: pgmap v15109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:50 smithi161 ceph-mon[91579]: pgmap v15110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:50 smithi188 ceph-mon[92425]: pgmap v15110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:50.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:52 smithi161 ceph-mon[91579]: pgmap v15111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:52 smithi188 ceph-mon[92425]: pgmap v15111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:55 smithi161 ceph-mon[91579]: pgmap v15112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:55 smithi188 ceph-mon[92425]: pgmap v15112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:02:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:57 smithi161 ceph-mon[91579]: pgmap v15113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:57 smithi188 ceph-mon[92425]: pgmap v15113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:02:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:02:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:59 smithi161 ceph-mon[91579]: pgmap v15114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:02:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:02:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:02:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:02:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:59 smithi188 ceph-mon[92425]: pgmap v15114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:02:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:02:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:02:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:02:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:03:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:01 smithi161 ceph-mon[91579]: pgmap v15115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:01 smithi188 ceph-mon[92425]: pgmap v15115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:03 smithi161 ceph-mon[91579]: pgmap v15116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:03 smithi188 ceph-mon[92425]: pgmap v15116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:05 smithi161 ceph-mon[91579]: pgmap v15117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:05 smithi188 ceph-mon[92425]: pgmap v15117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:07 smithi161 ceph-mon[91579]: pgmap v15118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:07 smithi188 ceph-mon[92425]: pgmap v15118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:09 smithi161 ceph-mon[91579]: pgmap v15119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:09 smithi188 ceph-mon[92425]: pgmap v15119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:11 smithi161 ceph-mon[91579]: pgmap v15120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:11 smithi188 ceph-mon[92425]: pgmap v15120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:12.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:12 smithi161 ceph-mon[91579]: pgmap v15121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:12.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:12 smithi188 ceph-mon[92425]: pgmap v15121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:14 smithi161 ceph-mon[91579]: pgmap v15122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:14 smithi188 ceph-mon[92425]: pgmap v15122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:16.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:16 smithi161 ceph-mon[91579]: pgmap v15123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:16 smithi188 ceph-mon[92425]: pgmap v15123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:19 smithi161 ceph-mon[91579]: pgmap v15124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:19 smithi188 ceph-mon[92425]: pgmap v15124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:21 smithi161 ceph-mon[91579]: pgmap v15125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:21 smithi188 ceph-mon[92425]: pgmap v15125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:23 smithi161 ceph-mon[91579]: pgmap v15126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:23 smithi188 ceph-mon[92425]: pgmap v15126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:25 smithi161 ceph-mon[91579]: pgmap v15127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:25 smithi188 ceph-mon[92425]: pgmap v15127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:27 smithi161 ceph-mon[91579]: pgmap v15128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:27 smithi188 ceph-mon[92425]: pgmap v15128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:29 smithi161 ceph-mon[91579]: pgmap v15129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:29 smithi188 ceph-mon[92425]: pgmap v15129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:31 smithi161 ceph-mon[91579]: pgmap v15130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:31 smithi188 ceph-mon[92425]: pgmap v15130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-mon[91579]: pgmap v15131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:33 smithi188 ceph-mon[92425]: pgmap v15131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:34.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:03:33.959+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:03:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:03:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:03:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:03:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:03:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:03:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:03:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:03:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:03:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:03:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:03:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:03:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:03:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:03:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:03:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:03:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:03:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:03:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:03:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:03:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:03:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:03:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:03:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:03:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:03:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:03:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:03:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:03:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:03:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:03:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:03:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:03:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:03:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:03:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:03:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:03:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:03:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:03:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:03:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:03:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:03:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:03:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:03:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:03:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:03:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=901] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:03:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=901] Local address: 172.21.15.161, port 59406 2022-07-03T19:03:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=901] Peer address: 172.21.15.101, port 22 2022-07-03T19:03:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=901] Connection lost 2022-07-03T19:03:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=901] Aborting connection 2022-07-03T19:03:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:03:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:03:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:03:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:03:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:03:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:03:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:03:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:03:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:03:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:35 smithi161 ceph-mon[91579]: pgmap v15132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:03:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:03:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:03:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:03:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:03:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:03:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:03:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:03:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:03:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:03:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:03:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:03:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:03:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:03:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:03:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:03:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:03:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:03:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:03:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:03:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:03:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:03:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:03:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:03:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:03:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:03:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:03:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:03:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:03:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:03:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:03:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:03:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:03:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:03:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:03:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:03:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:03:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:03:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:03:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:03:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:03:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:03:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:03:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:03:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:03:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=901] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:03:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=901] Local address: 172.21.15.161, port 59406 2022-07-03T19:03:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=901] Peer address: 172.21.15.101, port 22 2022-07-03T19:03:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=901] Connection lost 2022-07-03T19:03:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=901] Aborting connection 2022-07-03T19:03:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:03:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:35 smithi188 ceph-mon[92425]: pgmap v15132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:03:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:03:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:03:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:03:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:03:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:03:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:03:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:03:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:03:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:03:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:03:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:03:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:03:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:03:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:03:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:03:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:03:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:03:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:03:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:03:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:03:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:03:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:03:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:03:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:03:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:03:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:03:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:03:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:03:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:03:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:03:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:03:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:03:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:03:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:03:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:03:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:03:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:03:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:03:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:03:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:03:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:03:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:03:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:03:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:03:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=901] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:03:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=901] Local address: 172.21.15.161, port 59406 2022-07-03T19:03:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=901] Peer address: 172.21.15.101, port 22 2022-07-03T19:03:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=901] Connection lost 2022-07-03T19:03:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=901] Aborting connection 2022-07-03T19:03:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:37 smithi161 ceph-mon[91579]: pgmap v15133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:37 smithi188 ceph-mon[92425]: pgmap v15133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:39 smithi161 ceph-mon[91579]: pgmap v15134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:39 smithi188 ceph-mon[92425]: pgmap v15134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:40 smithi161 ceph-mon[91579]: pgmap v15135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:40 smithi188 ceph-mon[92425]: pgmap v15135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:43 smithi161 ceph-mon[91579]: pgmap v15136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:43 smithi188 ceph-mon[92425]: pgmap v15136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:43.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:45 smithi161 ceph-mon[91579]: pgmap v15137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:45 smithi188 ceph-mon[92425]: pgmap v15137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:47 smithi161 ceph-mon[91579]: pgmap v15138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:47 smithi188 ceph-mon[92425]: pgmap v15138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:49 smithi161 ceph-mon[91579]: pgmap v15139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:49 smithi188 ceph-mon[92425]: pgmap v15139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:51 smithi161 ceph-mon[91579]: pgmap v15140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:51 smithi188 ceph-mon[92425]: pgmap v15140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:53 smithi161 ceph-mon[91579]: pgmap v15141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:53.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:53 smithi188 ceph-mon[92425]: pgmap v15141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:55 smithi161 ceph-mon[91579]: pgmap v15142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:55 smithi188 ceph-mon[92425]: pgmap v15142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:03:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:57 smithi161 ceph-mon[91579]: pgmap v15143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:57 smithi188 ceph-mon[92425]: pgmap v15143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:03:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:03:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:59 smithi161 ceph-mon[91579]: pgmap v15144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:03:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:03:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:03:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:03:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:59 smithi188 ceph-mon[92425]: pgmap v15144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:03:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:03:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:03:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:03:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:04:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:00 smithi161 ceph-mon[91579]: pgmap v15145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:00 smithi188 ceph-mon[92425]: pgmap v15145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:03 smithi161 ceph-mon[91579]: pgmap v15146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:03 smithi188 ceph-mon[92425]: pgmap v15146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:05 smithi161 ceph-mon[91579]: pgmap v15147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:05.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:05 smithi188 ceph-mon[92425]: pgmap v15147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:07 smithi161 ceph-mon[91579]: pgmap v15148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:07 smithi188 ceph-mon[92425]: pgmap v15148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:09 smithi161 ceph-mon[91579]: pgmap v15149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:09 smithi188 ceph-mon[92425]: pgmap v15149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:11 smithi161 ceph-mon[91579]: pgmap v15150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:11 smithi188 ceph-mon[92425]: pgmap v15150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:13 smithi161 ceph-mon[91579]: pgmap v15151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:13 smithi188 ceph-mon[92425]: pgmap v15151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:15 smithi161 ceph-mon[91579]: pgmap v15152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:15 smithi188 ceph-mon[92425]: pgmap v15152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:17 smithi161 ceph-mon[91579]: pgmap v15153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:17 smithi188 ceph-mon[92425]: pgmap v15153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:19 smithi161 ceph-mon[91579]: pgmap v15154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:19 smithi188 ceph-mon[92425]: pgmap v15154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:21 smithi161 ceph-mon[91579]: pgmap v15155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:21 smithi188 ceph-mon[92425]: pgmap v15155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:22 smithi161 ceph-mon[91579]: pgmap v15156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:22 smithi188 ceph-mon[92425]: pgmap v15156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:24 smithi161 ceph-mon[91579]: pgmap v15157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:24 smithi188 ceph-mon[92425]: pgmap v15157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:26 smithi161 ceph-mon[91579]: pgmap v15158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:26 smithi188 ceph-mon[92425]: pgmap v15158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:29 smithi161 ceph-mon[91579]: pgmap v15159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:29 smithi188 ceph-mon[92425]: pgmap v15159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:31 smithi161 ceph-mon[91579]: pgmap v15160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:31 smithi188 ceph-mon[92425]: pgmap v15160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:33 smithi161 ceph-mon[91579]: pgmap v15161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:33 smithi188 ceph-mon[92425]: pgmap v15161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:04:34.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:04:34.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:04:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:04:34.458+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:04:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:04:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:04:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:04:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:04:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:04:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:04:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:04:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:04:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:04:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:04:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:04:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:04:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:04:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:04:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:04:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:04:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:04:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:04:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:04:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:04:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:04:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:04:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:04:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:04:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:04:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:04:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:04:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:04:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:04:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:04:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:04:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:04:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:04:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:04:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:04:34.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:04:34.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:04:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:04:34.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:04:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:04:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:04:34.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:04:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:04:34.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:04:34.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=903] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:04:34.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=903] Local address: 172.21.15.161, port 59434 2022-07-03T19:04:34.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=903] Peer address: 172.21.15.101, port 22 2022-07-03T19:04:34.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=903] Connection lost 2022-07-03T19:04:34.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=903] Aborting connection 2022-07-03T19:04:34.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:04:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:04:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:04:34.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:04:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:35 smithi161 ceph-mon[91579]: pgmap v15162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:04:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:04:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:04:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:04:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:04:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:04:35.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:04:35.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:04:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:04:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:04:35.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:04:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:04:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:04:35.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:04:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:04:35.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:04:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:04:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:04:35.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:04:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:04:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:04:35.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:04:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:04:35.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:04:35.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:04:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:04:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:04:35.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:04:35.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:04:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:04:35.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:04:35.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:04:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:04:35.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:04:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:04:35.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:04:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:04:35.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=903] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:04:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=903] Local address: 172.21.15.161, port 59434 2022-07-03T19:04:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=903] Peer address: 172.21.15.101, port 22 2022-07-03T19:04:35.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=903] Connection lost 2022-07-03T19:04:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=903] Aborting connection 2022-07-03T19:04:35.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:35 smithi188 ceph-mon[92425]: pgmap v15162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:04:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:04:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:04:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:04:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:04:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:04:35.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:04:35.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:04:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:04:35.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:04:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:04:35.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:04:35.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:04:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:04:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:04:35.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:04:35.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:04:35.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:04:35.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:04:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:04:35.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:04:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:04:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:04:35.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:04:35.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:04:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:04:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:04:35.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:04:35.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:04:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:04:35.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=903] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=903] Local address: 172.21.15.161, port 59434 2022-07-03T19:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=903] Peer address: 172.21.15.101, port 22 2022-07-03T19:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=903] Connection lost 2022-07-03T19:04:35.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=903] Aborting connection 2022-07-03T19:04:35.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:37 smithi161 ceph-mon[91579]: pgmap v15163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:37 smithi188 ceph-mon[92425]: pgmap v15163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:39 smithi161 ceph-mon[91579]: pgmap v15164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:39 smithi188 ceph-mon[92425]: pgmap v15164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:41 smithi161 ceph-mon[91579]: pgmap v15165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:41 smithi188 ceph-mon[92425]: pgmap v15165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:43 smithi161 ceph-mon[91579]: pgmap v15166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:43 smithi188 ceph-mon[92425]: pgmap v15166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:44 smithi161 ceph-mon[91579]: pgmap v15167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:44 smithi188 ceph-mon[92425]: pgmap v15167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:46 smithi161 ceph-mon[91579]: pgmap v15168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:46 smithi188 ceph-mon[92425]: pgmap v15168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:49 smithi161 ceph-mon[91579]: pgmap v15169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:49 smithi188 ceph-mon[92425]: pgmap v15169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:51 smithi161 ceph-mon[91579]: pgmap v15170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:51 smithi188 ceph-mon[92425]: pgmap v15170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:53 smithi161 ceph-mon[91579]: pgmap v15171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:53 smithi188 ceph-mon[92425]: pgmap v15171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:55 smithi161 ceph-mon[91579]: pgmap v15172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:55 smithi188 ceph-mon[92425]: pgmap v15172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:04:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:57 smithi161 ceph-mon[91579]: pgmap v15173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:57 smithi188 ceph-mon[92425]: pgmap v15173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:04:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:04:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:59 smithi161 ceph-mon[91579]: pgmap v15174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:04:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:59 smithi188 ceph-mon[92425]: pgmap v15174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:04:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:04:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:04:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:04:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:05:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:01 smithi161 ceph-mon[91579]: pgmap v15175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:01 smithi188 ceph-mon[92425]: pgmap v15175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:02 smithi161 ceph-mon[91579]: pgmap v15176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:02 smithi188 ceph-mon[92425]: pgmap v15176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:04 smithi161 ceph-mon[91579]: pgmap v15177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:04 smithi188 ceph-mon[92425]: pgmap v15177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:06 smithi161 ceph-mon[91579]: pgmap v15178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:06 smithi188 ceph-mon[92425]: pgmap v15178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:09 smithi161 ceph-mon[91579]: pgmap v15179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:09 smithi188 ceph-mon[92425]: pgmap v15179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:11 smithi161 ceph-mon[91579]: pgmap v15180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:11 smithi188 ceph-mon[92425]: pgmap v15180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:13 smithi161 ceph-mon[91579]: pgmap v15181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:13 smithi188 ceph-mon[92425]: pgmap v15181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:15 smithi161 ceph-mon[91579]: pgmap v15182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:15 smithi188 ceph-mon[92425]: pgmap v15182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:17 smithi161 ceph-mon[91579]: pgmap v15183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:17 smithi188 ceph-mon[92425]: pgmap v15183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:19 smithi161 ceph-mon[91579]: pgmap v15184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:19 smithi188 ceph-mon[92425]: pgmap v15184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:21 smithi161 ceph-mon[91579]: pgmap v15185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:21 smithi188 ceph-mon[92425]: pgmap v15185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:22 smithi161 ceph-mon[91579]: pgmap v15186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:22 smithi188 ceph-mon[92425]: pgmap v15186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:24 smithi161 ceph-mon[91579]: pgmap v15187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:24 smithi188 ceph-mon[92425]: pgmap v15187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:26 smithi161 ceph-mon[91579]: pgmap v15188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:26 smithi188 ceph-mon[92425]: pgmap v15188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:28.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:29 smithi161 ceph-mon[91579]: pgmap v15189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:29 smithi188 ceph-mon[92425]: pgmap v15189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:31 smithi161 ceph-mon[91579]: pgmap v15190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:31 smithi188 ceph-mon[92425]: pgmap v15190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:33 smithi161 ceph-mon[91579]: pgmap v15191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:33 smithi188 ceph-mon[92425]: pgmap v15191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:33.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:35.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:05:34.957+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:05:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:05:35.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:05:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:05:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:05:35.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:05:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:05:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:05:35.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:05:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:05:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:05:35.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:05:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:05:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:05:35.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:05:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:05:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:05:35.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:05:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:05:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:05:35.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:05:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:05:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:05:35.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:05:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:05:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:05:35.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:05:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:05:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:05:35.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:05:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:05:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:05:35.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:05:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:05:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:05:35.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:05:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:05:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:05:35.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:05:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:05:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:05:35.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:05:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:05:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:05:35.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:05:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=905] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:05:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=905] Local address: 172.21.15.161, port 59462 2022-07-03T19:05:35.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=905] Peer address: 172.21.15.101, port 22 2022-07-03T19:05:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=905] Connection lost 2022-07-03T19:05:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=905] Aborting connection 2022-07-03T19:05:35.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:05:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:05:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:35 smithi161 ceph-mon[91579]: pgmap v15192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:05:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:05:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:05:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:35 smithi188 ceph-mon[92425]: pgmap v15192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:05:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:05:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:05:35.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:05:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:05:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:05:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:05:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:05:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:05:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:05:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:05:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:05:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:05:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:05:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:05:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:05:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:05:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:05:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:05:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:05:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:05:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:05:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:05:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:05:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:05:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:05:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:05:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:05:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:05:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:05:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:05:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:05:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:05:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:05:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:05:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:05:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:05:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:05:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:05:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:05:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:05:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:05:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:05:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:05:36.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:05:36.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:05:36.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:05:36.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:05:36.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:05:36.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=905] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:05:36.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=905] Local address: 172.21.15.161, port 59462 2022-07-03T19:05:36.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=905] Peer address: 172.21.15.101, port 22 2022-07-03T19:05:36.907 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=905] Connection lost 2022-07-03T19:05:36.908 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=905] Aborting connection 2022-07-03T19:05:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:05:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:05:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:05:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:05:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:05:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:05:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:05:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:05:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:05:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:05:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:05:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:05:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:05:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:05:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:05:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:05:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:05:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:05:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:05:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:05:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:05:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:05:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:05:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:05:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:05:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:05:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:05:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:05:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:05:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:05:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:05:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:05:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:05:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:05:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:05:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:05:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:05:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:05:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:05:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:05:36.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:05:36.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:05:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:05:36.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:05:36.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:05:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:05:36.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=905] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:05:36.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=905] Local address: 172.21.15.161, port 59462 2022-07-03T19:05:36.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=905] Peer address: 172.21.15.101, port 22 2022-07-03T19:05:36.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=905] Connection lost 2022-07-03T19:05:36.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=905] Aborting connection 2022-07-03T19:05:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:37 smithi161 ceph-mon[91579]: pgmap v15193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:37 smithi188 ceph-mon[92425]: pgmap v15193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:39 smithi161 ceph-mon[91579]: pgmap v15194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:39 smithi188 ceph-mon[92425]: pgmap v15194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:41 smithi161 ceph-mon[91579]: pgmap v15195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:41 smithi188 ceph-mon[92425]: pgmap v15195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:42 smithi161 ceph-mon[91579]: pgmap v15196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:42 smithi188 ceph-mon[92425]: pgmap v15196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:44 smithi161 ceph-mon[91579]: pgmap v15197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:44 smithi188 ceph-mon[92425]: pgmap v15197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:46 smithi161 ceph-mon[91579]: pgmap v15198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:46 smithi188 ceph-mon[92425]: pgmap v15198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:49 smithi161 ceph-mon[91579]: pgmap v15199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:49 smithi188 ceph-mon[92425]: pgmap v15199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:51 smithi161 ceph-mon[91579]: pgmap v15200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:51 smithi188 ceph-mon[92425]: pgmap v15200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:53 smithi161 ceph-mon[91579]: pgmap v15201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:53 smithi188 ceph-mon[92425]: pgmap v15201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:55 smithi161 ceph-mon[91579]: pgmap v15202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:55 smithi188 ceph-mon[92425]: pgmap v15202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:05:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:57 smithi161 ceph-mon[91579]: pgmap v15203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:57 smithi188 ceph-mon[92425]: pgmap v15203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:05:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:05:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:59 smithi161 ceph-mon[91579]: pgmap v15204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:05:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:05:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:05:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:05:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:59 smithi188 ceph-mon[92425]: pgmap v15204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:05:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:05:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:05:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:05:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:06:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:01 smithi161 ceph-mon[91579]: pgmap v15205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:01 smithi188 ceph-mon[92425]: pgmap v15205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:02 smithi161 ceph-mon[91579]: pgmap v15206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:02 smithi188 ceph-mon[92425]: pgmap v15206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:04 smithi161 ceph-mon[91579]: pgmap v15207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:04 smithi188 ceph-mon[92425]: pgmap v15207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:06 smithi161 ceph-mon[91579]: pgmap v15208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:06 smithi188 ceph-mon[92425]: pgmap v15208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:09 smithi161 ceph-mon[91579]: pgmap v15209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:09 smithi188 ceph-mon[92425]: pgmap v15209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:11 smithi161 ceph-mon[91579]: pgmap v15210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:11 smithi188 ceph-mon[92425]: pgmap v15210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:13 smithi161 ceph-mon[91579]: pgmap v15211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:13 smithi188 ceph-mon[92425]: pgmap v15211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:15 smithi161 ceph-mon[91579]: pgmap v15212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:15 smithi188 ceph-mon[92425]: pgmap v15212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:17 smithi161 ceph-mon[91579]: pgmap v15213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:17 smithi188 ceph-mon[92425]: pgmap v15213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:18.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:19 smithi161 ceph-mon[91579]: pgmap v15214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:19 smithi188 ceph-mon[92425]: pgmap v15214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:21 smithi161 ceph-mon[91579]: pgmap v15215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:21 smithi188 ceph-mon[92425]: pgmap v15215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:22 smithi161 ceph-mon[91579]: pgmap v15216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:22 smithi188 ceph-mon[92425]: pgmap v15216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:24 smithi161 ceph-mon[91579]: pgmap v15217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:24 smithi188 ceph-mon[92425]: pgmap v15217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:26 smithi161 ceph-mon[91579]: pgmap v15218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:26 smithi188 ceph-mon[92425]: pgmap v15218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:29 smithi161 ceph-mon[91579]: pgmap v15219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:29 smithi188 ceph-mon[92425]: pgmap v15219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:31 smithi161 ceph-mon[91579]: pgmap v15220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:31 smithi188 ceph-mon[92425]: pgmap v15220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:33 smithi161 ceph-mon[91579]: pgmap v15221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:33 smithi188 ceph-mon[92425]: pgmap v15221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-mon[91579]: pgmap v15222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:06:35.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:06:35.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:06:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:06:35.458+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:06:35.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:06:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:06:35.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:06:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:06:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:06:35.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:06:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:06:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:06:35.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:06:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:06:35.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:06:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:06:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:06:35.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:06:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:06:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:06:35.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:06:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:06:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:06:35.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:06:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:06:35.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:06:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:06:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:06:35.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:06:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:06:35.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:06:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:06:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:06:35.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:06:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:06:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:06:35.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:06:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:06:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:06:35.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:06:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:06:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:06:35.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:06:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:06:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:06:35.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:06:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:06:35.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:06:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=907] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:06:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=907] Local address: 172.21.15.161, port 59490 2022-07-03T19:06:35.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=907] Peer address: 172.21.15.101, port 22 2022-07-03T19:06:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=907] Connection lost 2022-07-03T19:06:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=907] Aborting connection 2022-07-03T19:06:35.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:06:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:35 smithi188 ceph-mon[92425]: pgmap v15222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:06:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:06:35.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:06:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:06:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:06:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:06:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:06:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:06:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:06:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:06:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:06:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:06:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=907] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=907] Local address: 172.21.15.161, port 59490 2022-07-03T19:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=907] Peer address: 172.21.15.101, port 22 2022-07-03T19:06:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=907] Connection lost 2022-07-03T19:06:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=907] Aborting connection 2022-07-03T19:06:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:06:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:06:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:06:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:06:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:06:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:06:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:06:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:06:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:06:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:06:36.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:06:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:06:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:06:36.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:06:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:06:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:06:36.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:06:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:06:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:06:36.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:06:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:06:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:06:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:06:36.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:06:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:06:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:06:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:06:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:06:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:06:36.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:06:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:06:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:06:36.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:06:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:06:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:06:36.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:06:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:06:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:06:36.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:06:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:06:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:06:36.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:06:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:06:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:06:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:06:36.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:06:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=907] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:06:36.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=907] Local address: 172.21.15.161, port 59490 2022-07-03T19:06:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=907] Peer address: 172.21.15.101, port 22 2022-07-03T19:06:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=907] Connection lost 2022-07-03T19:06:36.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=907] Aborting connection 2022-07-03T19:06:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:37 smithi161 ceph-mon[91579]: pgmap v15223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:37 smithi188 ceph-mon[92425]: pgmap v15223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:39 smithi161 ceph-mon[91579]: pgmap v15224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:39 smithi188 ceph-mon[92425]: pgmap v15224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:40 smithi161 ceph-mon[91579]: pgmap v15225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:40 smithi188 ceph-mon[92425]: pgmap v15225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:43 smithi161 ceph-mon[91579]: pgmap v15226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:43.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:43 smithi188 ceph-mon[92425]: pgmap v15226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:45 smithi161 ceph-mon[91579]: pgmap v15227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:45 smithi188 ceph-mon[92425]: pgmap v15227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:47 smithi161 ceph-mon[91579]: pgmap v15228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:47 smithi188 ceph-mon[92425]: pgmap v15228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:49 smithi161 ceph-mon[91579]: pgmap v15229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:49 smithi188 ceph-mon[92425]: pgmap v15229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:51 smithi161 ceph-mon[91579]: pgmap v15230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:51 smithi188 ceph-mon[92425]: pgmap v15230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:52 smithi161 ceph-mon[91579]: pgmap v15231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:52 smithi188 ceph-mon[92425]: pgmap v15231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:54 smithi161 ceph-mon[91579]: pgmap v15232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:54 smithi188 ceph-mon[92425]: pgmap v15232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:06:56.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:56 smithi161 ceph-mon[91579]: pgmap v15233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:56 smithi188 ceph-mon[92425]: pgmap v15233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:06:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:06:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:59 smithi161 ceph-mon[91579]: pgmap v15234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:06:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:06:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:06:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:06:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:59 smithi188 ceph-mon[92425]: pgmap v15234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:06:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:06:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:07:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:01 smithi161 ceph-mon[91579]: pgmap v15235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:01 smithi188 ceph-mon[92425]: pgmap v15235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:03 smithi161 ceph-mon[91579]: pgmap v15236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:03 smithi188 ceph-mon[92425]: pgmap v15236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:05 smithi161 ceph-mon[91579]: pgmap v15237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:05 smithi188 ceph-mon[92425]: pgmap v15237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:05.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:07 smithi161 ceph-mon[91579]: pgmap v15238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:07 smithi188 ceph-mon[92425]: pgmap v15238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:09 smithi161 ceph-mon[91579]: pgmap v15239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:09 smithi188 ceph-mon[92425]: pgmap v15239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:10 smithi161 ceph-mon[91579]: pgmap v15240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:10 smithi188 ceph-mon[92425]: pgmap v15240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:13.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:13 smithi161 ceph-mon[91579]: pgmap v15241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:13.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:13 smithi188 ceph-mon[92425]: pgmap v15241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:15 smithi161 ceph-mon[91579]: pgmap v15242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:15 smithi188 ceph-mon[92425]: pgmap v15242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:17 smithi161 ceph-mon[91579]: pgmap v15243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:17 smithi188 ceph-mon[92425]: pgmap v15243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:19 smithi161 ceph-mon[91579]: pgmap v15244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:19 smithi188 ceph-mon[92425]: pgmap v15244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:21 smithi161 ceph-mon[91579]: pgmap v15245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:21 smithi188 ceph-mon[92425]: pgmap v15245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:22 smithi161 ceph-mon[91579]: pgmap v15246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:22 smithi188 ceph-mon[92425]: pgmap v15246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:24 smithi161 ceph-mon[91579]: pgmap v15247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:24 smithi188 ceph-mon[92425]: pgmap v15247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:26.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:26 smithi161 ceph-mon[91579]: pgmap v15248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:26 smithi188 ceph-mon[92425]: pgmap v15248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:29 smithi161 ceph-mon[91579]: pgmap v15249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:29 smithi188 ceph-mon[92425]: pgmap v15249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:31 smithi161 ceph-mon[91579]: pgmap v15250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:31 smithi188 ceph-mon[92425]: pgmap v15250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:33 smithi161 ceph-mon[91579]: pgmap v15251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:33.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:33 smithi188 ceph-mon[92425]: pgmap v15251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-mon[91579]: pgmap v15252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:35 smithi188 ceph-mon[92425]: pgmap v15252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:07:35.959+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:07:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:07:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:07:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:07:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:07:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:07:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:07:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:07:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:07:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:07:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:07:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:07:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:07:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:07:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:07:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:07:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:07:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:07:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:07:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:07:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:07:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:07:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:07:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:07:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:07:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:07:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:07:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:07:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:07:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:07:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:07:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:07:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:07:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:07:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:07:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:07:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:07:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:07:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:07:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:07:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:07:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:07:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:07:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:07:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:07:36.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=909] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:07:36.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=909] Local address: 172.21.15.161, port 59518 2022-07-03T19:07:36.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=909] Peer address: 172.21.15.101, port 22 2022-07-03T19:07:36.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=909] Connection lost 2022-07-03T19:07:36.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=909] Aborting connection 2022-07-03T19:07:36.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:07:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:07:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:07:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:07:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:07:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:07:36.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:07:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:07:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:36.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:07:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:07:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:07:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:07:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:07:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:07:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:07:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:07:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:07:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:07:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=909] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=909] Local address: 172.21.15.161, port 59518 2022-07-03T19:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=909] Peer address: 172.21.15.101, port 22 2022-07-03T19:07:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=909] Connection lost 2022-07-03T19:07:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=909] Aborting connection 2022-07-03T19:07:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:37 smithi161 ceph-mon[91579]: pgmap v15253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:07:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:07:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:07:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:07:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:07:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:07:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:07:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=909] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=909] Local address: 172.21.15.161, port 59518 2022-07-03T19:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=909] Peer address: 172.21.15.101, port 22 2022-07-03T19:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=909] Connection lost 2022-07-03T19:07:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=909] Aborting connection 2022-07-03T19:07:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:37 smithi188 ceph-mon[92425]: pgmap v15253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:38 smithi161 ceph-mon[91579]: pgmap v15254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:38 smithi188 ceph-mon[92425]: pgmap v15254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:41 smithi161 ceph-mon[91579]: pgmap v15255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:41 smithi188 ceph-mon[92425]: pgmap v15255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:43 smithi161 ceph-mon[91579]: pgmap v15256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:43 smithi188 ceph-mon[92425]: pgmap v15256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:45 smithi161 ceph-mon[91579]: pgmap v15257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:45 smithi188 ceph-mon[92425]: pgmap v15257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:47 smithi161 ceph-mon[91579]: pgmap v15258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:47 smithi188 ceph-mon[92425]: pgmap v15258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:49 smithi161 ceph-mon[91579]: pgmap v15259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:49 smithi188 ceph-mon[92425]: pgmap v15259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:51 smithi161 ceph-mon[91579]: pgmap v15260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:51 smithi188 ceph-mon[92425]: pgmap v15260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:52 smithi161 ceph-mon[91579]: pgmap v15261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:52 smithi188 ceph-mon[92425]: pgmap v15261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:54 smithi161 ceph-mon[91579]: pgmap v15262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:54 smithi188 ceph-mon[92425]: pgmap v15262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:07:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:56 smithi161 ceph-mon[91579]: pgmap v15263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:56 smithi188 ceph-mon[92425]: pgmap v15263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:07:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:07:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:59 smithi161 ceph-mon[91579]: pgmap v15264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:07:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:07:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:07:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:07:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:59 smithi188 ceph-mon[92425]: pgmap v15264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:07:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:07:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:07:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:07:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:08:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:01 smithi161 ceph-mon[91579]: pgmap v15265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:01 smithi188 ceph-mon[92425]: pgmap v15265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:03 smithi161 ceph-mon[91579]: pgmap v15266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:03 smithi188 ceph-mon[92425]: pgmap v15266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:05 smithi161 ceph-mon[91579]: pgmap v15267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:05 smithi188 ceph-mon[92425]: pgmap v15267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:06 smithi188 ceph-mon[92425]: pgmap v15268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:06 smithi161 ceph-mon[91579]: pgmap v15268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:09 smithi161 ceph-mon[91579]: pgmap v15269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:09 smithi188 ceph-mon[92425]: pgmap v15269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:11 smithi161 ceph-mon[91579]: pgmap v15270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:11 smithi188 ceph-mon[92425]: pgmap v15270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:13 smithi161 ceph-mon[91579]: pgmap v15271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:13 smithi188 ceph-mon[92425]: pgmap v15271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:15 smithi161 ceph-mon[91579]: pgmap v15272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:15 smithi188 ceph-mon[92425]: pgmap v15272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:16 smithi161 ceph-mon[91579]: pgmap v15273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:16 smithi188 ceph-mon[92425]: pgmap v15273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:19 smithi161 ceph-mon[91579]: pgmap v15274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:19 smithi188 ceph-mon[92425]: pgmap v15274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:21 smithi161 ceph-mon[91579]: pgmap v15275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:21 smithi188 ceph-mon[92425]: pgmap v15275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:23 smithi161 ceph-mon[91579]: pgmap v15276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:23 smithi188 ceph-mon[92425]: pgmap v15276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:24 smithi188 ceph-mon[92425]: pgmap v15277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:24 smithi161 ceph-mon[91579]: pgmap v15277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:26 smithi188 ceph-mon[92425]: pgmap v15278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:26 smithi161 ceph-mon[91579]: pgmap v15278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:29 smithi161 ceph-mon[91579]: pgmap v15279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:29 smithi188 ceph-mon[92425]: pgmap v15279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:31 smithi161 ceph-mon[91579]: pgmap v15280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:31 smithi188 ceph-mon[92425]: pgmap v15280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:33 smithi161 ceph-mon[91579]: pgmap v15281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:33 smithi188 ceph-mon[92425]: pgmap v15281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:34 smithi188 ceph-mon[92425]: pgmap v15282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:34 smithi161 ceph-mon[91579]: pgmap v15282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-mon[91579]: pgmap v15283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:36.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:08:36.895+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:08:36.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:08:36.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:08:36.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:08:36.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:08:36.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:08:36.906 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:36 smithi188 ceph-mon[92425]: pgmap v15283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:37.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:08:37.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:08:37.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:08:37.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:08:37.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:08:37.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=911] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=911] Local address: 172.21.15.161, port 59546 2022-07-03T19:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=911] Peer address: 172.21.15.101, port 22 2022-07-03T19:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=911] Connection lost 2022-07-03T19:08:37.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=911] Aborting connection 2022-07-03T19:08:37.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:08:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:08:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:08:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:08:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:08:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:08:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:08:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:08:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:08:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:08:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:08:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:08:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:08:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:08:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:08:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:08:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:08:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:08:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:08:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:08:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:08:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:08:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:08:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:08:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:08:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:08:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:08:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:08:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:08:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:08:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:08:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:08:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:08:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:08:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:08:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:08:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:08:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:08:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:08:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:08:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:08:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:08:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:08:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:08:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:08:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:08:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:08:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:08:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:08:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:08:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=911] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:08:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=911] Local address: 172.21.15.161, port 59546 2022-07-03T19:08:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=911] Peer address: 172.21.15.101, port 22 2022-07-03T19:08:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=911] Connection lost 2022-07-03T19:08:37.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=911] Aborting connection 2022-07-03T19:08:38.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:08:37.778+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T19:08:38.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:08:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:08:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:08:38.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:08:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:08:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:08:38.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:08:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:08:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:08:38.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:08:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:08:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:08:38.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:08:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:08:38.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:08:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:08:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:08:38.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:08:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:08:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:08:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:08:38.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:08:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:08:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:08:38.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:08:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:08:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:08:38.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:08:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:08:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:08:38.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:08:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:08:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:08:38.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:08:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:08:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:08:38.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:08:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:08:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:08:38.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:08:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:08:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:08:38.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:08:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:08:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:08:38.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:08:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:08:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:08:38.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:08:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=911] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:08:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=911] Local address: 172.21.15.161, port 59546 2022-07-03T19:08:38.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=911] Peer address: 172.21.15.101, port 22 2022-07-03T19:08:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=911] Connection lost 2022-07-03T19:08:38.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=911] Aborting connection 2022-07-03T19:08:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:38 smithi188 ceph-mon[92425]: pgmap v15284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:08:39.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:38 smithi161 ceph-mon[91579]: pgmap v15284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:41 smithi161 ceph-mon[91579]: pgmap v15285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:41 smithi188 ceph-mon[92425]: pgmap v15285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:43 smithi161 ceph-mon[91579]: pgmap v15286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:43 smithi188 ceph-mon[92425]: pgmap v15286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:45 smithi161 ceph-mon[91579]: pgmap v15287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:45 smithi188 ceph-mon[92425]: pgmap v15287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:47 smithi161 ceph-mon[91579]: pgmap v15288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:47 smithi188 ceph-mon[92425]: pgmap v15288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:49 smithi161 ceph-mon[91579]: pgmap v15289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:49 smithi188 ceph-mon[92425]: pgmap v15289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:51 smithi161 ceph-mon[91579]: pgmap v15290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:51 smithi188 ceph-mon[92425]: pgmap v15290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:52 smithi161 ceph-mon[91579]: pgmap v15291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:52 smithi188 ceph-mon[92425]: pgmap v15291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:55 smithi161 ceph-mon[91579]: pgmap v15292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:55 smithi188 ceph-mon[92425]: pgmap v15292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:08:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:57 smithi161 ceph-mon[91579]: pgmap v15293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:57 smithi188 ceph-mon[92425]: pgmap v15293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:08:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:59 smithi161 ceph-mon[91579]: pgmap v15294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:08:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:08:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:08:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:08:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:59 smithi188 ceph-mon[92425]: pgmap v15294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:08:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:08:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:08:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:08:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:09:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:00 smithi161 ceph-mon[91579]: pgmap v15295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:00 smithi188 ceph-mon[92425]: pgmap v15295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:03 smithi161 ceph-mon[91579]: pgmap v15296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:03 smithi188 ceph-mon[92425]: pgmap v15296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:05 smithi161 ceph-mon[91579]: pgmap v15297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:05 smithi188 ceph-mon[92425]: pgmap v15297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:07 smithi161 ceph-mon[91579]: pgmap v15298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:07 smithi188 ceph-mon[92425]: pgmap v15298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:09 smithi161 ceph-mon[91579]: pgmap v15299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:09 smithi188 ceph-mon[92425]: pgmap v15299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:10 smithi161 ceph-mon[91579]: pgmap v15300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:10 smithi188 ceph-mon[92425]: pgmap v15300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:13 smithi161 ceph-mon[91579]: pgmap v15301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:13 smithi188 ceph-mon[92425]: pgmap v15301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:15 smithi161 ceph-mon[91579]: pgmap v15302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:15.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:15 smithi188 ceph-mon[92425]: pgmap v15302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:17 smithi161 ceph-mon[91579]: pgmap v15303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:17 smithi188 ceph-mon[92425]: pgmap v15303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:19 smithi161 ceph-mon[91579]: pgmap v15304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:19 smithi188 ceph-mon[92425]: pgmap v15304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:20 smithi161 ceph-mon[91579]: pgmap v15305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:20.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:20 smithi188 ceph-mon[92425]: pgmap v15305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:23 smithi161 ceph-mon[91579]: pgmap v15306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:23 smithi188 ceph-mon[92425]: pgmap v15306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:25 smithi161 ceph-mon[91579]: pgmap v15307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:25 smithi188 ceph-mon[92425]: pgmap v15307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:27 smithi161 ceph-mon[91579]: pgmap v15308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:27 smithi188 ceph-mon[92425]: pgmap v15308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:29 smithi161 ceph-mon[91579]: pgmap v15309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:29 smithi188 ceph-mon[92425]: pgmap v15309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:31 smithi161 ceph-mon[91579]: pgmap v15310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:31 smithi188 ceph-mon[92425]: pgmap v15310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:32 smithi161 ceph-mon[91579]: pgmap v15311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:32 smithi188 ceph-mon[92425]: pgmap v15311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:35 smithi161 ceph-mon[91579]: pgmap v15312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:35 smithi188 ceph-mon[92425]: pgmap v15312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:37 smithi188 ceph-mon[92425]: pgmap v15313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:09:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:09:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:09:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-mon[91579]: pgmap v15313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:09:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:09:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:09:37.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:09:37.395+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:09:37.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:09:37.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:09:37.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:09:37.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:09:37.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:09:37.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:09:37.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:09:37.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:09:37.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:09:37.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:09:37.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:09:37.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:09:37.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:09:37.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:09:37.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:09:37.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:09:37.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:09:37.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:09:37.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:09:37.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:09:37.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:09:37.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:09:37.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:09:37.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:09:37.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:09:37.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:09:37.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:09:37.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:09:37.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:09:37.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:09:37.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:09:37.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:09:37.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:09:37.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:09:37.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:09:37.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:09:37.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:09:37.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:09:37.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:09:37.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:09:37.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:09:37.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:09:37.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:09:37.910 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:09:37.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=913] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:09:37.911 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=913] Local address: 172.21.15.161, port 59580 2022-07-03T19:09:37.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=913] Peer address: 172.21.15.101, port 22 2022-07-03T19:09:37.912 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=913] Connection lost 2022-07-03T19:09:37.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=913] Aborting connection 2022-07-03T19:09:37.913 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:09:38.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:09:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:09:38.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:09:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:09:38.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:09:38.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:09:38.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:09:38.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:09:38.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:09:38.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:09:38.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:09:38.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:09:38.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:09:38.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:09:38.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:09:38.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:09:38.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:09:38.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:09:38.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:09:38.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:09:38.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:09:38.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:09:38.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:09:38.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:09:38.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:09:38.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:09:38.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:09:38.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:09:38.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:09:38.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:09:38.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:09:38.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:09:38.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:09:38.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:09:38.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:09:38.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:09:38.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:09:38.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:09:38.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:09:38.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:09:38.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:09:38.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:09:38.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:09:38.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:09:38.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:09:38.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:09:38.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=913] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:09:38.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=913] Local address: 172.21.15.161, port 59580 2022-07-03T19:09:38.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=913] Peer address: 172.21.15.101, port 22 2022-07-03T19:09:38.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=913] Connection lost 2022-07-03T19:09:38.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=913] Aborting connection 2022-07-03T19:09:38.905 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:38 smithi161 ceph-mon[91579]: pgmap v15314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:38.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:09:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:09:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:09:38.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:09:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:09:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:09:38.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:09:38.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:09:38.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:09:38.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:09:38.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:09:38.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:09:38.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:09:38.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:09:38.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:09:38.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:09:38.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:09:38.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:09:38.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:09:38.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:09:38.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:09:38.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:09:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:09:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:09:38.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:09:38.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:09:38.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:09:38.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:09:38.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:09:38.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:09:38.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:09:38.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:09:38.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:09:38.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:09:38.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:09:38.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:09:38.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:09:38.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:09:38.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:09:38.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:09:38.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:09:38.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:09:38.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:09:38.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:09:38.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:09:38.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:09:38.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=913] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:09:38.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=913] Local address: 172.21.15.161, port 59580 2022-07-03T19:09:38.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=913] Peer address: 172.21.15.101, port 22 2022-07-03T19:09:38.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=913] Connection lost 2022-07-03T19:09:38.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=913] Aborting connection 2022-07-03T19:09:38.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:38 smithi188 ceph-mon[92425]: pgmap v15314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:41.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:41 smithi161 ceph-mon[91579]: pgmap v15315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:41.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:41 smithi188 ceph-mon[92425]: pgmap v15315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:43 smithi188 ceph-mon[92425]: pgmap v15316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:43 smithi161 ceph-mon[91579]: pgmap v15316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:45 smithi161 ceph-mon[91579]: pgmap v15317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:45 smithi188 ceph-mon[92425]: pgmap v15317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:46 smithi161 ceph-mon[91579]: pgmap v15318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:46 smithi188 ceph-mon[92425]: pgmap v15318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:49 smithi161 ceph-mon[91579]: pgmap v15319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:49 smithi188 ceph-mon[92425]: pgmap v15319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:51 smithi161 ceph-mon[91579]: pgmap v15320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:51 smithi188 ceph-mon[92425]: pgmap v15320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:52 smithi161 ceph-mon[91579]: pgmap v15321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:52 smithi188 ceph-mon[92425]: pgmap v15321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:55 smithi161 ceph-mon[91579]: pgmap v15322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:55 smithi188 ceph-mon[92425]: pgmap v15322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:09:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:56 smithi188 ceph-mon[92425]: pgmap v15323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:56 smithi161 ceph-mon[91579]: pgmap v15323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:09:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:59 smithi161 ceph-mon[91579]: pgmap v15324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:09:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:09:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:09:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:09:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:59 smithi188 ceph-mon[92425]: pgmap v15324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:09:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:09:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:09:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:09:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:10:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:10:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:10:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:01 smithi161 ceph-mon[91579]: pgmap v15325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:01 smithi188 ceph-mon[92425]: pgmap v15325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:02 smithi161 ceph-mon[91579]: pgmap v15326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:02 smithi188 ceph-mon[92425]: pgmap v15326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:05 smithi161 ceph-mon[91579]: pgmap v15327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:05 smithi188 ceph-mon[92425]: pgmap v15327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:06 smithi161 ceph-mon[91579]: pgmap v15328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:06 smithi188 ceph-mon[92425]: pgmap v15328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:09 smithi161 ceph-mon[91579]: pgmap v15329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:09 smithi188 ceph-mon[92425]: pgmap v15329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:10 smithi161 ceph-mon[91579]: pgmap v15330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:10.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:10.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:10 smithi188 ceph-mon[92425]: pgmap v15330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:10.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:13 smithi188 ceph-mon[92425]: pgmap v15331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:13 smithi161 ceph-mon[91579]: pgmap v15331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:15 smithi161 ceph-mon[91579]: pgmap v15332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:15 smithi188 ceph-mon[92425]: pgmap v15332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:16 smithi161 ceph-mon[91579]: pgmap v15333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:16 smithi188 ceph-mon[92425]: pgmap v15333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:19 smithi161 ceph-mon[91579]: pgmap v15334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:19 smithi188 ceph-mon[92425]: pgmap v15334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:20.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:21 smithi161 ceph-mon[91579]: pgmap v15335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:21 smithi188 ceph-mon[92425]: pgmap v15335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:22.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:22 smithi161 ceph-mon[91579]: pgmap v15336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:22.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:22 smithi188 ceph-mon[92425]: pgmap v15336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:24 smithi188 ceph-mon[92425]: pgmap v15337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:24 smithi161 ceph-mon[91579]: pgmap v15337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:26 smithi161 ceph-mon[91579]: pgmap v15338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:27.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:26 smithi188 ceph-mon[92425]: pgmap v15338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:29 smithi161 ceph-mon[91579]: pgmap v15339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:29 smithi188 ceph-mon[92425]: pgmap v15339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:30 smithi161 ceph-mon[91579]: pgmap v15340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:30 smithi188 ceph-mon[92425]: pgmap v15340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:33 smithi188 ceph-mon[92425]: pgmap v15341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:33 smithi161 ceph-mon[91579]: pgmap v15341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:34 smithi188 ceph-mon[92425]: pgmap v15342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:34 smithi161 ceph-mon[91579]: pgmap v15342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:36 smithi161 ceph-mon[91579]: pgmap v15343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:36 smithi188 ceph-mon[92425]: pgmap v15343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:10:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:10:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:10:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:10:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:10:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:10:38.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:10:37.892+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:10:38.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:10:38.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:10:38.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:10:38.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:10:38.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:10:38.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:10:38.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:10:38.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:10:38.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:10:38.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:10:38.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:10:38.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:10:38.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:10:38.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:10:38.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:10:38.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:10:38.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:10:38.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:10:38.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:10:38.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:10:38.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:10:38.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:10:38.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:10:38.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:10:38.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:10:38.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:10:38.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:10:38.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:10:38.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:10:38.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:10:38.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:10:38.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:10:38.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:10:38.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:10:38.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:10:38.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:10:38.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:10:38.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:10:38.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:10:38.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:10:38.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:10:38.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:10:38.407 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:10:38.408 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:10:38.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=915] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:10:38.409 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=915] Local address: 172.21.15.161, port 59584 2022-07-03T19:10:38.410 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=915] Peer address: 172.21.15.101, port 22 2022-07-03T19:10:38.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=915] Connection lost 2022-07-03T19:10:38.411 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=915] Aborting connection 2022-07-03T19:10:38.412 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:10:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:10:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:10:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:10:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:10:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:10:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:10:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:10:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:10:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:10:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:10:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:10:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:10:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:10:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:10:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:10:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:10:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:10:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:10:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:10:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:10:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:10:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:10:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:10:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:10:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:10:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:10:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:10:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=915] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=915] Local address: 172.21.15.161, port 59584 2022-07-03T19:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=915] Peer address: 172.21.15.101, port 22 2022-07-03T19:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=915] Connection lost 2022-07-03T19:10:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=915] Aborting connection 2022-07-03T19:10:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:39 smithi161 ceph-mon[91579]: pgmap v15344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:10:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:10:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:10:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:10:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:10:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:10:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:10:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:10:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:10:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:10:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:10:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:10:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:10:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:10:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:10:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:10:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:10:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:10:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:10:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:10:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:10:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:10:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:10:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:10:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:10:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:10:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:10:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:10:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:10:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:10:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:10:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=915] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=915] Local address: 172.21.15.161, port 59584 2022-07-03T19:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=915] Peer address: 172.21.15.101, port 22 2022-07-03T19:10:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=915] Connection lost 2022-07-03T19:10:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=915] Aborting connection 2022-07-03T19:10:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:39 smithi188 ceph-mon[92425]: pgmap v15344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:40 smithi161 ceph-mon[91579]: pgmap v15345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:40 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:40 smithi188 ceph-mon[92425]: pgmap v15345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:40.948 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:40.949 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:40.949 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:40 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 118.6M 2022-07-03T19:10:41.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 124418629: error parsing value: Value '124418629' is below minimum 939524096 2022-07-03T19:10:41.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:41.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:41 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:10:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 118.6M 2022-07-03T19:10:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 124418629: error parsing value: Value '124418629' is below minimum 939524096 2022-07-03T19:10:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:41 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:10:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:42 smithi188 ceph-mon[92425]: pgmap v15346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:42 smithi161 ceph-mon[91579]: pgmap v15346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:44 smithi161 ceph-mon[91579]: pgmap v15347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:44 smithi188 ceph-mon[92425]: pgmap v15347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:46.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:46 smithi161 ceph-mon[91579]: pgmap v15348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:46 smithi188 ceph-mon[92425]: pgmap v15348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:49 smithi188 ceph-mon[92425]: pgmap v15349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:49 smithi161 ceph-mon[91579]: pgmap v15349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:51.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:51 smithi188 ceph-mon[92425]: pgmap v15350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:51.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:51 smithi161 ceph-mon[91579]: pgmap v15350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:53 smithi188 ceph-mon[92425]: pgmap v15351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:53 smithi161 ceph-mon[91579]: pgmap v15351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:10:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:54 smithi161 ceph-mon[91579]: pgmap v15352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:54 smithi188 ceph-mon[92425]: pgmap v15352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:10:56.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:56 smithi161 ceph-mon[91579]: pgmap v15353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:10:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:56 smithi188 ceph-mon[92425]: pgmap v15353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:10:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:59 smithi188 ceph-mon[92425]: pgmap v15354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:10:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:10:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:10:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:59 smithi161 ceph-mon[91579]: pgmap v15354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:10:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:10:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:10:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:10:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:11:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:01 smithi188 ceph-mon[92425]: pgmap v15355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:11:01.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:01 smithi161 ceph-mon[91579]: pgmap v15355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:11:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:03 smithi188 ceph-mon[92425]: pgmap v15356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:11:03.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:03 smithi161 ceph-mon[91579]: pgmap v15356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:11:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:04 smithi161 ceph-mon[91579]: pgmap v15357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:04 smithi188 ceph-mon[92425]: pgmap v15357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:06 smithi161 ceph-mon[91579]: pgmap v15358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:06 smithi188 ceph-mon[92425]: pgmap v15358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:09 smithi188 ceph-mon[92425]: pgmap v15359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:09 smithi161 ceph-mon[91579]: pgmap v15359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:11 smithi188 ceph-mon[92425]: pgmap v15360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:11 smithi161 ceph-mon[91579]: pgmap v15360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:13 smithi188 ceph-mon[92425]: pgmap v15361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:13 smithi161 ceph-mon[91579]: pgmap v15361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:14 smithi161 ceph-mon[91579]: pgmap v15362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:14 smithi188 ceph-mon[92425]: pgmap v15362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:16.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:16 smithi188 ceph-mon[92425]: pgmap v15363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:17.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:16 smithi161 ceph-mon[91579]: pgmap v15363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:19 smithi188 ceph-mon[92425]: pgmap v15364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:19 smithi161 ceph-mon[91579]: pgmap v15364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:21 smithi188 ceph-mon[92425]: pgmap v15365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:21 smithi161 ceph-mon[91579]: pgmap v15365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:23 smithi161 ceph-mon[91579]: pgmap v15366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:23 smithi188 ceph-mon[92425]: pgmap v15366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:24 smithi161 ceph-mon[91579]: pgmap v15367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:24 smithi188 ceph-mon[92425]: pgmap v15367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:26.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:26 smithi161 ceph-mon[91579]: pgmap v15368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:26.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:26 smithi188 ceph-mon[92425]: pgmap v15368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:29 smithi188 ceph-mon[92425]: pgmap v15369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:29 smithi161 ceph-mon[91579]: pgmap v15369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:31 smithi161 ceph-mon[91579]: pgmap v15370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:31 smithi188 ceph-mon[92425]: pgmap v15370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:33 smithi161 ceph-mon[91579]: pgmap v15371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:33 smithi188 ceph-mon[92425]: pgmap v15371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:34 smithi161 ceph-mon[91579]: pgmap v15372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:34 smithi188 ceph-mon[92425]: pgmap v15372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:36 smithi188 ceph-mon[92425]: pgmap v15373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:36 smithi161 ceph-mon[91579]: pgmap v15373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:39 smithi161 ceph-mon[91579]: pgmap v15374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:39 smithi188 ceph-mon[92425]: pgmap v15374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:41 smithi188 ceph-mon[92425]: pgmap v15375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:41.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:11:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:11:41.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:11:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-mon[91579]: pgmap v15375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:11:41.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:11:41.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:11:41.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:11:41.144+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:11:41.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:11:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:11:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:11:41.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:11:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:11:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:11:41.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:11:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:11:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:11:41.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:11:41.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:11:41.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:11:41.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:11:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:11:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:11:41.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:11:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:11:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:11:41.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:11:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:11:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:11:41.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:11:41.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:11:41.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:11:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:11:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:11:41.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:11:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:11:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:11:41.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:11:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:11:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:11:41.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:11:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:11:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:11:41.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:11:41.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:11:41.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:11:41.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:11:41.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:11:41.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:11:41.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:11:41.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:11:41.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:11:41.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=917] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:11:41.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=917] Local address: 172.21.15.161, port 59588 2022-07-03T19:11:41.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=917] Peer address: 172.21.15.101, port 22 2022-07-03T19:11:41.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=917] Connection lost 2022-07-03T19:11:41.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=917] Aborting connection 2022-07-03T19:11:41.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:11:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:11:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:42 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:11:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:11:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:11:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:11:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:11:42.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:11:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:11:42.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:11:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:11:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:11:42.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:11:42.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:11:42.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:11:42.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:11:42.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:11:42.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:11:42.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:11:42.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:11:42.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:11:42.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:11:42.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:11:42.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:11:42.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:11:42.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:11:42.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:11:42.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:11:42.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:11:42.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:11:42.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:11:42.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:11:42.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:11:42.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:11:42.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:11:42.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:11:42.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:11:42.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:11:42.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:11:42.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:11:42.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:11:42.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:11:42.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:11:42.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:11:42.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:11:42.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:11:42.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:11:42.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:11:42.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=917] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:11:42.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=917] Local address: 172.21.15.161, port 59588 2022-07-03T19:11:42.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=917] Peer address: 172.21.15.101, port 22 2022-07-03T19:11:42.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=917] Connection lost 2022-07-03T19:11:42.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=917] Aborting connection 2022-07-03T19:11:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:42 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:11:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:11:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:11:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:11:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:11:42.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:11:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:11:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:11:42.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:11:42.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:11:42.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:11:42.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:11:42.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:11:42.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:11:42.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:11:42.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:11:42.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:11:42.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:11:42.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:11:42.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:11:42.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:11:42.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:11:42.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:11:42.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:11:42.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:11:42.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:11:42.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:11:42.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:11:42.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:11:42.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:11:42.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:11:42.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:11:42.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:11:42.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:11:42.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:11:42.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:11:42.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:11:42.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:11:42.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:11:42.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:11:42.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:11:42.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:11:42.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:11:42.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:11:42.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:11:42.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:11:42.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=917] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:11:42.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=917] Local address: 172.21.15.161, port 59588 2022-07-03T19:11:42.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=917] Peer address: 172.21.15.101, port 22 2022-07-03T19:11:42.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=917] Connection lost 2022-07-03T19:11:42.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=917] Aborting connection 2022-07-03T19:11:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:43 smithi161 ceph-mon[91579]: pgmap v15376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:43 smithi188 ceph-mon[92425]: pgmap v15376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:44 smithi161 ceph-mon[91579]: pgmap v15377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:44 smithi188 ceph-mon[92425]: pgmap v15377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:46.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:46 smithi188 ceph-mon[92425]: pgmap v15378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:46 smithi161 ceph-mon[91579]: pgmap v15378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:49 smithi161 ceph-mon[91579]: pgmap v15379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:49 smithi188 ceph-mon[92425]: pgmap v15379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:51 smithi161 ceph-mon[91579]: pgmap v15380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:51 smithi188 ceph-mon[92425]: pgmap v15380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:53 smithi161 ceph-mon[91579]: pgmap v15381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:53 smithi188 ceph-mon[92425]: pgmap v15381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:54 smithi161 ceph-mon[91579]: pgmap v15382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:54 smithi188 ceph-mon[92425]: pgmap v15382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:11:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:56 smithi188 ceph-mon[92425]: pgmap v15383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:56 smithi161 ceph-mon[91579]: pgmap v15383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:11:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:59 smithi161 ceph-mon[91579]: pgmap v15384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:11:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:11:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:11:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:11:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:59 smithi188 ceph-mon[92425]: pgmap v15384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:11:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:11:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:11:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:11:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:12:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:01 smithi161 ceph-mon[91579]: pgmap v15385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:01 smithi188 ceph-mon[92425]: pgmap v15385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:03 smithi161 ceph-mon[91579]: pgmap v15386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:03 smithi188 ceph-mon[92425]: pgmap v15386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:04 smithi188 ceph-mon[92425]: pgmap v15387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:04 smithi161 ceph-mon[91579]: pgmap v15387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:06 smithi188 ceph-mon[92425]: pgmap v15388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:06 smithi161 ceph-mon[91579]: pgmap v15388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:09 smithi161 ceph-mon[91579]: pgmap v15389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:09 smithi188 ceph-mon[92425]: pgmap v15389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:11 smithi161 ceph-mon[91579]: pgmap v15390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:11 smithi188 ceph-mon[92425]: pgmap v15390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:13 smithi161 ceph-mon[91579]: pgmap v15391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:13 smithi188 ceph-mon[92425]: pgmap v15391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:14 smithi188 ceph-mon[92425]: pgmap v15392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:14 smithi161 ceph-mon[91579]: pgmap v15392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:16 smithi188 ceph-mon[92425]: pgmap v15393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:16 smithi161 ceph-mon[91579]: pgmap v15393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:19 smithi161 ceph-mon[91579]: pgmap v15394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:19 smithi188 ceph-mon[92425]: pgmap v15394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:21 smithi161 ceph-mon[91579]: pgmap v15395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:21 smithi188 ceph-mon[92425]: pgmap v15395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:23 smithi161 ceph-mon[91579]: pgmap v15396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:23 smithi188 ceph-mon[92425]: pgmap v15396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:24 smithi188 ceph-mon[92425]: pgmap v15397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:24 smithi161 ceph-mon[91579]: pgmap v15397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:26 smithi188 ceph-mon[92425]: pgmap v15398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:26 smithi161 ceph-mon[91579]: pgmap v15398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:29 smithi161 ceph-mon[91579]: pgmap v15399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:29 smithi188 ceph-mon[92425]: pgmap v15399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:31 smithi161 ceph-mon[91579]: pgmap v15400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:31 smithi188 ceph-mon[92425]: pgmap v15400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:33 smithi161 ceph-mon[91579]: pgmap v15401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:33 smithi188 ceph-mon[92425]: pgmap v15401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:34 smithi188 ceph-mon[92425]: pgmap v15402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:34 smithi161 ceph-mon[91579]: pgmap v15402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:36 smithi188 ceph-mon[92425]: pgmap v15403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:36 smithi161 ceph-mon[91579]: pgmap v15403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:39 smithi161 ceph-mon[91579]: pgmap v15404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:39 smithi188 ceph-mon[92425]: pgmap v15404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-mon[91579]: pgmap v15405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:41 smithi188 ceph-mon[92425]: pgmap v15405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:42.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:12:41.650+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:12:42.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:12:42.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:12:42.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:12:42.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:12:42.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:12:42.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:12:42.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:12:42.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:12:42.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:12:42.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:12:42.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:12:42.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:12:42.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:12:42.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:12:42.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:12:42.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:12:42.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:12:42.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:12:42.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:12:42.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:12:42.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:12:42.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:12:42.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:12:42.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:12:42.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:12:42.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:12:42.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:12:42.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:12:42.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:12:42.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:12:42.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:12:42.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:12:42.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:12:42.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:12:42.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:12:42.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:12:42.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:12:42.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:12:42.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:12:42.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:12:42.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:12:42.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:12:42.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:12:42.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:12:42.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=919] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:12:42.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=919] Local address: 172.21.15.161, port 59592 2022-07-03T19:12:42.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=919] Peer address: 172.21.15.101, port 22 2022-07-03T19:12:42.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=919] Connection lost 2022-07-03T19:12:42.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=919] Aborting connection 2022-07-03T19:12:42.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:12:41 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:12:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:12:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:12:42.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:12:42.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:12:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:12:42.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:12:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:12:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:12:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:12:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:12:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:12:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:12:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:12:43.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:12:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:12:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:12:43.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:12:43.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:12:43.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:12:43.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:12:43.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:12:43.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:12:43.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:12:43.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:12:43.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:12:43.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:12:43.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:12:43.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:12:43.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:12:43.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:12:43.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:12:43.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:12:43.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:12:43.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:12:43.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:12:43.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:12:43.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:12:43.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:12:43.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:12:43.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:12:43.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:12:43.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:12:43.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:12:43.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:12:43.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:12:43.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:12:43.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:12:43.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:12:43.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:12:43.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:12:43.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:12:43.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:12:43.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=919] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:12:43.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=919] Local address: 172.21.15.161, port 59592 2022-07-03T19:12:43.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=919] Peer address: 172.21.15.101, port 22 2022-07-03T19:12:43.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=919] Connection lost 2022-07-03T19:12:43.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=919] Aborting connection 2022-07-03T19:12:43.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:43 smithi161 ceph-mon[91579]: pgmap v15406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:12:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:12:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:12:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:12:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:12:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:12:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:12:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:12:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:12:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:12:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:12:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:12:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:12:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:12:43.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:12:43.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:12:43.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:12:43.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:12:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:12:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:12:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:12:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:12:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:12:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:12:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:12:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:12:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:12:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:12:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:12:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:12:43.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:12:43.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:12:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:12:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:12:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:12:43.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:12:43.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:12:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:12:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:12:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:12:43.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:12:43.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:12:43.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:12:43.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:12:43.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:12:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:12:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=919] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:12:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=919] Local address: 172.21.15.161, port 59592 2022-07-03T19:12:43.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=919] Peer address: 172.21.15.101, port 22 2022-07-03T19:12:43.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=919] Connection lost 2022-07-03T19:12:43.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=919] Aborting connection 2022-07-03T19:12:43.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:43 smithi188 ceph-mon[92425]: pgmap v15406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:44 smithi188 ceph-mon[92425]: pgmap v15407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:44 smithi161 ceph-mon[91579]: pgmap v15407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:46 smithi188 ceph-mon[92425]: pgmap v15408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:46 smithi161 ceph-mon[91579]: pgmap v15408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:49 smithi161 ceph-mon[91579]: pgmap v15409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:49 smithi188 ceph-mon[92425]: pgmap v15409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:51 smithi161 ceph-mon[91579]: pgmap v15410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:51 smithi188 ceph-mon[92425]: pgmap v15410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:53 smithi161 ceph-mon[91579]: pgmap v15411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:53 smithi188 ceph-mon[92425]: pgmap v15411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:54 smithi188 ceph-mon[92425]: pgmap v15412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:54 smithi161 ceph-mon[91579]: pgmap v15412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:12:56.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:56 smithi188 ceph-mon[92425]: pgmap v15413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:56 smithi161 ceph-mon[91579]: pgmap v15413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:12:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:59 smithi161 ceph-mon[91579]: pgmap v15414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:12:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:12:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:12:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:12:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:59 smithi188 ceph-mon[92425]: pgmap v15414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:12:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:12:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:12:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:12:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:01 smithi161 ceph-mon[91579]: pgmap v15415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:01 smithi188 ceph-mon[92425]: pgmap v15415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:03 smithi161 ceph-mon[91579]: pgmap v15416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:03 smithi188 ceph-mon[92425]: pgmap v15416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:04 smithi188 ceph-mon[92425]: pgmap v15417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:04 smithi161 ceph-mon[91579]: pgmap v15417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:06 smithi188 ceph-mon[92425]: pgmap v15418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:06 smithi161 ceph-mon[91579]: pgmap v15418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:09 smithi161 ceph-mon[91579]: pgmap v15419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:09 smithi188 ceph-mon[92425]: pgmap v15419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:11 smithi161 ceph-mon[91579]: pgmap v15420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:11 smithi188 ceph-mon[92425]: pgmap v15420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:13 smithi161 ceph-mon[91579]: pgmap v15421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:13 smithi188 ceph-mon[92425]: pgmap v15421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:14 smithi188 ceph-mon[92425]: pgmap v15422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:14 smithi161 ceph-mon[91579]: pgmap v15422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:16 smithi188 ceph-mon[92425]: pgmap v15423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:16 smithi161 ceph-mon[91579]: pgmap v15423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:19 smithi161 ceph-mon[91579]: pgmap v15424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:19 smithi188 ceph-mon[92425]: pgmap v15424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:21 smithi161 ceph-mon[91579]: pgmap v15425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:21 smithi188 ceph-mon[92425]: pgmap v15425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:23 smithi161 ceph-mon[91579]: pgmap v15426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:23 smithi188 ceph-mon[92425]: pgmap v15426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:24 smithi188 ceph-mon[92425]: pgmap v15427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:24 smithi161 ceph-mon[91579]: pgmap v15427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:26.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:26 smithi188 ceph-mon[92425]: pgmap v15428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:26 smithi161 ceph-mon[91579]: pgmap v15428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:29 smithi161 ceph-mon[91579]: pgmap v15429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:29 smithi188 ceph-mon[92425]: pgmap v15429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:31 smithi161 ceph-mon[91579]: pgmap v15430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:31 smithi188 ceph-mon[92425]: pgmap v15430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:33 smithi161 ceph-mon[91579]: pgmap v15431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:33 smithi188 ceph-mon[92425]: pgmap v15431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:34 smithi188 ceph-mon[92425]: pgmap v15432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:34 smithi161 ceph-mon[91579]: pgmap v15432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:36 smithi161 ceph-mon[91579]: pgmap v15433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:36 smithi188 ceph-mon[92425]: pgmap v15433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:39 smithi161 ceph-mon[91579]: pgmap v15434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:39 smithi188 ceph-mon[92425]: pgmap v15434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:41 smithi161 ceph-mon[91579]: pgmap v15435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:41 smithi188 ceph-mon[92425]: pgmap v15435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:13:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:13:42.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:13:42.616 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:13:42.617 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:13:42.617 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:13:42.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:13:42.144+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:13:42.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:13:42.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:13:42.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:13:42.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:13:42.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:13:42.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:13:42.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:13:42.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:13:42.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:13:42.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:13:42.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:13:42.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:13:42.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:13:42.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:13:42.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:13:42.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:13:42.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:13:42.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:13:42.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:13:42.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:13:42.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:13:42.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:13:42.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:13:42.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:13:42.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:13:42.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:13:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:13:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:13:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:13:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:13:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:13:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:13:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:13:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:13:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:13:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:13:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:13:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:13:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:13:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:13:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:13:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:13:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:13:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:13:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=921] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:13:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=921] Local address: 172.21.15.161, port 59596 2022-07-03T19:13:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=921] Peer address: 172.21.15.101, port 22 2022-07-03T19:13:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=921] Connection lost 2022-07-03T19:13:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=921] Aborting connection 2022-07-03T19:13:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:13:43.470 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:13:43.471 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:13:43.471 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:13:43.472 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:13:43.472 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:13:43.473 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:13:43.473 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:13:43.474 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:13:43.474 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:13:43.474 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:13:43.475 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:13:43.475 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:13:43.475 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:13:43.476 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:13:43.476 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:13:43.477 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:13:43.477 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:13:43.477 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:13:43.478 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:13:43.478 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:13:43.478 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:13:43.479 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:13:43.479 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:13:43.479 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:13:43.480 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:13:43.480 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:13:43.480 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:13:43.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:13:43.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:13:43.481 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:13:43.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:13:43.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:13:43.482 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:13:43.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:13:43.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:13:43.483 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:13:43.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:13:43.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:13:43.484 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:13:43.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:13:43.485 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:13:43.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:13:43.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:13:43.486 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:13:43.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:13:43.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:13:43.487 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=921] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:13:43.488 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=921] Local address: 172.21.15.161, port 59596 2022-07-03T19:13:43.488 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=921] Peer address: 172.21.15.101, port 22 2022-07-03T19:13:43.488 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=921] Connection lost 2022-07-03T19:13:43.489 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=921] Aborting connection 2022-07-03T19:13:43.489 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:43 smithi161 ceph-mon[91579]: pgmap v15436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:13:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:13:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:13:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:13:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:13:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:13:43.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:13:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:13:43.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:13:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:13:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:13:43.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:13:43.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:13:43.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:13:43.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:13:43.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:13:43.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:13:43.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:13:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:13:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:13:43.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:13:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:13:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:13:43.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:13:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:13:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:13:43.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:13:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:13:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:13:43.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:13:43.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:13:43.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:13:43.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:13:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:13:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:13:43.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:13:43.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:13:43.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:13:43.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:13:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:13:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:13:43.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:13:43.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:13:43.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:13:43.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:13:43.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=921] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:13:43.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=921] Local address: 172.21.15.161, port 59596 2022-07-03T19:13:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=921] Peer address: 172.21.15.101, port 22 2022-07-03T19:13:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=921] Connection lost 2022-07-03T19:13:43.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=921] Aborting connection 2022-07-03T19:13:43.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:43 smithi188 ceph-mon[92425]: pgmap v15436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:44 smithi188 ceph-mon[92425]: pgmap v15437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:45.068 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:44 smithi161 ceph-mon[91579]: pgmap v15437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:45 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T19:13:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:13:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:45 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:13:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:45 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T19:13:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:13:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:45 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:13:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:46 smithi188 ceph-mon[92425]: pgmap v15438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:46 smithi161 ceph-mon[91579]: pgmap v15438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:49 smithi161 ceph-mon[91579]: pgmap v15439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:49 smithi188 ceph-mon[92425]: pgmap v15439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:51 smithi161 ceph-mon[91579]: pgmap v15440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:51 smithi188 ceph-mon[92425]: pgmap v15440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:53 smithi161 ceph-mon[91579]: pgmap v15441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:53 smithi188 ceph-mon[92425]: pgmap v15441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:54 smithi188 ceph-mon[92425]: pgmap v15442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:54 smithi161 ceph-mon[91579]: pgmap v15442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:13:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:56 smithi188 ceph-mon[92425]: pgmap v15443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:56 smithi161 ceph-mon[91579]: pgmap v15443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:13:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:59 smithi161 ceph-mon[91579]: pgmap v15444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:13:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:13:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:13:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:13:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:59 smithi188 ceph-mon[92425]: pgmap v15444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:13:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:13:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:13:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:13:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:14:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:01 smithi161 ceph-mon[91579]: pgmap v15445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:01 smithi188 ceph-mon[92425]: pgmap v15445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:03 smithi161 ceph-mon[91579]: pgmap v15446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:03 smithi188 ceph-mon[92425]: pgmap v15446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:04 smithi188 ceph-mon[92425]: pgmap v15447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:04 smithi161 ceph-mon[91579]: pgmap v15447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:06 smithi161 ceph-mon[91579]: pgmap v15448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:06 smithi188 ceph-mon[92425]: pgmap v15448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:09 smithi161 ceph-mon[91579]: pgmap v15449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:09 smithi188 ceph-mon[92425]: pgmap v15449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:11 smithi161 ceph-mon[91579]: pgmap v15450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:11 smithi188 ceph-mon[92425]: pgmap v15450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:13 smithi161 ceph-mon[91579]: pgmap v15451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:13 smithi188 ceph-mon[92425]: pgmap v15451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:14 smithi188 ceph-mon[92425]: pgmap v15452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:14 smithi161 ceph-mon[91579]: pgmap v15452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:16 smithi161 ceph-mon[91579]: pgmap v15453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:16 smithi188 ceph-mon[92425]: pgmap v15453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:19 smithi161 ceph-mon[91579]: pgmap v15454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:19 smithi188 ceph-mon[92425]: pgmap v15454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:21 smithi161 ceph-mon[91579]: pgmap v15455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:21 smithi188 ceph-mon[92425]: pgmap v15455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:23 smithi161 ceph-mon[91579]: pgmap v15456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:23 smithi188 ceph-mon[92425]: pgmap v15456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:24 smithi161 ceph-mon[91579]: pgmap v15457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:24 smithi188 ceph-mon[92425]: pgmap v15457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:26 smithi161 ceph-mon[91579]: pgmap v15458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:26 smithi188 ceph-mon[92425]: pgmap v15458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:29 smithi161 ceph-mon[91579]: pgmap v15459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:29 smithi188 ceph-mon[92425]: pgmap v15459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:31 smithi161 ceph-mon[91579]: pgmap v15460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:31 smithi188 ceph-mon[92425]: pgmap v15460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:33 smithi161 ceph-mon[91579]: pgmap v15461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:33 smithi188 ceph-mon[92425]: pgmap v15461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:34 smithi161 ceph-mon[91579]: pgmap v15462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:34 smithi188 ceph-mon[92425]: pgmap v15462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:36 smithi161 ceph-mon[91579]: pgmap v15463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:36 smithi188 ceph-mon[92425]: pgmap v15463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:39 smithi161 ceph-mon[91579]: pgmap v15464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:39 smithi188 ceph-mon[92425]: pgmap v15464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:41 smithi161 ceph-mon[91579]: pgmap v15465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:41 smithi188 ceph-mon[92425]: pgmap v15465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:43 smithi161 ceph-mon[91579]: pgmap v15466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:43 smithi188 ceph-mon[92425]: pgmap v15466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:44 smithi161 ceph-mon[91579]: pgmap v15467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:44 smithi188 ceph-mon[92425]: pgmap v15467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:45.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:14:45.412+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:14:45.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:14:45.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:14:45.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:14:45.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:14:45.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:14:45.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:14:45.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:14:45.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:14:45.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:14:45.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:14:45.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:14:45.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:14:45.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:14:45.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:14:45.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:14:45.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:14:45.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:14:45.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:14:45.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:14:45.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:14:45.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:14:45.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:14:45.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:14:45.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:14:45.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:14:45.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:14:45.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:14:45.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:14:45.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:14:45.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:14:45.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:14:45.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:14:45.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:14:45.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:14:45.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:14:45.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:14:45.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:14:45.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:14:45.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:14:45.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:14:45.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:14:45.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:14:45.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:14:45.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:14:45.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=923] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:14:45.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=923] Local address: 172.21.15.161, port 59600 2022-07-03T19:14:45.685 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=923] Peer address: 172.21.15.101, port 22 2022-07-03T19:14:45.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=923] Connection lost 2022-07-03T19:14:45.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=923] Aborting connection 2022-07-03T19:14:45.686 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:14:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:14:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:14:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:14:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:14:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:14:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:14:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:14:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:14:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:14:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:14:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:14:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:14:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=923] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:14:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=923] Local address: 172.21.15.161, port 59600 2022-07-03T19:14:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=923] Peer address: 172.21.15.101, port 22 2022-07-03T19:14:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=923] Connection lost 2022-07-03T19:14:45.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=923] Aborting connection 2022-07-03T19:14:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:14:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:14:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:14:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:14:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:14:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:14:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:14:46.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:14:46.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:14:46.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:14:46.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:14:46.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:14:46.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:14:46.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:14:46.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:14:46.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:14:46.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:14:46.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:14:46.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:14:46.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:14:46.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:14:46.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=923] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:14:46.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=923] Local address: 172.21.15.161, port 59600 2022-07-03T19:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=923] Peer address: 172.21.15.101, port 22 2022-07-03T19:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=923] Connection lost 2022-07-03T19:14:46.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=923] Aborting connection 2022-07-03T19:14:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:46 smithi161 ceph-mon[91579]: pgmap v15468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:46 smithi188 ceph-mon[92425]: pgmap v15468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:49 smithi161 ceph-mon[91579]: pgmap v15469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:49 smithi188 ceph-mon[92425]: pgmap v15469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:51 smithi161 ceph-mon[91579]: pgmap v15470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:51 smithi188 ceph-mon[92425]: pgmap v15470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:53 smithi161 ceph-mon[91579]: pgmap v15471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:53 smithi188 ceph-mon[92425]: pgmap v15471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:54 smithi161 ceph-mon[91579]: pgmap v15472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:54 smithi188 ceph-mon[92425]: pgmap v15472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:14:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:56 smithi161 ceph-mon[91579]: pgmap v15473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:56 smithi188 ceph-mon[92425]: pgmap v15473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:14:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:59 smithi161 ceph-mon[91579]: pgmap v15474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:14:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:14:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:14:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:14:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:59 smithi188 ceph-mon[92425]: pgmap v15474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:14:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:14:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:14:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:14:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:15:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:01 smithi161 ceph-mon[91579]: pgmap v15475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:01 smithi188 ceph-mon[92425]: pgmap v15475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:03 smithi161 ceph-mon[91579]: pgmap v15476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:03 smithi188 ceph-mon[92425]: pgmap v15476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:04 smithi161 ceph-mon[91579]: pgmap v15477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:04 smithi188 ceph-mon[92425]: pgmap v15477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:06 smithi161 ceph-mon[91579]: pgmap v15478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:06 smithi188 ceph-mon[92425]: pgmap v15478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:09 smithi161 ceph-mon[91579]: pgmap v15479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:09 smithi188 ceph-mon[92425]: pgmap v15479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:11 smithi161 ceph-mon[91579]: pgmap v15480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:11 smithi188 ceph-mon[92425]: pgmap v15480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:13 smithi161 ceph-mon[91579]: pgmap v15481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:13 smithi188 ceph-mon[92425]: pgmap v15481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:14 smithi161 ceph-mon[91579]: pgmap v15482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:14 smithi188 ceph-mon[92425]: pgmap v15482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:16 smithi161 ceph-mon[91579]: pgmap v15483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:16 smithi188 ceph-mon[92425]: pgmap v15483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:19 smithi161 ceph-mon[91579]: pgmap v15484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:19 smithi188 ceph-mon[92425]: pgmap v15484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:21 smithi161 ceph-mon[91579]: pgmap v15485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:21 smithi188 ceph-mon[92425]: pgmap v15485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:23 smithi161 ceph-mon[91579]: pgmap v15486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:23 smithi188 ceph-mon[92425]: pgmap v15486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:24 smithi161 ceph-mon[91579]: pgmap v15487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:24 smithi188 ceph-mon[92425]: pgmap v15487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:26 smithi161 ceph-mon[91579]: pgmap v15488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:26 smithi188 ceph-mon[92425]: pgmap v15488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:29 smithi161 ceph-mon[91579]: pgmap v15489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:29 smithi188 ceph-mon[92425]: pgmap v15489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:31 smithi161 ceph-mon[91579]: pgmap v15490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:31 smithi188 ceph-mon[92425]: pgmap v15490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:33 smithi161 ceph-mon[91579]: pgmap v15491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:33 smithi188 ceph-mon[92425]: pgmap v15491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:34 smithi161 ceph-mon[91579]: pgmap v15492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:34 smithi188 ceph-mon[92425]: pgmap v15492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:36 smithi161 ceph-mon[91579]: pgmap v15493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:36 smithi188 ceph-mon[92425]: pgmap v15493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:39 smithi161 ceph-mon[91579]: pgmap v15494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:39 smithi188 ceph-mon[92425]: pgmap v15494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:41 smithi161 ceph-mon[91579]: pgmap v15495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:41 smithi188 ceph-mon[92425]: pgmap v15495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:43 smithi161 ceph-mon[91579]: pgmap v15496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:43 smithi188 ceph-mon[92425]: pgmap v15496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:44 smithi161 ceph-mon[91579]: pgmap v15497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:44 smithi188 ceph-mon[92425]: pgmap v15497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:45.933 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:45.934 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:15:45.923+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:15:45.934 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:15:45.935 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:15:45.935 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:15:45.936 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:15:45.936 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:15:45.936 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:15:45.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:15:45.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:15:45.937 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:15:45.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:15:45.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:15:45.938 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:15:45.939 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:15:45.939 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:15:45.939 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:15:45.940 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:15:45.940 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:15:45.940 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:15:45.941 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:15:45.941 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:15:45.941 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:15:45.942 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:15:45.942 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:15:45.942 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:15:45.943 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:15:45.943 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:15:45.943 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:15:45.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:15:45.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:15:45.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:15:45.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:15:45.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:15:45.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:15:45.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:15:45.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:15:45.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:15:45.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:15:45.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:15:45.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:15:45.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:15:45.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:15:45.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:15:45.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:15:45.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:15:45.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=925] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:15:45.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=925] Local address: 172.21.15.161, port 59604 2022-07-03T19:15:45.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=925] Peer address: 172.21.15.101, port 22 2022-07-03T19:15:45.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=925] Connection lost 2022-07-03T19:15:45.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=925] Aborting connection 2022-07-03T19:15:45.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:15:45 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:15:45.952 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:15:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:15:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:15:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:15:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:15:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:15:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:15:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:15:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:15:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:15:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:15:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:15:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:15:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:15:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:15:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:15:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:15:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:15:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:15:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:15:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:15:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:15:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:15:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:15:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:15:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:15:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:15:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:15:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:15:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:15:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:15:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:15:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:15:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:15:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:15:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:15:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:15:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:15:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:15:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:15:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:15:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:15:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:15:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:15:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:15:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:15:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:15:47.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:15:47.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:15:47.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=925] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:15:47.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=925] Local address: 172.21.15.161, port 59604 2022-07-03T19:15:47.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=925] Peer address: 172.21.15.101, port 22 2022-07-03T19:15:47.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=925] Connection lost 2022-07-03T19:15:47.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=925] Aborting connection 2022-07-03T19:15:47.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:46 smithi161 ceph-mon[91579]: pgmap v15498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:15:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:15:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:15:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:15:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:15:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:15:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:15:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:15:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:15:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:15:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:15:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:15:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:15:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:15:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:15:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:15:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:15:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:15:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:15:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:15:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:15:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:15:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:15:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:15:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:15:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:15:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:15:47.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:15:47.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:15:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:15:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:15:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:15:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:15:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:15:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:15:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:15:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:15:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:15:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:15:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:15:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:15:47.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:15:47.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:15:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:15:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:15:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:15:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:15:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:15:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:15:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=925] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:15:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=925] Local address: 172.21.15.161, port 59604 2022-07-03T19:15:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=925] Peer address: 172.21.15.101, port 22 2022-07-03T19:15:47.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=925] Connection lost 2022-07-03T19:15:47.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=925] Aborting connection 2022-07-03T19:15:47.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:46 smithi188 ceph-mon[92425]: pgmap v15498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:49 smithi161 ceph-mon[91579]: pgmap v15499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:49 smithi188 ceph-mon[92425]: pgmap v15499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:51 smithi161 ceph-mon[91579]: pgmap v15500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:51 smithi188 ceph-mon[92425]: pgmap v15500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:53 smithi161 ceph-mon[91579]: pgmap v15501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:53 smithi188 ceph-mon[92425]: pgmap v15501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:54 smithi161 ceph-mon[91579]: pgmap v15502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:54 smithi188 ceph-mon[92425]: pgmap v15502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:15:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:56 smithi161 ceph-mon[91579]: pgmap v15503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:56 smithi188 ceph-mon[92425]: pgmap v15503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:15:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:59 smithi161 ceph-mon[91579]: pgmap v15504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:15:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:15:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:15:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:15:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:59 smithi188 ceph-mon[92425]: pgmap v15504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:15:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:15:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:15:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:15:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:16:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:01 smithi161 ceph-mon[91579]: pgmap v15505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:01 smithi188 ceph-mon[92425]: pgmap v15505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:03 smithi161 ceph-mon[91579]: pgmap v15506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:03 smithi188 ceph-mon[92425]: pgmap v15506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:04 smithi161 ceph-mon[91579]: pgmap v15507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:04 smithi188 ceph-mon[92425]: pgmap v15507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:06 smithi161 ceph-mon[91579]: pgmap v15508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:06 smithi188 ceph-mon[92425]: pgmap v15508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:09 smithi161 ceph-mon[91579]: pgmap v15509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:09 smithi188 ceph-mon[92425]: pgmap v15509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:11 smithi161 ceph-mon[91579]: pgmap v15510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:11 smithi188 ceph-mon[92425]: pgmap v15510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:13 smithi161 ceph-mon[91579]: pgmap v15511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:13 smithi188 ceph-mon[92425]: pgmap v15511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:14 smithi161 ceph-mon[91579]: pgmap v15512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:14 smithi188 ceph-mon[92425]: pgmap v15512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:16 smithi161 ceph-mon[91579]: pgmap v15513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:16 smithi188 ceph-mon[92425]: pgmap v15513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:19 smithi161 ceph-mon[91579]: pgmap v15514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:19 smithi188 ceph-mon[92425]: pgmap v15514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:21 smithi161 ceph-mon[91579]: pgmap v15515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:21 smithi188 ceph-mon[92425]: pgmap v15515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:23 smithi161 ceph-mon[91579]: pgmap v15516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:23 smithi188 ceph-mon[92425]: pgmap v15516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:24 smithi161 ceph-mon[91579]: pgmap v15517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:24 smithi188 ceph-mon[92425]: pgmap v15517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:26 smithi161 ceph-mon[91579]: pgmap v15518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:26 smithi188 ceph-mon[92425]: pgmap v15518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:29 smithi161 ceph-mon[91579]: pgmap v15519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:29 smithi188 ceph-mon[92425]: pgmap v15519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:31 smithi161 ceph-mon[91579]: pgmap v15520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:31 smithi188 ceph-mon[92425]: pgmap v15520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:33 smithi161 ceph-mon[91579]: pgmap v15521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:33 smithi188 ceph-mon[92425]: pgmap v15521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:34 smithi161 ceph-mon[91579]: pgmap v15522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:34 smithi188 ceph-mon[92425]: pgmap v15522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:36 smithi161 ceph-mon[91579]: pgmap v15523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:36 smithi188 ceph-mon[92425]: pgmap v15523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:39 smithi161 ceph-mon[91579]: pgmap v15524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:39 smithi188 ceph-mon[92425]: pgmap v15524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:41 smithi161 ceph-mon[91579]: pgmap v15525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:41 smithi188 ceph-mon[92425]: pgmap v15525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:43 smithi161 ceph-mon[91579]: pgmap v15526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:43 smithi188 ceph-mon[92425]: pgmap v15526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:44 smithi161 ceph-mon[91579]: pgmap v15527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:44 smithi188 ceph-mon[92425]: pgmap v15527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:46.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:16:46.421+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:16:46.702 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:16:46.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:16:46.703 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:16:46.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:16:46.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:16:46.704 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:16:46.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:16:46.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:16:46.705 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:16:46.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:16:46.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:16:46.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:16:46.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:16:46.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:16:46.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:16:46.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:16:46.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:16:46.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:16:46.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:16:46.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:16:46.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:16:46.710 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:16:46.710 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:16:46.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:16:46.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:16:46.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:16:46.712 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:16:46.712 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:16:46.713 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:16:46.714 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:16:46.714 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:16:46.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:16:46.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:16:46.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:16:46.716 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:16:46.716 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:16:46.716 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:16:46.717 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:16:46.718 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:16:46.718 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:16:46.719 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:16:46.720 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:16:46.720 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:16:46.721 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:16:46.722 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=927] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:16:46.722 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=927] Local address: 172.21.15.161, port 59608 2022-07-03T19:16:46.723 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=927] Peer address: 172.21.15.101, port 22 2022-07-03T19:16:46.723 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=927] Connection lost 2022-07-03T19:16:46.724 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=927] Aborting connection 2022-07-03T19:16:46.725 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:16:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: pgmap v15528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:16:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:16:47.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:16:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:16:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:46 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:16:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:16:47.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:16:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:16:47.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:16:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:16:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:16:47.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:16:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:16:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:16:47.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:16:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:16:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:16:47.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:16:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:16:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:16:47.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:16:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:16:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:16:47.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:16:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:16:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:16:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:16:47.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:16:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:16:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:16:47.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:16:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:16:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:16:47.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:16:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:16:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:16:47.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:16:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:16:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:16:47.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:16:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:16:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:16:47.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:16:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:16:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:16:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:16:47.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:16:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:16:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:16:47.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=927] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:16:47.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=927] Local address: 172.21.15.161, port 59608 2022-07-03T19:16:47.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=927] Peer address: 172.21.15.101, port 22 2022-07-03T19:16:47.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=927] Connection lost 2022-07-03T19:16:47.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=927] Aborting connection 2022-07-03T19:16:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: pgmap v15528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:16:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:16:47.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:16:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:16:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:46 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:16:47.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:16:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:16:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:16:47.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:16:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:16:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:16:47.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:16:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:16:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:16:47.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:16:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:16:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:16:47.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:16:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:16:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:16:47.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:16:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:16:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:16:47.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:16:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:16:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:16:47.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:16:47.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:16:47.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:16:47.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:16:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:16:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:16:47.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:16:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:16:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:16:47.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:16:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:16:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:16:47.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:16:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:16:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:16:47.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:16:47.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:16:47.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:16:47.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:16:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:16:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:16:47.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:16:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:16:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=927] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:16:47.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=927] Local address: 172.21.15.161, port 59608 2022-07-03T19:16:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=927] Peer address: 172.21.15.101, port 22 2022-07-03T19:16:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=927] Connection lost 2022-07-03T19:16:47.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=927] Aborting connection 2022-07-03T19:16:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:49 smithi161 ceph-mon[91579]: pgmap v15529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:49 smithi188 ceph-mon[92425]: pgmap v15529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:51 smithi161 ceph-mon[91579]: pgmap v15530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:51 smithi188 ceph-mon[92425]: pgmap v15530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:53 smithi161 ceph-mon[91579]: pgmap v15531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:53 smithi188 ceph-mon[92425]: pgmap v15531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:54 smithi161 ceph-mon[91579]: pgmap v15532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:54 smithi188 ceph-mon[92425]: pgmap v15532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:16:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:56 smithi161 ceph-mon[91579]: pgmap v15533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:56 smithi188 ceph-mon[92425]: pgmap v15533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:16:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:59 smithi161 ceph-mon[91579]: pgmap v15534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:16:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:16:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:16:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:16:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:59 smithi188 ceph-mon[92425]: pgmap v15534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:16:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:16:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:16:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:16:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:17:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:01 smithi161 ceph-mon[91579]: pgmap v15535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:01 smithi188 ceph-mon[92425]: pgmap v15535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:03 smithi161 ceph-mon[91579]: pgmap v15536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:03 smithi188 ceph-mon[92425]: pgmap v15536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:04 smithi161 ceph-mon[91579]: pgmap v15537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:04 smithi188 ceph-mon[92425]: pgmap v15537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:06 smithi161 ceph-mon[91579]: pgmap v15538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:06 smithi188 ceph-mon[92425]: pgmap v15538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:09 smithi161 ceph-mon[91579]: pgmap v15539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:09 smithi188 ceph-mon[92425]: pgmap v15539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:11 smithi161 ceph-mon[91579]: pgmap v15540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:11 smithi188 ceph-mon[92425]: pgmap v15540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:13 smithi161 ceph-mon[91579]: pgmap v15541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:13 smithi188 ceph-mon[92425]: pgmap v15541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:14 smithi161 ceph-mon[91579]: pgmap v15542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:14 smithi188 ceph-mon[92425]: pgmap v15542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:16 smithi161 ceph-mon[91579]: pgmap v15543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:16 smithi188 ceph-mon[92425]: pgmap v15543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:19 smithi161 ceph-mon[91579]: pgmap v15544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:19 smithi188 ceph-mon[92425]: pgmap v15544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:21 smithi161 ceph-mon[91579]: pgmap v15545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:21 smithi188 ceph-mon[92425]: pgmap v15545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:23 smithi161 ceph-mon[91579]: pgmap v15546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:23 smithi188 ceph-mon[92425]: pgmap v15546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:24 smithi161 ceph-mon[91579]: pgmap v15547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:24 smithi188 ceph-mon[92425]: pgmap v15547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:26 smithi161 ceph-mon[91579]: pgmap v15548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:26 smithi188 ceph-mon[92425]: pgmap v15548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:29 smithi161 ceph-mon[91579]: pgmap v15549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:29 smithi188 ceph-mon[92425]: pgmap v15549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:31 smithi161 ceph-mon[91579]: pgmap v15550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:31 smithi188 ceph-mon[92425]: pgmap v15550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:33 smithi161 ceph-mon[91579]: pgmap v15551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:33 smithi188 ceph-mon[92425]: pgmap v15551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:34 smithi161 ceph-mon[91579]: pgmap v15552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:34 smithi188 ceph-mon[92425]: pgmap v15552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:36 smithi161 ceph-mon[91579]: pgmap v15553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:36 smithi188 ceph-mon[92425]: pgmap v15553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:39 smithi161 ceph-mon[91579]: pgmap v15554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:39 smithi188 ceph-mon[92425]: pgmap v15554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:41 smithi161 ceph-mon[91579]: pgmap v15555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:41.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:41 smithi188 ceph-mon[92425]: pgmap v15555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:43 smithi161 ceph-mon[91579]: pgmap v15556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:43 smithi188 ceph-mon[92425]: pgmap v15556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:44 smithi161 ceph-mon[91579]: pgmap v15557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:44 smithi188 ceph-mon[92425]: pgmap v15557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:46.976 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-mon[91579]: pgmap v15558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:46.977 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:17:46.919+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:17:46.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:17:46.978 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:17:46.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:17:46.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:17:46.979 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:17:46.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:17:46.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:17:46.980 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:17:46.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:17:46.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:17:46.981 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:17:46.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:17:46.982 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:17:46.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:17:46.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:17:46.983 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:17:46.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:17:46.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:17:46.984 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:17:46.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:17:46.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:17:46.985 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:17:46.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:17:46.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:17:46.986 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:17:46.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:17:46.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:17:46.987 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:17:46.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:17:46.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:17:46.988 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:17:46.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:17:46.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:17:46.989 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:17:46.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:17:46.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:17:46.990 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:17:46.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:17:46.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:17:46.991 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:17:46.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:17:46.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:17:46.992 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:17:46.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:17:46.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=929] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:17:46.993 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=929] Local address: 172.21.15.161, port 59612 2022-07-03T19:17:46.994 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=929] Peer address: 172.21.15.101, port 22 2022-07-03T19:17:46.994 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=929] Connection lost 2022-07-03T19:17:46.994 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=929] Aborting connection 2022-07-03T19:17:46.995 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:17:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:17:46.995 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:46 smithi188 ceph-mon[92425]: pgmap v15558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:17:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:17:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:17:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:17:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:17:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:17:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:17:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:17:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:17:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:17:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:17:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:17:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:17:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:17:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:17:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:17:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:17:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:17:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:17:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:17:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:17:48.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:17:48.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:17:48.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:17:48.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:17:48.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:17:48.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:17:48.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:17:48.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:17:48.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:17:48.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:17:48.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:17:48.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:17:48.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:17:48.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:17:48.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:17:48.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:17:48.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:17:48.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:17:48.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:17:48.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:17:48.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:17:48.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:17:48.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:17:48.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:17:48.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:17:48.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:17:48.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:17:48.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:17:48.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=929] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:17:48.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=929] Local address: 172.21.15.161, port 59612 2022-07-03T19:17:48.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=929] Peer address: 172.21.15.101, port 22 2022-07-03T19:17:48.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=929] Connection lost 2022-07-03T19:17:48.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=929] Aborting connection 2022-07-03T19:17:48.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:17:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:17:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:17:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:17:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:17:48.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:17:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:17:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:17:48.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:17:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:17:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:17:48.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:17:48.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:17:48.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:17:48.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:17:48.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:17:48.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:17:48.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:17:48.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:17:48.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:17:48.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:17:48.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:17:48.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:17:48.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:17:48.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:17:48.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:17:48.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:17:48.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:17:48.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:17:48.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:17:48.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:17:48.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:17:48.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:17:48.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:17:48.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:17:48.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:17:48.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:17:48.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:17:48.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:17:48.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:17:48.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:17:48.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:17:48.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:17:48.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:17:48.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:17:48.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:17:48.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:17:48.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:17:48.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:17:48.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=929] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:17:48.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=929] Local address: 172.21.15.161, port 59612 2022-07-03T19:17:48.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=929] Peer address: 172.21.15.101, port 22 2022-07-03T19:17:48.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=929] Connection lost 2022-07-03T19:17:48.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=929] Aborting connection 2022-07-03T19:17:48.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:48 smithi188 ceph-mon[92425]: pgmap v15559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:48 smithi161 ceph-mon[91579]: pgmap v15559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:51 smithi161 ceph-mon[91579]: pgmap v15560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:51 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:51 smithi188 ceph-mon[92425]: pgmap v15560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:51 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:17:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:52 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T19:17:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:52 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T19:17:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:53 smithi161 ceph-mon[91579]: pgmap v15561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:53 smithi188 ceph-mon[92425]: pgmap v15561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:54 smithi161 ceph-mon[91579]: pgmap v15562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:54 smithi188 ceph-mon[92425]: pgmap v15562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:17:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:56 smithi161 ceph-mon[91579]: pgmap v15563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:56 smithi188 ceph-mon[92425]: pgmap v15563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:17:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:59 smithi161 ceph-mon[91579]: pgmap v15564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:17:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:17:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:17:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:17:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:59 smithi188 ceph-mon[92425]: pgmap v15564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:17:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:17:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:17:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:17:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:18:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:01 smithi161 ceph-mon[91579]: pgmap v15565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:01 smithi188 ceph-mon[92425]: pgmap v15565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:03 smithi161 ceph-mon[91579]: pgmap v15566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:03 smithi188 ceph-mon[92425]: pgmap v15566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:04 smithi161 ceph-mon[91579]: pgmap v15567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:04 smithi188 ceph-mon[92425]: pgmap v15567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:06 smithi161 ceph-mon[91579]: pgmap v15568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:06 smithi188 ceph-mon[92425]: pgmap v15568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:09 smithi161 ceph-mon[91579]: pgmap v15569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:09 smithi188 ceph-mon[92425]: pgmap v15569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:11 smithi161 ceph-mon[91579]: pgmap v15570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:11 smithi188 ceph-mon[92425]: pgmap v15570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:13 smithi161 ceph-mon[91579]: pgmap v15571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:13 smithi188 ceph-mon[92425]: pgmap v15571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:14 smithi161 ceph-mon[91579]: pgmap v15572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:14 smithi188 ceph-mon[92425]: pgmap v15572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:16 smithi161 ceph-mon[91579]: pgmap v15573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:16 smithi188 ceph-mon[92425]: pgmap v15573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:19 smithi161 ceph-mon[91579]: pgmap v15574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:19 smithi188 ceph-mon[92425]: pgmap v15574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:21 smithi161 ceph-mon[91579]: pgmap v15575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:21 smithi188 ceph-mon[92425]: pgmap v15575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:23 smithi161 ceph-mon[91579]: pgmap v15576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:23 smithi188 ceph-mon[92425]: pgmap v15576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:24 smithi161 ceph-mon[91579]: pgmap v15577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:24 smithi188 ceph-mon[92425]: pgmap v15577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:26 smithi161 ceph-mon[91579]: pgmap v15578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:26 smithi188 ceph-mon[92425]: pgmap v15578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:29 smithi161 ceph-mon[91579]: pgmap v15579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:29 smithi188 ceph-mon[92425]: pgmap v15579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:31 smithi161 ceph-mon[91579]: pgmap v15580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:31 smithi188 ceph-mon[92425]: pgmap v15580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:33 smithi161 ceph-mon[91579]: pgmap v15581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:33 smithi188 ceph-mon[92425]: pgmap v15581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:34 smithi161 ceph-mon[91579]: pgmap v15582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:34 smithi188 ceph-mon[92425]: pgmap v15582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:36 smithi161 ceph-mon[91579]: pgmap v15583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:36 smithi188 ceph-mon[92425]: pgmap v15583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:39 smithi161 ceph-mon[91579]: pgmap v15584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:39 smithi188 ceph-mon[92425]: pgmap v15584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:41 smithi161 ceph-mon[91579]: pgmap v15585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:41 smithi188 ceph-mon[92425]: pgmap v15585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:43 smithi161 ceph-mon[91579]: pgmap v15586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:43 smithi188 ceph-mon[92425]: pgmap v15586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:44 smithi161 ceph-mon[91579]: pgmap v15587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:44 smithi188 ceph-mon[92425]: pgmap v15587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:46 smithi161 ceph-mon[91579]: pgmap v15588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:46 smithi188 ceph-mon[92425]: pgmap v15588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:49 smithi161 ceph-mon[91579]: pgmap v15589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:49 smithi188 ceph-mon[92425]: pgmap v15589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:51.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:18:50.735+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:18:51.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:18:51.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:18:51.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:18:51.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:18:51.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:18:51.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:18:51.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:18:51.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:18:51.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:18:51.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:18:51.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:18:51.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:18:51.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:18:51.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:18:51.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:18:51.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:18:51.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:18:51.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:18:51.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:18:51.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:18:51.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:18:51.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:18:51.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:18:51.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:18:51.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:18:51.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:18:51.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:18:51.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:18:51.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:18:51.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:18:51.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:18:51.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:18:51.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:18:51.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:18:51.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:18:51.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:18:51.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:18:51.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:18:51.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:18:51.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:18:51.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:18:51.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:18:51.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:18:51.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:18:51.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=931] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:18:51.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=931] Local address: 172.21.15.161, port 59616 2022-07-03T19:18:51.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=931] Peer address: 172.21.15.101, port 22 2022-07-03T19:18:51.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=931] Connection lost 2022-07-03T19:18:51.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=931] Aborting connection 2022-07-03T19:18:51.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:18:50 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:18:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:51 smithi161 ceph-mon[91579]: pgmap v15590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:18:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:18:51.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:18:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:51 smithi188 ceph-mon[92425]: pgmap v15590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:18:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:18:51.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:18:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:18:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:18:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:18:52.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:18:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:18:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:18:52.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:18:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:18:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:18:52.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:18:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:18:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:18:52.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:18:52.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:18:52.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:18:52.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:18:52.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:18:52.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:18:52.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:18:52.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:18:52.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:18:52.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:18:52.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:18:52.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:18:52.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:18:52.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:18:52.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:18:52.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:18:52.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:18:52.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:18:52.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:18:52.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:18:52.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:18:52.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:18:52.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:18:52.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:18:52.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:18:52.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:18:52.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:18:52.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:18:52.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:18:52.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:18:52.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:18:52.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:18:52.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:18:52.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:18:52.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=931] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:18:52.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=931] Local address: 172.21.15.161, port 59616 2022-07-03T19:18:52.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=931] Peer address: 172.21.15.101, port 22 2022-07-03T19:18:52.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=931] Connection lost 2022-07-03T19:18:52.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=931] Aborting connection 2022-07-03T19:18:52.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:18:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:18:52.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:18:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:18:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:18:52.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:18:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:18:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:18:52.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:18:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:18:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:18:52.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:18:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:18:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:18:52.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:18:52.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:18:52.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:18:52.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:18:52.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:18:52.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:18:52.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:18:52.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:18:52.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:18:52.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:18:52.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:18:52.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:18:52.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:18:52.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:18:52.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:18:52.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:18:52.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:18:52.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:18:52.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:18:52.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:18:52.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:18:52.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:18:52.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:18:52.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:18:52.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:18:52.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:18:52.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:18:52.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:18:52.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:18:52.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:18:52.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:18:52.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:18:52.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=931] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:18:52.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=931] Local address: 172.21.15.161, port 59616 2022-07-03T19:18:52.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=931] Peer address: 172.21.15.101, port 22 2022-07-03T19:18:52.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=931] Connection lost 2022-07-03T19:18:52.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=931] Aborting connection 2022-07-03T19:18:52.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:52.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:18:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:53 smithi161 ceph-mon[91579]: pgmap v15591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:53 smithi188 ceph-mon[92425]: pgmap v15591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:54 smithi161 ceph-mon[91579]: pgmap v15592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:54 smithi188 ceph-mon[92425]: pgmap v15592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:18:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:56 smithi161 ceph-mon[91579]: pgmap v15593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:56 smithi188 ceph-mon[92425]: pgmap v15593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:18:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:59 smithi161 ceph-mon[91579]: pgmap v15594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:18:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:18:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:18:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:18:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:59 smithi188 ceph-mon[92425]: pgmap v15594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:18:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:18:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:18:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:18:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:19:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:01 smithi161 ceph-mon[91579]: pgmap v15595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:01 smithi188 ceph-mon[92425]: pgmap v15595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:03 smithi161 ceph-mon[91579]: pgmap v15596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:03 smithi188 ceph-mon[92425]: pgmap v15596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:04 smithi161 ceph-mon[91579]: pgmap v15597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:04 smithi188 ceph-mon[92425]: pgmap v15597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:06 smithi161 ceph-mon[91579]: pgmap v15598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:06 smithi188 ceph-mon[92425]: pgmap v15598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:09 smithi161 ceph-mon[91579]: pgmap v15599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:09 smithi188 ceph-mon[92425]: pgmap v15599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:11 smithi161 ceph-mon[91579]: pgmap v15600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:11 smithi188 ceph-mon[92425]: pgmap v15600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:13 smithi161 ceph-mon[91579]: pgmap v15601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:13 smithi188 ceph-mon[92425]: pgmap v15601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:14 smithi161 ceph-mon[91579]: pgmap v15602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:14 smithi188 ceph-mon[92425]: pgmap v15602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:16 smithi161 ceph-mon[91579]: pgmap v15603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:16 smithi188 ceph-mon[92425]: pgmap v15603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:19 smithi161 ceph-mon[91579]: pgmap v15604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:19 smithi188 ceph-mon[92425]: pgmap v15604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:21 smithi161 ceph-mon[91579]: pgmap v15605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:21 smithi188 ceph-mon[92425]: pgmap v15605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:23 smithi161 ceph-mon[91579]: pgmap v15606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:23 smithi188 ceph-mon[92425]: pgmap v15606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:24 smithi161 ceph-mon[91579]: pgmap v15607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:24 smithi188 ceph-mon[92425]: pgmap v15607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:26 smithi161 ceph-mon[91579]: pgmap v15608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:26 smithi188 ceph-mon[92425]: pgmap v15608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:29 smithi161 ceph-mon[91579]: pgmap v15609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:29 smithi188 ceph-mon[92425]: pgmap v15609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:31 smithi161 ceph-mon[91579]: pgmap v15610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:31 smithi188 ceph-mon[92425]: pgmap v15610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:33 smithi161 ceph-mon[91579]: pgmap v15611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:33 smithi188 ceph-mon[92425]: pgmap v15611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:34 smithi161 ceph-mon[91579]: pgmap v15612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:34 smithi188 ceph-mon[92425]: pgmap v15612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:36 smithi161 ceph-mon[91579]: pgmap v15613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:36 smithi188 ceph-mon[92425]: pgmap v15613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:39 smithi161 ceph-mon[91579]: pgmap v15614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:39 smithi188 ceph-mon[92425]: pgmap v15614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:41 smithi161 ceph-mon[91579]: pgmap v15615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:41 smithi188 ceph-mon[92425]: pgmap v15615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:43 smithi161 ceph-mon[91579]: pgmap v15616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:43 smithi188 ceph-mon[92425]: pgmap v15616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:44 smithi161 ceph-mon[91579]: pgmap v15617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:44 smithi188 ceph-mon[92425]: pgmap v15617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:46 smithi161 ceph-mon[91579]: pgmap v15618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:46 smithi188 ceph-mon[92425]: pgmap v15618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:49 smithi161 ceph-mon[91579]: pgmap v15619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:49 smithi188 ceph-mon[92425]: pgmap v15619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-mon[91579]: pgmap v15620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:51.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:51 smithi188 ceph-mon[92425]: pgmap v15620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:51.655 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:52.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:19:51.645+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:19:52.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:19:52.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:19:52.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:19:52.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:19:52.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:19:52.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:19:52.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:19:52.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:19:52.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:19:52.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:19:52.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:19:52.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:19:52.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:19:52.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:19:52.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:19:52.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:19:52.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:19:52.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:19:52.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:19:52.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:19:52.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:19:52.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:19:52.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:19:52.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:19:52.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:19:52.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:19:52.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:19:52.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:19:52.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:19:52.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:19:52.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:19:52.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:19:52.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:19:52.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:19:52.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:19:52.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:19:52.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:19:52.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:19:52.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:19:52.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:19:52.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:19:52.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:19:52.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:19:52.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:19:52.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=933] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:19:52.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=933] Local address: 172.21.15.161, port 59620 2022-07-03T19:19:52.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=933] Peer address: 172.21.15.101, port 22 2022-07-03T19:19:52.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=933] Connection lost 2022-07-03T19:19:52.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=933] Aborting connection 2022-07-03T19:19:52.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:19:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:19:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:19:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:19:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:52 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:19:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:52 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:19:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:19:52.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:19:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:19:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:19:52.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:19:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:19:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:19:52.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:19:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:19:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:19:52.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:19:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:19:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:19:52.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:19:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:19:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:19:52.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:19:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:19:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:19:52.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:19:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:19:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:19:52.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:19:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:19:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:19:52.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:19:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:19:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:19:52.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:19:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:19:52.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:19:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:19:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:19:52.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:19:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:19:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:19:52.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:19:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:19:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:19:52.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:19:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:19:52.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:19:52.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:19:52.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:19:52.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=933] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:19:52.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=933] Local address: 172.21.15.161, port 59620 2022-07-03T19:19:52.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=933] Peer address: 172.21.15.101, port 22 2022-07-03T19:19:52.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=933] Connection lost 2022-07-03T19:19:52.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=933] Aborting connection 2022-07-03T19:19:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:19:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:19:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:19:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:52 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:19:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:52 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:19:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:19:52.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:19:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:19:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:19:52.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:19:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:19:52.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:19:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:19:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:19:52.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:19:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:19:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:19:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:19:52.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:19:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:19:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:19:52.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:19:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:19:52.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:19:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:19:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:19:52.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:19:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:19:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:19:52.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:19:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:19:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:19:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:19:52.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:19:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:19:52.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:19:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:19:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:19:52.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:19:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:19:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:19:52.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:19:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:19:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:19:52.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:19:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:19:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:19:52.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:19:52.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:19:52.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=933] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:19:52.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=933] Local address: 172.21.15.161, port 59620 2022-07-03T19:19:52.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=933] Peer address: 172.21.15.101, port 22 2022-07-03T19:19:52.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=933] Connection lost 2022-07-03T19:19:52.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=933] Aborting connection 2022-07-03T19:19:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:53 smithi161 ceph-mon[91579]: pgmap v15621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:53 smithi188 ceph-mon[92425]: pgmap v15621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:54 smithi161 ceph-mon[91579]: pgmap v15622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:54 smithi188 ceph-mon[92425]: pgmap v15622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:19:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:56 smithi161 ceph-mon[91579]: pgmap v15623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:56 smithi188 ceph-mon[92425]: pgmap v15623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:19:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:59 smithi161 ceph-mon[91579]: pgmap v15624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:19:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:19:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:19:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:19:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:59 smithi188 ceph-mon[92425]: pgmap v15624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:19:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:19:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:19:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:19:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:20:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:20:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:20:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:01 smithi161 ceph-mon[91579]: pgmap v15625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:01 smithi188 ceph-mon[92425]: pgmap v15625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:03 smithi161 ceph-mon[91579]: pgmap v15626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:03 smithi188 ceph-mon[92425]: pgmap v15626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:04 smithi161 ceph-mon[91579]: pgmap v15627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:04 smithi188 ceph-mon[92425]: pgmap v15627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:06 smithi161 ceph-mon[91579]: pgmap v15628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:06 smithi188 ceph-mon[92425]: pgmap v15628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:09 smithi161 ceph-mon[91579]: pgmap v15629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:09 smithi188 ceph-mon[92425]: pgmap v15629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:11 smithi161 ceph-mon[91579]: pgmap v15630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:11 smithi188 ceph-mon[92425]: pgmap v15630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:13 smithi161 ceph-mon[91579]: pgmap v15631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:13 smithi188 ceph-mon[92425]: pgmap v15631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:14 smithi161 ceph-mon[91579]: pgmap v15632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:14 smithi188 ceph-mon[92425]: pgmap v15632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:16 smithi161 ceph-mon[91579]: pgmap v15633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:16 smithi188 ceph-mon[92425]: pgmap v15633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:19 smithi161 ceph-mon[91579]: pgmap v15634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:19 smithi188 ceph-mon[92425]: pgmap v15634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:21 smithi161 ceph-mon[91579]: pgmap v15635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:21 smithi188 ceph-mon[92425]: pgmap v15635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:23 smithi161 ceph-mon[91579]: pgmap v15636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:23 smithi188 ceph-mon[92425]: pgmap v15636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:24 smithi161 ceph-mon[91579]: pgmap v15637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:24 smithi188 ceph-mon[92425]: pgmap v15637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:26 smithi161 ceph-mon[91579]: pgmap v15638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:26 smithi188 ceph-mon[92425]: pgmap v15638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:29 smithi161 ceph-mon[91579]: pgmap v15639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:29 smithi188 ceph-mon[92425]: pgmap v15639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:31 smithi161 ceph-mon[91579]: pgmap v15640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:31 smithi188 ceph-mon[92425]: pgmap v15640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:33 smithi161 ceph-mon[91579]: pgmap v15641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:33 smithi188 ceph-mon[92425]: pgmap v15641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:34 smithi161 ceph-mon[91579]: pgmap v15642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:34 smithi188 ceph-mon[92425]: pgmap v15642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:36 smithi161 ceph-mon[91579]: pgmap v15643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:36 smithi188 ceph-mon[92425]: pgmap v15643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:39 smithi161 ceph-mon[91579]: pgmap v15644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:39 smithi188 ceph-mon[92425]: pgmap v15644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:41 smithi161 ceph-mon[91579]: pgmap v15645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:41.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:41 smithi188 ceph-mon[92425]: pgmap v15645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:41.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:43 smithi161 ceph-mon[91579]: pgmap v15646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:43 smithi188 ceph-mon[92425]: pgmap v15646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:44 smithi161 ceph-mon[91579]: pgmap v15647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:44 smithi188 ceph-mon[92425]: pgmap v15647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:46 smithi161 ceph-mon[91579]: pgmap v15648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:46 smithi188 ceph-mon[92425]: pgmap v15648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:49 smithi161 ceph-mon[91579]: pgmap v15649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:49 smithi188 ceph-mon[92425]: pgmap v15649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:51 smithi161 ceph-mon[91579]: pgmap v15650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:51 smithi188 ceph-mon[92425]: pgmap v15650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:20:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:20:52.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:20:52.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:20:52.152+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:20:52.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:20:52.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:20:52.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:20:52.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:20:52.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:20:52.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:20:52.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:20:52.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:20:52.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:20:52.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:20:52.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:20:52.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:20:52.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:20:52.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:20:52.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:20:52.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:20:52.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:20:52.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:20:52.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:20:52.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:20:52.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:20:52.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:20:52.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:20:52.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:20:52.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:20:52.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:20:52.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:20:52.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:20:52.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:20:52.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:20:52.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:20:52.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:20:52.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:20:52.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:20:52.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:20:52.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:20:52.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:20:52.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:20:52.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:20:52.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:20:52.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:20:52.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:20:52.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:20:52.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:20:52.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=935] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:20:52.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=935] Local address: 172.21.15.161, port 59624 2022-07-03T19:20:52.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=935] Peer address: 172.21.15.101, port 22 2022-07-03T19:20:52.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=935] Connection lost 2022-07-03T19:20:52.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=935] Aborting connection 2022-07-03T19:20:52.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:20:52.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:20:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:20:52.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:20:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:20:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:20:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:20:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:20:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:20:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:20:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:20:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:20:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:20:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:20:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:20:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:20:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:20:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:20:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:20:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:20:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:20:53.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:20:53.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:20:53.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:20:53.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:20:53.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:20:53.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:20:53.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:20:53.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:20:53.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:20:53.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:20:53.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:20:53.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:20:53.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:20:53.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:20:53.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:20:53.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:20:53.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:20:53.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:20:53.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:20:53.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:20:53.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:20:53.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:20:53.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:20:53.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:20:53.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:20:53.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:20:53.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:20:53.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=935] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:20:53.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=935] Local address: 172.21.15.161, port 59624 2022-07-03T19:20:53.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=935] Peer address: 172.21.15.101, port 22 2022-07-03T19:20:53.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=935] Connection lost 2022-07-03T19:20:53.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=935] Aborting connection 2022-07-03T19:20:53.659 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:53 smithi161 ceph-mon[91579]: pgmap v15651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:20:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:20:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:20:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:20:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:20:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:20:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:20:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:20:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:20:53.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:20:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:20:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:20:53.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:20:53.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:20:53.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:20:53.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:20:53.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:20:53.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:20:53.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:20:53.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:20:53.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:20:53.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:20:53.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:20:53.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:20:53.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:20:53.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:20:53.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:20:53.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:20:53.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:20:53.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:20:53.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:20:53.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:20:53.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:20:53.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:20:53.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:20:53.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:20:53.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:20:53.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:20:53.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:20:53.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:20:53.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:20:53.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:20:53.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:20:53.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:20:53.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:20:53.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:20:53.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=935] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:20:53.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=935] Local address: 172.21.15.161, port 59624 2022-07-03T19:20:53.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=935] Peer address: 172.21.15.101, port 22 2022-07-03T19:20:53.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=935] Connection lost 2022-07-03T19:20:53.691 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=935] Aborting connection 2022-07-03T19:20:53.692 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:53 smithi188 ceph-mon[92425]: pgmap v15651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:54 smithi161 ceph-mon[91579]: pgmap v15652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:54 smithi188 ceph-mon[92425]: pgmap v15652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 112.2M 2022-07-03T19:20:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 117707743: error parsing value: Value '117707743' is below minimum 939524096 2022-07-03T19:20:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:20:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 112.2M 2022-07-03T19:20:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 117707743: error parsing value: Value '117707743' is below minimum 939524096 2022-07-03T19:20:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:20:56.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:20:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:56 smithi161 ceph-mon[91579]: pgmap v15653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:56 smithi188 ceph-mon[92425]: pgmap v15653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:20:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:59 smithi161 ceph-mon[91579]: pgmap v15654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:20:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:20:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:20:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:20:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:59 smithi188 ceph-mon[92425]: pgmap v15654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:20:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:20:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:20:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:20:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:21:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:01 smithi161 ceph-mon[91579]: pgmap v15655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:01 smithi188 ceph-mon[92425]: pgmap v15655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:03 smithi161 ceph-mon[91579]: pgmap v15656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:03 smithi188 ceph-mon[92425]: pgmap v15656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:04 smithi161 ceph-mon[91579]: pgmap v15657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:04 smithi188 ceph-mon[92425]: pgmap v15657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:06 smithi161 ceph-mon[91579]: pgmap v15658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:06 smithi188 ceph-mon[92425]: pgmap v15658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:09 smithi161 ceph-mon[91579]: pgmap v15659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:09 smithi188 ceph-mon[92425]: pgmap v15659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:11 smithi161 ceph-mon[91579]: pgmap v15660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:11 smithi188 ceph-mon[92425]: pgmap v15660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:13 smithi161 ceph-mon[91579]: pgmap v15661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:13 smithi188 ceph-mon[92425]: pgmap v15661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:14 smithi161 ceph-mon[91579]: pgmap v15662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:14 smithi188 ceph-mon[92425]: pgmap v15662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:16 smithi161 ceph-mon[91579]: pgmap v15663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:16 smithi188 ceph-mon[92425]: pgmap v15663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:19 smithi161 ceph-mon[91579]: pgmap v15664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:19 smithi188 ceph-mon[92425]: pgmap v15664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:21 smithi161 ceph-mon[91579]: pgmap v15665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:21 smithi188 ceph-mon[92425]: pgmap v15665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:23 smithi161 ceph-mon[91579]: pgmap v15666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:23 smithi188 ceph-mon[92425]: pgmap v15666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:24 smithi161 ceph-mon[91579]: pgmap v15667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:24 smithi188 ceph-mon[92425]: pgmap v15667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:26 smithi161 ceph-mon[91579]: pgmap v15668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:26 smithi188 ceph-mon[92425]: pgmap v15668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:29 smithi161 ceph-mon[91579]: pgmap v15669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:29 smithi188 ceph-mon[92425]: pgmap v15669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:31 smithi161 ceph-mon[91579]: pgmap v15670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:31 smithi188 ceph-mon[92425]: pgmap v15670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:33 smithi161 ceph-mon[91579]: pgmap v15671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:33 smithi188 ceph-mon[92425]: pgmap v15671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:34 smithi161 ceph-mon[91579]: pgmap v15672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:34 smithi188 ceph-mon[92425]: pgmap v15672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:36 smithi161 ceph-mon[91579]: pgmap v15673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:36 smithi188 ceph-mon[92425]: pgmap v15673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:21:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:39 smithi161 ceph-mon[91579]: pgmap v15674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:39 smithi188 ceph-mon[92425]: pgmap v15674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:41 smithi161 ceph-mon[91579]: pgmap v15675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:41 smithi188 ceph-mon[92425]: pgmap v15675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:43 smithi161 ceph-mon[91579]: pgmap v15676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:43 smithi188 ceph-mon[92425]: pgmap v15676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:44 smithi161 ceph-mon[91579]: pgmap v15677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:44 smithi188 ceph-mon[92425]: pgmap v15677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:46 smithi161 ceph-mon[91579]: pgmap v15678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:46 smithi188 ceph-mon[92425]: pgmap v15678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:49 smithi161 ceph-mon[91579]: pgmap v15679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:49 smithi188 ceph-mon[92425]: pgmap v15679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:51 smithi161 ceph-mon[91579]: pgmap v15680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:51 smithi188 ceph-mon[92425]: pgmap v15680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:53 smithi161 ceph-mon[91579]: pgmap v15681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:53 smithi188 ceph-mon[92425]: pgmap v15681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:54 smithi161 ceph-mon[91579]: pgmap v15682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:54 smithi188 ceph-mon[92425]: pgmap v15682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:55.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:21:55.306+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:21:55.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:21:55.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:21:55.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:21:55.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:21:55.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:21:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:21:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:21:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:21:55.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:21:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:21:55.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:21:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:21:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:21:55.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:21:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:21:55.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:21:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:21:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:21:55.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:21:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:21:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:21:55.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:21:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:21:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:21:55.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:21:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:21:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:21:55.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:21:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:21:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:21:55.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:21:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:21:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:21:55.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:21:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:21:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:21:55.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:21:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:21:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:21:55.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:21:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:21:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:21:55.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:21:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:21:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=937] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:21:55.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=937] Local address: 172.21.15.161, port 59628 2022-07-03T19:21:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=937] Peer address: 172.21.15.101, port 22 2022-07-03T19:21:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=937] Connection lost 2022-07-03T19:21:55.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=937] Aborting connection 2022-07-03T19:21:55.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:21:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:21:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:21:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:21:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:21:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:21:56.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:21:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:21:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:21:56.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:21:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:21:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:21:56.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:21:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:21:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:21:56.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:21:56.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:21:56.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:21:56.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:21:56.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:21:56.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:21:56.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:21:56.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:21:56.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:21:56.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:21:56.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:21:56.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:21:56.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:21:56.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:21:56.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:21:56.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:21:56.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:21:56.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:21:56.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:21:56.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:21:56.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:21:56.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:21:56.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:21:56.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:21:56.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:21:56.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:21:56.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:21:56.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:21:56.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:21:56.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:21:56.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:21:56.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:21:56.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:21:56.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:21:56.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:21:56.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=937] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:21:56.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=937] Local address: 172.21.15.161, port 59628 2022-07-03T19:21:56.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=937] Peer address: 172.21.15.101, port 22 2022-07-03T19:21:56.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=937] Connection lost 2022-07-03T19:21:56.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=937] Aborting connection 2022-07-03T19:21:56.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:21:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:21:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:21:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:21:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:21:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:21:56.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:21:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:21:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:21:56.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:21:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:21:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:21:56.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:21:56.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:21:56.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:21:56.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:21:56.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:21:56.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:21:56.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:21:56.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:21:56.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:21:56.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:21:56.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:21:56.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:21:56.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:21:56.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:21:56.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:21:56.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:21:56.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:21:56.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:21:56.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:21:56.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:21:56.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:21:56.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:21:56.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:21:56.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:21:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:21:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:21:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:21:56.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:21:56.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:21:56.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:21:56.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:21:56.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:21:56.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:21:56.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:21:56.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:21:56.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:21:56.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:21:56.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=937] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:21:56.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=937] Local address: 172.21.15.161, port 59628 2022-07-03T19:21:56.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=937] Peer address: 172.21.15.101, port 22 2022-07-03T19:21:56.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=937] Connection lost 2022-07-03T19:21:56.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=937] Aborting connection 2022-07-03T19:21:56.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:21:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:56 smithi161 ceph-mon[91579]: pgmap v15683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:56 smithi188 ceph-mon[92425]: pgmap v15683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:21:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:59 smithi161 ceph-mon[91579]: pgmap v15684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:21:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:21:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:21:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:21:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:59 smithi188 ceph-mon[92425]: pgmap v15684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:21:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:21:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:21:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:21:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:22:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:01 smithi161 ceph-mon[91579]: pgmap v15685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:01 smithi188 ceph-mon[92425]: pgmap v15685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:03 smithi161 ceph-mon[91579]: pgmap v15686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:03 smithi188 ceph-mon[92425]: pgmap v15686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:04 smithi161 ceph-mon[91579]: pgmap v15687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:04 smithi188 ceph-mon[92425]: pgmap v15687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:06 smithi161 ceph-mon[91579]: pgmap v15688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:06 smithi188 ceph-mon[92425]: pgmap v15688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:09 smithi161 ceph-mon[91579]: pgmap v15689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:09 smithi188 ceph-mon[92425]: pgmap v15689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:11 smithi161 ceph-mon[91579]: pgmap v15690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:11 smithi188 ceph-mon[92425]: pgmap v15690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:13 smithi161 ceph-mon[91579]: pgmap v15691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:13 smithi188 ceph-mon[92425]: pgmap v15691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:14 smithi161 ceph-mon[91579]: pgmap v15692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:14 smithi188 ceph-mon[92425]: pgmap v15692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:16 smithi161 ceph-mon[91579]: pgmap v15693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:16 smithi188 ceph-mon[92425]: pgmap v15693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:19 smithi161 ceph-mon[91579]: pgmap v15694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:19 smithi188 ceph-mon[92425]: pgmap v15694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:21 smithi161 ceph-mon[91579]: pgmap v15695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:21 smithi188 ceph-mon[92425]: pgmap v15695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:23 smithi161 ceph-mon[91579]: pgmap v15696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:23 smithi188 ceph-mon[92425]: pgmap v15696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:24 smithi161 ceph-mon[91579]: pgmap v15697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:24 smithi188 ceph-mon[92425]: pgmap v15697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:26 smithi161 ceph-mon[91579]: pgmap v15698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:26 smithi188 ceph-mon[92425]: pgmap v15698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:29 smithi161 ceph-mon[91579]: pgmap v15699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:29 smithi188 ceph-mon[92425]: pgmap v15699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:31 smithi161 ceph-mon[91579]: pgmap v15700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:31 smithi188 ceph-mon[92425]: pgmap v15700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:33 smithi161 ceph-mon[91579]: pgmap v15701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:33 smithi188 ceph-mon[92425]: pgmap v15701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:34 smithi161 ceph-mon[91579]: pgmap v15702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:34 smithi188 ceph-mon[92425]: pgmap v15702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:36 smithi161 ceph-mon[91579]: pgmap v15703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:36 smithi188 ceph-mon[92425]: pgmap v15703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:39 smithi161 ceph-mon[91579]: pgmap v15704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:39 smithi188 ceph-mon[92425]: pgmap v15704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:41 smithi161 ceph-mon[91579]: pgmap v15705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:41 smithi188 ceph-mon[92425]: pgmap v15705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:43 smithi161 ceph-mon[91579]: pgmap v15706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:43 smithi188 ceph-mon[92425]: pgmap v15706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:44 smithi161 ceph-mon[91579]: pgmap v15707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:44 smithi188 ceph-mon[92425]: pgmap v15707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:46 smithi161 ceph-mon[91579]: pgmap v15708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:46 smithi188 ceph-mon[92425]: pgmap v15708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:49 smithi161 ceph-mon[91579]: pgmap v15709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:49 smithi188 ceph-mon[92425]: pgmap v15709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:51 smithi161 ceph-mon[91579]: pgmap v15710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:51 smithi188 ceph-mon[92425]: pgmap v15710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:53 smithi188 ceph-mon[92425]: pgmap v15711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:53 smithi161 ceph-mon[91579]: pgmap v15711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:54 smithi161 ceph-mon[91579]: pgmap v15712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:54 smithi188 ceph-mon[92425]: pgmap v15712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:22:56.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:22:55.816+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:22:56.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:22:56.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:22:56.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:22:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:22:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:22:56.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:22:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:22:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:22:56.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:22:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:22:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:22:56.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:22:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:22:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:22:56.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:22:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:22:56.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:22:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:22:56.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:22:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:22:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:22:56.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:22:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:22:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:22:56.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:22:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:22:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:22:56.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:22:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:22:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:22:56.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:22:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:22:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:22:56.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:22:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:22:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:22:56.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:22:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:22:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:22:56.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:22:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:22:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:22:56.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:22:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:22:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=939] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:22:56.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=939] Local address: 172.21.15.161, port 59632 2022-07-03T19:22:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=939] Peer address: 172.21.15.101, port 22 2022-07-03T19:22:56.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=939] Connection lost 2022-07-03T19:22:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=939] Aborting connection 2022-07-03T19:22:56.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:22:55 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:22:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:22:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:22:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:22:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:22:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:22:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:22:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:22:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:22:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:22:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:22:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:22:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:22:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:22:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:22:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:22:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:22:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:22:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:22:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:22:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:22:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:22:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:22:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:22:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:22:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:22:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:22:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:22:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:22:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:22:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:22:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:22:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:22:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:22:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:22:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:22:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:22:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:22:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:22:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:22:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:22:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:22:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:22:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:22:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:22:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:22:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:22:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:22:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:22:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:22:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:22:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=939] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:22:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=939] Local address: 172.21.15.161, port 59632 2022-07-03T19:22:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=939] Peer address: 172.21.15.101, port 22 2022-07-03T19:22:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=939] Connection lost 2022-07-03T19:22:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=939] Aborting connection 2022-07-03T19:22:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:56 smithi161 ceph-mon[91579]: pgmap v15713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:22:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:22:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:22:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:22:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:22:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:22:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:22:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:22:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:22:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:22:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:22:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:22:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:22:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:22:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:22:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:22:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:22:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:22:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:22:57.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:22:57.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:22:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:22:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:22:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:22:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:22:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:22:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:22:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:22:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:22:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:22:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:22:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:22:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:22:57.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:22:57.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:22:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:22:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:22:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:22:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:22:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:22:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:22:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:22:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:22:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:22:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:22:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:22:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:22:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:22:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=939] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:22:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=939] Local address: 172.21.15.161, port 59632 2022-07-03T19:22:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=939] Peer address: 172.21.15.101, port 22 2022-07-03T19:22:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=939] Connection lost 2022-07-03T19:22:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=939] Aborting connection 2022-07-03T19:22:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:56 smithi188 ceph-mon[92425]: pgmap v15713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:22:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:59 smithi161 ceph-mon[91579]: pgmap v15714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:22:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:22:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:22:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:22:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:59 smithi188 ceph-mon[92425]: pgmap v15714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:22:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:22:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:22:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:22:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:23:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:01 smithi188 ceph-mon[92425]: pgmap v15715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:01 smithi161 ceph-mon[91579]: pgmap v15715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:03 smithi188 ceph-mon[92425]: pgmap v15716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:03 smithi161 ceph-mon[91579]: pgmap v15716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:04 smithi161 ceph-mon[91579]: pgmap v15717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:04 smithi188 ceph-mon[92425]: pgmap v15717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:06 smithi161 ceph-mon[91579]: pgmap v15718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:06 smithi188 ceph-mon[92425]: pgmap v15718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:09 smithi188 ceph-mon[92425]: pgmap v15719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:09 smithi161 ceph-mon[91579]: pgmap v15719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:11 smithi188 ceph-mon[92425]: pgmap v15720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:11 smithi161 ceph-mon[91579]: pgmap v15720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:13 smithi188 ceph-mon[92425]: pgmap v15721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:13 smithi161 ceph-mon[91579]: pgmap v15721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:14 smithi161 ceph-mon[91579]: pgmap v15722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:14 smithi188 ceph-mon[92425]: pgmap v15722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:16 smithi161 ceph-mon[91579]: pgmap v15723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:16 smithi188 ceph-mon[92425]: pgmap v15723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:19 smithi188 ceph-mon[92425]: pgmap v15724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:19 smithi161 ceph-mon[91579]: pgmap v15724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:21 smithi188 ceph-mon[92425]: pgmap v15725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:21 smithi161 ceph-mon[91579]: pgmap v15725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:23 smithi188 ceph-mon[92425]: pgmap v15726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:23 smithi161 ceph-mon[91579]: pgmap v15726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:24 smithi161 ceph-mon[91579]: pgmap v15727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:24 smithi188 ceph-mon[92425]: pgmap v15727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:26 smithi161 ceph-mon[91579]: pgmap v15728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:26 smithi188 ceph-mon[92425]: pgmap v15728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:29 smithi188 ceph-mon[92425]: pgmap v15729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:29 smithi161 ceph-mon[91579]: pgmap v15729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:31 smithi188 ceph-mon[92425]: pgmap v15730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:31 smithi161 ceph-mon[91579]: pgmap v15730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:33 smithi188 ceph-mon[92425]: pgmap v15731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:33 smithi161 ceph-mon[91579]: pgmap v15731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:34 smithi161 ceph-mon[91579]: pgmap v15732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:34 smithi188 ceph-mon[92425]: pgmap v15732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:36 smithi161 ceph-mon[91579]: pgmap v15733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:36 smithi188 ceph-mon[92425]: pgmap v15733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:39 smithi188 ceph-mon[92425]: pgmap v15734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:39 smithi161 ceph-mon[91579]: pgmap v15734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:41 smithi188 ceph-mon[92425]: pgmap v15735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:41 smithi161 ceph-mon[91579]: pgmap v15735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:43 smithi188 ceph-mon[92425]: pgmap v15736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:43 smithi161 ceph-mon[91579]: pgmap v15736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:44 smithi161 ceph-mon[91579]: pgmap v15737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:44 smithi188 ceph-mon[92425]: pgmap v15737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:46 smithi161 ceph-mon[91579]: pgmap v15738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:46 smithi188 ceph-mon[92425]: pgmap v15738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:49 smithi188 ceph-mon[92425]: pgmap v15739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:49 smithi161 ceph-mon[91579]: pgmap v15739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:51 smithi188 ceph-mon[92425]: pgmap v15740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:51 smithi161 ceph-mon[91579]: pgmap v15740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:53 smithi188 ceph-mon[92425]: pgmap v15741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:53 smithi161 ceph-mon[91579]: pgmap v15741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:54 smithi161 ceph-mon[91579]: pgmap v15742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:54 smithi188 ceph-mon[92425]: pgmap v15742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:23:56.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:23:56.321+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:23:56.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:23:56.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:23:56.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:23:56.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:23:56.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:23:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:23:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:23:56.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:23:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:23:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:23:56.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:23:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:23:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:23:56.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:23:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:23:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:23:56.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:23:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:23:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:23:56.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:23:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:23:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:23:56.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:23:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:23:56.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:23:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:23:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:23:56.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:23:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:23:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:23:56.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:23:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:23:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:23:56.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:23:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:23:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:23:56.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:23:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:23:56.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:23:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:23:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:23:56.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:23:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:23:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:23:56.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=941] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:23:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=941] Local address: 172.21.15.161, port 59636 2022-07-03T19:23:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=941] Peer address: 172.21.15.101, port 22 2022-07-03T19:23:56.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=941] Connection lost 2022-07-03T19:23:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=941] Aborting connection 2022-07-03T19:23:56.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:23:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:23:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:23:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:23:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:23:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:23:57.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:23:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:23:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:23:57.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:23:57.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:23:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:23:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:23:57.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:23:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:23:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:23:57.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:23:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:23:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:23:57.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:23:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:23:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:23:57.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:23:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:23:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:23:57.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:23:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:23:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:23:57.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:23:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:23:57.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:23:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:23:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:23:57.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:23:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:23:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:23:57.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:23:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:23:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:23:57.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:23:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:23:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:23:57.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:23:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:23:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:23:57.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:23:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:23:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:23:57.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:23:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:23:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=941] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:23:57.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=941] Local address: 172.21.15.161, port 59636 2022-07-03T19:23:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=941] Peer address: 172.21.15.101, port 22 2022-07-03T19:23:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=941] Connection lost 2022-07-03T19:23:57.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=941] Aborting connection 2022-07-03T19:23:57.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:56 smithi161 ceph-mon[91579]: pgmap v15743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:23:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:23:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:23:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:23:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:23:57.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:23:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:23:57.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:23:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:23:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:23:57.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:23:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:23:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:23:57.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:23:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:23:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:23:57.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:23:57.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:23:57.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:23:57.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:23:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:23:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:23:57.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:23:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:23:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:23:57.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:23:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:23:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:23:57.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:23:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:23:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:23:57.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:23:57.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:23:57.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:23:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:23:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:23:57.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:23:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:23:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:23:57.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:23:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:23:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:23:57.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:23:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:23:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:23:57.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:23:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:23:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:23:57.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:23:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=941] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:23:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=941] Local address: 172.21.15.161, port 59636 2022-07-03T19:23:57.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=941] Peer address: 172.21.15.101, port 22 2022-07-03T19:23:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=941] Connection lost 2022-07-03T19:23:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=941] Aborting connection 2022-07-03T19:23:57.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:56 smithi188 ceph-mon[92425]: pgmap v15743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:23:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:59 smithi188 ceph-mon[92425]: pgmap v15744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:23:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:23:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:23:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:23:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:59 smithi161 ceph-mon[91579]: pgmap v15744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:23:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:23:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:23:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:23:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:24:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:01 smithi188 ceph-mon[92425]: pgmap v15745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:01 smithi161 ceph-mon[91579]: pgmap v15745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:03 smithi188 ceph-mon[92425]: pgmap v15746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:03 smithi161 ceph-mon[91579]: pgmap v15746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:04 smithi161 ceph-mon[91579]: pgmap v15747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:04 smithi188 ceph-mon[92425]: pgmap v15747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:06 smithi161 ceph-mon[91579]: pgmap v15748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:06 smithi188 ceph-mon[92425]: pgmap v15748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:09 smithi188 ceph-mon[92425]: pgmap v15749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:09 smithi161 ceph-mon[91579]: pgmap v15749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:11 smithi188 ceph-mon[92425]: pgmap v15750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:11 smithi161 ceph-mon[91579]: pgmap v15750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:13 smithi188 ceph-mon[92425]: pgmap v15751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:13 smithi161 ceph-mon[91579]: pgmap v15751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:14 smithi161 ceph-mon[91579]: pgmap v15752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:14 smithi188 ceph-mon[92425]: pgmap v15752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:16 smithi161 ceph-mon[91579]: pgmap v15753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:16 smithi188 ceph-mon[92425]: pgmap v15753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:19 smithi188 ceph-mon[92425]: pgmap v15754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:19 smithi161 ceph-mon[91579]: pgmap v15754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:21 smithi188 ceph-mon[92425]: pgmap v15755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:21 smithi161 ceph-mon[91579]: pgmap v15755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:23 smithi188 ceph-mon[92425]: pgmap v15756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:23 smithi161 ceph-mon[91579]: pgmap v15756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:24 smithi161 ceph-mon[91579]: pgmap v15757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:24 smithi188 ceph-mon[92425]: pgmap v15757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:26 smithi161 ceph-mon[91579]: pgmap v15758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:26 smithi188 ceph-mon[92425]: pgmap v15758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:29 smithi188 ceph-mon[92425]: pgmap v15759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:29 smithi161 ceph-mon[91579]: pgmap v15759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:31 smithi188 ceph-mon[92425]: pgmap v15760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:31 smithi161 ceph-mon[91579]: pgmap v15760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:33 smithi188 ceph-mon[92425]: pgmap v15761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:33 smithi161 ceph-mon[91579]: pgmap v15761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:34 smithi161 ceph-mon[91579]: pgmap v15762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:34 smithi188 ceph-mon[92425]: pgmap v15762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:36 smithi161 ceph-mon[91579]: pgmap v15763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:36 smithi188 ceph-mon[92425]: pgmap v15763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:39 smithi188 ceph-mon[92425]: pgmap v15764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:39 smithi161 ceph-mon[91579]: pgmap v15764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:41 smithi188 ceph-mon[92425]: pgmap v15765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:41 smithi161 ceph-mon[91579]: pgmap v15765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:43 smithi188 ceph-mon[92425]: pgmap v15766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:43 smithi161 ceph-mon[91579]: pgmap v15766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:44 smithi161 ceph-mon[91579]: pgmap v15767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:45.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:44 smithi188 ceph-mon[92425]: pgmap v15767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:46 smithi161 ceph-mon[91579]: pgmap v15768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:46 smithi188 ceph-mon[92425]: pgmap v15768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:49 smithi188 ceph-mon[92425]: pgmap v15769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:49 smithi161 ceph-mon[91579]: pgmap v15769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:51 smithi161 ceph-mon[91579]: pgmap v15770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:51 smithi188 ceph-mon[92425]: pgmap v15770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:53 smithi161 ceph-mon[91579]: pgmap v15771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:53 smithi188 ceph-mon[92425]: pgmap v15771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:54 smithi161 ceph-mon[91579]: pgmap v15772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:54 smithi188 ceph-mon[92425]: pgmap v15772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:24:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-mon[91579]: pgmap v15773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:24:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:24:57.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:24:57.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:24:56.822+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:24:57.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:24:57.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:24:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:24:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:24:57.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:24:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:24:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:24:57.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:24:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:24:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:24:57.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:24:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:24:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:24:57.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:24:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:24:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:24:57.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:24:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:24:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:24:57.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:24:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:24:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:24:57.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:24:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:24:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:24:57.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:24:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:24:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:24:57.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:24:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:24:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:24:57.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:24:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:24:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:24:57.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:24:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:24:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:24:57.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:24:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:24:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:24:57.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:24:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:24:57.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:24:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:24:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=943] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:24:57.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=943] Local address: 172.21.15.161, port 59666 2022-07-03T19:24:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=943] Peer address: 172.21.15.101, port 22 2022-07-03T19:24:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=943] Connection lost 2022-07-03T19:24:57.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=943] Aborting connection 2022-07-03T19:24:57.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:24:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:24:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:56 smithi188 ceph-mon[92425]: pgmap v15773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:24:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:24:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:24:57.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:24:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:24:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:24:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:24:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:24:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:24:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:24:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:24:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:24:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:24:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:24:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:24:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:24:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:24:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:24:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:24:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:24:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:24:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:24:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:24:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:24:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:24:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:24:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:24:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:24:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:24:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:24:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:24:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:24:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:24:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:24:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:24:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:24:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:24:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:24:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:24:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:24:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:24:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:24:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:24:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:24:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:24:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:24:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:24:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:24:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=943] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:24:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=943] Local address: 172.21.15.161, port 59666 2022-07-03T19:24:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=943] Peer address: 172.21.15.101, port 22 2022-07-03T19:24:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=943] Connection lost 2022-07-03T19:24:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=943] Aborting connection 2022-07-03T19:24:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:24:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:24:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:24:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:24:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:24:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:24:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:24:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:24:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:24:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:24:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:24:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:24:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:24:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:24:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:24:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:24:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:24:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:24:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:24:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:24:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:24:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:24:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:24:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:24:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:24:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:24:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:24:58.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:24:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:24:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:24:58.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:24:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:24:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:24:58.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:24:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:24:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:24:58.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:24:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:24:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:24:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:24:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:24:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:24:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:24:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:24:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:24:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:24:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=943] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:24:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=943] Local address: 172.21.15.161, port 59666 2022-07-03T19:24:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=943] Peer address: 172.21.15.101, port 22 2022-07-03T19:24:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=943] Connection lost 2022-07-03T19:24:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=943] Aborting connection 2022-07-03T19:24:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:58 smithi161 ceph-mon[91579]: pgmap v15774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:24:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:58 smithi188 ceph-mon[92425]: pgmap v15774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:25:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:25:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:25:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:25:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:00 smithi161 ceph-mon[91579]: pgmap v15775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:00 smithi188 ceph-mon[92425]: pgmap v15775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:03 smithi161 ceph-mon[91579]: pgmap v15776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:03 smithi188 ceph-mon[92425]: pgmap v15776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:04 smithi161 ceph-mon[91579]: pgmap v15777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:04 smithi188 ceph-mon[92425]: pgmap v15777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:06 smithi161 ceph-mon[91579]: pgmap v15778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:06 smithi188 ceph-mon[92425]: pgmap v15778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:09 smithi161 ceph-mon[91579]: pgmap v15779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:09 smithi188 ceph-mon[92425]: pgmap v15779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:11 smithi161 ceph-mon[91579]: pgmap v15780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:11 smithi188 ceph-mon[92425]: pgmap v15780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:13 smithi161 ceph-mon[91579]: pgmap v15781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:13 smithi188 ceph-mon[92425]: pgmap v15781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:14 smithi161 ceph-mon[91579]: pgmap v15782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:14 smithi188 ceph-mon[92425]: pgmap v15782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:16 smithi161 ceph-mon[91579]: pgmap v15783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:16 smithi188 ceph-mon[92425]: pgmap v15783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:19 smithi161 ceph-mon[91579]: pgmap v15784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:19 smithi188 ceph-mon[92425]: pgmap v15784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:21 smithi161 ceph-mon[91579]: pgmap v15785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:21 smithi188 ceph-mon[92425]: pgmap v15785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:23 smithi161 ceph-mon[91579]: pgmap v15786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:23 smithi188 ceph-mon[92425]: pgmap v15786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:24 smithi161 ceph-mon[91579]: pgmap v15787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:24 smithi188 ceph-mon[92425]: pgmap v15787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:26 smithi161 ceph-mon[91579]: pgmap v15788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:26 smithi188 ceph-mon[92425]: pgmap v15788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:29 smithi161 ceph-mon[91579]: pgmap v15789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:29 smithi188 ceph-mon[92425]: pgmap v15789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:31 smithi161 ceph-mon[91579]: pgmap v15790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:31 smithi188 ceph-mon[92425]: pgmap v15790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:33 smithi161 ceph-mon[91579]: pgmap v15791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:33 smithi188 ceph-mon[92425]: pgmap v15791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:34 smithi161 ceph-mon[91579]: pgmap v15792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:34 smithi188 ceph-mon[92425]: pgmap v15792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:36 smithi161 ceph-mon[91579]: pgmap v15793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:36 smithi188 ceph-mon[92425]: pgmap v15793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:39 smithi161 ceph-mon[91579]: pgmap v15794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:39 smithi188 ceph-mon[92425]: pgmap v15794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:41 smithi161 ceph-mon[91579]: pgmap v15795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:41 smithi188 ceph-mon[92425]: pgmap v15795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:43 smithi161 ceph-mon[91579]: pgmap v15796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:43 smithi188 ceph-mon[92425]: pgmap v15796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:44 smithi161 ceph-mon[91579]: pgmap v15797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:44 smithi188 ceph-mon[92425]: pgmap v15797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:46 smithi161 ceph-mon[91579]: pgmap v15798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:46 smithi188 ceph-mon[92425]: pgmap v15798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:49 smithi161 ceph-mon[91579]: pgmap v15799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:49 smithi188 ceph-mon[92425]: pgmap v15799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:51 smithi161 ceph-mon[91579]: pgmap v15800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:51 smithi188 ceph-mon[92425]: pgmap v15800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:53 smithi161 ceph-mon[91579]: pgmap v15801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:53 smithi188 ceph-mon[92425]: pgmap v15801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:54 smithi161 ceph-mon[91579]: pgmap v15802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:54 smithi188 ceph-mon[92425]: pgmap v15802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:25:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:56 smithi161 ceph-mon[91579]: pgmap v15803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:56 smithi188 ceph-mon[92425]: pgmap v15803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:25:57.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:25:57.322+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:25:57.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:25:57.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:25:57.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:25:57.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:25:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:25:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:25:57.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:25:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:25:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:25:57.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:25:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:25:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:25:57.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:25:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:25:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:25:57.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:25:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:25:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:25:57.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:25:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:25:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:25:57.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:25:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:25:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:25:57.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:25:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:25:57.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:25:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:25:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:25:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:25:57.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:25:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:25:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:25:57.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:25:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:25:57.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:25:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:25:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:25:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:25:57.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:25:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:25:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:25:57.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:25:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:25:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=945] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:25:57.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=945] Local address: 172.21.15.161, port 59670 2022-07-03T19:25:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=945] Peer address: 172.21.15.101, port 22 2022-07-03T19:25:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=945] Connection lost 2022-07-03T19:25:57.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=945] Aborting connection 2022-07-03T19:25:57.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:25:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:25:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:25:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:25:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:25:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:25:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:25:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:25:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:25:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:25:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:25:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:25:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:25:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:25:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:25:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:25:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:25:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:25:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:25:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:25:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:25:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:25:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:25:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:25:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:25:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:25:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:25:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:25:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:25:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:25:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:25:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:25:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:25:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:25:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:25:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:25:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:25:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:25:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:25:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:25:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:25:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:25:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:25:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:25:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:25:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:25:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:25:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:25:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:25:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:25:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=945] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:25:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=945] Local address: 172.21.15.161, port 59670 2022-07-03T19:25:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=945] Peer address: 172.21.15.101, port 22 2022-07-03T19:25:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=945] Connection lost 2022-07-03T19:25:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=945] Aborting connection 2022-07-03T19:25:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:25:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:25:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:25:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:25:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:25:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:25:58.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:25:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:25:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:25:58.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:25:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:25:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:25:58.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:25:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:25:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:25:58.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:25:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:25:58.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:25:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:25:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:25:58.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:25:58.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:25:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:25:58.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:25:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:25:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:25:58.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:25:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:25:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:25:58.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:25:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:25:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:25:58.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:25:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:25:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:25:58.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:25:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:25:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:25:58.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:25:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:25:58.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:25:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:25:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:25:58.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:25:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:25:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:25:58.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:25:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:25:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:25:58.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=945] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:25:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=945] Local address: 172.21.15.161, port 59670 2022-07-03T19:25:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=945] Peer address: 172.21.15.101, port 22 2022-07-03T19:25:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=945] Connection lost 2022-07-03T19:25:58.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=945] Aborting connection 2022-07-03T19:25:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:58 smithi161 ceph-mon[91579]: pgmap v15804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:25:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:58 smithi188 ceph-mon[92425]: pgmap v15804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:26:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:26:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:26:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:26:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:26:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:26:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:26:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:26:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:00 smithi161 ceph-mon[91579]: pgmap v15805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:00 smithi188 ceph-mon[92425]: pgmap v15805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:03 smithi161 ceph-mon[91579]: pgmap v15806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:03 smithi188 ceph-mon[92425]: pgmap v15806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:04 smithi161 ceph-mon[91579]: pgmap v15807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:04 smithi188 ceph-mon[92425]: pgmap v15807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:06 smithi161 ceph-mon[91579]: pgmap v15808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:06 smithi188 ceph-mon[92425]: pgmap v15808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:09 smithi161 ceph-mon[91579]: pgmap v15809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:09 smithi188 ceph-mon[92425]: pgmap v15809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:11 smithi161 ceph-mon[91579]: pgmap v15810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:11 smithi188 ceph-mon[92425]: pgmap v15810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:13 smithi161 ceph-mon[91579]: pgmap v15811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:13 smithi188 ceph-mon[92425]: pgmap v15811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:14 smithi161 ceph-mon[91579]: pgmap v15812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:14 smithi188 ceph-mon[92425]: pgmap v15812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:16 smithi161 ceph-mon[91579]: pgmap v15813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:16 smithi188 ceph-mon[92425]: pgmap v15813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:19 smithi161 ceph-mon[91579]: pgmap v15814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:19 smithi188 ceph-mon[92425]: pgmap v15814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:21 smithi161 ceph-mon[91579]: pgmap v15815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:21 smithi188 ceph-mon[92425]: pgmap v15815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:23 smithi161 ceph-mon[91579]: pgmap v15816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:23 smithi188 ceph-mon[92425]: pgmap v15816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:25.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:24 smithi161 ceph-mon[91579]: pgmap v15817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:24 smithi188 ceph-mon[92425]: pgmap v15817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:26 smithi161 ceph-mon[91579]: pgmap v15818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:26 smithi188 ceph-mon[92425]: pgmap v15818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:29 smithi161 ceph-mon[91579]: pgmap v15819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:29 smithi188 ceph-mon[92425]: pgmap v15819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:31 smithi161 ceph-mon[91579]: pgmap v15820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:31 smithi188 ceph-mon[92425]: pgmap v15820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:33 smithi161 ceph-mon[91579]: pgmap v15821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:33 smithi188 ceph-mon[92425]: pgmap v15821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:34 smithi161 ceph-mon[91579]: pgmap v15822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:34 smithi188 ceph-mon[92425]: pgmap v15822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:36 smithi161 ceph-mon[91579]: pgmap v15823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:36 smithi188 ceph-mon[92425]: pgmap v15823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:39 smithi161 ceph-mon[91579]: pgmap v15824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:39 smithi188 ceph-mon[92425]: pgmap v15824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:41 smithi161 ceph-mon[91579]: pgmap v15825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:41 smithi188 ceph-mon[92425]: pgmap v15825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:43 smithi161 ceph-mon[91579]: pgmap v15826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:43 smithi188 ceph-mon[92425]: pgmap v15826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:44 smithi161 ceph-mon[91579]: pgmap v15827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:44 smithi188 ceph-mon[92425]: pgmap v15827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:46 smithi161 ceph-mon[91579]: pgmap v15828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:46 smithi188 ceph-mon[92425]: pgmap v15828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:49 smithi161 ceph-mon[91579]: pgmap v15829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:49 smithi188 ceph-mon[92425]: pgmap v15829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:51 smithi161 ceph-mon[91579]: pgmap v15830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:51 smithi188 ceph-mon[92425]: pgmap v15830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:53 smithi161 ceph-mon[91579]: pgmap v15831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:53 smithi188 ceph-mon[92425]: pgmap v15831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:54 smithi161 ceph-mon[91579]: pgmap v15832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:54 smithi188 ceph-mon[92425]: pgmap v15832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:26:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:56 smithi161 ceph-mon[91579]: pgmap v15833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:56 smithi188 ceph-mon[92425]: pgmap v15833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:26:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:26:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:26:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:26:58.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:26:57.816+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:26:58.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:26:58.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:26:58.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:26:58.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:26:58.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:26:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:26:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:26:58.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:26:58.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:26:58.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:26:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:26:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:26:58.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:26:58.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:26:58.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:26:58.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:26:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:26:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:26:58.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:26:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:26:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:26:58.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:26:58.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:26:58.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:26:58.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:26:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:26:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:26:58.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:26:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:26:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:26:58.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:26:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:26:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:26:58.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:26:58.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:26:58.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:26:58.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:26:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:26:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:26:58.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:26:58.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:26:58.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:26:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:26:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:26:58.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=947] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:26:58.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=947] Local address: 172.21.15.161, port 59674 2022-07-03T19:26:58.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=947] Peer address: 172.21.15.101, port 22 2022-07-03T19:26:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=947] Connection lost 2022-07-03T19:26:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=947] Aborting connection 2022-07-03T19:26:58.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:26:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:26:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:26:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:26:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:26:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:26:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:26:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:26:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:26:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:26:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:26:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:26:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:26:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:26:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:26:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:26:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:26:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:26:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:26:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:26:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:26:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:26:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:26:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:26:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:26:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:26:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:26:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:26:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:26:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:26:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:26:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:26:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:26:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:26:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:26:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:26:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:26:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:26:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:26:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:26:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:26:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:26:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:26:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:26:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:26:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:26:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:26:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:26:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:26:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:26:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=947] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:26:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=947] Local address: 172.21.15.161, port 59674 2022-07-03T19:26:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=947] Peer address: 172.21.15.101, port 22 2022-07-03T19:26:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=947] Connection lost 2022-07-03T19:26:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=947] Aborting connection 2022-07-03T19:26:59.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:58 smithi161 ceph-mon[91579]: pgmap v15834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:26:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:26:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:26:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:26:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:26:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:26:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:26:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:26:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:26:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:26:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:26:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:26:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:26:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:26:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:26:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:26:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:26:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:26:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:26:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:26:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:26:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:26:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:26:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:26:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:26:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:26:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:26:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:26:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:26:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:26:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:26:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:26:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:26:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:26:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:26:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:26:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:26:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:26:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:26:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:26:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:26:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:26:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:26:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:26:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:26:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:26:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:26:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=947] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:26:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=947] Local address: 172.21.15.161, port 59674 2022-07-03T19:26:59.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=947] Peer address: 172.21.15.101, port 22 2022-07-03T19:26:59.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=947] Connection lost 2022-07-03T19:26:59.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=947] Aborting connection 2022-07-03T19:26:59.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:58 smithi188 ceph-mon[92425]: pgmap v15834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:27:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:27:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:27:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:27:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:27:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:27:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:27:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:27:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:00 smithi161 ceph-mon[91579]: pgmap v15835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:00 smithi188 ceph-mon[92425]: pgmap v15835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:03 smithi161 ceph-mon[91579]: pgmap v15836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:03 smithi188 ceph-mon[92425]: pgmap v15836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:04 smithi161 ceph-mon[91579]: pgmap v15837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:04 smithi188 ceph-mon[92425]: pgmap v15837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:06 smithi161 ceph-mon[91579]: pgmap v15838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:06 smithi188 ceph-mon[92425]: pgmap v15838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:09 smithi161 ceph-mon[91579]: pgmap v15839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:09 smithi188 ceph-mon[92425]: pgmap v15839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:11 smithi161 ceph-mon[91579]: pgmap v15840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:11 smithi188 ceph-mon[92425]: pgmap v15840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:13 smithi161 ceph-mon[91579]: pgmap v15841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:13 smithi188 ceph-mon[92425]: pgmap v15841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:14 smithi161 ceph-mon[91579]: pgmap v15842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:14 smithi188 ceph-mon[92425]: pgmap v15842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:16 smithi161 ceph-mon[91579]: pgmap v15843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:16 smithi188 ceph-mon[92425]: pgmap v15843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:19 smithi161 ceph-mon[91579]: pgmap v15844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:19 smithi188 ceph-mon[92425]: pgmap v15844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:21 smithi161 ceph-mon[91579]: pgmap v15845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:21 smithi188 ceph-mon[92425]: pgmap v15845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:23 smithi161 ceph-mon[91579]: pgmap v15846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:23 smithi188 ceph-mon[92425]: pgmap v15846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:24 smithi161 ceph-mon[91579]: pgmap v15847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:24 smithi188 ceph-mon[92425]: pgmap v15847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:26.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:26 smithi161 ceph-mon[91579]: pgmap v15848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:26 smithi188 ceph-mon[92425]: pgmap v15848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:29 smithi161 ceph-mon[91579]: pgmap v15849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:29 smithi188 ceph-mon[92425]: pgmap v15849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:31 smithi161 ceph-mon[91579]: pgmap v15850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:31 smithi188 ceph-mon[92425]: pgmap v15850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:33 smithi161 ceph-mon[91579]: pgmap v15851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:33 smithi188 ceph-mon[92425]: pgmap v15851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:34 smithi161 ceph-mon[91579]: pgmap v15852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:34 smithi188 ceph-mon[92425]: pgmap v15852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:36 smithi161 ceph-mon[91579]: pgmap v15853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:36 smithi188 ceph-mon[92425]: pgmap v15853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:39 smithi161 ceph-mon[91579]: pgmap v15854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:39 smithi188 ceph-mon[92425]: pgmap v15854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:41 smithi161 ceph-mon[91579]: pgmap v15855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:41 smithi188 ceph-mon[92425]: pgmap v15855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:43 smithi161 ceph-mon[91579]: pgmap v15856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:43 smithi188 ceph-mon[92425]: pgmap v15856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:44 smithi161 ceph-mon[91579]: pgmap v15857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:44 smithi188 ceph-mon[92425]: pgmap v15857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:46 smithi161 ceph-mon[91579]: pgmap v15858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:46 smithi188 ceph-mon[92425]: pgmap v15858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:49 smithi161 ceph-mon[91579]: pgmap v15859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:49 smithi188 ceph-mon[92425]: pgmap v15859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:51 smithi161 ceph-mon[91579]: pgmap v15860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:51 smithi188 ceph-mon[92425]: pgmap v15860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:53 smithi161 ceph-mon[91579]: pgmap v15861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:53 smithi188 ceph-mon[92425]: pgmap v15861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:54 smithi161 ceph-mon[91579]: pgmap v15862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:54 smithi188 ceph-mon[92425]: pgmap v15862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:27:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:27:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:56 smithi161 ceph-mon[91579]: pgmap v15863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:56 smithi188 ceph-mon[92425]: pgmap v15863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:27:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:27:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:27:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:27:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:27:58.320+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:27:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:27:58.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:27:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:27:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:27:58.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:27:58.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:27:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:27:58.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:27:58.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:27:58.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:27:58.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:27:58.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:27:58.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:27:58.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:27:58.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:27:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:27:58.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:27:58.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:27:58.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:27:58.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:27:58.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=949] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:27:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=949] Local address: 172.21.15.161, port 59678 2022-07-03T19:27:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=949] Peer address: 172.21.15.101, port 22 2022-07-03T19:27:58.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=949] Connection lost 2022-07-03T19:27:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=949] Aborting connection 2022-07-03T19:27:58.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:27:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:27:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:27:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:27:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:28:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:28:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:28:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:28:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:28:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:28:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:28:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:28:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:28:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:28:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:28:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:28:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:28:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:28:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:28:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:28:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:28:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:28:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:28:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:28:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:28:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:28:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:28:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:28:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:28:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:28:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:28:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:28:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:28:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:28:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:28:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:28:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:28:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:28:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:28:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:28:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:28:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:28:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:28:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:28:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:28:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:28:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=949] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:28:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=949] Local address: 172.21.15.161, port 59678 2022-07-03T19:28:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=949] Peer address: 172.21.15.101, port 22 2022-07-03T19:28:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=949] Connection lost 2022-07-03T19:28:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=949] Aborting connection 2022-07-03T19:28:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: pgmap v15864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:28:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:28:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:28:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:28:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:28:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:28:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:28:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:28:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:28:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:28:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:28:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:28:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:28:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:28:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:28:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:28:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:28:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:28:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:28:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:28:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:28:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:28:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:28:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:28:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:28:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:28:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:28:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:28:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:28:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:28:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:28:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:28:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:28:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:28:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:28:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:28:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:28:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:28:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:28:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:28:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:28:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:28:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:28:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:28:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:28:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:28:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=949] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:28:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=949] Local address: 172.21.15.161, port 59678 2022-07-03T19:28:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=949] Peer address: 172.21.15.101, port 22 2022-07-03T19:28:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=949] Connection lost 2022-07-03T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=949] Aborting connection 2022-07-03T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: pgmap v15864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.184 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:28:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:00.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:00.186 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:01 smithi161 ceph-mon[91579]: pgmap v15865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:01 smithi188 ceph-mon[92425]: pgmap v15865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:03 smithi161 ceph-mon[91579]: pgmap v15866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:03 smithi188 ceph-mon[92425]: pgmap v15866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:04 smithi161 ceph-mon[91579]: pgmap v15867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:04 smithi188 ceph-mon[92425]: pgmap v15867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:06 smithi161 ceph-mon[91579]: pgmap v15868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:06 smithi188 ceph-mon[92425]: pgmap v15868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:09 smithi161 ceph-mon[91579]: pgmap v15869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:09 smithi188 ceph-mon[92425]: pgmap v15869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:11 smithi161 ceph-mon[91579]: pgmap v15870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:11 smithi188 ceph-mon[92425]: pgmap v15870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:13 smithi161 ceph-mon[91579]: pgmap v15871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:13 smithi188 ceph-mon[92425]: pgmap v15871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:14 smithi161 ceph-mon[91579]: pgmap v15872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:14 smithi188 ceph-mon[92425]: pgmap v15872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:16 smithi161 ceph-mon[91579]: pgmap v15873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:16 smithi188 ceph-mon[92425]: pgmap v15873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:19 smithi161 ceph-mon[91579]: pgmap v15874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:19 smithi188 ceph-mon[92425]: pgmap v15874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:21 smithi161 ceph-mon[91579]: pgmap v15875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:21 smithi188 ceph-mon[92425]: pgmap v15875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:23 smithi161 ceph-mon[91579]: pgmap v15876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:23 smithi188 ceph-mon[92425]: pgmap v15876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:24 smithi161 ceph-mon[91579]: pgmap v15877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:24 smithi188 ceph-mon[92425]: pgmap v15877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:26 smithi161 ceph-mon[91579]: pgmap v15878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:26 smithi188 ceph-mon[92425]: pgmap v15878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:29 smithi161 ceph-mon[91579]: pgmap v15879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:29 smithi188 ceph-mon[92425]: pgmap v15879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:31 smithi161 ceph-mon[91579]: pgmap v15880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:31 smithi188 ceph-mon[92425]: pgmap v15880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:33 smithi161 ceph-mon[91579]: pgmap v15881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:33 smithi188 ceph-mon[92425]: pgmap v15881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:34 smithi161 ceph-mon[91579]: pgmap v15882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:34 smithi188 ceph-mon[92425]: pgmap v15882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:36 smithi161 ceph-mon[91579]: pgmap v15883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:36 smithi188 ceph-mon[92425]: pgmap v15883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:39 smithi161 ceph-mon[91579]: pgmap v15884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:39 smithi188 ceph-mon[92425]: pgmap v15884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:41 smithi161 ceph-mon[91579]: pgmap v15885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:41 smithi188 ceph-mon[92425]: pgmap v15885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:43 smithi161 ceph-mon[91579]: pgmap v15886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:43 smithi188 ceph-mon[92425]: pgmap v15886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:44 smithi161 ceph-mon[91579]: pgmap v15887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:44 smithi188 ceph-mon[92425]: pgmap v15887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:46 smithi161 ceph-mon[91579]: pgmap v15888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:46 smithi188 ceph-mon[92425]: pgmap v15888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:49 smithi161 ceph-mon[91579]: pgmap v15889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:49 smithi188 ceph-mon[92425]: pgmap v15889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:51 smithi161 ceph-mon[91579]: pgmap v15890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:51 smithi188 ceph-mon[92425]: pgmap v15890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:53 smithi161 ceph-mon[91579]: pgmap v15891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:53 smithi188 ceph-mon[92425]: pgmap v15891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:54 smithi161 ceph-mon[91579]: pgmap v15892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:54 smithi188 ceph-mon[92425]: pgmap v15892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:28:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:56 smithi161 ceph-mon[91579]: pgmap v15893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:56 smithi188 ceph-mon[92425]: pgmap v15893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: pgmap v15894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:28:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:28:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:28:59.246+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:28:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:28:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:28:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:28:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:28:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:28:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:28:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:28:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:28:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:28:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:28:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:28:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:28:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:28:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:28:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:28:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=951] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=951] Local address: 172.21.15.161, port 59682 2022-07-03T19:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=951] Peer address: 172.21.15.101, port 22 2022-07-03T19:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=951] Connection lost 2022-07-03T19:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=951] Aborting connection 2022-07-03T19:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:28:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: pgmap v15894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:28:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:28:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:28:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:28:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:28:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:29:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:29:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:29:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:29:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:29:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:29:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:29:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:29:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:29:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:29:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:29:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:29:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:29:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:29:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:29:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:29:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:29:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:29:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:29:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:29:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:29:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:29:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:29:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:29:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:29:00.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:29:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:29:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:29:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:29:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:29:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:29:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:29:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:29:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:29:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:29:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:29:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:29:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:29:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:29:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:29:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:29:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:29:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:29:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:29:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:29:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:29:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:29:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=951] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:29:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=951] Local address: 172.21.15.161, port 59682 2022-07-03T19:29:00.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=951] Peer address: 172.21.15.101, port 22 2022-07-03T19:29:00.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=951] Connection lost 2022-07-03T19:29:00.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=951] Aborting connection 2022-07-03T19:29:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:29:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:29:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:29:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:29:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:29:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:29:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:29:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:29:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:29:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:29:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:29:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:29:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:29:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:29:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:29:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:29:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:29:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:29:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:29:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:29:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:29:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:29:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:29:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:29:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:29:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:29:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:29:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:29:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:29:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:29:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:29:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:29:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:29:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:29:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:29:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:29:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:29:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:29:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:29:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:29:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:29:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:29:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:29:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:29:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:29:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:29:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=951] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:29:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=951] Local address: 172.21.15.161, port 59682 2022-07-03T19:29:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=951] Peer address: 172.21.15.101, port 22 2022-07-03T19:29:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=951] Connection lost 2022-07-03T19:29:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=951] Aborting connection 2022-07-03T19:29:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:01 smithi161 ceph-mon[91579]: pgmap v15895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:01 smithi188 ceph-mon[92425]: pgmap v15895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:03 smithi161 ceph-mon[91579]: pgmap v15896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:03 smithi188 ceph-mon[92425]: pgmap v15896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:04 smithi161 ceph-mon[91579]: pgmap v15897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:04 smithi188 ceph-mon[92425]: pgmap v15897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:06.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:06 smithi161 ceph-mon[91579]: pgmap v15898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:06 smithi188 ceph-mon[92425]: pgmap v15898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:09 smithi161 ceph-mon[91579]: pgmap v15899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:09 smithi188 ceph-mon[92425]: pgmap v15899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:11 smithi161 ceph-mon[91579]: pgmap v15900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:11 smithi188 ceph-mon[92425]: pgmap v15900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:13 smithi161 ceph-mon[91579]: pgmap v15901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:13 smithi188 ceph-mon[92425]: pgmap v15901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:14 smithi161 ceph-mon[91579]: pgmap v15902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:14 smithi188 ceph-mon[92425]: pgmap v15902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:16.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:16 smithi161 ceph-mon[91579]: pgmap v15903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:16 smithi188 ceph-mon[92425]: pgmap v15903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:19 smithi161 ceph-mon[91579]: pgmap v15904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:19 smithi188 ceph-mon[92425]: pgmap v15904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:21 smithi161 ceph-mon[91579]: pgmap v15905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:21 smithi188 ceph-mon[92425]: pgmap v15905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:23 smithi161 ceph-mon[91579]: pgmap v15906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:23 smithi188 ceph-mon[92425]: pgmap v15906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:24 smithi161 ceph-mon[91579]: pgmap v15907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:24 smithi188 ceph-mon[92425]: pgmap v15907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:26 smithi188 ceph-mon[92425]: pgmap v15908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:26 smithi161 ceph-mon[91579]: pgmap v15908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:29 smithi161 ceph-mon[91579]: pgmap v15909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:29 smithi188 ceph-mon[92425]: pgmap v15909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:31 smithi161 ceph-mon[91579]: pgmap v15910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:31 smithi188 ceph-mon[92425]: pgmap v15910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:33 smithi161 ceph-mon[91579]: pgmap v15911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:33 smithi188 ceph-mon[92425]: pgmap v15911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:34 smithi161 ceph-mon[91579]: pgmap v15912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:34 smithi188 ceph-mon[92425]: pgmap v15912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:36 smithi188 ceph-mon[92425]: pgmap v15913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:36 smithi161 ceph-mon[91579]: pgmap v15913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:39 smithi161 ceph-mon[91579]: pgmap v15914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:39 smithi188 ceph-mon[92425]: pgmap v15914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:41 smithi161 ceph-mon[91579]: pgmap v15915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:41 smithi188 ceph-mon[92425]: pgmap v15915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:43 smithi161 ceph-mon[91579]: pgmap v15916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:43 smithi188 ceph-mon[92425]: pgmap v15916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:44 smithi161 ceph-mon[91579]: pgmap v15917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:44 smithi188 ceph-mon[92425]: pgmap v15917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:46.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:46 smithi188 ceph-mon[92425]: pgmap v15918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:46 smithi161 ceph-mon[91579]: pgmap v15918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:49 smithi161 ceph-mon[91579]: pgmap v15919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:49 smithi188 ceph-mon[92425]: pgmap v15919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:51 smithi161 ceph-mon[91579]: pgmap v15920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:51 smithi188 ceph-mon[92425]: pgmap v15920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:53 smithi161 ceph-mon[91579]: pgmap v15921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:53 smithi188 ceph-mon[92425]: pgmap v15921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:54 smithi188 ceph-mon[92425]: pgmap v15922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:54 smithi161 ceph-mon[91579]: pgmap v15922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:29:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:56 smithi188 ceph-mon[92425]: pgmap v15923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:56 smithi161 ceph-mon[91579]: pgmap v15923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:29:59.762 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:59 smithi188 ceph-mon[92425]: pgmap v15924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:59.762 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:29:59.763 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:29:59.763 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:29:59.763 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:29:59.768 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-mon[91579]: pgmap v15924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:29:59.769 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:29:59.769 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:29:59.770 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:29:59.770 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:29:59.771 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:29:59.751+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:29:59.771 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:29:59.771 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:29:59.772 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:29:59.772 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:29:59.772 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:29:59.773 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:29:59.773 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:29:59.773 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:29:59.774 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:29:59.774 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:29:59.774 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:29:59.775 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:29:59.775 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:29:59.776 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:29:59.776 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:29:59.776 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:29:59.777 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:29:59.777 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:29:59.777 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:29:59.778 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:29:59.778 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:29:59.779 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:29:59.779 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:29:59.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:29:59.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:29:59.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:29:59.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:29:59.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:29:59.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:29:59.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:29:59.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:29:59.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:29:59.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:29:59.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:29:59.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:29:59.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:29:59.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:29:59.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:29:59.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:29:59.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:29:59.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:29:59.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:29:59.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:29:59.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:29:59.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=953] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:29:59.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=953] Local address: 172.21.15.161, port 59686 2022-07-03T19:29:59.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=953] Peer address: 172.21.15.101, port 22 2022-07-03T19:29:59.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=953] Connection lost 2022-07-03T19:29:59.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=953] Aborting connection 2022-07-03T19:29:59.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:29:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:30:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:30:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:30:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:30:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:30:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:30:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:30:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:30:00.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:30:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:30:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:30:00.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:30:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:30:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:30:00.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:30:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:30:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:30:00.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:30:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:30:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:30:00.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:30:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:30:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:30:00.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:30:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:30:00.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:30:00.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:30:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:30:00.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:30:00.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:30:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:30:00.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:30:00.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:30:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:30:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:30:00.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:30:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:30:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:30:00.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:30:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:30:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:30:00.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:30:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:30:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:30:00.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:30:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:30:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:30:00.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:30:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:30:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:30:00.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=953] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:30:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=953] Local address: 172.21.15.161, port 59686 2022-07-03T19:30:00.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=953] Peer address: 172.21.15.101, port 22 2022-07-03T19:30:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=953] Connection lost 2022-07-03T19:30:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=953] Aborting connection 2022-07-03T19:30:00.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:30:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:30:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:30:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:30:00.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:30:00.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:30:00.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:30:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:30:00.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:30:00.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:30:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:30:00.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:30:00.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:30:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:30:00.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:30:00.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:30:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:30:00.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:30:00.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:30:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:30:00.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:30:00.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:30:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:30:00.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:30:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:30:00.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:30:00.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:30:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:30:00.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:30:00.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:30:00.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:30:00.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:30:00.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:30:00.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:30:00.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:30:00.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:30:00.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:30:00.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:30:00.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:30:00.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:30:00.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:30:00.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:30:00.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:30:00.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:30:00.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:30:00.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:30:00.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:30:00.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:30:00.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:30:00.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:30:00.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=953] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:30:00.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=953] Local address: 172.21.15.161, port 59686 2022-07-03T19:30:00.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=953] Peer address: 172.21.15.101, port 22 2022-07-03T19:30:00.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=953] Connection lost 2022-07-03T19:30:00.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=953] Aborting connection 2022-07-03T19:30:00.947 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:30:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:01 smithi161 ceph-mon[91579]: pgmap v15925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:01 smithi188 ceph-mon[92425]: pgmap v15925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:03 smithi161 ceph-mon[91579]: pgmap v15926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:03 smithi188 ceph-mon[92425]: pgmap v15926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:04 smithi188 ceph-mon[92425]: pgmap v15927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:04 smithi161 ceph-mon[91579]: pgmap v15927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:06 smithi188 ceph-mon[92425]: pgmap v15928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:06 smithi161 ceph-mon[91579]: pgmap v15928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:09 smithi161 ceph-mon[91579]: pgmap v15929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:09 smithi188 ceph-mon[92425]: pgmap v15929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:11 smithi161 ceph-mon[91579]: pgmap v15930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:11 smithi188 ceph-mon[92425]: pgmap v15930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:13 smithi161 ceph-mon[91579]: pgmap v15931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:13 smithi188 ceph-mon[92425]: pgmap v15931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:14 smithi188 ceph-mon[92425]: pgmap v15932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:14 smithi161 ceph-mon[91579]: pgmap v15932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:16 smithi188 ceph-mon[92425]: pgmap v15933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:16 smithi161 ceph-mon[91579]: pgmap v15933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:19 smithi161 ceph-mon[91579]: pgmap v15934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:19 smithi188 ceph-mon[92425]: pgmap v15934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:21 smithi161 ceph-mon[91579]: pgmap v15935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:21 smithi188 ceph-mon[92425]: pgmap v15935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:23 smithi161 ceph-mon[91579]: pgmap v15936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:23 smithi188 ceph-mon[92425]: pgmap v15936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:24 smithi188 ceph-mon[92425]: pgmap v15937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:24 smithi161 ceph-mon[91579]: pgmap v15937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:26 smithi188 ceph-mon[92425]: pgmap v15938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:26 smithi161 ceph-mon[91579]: pgmap v15938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:29 smithi161 ceph-mon[91579]: pgmap v15939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:29 smithi188 ceph-mon[92425]: pgmap v15939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:31 smithi161 ceph-mon[91579]: pgmap v15940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:31 smithi188 ceph-mon[92425]: pgmap v15940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:33 smithi161 ceph-mon[91579]: pgmap v15941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:33 smithi188 ceph-mon[92425]: pgmap v15941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:34 smithi188 ceph-mon[92425]: pgmap v15942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:34 smithi161 ceph-mon[91579]: pgmap v15942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:36 smithi188 ceph-mon[92425]: pgmap v15943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:36 smithi161 ceph-mon[91579]: pgmap v15943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:39 smithi161 ceph-mon[91579]: pgmap v15944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:39 smithi188 ceph-mon[92425]: pgmap v15944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:41 smithi161 ceph-mon[91579]: pgmap v15945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:41 smithi188 ceph-mon[92425]: pgmap v15945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:43 smithi161 ceph-mon[91579]: pgmap v15946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:43 smithi188 ceph-mon[92425]: pgmap v15946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:44 smithi188 ceph-mon[92425]: pgmap v15947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:44 smithi161 ceph-mon[91579]: pgmap v15947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:46 smithi188 ceph-mon[92425]: pgmap v15948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:46 smithi161 ceph-mon[91579]: pgmap v15948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:49 smithi161 ceph-mon[91579]: pgmap v15949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:49 smithi188 ceph-mon[92425]: pgmap v15949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:51 smithi161 ceph-mon[91579]: pgmap v15950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:51 smithi188 ceph-mon[92425]: pgmap v15950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:53 smithi161 ceph-mon[91579]: pgmap v15951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:53 smithi188 ceph-mon[92425]: pgmap v15951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:54 smithi188 ceph-mon[92425]: pgmap v15952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:54 smithi161 ceph-mon[91579]: pgmap v15952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:30:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:56 smithi188 ceph-mon[92425]: pgmap v15953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:56 smithi161 ceph-mon[91579]: pgmap v15953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:30:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:59 smithi161 ceph-mon[91579]: pgmap v15954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:30:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:30:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:30:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:30:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:59 smithi188 ceph-mon[92425]: pgmap v15954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:30:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:30:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:30:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:30:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:31:00.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:31:00.255+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:31:00.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:31:00.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:31:00.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:31:00.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:31:00.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:31:00.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:31:00.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:31:00.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:31:00.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:31:00.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:31:00.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:31:00.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:31:00.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:31:00.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:31:00.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:31:00.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:31:00.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:31:00.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:31:00.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:31:00.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:31:00.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:31:00.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:31:00.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:31:00.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:31:00.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:31:00.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:31:00.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:31:00.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:31:00.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:31:00.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:31:00.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:31:00.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:31:00.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:31:00.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:31:00.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:31:00.541 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:31:00.541 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:31:00.541 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:31:00.542 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:31:00.542 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:31:00.543 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:31:00.543 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:31:00.543 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:31:00.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:31:00.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=955] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:31:00.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=955] Local address: 172.21.15.161, port 59690 2022-07-03T19:31:00.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=955] Peer address: 172.21.15.101, port 22 2022-07-03T19:31:00.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=955] Connection lost 2022-07-03T19:31:00.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=955] Aborting connection 2022-07-03T19:31:00.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:31:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:31:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:31:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:31:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:31:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:31:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:31:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:31:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:31:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:31:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:31:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:31:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:31:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:31:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:31:01.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:31:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:31:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:31:01.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:31:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:31:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:31:01.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:31:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:31:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:31:01.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:31:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:31:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:31:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:31:01.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:31:01.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:31:01.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:31:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:31:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:31:01.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:31:01.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:31:01.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:31:01.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:31:01.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:31:01.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:31:01.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:31:01.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:31:01.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:31:01.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:31:01.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:31:01.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:31:01.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:31:01.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:31:01.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:31:01.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:31:01.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:31:01.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=955] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:31:01.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=955] Local address: 172.21.15.161, port 59690 2022-07-03T19:31:01.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=955] Peer address: 172.21.15.101, port 22 2022-07-03T19:31:01.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=955] Connection lost 2022-07-03T19:31:01.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=955] Aborting connection 2022-07-03T19:31:01.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:01 smithi161 ceph-mon[91579]: pgmap v15955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:01.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:31:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:31:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:31:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:31:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:31:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:31:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:31:01.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:31:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:31:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:31:01.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:31:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:31:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:31:01.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:31:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:31:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:31:01.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:31:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:31:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:31:01.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:31:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:31:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:31:01.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:31:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:31:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:31:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:31:01.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:31:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:31:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:31:01.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:31:01.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:31:01.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:31:01.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:31:01.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:31:01.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:31:01.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:31:01.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:31:01.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:31:01.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:31:01.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:31:01.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:31:01.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:31:01.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:31:01.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:31:01.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:31:01.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:31:01.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=955] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:31:01.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=955] Local address: 172.21.15.161, port 59690 2022-07-03T19:31:01.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=955] Peer address: 172.21.15.101, port 22 2022-07-03T19:31:01.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=955] Connection lost 2022-07-03T19:31:01.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=955] Aborting connection 2022-07-03T19:31:01.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:01 smithi188 ceph-mon[92425]: pgmap v15955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:01.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: pgmap v15956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:03 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: pgmap v15956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:31:03.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:03.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:03 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:31:04.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:04 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 106.6M 2022-07-03T19:31:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:04 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 111802163: error parsing value: Value '111802163' is below minimum 939524096 2022-07-03T19:31:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:04 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 106.6M 2022-07-03T19:31:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:04 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 111802163: error parsing value: Value '111802163' is below minimum 939524096 2022-07-03T19:31:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:05 smithi161 ceph-mon[91579]: pgmap v15957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:05 smithi188 ceph-mon[92425]: pgmap v15957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:07 smithi161 ceph-mon[91579]: pgmap v15958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:07 smithi188 ceph-mon[92425]: pgmap v15958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:09 smithi161 ceph-mon[91579]: pgmap v15959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:09 smithi188 ceph-mon[92425]: pgmap v15959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:11 smithi161 ceph-mon[91579]: pgmap v15960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:11 smithi188 ceph-mon[92425]: pgmap v15960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:13 smithi161 ceph-mon[91579]: pgmap v15961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:13 smithi188 ceph-mon[92425]: pgmap v15961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:14 smithi188 ceph-mon[92425]: pgmap v15962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:14 smithi161 ceph-mon[91579]: pgmap v15962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:16 smithi188 ceph-mon[92425]: pgmap v15963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:16 smithi161 ceph-mon[91579]: pgmap v15963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:19 smithi161 ceph-mon[91579]: pgmap v15964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:19 smithi188 ceph-mon[92425]: pgmap v15964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:21 smithi161 ceph-mon[91579]: pgmap v15965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:21 smithi188 ceph-mon[92425]: pgmap v15965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:23 smithi161 ceph-mon[91579]: pgmap v15966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:23 smithi188 ceph-mon[92425]: pgmap v15966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:24 smithi188 ceph-mon[92425]: pgmap v15967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:24 smithi161 ceph-mon[91579]: pgmap v15967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:26 smithi188 ceph-mon[92425]: pgmap v15968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:26 smithi161 ceph-mon[91579]: pgmap v15968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:31:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:29 smithi161 ceph-mon[91579]: pgmap v15969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:29 smithi188 ceph-mon[92425]: pgmap v15969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:31 smithi161 ceph-mon[91579]: pgmap v15970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:31 smithi188 ceph-mon[92425]: pgmap v15970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:33 smithi161 ceph-mon[91579]: pgmap v15971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:33 smithi188 ceph-mon[92425]: pgmap v15971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:34 smithi188 ceph-mon[92425]: pgmap v15972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:34 smithi161 ceph-mon[91579]: pgmap v15972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:36 smithi188 ceph-mon[92425]: pgmap v15973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:36 smithi161 ceph-mon[91579]: pgmap v15973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:39 smithi161 ceph-mon[91579]: pgmap v15974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:39 smithi188 ceph-mon[92425]: pgmap v15974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:41 smithi161 ceph-mon[91579]: pgmap v15975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:41 smithi188 ceph-mon[92425]: pgmap v15975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:43 smithi161 ceph-mon[91579]: pgmap v15976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:43 smithi188 ceph-mon[92425]: pgmap v15976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:44 smithi188 ceph-mon[92425]: pgmap v15977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:44 smithi161 ceph-mon[91579]: pgmap v15977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:46 smithi188 ceph-mon[92425]: pgmap v15978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:46 smithi161 ceph-mon[91579]: pgmap v15978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:49 smithi161 ceph-mon[91579]: pgmap v15979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:49 smithi188 ceph-mon[92425]: pgmap v15979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:51 smithi161 ceph-mon[91579]: pgmap v15980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:51 smithi188 ceph-mon[92425]: pgmap v15980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:53 smithi161 ceph-mon[91579]: pgmap v15981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:53 smithi188 ceph-mon[92425]: pgmap v15981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:54 smithi188 ceph-mon[92425]: pgmap v15982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:54 smithi161 ceph-mon[91579]: pgmap v15982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:31:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:56 smithi188 ceph-mon[92425]: pgmap v15983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:56 smithi161 ceph-mon[91579]: pgmap v15983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:31:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:59 smithi161 ceph-mon[91579]: pgmap v15984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:31:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:31:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:31:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:31:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:59 smithi188 ceph-mon[92425]: pgmap v15984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:31:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:31:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:31:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:31:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:32:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:01 smithi161 ceph-mon[91579]: pgmap v15985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:01 smithi188 ceph-mon[92425]: pgmap v15985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-mon[91579]: pgmap v15986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:32:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:32:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:32:03.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:32:03.506+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:32:03.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:32:03.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:32:03.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:32:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:32:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:32:03.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:32:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:32:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:32:03.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:32:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:32:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:32:03.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:32:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:32:03.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:32:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:32:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:32:03.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:32:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:32:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:32:03.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:32:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:32:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:32:03.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:32:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:32:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:32:03.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:32:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:32:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:32:03.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:32:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:32:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:32:03.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:32:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:32:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:32:03.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:32:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:32:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:32:03.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:32:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:32:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:32:03.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:32:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:32:03.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:32:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:32:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=957] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:32:03.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=957] Local address: 172.21.15.161, port 59694 2022-07-03T19:32:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=957] Peer address: 172.21.15.101, port 22 2022-07-03T19:32:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=957] Connection lost 2022-07-03T19:32:03.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=957] Aborting connection 2022-07-03T19:32:03.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:32:03 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:32:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:03 smithi188 ceph-mon[92425]: pgmap v15986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:32:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:32:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:32:04.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:32:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:32:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:32:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:32:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:32:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:32:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:32:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:32:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:32:04.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:32:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:32:04.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:32:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:32:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:32:04.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:32:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:32:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:32:04.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:32:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:32:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:32:04.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:32:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:32:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:32:04.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:32:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:32:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:32:04.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:32:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:32:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:32:04.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:32:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:32:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:32:04.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:32:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:32:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:32:04.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:32:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:32:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:32:04.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:32:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:32:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:32:04.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:32:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:32:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:32:04.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:32:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:32:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=957] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:32:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=957] Local address: 172.21.15.161, port 59694 2022-07-03T19:32:04.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=957] Peer address: 172.21.15.101, port 22 2022-07-03T19:32:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=957] Connection lost 2022-07-03T19:32:04.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=957] Aborting connection 2022-07-03T19:32:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:32:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:32:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:32:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:32:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:32:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:32:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:32:04.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:32:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:32:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:32:04.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:32:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:32:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:32:04.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:32:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:32:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:32:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:32:04.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:32:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:32:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:32:04.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:32:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:32:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:32:04.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:32:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:32:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:32:04.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:32:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:32:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:32:04.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:32:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:32:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:32:04.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:32:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:32:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:32:04.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:32:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:32:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:32:04.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:32:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:32:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:32:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:32:04.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:32:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:32:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:32:04.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:32:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=957] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:32:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=957] Local address: 172.21.15.161, port 59694 2022-07-03T19:32:04.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=957] Peer address: 172.21.15.101, port 22 2022-07-03T19:32:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=957] Connection lost 2022-07-03T19:32:04.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=957] Aborting connection 2022-07-03T19:32:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:05 smithi161 ceph-mon[91579]: pgmap v15987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:05 smithi188 ceph-mon[92425]: pgmap v15987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:07 smithi161 ceph-mon[91579]: pgmap v15988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:07 smithi188 ceph-mon[92425]: pgmap v15988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:09 smithi161 ceph-mon[91579]: pgmap v15989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:09 smithi188 ceph-mon[92425]: pgmap v15989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:11 smithi161 ceph-mon[91579]: pgmap v15990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:11 smithi188 ceph-mon[92425]: pgmap v15990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:13 smithi161 ceph-mon[91579]: pgmap v15991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:13 smithi188 ceph-mon[92425]: pgmap v15991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:14 smithi161 ceph-mon[91579]: pgmap v15992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:14 smithi188 ceph-mon[92425]: pgmap v15992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:16 smithi161 ceph-mon[91579]: pgmap v15993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:16 smithi188 ceph-mon[92425]: pgmap v15993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:19 smithi161 ceph-mon[91579]: pgmap v15994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:19 smithi188 ceph-mon[92425]: pgmap v15994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:21 smithi161 ceph-mon[91579]: pgmap v15995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:21 smithi188 ceph-mon[92425]: pgmap v15995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:23 smithi161 ceph-mon[91579]: pgmap v15996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:23 smithi188 ceph-mon[92425]: pgmap v15996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:24 smithi161 ceph-mon[91579]: pgmap v15997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:24 smithi188 ceph-mon[92425]: pgmap v15997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:26 smithi161 ceph-mon[91579]: pgmap v15998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:26 smithi188 ceph-mon[92425]: pgmap v15998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:29 smithi161 ceph-mon[91579]: pgmap v15999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:29 smithi188 ceph-mon[92425]: pgmap v15999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:31 smithi161 ceph-mon[91579]: pgmap v16000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:31 smithi188 ceph-mon[92425]: pgmap v16000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:33 smithi161 ceph-mon[91579]: pgmap v16001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:33 smithi188 ceph-mon[92425]: pgmap v16001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:34 smithi161 ceph-mon[91579]: pgmap v16002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:34 smithi188 ceph-mon[92425]: pgmap v16002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:36 smithi161 ceph-mon[91579]: pgmap v16003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:36 smithi188 ceph-mon[92425]: pgmap v16003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:39 smithi161 ceph-mon[91579]: pgmap v16004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:39 smithi188 ceph-mon[92425]: pgmap v16004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:41 smithi161 ceph-mon[91579]: pgmap v16005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:41 smithi188 ceph-mon[92425]: pgmap v16005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:43 smithi161 ceph-mon[91579]: pgmap v16006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:43.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:43 smithi188 ceph-mon[92425]: pgmap v16006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:44 smithi161 ceph-mon[91579]: pgmap v16007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:44 smithi188 ceph-mon[92425]: pgmap v16007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:46 smithi161 ceph-mon[91579]: pgmap v16008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:46 smithi188 ceph-mon[92425]: pgmap v16008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:49 smithi161 ceph-mon[91579]: pgmap v16009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:49 smithi188 ceph-mon[92425]: pgmap v16009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:51 smithi161 ceph-mon[91579]: pgmap v16010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:51 smithi188 ceph-mon[92425]: pgmap v16010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:53 smithi161 ceph-mon[91579]: pgmap v16011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:53.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:53 smithi188 ceph-mon[92425]: pgmap v16011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:54 smithi161 ceph-mon[91579]: pgmap v16012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:54 smithi188 ceph-mon[92425]: pgmap v16012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:32:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:56 smithi161 ceph-mon[91579]: pgmap v16013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:56 smithi188 ceph-mon[92425]: pgmap v16013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:32:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:59 smithi161 ceph-mon[91579]: pgmap v16014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:32:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:32:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:32:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:32:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:59 smithi188 ceph-mon[92425]: pgmap v16014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:32:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:32:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:32:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:32:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:33:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:01 smithi161 ceph-mon[91579]: pgmap v16015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:01 smithi188 ceph-mon[92425]: pgmap v16015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:03 smithi161 ceph-mon[91579]: pgmap v16016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:03 smithi188 ceph-mon[92425]: pgmap v16016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:04.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:33:04.008+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:33:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:33:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:33:04.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:33:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:33:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:33:04.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:33:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:33:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:33:04.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:33:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:33:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:33:04.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:33:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:33:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:33:04.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:33:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:33:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:33:04.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:33:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:33:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:33:04.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:33:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:33:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:33:04.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:33:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:33:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:33:04.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:33:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:33:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:33:04.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:33:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:33:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:33:04.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:33:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:33:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:33:04.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:33:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:33:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:33:04.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:33:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:33:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:33:04.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:33:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:33:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:33:04.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=959] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:33:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=959] Local address: 172.21.15.161, port 59698 2022-07-03T19:33:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=959] Peer address: 172.21.15.101, port 22 2022-07-03T19:33:04.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=959] Connection lost 2022-07-03T19:33:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=959] Aborting connection 2022-07-03T19:33:04.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:33:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:33:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:33:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:33:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:33:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:33:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:33:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:33:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:33:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:33:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:33:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:33:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:33:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:33:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:33:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:33:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:33:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:33:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:33:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:33:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:33:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:33:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:33:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:33:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:33:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:33:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:33:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:33:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:33:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:33:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:33:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:33:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:33:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:33:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:33:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:33:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:33:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:33:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:33:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:33:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:33:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:33:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:33:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:33:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:33:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:33:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:33:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:33:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=959] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:33:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=959] Local address: 172.21.15.161, port 59698 2022-07-03T19:33:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=959] Peer address: 172.21.15.101, port 22 2022-07-03T19:33:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=959] Connection lost 2022-07-03T19:33:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=959] Aborting connection 2022-07-03T19:33:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:04 smithi161 ceph-mon[91579]: pgmap v16017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:33:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:33:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:33:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:33:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:33:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:33:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:33:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:33:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:33:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:33:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:33:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:33:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:33:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:33:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:33:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:33:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:33:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:33:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:33:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:33:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:33:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:33:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:33:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:33:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:33:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:33:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:33:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:33:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:33:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:33:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:33:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:33:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:33:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:33:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:33:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:33:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:33:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:33:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:33:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:33:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:33:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:33:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:33:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:33:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:33:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:33:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:33:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:33:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:33:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=959] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:33:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=959] Local address: 172.21.15.161, port 59698 2022-07-03T19:33:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=959] Peer address: 172.21.15.101, port 22 2022-07-03T19:33:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=959] Connection lost 2022-07-03T19:33:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=959] Aborting connection 2022-07-03T19:33:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:04 smithi188 ceph-mon[92425]: pgmap v16017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:06 smithi161 ceph-mon[91579]: pgmap v16018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:06 smithi188 ceph-mon[92425]: pgmap v16018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:09 smithi161 ceph-mon[91579]: pgmap v16019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:09 smithi188 ceph-mon[92425]: pgmap v16019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:11 smithi161 ceph-mon[91579]: pgmap v16020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:11 smithi188 ceph-mon[92425]: pgmap v16020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:13 smithi161 ceph-mon[91579]: pgmap v16021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:13 smithi188 ceph-mon[92425]: pgmap v16021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:14 smithi161 ceph-mon[91579]: pgmap v16022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:14 smithi188 ceph-mon[92425]: pgmap v16022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:16 smithi161 ceph-mon[91579]: pgmap v16023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:16 smithi188 ceph-mon[92425]: pgmap v16023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:19 smithi161 ceph-mon[91579]: pgmap v16024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:19 smithi188 ceph-mon[92425]: pgmap v16024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:21 smithi161 ceph-mon[91579]: pgmap v16025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:21 smithi188 ceph-mon[92425]: pgmap v16025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:23 smithi161 ceph-mon[91579]: pgmap v16026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:23 smithi188 ceph-mon[92425]: pgmap v16026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:24 smithi161 ceph-mon[91579]: pgmap v16027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:24 smithi188 ceph-mon[92425]: pgmap v16027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:26 smithi161 ceph-mon[91579]: pgmap v16028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:26 smithi188 ceph-mon[92425]: pgmap v16028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:29 smithi161 ceph-mon[91579]: pgmap v16029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:29 smithi188 ceph-mon[92425]: pgmap v16029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:31 smithi161 ceph-mon[91579]: pgmap v16030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:31 smithi188 ceph-mon[92425]: pgmap v16030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:33 smithi161 ceph-mon[91579]: pgmap v16031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:33 smithi188 ceph-mon[92425]: pgmap v16031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:34 smithi161 ceph-mon[91579]: pgmap v16032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:34 smithi188 ceph-mon[92425]: pgmap v16032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:36 smithi161 ceph-mon[91579]: pgmap v16033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:36 smithi188 ceph-mon[92425]: pgmap v16033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:39 smithi161 ceph-mon[91579]: pgmap v16034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:39 smithi188 ceph-mon[92425]: pgmap v16034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:41 smithi161 ceph-mon[91579]: pgmap v16035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:41 smithi188 ceph-mon[92425]: pgmap v16035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:43 smithi161 ceph-mon[91579]: pgmap v16036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:43 smithi188 ceph-mon[92425]: pgmap v16036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:44 smithi161 ceph-mon[91579]: pgmap v16037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:44 smithi188 ceph-mon[92425]: pgmap v16037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:46 smithi161 ceph-mon[91579]: pgmap v16038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:46 smithi188 ceph-mon[92425]: pgmap v16038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:49 smithi161 ceph-mon[91579]: pgmap v16039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:49 smithi188 ceph-mon[92425]: pgmap v16039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:51 smithi161 ceph-mon[91579]: pgmap v16040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:51 smithi188 ceph-mon[92425]: pgmap v16040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:53 smithi161 ceph-mon[91579]: pgmap v16041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:53 smithi188 ceph-mon[92425]: pgmap v16041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:54 smithi161 ceph-mon[91579]: pgmap v16042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:54 smithi188 ceph-mon[92425]: pgmap v16042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:33:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:56 smithi161 ceph-mon[91579]: pgmap v16043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:56 smithi188 ceph-mon[92425]: pgmap v16043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:33:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:59 smithi161 ceph-mon[91579]: pgmap v16044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:33:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:33:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:33:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:33:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:59 smithi188 ceph-mon[92425]: pgmap v16044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:33:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:33:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:33:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:33:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:34:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:01 smithi161 ceph-mon[91579]: pgmap v16045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:01 smithi188 ceph-mon[92425]: pgmap v16045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:03 smithi161 ceph-mon[91579]: pgmap v16046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:03 smithi188 ceph-mon[92425]: pgmap v16046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:04.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:34:04.509+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:34:04.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:34:04.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:34:04.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:34:04.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:34:04.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:34:04.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:34:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:34:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:34:04.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:34:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:34:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:34:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:34:04.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:34:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:34:04.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:34:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:34:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:34:04.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:34:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:34:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:34:04.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:34:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:34:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:34:04.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:34:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:34:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:34:04.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:34:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:34:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:34:04.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:34:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:34:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:34:04.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:34:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:34:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:34:04.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:34:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:34:04.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:34:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:34:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:34:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:34:04.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:34:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:34:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:34:04.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=961] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:34:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=961] Local address: 172.21.15.161, port 59702 2022-07-03T19:34:04.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=961] Peer address: 172.21.15.101, port 22 2022-07-03T19:34:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=961] Connection lost 2022-07-03T19:34:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=961] Aborting connection 2022-07-03T19:34:04.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:34:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:34:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:34:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:34:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:34:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:34:05.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:34:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:34:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:34:05.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:34:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:34:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:34:05.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:34:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:34:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:34:05.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:34:05.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:34:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:34:05.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:34:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:34:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:34:05.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:34:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:34:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:34:05.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:34:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:34:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:34:05.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:34:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:34:05.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:34:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:34:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:34:05.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:34:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:34:05.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:34:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:34:05.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:34:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:34:05.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:34:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:34:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:34:05.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:34:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:34:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:34:05.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:34:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:34:05.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:34:05.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:34:05.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=961] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:34:05.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=961] Local address: 172.21.15.161, port 59702 2022-07-03T19:34:05.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=961] Peer address: 172.21.15.101, port 22 2022-07-03T19:34:05.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=961] Connection lost 2022-07-03T19:34:05.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=961] Aborting connection 2022-07-03T19:34:05.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:04 smithi161 ceph-mon[91579]: pgmap v16047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:34:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:34:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:34:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:34:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:34:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:34:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:34:05.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:34:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:34:05.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:34:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:34:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:34:05.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:34:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:34:05.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:34:05.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:34:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:34:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:34:05.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:34:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:34:05.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:34:05.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:34:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:34:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:34:05.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:34:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:34:05.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:34:05.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:34:05.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:34:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:34:05.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:34:05.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:34:05.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:34:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:34:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:34:05.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=961] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:34:05.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=961] Local address: 172.21.15.161, port 59702 2022-07-03T19:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=961] Peer address: 172.21.15.101, port 22 2022-07-03T19:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=961] Connection lost 2022-07-03T19:34:05.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=961] Aborting connection 2022-07-03T19:34:05.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:04 smithi188 ceph-mon[92425]: pgmap v16047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:06 smithi161 ceph-mon[91579]: pgmap v16048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:06 smithi188 ceph-mon[92425]: pgmap v16048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:09 smithi161 ceph-mon[91579]: pgmap v16049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:09 smithi188 ceph-mon[92425]: pgmap v16049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:11 smithi161 ceph-mon[91579]: pgmap v16050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:11 smithi188 ceph-mon[92425]: pgmap v16050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:13 smithi161 ceph-mon[91579]: pgmap v16051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:13 smithi188 ceph-mon[92425]: pgmap v16051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:14 smithi161 ceph-mon[91579]: pgmap v16052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:14 smithi188 ceph-mon[92425]: pgmap v16052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:16 smithi161 ceph-mon[91579]: pgmap v16053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:16 smithi188 ceph-mon[92425]: pgmap v16053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:19 smithi161 ceph-mon[91579]: pgmap v16054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:19 smithi188 ceph-mon[92425]: pgmap v16054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:21 smithi161 ceph-mon[91579]: pgmap v16055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:21 smithi188 ceph-mon[92425]: pgmap v16055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:23 smithi161 ceph-mon[91579]: pgmap v16056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:23.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:23 smithi188 ceph-mon[92425]: pgmap v16056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:24 smithi161 ceph-mon[91579]: pgmap v16057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:24 smithi188 ceph-mon[92425]: pgmap v16057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:26 smithi161 ceph-mon[91579]: pgmap v16058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:26 smithi188 ceph-mon[92425]: pgmap v16058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:29 smithi161 ceph-mon[91579]: pgmap v16059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:29 smithi188 ceph-mon[92425]: pgmap v16059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:31 smithi161 ceph-mon[91579]: pgmap v16060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:31 smithi188 ceph-mon[92425]: pgmap v16060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:33 smithi161 ceph-mon[91579]: pgmap v16061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:33.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:33 smithi188 ceph-mon[92425]: pgmap v16061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:34 smithi161 ceph-mon[91579]: pgmap v16062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:34 smithi188 ceph-mon[92425]: pgmap v16062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:36 smithi161 ceph-mon[91579]: pgmap v16063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:36 smithi188 ceph-mon[92425]: pgmap v16063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:39 smithi161 ceph-mon[91579]: pgmap v16064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:39 smithi188 ceph-mon[92425]: pgmap v16064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:41 smithi161 ceph-mon[91579]: pgmap v16065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:41 smithi188 ceph-mon[92425]: pgmap v16065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:43 smithi161 ceph-mon[91579]: pgmap v16066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:43 smithi188 ceph-mon[92425]: pgmap v16066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:44 smithi161 ceph-mon[91579]: pgmap v16067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:44 smithi188 ceph-mon[92425]: pgmap v16067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:46 smithi161 ceph-mon[91579]: pgmap v16068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:46 smithi188 ceph-mon[92425]: pgmap v16068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:49 smithi161 ceph-mon[91579]: pgmap v16069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:49 smithi188 ceph-mon[92425]: pgmap v16069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:51 smithi161 ceph-mon[91579]: pgmap v16070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:51 smithi188 ceph-mon[92425]: pgmap v16070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:53 smithi161 ceph-mon[91579]: pgmap v16071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:53 smithi188 ceph-mon[92425]: pgmap v16071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:54 smithi161 ceph-mon[91579]: pgmap v16072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:54 smithi188 ceph-mon[92425]: pgmap v16072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:34:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:56 smithi161 ceph-mon[91579]: pgmap v16073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:56 smithi188 ceph-mon[92425]: pgmap v16073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:34:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:59 smithi161 ceph-mon[91579]: pgmap v16074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:34:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:34:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:34:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:34:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:59 smithi188 ceph-mon[92425]: pgmap v16074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:34:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:34:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:34:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:34:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:35:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:01 smithi161 ceph-mon[91579]: pgmap v16075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:01 smithi188 ceph-mon[92425]: pgmap v16075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:03 smithi161 ceph-mon[91579]: pgmap v16076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:03.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:03 smithi188 ceph-mon[92425]: pgmap v16076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:04 smithi161 ceph-mon[91579]: pgmap v16077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:05.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:35:05.006+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:35:05.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:35:05.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:35:05.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:35:05.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:35:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:35:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:35:05.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:35:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:35:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:35:05.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:35:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:35:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:35:05.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:35:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:35:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:35:05.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:35:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:35:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:35:05.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:35:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:35:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:35:05.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:35:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:35:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:35:05.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:35:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:35:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:35:05.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:35:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:35:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:35:05.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:35:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:35:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:35:05.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:35:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:35:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:35:05.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:35:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:35:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:35:05.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:35:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:35:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:35:05.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:35:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:35:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=963] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:35:05.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=963] Local address: 172.21.15.161, port 59706 2022-07-03T19:35:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=963] Peer address: 172.21.15.101, port 22 2022-07-03T19:35:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=963] Connection lost 2022-07-03T19:35:05.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=963] Aborting connection 2022-07-03T19:35:05.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:35:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:04 smithi188 ceph-mon[92425]: pgmap v16077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:35:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:35:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:35:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:35:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:35:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:35:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:35:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:35:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:35:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:35:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:35:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:35:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:35:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:35:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:35:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:35:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:35:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:35:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:35:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:35:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:35:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:35:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:35:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:35:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:35:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:35:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:35:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:35:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:35:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:35:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:35:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:35:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:35:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:35:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:35:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:35:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:35:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:35:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:35:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:35:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:35:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:35:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:35:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:35:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:35:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:35:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:35:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:35:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:35:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=963] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:35:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=963] Local address: 172.21.15.161, port 59706 2022-07-03T19:35:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=963] Peer address: 172.21.15.101, port 22 2022-07-03T19:35:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=963] Connection lost 2022-07-03T19:35:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=963] Aborting connection 2022-07-03T19:35:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:35:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:35:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:35:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:35:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:35:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:35:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:35:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:35:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:35:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:35:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:35:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:35:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:35:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:35:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:35:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:35:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:35:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:35:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:35:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:35:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:35:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:35:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:35:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:35:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:35:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:35:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:35:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:35:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:35:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:35:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:35:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:35:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:35:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:35:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:35:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:35:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:35:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:35:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:35:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:35:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:35:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:35:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:35:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:35:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:35:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:35:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:35:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:35:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:35:06.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=963] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:35:06.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=963] Local address: 172.21.15.161, port 59706 2022-07-03T19:35:06.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=963] Peer address: 172.21.15.101, port 22 2022-07-03T19:35:06.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=963] Connection lost 2022-07-03T19:35:06.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=963] Aborting connection 2022-07-03T19:35:06.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:06 smithi161 ceph-mon[91579]: pgmap v16078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:06 smithi188 ceph-mon[92425]: pgmap v16078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:09 smithi161 ceph-mon[91579]: pgmap v16079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:09 smithi188 ceph-mon[92425]: pgmap v16079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:11 smithi161 ceph-mon[91579]: pgmap v16080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:11.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:11 smithi188 ceph-mon[92425]: pgmap v16080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:13.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:13 smithi161 ceph-mon[91579]: pgmap v16081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:13.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:13 smithi188 ceph-mon[92425]: pgmap v16081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:14 smithi161 ceph-mon[91579]: pgmap v16082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:14 smithi188 ceph-mon[92425]: pgmap v16082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:16 smithi161 ceph-mon[91579]: pgmap v16083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:16 smithi188 ceph-mon[92425]: pgmap v16083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:19 smithi161 ceph-mon[91579]: pgmap v16084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:19 smithi188 ceph-mon[92425]: pgmap v16084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:21.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:21 smithi161 ceph-mon[91579]: pgmap v16085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:21 smithi188 ceph-mon[92425]: pgmap v16085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:23 smithi161 ceph-mon[91579]: pgmap v16086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:23.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:23 smithi188 ceph-mon[92425]: pgmap v16086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:24 smithi161 ceph-mon[91579]: pgmap v16087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:24 smithi188 ceph-mon[92425]: pgmap v16087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:26 smithi161 ceph-mon[91579]: pgmap v16088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:26 smithi188 ceph-mon[92425]: pgmap v16088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:29 smithi161 ceph-mon[91579]: pgmap v16089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:29 smithi188 ceph-mon[92425]: pgmap v16089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:31.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:31 smithi161 ceph-mon[91579]: pgmap v16090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:31 smithi188 ceph-mon[92425]: pgmap v16090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:33 smithi161 ceph-mon[91579]: pgmap v16091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:33.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:33 smithi188 ceph-mon[92425]: pgmap v16091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:34 smithi161 ceph-mon[91579]: pgmap v16092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:34 smithi188 ceph-mon[92425]: pgmap v16092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:36 smithi161 ceph-mon[91579]: pgmap v16093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:36 smithi188 ceph-mon[92425]: pgmap v16093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:39 smithi161 ceph-mon[91579]: pgmap v16094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:39 smithi188 ceph-mon[92425]: pgmap v16094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:41 smithi161 ceph-mon[91579]: pgmap v16095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:41 smithi188 ceph-mon[92425]: pgmap v16095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:43 smithi161 ceph-mon[91579]: pgmap v16096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:43 smithi188 ceph-mon[92425]: pgmap v16096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:44 smithi161 ceph-mon[91579]: pgmap v16097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:44 smithi188 ceph-mon[92425]: pgmap v16097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:46 smithi161 ceph-mon[91579]: pgmap v16098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:46 smithi188 ceph-mon[92425]: pgmap v16098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:49 smithi161 ceph-mon[91579]: pgmap v16099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:49 smithi188 ceph-mon[92425]: pgmap v16099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:51.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:51 smithi161 ceph-mon[91579]: pgmap v16100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:51 smithi188 ceph-mon[92425]: pgmap v16100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:53.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:53 smithi161 ceph-mon[91579]: pgmap v16101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:53.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:53 smithi188 ceph-mon[92425]: pgmap v16101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:54 smithi161 ceph-mon[91579]: pgmap v16102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:54 smithi188 ceph-mon[92425]: pgmap v16102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:35:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:56 smithi161 ceph-mon[91579]: pgmap v16103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:56 smithi188 ceph-mon[92425]: pgmap v16103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:35:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:59 smithi161 ceph-mon[91579]: pgmap v16104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:35:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:35:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:35:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:35:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:59 smithi188 ceph-mon[92425]: pgmap v16104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:35:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:35:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:35:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:35:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:36:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:01 smithi161 ceph-mon[91579]: pgmap v16105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:01 smithi188 ceph-mon[92425]: pgmap v16105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:03.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:03 smithi161 ceph-mon[91579]: pgmap v16106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:03.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:03 smithi188 ceph-mon[92425]: pgmap v16106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:04 smithi161 ceph-mon[91579]: pgmap v16107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:04 smithi188 ceph-mon[92425]: pgmap v16107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:05.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:36:05.509+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:36:05.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:36:05.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:36:05.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:36:05.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:36:05.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:36:05.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:36:05.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:36:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:36:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:36:05.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:36:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:36:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:36:05.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:36:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:36:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:36:05.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:36:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:36:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:36:05.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:36:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:36:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:36:05.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:36:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:36:05.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:36:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:36:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:36:05.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:36:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:36:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:36:05.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:36:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:36:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:36:05.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:36:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:36:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:36:05.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:36:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:36:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:36:05.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:36:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:36:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:36:05.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:36:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:36:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:36:05.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=965] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:36:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=965] Local address: 172.21.15.161, port 59710 2022-07-03T19:36:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=965] Peer address: 172.21.15.101, port 22 2022-07-03T19:36:05.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=965] Connection lost 2022-07-03T19:36:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=965] Aborting connection 2022-07-03T19:36:05.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:36:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:36:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:36:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:36:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:36:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:36:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:36:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:36:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:36:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:36:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:36:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:36:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:36:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:36:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:36:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:36:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:36:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:36:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:36:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:36:06.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:36:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:36:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:36:06.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:36:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:36:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:36:06.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:36:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:36:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:36:06.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:36:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:36:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:36:06.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:36:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:36:06.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:36:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:36:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:36:06.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:36:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:36:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:36:06.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:36:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:36:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:36:06.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:36:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:36:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:36:06.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:36:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:36:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:36:06.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:36:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=965] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:36:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=965] Local address: 172.21.15.161, port 59710 2022-07-03T19:36:06.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=965] Peer address: 172.21.15.101, port 22 2022-07-03T19:36:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=965] Connection lost 2022-07-03T19:36:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=965] Aborting connection 2022-07-03T19:36:06.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:06.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:36:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:36:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:36:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:36:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:36:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:36:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:36:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:36:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:36:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:36:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:36:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:36:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:36:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:36:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:36:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:36:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:36:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:36:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:36:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:36:06.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:36:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:36:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:36:06.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:36:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:36:06.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:36:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:36:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:36:06.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:36:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:36:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:36:06.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:36:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:36:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:36:06.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:36:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:36:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:36:06.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:36:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:36:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:36:06.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:36:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:36:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:36:06.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:36:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:36:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:36:06.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:36:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:36:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:36:06.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=965] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:36:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=965] Local address: 172.21.15.161, port 59710 2022-07-03T19:36:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=965] Peer address: 172.21.15.101, port 22 2022-07-03T19:36:06.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=965] Connection lost 2022-07-03T19:36:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=965] Aborting connection 2022-07-03T19:36:06.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:06 smithi161 ceph-mon[91579]: pgmap v16108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:06 smithi188 ceph-mon[92425]: pgmap v16108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:09 smithi161 ceph-mon[91579]: pgmap v16109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:09 smithi188 ceph-mon[92425]: pgmap v16109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:11 smithi161 ceph-mon[91579]: pgmap v16110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:11 smithi188 ceph-mon[92425]: pgmap v16110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:13.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:13 smithi188 ceph-mon[92425]: pgmap v16111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:13 smithi161 ceph-mon[91579]: pgmap v16111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:14 smithi161 ceph-mon[91579]: pgmap v16112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:14 smithi188 ceph-mon[92425]: pgmap v16112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:16 smithi161 ceph-mon[91579]: pgmap v16113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:16 smithi188 ceph-mon[92425]: pgmap v16113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:19 smithi161 ceph-mon[91579]: pgmap v16114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:19 smithi188 ceph-mon[92425]: pgmap v16114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:21 smithi188 ceph-mon[92425]: pgmap v16115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:21 smithi161 ceph-mon[91579]: pgmap v16115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:23 smithi188 ceph-mon[92425]: pgmap v16116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:23 smithi161 ceph-mon[91579]: pgmap v16116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:24 smithi161 ceph-mon[91579]: pgmap v16117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:24 smithi188 ceph-mon[92425]: pgmap v16117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:26 smithi161 ceph-mon[91579]: pgmap v16118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:26 smithi188 ceph-mon[92425]: pgmap v16118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:29 smithi188 ceph-mon[92425]: pgmap v16119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:29 smithi161 ceph-mon[91579]: pgmap v16119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:31 smithi188 ceph-mon[92425]: pgmap v16120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:31 smithi161 ceph-mon[91579]: pgmap v16120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:33 smithi188 ceph-mon[92425]: pgmap v16121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:33 smithi161 ceph-mon[91579]: pgmap v16121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:34 smithi161 ceph-mon[91579]: pgmap v16122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:34 smithi188 ceph-mon[92425]: pgmap v16122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:36 smithi161 ceph-mon[91579]: pgmap v16123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:36 smithi188 ceph-mon[92425]: pgmap v16123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:39 smithi188 ceph-mon[92425]: pgmap v16124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:39 smithi161 ceph-mon[91579]: pgmap v16124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:41 smithi188 ceph-mon[92425]: pgmap v16125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:41 smithi161 ceph-mon[91579]: pgmap v16125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:43 smithi188 ceph-mon[92425]: pgmap v16126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:43 smithi161 ceph-mon[91579]: pgmap v16126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:44 smithi161 ceph-mon[91579]: pgmap v16127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:44 smithi188 ceph-mon[92425]: pgmap v16127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:46 smithi161 ceph-mon[91579]: pgmap v16128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:46 smithi188 ceph-mon[92425]: pgmap v16128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:49 smithi188 ceph-mon[92425]: pgmap v16129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:50.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:49 smithi161 ceph-mon[91579]: pgmap v16129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:51 smithi188 ceph-mon[92425]: pgmap v16130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:51 smithi161 ceph-mon[91579]: pgmap v16130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:53.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:53 smithi188 ceph-mon[92425]: pgmap v16131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:53 smithi161 ceph-mon[91579]: pgmap v16131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:54 smithi161 ceph-mon[91579]: pgmap v16132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:54 smithi188 ceph-mon[92425]: pgmap v16132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:36:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:56 smithi161 ceph-mon[91579]: pgmap v16133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:56 smithi188 ceph-mon[92425]: pgmap v16133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:36:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:59 smithi188 ceph-mon[92425]: pgmap v16134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:36:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:36:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:36:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:36:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:37:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:59 smithi161 ceph-mon[91579]: pgmap v16134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:37:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:37:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:37:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:37:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:01 smithi188 ceph-mon[92425]: pgmap v16135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:01 smithi161 ceph-mon[91579]: pgmap v16135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:03 smithi188 ceph-mon[92425]: pgmap v16136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:03 smithi161 ceph-mon[91579]: pgmap v16136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:04 smithi161 ceph-mon[91579]: pgmap v16137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:04 smithi188 ceph-mon[92425]: pgmap v16137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:37:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:37:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:37:06.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:37:06.008+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:37:06.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:37:06.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:37:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:37:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:37:06.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:37:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:37:06.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:37:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:37:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:37:06.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:37:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:37:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:37:06.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:37:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:37:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:37:06.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:37:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:37:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:37:06.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:37:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:37:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:37:06.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:37:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:37:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:37:06.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:37:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:37:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:37:06.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:37:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:37:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:37:06.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:37:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:37:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:37:06.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:37:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:37:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:37:06.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:37:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:37:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:37:06.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:37:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:37:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:37:06.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:37:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:37:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=967] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:37:06.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=967] Local address: 172.21.15.161, port 59714 2022-07-03T19:37:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=967] Peer address: 172.21.15.101, port 22 2022-07-03T19:37:06.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=967] Connection lost 2022-07-03T19:37:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=967] Aborting connection 2022-07-03T19:37:06.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:37:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:37:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:37:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:37:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:37:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:37:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:37:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:37:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:37:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:37:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:37:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:37:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:37:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:37:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:37:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:37:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:37:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:37:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:37:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:37:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:37:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:37:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:37:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:37:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:37:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:37:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:37:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:37:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:37:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:37:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:37:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:37:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:37:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:37:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:37:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:37:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:37:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:37:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:37:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:37:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:37:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:37:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:37:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:37:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:37:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:37:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:37:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:37:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:37:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:37:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=967] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:37:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=967] Local address: 172.21.15.161, port 59714 2022-07-03T19:37:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=967] Peer address: 172.21.15.101, port 22 2022-07-03T19:37:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=967] Connection lost 2022-07-03T19:37:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=967] Aborting connection 2022-07-03T19:37:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:06 smithi161 ceph-mon[91579]: pgmap v16138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:37:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:37:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:37:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:37:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:37:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:37:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:37:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:37:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:37:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:37:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:37:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:37:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:37:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:37:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:37:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:37:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:37:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:37:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:37:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:37:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:37:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:37:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:37:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:37:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:37:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:37:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:37:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:37:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:37:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:37:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:37:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:37:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:37:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:37:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:37:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:37:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:37:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:37:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:37:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:37:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:37:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=967] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:37:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=967] Local address: 172.21.15.161, port 59714 2022-07-03T19:37:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=967] Peer address: 172.21.15.101, port 22 2022-07-03T19:37:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=967] Connection lost 2022-07-03T19:37:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=967] Aborting connection 2022-07-03T19:37:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:06 smithi188 ceph-mon[92425]: pgmap v16138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:09 smithi188 ceph-mon[92425]: pgmap v16139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:10.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:09 smithi161 ceph-mon[91579]: pgmap v16139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:11 smithi188 ceph-mon[92425]: pgmap v16140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:11 smithi161 ceph-mon[91579]: pgmap v16140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:13 smithi188 ceph-mon[92425]: pgmap v16141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:13 smithi161 ceph-mon[91579]: pgmap v16141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:14 smithi161 ceph-mon[91579]: pgmap v16142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:14 smithi188 ceph-mon[92425]: pgmap v16142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:16 smithi161 ceph-mon[91579]: pgmap v16143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:16 smithi188 ceph-mon[92425]: pgmap v16143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:19 smithi188 ceph-mon[92425]: pgmap v16144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:19 smithi161 ceph-mon[91579]: pgmap v16144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:21 smithi188 ceph-mon[92425]: pgmap v16145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:21 smithi161 ceph-mon[91579]: pgmap v16145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:23 smithi188 ceph-mon[92425]: pgmap v16146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:23 smithi161 ceph-mon[91579]: pgmap v16146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:24 smithi161 ceph-mon[91579]: pgmap v16147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:24 smithi188 ceph-mon[92425]: pgmap v16147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:26 smithi161 ceph-mon[91579]: pgmap v16148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:26 smithi188 ceph-mon[92425]: pgmap v16148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:29 smithi188 ceph-mon[92425]: pgmap v16149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:29 smithi161 ceph-mon[91579]: pgmap v16149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:31 smithi188 ceph-mon[92425]: pgmap v16150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:31 smithi161 ceph-mon[91579]: pgmap v16150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:33 smithi188 ceph-mon[92425]: pgmap v16151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:33 smithi161 ceph-mon[91579]: pgmap v16151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:34 smithi161 ceph-mon[91579]: pgmap v16152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:34 smithi188 ceph-mon[92425]: pgmap v16152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:36 smithi161 ceph-mon[91579]: pgmap v16153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:36 smithi188 ceph-mon[92425]: pgmap v16153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:39 smithi188 ceph-mon[92425]: pgmap v16154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:39 smithi161 ceph-mon[91579]: pgmap v16154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:41 smithi188 ceph-mon[92425]: pgmap v16155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:41 smithi161 ceph-mon[91579]: pgmap v16155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:43 smithi188 ceph-mon[92425]: pgmap v16156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:43 smithi161 ceph-mon[91579]: pgmap v16156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:44 smithi161 ceph-mon[91579]: pgmap v16157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:44 smithi188 ceph-mon[92425]: pgmap v16157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:47 smithi161 ceph-mon[91579]: pgmap v16158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:47 smithi188 ceph-mon[92425]: pgmap v16158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:49 smithi188 ceph-mon[92425]: pgmap v16159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:50.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:49 smithi161 ceph-mon[91579]: pgmap v16159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:51.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:51 smithi188 ceph-mon[92425]: pgmap v16160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:51 smithi161 ceph-mon[91579]: pgmap v16160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:53 smithi188 ceph-mon[92425]: pgmap v16161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:53 smithi161 ceph-mon[91579]: pgmap v16161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:54 smithi161 ceph-mon[91579]: pgmap v16162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:54 smithi188 ceph-mon[92425]: pgmap v16162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:37:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:57 smithi161 ceph-mon[91579]: pgmap v16163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:57 smithi188 ceph-mon[92425]: pgmap v16163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:37:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:59 smithi188 ceph-mon[92425]: pgmap v16164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:37:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:37:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:37:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:37:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:59 smithi161 ceph-mon[91579]: pgmap v16164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:01 smithi188 ceph-mon[92425]: pgmap v16165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:01 smithi161 ceph-mon[91579]: pgmap v16165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:03 smithi161 ceph-mon[91579]: pgmap v16166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:03 smithi188 ceph-mon[92425]: pgmap v16166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:04 smithi161 ceph-mon[91579]: pgmap v16167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:04 smithi188 ceph-mon[92425]: pgmap v16167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:06.505 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:38:06.502+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:38:06.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:38:06.506 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:38:06.507 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:38:06.507 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:38:06.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:38:06.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:38:06.508 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:38:06.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:38:06.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:38:06.509 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:38:06.510 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:38:06.510 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:38:06.510 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:38:06.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:38:06.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:38:06.511 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:38:06.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:38:06.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:38:06.512 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:38:06.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:38:06.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:38:06.513 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:38:06.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:38:06.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:38:06.514 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:38:06.515 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:38:06.515 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:38:06.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:38:06.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:38:06.516 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:38:06.517 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:38:06.517 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:38:06.518 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:38:06.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:38:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:38:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:38:06.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:38:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:38:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:38:06.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:38:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:38:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:38:06.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:38:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:38:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=969] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:38:06.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=969] Local address: 172.21.15.161, port 59718 2022-07-03T19:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=969] Peer address: 172.21.15.101, port 22 2022-07-03T19:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=969] Connection lost 2022-07-03T19:38:06.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=969] Aborting connection 2022-07-03T19:38:06.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:38:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:38:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:38:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:38:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:38:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:38:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:38:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:38:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:38:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:38:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:38:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:38:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:38:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:38:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:38:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:38:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:38:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:38:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:38:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:38:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:38:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:38:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:38:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:38:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:38:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:38:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:38:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:38:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:38:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:38:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:38:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:38:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:38:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:38:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:38:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:38:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:38:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:38:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:38:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:38:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:38:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:38:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:38:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:38:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:38:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:38:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:38:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:38:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:38:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:38:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:38:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=969] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:38:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=969] Local address: 172.21.15.161, port 59718 2022-07-03T19:38:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=969] Peer address: 172.21.15.101, port 22 2022-07-03T19:38:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=969] Connection lost 2022-07-03T19:38:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=969] Aborting connection 2022-07-03T19:38:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: pgmap v16168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:38:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:38:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:38:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:38:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:38:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:38:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:38:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:38:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:38:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:38:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:38:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:38:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:38:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:38:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:38:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:38:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:38:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:38:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:38:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:38:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:38:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:38:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:38:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:38:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:38:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:38:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:38:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:38:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:38:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:38:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:38:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:38:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:38:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:38:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:38:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:38:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:38:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:38:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:38:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:38:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:38:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:38:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:38:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:38:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:38:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:38:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:38:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:38:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:38:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=969] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:38:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=969] Local address: 172.21.15.161, port 59718 2022-07-03T19:38:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=969] Peer address: 172.21.15.101, port 22 2022-07-03T19:38:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=969] Connection lost 2022-07-03T19:38:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=969] Aborting connection 2022-07-03T19:38:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: pgmap v16168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:38:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:09 smithi188 ceph-mon[92425]: pgmap v16169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:10.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:09 smithi161 ceph-mon[91579]: pgmap v16169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:11 smithi161 ceph-mon[91579]: pgmap v16170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:11 smithi188 ceph-mon[92425]: pgmap v16170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:13 smithi161 ceph-mon[91579]: pgmap v16171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:13 smithi188 ceph-mon[92425]: pgmap v16171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:14 smithi161 ceph-mon[91579]: pgmap v16172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:14 smithi188 ceph-mon[92425]: pgmap v16172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:16.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:17 smithi161 ceph-mon[91579]: pgmap v16173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:17 smithi188 ceph-mon[92425]: pgmap v16173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:19 smithi161 ceph-mon[91579]: pgmap v16174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:19 smithi188 ceph-mon[92425]: pgmap v16174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:21 smithi161 ceph-mon[91579]: pgmap v16175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:21 smithi188 ceph-mon[92425]: pgmap v16175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:24.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:23 smithi161 ceph-mon[91579]: pgmap v16176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:23 smithi188 ceph-mon[92425]: pgmap v16176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:24 smithi161 ceph-mon[91579]: pgmap v16177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:25 smithi188 ceph-mon[92425]: pgmap v16177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:27 smithi161 ceph-mon[91579]: pgmap v16178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:27 smithi188 ceph-mon[92425]: pgmap v16178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:29 smithi161 ceph-mon[91579]: pgmap v16179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:29 smithi188 ceph-mon[92425]: pgmap v16179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:31 smithi161 ceph-mon[91579]: pgmap v16180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:31 smithi188 ceph-mon[92425]: pgmap v16180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:33 smithi161 ceph-mon[91579]: pgmap v16181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:33 smithi188 ceph-mon[92425]: pgmap v16181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:34 smithi161 ceph-mon[91579]: pgmap v16182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:35 smithi188 ceph-mon[92425]: pgmap v16182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:37 smithi161 ceph-mon[91579]: pgmap v16183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:37 smithi188 ceph-mon[92425]: pgmap v16183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:39 smithi161 ceph-mon[91579]: pgmap v16184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:39 smithi188 ceph-mon[92425]: pgmap v16184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:41 smithi161 ceph-mon[91579]: pgmap v16185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:41 smithi188 ceph-mon[92425]: pgmap v16185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:43 smithi161 ceph-mon[91579]: pgmap v16186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:43 smithi188 ceph-mon[92425]: pgmap v16186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:45 smithi161 ceph-mon[91579]: pgmap v16187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:45 smithi188 ceph-mon[92425]: pgmap v16187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:47 smithi161 ceph-mon[91579]: pgmap v16188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:47 smithi188 ceph-mon[92425]: pgmap v16188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:49 smithi161 ceph-mon[91579]: pgmap v16189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:49 smithi188 ceph-mon[92425]: pgmap v16189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:51 smithi161 ceph-mon[91579]: pgmap v16190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:51 smithi188 ceph-mon[92425]: pgmap v16190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:52.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:53 smithi161 ceph-mon[91579]: pgmap v16191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:53 smithi188 ceph-mon[92425]: pgmap v16191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:55 smithi161 ceph-mon[91579]: pgmap v16192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:55 smithi188 ceph-mon[92425]: pgmap v16192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:38:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:57 smithi161 ceph-mon[91579]: pgmap v16193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:57 smithi188 ceph-mon[92425]: pgmap v16193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:38:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:38:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:59 smithi161 ceph-mon[91579]: pgmap v16194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:59 smithi188 ceph-mon[92425]: pgmap v16194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:38:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:38:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:38:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:39:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:01 smithi161 ceph-mon[91579]: pgmap v16195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:01 smithi188 ceph-mon[92425]: pgmap v16195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:03 smithi161 ceph-mon[91579]: pgmap v16196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:03 smithi188 ceph-mon[92425]: pgmap v16196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:05 smithi161 ceph-mon[91579]: pgmap v16197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:05 smithi188 ceph-mon[92425]: pgmap v16197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:06.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-mon[91579]: pgmap v16198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:07 smithi188 ceph-mon[92425]: pgmap v16198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:07.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:39:07.430+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:39:07.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:39:07.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:39:07.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:39:07.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:39:07.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:39:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:39:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:39:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:39:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:39:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:39:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:39:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:39:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:39:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:39:07.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:39:07.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:39:07.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:39:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:39:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:39:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:39:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:39:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:39:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:39:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:39:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:39:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:39:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:39:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:39:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:39:07.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:39:07.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:39:07.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:39:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:39:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:39:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:39:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:39:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:39:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:39:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:39:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:39:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:39:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:39:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:39:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:39:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=971] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:39:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=971] Local address: 172.21.15.161, port 59764 2022-07-03T19:39:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=971] Peer address: 172.21.15.101, port 22 2022-07-03T19:39:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=971] Connection lost 2022-07-03T19:39:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=971] Aborting connection 2022-07-03T19:39:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:39:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:39:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:39:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:39:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:39:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:39:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:39:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:39:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:39:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:39:08.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:39:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:39:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:39:08.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:39:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:39:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:39:08.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:39:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:39:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:39:08.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:39:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:39:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:39:08.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:39:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:39:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:39:08.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:39:08.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:39:08.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:39:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:39:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:39:08.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:39:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:39:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:39:08.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:39:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:39:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:39:08.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:39:08.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:39:08.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:39:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:39:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:39:08.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:39:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:39:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:39:08.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:39:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:39:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:39:08.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:39:08.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:39:08.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:39:08.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=971] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:39:08.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=971] Local address: 172.21.15.161, port 59764 2022-07-03T19:39:08.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=971] Peer address: 172.21.15.101, port 22 2022-07-03T19:39:08.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=971] Connection lost 2022-07-03T19:39:08.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=971] Aborting connection 2022-07-03T19:39:08.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:39:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:39:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:39:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:39:08.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:39:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:39:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:39:08.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:39:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:39:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:39:08.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:39:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:39:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:39:08.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:39:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:39:08.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:39:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:39:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:39:08.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:39:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:39:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:39:08.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:39:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:39:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:39:08.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:39:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:39:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:39:08.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:39:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:39:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:39:08.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:39:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:39:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:39:08.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:39:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:39:08.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:39:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:39:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:39:08.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:39:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:39:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:39:08.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:39:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:39:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=971] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:39:08.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=971] Local address: 172.21.15.161, port 59764 2022-07-03T19:39:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=971] Peer address: 172.21.15.101, port 22 2022-07-03T19:39:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=971] Connection lost 2022-07-03T19:39:08.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=971] Aborting connection 2022-07-03T19:39:08.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:09 smithi161 ceph-mon[91579]: pgmap v16199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:09 smithi188 ceph-mon[92425]: pgmap v16199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:11 smithi161 ceph-mon[91579]: pgmap v16200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:11 smithi188 ceph-mon[92425]: pgmap v16200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:13 smithi161 ceph-mon[91579]: pgmap v16201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:13 smithi188 ceph-mon[92425]: pgmap v16201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:15 smithi161 ceph-mon[91579]: pgmap v16202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:15 smithi188 ceph-mon[92425]: pgmap v16202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:16.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:17 smithi161 ceph-mon[91579]: pgmap v16203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:17 smithi188 ceph-mon[92425]: pgmap v16203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:19 smithi161 ceph-mon[91579]: pgmap v16204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:19 smithi188 ceph-mon[92425]: pgmap v16204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:21 smithi161 ceph-mon[91579]: pgmap v16205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:21 smithi188 ceph-mon[92425]: pgmap v16205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:23 smithi161 ceph-mon[91579]: pgmap v16206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:23 smithi188 ceph-mon[92425]: pgmap v16206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:25 smithi161 ceph-mon[91579]: pgmap v16207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:25 smithi188 ceph-mon[92425]: pgmap v16207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:27 smithi161 ceph-mon[91579]: pgmap v16208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:27 smithi188 ceph-mon[92425]: pgmap v16208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:29 smithi161 ceph-mon[91579]: pgmap v16209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:29 smithi188 ceph-mon[92425]: pgmap v16209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:31 smithi161 ceph-mon[91579]: pgmap v16210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:31 smithi188 ceph-mon[92425]: pgmap v16210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:33 smithi161 ceph-mon[91579]: pgmap v16211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:33 smithi188 ceph-mon[92425]: pgmap v16211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:35 smithi161 ceph-mon[91579]: pgmap v16212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:35 smithi188 ceph-mon[92425]: pgmap v16212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:37 smithi161 ceph-mon[91579]: pgmap v16213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:37 smithi188 ceph-mon[92425]: pgmap v16213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:39 smithi161 ceph-mon[91579]: pgmap v16214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:39 smithi188 ceph-mon[92425]: pgmap v16214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:41 smithi161 ceph-mon[91579]: pgmap v16215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:41 smithi188 ceph-mon[92425]: pgmap v16215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:43 smithi161 ceph-mon[91579]: pgmap v16216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:43 smithi188 ceph-mon[92425]: pgmap v16216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:45 smithi161 ceph-mon[91579]: pgmap v16217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:45 smithi188 ceph-mon[92425]: pgmap v16217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:47 smithi161 ceph-mon[91579]: pgmap v16218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:47 smithi188 ceph-mon[92425]: pgmap v16218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:48.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:49 smithi161 ceph-mon[91579]: pgmap v16219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:49 smithi188 ceph-mon[92425]: pgmap v16219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:51 smithi161 ceph-mon[91579]: pgmap v16220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:51 smithi188 ceph-mon[92425]: pgmap v16220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:53 smithi161 ceph-mon[91579]: pgmap v16221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:53 smithi188 ceph-mon[92425]: pgmap v16221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:55 smithi161 ceph-mon[91579]: pgmap v16222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:55 smithi188 ceph-mon[92425]: pgmap v16222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:39:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:57 smithi161 ceph-mon[91579]: pgmap v16223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:57 smithi188 ceph-mon[92425]: pgmap v16223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:39:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:39:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:59 smithi161 ceph-mon[91579]: pgmap v16224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:39:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:39:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:39:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:39:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:59 smithi188 ceph-mon[92425]: pgmap v16224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:39:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:39:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:39:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:39:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:40:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:40:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:40:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:01 smithi161 ceph-mon[91579]: pgmap v16225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:01 smithi188 ceph-mon[92425]: pgmap v16225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:03 smithi161 ceph-mon[91579]: pgmap v16226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:03 smithi188 ceph-mon[92425]: pgmap v16226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:05 smithi161 ceph-mon[91579]: pgmap v16227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:05 smithi188 ceph-mon[92425]: pgmap v16227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-mon[91579]: pgmap v16228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:07 smithi188 ceph-mon[92425]: pgmap v16228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:40:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:40:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:40:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:40:07.934+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:40:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:40:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:40:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:40:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:40:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:40:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:40:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:40:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:40:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:40:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:40:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:40:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:40:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:40:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:40:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:40:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:40:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:40:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:40:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:40:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:40:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:40:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:40:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:40:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:40:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:40:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:40:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:40:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:40:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:40:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:40:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:40:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:40:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:40:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:40:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:40:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:40:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:40:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:40:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:40:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:40:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:40:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:40:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:40:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:40:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=973] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:40:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=973] Local address: 172.21.15.161, port 59794 2022-07-03T19:40:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=973] Peer address: 172.21.15.101, port 22 2022-07-03T19:40:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=973] Connection lost 2022-07-03T19:40:08.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=973] Aborting connection 2022-07-03T19:40:08.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:40:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:40:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:40:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:40:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:40:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:40:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:40:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:40:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:40:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:40:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:40:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:40:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:40:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:40:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:40:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:40:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:40:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:40:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:40:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:40:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:40:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:40:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:40:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:40:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:40:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:40:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:40:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:40:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:40:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:40:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:40:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:40:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:40:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:40:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:40:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:40:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:40:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:40:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:40:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:40:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:40:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:40:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:40:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:40:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:40:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:40:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:40:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:40:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=973] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:40:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=973] Local address: 172.21.15.161, port 59794 2022-07-03T19:40:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=973] Peer address: 172.21.15.101, port 22 2022-07-03T19:40:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=973] Connection lost 2022-07-03T19:40:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=973] Aborting connection 2022-07-03T19:40:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:09 smithi161 ceph-mon[91579]: pgmap v16229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:40:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:40:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:40:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:40:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:40:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:40:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:40:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:40:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:40:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:40:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:40:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:40:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:40:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:40:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:40:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:40:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:40:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:40:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:40:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:40:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:40:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:40:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:40:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:40:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:40:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:40:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:40:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:40:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:40:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:40:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:40:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:40:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:40:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:40:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:40:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:40:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:40:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:40:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:40:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:40:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:40:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:40:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:40:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:40:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:40:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:40:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=973] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:40:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=973] Local address: 172.21.15.161, port 59794 2022-07-03T19:40:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=973] Peer address: 172.21.15.101, port 22 2022-07-03T19:40:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=973] Connection lost 2022-07-03T19:40:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=973] Aborting connection 2022-07-03T19:40:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:09 smithi188 ceph-mon[92425]: pgmap v16229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:11 smithi161 ceph-mon[91579]: pgmap v16230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:11 smithi188 ceph-mon[92425]: pgmap v16230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:13 smithi161 ceph-mon[91579]: pgmap v16231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:13 smithi188 ceph-mon[92425]: pgmap v16231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:15 smithi161 ceph-mon[91579]: pgmap v16232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:15 smithi188 ceph-mon[92425]: pgmap v16232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:17 smithi161 ceph-mon[91579]: pgmap v16233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:17 smithi188 ceph-mon[92425]: pgmap v16233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:19 smithi161 ceph-mon[91579]: pgmap v16234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:19 smithi188 ceph-mon[92425]: pgmap v16234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:21 smithi161 ceph-mon[91579]: pgmap v16235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:21 smithi188 ceph-mon[92425]: pgmap v16235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:23 smithi161 ceph-mon[91579]: pgmap v16236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:23 smithi188 ceph-mon[92425]: pgmap v16236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:25 smithi161 ceph-mon[91579]: pgmap v16237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:25 smithi188 ceph-mon[92425]: pgmap v16237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:27 smithi161 ceph-mon[91579]: pgmap v16238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:27 smithi188 ceph-mon[92425]: pgmap v16238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:29 smithi188 ceph-mon[92425]: pgmap v16239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:29 smithi161 ceph-mon[91579]: pgmap v16239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:31 smithi161 ceph-mon[91579]: pgmap v16240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:31 smithi188 ceph-mon[92425]: pgmap v16240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:33 smithi161 ceph-mon[91579]: pgmap v16241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:33 smithi188 ceph-mon[92425]: pgmap v16241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:35 smithi161 ceph-mon[91579]: pgmap v16242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:35 smithi188 ceph-mon[92425]: pgmap v16242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:37 smithi161 ceph-mon[91579]: pgmap v16243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:37 smithi188 ceph-mon[92425]: pgmap v16243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:39 smithi188 ceph-mon[92425]: pgmap v16244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:39 smithi161 ceph-mon[91579]: pgmap v16244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:41 smithi161 ceph-mon[91579]: pgmap v16245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:41 smithi188 ceph-mon[92425]: pgmap v16245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:43 smithi161 ceph-mon[91579]: pgmap v16246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:43 smithi188 ceph-mon[92425]: pgmap v16246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:45 smithi161 ceph-mon[91579]: pgmap v16247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:45 smithi188 ceph-mon[92425]: pgmap v16247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:47 smithi188 ceph-mon[92425]: pgmap v16248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:47 smithi161 ceph-mon[91579]: pgmap v16248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:49 smithi188 ceph-mon[92425]: pgmap v16249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:49 smithi161 ceph-mon[91579]: pgmap v16249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:51 smithi161 ceph-mon[91579]: pgmap v16250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:51 smithi188 ceph-mon[92425]: pgmap v16250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:53 smithi161 ceph-mon[91579]: pgmap v16251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:53 smithi188 ceph-mon[92425]: pgmap v16251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:55 smithi161 ceph-mon[91579]: pgmap v16252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:55 smithi188 ceph-mon[92425]: pgmap v16252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:40:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:57 smithi188 ceph-mon[92425]: pgmap v16253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:57 smithi161 ceph-mon[91579]: pgmap v16253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:40:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:40:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:59 smithi188 ceph-mon[92425]: pgmap v16254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:40:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:40:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:40:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:40:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:59 smithi161 ceph-mon[91579]: pgmap v16254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:40:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:40:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:40:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:40:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:41:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:01 smithi161 ceph-mon[91579]: pgmap v16255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:01 smithi188 ceph-mon[92425]: pgmap v16255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:03 smithi161 ceph-mon[91579]: pgmap v16256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:03 smithi188 ceph-mon[92425]: pgmap v16256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:05 smithi161 ceph-mon[91579]: pgmap v16257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:05 smithi188 ceph-mon[92425]: pgmap v16257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:06.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:07 smithi188 ceph-mon[92425]: pgmap v16258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:07 smithi161 ceph-mon[91579]: pgmap v16258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:41:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:08.442 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:08.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:41:08.441+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:41:08.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:41:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:41:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:41:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:41:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:41:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:41:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:41:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:41:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:41:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:41:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:41:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:41:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:41:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:41:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:41:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:41:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:41:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:41:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:41:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:41:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:41:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:41:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:41:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:41:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:41:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:41:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:41:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:41:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:41:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:41:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:41:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:41:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:41:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:41:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:41:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:41:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:41:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:41:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:41:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:41:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:41:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:41:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:41:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:41:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=975] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:41:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=975] Local address: 172.21.15.161, port 59822 2022-07-03T19:41:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=975] Peer address: 172.21.15.101, port 22 2022-07-03T19:41:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=975] Connection lost 2022-07-03T19:41:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=975] Aborting connection 2022-07-03T19:41:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:41:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:41:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:41:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:41:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:41:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:41:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:41:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:41:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:41:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:41:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:41:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:41:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:41:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:41:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:41:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:41:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:41:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:41:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:41:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:41:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:41:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:41:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:41:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:41:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:41:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:41:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:41:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:41:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:41:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:41:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:41:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:41:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:41:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:41:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:41:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:41:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:41:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:41:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:41:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:41:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:41:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:41:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:41:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:41:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:41:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:41:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:41:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:41:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:41:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:41:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:41:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=975] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:41:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=975] Local address: 172.21.15.161, port 59822 2022-07-03T19:41:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=975] Peer address: 172.21.15.101, port 22 2022-07-03T19:41:09.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=975] Connection lost 2022-07-03T19:41:09.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=975] Aborting connection 2022-07-03T19:41:09.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:09 smithi188 ceph-mon[92425]: pgmap v16259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:41:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:41:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:41:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:41:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:41:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:41:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:41:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:41:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:41:09.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:41:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:41:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:41:09.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:41:09.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:41:09.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:41:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:41:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:41:09.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:41:09.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:41:09.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:41:09.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:41:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:41:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:41:09.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:41:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:41:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:41:09.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:41:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:41:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:41:09.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:41:09.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:41:09.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:41:09.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:41:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:41:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:41:09.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:41:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:41:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:41:09.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:41:09.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:41:09.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:41:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:41:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:41:09.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:41:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:41:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:41:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:41:09.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:41:09.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:41:09.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=975] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:41:09.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=975] Local address: 172.21.15.161, port 59822 2022-07-03T19:41:09.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=975] Peer address: 172.21.15.101, port 22 2022-07-03T19:41:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=975] Connection lost 2022-07-03T19:41:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=975] Aborting connection 2022-07-03T19:41:09.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:09 smithi161 ceph-mon[91579]: pgmap v16259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: pgmap v16260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:11 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: pgmap v16260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:41:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:12.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:11 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:41:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:12 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 100.2M 2022-07-03T19:41:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:12 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 105091276: error parsing value: Value '105091276' is below minimum 939524096 2022-07-03T19:41:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:12 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 100.2M 2022-07-03T19:41:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:12 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 105091276: error parsing value: Value '105091276' is below minimum 939524096 2022-07-03T19:41:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:13 smithi161 ceph-mon[91579]: pgmap v16261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:13 smithi188 ceph-mon[92425]: pgmap v16261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:15 smithi161 ceph-mon[91579]: pgmap v16262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:15 smithi188 ceph-mon[92425]: pgmap v16262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:16.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:17 smithi188 ceph-mon[92425]: pgmap v16263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:17 smithi161 ceph-mon[91579]: pgmap v16263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:19 smithi188 ceph-mon[92425]: pgmap v16264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:19 smithi161 ceph-mon[91579]: pgmap v16264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:21 smithi161 ceph-mon[91579]: pgmap v16265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:21 smithi188 ceph-mon[92425]: pgmap v16265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:23 smithi161 ceph-mon[91579]: pgmap v16266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:23 smithi188 ceph-mon[92425]: pgmap v16266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:25 smithi161 ceph-mon[91579]: pgmap v16267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:25 smithi188 ceph-mon[92425]: pgmap v16267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:27 smithi188 ceph-mon[92425]: pgmap v16268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:27 smithi161 ceph-mon[91579]: pgmap v16268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:29 smithi188 ceph-mon[92425]: pgmap v16269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:29 smithi161 ceph-mon[91579]: pgmap v16269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:31 smithi161 ceph-mon[91579]: pgmap v16270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:31 smithi188 ceph-mon[92425]: pgmap v16270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:33 smithi161 ceph-mon[91579]: pgmap v16271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:33 smithi188 ceph-mon[92425]: pgmap v16271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:35 smithi161 ceph-mon[91579]: pgmap v16272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:35 smithi188 ceph-mon[92425]: pgmap v16272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:37 smithi188 ceph-mon[92425]: pgmap v16273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:37 smithi161 ceph-mon[91579]: pgmap v16273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:39 smithi188 ceph-mon[92425]: pgmap v16274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:39 smithi161 ceph-mon[91579]: pgmap v16274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:41 smithi161 ceph-mon[91579]: pgmap v16275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:41 smithi188 ceph-mon[92425]: pgmap v16275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:43 smithi161 ceph-mon[91579]: pgmap v16276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:43 smithi188 ceph-mon[92425]: pgmap v16276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:45 smithi161 ceph-mon[91579]: pgmap v16277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:45 smithi188 ceph-mon[92425]: pgmap v16277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:47 smithi188 ceph-mon[92425]: pgmap v16278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:47 smithi161 ceph-mon[91579]: pgmap v16278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:49 smithi161 ceph-mon[91579]: pgmap v16279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:49 smithi188 ceph-mon[92425]: pgmap v16279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:51 smithi161 ceph-mon[91579]: pgmap v16280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:51 smithi188 ceph-mon[92425]: pgmap v16280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:53 smithi161 ceph-mon[91579]: pgmap v16281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:53 smithi188 ceph-mon[92425]: pgmap v16281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:55 smithi161 ceph-mon[91579]: pgmap v16282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:55 smithi188 ceph-mon[92425]: pgmap v16282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:41:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:57 smithi161 ceph-mon[91579]: pgmap v16283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:57 smithi188 ceph-mon[92425]: pgmap v16283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:41:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:41:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:59 smithi161 ceph-mon[91579]: pgmap v16284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:41:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:41:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:41:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:41:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:59 smithi188 ceph-mon[92425]: pgmap v16284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:41:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:41:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:41:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:41:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:42:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:01 smithi161 ceph-mon[91579]: pgmap v16285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:01 smithi188 ceph-mon[92425]: pgmap v16285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:03 smithi161 ceph-mon[91579]: pgmap v16286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:03 smithi188 ceph-mon[92425]: pgmap v16286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:05 smithi161 ceph-mon[91579]: pgmap v16287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:05 smithi188 ceph-mon[92425]: pgmap v16287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:07 smithi161 ceph-mon[91579]: pgmap v16288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:07 smithi188 ceph-mon[92425]: pgmap v16288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:09 smithi161 ceph-mon[91579]: pgmap v16289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:09 smithi188 ceph-mon[92425]: pgmap v16289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:11.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-mon[91579]: pgmap v16290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:42:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:42:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:42:11.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:42:11.590+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:42:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:42:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:42:11.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:42:11.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:42:11.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:42:11.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:42:11.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:42:11.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:42:11.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:42:11.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:42:11.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:42:11.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:42:11.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:42:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:42:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:42:11.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:42:11.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:42:11.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:42:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:42:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:42:11.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:42:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:42:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:42:11.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:42:11.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:42:11.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:42:11.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:42:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:42:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:42:11.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:42:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:42:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:42:11.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:42:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:42:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:42:11.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:42:11.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:42:11.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:42:11.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:42:11.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:42:11.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:42:11.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:42:11.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:42:11.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:42:11.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=977] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:42:11.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=977] Local address: 172.21.15.161, port 59850 2022-07-03T19:42:11.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=977] Peer address: 172.21.15.101, port 22 2022-07-03T19:42:11.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=977] Connection lost 2022-07-03T19:42:11.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=977] Aborting connection 2022-07-03T19:42:11.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:42:11 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:42:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:11 smithi188 ceph-mon[92425]: pgmap v16290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:42:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:42:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:42:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:12 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:42:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:12 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:42:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:42:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:42:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:42:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:42:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:42:13.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:42:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:42:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:42:13.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:42:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:42:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:42:13.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:42:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:42:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:42:13.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:42:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:42:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:42:13.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:42:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:42:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:42:13.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:42:13.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:42:13.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:42:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:42:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:42:13.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:42:13.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:42:13.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:42:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:42:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:42:13.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:42:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:42:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:42:13.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:42:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:42:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:42:13.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:42:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:42:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:42:13.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:42:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:42:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:42:13.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=977] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:42:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=977] Local address: 172.21.15.161, port 59850 2022-07-03T19:42:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=977] Peer address: 172.21.15.101, port 22 2022-07-03T19:42:13.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=977] Connection lost 2022-07-03T19:42:13.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=977] Aborting connection 2022-07-03T19:42:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:12 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:42:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:12 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:42:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:42:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:42:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:42:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:42:13.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:42:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:42:13.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:42:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:42:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:42:13.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:42:13.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:42:13.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:42:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:42:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:42:13.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:42:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:42:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:42:13.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:42:13.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:42:13.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:42:13.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:42:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:42:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:42:13.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:42:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:42:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:42:13.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:42:13.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:42:13.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:42:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:42:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:42:13.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:42:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:42:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:42:13.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:42:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:42:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:42:13.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:42:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:42:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:42:13.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:42:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:42:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=977] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:42:13.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=977] Local address: 172.21.15.161, port 59850 2022-07-03T19:42:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=977] Peer address: 172.21.15.101, port 22 2022-07-03T19:42:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=977] Connection lost 2022-07-03T19:42:13.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=977] Aborting connection 2022-07-03T19:42:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:13 smithi161 ceph-mon[91579]: pgmap v16291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:13 smithi188 ceph-mon[92425]: pgmap v16291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:15 smithi161 ceph-mon[91579]: pgmap v16292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:15 smithi188 ceph-mon[92425]: pgmap v16292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:17 smithi161 ceph-mon[91579]: pgmap v16293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:17 smithi188 ceph-mon[92425]: pgmap v16293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:19 smithi161 ceph-mon[91579]: pgmap v16294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:19 smithi188 ceph-mon[92425]: pgmap v16294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:21 smithi161 ceph-mon[91579]: pgmap v16295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:21 smithi188 ceph-mon[92425]: pgmap v16295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:23 smithi161 ceph-mon[91579]: pgmap v16296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:23 smithi188 ceph-mon[92425]: pgmap v16296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:25 smithi161 ceph-mon[91579]: pgmap v16297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:25 smithi188 ceph-mon[92425]: pgmap v16297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:27 smithi161 ceph-mon[91579]: pgmap v16298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:27 smithi188 ceph-mon[92425]: pgmap v16298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:29 smithi161 ceph-mon[91579]: pgmap v16299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:29 smithi188 ceph-mon[92425]: pgmap v16299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:32.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:31 smithi161 ceph-mon[91579]: pgmap v16300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:32.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:31 smithi188 ceph-mon[92425]: pgmap v16300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:33 smithi161 ceph-mon[91579]: pgmap v16301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:33 smithi188 ceph-mon[92425]: pgmap v16301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:35 smithi161 ceph-mon[91579]: pgmap v16302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:35 smithi188 ceph-mon[92425]: pgmap v16302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:37 smithi161 ceph-mon[91579]: pgmap v16303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:37 smithi188 ceph-mon[92425]: pgmap v16303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:39 smithi161 ceph-mon[91579]: pgmap v16304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:39 smithi188 ceph-mon[92425]: pgmap v16304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:41 smithi161 ceph-mon[91579]: pgmap v16305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:41 smithi188 ceph-mon[92425]: pgmap v16305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:43 smithi161 ceph-mon[91579]: pgmap v16306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:43 smithi188 ceph-mon[92425]: pgmap v16306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:45 smithi161 ceph-mon[91579]: pgmap v16307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:45 smithi188 ceph-mon[92425]: pgmap v16307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:47 smithi161 ceph-mon[91579]: pgmap v16308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:47 smithi188 ceph-mon[92425]: pgmap v16308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:49 smithi161 ceph-mon[91579]: pgmap v16309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:49 smithi188 ceph-mon[92425]: pgmap v16309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:51 smithi161 ceph-mon[91579]: pgmap v16310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:51 smithi188 ceph-mon[92425]: pgmap v16310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:53 smithi161 ceph-mon[91579]: pgmap v16311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:53 smithi188 ceph-mon[92425]: pgmap v16311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:55 smithi161 ceph-mon[91579]: pgmap v16312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:55 smithi188 ceph-mon[92425]: pgmap v16312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:42:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:57 smithi161 ceph-mon[91579]: pgmap v16313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:57 smithi188 ceph-mon[92425]: pgmap v16313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:42:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:42:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:59 smithi161 ceph-mon[91579]: pgmap v16314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:42:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:42:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:42:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:42:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:59 smithi188 ceph-mon[92425]: pgmap v16314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:42:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:42:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:42:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:42:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:43:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:01 smithi161 ceph-mon[91579]: pgmap v16315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:01 smithi188 ceph-mon[92425]: pgmap v16315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:03 smithi161 ceph-mon[91579]: pgmap v16316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:03 smithi188 ceph-mon[92425]: pgmap v16316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:05 smithi161 ceph-mon[91579]: pgmap v16317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:05 smithi188 ceph-mon[92425]: pgmap v16317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:07 smithi161 ceph-mon[91579]: pgmap v16318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:07 smithi188 ceph-mon[92425]: pgmap v16318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:09 smithi161 ceph-mon[91579]: pgmap v16319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:09 smithi188 ceph-mon[92425]: pgmap v16319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:12.093 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:11 smithi161 ceph-mon[91579]: pgmap v16320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:12.093 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:12.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:11 smithi188 ceph-mon[92425]: pgmap v16320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:12.103 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:12.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:43:12.093+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:43:12.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:43:12.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:43:12.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:43:12.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:43:12.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:43:12.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:43:12.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:43:12.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:43:12.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:43:12.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:43:12.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:43:12.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:43:12.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:43:12.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:43:12.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:43:12.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:43:12.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:43:12.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:43:12.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:43:12.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:43:12.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:43:12.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:43:12.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:43:12.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:43:12.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:43:12.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:43:12.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:43:12.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:43:12.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:43:12.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:43:12.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:43:12.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:43:12.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:43:12.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:43:12.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:43:12.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:43:12.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:43:12.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:43:12.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:43:12.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:43:12.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:43:12.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:43:12.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:43:12.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:43:12.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=979] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:43:12.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=979] Local address: 172.21.15.161, port 59878 2022-07-03T19:43:12.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=979] Peer address: 172.21.15.101, port 22 2022-07-03T19:43:12.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=979] Connection lost 2022-07-03T19:43:12.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=979] Aborting connection 2022-07-03T19:43:12.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:43:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:43:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:43:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:43:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:43:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:43:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:43:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:13 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:43:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:13 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:43:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:43:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:43:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:43:14.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:43:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:43:14.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:43:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:43:14.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:43:14.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:43:14.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:43:14.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:43:14.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:43:14.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:43:14.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:43:14.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:43:14.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:43:14.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:43:14.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:43:14.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:43:14.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:43:14.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:43:14.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:43:14.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:43:14.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:43:14.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:43:14.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:43:14.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:43:14.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:43:14.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:43:14.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:43:14.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:43:14.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:43:14.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:43:14.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:43:14.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:43:14.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:43:14.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:43:14.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:43:14.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:43:14.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:43:14.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:43:14.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:43:14.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:43:14.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:43:14.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=979] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:43:14.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=979] Local address: 172.21.15.161, port 59878 2022-07-03T19:43:14.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=979] Peer address: 172.21.15.101, port 22 2022-07-03T19:43:14.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=979] Connection lost 2022-07-03T19:43:14.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=979] Aborting connection 2022-07-03T19:43:14.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:13 smithi161 ceph-mon[91579]: pgmap v16321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:14.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:13 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:43:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:13 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:43:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:43:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:43:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:43:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:43:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:43:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:43:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:43:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:43:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:43:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:43:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:43:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:43:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:43:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:43:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:43:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:43:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:43:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:43:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:43:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:43:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:43:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:43:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:43:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:43:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:43:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:43:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:43:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:43:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:43:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:43:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:43:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:43:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:43:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:43:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:43:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:43:14.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:43:14.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:43:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:43:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:43:14.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:43:14.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:43:14.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:43:14.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:43:14.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=979] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:43:14.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=979] Local address: 172.21.15.161, port 59878 2022-07-03T19:43:14.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=979] Peer address: 172.21.15.101, port 22 2022-07-03T19:43:14.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=979] Connection lost 2022-07-03T19:43:14.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=979] Aborting connection 2022-07-03T19:43:14.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:13 smithi188 ceph-mon[92425]: pgmap v16321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:14.185 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:15 smithi161 ceph-mon[91579]: pgmap v16322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:15 smithi188 ceph-mon[92425]: pgmap v16322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:17 smithi161 ceph-mon[91579]: pgmap v16323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:17 smithi188 ceph-mon[92425]: pgmap v16323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:19 smithi161 ceph-mon[91579]: pgmap v16324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:19 smithi188 ceph-mon[92425]: pgmap v16324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:21 smithi161 ceph-mon[91579]: pgmap v16325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:21 smithi188 ceph-mon[92425]: pgmap v16325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:23 smithi161 ceph-mon[91579]: pgmap v16326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:23 smithi188 ceph-mon[92425]: pgmap v16326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:25 smithi161 ceph-mon[91579]: pgmap v16327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:25 smithi188 ceph-mon[92425]: pgmap v16327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:27 smithi161 ceph-mon[91579]: pgmap v16328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:27 smithi188 ceph-mon[92425]: pgmap v16328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:29 smithi161 ceph-mon[91579]: pgmap v16329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:29 smithi188 ceph-mon[92425]: pgmap v16329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:31 smithi161 ceph-mon[91579]: pgmap v16330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:31 smithi188 ceph-mon[92425]: pgmap v16330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:33 smithi161 ceph-mon[91579]: pgmap v16331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:33 smithi188 ceph-mon[92425]: pgmap v16331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:35 smithi161 ceph-mon[91579]: pgmap v16332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:35 smithi188 ceph-mon[92425]: pgmap v16332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:37 smithi161 ceph-mon[91579]: pgmap v16333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:37 smithi188 ceph-mon[92425]: pgmap v16333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:39 smithi161 ceph-mon[91579]: pgmap v16334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:39 smithi188 ceph-mon[92425]: pgmap v16334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:41 smithi161 ceph-mon[91579]: pgmap v16335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:41 smithi188 ceph-mon[92425]: pgmap v16335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:43 smithi161 ceph-mon[91579]: pgmap v16336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:43 smithi188 ceph-mon[92425]: pgmap v16336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:45 smithi161 ceph-mon[91579]: pgmap v16337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:45 smithi188 ceph-mon[92425]: pgmap v16337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:47 smithi161 ceph-mon[91579]: pgmap v16338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:47 smithi188 ceph-mon[92425]: pgmap v16338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:49 smithi161 ceph-mon[91579]: pgmap v16339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:49 smithi188 ceph-mon[92425]: pgmap v16339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:51 smithi161 ceph-mon[91579]: pgmap v16340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:51 smithi188 ceph-mon[92425]: pgmap v16340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:53 smithi161 ceph-mon[91579]: pgmap v16341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:53 smithi188 ceph-mon[92425]: pgmap v16341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:55 smithi161 ceph-mon[91579]: pgmap v16342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:55 smithi188 ceph-mon[92425]: pgmap v16342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:43:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:57 smithi161 ceph-mon[91579]: pgmap v16343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:57 smithi188 ceph-mon[92425]: pgmap v16343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:43:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:43:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:59 smithi161 ceph-mon[91579]: pgmap v16344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:43:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:43:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:43:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:43:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:59 smithi188 ceph-mon[92425]: pgmap v16344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:43:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:43:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:43:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:43:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:44:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:01 smithi161 ceph-mon[91579]: pgmap v16345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:01 smithi188 ceph-mon[92425]: pgmap v16345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:03 smithi161 ceph-mon[91579]: pgmap v16346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:03 smithi188 ceph-mon[92425]: pgmap v16346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:05 smithi161 ceph-mon[91579]: pgmap v16347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:05 smithi188 ceph-mon[92425]: pgmap v16347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:07 smithi161 ceph-mon[91579]: pgmap v16348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:07 smithi188 ceph-mon[92425]: pgmap v16348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:09 smithi161 ceph-mon[91579]: pgmap v16349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:09 smithi188 ceph-mon[92425]: pgmap v16349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:11 smithi161 ceph-mon[91579]: pgmap v16350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:11 smithi188 ceph-mon[92425]: pgmap v16350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:12.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:44:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:44:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:44:12.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:44:12.594+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:44:12.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:44:12.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:44:12.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:44:12.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:44:12.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:44:12.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:44:12.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:44:12.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:44:12.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:44:12.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:44:12.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:44:12.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:44:12.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:44:12.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:44:12.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:44:12.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:44:12.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:44:12.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:44:12.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:44:12.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:44:12.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:44:12.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:44:12.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:44:12.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:44:12.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:44:12.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:44:12.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:44:12.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:44:12.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:44:12.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:44:12.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:44:12.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:44:12.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:44:12.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:44:12.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:44:12.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:44:12.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:44:12.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:44:12.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:44:12.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:44:12.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:44:12.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:44:12.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:44:12.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:44:12.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=981] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:44:12.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=981] Local address: 172.21.15.161, port 59906 2022-07-03T19:44:12.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=981] Peer address: 172.21.15.101, port 22 2022-07-03T19:44:12.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=981] Connection lost 2022-07-03T19:44:12.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=981] Aborting connection 2022-07-03T19:44:12.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:44:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:44:12.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:44:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:44:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:44:14.063 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:13 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:44:14.064 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:13 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:44:14.064 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:44:14.064 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:44:14.065 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:44:14.065 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:44:14.065 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:44:14.066 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:44:14.066 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:44:14.067 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:44:14.067 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:44:14.067 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:44:14.068 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:44:14.068 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:44:14.068 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:44:14.069 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:44:14.069 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:44:14.069 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:44:14.070 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:44:14.070 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:44:14.070 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:44:14.071 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:44:14.071 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:44:14.071 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:44:14.072 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:44:14.072 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:44:14.072 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:44:14.073 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:44:14.073 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:44:14.073 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:44:14.073 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:44:14.074 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:44:14.074 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:44:14.074 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:44:14.075 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:44:14.075 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:44:14.075 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:44:14.076 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:44:14.076 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:44:14.076 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:44:14.077 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:44:14.077 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:44:14.077 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:44:14.078 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:44:14.078 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:44:14.078 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:44:14.079 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=981] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:44:14.079 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=981] Local address: 172.21.15.161, port 59906 2022-07-03T19:44:14.079 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=981] Peer address: 172.21.15.101, port 22 2022-07-03T19:44:14.080 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=981] Connection lost 2022-07-03T19:44:14.080 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=981] Aborting connection 2022-07-03T19:44:14.080 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:13 smithi161 ceph-mon[91579]: pgmap v16351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:14.081 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:13 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:13 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:44:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:44:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:44:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:44:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:44:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:44:14.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:44:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:44:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:44:14.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:44:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:44:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:44:14.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:44:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:44:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:44:14.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:44:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:44:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:44:14.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:44:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:44:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:44:14.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:44:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:44:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:44:14.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:44:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:44:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:44:14.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:44:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:44:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:44:14.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:44:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:44:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:44:14.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:44:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:44:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:44:14.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:44:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:44:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:44:14.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:44:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:44:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:44:14.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:44:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:44:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=981] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:44:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=981] Local address: 172.21.15.161, port 59906 2022-07-03T19:44:14.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=981] Peer address: 172.21.15.101, port 22 2022-07-03T19:44:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=981] Connection lost 2022-07-03T19:44:14.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=981] Aborting connection 2022-07-03T19:44:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:13 smithi188 ceph-mon[92425]: pgmap v16351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:14.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:15 smithi161 ceph-mon[91579]: pgmap v16352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:15 smithi188 ceph-mon[92425]: pgmap v16352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:16 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T19:44:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:44:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:44:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:16 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T19:44:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:44:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:44:17.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:17 smithi161 ceph-mon[91579]: pgmap v16353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:17 smithi188 ceph-mon[92425]: pgmap v16353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:19 smithi161 ceph-mon[91579]: pgmap v16354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:19 smithi188 ceph-mon[92425]: pgmap v16354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:21 smithi161 ceph-mon[91579]: pgmap v16355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:21 smithi188 ceph-mon[92425]: pgmap v16355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:23 smithi161 ceph-mon[91579]: pgmap v16356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:23 smithi188 ceph-mon[92425]: pgmap v16356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:25 smithi161 ceph-mon[91579]: pgmap v16357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:25 smithi188 ceph-mon[92425]: pgmap v16357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:27 smithi161 ceph-mon[91579]: pgmap v16358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:27 smithi188 ceph-mon[92425]: pgmap v16358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:29 smithi161 ceph-mon[91579]: pgmap v16359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:29 smithi188 ceph-mon[92425]: pgmap v16359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:31 smithi161 ceph-mon[91579]: pgmap v16360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:31 smithi188 ceph-mon[92425]: pgmap v16360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:33 smithi161 ceph-mon[91579]: pgmap v16361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:33 smithi188 ceph-mon[92425]: pgmap v16361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:35 smithi161 ceph-mon[91579]: pgmap v16362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:35 smithi188 ceph-mon[92425]: pgmap v16362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:37 smithi161 ceph-mon[91579]: pgmap v16363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:37 smithi188 ceph-mon[92425]: pgmap v16363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:39 smithi161 ceph-mon[91579]: pgmap v16364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:39 smithi188 ceph-mon[92425]: pgmap v16364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:41 smithi161 ceph-mon[91579]: pgmap v16365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:41 smithi188 ceph-mon[92425]: pgmap v16365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:43 smithi161 ceph-mon[91579]: pgmap v16366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:43 smithi188 ceph-mon[92425]: pgmap v16366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:45 smithi161 ceph-mon[91579]: pgmap v16367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:45 smithi188 ceph-mon[92425]: pgmap v16367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:47 smithi161 ceph-mon[91579]: pgmap v16368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:47 smithi188 ceph-mon[92425]: pgmap v16368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:49 smithi161 ceph-mon[91579]: pgmap v16369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:49 smithi188 ceph-mon[92425]: pgmap v16369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:51 smithi161 ceph-mon[91579]: pgmap v16370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:51 smithi188 ceph-mon[92425]: pgmap v16370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:53 smithi161 ceph-mon[91579]: pgmap v16371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:53 smithi188 ceph-mon[92425]: pgmap v16371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:55 smithi161 ceph-mon[91579]: pgmap v16372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:55 smithi188 ceph-mon[92425]: pgmap v16372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:44:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:57 smithi161 ceph-mon[91579]: pgmap v16373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:57 smithi188 ceph-mon[92425]: pgmap v16373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:44:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:44:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:59 smithi161 ceph-mon[91579]: pgmap v16374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:44:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:44:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:44:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:44:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:44:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:59 smithi188 ceph-mon[92425]: pgmap v16374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:44:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:44:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:44:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:44:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:44:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:45:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:01 smithi161 ceph-mon[91579]: pgmap v16375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:01 smithi188 ceph-mon[92425]: pgmap v16375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:03 smithi161 ceph-mon[91579]: pgmap v16376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:03 smithi188 ceph-mon[92425]: pgmap v16376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:05 smithi161 ceph-mon[91579]: pgmap v16377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:05 smithi188 ceph-mon[92425]: pgmap v16377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:07 smithi161 ceph-mon[91579]: pgmap v16378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:07 smithi188 ceph-mon[92425]: pgmap v16378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:09 smithi161 ceph-mon[91579]: pgmap v16379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:09 smithi188 ceph-mon[92425]: pgmap v16379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:11 smithi161 ceph-mon[91579]: pgmap v16380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:11 smithi188 ceph-mon[92425]: pgmap v16380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:13 smithi161 ceph-mon[91579]: pgmap v16381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:13 smithi188 ceph-mon[92425]: pgmap v16381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-mon[91579]: pgmap v16382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:15 smithi188 ceph-mon[92425]: pgmap v16382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:16.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:45:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:45:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:45:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:45:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:45:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:45:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:45:16.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:45:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:45:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:45:16.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:45:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:45:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:45:16.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:45:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:45:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:45:16.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:45:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:45:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:45:16.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:45:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:45:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:45:16.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:45:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:45:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:45:16.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:45:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:45:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:45:16.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:45:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:45:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:45:16.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:45:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:45:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:45:16.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:45:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:45:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:45:16.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:45:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:45:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:45:16.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:45:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:45:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:45:16.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:45:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:45:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:45:16.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:45:16.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:45:16.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:45:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=983] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:45:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=983] Local address: 172.21.15.161, port 59936 2022-07-03T19:45:16.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=983] Peer address: 172.21.15.101, port 22 2022-07-03T19:45:16.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=983] Connection lost 2022-07-03T19:45:16.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=983] Aborting connection 2022-07-03T19:45:16.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:45:15.863+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:45:16.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:45:16.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:45:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:45:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:45:16.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:45:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:45:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:45:16.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:45:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:45:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:45:16.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:45:16.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:45:16.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:45:16.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:45:16.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:45:16.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:45:16.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:45:16.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:45:16.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:45:16.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:45:16.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:45:16.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:45:16.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:45:16.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:45:16.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:45:16.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:45:16.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:45:16.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:45:16.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:45:16.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:45:16.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:45:16.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:45:16.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:45:16.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:45:16.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:45:16.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:45:16.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:45:16.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:45:16.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:45:16.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:45:16.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:45:16.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:45:16.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:45:16.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:45:16.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=983] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:45:16.162 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=983] Local address: 172.21.15.161, port 59936 2022-07-03T19:45:16.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=983] Peer address: 172.21.15.101, port 22 2022-07-03T19:45:16.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=983] Connection lost 2022-07-03T19:45:16.163 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=983] Aborting connection 2022-07-03T19:45:16.164 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:45:15 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:45:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:45:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:45:16.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:45:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:45:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:45:16.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:45:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:45:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:45:16.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:45:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:45:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:45:16.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:45:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:45:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:45:16.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:45:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:45:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:45:16.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:45:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:45:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:45:16.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:45:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:45:16.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:45:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:45:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:45:16.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:45:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:45:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:45:16.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:45:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:45:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:45:16.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:45:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:45:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:45:16.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:45:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:45:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:45:16.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:45:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:45:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:45:16.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:45:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:45:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:45:16.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:45:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:45:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:45:16.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:45:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:45:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:45:16.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=983] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:45:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=983] Local address: 172.21.15.161, port 59936 2022-07-03T19:45:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=983] Peer address: 172.21.15.101, port 22 2022-07-03T19:45:16.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=983] Connection lost 2022-07-03T19:45:16.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=983] Aborting connection 2022-07-03T19:45:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:17 smithi161 ceph-mon[91579]: pgmap v16383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:17 smithi188 ceph-mon[92425]: pgmap v16383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:19 smithi161 ceph-mon[91579]: pgmap v16384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:19 smithi188 ceph-mon[92425]: pgmap v16384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:21 smithi161 ceph-mon[91579]: pgmap v16385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:21 smithi188 ceph-mon[92425]: pgmap v16385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:23 smithi161 ceph-mon[91579]: pgmap v16386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:23 smithi188 ceph-mon[92425]: pgmap v16386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:25 smithi161 ceph-mon[91579]: pgmap v16387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:25 smithi188 ceph-mon[92425]: pgmap v16387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:27 smithi161 ceph-mon[91579]: pgmap v16388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:27 smithi188 ceph-mon[92425]: pgmap v16388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:29 smithi161 ceph-mon[91579]: pgmap v16389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:29 smithi188 ceph-mon[92425]: pgmap v16389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:31 smithi161 ceph-mon[91579]: pgmap v16390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:31 smithi188 ceph-mon[92425]: pgmap v16390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:33 smithi161 ceph-mon[91579]: pgmap v16391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:33 smithi188 ceph-mon[92425]: pgmap v16391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:35 smithi161 ceph-mon[91579]: pgmap v16392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:35 smithi188 ceph-mon[92425]: pgmap v16392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:37 smithi161 ceph-mon[91579]: pgmap v16393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:37 smithi188 ceph-mon[92425]: pgmap v16393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:39 smithi161 ceph-mon[91579]: pgmap v16394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:39 smithi188 ceph-mon[92425]: pgmap v16394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:41 smithi161 ceph-mon[91579]: pgmap v16395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:41 smithi188 ceph-mon[92425]: pgmap v16395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:43 smithi161 ceph-mon[91579]: pgmap v16396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:43 smithi188 ceph-mon[92425]: pgmap v16396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:45 smithi161 ceph-mon[91579]: pgmap v16397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:45 smithi188 ceph-mon[92425]: pgmap v16397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:47 smithi161 ceph-mon[91579]: pgmap v16398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:47 smithi188 ceph-mon[92425]: pgmap v16398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:49 smithi161 ceph-mon[91579]: pgmap v16399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:49 smithi188 ceph-mon[92425]: pgmap v16399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:51 smithi161 ceph-mon[91579]: pgmap v16400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:51 smithi188 ceph-mon[92425]: pgmap v16400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:53 smithi161 ceph-mon[91579]: pgmap v16401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:53 smithi188 ceph-mon[92425]: pgmap v16401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:55 smithi161 ceph-mon[91579]: pgmap v16402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:55 smithi188 ceph-mon[92425]: pgmap v16402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:45:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:57 smithi161 ceph-mon[91579]: pgmap v16403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:57 smithi188 ceph-mon[92425]: pgmap v16403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:45:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:45:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:59 smithi161 ceph-mon[91579]: pgmap v16404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:45:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:45:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:45:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:45:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:59 smithi188 ceph-mon[92425]: pgmap v16404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:45:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:45:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:45:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:45:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:46:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:01 smithi161 ceph-mon[91579]: pgmap v16405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:01 smithi188 ceph-mon[92425]: pgmap v16405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:03 smithi161 ceph-mon[91579]: pgmap v16406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:03 smithi188 ceph-mon[92425]: pgmap v16406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:05 smithi161 ceph-mon[91579]: pgmap v16407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:05 smithi188 ceph-mon[92425]: pgmap v16407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:07 smithi161 ceph-mon[91579]: pgmap v16408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:07 smithi188 ceph-mon[92425]: pgmap v16408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:09 smithi161 ceph-mon[91579]: pgmap v16409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:09 smithi188 ceph-mon[92425]: pgmap v16409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:11 smithi161 ceph-mon[91579]: pgmap v16410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:11 smithi188 ceph-mon[92425]: pgmap v16410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:13 smithi161 ceph-mon[91579]: pgmap v16411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:13 smithi188 ceph-mon[92425]: pgmap v16411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:15 smithi161 ceph-mon[91579]: pgmap v16412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:15 smithi188 ceph-mon[92425]: pgmap v16412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:16.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:46:16.382+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:46:16.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:46:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:46:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:46:16.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:46:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:46:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:46:16.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:46:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:46:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:46:16.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:46:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:46:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:46:16.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:46:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:46:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:46:16.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:46:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:46:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:46:16.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:46:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:46:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:46:16.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:46:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:46:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:46:16.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:46:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:46:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:46:16.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:46:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:46:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:46:16.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:46:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:46:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:46:16.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:46:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:46:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:46:16.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:46:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:46:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:46:16.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:46:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:46:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:46:16.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:46:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:46:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=985] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:46:16.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=985] Local address: 172.21.15.161, port 59964 2022-07-03T19:46:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=985] Peer address: 172.21.15.101, port 22 2022-07-03T19:46:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=985] Connection lost 2022-07-03T19:46:16.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=985] Aborting connection 2022-07-03T19:46:16.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:46:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:46:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:46:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:46:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:46:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:46:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:46:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:46:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:46:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:46:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:46:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:46:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:46:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:46:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:46:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:46:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:46:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:46:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:46:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:46:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:46:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:46:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:46:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:46:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:46:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:46:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:46:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:46:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:46:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:46:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:46:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=985] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:46:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=985] Local address: 172.21.15.161, port 59964 2022-07-03T19:46:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=985] Peer address: 172.21.15.101, port 22 2022-07-03T19:46:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=985] Connection lost 2022-07-03T19:46:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=985] Aborting connection 2022-07-03T19:46:17.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:17 smithi161 ceph-mon[91579]: pgmap v16413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:46:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:46:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:46:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:46:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:46:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:46:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:46:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:46:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:46:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:46:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:46:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:46:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:46:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:46:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:46:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:46:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:46:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:46:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:46:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:46:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:46:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:46:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:46:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:46:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:46:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:46:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:46:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:46:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:46:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:46:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:46:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:46:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:46:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:46:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:46:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=985] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:46:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=985] Local address: 172.21.15.161, port 59964 2022-07-03T19:46:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=985] Peer address: 172.21.15.101, port 22 2022-07-03T19:46:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=985] Connection lost 2022-07-03T19:46:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=985] Aborting connection 2022-07-03T19:46:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:17 smithi188 ceph-mon[92425]: pgmap v16413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:19 smithi161 ceph-mon[91579]: pgmap v16414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:19 smithi188 ceph-mon[92425]: pgmap v16414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:21 smithi161 ceph-mon[91579]: pgmap v16415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:21 smithi188 ceph-mon[92425]: pgmap v16415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:23 smithi161 ceph-mon[91579]: pgmap v16416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:23 smithi188 ceph-mon[92425]: pgmap v16416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:25 smithi161 ceph-mon[91579]: pgmap v16417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:25 smithi188 ceph-mon[92425]: pgmap v16417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:27 smithi161 ceph-mon[91579]: pgmap v16418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:27 smithi188 ceph-mon[92425]: pgmap v16418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:29 smithi161 ceph-mon[91579]: pgmap v16419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:29 smithi188 ceph-mon[92425]: pgmap v16419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:31 smithi161 ceph-mon[91579]: pgmap v16420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:31 smithi188 ceph-mon[92425]: pgmap v16420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:33 smithi161 ceph-mon[91579]: pgmap v16421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:33 smithi188 ceph-mon[92425]: pgmap v16421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:35 smithi161 ceph-mon[91579]: pgmap v16422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:35 smithi188 ceph-mon[92425]: pgmap v16422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:37 smithi161 ceph-mon[91579]: pgmap v16423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:37 smithi188 ceph-mon[92425]: pgmap v16423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:39 smithi161 ceph-mon[91579]: pgmap v16424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:39 smithi188 ceph-mon[92425]: pgmap v16424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:41 smithi161 ceph-mon[91579]: pgmap v16425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:41 smithi188 ceph-mon[92425]: pgmap v16425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:43 smithi161 ceph-mon[91579]: pgmap v16426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:43 smithi188 ceph-mon[92425]: pgmap v16426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:45 smithi161 ceph-mon[91579]: pgmap v16427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:45 smithi188 ceph-mon[92425]: pgmap v16427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:47 smithi161 ceph-mon[91579]: pgmap v16428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:47 smithi188 ceph-mon[92425]: pgmap v16428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:49 smithi161 ceph-mon[91579]: pgmap v16429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:49 smithi188 ceph-mon[92425]: pgmap v16429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:51 smithi161 ceph-mon[91579]: pgmap v16430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:51 smithi188 ceph-mon[92425]: pgmap v16430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:53 smithi161 ceph-mon[91579]: pgmap v16431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:53 smithi188 ceph-mon[92425]: pgmap v16431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:55 smithi161 ceph-mon[91579]: pgmap v16432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:55 smithi188 ceph-mon[92425]: pgmap v16432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:46:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:57 smithi161 ceph-mon[91579]: pgmap v16433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:57 smithi188 ceph-mon[92425]: pgmap v16433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:46:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:46:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:59 smithi161 ceph-mon[91579]: pgmap v16434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:46:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:46:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:46:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:46:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:59 smithi188 ceph-mon[92425]: pgmap v16434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:46:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:46:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:46:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:46:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:47:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:01 smithi161 ceph-mon[91579]: pgmap v16435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:01 smithi188 ceph-mon[92425]: pgmap v16435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:03 smithi161 ceph-mon[91579]: pgmap v16436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:03 smithi188 ceph-mon[92425]: pgmap v16436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:05 smithi161 ceph-mon[91579]: pgmap v16437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:05 smithi188 ceph-mon[92425]: pgmap v16437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:07 smithi161 ceph-mon[91579]: pgmap v16438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:07 smithi188 ceph-mon[92425]: pgmap v16438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:09 smithi161 ceph-mon[91579]: pgmap v16439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:09 smithi188 ceph-mon[92425]: pgmap v16439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:11 smithi161 ceph-mon[91579]: pgmap v16440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:11 smithi188 ceph-mon[92425]: pgmap v16440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:13 smithi161 ceph-mon[91579]: pgmap v16441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:13 smithi188 ceph-mon[92425]: pgmap v16441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:15 smithi161 ceph-mon[91579]: pgmap v16442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:15 smithi188 ceph-mon[92425]: pgmap v16442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:16.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:17.313 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:47:16.879+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:47:17.314 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:47:17.314 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:47:17.315 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:47:17.315 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:47:17.316 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:47:17.316 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:47:17.316 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:47:17.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:47:17.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:47:17.317 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:47:17.318 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:47:17.318 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:47:17.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:47:17.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:47:17.319 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:47:17.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:47:17.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:47:17.320 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:47:17.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:47:17.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:47:17.321 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:47:17.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:47:17.322 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:47:17.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:47:17.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:47:17.323 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:47:17.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:47:17.324 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:47:17.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:47:17.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:47:17.325 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:47:17.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:47:17.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:47:17.326 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:47:17.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:47:17.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:47:17.327 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:47:17.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:47:17.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:47:17.328 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:47:17.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:47:17.329 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:47:17.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:47:17.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:47:17.330 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=987] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:47:17.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=987] Local address: 172.21.15.161, port 59992 2022-07-03T19:47:17.331 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=987] Peer address: 172.21.15.101, port 22 2022-07-03T19:47:17.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=987] Connection lost 2022-07-03T19:47:17.332 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=987] Aborting connection 2022-07-03T19:47:17.333 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:47:16 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:47:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: pgmap v16443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:47:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:47:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:47:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:47:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:17 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:47:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:47:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:47:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:47:17.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:47:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:47:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:47:17.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:47:17.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:47:17.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:47:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:47:17.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:47:17.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:47:17.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:47:17.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:47:17.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:47:17.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:47:17.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:47:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:47:17.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:47:17.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:47:17.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:47:17.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=987] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=987] Local address: 172.21.15.161, port 59992 2022-07-03T19:47:17.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=987] Peer address: 172.21.15.101, port 22 2022-07-03T19:47:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=987] Connection lost 2022-07-03T19:47:17.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=987] Aborting connection 2022-07-03T19:47:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: pgmap v16443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:47:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:47:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:47:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:47:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:17 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:47:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:47:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:47:17.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:47:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:47:17.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:47:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:47:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:47:17.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:47:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:47:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:47:17.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:47:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:47:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:47:17.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:47:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:47:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:47:17.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:47:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:47:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:47:17.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:47:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:47:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:47:17.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:47:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:47:17.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:47:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:47:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:47:17.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:47:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:47:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:47:17.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:47:17.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:47:17.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:47:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:47:17.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:47:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:47:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:47:17.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:47:17.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=987] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=987] Local address: 172.21.15.161, port 59992 2022-07-03T19:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=987] Peer address: 172.21.15.101, port 22 2022-07-03T19:47:17.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=987] Connection lost 2022-07-03T19:47:17.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=987] Aborting connection 2022-07-03T19:47:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:19 smithi161 ceph-mon[91579]: pgmap v16444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:19 smithi188 ceph-mon[92425]: pgmap v16444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:21 smithi161 ceph-mon[91579]: pgmap v16445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:21 smithi188 ceph-mon[92425]: pgmap v16445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:23 smithi161 ceph-mon[91579]: pgmap v16446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:23 smithi188 ceph-mon[92425]: pgmap v16446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:25 smithi161 ceph-mon[91579]: pgmap v16447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:25 smithi188 ceph-mon[92425]: pgmap v16447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:27 smithi161 ceph-mon[91579]: pgmap v16448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:27 smithi188 ceph-mon[92425]: pgmap v16448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:29 smithi161 ceph-mon[91579]: pgmap v16449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:29 smithi188 ceph-mon[92425]: pgmap v16449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:31 smithi161 ceph-mon[91579]: pgmap v16450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:31 smithi188 ceph-mon[92425]: pgmap v16450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:33 smithi161 ceph-mon[91579]: pgmap v16451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:33 smithi188 ceph-mon[92425]: pgmap v16451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:35 smithi161 ceph-mon[91579]: pgmap v16452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:35 smithi188 ceph-mon[92425]: pgmap v16452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:37 smithi161 ceph-mon[91579]: pgmap v16453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:37 smithi188 ceph-mon[92425]: pgmap v16453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:39 smithi161 ceph-mon[91579]: pgmap v16454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:39 smithi188 ceph-mon[92425]: pgmap v16454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:41 smithi161 ceph-mon[91579]: pgmap v16455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:41 smithi188 ceph-mon[92425]: pgmap v16455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:43 smithi161 ceph-mon[91579]: pgmap v16456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:43 smithi188 ceph-mon[92425]: pgmap v16456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:45 smithi161 ceph-mon[91579]: pgmap v16457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:45 smithi188 ceph-mon[92425]: pgmap v16457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:46.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:47 smithi161 ceph-mon[91579]: pgmap v16458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:47 smithi188 ceph-mon[92425]: pgmap v16458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:49 smithi161 ceph-mon[91579]: pgmap v16459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:49 smithi188 ceph-mon[92425]: pgmap v16459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:51 smithi161 ceph-mon[91579]: pgmap v16460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:51 smithi188 ceph-mon[92425]: pgmap v16460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:53 smithi161 ceph-mon[91579]: pgmap v16461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:53 smithi188 ceph-mon[92425]: pgmap v16461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:55 smithi161 ceph-mon[91579]: pgmap v16462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:55 smithi188 ceph-mon[92425]: pgmap v16462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:47:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:57 smithi161 ceph-mon[91579]: pgmap v16463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:57 smithi188 ceph-mon[92425]: pgmap v16463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:47:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:47:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:59 smithi161 ceph-mon[91579]: pgmap v16464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:47:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:47:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:47:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:47:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:59 smithi188 ceph-mon[92425]: pgmap v16464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:47:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:47:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:47:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:47:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:48:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:01 smithi161 ceph-mon[91579]: pgmap v16465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:01 smithi188 ceph-mon[92425]: pgmap v16465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:03 smithi161 ceph-mon[91579]: pgmap v16466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:03 smithi188 ceph-mon[92425]: pgmap v16466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:05 smithi188 ceph-mon[92425]: pgmap v16467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:05 smithi161 ceph-mon[91579]: pgmap v16467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:07 smithi161 ceph-mon[91579]: pgmap v16468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:07 smithi188 ceph-mon[92425]: pgmap v16468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:09 smithi161 ceph-mon[91579]: pgmap v16469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:09 smithi188 ceph-mon[92425]: pgmap v16469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:11 smithi161 ceph-mon[91579]: pgmap v16470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:11 smithi188 ceph-mon[92425]: pgmap v16470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:13 smithi161 ceph-mon[91579]: pgmap v16471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:13 smithi188 ceph-mon[92425]: pgmap v16471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:15 smithi188 ceph-mon[92425]: pgmap v16472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:15 smithi161 ceph-mon[91579]: pgmap v16472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-mon[91579]: pgmap v16473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:17.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:48:17.378+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:48:17.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:48:17.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:48:17.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:48:17.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:48:17.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:48:17.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:48:17.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:48:17.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:48:17.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:48:17.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:48:17.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:48:17.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:48:17.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:48:17.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:48:17.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:48:17.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:48:17.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:48:17.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:48:17.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:48:17.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:48:17.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:48:17.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:48:17.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:48:17.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:48:17.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:48:17.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:48:17.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:48:17.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:48:17.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:48:17.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:48:17.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:48:17.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:48:17.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:48:17.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:48:17.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:48:17.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:48:17.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:48:17.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:48:17.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:48:17.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:48:17.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:48:17.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:48:17.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:48:17.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:48:17.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=989] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:48:17.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=989] Local address: 172.21.15.161, port 60020 2022-07-03T19:48:17.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=989] Peer address: 172.21.15.101, port 22 2022-07-03T19:48:17.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=989] Connection lost 2022-07-03T19:48:17.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=989] Aborting connection 2022-07-03T19:48:17.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:48:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:48:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:17 smithi188 ceph-mon[92425]: pgmap v16473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:48:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:48:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:48:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:48:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:48:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:48:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:48:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:48:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:48:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:48:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:48:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:48:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:48:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:48:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:48:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:48:18.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:48:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:48:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:48:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:48:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:48:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:48:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:48:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:48:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:48:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:48:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:48:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:48:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:48:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:48:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:48:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:48:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:48:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:48:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:48:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:48:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:48:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:48:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:48:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:48:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:48:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:48:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:48:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:48:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:48:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:48:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:48:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:48:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:48:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=989] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:48:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=989] Local address: 172.21.15.161, port 60020 2022-07-03T19:48:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=989] Peer address: 172.21.15.101, port 22 2022-07-03T19:48:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=989] Connection lost 2022-07-03T19:48:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=989] Aborting connection 2022-07-03T19:48:18.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.648 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.649 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:48:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:48:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:48:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:48:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:48:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:48:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:48:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:48:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:48:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:48:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:48:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:48:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:48:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:48:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:48:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:48:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:48:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:48:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:48:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:48:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:48:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:48:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:48:18.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:48:18.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:48:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:48:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:48:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:48:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:48:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:48:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:48:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:48:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:48:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:48:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:48:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:48:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:48:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:48:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:48:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:48:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:48:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:48:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:48:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:48:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:48:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:48:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:48:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:48:18.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:48:18.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=989] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:48:18.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=989] Local address: 172.21.15.161, port 60020 2022-07-03T19:48:18.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=989] Peer address: 172.21.15.101, port 22 2022-07-03T19:48:18.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=989] Connection lost 2022-07-03T19:48:18.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=989] Aborting connection 2022-07-03T19:48:18.685 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:18.687 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:19.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:19 smithi188 ceph-mon[92425]: pgmap v16474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:19 smithi161 ceph-mon[91579]: pgmap v16474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:21 smithi161 ceph-mon[91579]: pgmap v16475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:21 smithi188 ceph-mon[92425]: pgmap v16475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:48:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:23.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:22 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T19:48:23.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:22 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T19:48:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:23 smithi161 ceph-mon[91579]: pgmap v16476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:23 smithi188 ceph-mon[92425]: pgmap v16476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:25 smithi188 ceph-mon[92425]: pgmap v16477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:25 smithi161 ceph-mon[91579]: pgmap v16477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:27 smithi161 ceph-mon[91579]: pgmap v16478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:27 smithi188 ceph-mon[92425]: pgmap v16478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:29 smithi161 ceph-mon[91579]: pgmap v16479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:29 smithi188 ceph-mon[92425]: pgmap v16479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:31 smithi161 ceph-mon[91579]: pgmap v16480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:31 smithi188 ceph-mon[92425]: pgmap v16480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:33 smithi161 ceph-mon[91579]: pgmap v16481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:33 smithi188 ceph-mon[92425]: pgmap v16481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:35 smithi188 ceph-mon[92425]: pgmap v16482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:35 smithi161 ceph-mon[91579]: pgmap v16482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:37 smithi161 ceph-mon[91579]: pgmap v16483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:37 smithi188 ceph-mon[92425]: pgmap v16483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:39 smithi161 ceph-mon[91579]: pgmap v16484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:39 smithi188 ceph-mon[92425]: pgmap v16484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:41 smithi161 ceph-mon[91579]: pgmap v16485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:41 smithi188 ceph-mon[92425]: pgmap v16485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:43 smithi161 ceph-mon[91579]: pgmap v16486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:43 smithi188 ceph-mon[92425]: pgmap v16486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:45 smithi188 ceph-mon[92425]: pgmap v16487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:45 smithi161 ceph-mon[91579]: pgmap v16487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:47 smithi161 ceph-mon[91579]: pgmap v16488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:47 smithi188 ceph-mon[92425]: pgmap v16488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:49 smithi161 ceph-mon[91579]: pgmap v16489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:49 smithi188 ceph-mon[92425]: pgmap v16489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:51 smithi161 ceph-mon[91579]: pgmap v16490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:51 smithi188 ceph-mon[92425]: pgmap v16490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:53 smithi161 ceph-mon[91579]: pgmap v16491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:53 smithi188 ceph-mon[92425]: pgmap v16491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:55 smithi188 ceph-mon[92425]: pgmap v16492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:55 smithi161 ceph-mon[91579]: pgmap v16492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:48:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:57 smithi161 ceph-mon[91579]: pgmap v16493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:57 smithi188 ceph-mon[92425]: pgmap v16493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:48:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:48:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:59 smithi161 ceph-mon[91579]: pgmap v16494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:48:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:48:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:48:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:48:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:59 smithi188 ceph-mon[92425]: pgmap v16494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:48:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:48:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:48:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:48:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:49:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:01 smithi161 ceph-mon[91579]: pgmap v16495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:01 smithi188 ceph-mon[92425]: pgmap v16495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:03 smithi161 ceph-mon[91579]: pgmap v16496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:03 smithi188 ceph-mon[92425]: pgmap v16496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:05 smithi188 ceph-mon[92425]: pgmap v16497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:05 smithi161 ceph-mon[91579]: pgmap v16497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:07 smithi161 ceph-mon[91579]: pgmap v16498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:07 smithi188 ceph-mon[92425]: pgmap v16498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:09 smithi161 ceph-mon[91579]: pgmap v16499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:09 smithi188 ceph-mon[92425]: pgmap v16499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:11 smithi161 ceph-mon[91579]: pgmap v16500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:11 smithi188 ceph-mon[92425]: pgmap v16500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:13 smithi161 ceph-mon[91579]: pgmap v16501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:13 smithi188 ceph-mon[92425]: pgmap v16501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:15 smithi188 ceph-mon[92425]: pgmap v16502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:15 smithi161 ceph-mon[91579]: pgmap v16502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:17 smithi161 ceph-mon[91579]: pgmap v16503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:17 smithi188 ceph-mon[92425]: pgmap v16503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:19 smithi161 ceph-mon[91579]: pgmap v16504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:19 smithi188 ceph-mon[92425]: pgmap v16504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:21.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:49:21.087+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:49:21.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:49:21.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:49:21.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:49:21.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:49:21.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:49:21.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:49:21.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:49:21.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:49:21.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:49:21.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:49:21.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:49:21.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:49:21.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:49:21.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:49:21.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:49:21.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:49:21.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:49:21.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:49:21.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:49:21.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:49:21.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:49:21.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:49:21.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:49:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:49:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:49:21.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:49:21.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:49:21.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:49:21.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:49:21.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:49:21.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:49:21.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:49:21.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:49:21.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:49:21.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:49:21.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:49:21.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:49:21.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:49:21.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:49:21.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:49:21.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:49:21.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:49:21.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:49:21.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:49:21.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=991] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:49:21.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=991] Local address: 172.21.15.161, port 60050 2022-07-03T19:49:21.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=991] Peer address: 172.21.15.101, port 22 2022-07-03T19:49:21.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=991] Connection lost 2022-07-03T19:49:21.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=991] Aborting connection 2022-07-03T19:49:21.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:49:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: pgmap v16505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:49:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:49:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:49:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:22.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:22.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:21 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: pgmap v16505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:49:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:49:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:49:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:22.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:22.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:21 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:22 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:49:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:22 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:49:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:49:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:49:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:49:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:49:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:49:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:49:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:49:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:49:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:49:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:49:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:49:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:49:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:49:23.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:49:23.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:49:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:49:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:49:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:49:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:49:23.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:49:23.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:49:23.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:49:23.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:49:23.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:49:23.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:49:23.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:49:23.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:49:23.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:49:23.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:49:23.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:49:23.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:49:23.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:49:23.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:49:23.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:49:23.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:49:23.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:49:23.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:49:23.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:49:23.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:49:23.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:49:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:49:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:49:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:49:23.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:49:23.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=991] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:49:23.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=991] Local address: 172.21.15.161, port 60050 2022-07-03T19:49:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=991] Peer address: 172.21.15.101, port 22 2022-07-03T19:49:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=991] Connection lost 2022-07-03T19:49:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=991] Aborting connection 2022-07-03T19:49:23.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:22 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:23.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:22 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:22 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:49:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:22 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:49:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:49:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:49:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:49:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:49:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:49:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:49:23.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:49:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:49:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:49:23.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:49:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:49:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:49:23.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:49:23.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:49:23.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:49:23.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:49:23.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:49:23.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:49:23.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:49:23.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:49:23.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:49:23.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:49:23.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:49:23.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:49:23.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:49:23.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:49:23.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:49:23.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:49:23.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:49:23.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:49:23.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:49:23.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:49:23.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:49:23.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:49:23.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:49:23.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:49:23.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:49:23.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:49:23.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:49:23.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:49:23.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:49:23.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:49:23.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:49:23.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:49:23.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=991] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:49:23.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=991] Local address: 172.21.15.161, port 60050 2022-07-03T19:49:23.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=991] Peer address: 172.21.15.101, port 22 2022-07-03T19:49:23.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=991] Connection lost 2022-07-03T19:49:23.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=991] Aborting connection 2022-07-03T19:49:23.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:22 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:23.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:22 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:49:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:23 smithi161 ceph-mon[91579]: pgmap v16506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:23 smithi188 ceph-mon[92425]: pgmap v16506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:25 smithi188 ceph-mon[92425]: pgmap v16507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:25 smithi161 ceph-mon[91579]: pgmap v16507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:27 smithi161 ceph-mon[91579]: pgmap v16508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:27 smithi188 ceph-mon[92425]: pgmap v16508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:29 smithi188 ceph-mon[92425]: pgmap v16509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:29 smithi161 ceph-mon[91579]: pgmap v16509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:31 smithi161 ceph-mon[91579]: pgmap v16510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:31 smithi188 ceph-mon[92425]: pgmap v16510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:33 smithi161 ceph-mon[91579]: pgmap v16511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:33 smithi188 ceph-mon[92425]: pgmap v16511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:35 smithi188 ceph-mon[92425]: pgmap v16512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:35 smithi161 ceph-mon[91579]: pgmap v16512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:37 smithi188 ceph-mon[92425]: pgmap v16513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:37 smithi161 ceph-mon[91579]: pgmap v16513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:39 smithi188 ceph-mon[92425]: pgmap v16514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:39 smithi161 ceph-mon[91579]: pgmap v16514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:41 smithi161 ceph-mon[91579]: pgmap v16515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:41 smithi188 ceph-mon[92425]: pgmap v16515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:43 smithi161 ceph-mon[91579]: pgmap v16516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:43 smithi188 ceph-mon[92425]: pgmap v16516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:45 smithi188 ceph-mon[92425]: pgmap v16517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:45.637 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:45 smithi161 ceph-mon[91579]: pgmap v16517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:47 smithi188 ceph-mon[92425]: pgmap v16518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:47 smithi161 ceph-mon[91579]: pgmap v16518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:49.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:49 smithi188 ceph-mon[92425]: pgmap v16519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:49 smithi161 ceph-mon[91579]: pgmap v16519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:51 smithi161 ceph-mon[91579]: pgmap v16520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:51 smithi188 ceph-mon[92425]: pgmap v16520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:53 smithi188 ceph-mon[92425]: pgmap v16521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:53 smithi161 ceph-mon[91579]: pgmap v16521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:55 smithi188 ceph-mon[92425]: pgmap v16522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:55 smithi161 ceph-mon[91579]: pgmap v16522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:56.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:49:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:57 smithi188 ceph-mon[92425]: pgmap v16523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:57 smithi161 ceph-mon[91579]: pgmap v16523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:49:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:49:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:59 smithi188 ceph-mon[92425]: pgmap v16524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:49:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:49:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:49:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:49:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:59 smithi161 ceph-mon[91579]: pgmap v16524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:49:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:49:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:49:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:49:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:50:00.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:50:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T19:50:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:01 smithi188 ceph-mon[92425]: pgmap v16525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:01.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:01 smithi161 ceph-mon[91579]: pgmap v16525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:03 smithi188 ceph-mon[92425]: pgmap v16526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:03 smithi161 ceph-mon[91579]: pgmap v16526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:05 smithi188 ceph-mon[92425]: pgmap v16527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:05 smithi161 ceph-mon[91579]: pgmap v16527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:07 smithi188 ceph-mon[92425]: pgmap v16528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:07 smithi161 ceph-mon[91579]: pgmap v16528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:09 smithi188 ceph-mon[92425]: pgmap v16529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:09 smithi161 ceph-mon[91579]: pgmap v16529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:11 smithi188 ceph-mon[92425]: pgmap v16530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:11 smithi161 ceph-mon[91579]: pgmap v16530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:13 smithi188 ceph-mon[92425]: pgmap v16531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:13 smithi161 ceph-mon[91579]: pgmap v16531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:15 smithi161 ceph-mon[91579]: pgmap v16532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:15 smithi188 ceph-mon[92425]: pgmap v16532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:17 smithi188 ceph-mon[92425]: pgmap v16533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:17.885 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:17 smithi161 ceph-mon[91579]: pgmap v16533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:19 smithi188 ceph-mon[92425]: pgmap v16534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:19 smithi161 ceph-mon[91579]: pgmap v16534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:21 smithi188 ceph-mon[92425]: pgmap v16535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-mon[91579]: pgmap v16535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:22.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:50:21.980+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:50:22.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:50:22.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:50:22.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:50:22.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:50:22.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:50:22.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:50:22.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:50:22.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:50:22.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:50:22.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:50:22.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:50:22.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:50:22.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:50:22.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:50:22.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:50:22.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:50:22.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:50:22.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:50:22.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:50:22.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:50:22.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:50:22.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:50:22.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:50:22.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:50:22.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:50:22.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:50:22.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:50:22.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:50:22.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:50:22.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:50:22.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:50:22.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:50:22.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:50:22.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:50:22.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:50:22.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:50:22.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:50:22.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:50:22.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:50:22.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:50:22.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:50:22.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:50:22.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:50:22.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:50:22.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=993] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:50:22.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=993] Local address: 172.21.15.161, port 60078 2022-07-03T19:50:22.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=993] Peer address: 172.21.15.101, port 22 2022-07-03T19:50:22.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=993] Connection lost 2022-07-03T19:50:22.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=993] Aborting connection 2022-07-03T19:50:22.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:50:21 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:50:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:50:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:50:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:50:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:22 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:50:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:22 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:50:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:50:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:50:23.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:50:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:50:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:50:23.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:50:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:50:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:50:23.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:50:23.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:50:23.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:50:23.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:50:23.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:50:23.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:50:23.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:50:23.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:50:23.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:50:23.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:50:23.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:50:23.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:50:23.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:50:23.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:50:23.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:50:23.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:50:23.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:50:23.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:50:23.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:50:23.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:50:23.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:50:23.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:50:23.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:50:23.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:50:23.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:50:23.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:50:23.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:50:23.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:50:23.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:50:23.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:50:23.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:50:23.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:50:23.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:50:23.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:50:23.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:50:23.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:50:23.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=993] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:50:23.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=993] Local address: 172.21.15.161, port 60078 2022-07-03T19:50:23.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=993] Peer address: 172.21.15.101, port 22 2022-07-03T19:50:23.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=993] Connection lost 2022-07-03T19:50:23.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=993] Aborting connection 2022-07-03T19:50:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:50:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:50:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:50:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:22 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:50:23.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:22 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:50:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:50:23.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:50:23.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:50:23.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:50:23.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:50:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:50:23.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:50:23.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:50:23.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:50:23.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:50:23.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:50:23.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:50:23.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:50:23.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:50:23.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:50:23.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:50:23.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:50:23.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:50:23.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:50:23.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:50:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:50:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:50:23.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:50:23.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:50:23.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:50:23.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:50:23.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:50:23.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:50:23.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:50:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:50:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:50:23.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:50:23.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:50:23.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:50:23.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:50:23.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:50:23.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:50:23.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:50:23.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:50:23.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:50:23.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:50:23.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:50:23.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:50:23.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:50:23.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=993] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:50:23.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=993] Local address: 172.21.15.161, port 60078 2022-07-03T19:50:23.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=993] Peer address: 172.21.15.101, port 22 2022-07-03T19:50:23.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=993] Connection lost 2022-07-03T19:50:23.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=993] Aborting connection 2022-07-03T19:50:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:23 smithi188 ceph-mon[92425]: pgmap v16536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:23 smithi161 ceph-mon[91579]: pgmap v16536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:25 smithi161 ceph-mon[91579]: pgmap v16537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:25 smithi188 ceph-mon[92425]: pgmap v16537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:27 smithi188 ceph-mon[92425]: pgmap v16538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:27 smithi161 ceph-mon[91579]: pgmap v16538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:29 smithi188 ceph-mon[92425]: pgmap v16539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:29 smithi161 ceph-mon[91579]: pgmap v16539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:31 smithi188 ceph-mon[92425]: pgmap v16540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:31 smithi161 ceph-mon[91579]: pgmap v16540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:33 smithi188 ceph-mon[92425]: pgmap v16541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:33 smithi161 ceph-mon[91579]: pgmap v16541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:35 smithi161 ceph-mon[91579]: pgmap v16542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:35 smithi188 ceph-mon[92425]: pgmap v16542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:37 smithi188 ceph-mon[92425]: pgmap v16543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:37 smithi161 ceph-mon[91579]: pgmap v16543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:39 smithi161 ceph-mon[91579]: pgmap v16544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:39 smithi188 ceph-mon[92425]: pgmap v16544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:41 smithi188 ceph-mon[92425]: pgmap v16545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:42.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:42.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:41 smithi161 ceph-mon[91579]: pgmap v16545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:43 smithi188 ceph-mon[92425]: pgmap v16546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:43 smithi161 ceph-mon[91579]: pgmap v16546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:45 smithi161 ceph-mon[91579]: pgmap v16547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:45 smithi188 ceph-mon[92425]: pgmap v16547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:47 smithi161 ceph-mon[91579]: pgmap v16548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:47 smithi188 ceph-mon[92425]: pgmap v16548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:49 smithi161 ceph-mon[91579]: pgmap v16549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:49 smithi188 ceph-mon[92425]: pgmap v16549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:51 smithi188 ceph-mon[92425]: pgmap v16550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:51 smithi161 ceph-mon[91579]: pgmap v16550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:53 smithi188 ceph-mon[92425]: pgmap v16551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:53 smithi161 ceph-mon[91579]: pgmap v16551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:55 smithi161 ceph-mon[91579]: pgmap v16552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:55 smithi188 ceph-mon[92425]: pgmap v16552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:50:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:57 smithi161 ceph-mon[91579]: pgmap v16553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:57 smithi188 ceph-mon[92425]: pgmap v16553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:50:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:50:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:59 smithi161 ceph-mon[91579]: pgmap v16554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:50:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:50:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:50:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:50:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:59 smithi188 ceph-mon[92425]: pgmap v16554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:50:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:50:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:51:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:01 smithi188 ceph-mon[92425]: pgmap v16555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:01 smithi161 ceph-mon[91579]: pgmap v16555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:03 smithi188 ceph-mon[92425]: pgmap v16556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:03 smithi161 ceph-mon[91579]: pgmap v16556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:05 smithi161 ceph-mon[91579]: pgmap v16557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:05 smithi188 ceph-mon[92425]: pgmap v16557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:07 smithi161 ceph-mon[91579]: pgmap v16558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:07 smithi188 ceph-mon[92425]: pgmap v16558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:09 smithi161 ceph-mon[91579]: pgmap v16559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:09 smithi188 ceph-mon[92425]: pgmap v16559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:11 smithi188 ceph-mon[92425]: pgmap v16560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:11 smithi161 ceph-mon[91579]: pgmap v16560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:13 smithi188 ceph-mon[92425]: pgmap v16561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:13 smithi161 ceph-mon[91579]: pgmap v16561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:15 smithi161 ceph-mon[91579]: pgmap v16562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:15 smithi188 ceph-mon[92425]: pgmap v16562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:17 smithi161 ceph-mon[91579]: pgmap v16563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:17 smithi188 ceph-mon[92425]: pgmap v16563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:19 smithi161 ceph-mon[91579]: pgmap v16564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:19 smithi188 ceph-mon[92425]: pgmap v16564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:21 smithi188 ceph-mon[92425]: pgmap v16565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:21 smithi161 ceph-mon[91579]: pgmap v16565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:22.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:51:22.487+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:51:22.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:51:22.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:51:22.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:51:22.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:51:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:51:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:51:22.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:51:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:51:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:51:22.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:51:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:51:22.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:51:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:51:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:51:22.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:51:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:51:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:51:22.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:51:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:51:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:51:22.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:51:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:51:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:51:22.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:51:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:51:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:51:22.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:51:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:51:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:51:22.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:51:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:51:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:51:22.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:51:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:51:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:51:22.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:51:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:51:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:51:22.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:51:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:51:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:51:22.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:51:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:51:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:51:22.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=995] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:51:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=995] Local address: 172.21.15.161, port 60106 2022-07-03T19:51:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=995] Peer address: 172.21.15.101, port 22 2022-07-03T19:51:22.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=995] Connection lost 2022-07-03T19:51:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=995] Aborting connection 2022-07-03T19:51:22.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:51:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:51:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:51:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:51:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:51:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:51:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:51:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:23 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:51:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:23 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:51:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:51:24.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:51:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:51:24.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:51:24.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:51:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:51:24.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:51:24.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:51:24.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:51:24.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:51:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:51:24.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:51:24.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:51:24.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:51:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:51:24.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:51:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:51:24.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:51:24.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:51:24.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:51:24.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:51:24.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:51:24.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:51:24.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:51:24.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:51:24.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:51:24.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:51:24.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:51:24.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:51:24.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:51:24.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:51:24.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:51:24.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:51:24.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:51:24.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:51:24.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:51:24.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:51:24.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:51:24.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:51:24.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:51:24.405 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:51:24.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:51:24.406 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:51:24.407 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:51:24.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=995] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:51:24.408 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=995] Local address: 172.21.15.161, port 60106 2022-07-03T19:51:24.409 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=995] Peer address: 172.21.15.101, port 22 2022-07-03T19:51:24.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=995] Connection lost 2022-07-03T19:51:24.410 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=995] Aborting connection 2022-07-03T19:51:24.411 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:23 smithi161 ceph-mon[91579]: pgmap v16566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:24.411 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:23 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:51:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:23 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:51:24.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:51:24.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:51:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:51:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:51:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:51:24.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:51:24.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:51:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:51:24.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:51:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:51:24.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:51:24.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:51:24.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:51:24.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:51:24.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:51:24.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:51:24.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:51:24.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:51:24.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:51:24.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:51:24.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:51:24.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:51:24.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:51:24.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:51:24.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:51:24.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:51:24.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:51:24.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:51:24.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:51:24.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:51:24.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:51:24.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:51:24.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:51:24.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:51:24.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:51:24.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:51:24.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:51:24.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:51:24.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:51:24.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:51:24.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:51:24.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:51:24.438 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:51:24.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:51:24.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=995] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:51:24.439 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=995] Local address: 172.21.15.161, port 60106 2022-07-03T19:51:24.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=995] Peer address: 172.21.15.101, port 22 2022-07-03T19:51:24.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=995] Connection lost 2022-07-03T19:51:24.440 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=995] Aborting connection 2022-07-03T19:51:24.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:23 smithi188 ceph-mon[92425]: pgmap v16566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:24.441 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:25 smithi161 ceph-mon[91579]: pgmap v16567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:25 smithi188 ceph-mon[92425]: pgmap v16567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 95812k 2022-07-03T19:51:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 98111955: error parsing value: Value '98111955' is below minimum 939524096 2022-07-03T19:51:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:26 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:51:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 95812k 2022-07-03T19:51:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 98111955: error parsing value: Value '98111955' is below minimum 939524096 2022-07-03T19:51:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:26 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T19:51:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:27 smithi161 ceph-mon[91579]: pgmap v16568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:27 smithi188 ceph-mon[92425]: pgmap v16568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:29 smithi161 ceph-mon[91579]: pgmap v16569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:29 smithi188 ceph-mon[92425]: pgmap v16569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:32.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:31 smithi188 ceph-mon[92425]: pgmap v16570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:32.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:31 smithi161 ceph-mon[91579]: pgmap v16570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:33 smithi188 ceph-mon[92425]: pgmap v16571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:33 smithi161 ceph-mon[91579]: pgmap v16571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:35 smithi161 ceph-mon[91579]: pgmap v16572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:35 smithi188 ceph-mon[92425]: pgmap v16572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:37 smithi161 ceph-mon[91579]: pgmap v16573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:37 smithi188 ceph-mon[92425]: pgmap v16573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:38.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:39 smithi161 ceph-mon[91579]: pgmap v16574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:39 smithi188 ceph-mon[92425]: pgmap v16574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:41 smithi188 ceph-mon[92425]: pgmap v16575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:41 smithi161 ceph-mon[91579]: pgmap v16575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:43 smithi161 ceph-mon[91579]: pgmap v16576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:43 smithi188 ceph-mon[92425]: pgmap v16576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:45 smithi161 ceph-mon[91579]: pgmap v16577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:45 smithi188 ceph-mon[92425]: pgmap v16577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:47 smithi161 ceph-mon[91579]: pgmap v16578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:47 smithi188 ceph-mon[92425]: pgmap v16578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:49 smithi161 ceph-mon[91579]: pgmap v16579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:49 smithi188 ceph-mon[92425]: pgmap v16579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:51 smithi161 ceph-mon[91579]: pgmap v16580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:51 smithi188 ceph-mon[92425]: pgmap v16580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T19:51:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:53 smithi161 ceph-mon[91579]: pgmap v16581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:53 smithi188 ceph-mon[92425]: pgmap v16581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:55 smithi161 ceph-mon[91579]: pgmap v16582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:55 smithi188 ceph-mon[92425]: pgmap v16582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:56.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:51:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:57 smithi161 ceph-mon[91579]: pgmap v16583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:57 smithi188 ceph-mon[92425]: pgmap v16583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:51:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:51:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:59 smithi161 ceph-mon[91579]: pgmap v16584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:51:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:51:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:51:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:51:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:59 smithi188 ceph-mon[92425]: pgmap v16584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:51:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:51:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:51:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:51:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:52:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:01 smithi161 ceph-mon[91579]: pgmap v16585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:01 smithi188 ceph-mon[92425]: pgmap v16585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:03 smithi161 ceph-mon[91579]: pgmap v16586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:03 smithi188 ceph-mon[92425]: pgmap v16586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:05 smithi161 ceph-mon[91579]: pgmap v16587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:05 smithi188 ceph-mon[92425]: pgmap v16587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:07 smithi161 ceph-mon[91579]: pgmap v16588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:07 smithi188 ceph-mon[92425]: pgmap v16588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:09 smithi161 ceph-mon[91579]: pgmap v16589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:09 smithi188 ceph-mon[92425]: pgmap v16589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:11 smithi161 ceph-mon[91579]: pgmap v16590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:11 smithi188 ceph-mon[92425]: pgmap v16590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:13 smithi161 ceph-mon[91579]: pgmap v16591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:13 smithi188 ceph-mon[92425]: pgmap v16591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:15 smithi161 ceph-mon[91579]: pgmap v16592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:15 smithi188 ceph-mon[92425]: pgmap v16592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:17 smithi161 ceph-mon[91579]: pgmap v16593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:17 smithi188 ceph-mon[92425]: pgmap v16593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:19 smithi161 ceph-mon[91579]: pgmap v16594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:19 smithi188 ceph-mon[92425]: pgmap v16594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:21 smithi161 ceph-mon[91579]: pgmap v16595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:21 smithi188 ceph-mon[92425]: pgmap v16595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:23 smithi161 ceph-mon[91579]: pgmap v16596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:23 smithi188 ceph-mon[92425]: pgmap v16596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-mon[91579]: pgmap v16597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:25 smithi188 ceph-mon[92425]: pgmap v16597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:26.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:52:25.742+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:52:26.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:52:26.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:52:26.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:52:26.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:52:26.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:52:26.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:52:26.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:52:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:52:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:52:26.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:52:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:52:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:52:26.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:52:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:52:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:52:26.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:52:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:52:26.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:52:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:52:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:52:26.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:52:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:52:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:52:26.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:52:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:52:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:52:26.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:52:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:52:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:52:26.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:52:26.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:52:26.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:52:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:52:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:52:26.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:52:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:52:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:52:26.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:52:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:52:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:52:26.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:52:26.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:52:26.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:52:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:52:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=997] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:52:26.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=997] Local address: 172.21.15.161, port 60136 2022-07-03T19:52:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=997] Peer address: 172.21.15.101, port 22 2022-07-03T19:52:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=997] Connection lost 2022-07-03T19:52:26.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=997] Aborting connection 2022-07-03T19:52:26.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:52:25 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:52:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:52:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:52:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:52:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:52:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:52:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:52:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:52:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:52:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:52:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:52:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:52:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:52:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:52:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:52:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:52:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:52:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:52:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:52:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:52:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:52:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:52:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:52:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:52:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:52:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:52:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:52:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:52:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:52:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=997] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=997] Local address: 172.21.15.161, port 60136 2022-07-03T19:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=997] Peer address: 172.21.15.101, port 22 2022-07-03T19:52:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=997] Connection lost 2022-07-03T19:52:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=997] Aborting connection 2022-07-03T19:52:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:52:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:52:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:52:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:52:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:52:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:52:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:52:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:52:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:52:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:52:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:52:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:52:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:52:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:52:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:52:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:52:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:52:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:52:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:52:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:52:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:52:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:52:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:52:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:52:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:52:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:52:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:52:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:52:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:52:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:52:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:52:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:52:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:52:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:52:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:52:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:52:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:52:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:52:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:52:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:52:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:52:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:52:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:52:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:52:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:52:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:52:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:52:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=997] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:52:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=997] Local address: 172.21.15.161, port 60136 2022-07-03T19:52:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=997] Peer address: 172.21.15.101, port 22 2022-07-03T19:52:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=997] Connection lost 2022-07-03T19:52:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=997] Aborting connection 2022-07-03T19:52:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:27 smithi161 ceph-mon[91579]: pgmap v16598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:27 smithi188 ceph-mon[92425]: pgmap v16598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:29 smithi161 ceph-mon[91579]: pgmap v16599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:29 smithi188 ceph-mon[92425]: pgmap v16599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:31 smithi161 ceph-mon[91579]: pgmap v16600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:31 smithi188 ceph-mon[92425]: pgmap v16600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:33 smithi161 ceph-mon[91579]: pgmap v16601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:33 smithi188 ceph-mon[92425]: pgmap v16601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:35 smithi161 ceph-mon[91579]: pgmap v16602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:35 smithi188 ceph-mon[92425]: pgmap v16602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:36.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:37 smithi161 ceph-mon[91579]: pgmap v16603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:37 smithi188 ceph-mon[92425]: pgmap v16603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:38.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:39 smithi161 ceph-mon[91579]: pgmap v16604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:39 smithi188 ceph-mon[92425]: pgmap v16604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:41 smithi161 ceph-mon[91579]: pgmap v16605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:41 smithi188 ceph-mon[92425]: pgmap v16605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:43 smithi161 ceph-mon[91579]: pgmap v16606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:43 smithi188 ceph-mon[92425]: pgmap v16606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:45 smithi161 ceph-mon[91579]: pgmap v16607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:45 smithi188 ceph-mon[92425]: pgmap v16607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:47 smithi161 ceph-mon[91579]: pgmap v16608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:47 smithi188 ceph-mon[92425]: pgmap v16608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:49 smithi161 ceph-mon[91579]: pgmap v16609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:49 smithi188 ceph-mon[92425]: pgmap v16609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:51 smithi161 ceph-mon[91579]: pgmap v16610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:51 smithi188 ceph-mon[92425]: pgmap v16610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:53 smithi161 ceph-mon[91579]: pgmap v16611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:53 smithi188 ceph-mon[92425]: pgmap v16611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:55 smithi161 ceph-mon[91579]: pgmap v16612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:55 smithi188 ceph-mon[92425]: pgmap v16612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:52:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:57 smithi161 ceph-mon[91579]: pgmap v16613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:57 smithi188 ceph-mon[92425]: pgmap v16613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:52:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:52:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:59 smithi161 ceph-mon[91579]: pgmap v16614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:52:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:52:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:59 smithi188 ceph-mon[92425]: pgmap v16614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:52:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:52:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:52:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:52:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:53:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:01 smithi161 ceph-mon[91579]: pgmap v16615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:01 smithi188 ceph-mon[92425]: pgmap v16615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:03 smithi161 ceph-mon[91579]: pgmap v16616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:03 smithi188 ceph-mon[92425]: pgmap v16616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:05 smithi161 ceph-mon[91579]: pgmap v16617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:05 smithi188 ceph-mon[92425]: pgmap v16617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:07 smithi161 ceph-mon[91579]: pgmap v16618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:07 smithi188 ceph-mon[92425]: pgmap v16618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:08.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:09 smithi161 ceph-mon[91579]: pgmap v16619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:09 smithi188 ceph-mon[92425]: pgmap v16619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:11 smithi161 ceph-mon[91579]: pgmap v16620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:11 smithi188 ceph-mon[92425]: pgmap v16620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:13 smithi161 ceph-mon[91579]: pgmap v16621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:13 smithi188 ceph-mon[92425]: pgmap v16621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:15 smithi161 ceph-mon[91579]: pgmap v16622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:15 smithi188 ceph-mon[92425]: pgmap v16622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:17 smithi161 ceph-mon[91579]: pgmap v16623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:17 smithi188 ceph-mon[92425]: pgmap v16623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:19 smithi161 ceph-mon[91579]: pgmap v16624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:19 smithi188 ceph-mon[92425]: pgmap v16624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:21 smithi161 ceph-mon[91579]: pgmap v16625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:21 smithi188 ceph-mon[92425]: pgmap v16625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:23 smithi161 ceph-mon[91579]: pgmap v16626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:23 smithi188 ceph-mon[92425]: pgmap v16626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:25 smithi161 ceph-mon[91579]: pgmap v16627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:25 smithi188 ceph-mon[92425]: pgmap v16627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:53:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:53:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:53:26.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:53:26.246+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:53:26.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:53:26.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:53:26.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:53:26.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:53:26.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:53:26.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:53:26.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:53:26.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:53:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:53:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:53:26.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:53:26.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:53:26.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:53:26.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:53:26.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:53:26.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:53:26.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:53:26.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:53:26.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:53:26.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:53:26.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:53:26.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:53:26.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:53:26.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:53:26.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:53:26.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:53:26.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:53:26.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:53:26.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:53:26.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:53:26.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:53:26.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:53:26.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:53:26.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:53:26.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:53:26.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:53:26.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:53:26.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:53:26.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:53:26.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:53:26.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:53:26.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:53:26.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:53:26.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:53:26.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=999] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:53:26.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=999] Local address: 172.21.15.161, port 60164 2022-07-03T19:53:26.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=999] Peer address: 172.21.15.101, port 22 2022-07-03T19:53:26.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=999] Connection lost 2022-07-03T19:53:26.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=999] Aborting connection 2022-07-03T19:53:26.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:53:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:53:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:53:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:53:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:53:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:53:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:53:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:53:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:53:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:53:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:53:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:53:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:53:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:53:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:53:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:53:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:53:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:53:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:53:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:53:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:53:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:53:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:53:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:53:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:53:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:53:27.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:53:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:53:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:53:27.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:53:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:53:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:53:27.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:53:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:53:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:53:27.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:53:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:53:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:53:27.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:53:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:53:27.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:53:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:53:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:53:27.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:53:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:53:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:53:27.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:53:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:53:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:53:27.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:53:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=999] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:53:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=999] Local address: 172.21.15.161, port 60164 2022-07-03T19:53:27.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=999] Peer address: 172.21.15.101, port 22 2022-07-03T19:53:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=999] Connection lost 2022-07-03T19:53:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=999] Aborting connection 2022-07-03T19:53:27.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:27.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:27 smithi161 ceph-mon[91579]: pgmap v16628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:53:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:53:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:53:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:53:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:53:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:53:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:53:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:53:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:53:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:53:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:53:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:53:27.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:53:27.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:53:27.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:53:27.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:53:27.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:53:27.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:53:27.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:53:27.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:53:27.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:53:27.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:53:27.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=999] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=999] Local address: 172.21.15.161, port 60164 2022-07-03T19:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=999] Peer address: 172.21.15.101, port 22 2022-07-03T19:53:27.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=999] Connection lost 2022-07-03T19:53:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=999] Aborting connection 2022-07-03T19:53:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:27.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:27 smithi188 ceph-mon[92425]: pgmap v16628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:29 smithi161 ceph-mon[91579]: pgmap v16629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:29 smithi188 ceph-mon[92425]: pgmap v16629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:31 smithi161 ceph-mon[91579]: pgmap v16630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:31 smithi188 ceph-mon[92425]: pgmap v16630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:33 smithi161 ceph-mon[91579]: pgmap v16631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:33 smithi188 ceph-mon[92425]: pgmap v16631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:35 smithi161 ceph-mon[91579]: pgmap v16632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:35 smithi188 ceph-mon[92425]: pgmap v16632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:37 smithi161 ceph-mon[91579]: pgmap v16633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:37 smithi188 ceph-mon[92425]: pgmap v16633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:38.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:39 smithi161 ceph-mon[91579]: pgmap v16634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:39 smithi188 ceph-mon[92425]: pgmap v16634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:41 smithi161 ceph-mon[91579]: pgmap v16635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:41 smithi188 ceph-mon[92425]: pgmap v16635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:43 smithi161 ceph-mon[91579]: pgmap v16636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:43 smithi188 ceph-mon[92425]: pgmap v16636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:45 smithi161 ceph-mon[91579]: pgmap v16637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:45 smithi188 ceph-mon[92425]: pgmap v16637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:47 smithi161 ceph-mon[91579]: pgmap v16638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:47 smithi188 ceph-mon[92425]: pgmap v16638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:49 smithi161 ceph-mon[91579]: pgmap v16639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:49 smithi188 ceph-mon[92425]: pgmap v16639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:51 smithi161 ceph-mon[91579]: pgmap v16640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:51 smithi188 ceph-mon[92425]: pgmap v16640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:53 smithi161 ceph-mon[91579]: pgmap v16641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:53 smithi188 ceph-mon[92425]: pgmap v16641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:55 smithi161 ceph-mon[91579]: pgmap v16642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:55 smithi188 ceph-mon[92425]: pgmap v16642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:53:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:57 smithi161 ceph-mon[91579]: pgmap v16643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:57 smithi188 ceph-mon[92425]: pgmap v16643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:53:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:53:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:59 smithi161 ceph-mon[91579]: pgmap v16644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:53:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:53:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:59 smithi188 ceph-mon[92425]: pgmap v16644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:53:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:53:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:54:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:01 smithi161 ceph-mon[91579]: pgmap v16645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:01 smithi188 ceph-mon[92425]: pgmap v16645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:03 smithi161 ceph-mon[91579]: pgmap v16646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:03 smithi188 ceph-mon[92425]: pgmap v16646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:05 smithi161 ceph-mon[91579]: pgmap v16647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:05 smithi188 ceph-mon[92425]: pgmap v16647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:07 smithi161 ceph-mon[91579]: pgmap v16648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:07 smithi188 ceph-mon[92425]: pgmap v16648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:08.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:09 smithi161 ceph-mon[91579]: pgmap v16649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:09 smithi188 ceph-mon[92425]: pgmap v16649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:11 smithi161 ceph-mon[91579]: pgmap v16650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:11 smithi188 ceph-mon[92425]: pgmap v16650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:13 smithi161 ceph-mon[91579]: pgmap v16651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:13 smithi188 ceph-mon[92425]: pgmap v16651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:15 smithi161 ceph-mon[91579]: pgmap v16652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:15 smithi188 ceph-mon[92425]: pgmap v16652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:17 smithi161 ceph-mon[91579]: pgmap v16653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:17 smithi188 ceph-mon[92425]: pgmap v16653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:19 smithi161 ceph-mon[91579]: pgmap v16654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:19 smithi188 ceph-mon[92425]: pgmap v16654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:21 smithi161 ceph-mon[91579]: pgmap v16655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:21 smithi188 ceph-mon[92425]: pgmap v16655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:22.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:23 smithi161 ceph-mon[91579]: pgmap v16656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:23 smithi188 ceph-mon[92425]: pgmap v16656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:25 smithi161 ceph-mon[91579]: pgmap v16657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:25 smithi188 ceph-mon[92425]: pgmap v16657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:26.800 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:26.802 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:54:26.745+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:54:26.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:54:26.803 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:54:26.804 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:54:26.804 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:54:26.805 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:54:26.805 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:54:26.806 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:54:26.806 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:54:26.807 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:54:26.807 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:54:26.807 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:54:26.808 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:54:26.808 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:54:26.809 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:54:26.810 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:54:26.810 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:54:26.810 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:54:26.811 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:54:26.811 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:54:26.812 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:54:26.812 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:54:26.813 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:54:26.813 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:54:26.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:54:26.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:54:26.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:54:26.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:54:26.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:54:26.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:54:26.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:54:26.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:54:26.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:54:26.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:54:26.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:54:26.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:54:26.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:54:26.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:54:26.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:54:26.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:54:26.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:54:26.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:54:26.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:54:26.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:54:26.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:54:26.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1001] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:54:26.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1001] Local address: 172.21.15.161, port 60220 2022-07-03T19:54:26.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1001] Peer address: 172.21.15.101, port 22 2022-07-03T19:54:26.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1001] Connection lost 2022-07-03T19:54:26.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1001] Aborting connection 2022-07-03T19:54:26.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:54:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:54:26.828 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:54:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:54:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:54:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:54:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:54:27.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:54:27.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:54:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:54:27.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:54:27.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:54:27.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:54:27.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:54:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:54:27.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:54:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:54:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:54:27.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:54:27.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:54:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:54:27.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:54:27.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:54:27.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:54:27.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:54:27.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:54:27.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:54:27.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:54:27.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:54:27.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:54:27.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1001] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:54:27.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1001] Local address: 172.21.15.161, port 60220 2022-07-03T19:54:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1001] Peer address: 172.21.15.101, port 22 2022-07-03T19:54:27.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1001] Connection lost 2022-07-03T19:54:27.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1001] Aborting connection 2022-07-03T19:54:27.900 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:27 smithi161 ceph-mon[91579]: pgmap v16658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:54:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:54:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:54:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:54:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:54:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:54:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:54:27.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:54:27.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:54:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:54:27.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:54:27.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:54:27.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:54:27.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:54:27.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:54:27.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:54:27.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:54:27.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:54:27.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:54:27.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:54:27.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:54:27.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:54:27.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:54:27.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:54:27.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:54:27.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:54:27.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:54:27.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1001] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:54:27.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1001] Local address: 172.21.15.161, port 60220 2022-07-03T19:54:27.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1001] Peer address: 172.21.15.101, port 22 2022-07-03T19:54:27.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1001] Connection lost 2022-07-03T19:54:27.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1001] Aborting connection 2022-07-03T19:54:27.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:27 smithi188 ceph-mon[92425]: pgmap v16658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:29 smithi161 ceph-mon[91579]: pgmap v16659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:29.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:29 smithi188 ceph-mon[92425]: pgmap v16659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:31 smithi161 ceph-mon[91579]: pgmap v16660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:31 smithi188 ceph-mon[92425]: pgmap v16660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:33 smithi161 ceph-mon[91579]: pgmap v16661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:33 smithi188 ceph-mon[92425]: pgmap v16661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:35 smithi161 ceph-mon[91579]: pgmap v16662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:35 smithi188 ceph-mon[92425]: pgmap v16662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:37 smithi161 ceph-mon[91579]: pgmap v16663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:37 smithi188 ceph-mon[92425]: pgmap v16663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:38.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:39 smithi161 ceph-mon[91579]: pgmap v16664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:39 smithi188 ceph-mon[92425]: pgmap v16664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:41 smithi161 ceph-mon[91579]: pgmap v16665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:41 smithi188 ceph-mon[92425]: pgmap v16665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:43 smithi161 ceph-mon[91579]: pgmap v16666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:43 smithi188 ceph-mon[92425]: pgmap v16666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:45 smithi161 ceph-mon[91579]: pgmap v16667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:45 smithi188 ceph-mon[92425]: pgmap v16667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:47 smithi161 ceph-mon[91579]: pgmap v16668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:47 smithi188 ceph-mon[92425]: pgmap v16668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:49 smithi161 ceph-mon[91579]: pgmap v16669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:49 smithi188 ceph-mon[92425]: pgmap v16669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:51 smithi161 ceph-mon[91579]: pgmap v16670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:51 smithi188 ceph-mon[92425]: pgmap v16670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:52.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:53 smithi161 ceph-mon[91579]: pgmap v16671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:54.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:53 smithi188 ceph-mon[92425]: pgmap v16671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:54.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:55 smithi161 ceph-mon[91579]: pgmap v16672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:55 smithi188 ceph-mon[92425]: pgmap v16672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:54:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:57 smithi161 ceph-mon[91579]: pgmap v16673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:57 smithi188 ceph-mon[92425]: pgmap v16673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:54:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:54:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:59 smithi161 ceph-mon[91579]: pgmap v16674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:54:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:54:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:54:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:54:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:59 smithi188 ceph-mon[92425]: pgmap v16674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:54:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:54:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:54:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:54:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:55:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:01 smithi161 ceph-mon[91579]: pgmap v16675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:02.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:01 smithi188 ceph-mon[92425]: pgmap v16675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:03 smithi161 ceph-mon[91579]: pgmap v16676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:03 smithi188 ceph-mon[92425]: pgmap v16676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:05 smithi161 ceph-mon[91579]: pgmap v16677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:05 smithi188 ceph-mon[92425]: pgmap v16677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:07 smithi161 ceph-mon[91579]: pgmap v16678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:07 smithi188 ceph-mon[92425]: pgmap v16678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:08.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:09 smithi161 ceph-mon[91579]: pgmap v16679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:09 smithi188 ceph-mon[92425]: pgmap v16679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:11 smithi161 ceph-mon[91579]: pgmap v16680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:11 smithi188 ceph-mon[92425]: pgmap v16680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:11 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:13 smithi161 ceph-mon[91579]: pgmap v16681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:13 smithi188 ceph-mon[92425]: pgmap v16681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:15 smithi161 ceph-mon[91579]: pgmap v16682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:15 smithi188 ceph-mon[92425]: pgmap v16682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:17 smithi161 ceph-mon[91579]: pgmap v16683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:17 smithi188 ceph-mon[92425]: pgmap v16683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:19 smithi161 ceph-mon[91579]: pgmap v16684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:19 smithi188 ceph-mon[92425]: pgmap v16684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:21 smithi161 ceph-mon[91579]: pgmap v16685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:21 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:21 smithi188 ceph-mon[92425]: pgmap v16685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:21 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:23 smithi161 ceph-mon[91579]: pgmap v16686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:23 smithi188 ceph-mon[92425]: pgmap v16686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:25 smithi161 ceph-mon[91579]: pgmap v16687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:25 smithi188 ceph-mon[92425]: pgmap v16687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:27.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:55:27.244+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:55:27.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:55:27.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:55:27.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:55:27.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:55:27.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:55:27.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:55:27.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:55:27.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:55:27.549 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:55:27.550 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:55:27.550 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:55:27.550 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:55:27.551 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:55:27.551 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:55:27.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:55:27.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:55:27.552 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:55:27.553 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:55:27.554 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:55:27.554 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:55:27.555 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:55:27.556 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:55:27.556 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:55:27.557 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:55:27.557 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:55:27.558 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:55:27.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:55:27.559 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:55:27.561 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-mon[91579]: pgmap v16688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:27.561 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:55:27.562 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:55:27.563 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:55:27.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:55:27.564 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:55:27.565 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:55:27.565 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:55:27.566 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:55:27.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:55:27.567 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:55:27.568 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:55:27.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:55:27.569 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:55:27.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:55:27.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:55:27.571 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:55:27.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:55:27.572 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:55:27.573 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:55:27.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1003] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:55:27.574 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1003] Local address: 172.21.15.161, port 60248 2022-07-03T19:55:27.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1003] Peer address: 172.21.15.101, port 22 2022-07-03T19:55:27.575 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1003] Connection lost 2022-07-03T19:55:27.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1003] Aborting connection 2022-07-03T19:55:27.576 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:55:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:27 smithi188 ceph-mon[92425]: pgmap v16688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:55:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:55:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:55:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:55:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:55:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:55:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:55:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:55:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:55:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:55:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:55:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:55:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:55:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:55:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:55:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:55:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:55:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:55:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:55:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:55:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:55:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:55:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:55:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:55:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:55:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:55:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:55:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:55:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:55:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:55:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:55:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:55:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:55:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:55:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:55:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:55:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1003] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:55:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1003] Local address: 172.21.15.161, port 60248 2022-07-03T19:55:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1003] Peer address: 172.21.15.101, port 22 2022-07-03T19:55:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1003] Connection lost 2022-07-03T19:55:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1003] Aborting connection 2022-07-03T19:55:28.899 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:55:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:55:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:55:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:55:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:55:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:55:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:55:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:55:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:55:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:55:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:55:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:55:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:55:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:55:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:55:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:55:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:55:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:55:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:55:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:55:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:55:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:55:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:55:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:55:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:55:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:55:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:55:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1003] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:55:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1003] Local address: 172.21.15.161, port 60248 2022-07-03T19:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1003] Peer address: 172.21.15.101, port 22 2022-07-03T19:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1003] Connection lost 2022-07-03T19:55:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1003] Aborting connection 2022-07-03T19:55:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:29 smithi161 ceph-mon[91579]: pgmap v16689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:29.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:29 smithi188 ceph-mon[92425]: pgmap v16689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:31 smithi161 ceph-mon[91579]: pgmap v16690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:31 smithi188 ceph-mon[92425]: pgmap v16690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:33 smithi161 ceph-mon[91579]: pgmap v16691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:33 smithi188 ceph-mon[92425]: pgmap v16691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:35 smithi161 ceph-mon[91579]: pgmap v16692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:35 smithi188 ceph-mon[92425]: pgmap v16692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:37 smithi161 ceph-mon[91579]: pgmap v16693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:37 smithi188 ceph-mon[92425]: pgmap v16693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:39 smithi161 ceph-mon[91579]: pgmap v16694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:39 smithi188 ceph-mon[92425]: pgmap v16694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:41 smithi161 ceph-mon[91579]: pgmap v16695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:41 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:41 smithi188 ceph-mon[92425]: pgmap v16695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:41 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:43 smithi161 ceph-mon[91579]: pgmap v16696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:43 smithi188 ceph-mon[92425]: pgmap v16696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:45 smithi161 ceph-mon[91579]: pgmap v16697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:45 smithi188 ceph-mon[92425]: pgmap v16697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:47 smithi161 ceph-mon[91579]: pgmap v16698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:47 smithi188 ceph-mon[92425]: pgmap v16698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:49 smithi161 ceph-mon[91579]: pgmap v16699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:49 smithi188 ceph-mon[92425]: pgmap v16699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:51 smithi161 ceph-mon[91579]: pgmap v16700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:51 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:51 smithi188 ceph-mon[92425]: pgmap v16700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:51 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:53 smithi161 ceph-mon[91579]: pgmap v16701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:53 smithi188 ceph-mon[92425]: pgmap v16701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:55 smithi161 ceph-mon[91579]: pgmap v16702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:55 smithi188 ceph-mon[92425]: pgmap v16702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:55:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:57 smithi161 ceph-mon[91579]: pgmap v16703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:57 smithi188 ceph-mon[92425]: pgmap v16703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:55:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:55:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:59 smithi161 ceph-mon[91579]: pgmap v16704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:55:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:55:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:55:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:55:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:59 smithi188 ceph-mon[92425]: pgmap v16704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:55:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:55:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:55:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:55:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:56:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:01 smithi161 ceph-mon[91579]: pgmap v16705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:01 smithi188 ceph-mon[92425]: pgmap v16705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:03 smithi161 ceph-mon[91579]: pgmap v16706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:04 smithi188 ceph-mon[92425]: pgmap v16706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:05 smithi161 ceph-mon[91579]: pgmap v16707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:05 smithi188 ceph-mon[92425]: pgmap v16707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:07 smithi161 ceph-mon[91579]: pgmap v16708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:07 smithi188 ceph-mon[92425]: pgmap v16708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:08.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:09 smithi161 ceph-mon[91579]: pgmap v16709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:09 smithi188 ceph-mon[92425]: pgmap v16709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:11 smithi161 ceph-mon[91579]: pgmap v16710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:12.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:11 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:12 smithi188 ceph-mon[92425]: pgmap v16710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:14 smithi161 ceph-mon[91579]: pgmap v16711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:14 smithi188 ceph-mon[92425]: pgmap v16711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:15 smithi161 ceph-mon[91579]: pgmap v16712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:15 smithi188 ceph-mon[92425]: pgmap v16712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:17 smithi161 ceph-mon[91579]: pgmap v16713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:17 smithi188 ceph-mon[92425]: pgmap v16713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:18.570 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:56:18.142+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T19:56:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T19:56:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:19 smithi161 ceph-mon[91579]: pgmap v16714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:19 smithi188 ceph-mon[92425]: pgmap v16714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:22 smithi161 ceph-mon[91579]: pgmap v16715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:22 smithi188 ceph-mon[92425]: pgmap v16715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:24 smithi161 ceph-mon[91579]: pgmap v16716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:24 smithi188 ceph-mon[92425]: pgmap v16716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:25 smithi161 ceph-mon[91579]: pgmap v16717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:25 smithi188 ceph-mon[92425]: pgmap v16717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-mon[91579]: pgmap v16718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:27.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:56:27.748+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:56:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:56:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:56:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:56:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:56:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:56:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:56:27.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:56:27.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:56:27.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:56:27.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:56:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:56:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:56:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:56:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:56:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:56:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:56:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:56:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:56:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:56:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:56:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:56:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:56:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:56:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:56:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:56:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:56:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:56:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:56:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:56:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:56:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:56:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:56:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:56:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:56:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:56:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:56:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:56:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:56:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:56:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:56:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:56:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:56:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:56:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:56:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1005] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:56:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1005] Local address: 172.21.15.161, port 60274 2022-07-03T19:56:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1005] Peer address: 172.21.15.101, port 22 2022-07-03T19:56:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1005] Connection lost 2022-07-03T19:56:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1005] Aborting connection 2022-07-03T19:56:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:56:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:56:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:27 smithi188 ceph-mon[92425]: pgmap v16718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:56:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:56:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:56:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:56:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:56:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:56:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:56:28.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:56:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:56:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:56:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:56:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:56:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:56:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:56:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:56:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:56:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:56:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:56:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:56:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:56:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:56:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:56:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:56:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:56:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:56:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:56:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:56:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:56:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:56:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:56:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:56:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:56:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:56:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:56:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:56:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:56:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:56:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:56:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:56:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:56:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:56:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:56:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:56:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:56:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:56:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:56:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:56:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:56:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:56:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1005] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:56:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1005] Local address: 172.21.15.161, port 60274 2022-07-03T19:56:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1005] Peer address: 172.21.15.101, port 22 2022-07-03T19:56:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1005] Connection lost 2022-07-03T19:56:28.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1005] Aborting connection 2022-07-03T19:56:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:56:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:56:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:56:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:56:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:56:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:56:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:56:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:56:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:56:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:56:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:56:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:56:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:56:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:56:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:56:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:56:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:56:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:56:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:56:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:56:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:56:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:56:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:56:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:56:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:56:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:56:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:56:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:56:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:56:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:56:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:56:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:56:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:56:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:56:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:56:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:56:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:56:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:56:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:56:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:56:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:56:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:56:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:56:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:56:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:56:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:56:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:56:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:56:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:56:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1005] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:56:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1005] Local address: 172.21.15.161, port 60274 2022-07-03T19:56:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1005] Peer address: 172.21.15.101, port 22 2022-07-03T19:56:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1005] Connection lost 2022-07-03T19:56:28.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1005] Aborting connection 2022-07-03T19:56:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:29 smithi161 ceph-mon[91579]: pgmap v16719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:29.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:29 smithi188 ceph-mon[92425]: pgmap v16719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:32 smithi161 ceph-mon[91579]: pgmap v16720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:32 smithi188 ceph-mon[92425]: pgmap v16720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:34 smithi161 ceph-mon[91579]: pgmap v16721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:34 smithi188 ceph-mon[92425]: pgmap v16721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:35 smithi161 ceph-mon[91579]: pgmap v16722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:35 smithi188 ceph-mon[92425]: pgmap v16722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:37 smithi161 ceph-mon[91579]: pgmap v16723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:37 smithi188 ceph-mon[92425]: pgmap v16723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:40 smithi161 ceph-mon[91579]: pgmap v16724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:40 smithi188 ceph-mon[92425]: pgmap v16724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:42.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:42 smithi161 ceph-mon[91579]: pgmap v16725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:42.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:42 smithi188 ceph-mon[92425]: pgmap v16725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:44 smithi161 ceph-mon[91579]: pgmap v16726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:44 smithi188 ceph-mon[92425]: pgmap v16726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:45 smithi161 ceph-mon[91579]: pgmap v16727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:45 smithi188 ceph-mon[92425]: pgmap v16727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:46.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:47 smithi161 ceph-mon[91579]: pgmap v16728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:47 smithi188 ceph-mon[92425]: pgmap v16728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:50 smithi161 ceph-mon[91579]: pgmap v16729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:50 smithi188 ceph-mon[92425]: pgmap v16729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:52 smithi161 ceph-mon[91579]: pgmap v16730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:52 smithi188 ceph-mon[92425]: pgmap v16730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:54 smithi161 ceph-mon[91579]: pgmap v16731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:54 smithi188 ceph-mon[92425]: pgmap v16731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:56:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:55 smithi161 ceph-mon[91579]: pgmap v16732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:55 smithi188 ceph-mon[92425]: pgmap v16732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:56:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:57 smithi161 ceph-mon[91579]: pgmap v16733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:57 smithi188 ceph-mon[92425]: pgmap v16733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:56:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:56:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:56:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:56:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:56:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:56:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:56:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:56:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:57:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:00 smithi161 ceph-mon[91579]: pgmap v16734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:00 smithi188 ceph-mon[92425]: pgmap v16734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:02 smithi161 ceph-mon[91579]: pgmap v16735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:02 smithi188 ceph-mon[92425]: pgmap v16735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:04 smithi161 ceph-mon[91579]: pgmap v16736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:04 smithi188 ceph-mon[92425]: pgmap v16736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:05 smithi161 ceph-mon[91579]: pgmap v16737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:05 smithi188 ceph-mon[92425]: pgmap v16737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:07 smithi161 ceph-mon[91579]: pgmap v16738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:07 smithi188 ceph-mon[92425]: pgmap v16738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:10 smithi161 ceph-mon[91579]: pgmap v16739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:10 smithi188 ceph-mon[92425]: pgmap v16739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:12 smithi161 ceph-mon[91579]: pgmap v16740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:12 smithi188 ceph-mon[92425]: pgmap v16740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:14 smithi161 ceph-mon[91579]: pgmap v16741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:14 smithi188 ceph-mon[92425]: pgmap v16741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:15 smithi161 ceph-mon[91579]: pgmap v16742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:15 smithi188 ceph-mon[92425]: pgmap v16742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:17 smithi161 ceph-mon[91579]: pgmap v16743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:17 smithi188 ceph-mon[92425]: pgmap v16743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:20 smithi161 ceph-mon[91579]: pgmap v16744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:20 smithi188 ceph-mon[92425]: pgmap v16744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:22 smithi161 ceph-mon[91579]: pgmap v16745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:22 smithi188 ceph-mon[92425]: pgmap v16745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:24 smithi161 ceph-mon[91579]: pgmap v16746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:24 smithi188 ceph-mon[92425]: pgmap v16746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:25 smithi161 ceph-mon[91579]: pgmap v16747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:25 smithi188 ceph-mon[92425]: pgmap v16747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:27 smithi161 ceph-mon[91579]: pgmap v16748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:27 smithi188 ceph-mon[92425]: pgmap v16748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:28.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:57:28.249+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:57:28.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:57:28.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:57:28.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:57:28.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:57:28.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:57:28.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:57:28.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:57:28.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:57:28.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:57:28.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:57:28.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:57:28.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:57:28.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:57:28.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:57:28.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:57:28.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:57:28.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:57:28.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:57:28.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:57:28.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:57:28.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:57:28.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:57:28.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:57:28.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:57:28.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:57:28.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:57:28.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:57:28.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:57:28.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:57:28.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:57:28.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:57:28.617 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:57:28.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:57:28.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:57:28.618 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:57:28.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:57:28.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:57:28.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:57:28.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:57:28.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:57:28.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:57:28.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:57:28.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:57:28.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:57:28.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1007] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:57:28.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1007] Local address: 172.21.15.161, port 60278 2022-07-03T19:57:28.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1007] Peer address: 172.21.15.101, port 22 2022-07-03T19:57:28.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1007] Connection lost 2022-07-03T19:57:28.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1007] Aborting connection 2022-07-03T19:57:28.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:57:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:57:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:57:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:57:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:57:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:57:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:57:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:57:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:57:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:57:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:57:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:57:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:57:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:57:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:57:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:57:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:57:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:57:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:57:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:57:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:57:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:57:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:57:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:57:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1007] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1007] Local address: 172.21.15.161, port 60278 2022-07-03T19:57:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1007] Peer address: 172.21.15.101, port 22 2022-07-03T19:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1007] Connection lost 2022-07-03T19:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1007] Aborting connection 2022-07-03T19:57:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:29 smithi161 ceph-mon[91579]: pgmap v16749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:29.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:57:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:57:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:57:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:57:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:57:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:57:29.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:57:29.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:57:29.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:57:29.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:57:29.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:57:29.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:57:29.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:57:29.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:57:29.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:57:29.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:57:29.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:57:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:57:29.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1007] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:57:29.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1007] Local address: 172.21.15.161, port 60278 2022-07-03T19:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1007] Peer address: 172.21.15.101, port 22 2022-07-03T19:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1007] Connection lost 2022-07-03T19:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1007] Aborting connection 2022-07-03T19:57:29.928 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:29 smithi188 ceph-mon[92425]: pgmap v16749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:32 smithi161 ceph-mon[91579]: pgmap v16750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:32 smithi188 ceph-mon[92425]: pgmap v16750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:34 smithi161 ceph-mon[91579]: pgmap v16751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:34 smithi188 ceph-mon[92425]: pgmap v16751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:35 smithi161 ceph-mon[91579]: pgmap v16752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:35 smithi188 ceph-mon[92425]: pgmap v16752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:37 smithi161 ceph-mon[91579]: pgmap v16753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:37 smithi188 ceph-mon[92425]: pgmap v16753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:40 smithi161 ceph-mon[91579]: pgmap v16754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:40 smithi188 ceph-mon[92425]: pgmap v16754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:42 smithi161 ceph-mon[91579]: pgmap v16755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:42 smithi188 ceph-mon[92425]: pgmap v16755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:44 smithi161 ceph-mon[91579]: pgmap v16756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:44 smithi188 ceph-mon[92425]: pgmap v16756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:45 smithi161 ceph-mon[91579]: pgmap v16757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:45 smithi188 ceph-mon[92425]: pgmap v16757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:47 smithi161 ceph-mon[91579]: pgmap v16758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:47 smithi188 ceph-mon[92425]: pgmap v16758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:50 smithi161 ceph-mon[91579]: pgmap v16759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:50 smithi188 ceph-mon[92425]: pgmap v16759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:52 smithi161 ceph-mon[91579]: pgmap v16760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:52 smithi188 ceph-mon[92425]: pgmap v16760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:54 smithi161 ceph-mon[91579]: pgmap v16761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:54 smithi188 ceph-mon[92425]: pgmap v16761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:57:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:55 smithi161 ceph-mon[91579]: pgmap v16762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:55 smithi188 ceph-mon[92425]: pgmap v16762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:57:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:57 smithi161 ceph-mon[91579]: pgmap v16763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:57 smithi188 ceph-mon[92425]: pgmap v16763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:57:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:57:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:57:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:57:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:57:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:57:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:57:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:57:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:58:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:00 smithi161 ceph-mon[91579]: pgmap v16764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:00 smithi188 ceph-mon[92425]: pgmap v16764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:02 smithi161 ceph-mon[91579]: pgmap v16765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:02 smithi188 ceph-mon[92425]: pgmap v16765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:04 smithi161 ceph-mon[91579]: pgmap v16766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:04 smithi188 ceph-mon[92425]: pgmap v16766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:05 smithi161 ceph-mon[91579]: pgmap v16767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:05 smithi188 ceph-mon[92425]: pgmap v16767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:07 smithi161 ceph-mon[91579]: pgmap v16768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:07 smithi188 ceph-mon[92425]: pgmap v16768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:10 smithi161 ceph-mon[91579]: pgmap v16769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:10 smithi188 ceph-mon[92425]: pgmap v16769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:12 smithi161 ceph-mon[91579]: pgmap v16770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:12 smithi188 ceph-mon[92425]: pgmap v16770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:14 smithi161 ceph-mon[91579]: pgmap v16771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:14 smithi188 ceph-mon[92425]: pgmap v16771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:15 smithi161 ceph-mon[91579]: pgmap v16772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:15 smithi188 ceph-mon[92425]: pgmap v16772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:16.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:17 smithi188 ceph-mon[92425]: pgmap v16773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:17 smithi161 ceph-mon[91579]: pgmap v16773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:20 smithi161 ceph-mon[91579]: pgmap v16774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:20 smithi188 ceph-mon[92425]: pgmap v16774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:22 smithi161 ceph-mon[91579]: pgmap v16775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:22 smithi188 ceph-mon[92425]: pgmap v16775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:24 smithi161 ceph-mon[91579]: pgmap v16776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:24 smithi188 ceph-mon[92425]: pgmap v16776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:25 smithi161 ceph-mon[91579]: pgmap v16777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:25 smithi188 ceph-mon[92425]: pgmap v16777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:27 smithi188 ceph-mon[92425]: pgmap v16778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:27 smithi161 ceph-mon[91579]: pgmap v16778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:58:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:58:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:58:29.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:58:28.747+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:58:29.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:58:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:58:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:58:29.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:58:29.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:58:29.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:58:29.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:58:29.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:58:29.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:58:29.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:58:29.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:58:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:58:29.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:58:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:58:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:58:29.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:58:29.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:58:29.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:58:29.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:58:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:58:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:58:29.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1009] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1009] Local address: 172.21.15.161, port 60282 2022-07-03T19:58:29.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1009] Peer address: 172.21.15.101, port 22 2022-07-03T19:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1009] Connection lost 2022-07-03T19:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1009] Aborting connection 2022-07-03T19:58:29.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:58:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:58:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:58:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:58:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:58:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:58:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:58:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:58:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:58:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:58:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:58:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:58:30.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:58:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:58:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:58:30.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:58:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:58:30.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:58:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:58:30.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:58:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:58:30.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:58:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:58:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:58:30.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:58:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:58:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:58:30.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:58:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:58:30.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:58:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:58:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:58:30.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:58:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:58:30.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:58:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:58:30.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:58:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:58:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:58:30.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:58:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:58:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:58:30.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:58:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:58:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:58:30.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:58:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:58:30.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:58:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:58:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:58:30.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:58:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1009] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:58:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1009] Local address: 172.21.15.161, port 60282 2022-07-03T19:58:30.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1009] Peer address: 172.21.15.101, port 22 2022-07-03T19:58:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1009] Connection lost 2022-07-03T19:58:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1009] Aborting connection 2022-07-03T19:58:30.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: pgmap v16779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:58:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:58:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:58:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:58:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:58:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:58:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:58:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:58:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:58:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:58:30.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:58:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:58:30.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:58:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:58:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:58:30.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:58:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:58:30.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:58:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:58:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:58:30.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:58:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:58:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:58:30.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:58:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:58:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:58:30.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:58:30.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:58:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:58:30.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:58:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:58:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:58:30.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:58:30.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:58:30.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:58:30.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:58:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:58:30.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1009] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1009] Local address: 172.21.15.161, port 60282 2022-07-03T19:58:30.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1009] Peer address: 172.21.15.101, port 22 2022-07-03T19:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1009] Connection lost 2022-07-03T19:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1009] Aborting connection 2022-07-03T19:58:30.180 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: pgmap v16779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:30.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T19:58:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:32 smithi161 ceph-mon[91579]: pgmap v16780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:32 smithi188 ceph-mon[92425]: pgmap v16780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:34 smithi161 ceph-mon[91579]: pgmap v16781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:34 smithi188 ceph-mon[92425]: pgmap v16781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:35 smithi161 ceph-mon[91579]: pgmap v16782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:35 smithi188 ceph-mon[92425]: pgmap v16782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:37 smithi188 ceph-mon[92425]: pgmap v16783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:37 smithi161 ceph-mon[91579]: pgmap v16783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:40 smithi161 ceph-mon[91579]: pgmap v16784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:40 smithi188 ceph-mon[92425]: pgmap v16784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:42 smithi161 ceph-mon[91579]: pgmap v16785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:42 smithi188 ceph-mon[92425]: pgmap v16785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:44 smithi161 ceph-mon[91579]: pgmap v16786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:44 smithi188 ceph-mon[92425]: pgmap v16786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:45 smithi161 ceph-mon[91579]: pgmap v16787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:45 smithi188 ceph-mon[92425]: pgmap v16787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:47 smithi188 ceph-mon[92425]: pgmap v16788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:47 smithi161 ceph-mon[91579]: pgmap v16788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:50 smithi161 ceph-mon[91579]: pgmap v16789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:50 smithi188 ceph-mon[92425]: pgmap v16789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:52 smithi161 ceph-mon[91579]: pgmap v16790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:52 smithi188 ceph-mon[92425]: pgmap v16790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:54 smithi161 ceph-mon[91579]: pgmap v16791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:54 smithi188 ceph-mon[92425]: pgmap v16791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:58:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:55 smithi161 ceph-mon[91579]: pgmap v16792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:55 smithi188 ceph-mon[92425]: pgmap v16792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:58:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:57 smithi188 ceph-mon[92425]: pgmap v16793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:57 smithi161 ceph-mon[91579]: pgmap v16793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:58:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:58:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:58:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:58:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:58:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:58:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:58:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:58:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:59:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:00 smithi161 ceph-mon[91579]: pgmap v16794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:00 smithi188 ceph-mon[92425]: pgmap v16794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:02 smithi161 ceph-mon[91579]: pgmap v16795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:02 smithi188 ceph-mon[92425]: pgmap v16795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:04 smithi161 ceph-mon[91579]: pgmap v16796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:04 smithi188 ceph-mon[92425]: pgmap v16796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:05 smithi161 ceph-mon[91579]: pgmap v16797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:05 smithi188 ceph-mon[92425]: pgmap v16797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:07 smithi188 ceph-mon[92425]: pgmap v16798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:07 smithi161 ceph-mon[91579]: pgmap v16798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:10 smithi161 ceph-mon[91579]: pgmap v16799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:10 smithi188 ceph-mon[92425]: pgmap v16799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:12 smithi161 ceph-mon[91579]: pgmap v16800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:12 smithi188 ceph-mon[92425]: pgmap v16800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:14 smithi161 ceph-mon[91579]: pgmap v16801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:14 smithi188 ceph-mon[92425]: pgmap v16801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:15 smithi161 ceph-mon[91579]: pgmap v16802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:15 smithi188 ceph-mon[92425]: pgmap v16802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:17 smithi188 ceph-mon[92425]: pgmap v16803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:17 smithi161 ceph-mon[91579]: pgmap v16803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:20 smithi161 ceph-mon[91579]: pgmap v16804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:20 smithi188 ceph-mon[92425]: pgmap v16804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:22 smithi161 ceph-mon[91579]: pgmap v16805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:22 smithi188 ceph-mon[92425]: pgmap v16805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:22.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:24 smithi161 ceph-mon[91579]: pgmap v16806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:24 smithi188 ceph-mon[92425]: pgmap v16806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:25 smithi161 ceph-mon[91579]: pgmap v16807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:25 smithi188 ceph-mon[92425]: pgmap v16807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:27 smithi188 ceph-mon[92425]: pgmap v16808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:27 smithi161 ceph-mon[91579]: pgmap v16808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:30.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:59:29.681+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:59:30.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:59:30.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:59:30.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T19:59:30.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:59:30.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:59:30.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:59:30.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T19:59:30.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:59:30.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T19:59:30.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:59:30.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T19:59:30.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:59:30.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:59:30.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T19:59:30.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:59:30.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T19:59:30.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:59:30.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T19:59:30.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:59:30.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T19:59:30.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:59:30.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:59:30.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:59:30.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:59:30.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:59:30.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T19:59:30.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:59:30.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:59:30.071 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:59:30.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T19:59:30.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:59:30.072 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T19:59:30.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:59:30.073 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:59:30.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:59:30.074 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T19:59:30.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:59:30.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T19:59:30.075 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:59:30.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T19:59:30.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:59:30.076 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T19:59:30.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:59:30.077 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:59:30.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1011] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:59:30.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1011] Local address: 172.21.15.161, port 60286 2022-07-03T19:59:30.078 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1011] Peer address: 172.21.15.101, port 22 2022-07-03T19:59:30.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1011] Connection lost 2022-07-03T19:59:30.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1011] Aborting connection 2022-07-03T19:59:30.079 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 19:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T19:59:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:30 smithi161 ceph-mon[91579]: pgmap v16809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:59:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:59:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:59:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:30 smithi188 ceph-mon[92425]: pgmap v16809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T19:59:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T19:59:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T19:59:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:59:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:59:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:59:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:59:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T19:59:31.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:59:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:59:31.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:59:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T19:59:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:59:31.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T19:59:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:59:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T19:59:31.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:59:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:59:31.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:59:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T19:59:31.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T19:59:31.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:59:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T19:59:31.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:59:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T19:59:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:59:31.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:59:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:59:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:59:31.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:59:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:59:31.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:59:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:59:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:59:31.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T19:59:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:59:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T19:59:31.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:59:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:59:31.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:59:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:59:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:59:31.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T19:59:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:59:31.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:59:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:59:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T19:59:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:59:31.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:59:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1011] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:59:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1011] Local address: 172.21.15.161, port 60286 2022-07-03T19:59:31.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1011] Peer address: 172.21.15.101, port 22 2022-07-03T19:59:31.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1011] Connection lost 2022-07-03T19:59:31.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1011] Aborting connection 2022-07-03T19:59:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T19:59:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T19:59:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:59:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T19:59:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T19:59:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T19:59:31.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T19:59:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T19:59:31.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T19:59:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T19:59:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T19:59:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T19:59:31.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T19:59:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T19:59:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:59:31.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T19:59:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T19:59:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T19:59:31.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T19:59:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T19:59:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T19:59:31.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T19:59:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T19:59:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T19:59:31.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T19:59:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T19:59:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T19:59:31.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T19:59:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T19:59:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T19:59:31.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T19:59:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T19:59:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T19:59:31.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T19:59:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T19:59:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T19:59:31.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T19:59:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T19:59:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T19:59:31.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T19:59:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T19:59:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T19:59:31.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T19:59:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T19:59:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T19:59:31.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T19:59:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1011] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T19:59:31.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1011] Local address: 172.21.15.161, port 60286 2022-07-03T19:59:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1011] Peer address: 172.21.15.101, port 22 2022-07-03T19:59:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1011] Connection lost 2022-07-03T19:59:31.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1011] Aborting connection 2022-07-03T19:59:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:32 smithi161 ceph-mon[91579]: pgmap v16810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:32 smithi188 ceph-mon[92425]: pgmap v16810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:34 smithi161 ceph-mon[91579]: pgmap v16811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:34 smithi188 ceph-mon[92425]: pgmap v16811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:35 smithi161 ceph-mon[91579]: pgmap v16812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:35 smithi188 ceph-mon[92425]: pgmap v16812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:37 smithi161 ceph-mon[91579]: pgmap v16813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:37 smithi188 ceph-mon[92425]: pgmap v16813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:40 smithi161 ceph-mon[91579]: pgmap v16814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:40 smithi188 ceph-mon[92425]: pgmap v16814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:42 smithi161 ceph-mon[91579]: pgmap v16815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:42 smithi188 ceph-mon[92425]: pgmap v16815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:44 smithi161 ceph-mon[91579]: pgmap v16816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:44 smithi188 ceph-mon[92425]: pgmap v16816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:45 smithi161 ceph-mon[91579]: pgmap v16817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:45 smithi188 ceph-mon[92425]: pgmap v16817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:47 smithi161 ceph-mon[91579]: pgmap v16818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:47 smithi188 ceph-mon[92425]: pgmap v16818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:50 smithi161 ceph-mon[91579]: pgmap v16819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:50 smithi188 ceph-mon[92425]: pgmap v16819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:52 smithi161 ceph-mon[91579]: pgmap v16820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:52 smithi188 ceph-mon[92425]: pgmap v16820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:54 smithi161 ceph-mon[91579]: pgmap v16821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:54 smithi188 ceph-mon[92425]: pgmap v16821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T19:59:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:55 smithi161 ceph-mon[91579]: pgmap v16822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:55 smithi188 ceph-mon[92425]: pgmap v16822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T19:59:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:57 smithi161 ceph-mon[91579]: pgmap v16823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:57 smithi188 ceph-mon[92425]: pgmap v16823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T19:59:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:59:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:59:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:59:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 19:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:59:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:59:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T19:59:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T19:59:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 19:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:00:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:00 smithi161 ceph-mon[91579]: pgmap v16824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:00:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:00 smithi188 ceph-mon[92425]: pgmap v16824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:00:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:02 smithi161 ceph-mon[91579]: pgmap v16825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:02 smithi188 ceph-mon[92425]: pgmap v16825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:04 smithi161 ceph-mon[91579]: pgmap v16826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:04 smithi188 ceph-mon[92425]: pgmap v16826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:05 smithi161 ceph-mon[91579]: pgmap v16827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:05 smithi188 ceph-mon[92425]: pgmap v16827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:07 smithi161 ceph-mon[91579]: pgmap v16828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:07 smithi188 ceph-mon[92425]: pgmap v16828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:10 smithi161 ceph-mon[91579]: pgmap v16829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:10 smithi188 ceph-mon[92425]: pgmap v16829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:12 smithi161 ceph-mon[91579]: pgmap v16830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:12 smithi188 ceph-mon[92425]: pgmap v16830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:14 smithi161 ceph-mon[91579]: pgmap v16831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:14 smithi188 ceph-mon[92425]: pgmap v16831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:15 smithi161 ceph-mon[91579]: pgmap v16832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:15 smithi188 ceph-mon[92425]: pgmap v16832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:17 smithi161 ceph-mon[91579]: pgmap v16833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:17 smithi188 ceph-mon[92425]: pgmap v16833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:20 smithi161 ceph-mon[91579]: pgmap v16834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:20 smithi188 ceph-mon[92425]: pgmap v16834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:22 smithi161 ceph-mon[91579]: pgmap v16835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:22 smithi188 ceph-mon[92425]: pgmap v16835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:24 smithi161 ceph-mon[91579]: pgmap v16836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:24 smithi188 ceph-mon[92425]: pgmap v16836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:25 smithi161 ceph-mon[91579]: pgmap v16837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:25 smithi188 ceph-mon[92425]: pgmap v16837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:27 smithi161 ceph-mon[91579]: pgmap v16838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:27 smithi188 ceph-mon[92425]: pgmap v16838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-mon[91579]: pgmap v16839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:30.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:00:30.179+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:00:30.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:00:30.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:00:30.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:00:30.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:00:30.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:00:30.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:00:30.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:00:30.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:00:30.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:00:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:00:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:00:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:00:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:00:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:00:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:00:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:00:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:00:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:00:30.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:00:30.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:00:30.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:00:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:00:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:00:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:00:30.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:00:30.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:00:30.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:00:30.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:00:30.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:00:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:00:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:00:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:00:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:00:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:00:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:00:30.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:00:30.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:00:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:00:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:00:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:00:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:00:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:00:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:00:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:00:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1013] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:00:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1013] Local address: 172.21.15.161, port 60290 2022-07-03T20:00:30.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1013] Peer address: 172.21.15.101, port 22 2022-07-03T20:00:30.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1013] Connection lost 2022-07-03T20:00:30.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1013] Aborting connection 2022-07-03T20:00:30.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:00:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:30 smithi188 ceph-mon[92425]: pgmap v16839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:00:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:00:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:00:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:00:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:00:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:00:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:00:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:00:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:00:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:00:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:00:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:00:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:00:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:00:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:00:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:00:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:00:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:00:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:00:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:00:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:00:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:00:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:00:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:00:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:00:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:00:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:00:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:00:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:00:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:00:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:00:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:00:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:00:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:00:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:00:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:00:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:00:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1013] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1013] Local address: 172.21.15.161, port 60290 2022-07-03T20:00:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1013] Peer address: 172.21.15.101, port 22 2022-07-03T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1013] Connection lost 2022-07-03T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1013] Aborting connection 2022-07-03T20:00:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:32 smithi161 ceph-mon[91579]: pgmap v16840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:00:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:00:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:00:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:00:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:00:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:00:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:00:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:00:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:00:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:00:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:00:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:00:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:00:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:00:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:00:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:00:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:00:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:00:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:00:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:00:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:00:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:00:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:00:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:00:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:00:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:00:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1013] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:00:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1013] Local address: 172.21.15.161, port 60290 2022-07-03T20:00:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1013] Peer address: 172.21.15.101, port 22 2022-07-03T20:00:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1013] Connection lost 2022-07-03T20:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1013] Aborting connection 2022-07-03T20:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:32 smithi188 ceph-mon[92425]: pgmap v16840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:34 smithi161 ceph-mon[91579]: pgmap v16841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:34 smithi188 ceph-mon[92425]: pgmap v16841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:35 smithi161 ceph-mon[91579]: pgmap v16842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:35 smithi188 ceph-mon[92425]: pgmap v16842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:37 smithi161 ceph-mon[91579]: pgmap v16843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:37 smithi188 ceph-mon[92425]: pgmap v16843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:40 smithi161 ceph-mon[91579]: pgmap v16844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:40 smithi188 ceph-mon[92425]: pgmap v16844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:42 smithi161 ceph-mon[91579]: pgmap v16845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:42 smithi188 ceph-mon[92425]: pgmap v16845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:44 smithi161 ceph-mon[91579]: pgmap v16846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:44 smithi188 ceph-mon[92425]: pgmap v16846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:45 smithi161 ceph-mon[91579]: pgmap v16847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:45 smithi188 ceph-mon[92425]: pgmap v16847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:47 smithi161 ceph-mon[91579]: pgmap v16848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:47 smithi188 ceph-mon[92425]: pgmap v16848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:50 smithi161 ceph-mon[91579]: pgmap v16849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:50 smithi188 ceph-mon[92425]: pgmap v16849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:52 smithi161 ceph-mon[91579]: pgmap v16850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:52 smithi188 ceph-mon[92425]: pgmap v16850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:54 smithi161 ceph-mon[91579]: pgmap v16851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:54 smithi188 ceph-mon[92425]: pgmap v16851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:00:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:55 smithi161 ceph-mon[91579]: pgmap v16852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:55 smithi188 ceph-mon[92425]: pgmap v16852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:00:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:57 smithi161 ceph-mon[91579]: pgmap v16853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:57 smithi188 ceph-mon[92425]: pgmap v16853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:00:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:00:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:00:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:00:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:00:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:00:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:00:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:00:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:01:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:00 smithi161 ceph-mon[91579]: pgmap v16854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:00 smithi188 ceph-mon[92425]: pgmap v16854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:02 smithi161 ceph-mon[91579]: pgmap v16855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:02 smithi188 ceph-mon[92425]: pgmap v16855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:04 smithi161 ceph-mon[91579]: pgmap v16856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:04 smithi188 ceph-mon[92425]: pgmap v16856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:05 smithi161 ceph-mon[91579]: pgmap v16857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:05 smithi188 ceph-mon[92425]: pgmap v16857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:07 smithi161 ceph-mon[91579]: pgmap v16858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:07 smithi188 ceph-mon[92425]: pgmap v16858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:10 smithi161 ceph-mon[91579]: pgmap v16859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:10 smithi188 ceph-mon[92425]: pgmap v16859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:12 smithi161 ceph-mon[91579]: pgmap v16860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:12 smithi188 ceph-mon[92425]: pgmap v16860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:14 smithi161 ceph-mon[91579]: pgmap v16861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:14 smithi188 ceph-mon[92425]: pgmap v16861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:15 smithi161 ceph-mon[91579]: pgmap v16862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:15 smithi188 ceph-mon[92425]: pgmap v16862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:17 smithi161 ceph-mon[91579]: pgmap v16863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:17 smithi188 ceph-mon[92425]: pgmap v16863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:20 smithi161 ceph-mon[91579]: pgmap v16864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:20 smithi188 ceph-mon[92425]: pgmap v16864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:22 smithi161 ceph-mon[91579]: pgmap v16865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:22 smithi188 ceph-mon[92425]: pgmap v16865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:24 smithi161 ceph-mon[91579]: pgmap v16866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:24 smithi188 ceph-mon[92425]: pgmap v16866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:25 smithi161 ceph-mon[91579]: pgmap v16867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:25 smithi188 ceph-mon[92425]: pgmap v16867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:27 smithi161 ceph-mon[91579]: pgmap v16868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:27 smithi188 ceph-mon[92425]: pgmap v16868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-mon[91579]: pgmap v16869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:30 smithi188 ceph-mon[92425]: pgmap v16869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:31.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:01:30.688+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:01:31.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:01:31.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:01:31.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:01:31.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:01:31.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:01:31.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:01:31.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:01:31.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:01:31.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:01:31.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:01:31.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:01:31.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:01:31.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:01:31.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:01:31.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:01:31.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:01:31.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:01:31.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:01:31.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:01:31.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:01:31.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:01:31.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:01:31.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:01:31.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:01:31.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:01:31.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:01:31.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:01:31.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:01:31.113 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:01:31.113 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:01:31.113 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:01:31.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:01:31.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:01:31.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:01:31.115 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:01:31.115 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:01:31.115 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:01:31.116 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:01:31.116 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:01:31.116 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:01:31.117 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:01:31.117 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:01:31.117 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:01:31.118 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:01:31.119 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:01:31.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:01:31.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:01:31.120 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1015] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:01:31.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1015] Local address: 172.21.15.161, port 60294 2022-07-03T20:01:31.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1015] Peer address: 172.21.15.101, port 22 2022-07-03T20:01:31.121 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1015] Connection lost 2022-07-03T20:01:31.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1015] Aborting connection 2022-07-03T20:01:31.122 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:01:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:01:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:01:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:01:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:01:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:01:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:01:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:01:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:01:32.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:01:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:01:32.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:01:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:01:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:01:32.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:01:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:01:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:01:32.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:01:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:01:32.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:01:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:01:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:01:32.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:01:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:01:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:01:32.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:01:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:01:32.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:01:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:01:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:01:32.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:01:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:01:32.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:01:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:01:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:01:32.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:01:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:01:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:01:32.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:01:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:01:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:01:32.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:01:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:01:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:01:32.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:01:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:01:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:01:32.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:01:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:01:32.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:01:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:01:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1015] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:01:32.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1015] Local address: 172.21.15.161, port 60294 2022-07-03T20:01:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1015] Peer address: 172.21.15.101, port 22 2022-07-03T20:01:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1015] Connection lost 2022-07-03T20:01:32.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1015] Aborting connection 2022-07-03T20:01:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:32 smithi161 ceph-mon[91579]: pgmap v16870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:32.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:01:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:01:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:01:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:01:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:01:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:01:32.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:01:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:01:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:01:32.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:01:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:01:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:01:32.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:01:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:01:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:01:32.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:01:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:01:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:01:32.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:01:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:01:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:01:32.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:01:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:01:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:01:32.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:01:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:01:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:01:32.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:01:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:01:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:01:32.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:01:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:01:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:01:32.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:01:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:01:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:01:32.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:01:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:01:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:01:32.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:01:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:01:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:01:32.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:01:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:01:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:01:32.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:01:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1015] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:01:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1015] Local address: 172.21.15.161, port 60294 2022-07-03T20:01:32.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1015] Peer address: 172.21.15.101, port 22 2022-07-03T20:01:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1015] Connection lost 2022-07-03T20:01:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1015] Aborting connection 2022-07-03T20:01:32.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:32 smithi188 ceph-mon[92425]: pgmap v16870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:01:32.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: pgmap v16871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.385 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.387 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:34 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: pgmap v16871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:01:34.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:34.421 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:34 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:01:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:35 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 88996k 2022-07-03T20:01:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:35 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 91132633: error parsing value: Value '91132633' is below minimum 939524096 2022-07-03T20:01:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:35 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 88996k 2022-07-03T20:01:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:35 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 91132633: error parsing value: Value '91132633' is below minimum 939524096 2022-07-03T20:01:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:36 smithi161 ceph-mon[91579]: pgmap v16872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:36 smithi188 ceph-mon[92425]: pgmap v16872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:38 smithi161 ceph-mon[91579]: pgmap v16873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:38 smithi188 ceph-mon[92425]: pgmap v16873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:40 smithi161 ceph-mon[91579]: pgmap v16874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:40 smithi188 ceph-mon[92425]: pgmap v16874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:42 smithi161 ceph-mon[91579]: pgmap v16875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:42.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:42 smithi188 ceph-mon[92425]: pgmap v16875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:43.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:43 smithi188 ceph-mon[92425]: pgmap v16876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:43 smithi161 ceph-mon[91579]: pgmap v16876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:45 smithi161 ceph-mon[91579]: pgmap v16877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:45 smithi188 ceph-mon[92425]: pgmap v16877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:47 smithi161 ceph-mon[91579]: pgmap v16878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:47 smithi188 ceph-mon[92425]: pgmap v16878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:50 smithi161 ceph-mon[91579]: pgmap v16879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:50 smithi188 ceph-mon[92425]: pgmap v16879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:52 smithi161 ceph-mon[91579]: pgmap v16880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:52.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:52 smithi188 ceph-mon[92425]: pgmap v16880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:52.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:54 smithi161 ceph-mon[91579]: pgmap v16881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:54 smithi188 ceph-mon[92425]: pgmap v16881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:01:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:55 smithi161 ceph-mon[91579]: pgmap v16882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:55 smithi188 ceph-mon[92425]: pgmap v16882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:01:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:57 smithi161 ceph-mon[91579]: pgmap v16883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:57 smithi188 ceph-mon[92425]: pgmap v16883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:01:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:01:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:01:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:01:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:01:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:01:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:01:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:01:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:02:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:00 smithi161 ceph-mon[91579]: pgmap v16884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:00 smithi188 ceph-mon[92425]: pgmap v16884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:02 smithi161 ceph-mon[91579]: pgmap v16885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:02 smithi188 ceph-mon[92425]: pgmap v16885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:04 smithi161 ceph-mon[91579]: pgmap v16886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:04 smithi188 ceph-mon[92425]: pgmap v16886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:05 smithi161 ceph-mon[91579]: pgmap v16887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:05 smithi188 ceph-mon[92425]: pgmap v16887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:07 smithi161 ceph-mon[91579]: pgmap v16888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:07 smithi188 ceph-mon[92425]: pgmap v16888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:10 smithi161 ceph-mon[91579]: pgmap v16889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:10 smithi188 ceph-mon[92425]: pgmap v16889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:12 smithi161 ceph-mon[91579]: pgmap v16890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:12 smithi188 ceph-mon[92425]: pgmap v16890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:14 smithi161 ceph-mon[91579]: pgmap v16891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:14 smithi188 ceph-mon[92425]: pgmap v16891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:15 smithi161 ceph-mon[91579]: pgmap v16892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:15 smithi188 ceph-mon[92425]: pgmap v16892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:17 smithi161 ceph-mon[91579]: pgmap v16893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:17 smithi188 ceph-mon[92425]: pgmap v16893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:20 smithi161 ceph-mon[91579]: pgmap v16894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:20 smithi188 ceph-mon[92425]: pgmap v16894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:22 smithi161 ceph-mon[91579]: pgmap v16895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:22 smithi188 ceph-mon[92425]: pgmap v16895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:24 smithi161 ceph-mon[91579]: pgmap v16896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:24 smithi188 ceph-mon[92425]: pgmap v16896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:25 smithi161 ceph-mon[91579]: pgmap v16897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:25 smithi188 ceph-mon[92425]: pgmap v16897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:27 smithi161 ceph-mon[91579]: pgmap v16898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:27 smithi188 ceph-mon[92425]: pgmap v16898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:30 smithi161 ceph-mon[91579]: pgmap v16899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:30 smithi188 ceph-mon[92425]: pgmap v16899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:32 smithi161 ceph-mon[91579]: pgmap v16900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:32.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:32 smithi188 ceph-mon[92425]: pgmap v16900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:34 smithi161 ceph-mon[91579]: pgmap v16901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:02:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:02:34.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:02:34.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:02:33.949+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:02:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:02:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:02:34.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:02:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:02:34.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:02:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:02:34.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:02:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:02:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:02:34.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:02:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:02:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:02:34.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:02:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:02:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:02:34.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:02:34.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:02:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:02:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:02:34.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:02:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:02:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:02:34.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:02:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:02:34.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:02:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:02:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:02:34.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:02:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:02:34.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:02:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:02:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:02:34.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:02:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:02:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:02:34.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:02:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:02:34.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:02:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:02:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:02:34.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:02:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:02:34.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:02:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:02:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1017] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:02:34.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1017] Local address: 172.21.15.161, port 60298 2022-07-03T20:02:34.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1017] Peer address: 172.21.15.101, port 22 2022-07-03T20:02:34.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1017] Connection lost 2022-07-03T20:02:34.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1017] Aborting connection 2022-07-03T20:02:34.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:02:33 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:02:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:34 smithi188 ceph-mon[92425]: pgmap v16901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:02:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:02:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:02:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:02:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:02:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:02:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:02:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:02:35.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:02:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:02:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:02:35.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:02:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:02:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:02:35.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:02:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:02:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:02:35.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:02:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:02:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:02:35.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:02:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:02:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:02:35.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:02:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:02:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:02:35.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:02:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:02:35.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:02:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:02:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:02:35.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:02:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:02:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:02:35.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:02:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:02:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:02:35.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:02:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:02:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:02:35.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:02:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:02:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:02:35.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:02:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:02:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:02:35.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:02:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:02:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:02:35.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1017] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:02:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1017] Local address: 172.21.15.161, port 60298 2022-07-03T20:02:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1017] Peer address: 172.21.15.101, port 22 2022-07-03T20:02:35.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1017] Connection lost 2022-07-03T20:02:35.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1017] Aborting connection 2022-07-03T20:02:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:02:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:02:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:02:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:02:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:02:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:02:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:02:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:02:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:02:35.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:02:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:02:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:02:35.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:02:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:02:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:02:35.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:02:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:02:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:02:35.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:02:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:02:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:02:35.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:02:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:02:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:02:35.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:02:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:02:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:02:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:02:35.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:02:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:02:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:02:35.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:02:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:02:35.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:02:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:02:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:02:35.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:02:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:02:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:02:35.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:02:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:02:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:02:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:02:35.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:02:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:02:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:02:35.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1017] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:02:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1017] Local address: 172.21.15.161, port 60298 2022-07-03T20:02:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1017] Peer address: 172.21.15.101, port 22 2022-07-03T20:02:35.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1017] Connection lost 2022-07-03T20:02:35.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1017] Aborting connection 2022-07-03T20:02:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:36 smithi161 ceph-mon[91579]: pgmap v16902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:36 smithi188 ceph-mon[92425]: pgmap v16902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:38 smithi188 ceph-mon[92425]: pgmap v16903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:38 smithi161 ceph-mon[91579]: pgmap v16903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:40 smithi188 ceph-mon[92425]: pgmap v16904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:40 smithi161 ceph-mon[91579]: pgmap v16904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:41.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:41 smithi188 ceph-mon[92425]: pgmap v16905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:41 smithi161 ceph-mon[91579]: pgmap v16905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:43.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:43 smithi188 ceph-mon[92425]: pgmap v16906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:43 smithi161 ceph-mon[91579]: pgmap v16906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:45 smithi161 ceph-mon[91579]: pgmap v16907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:45 smithi188 ceph-mon[92425]: pgmap v16907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:47 smithi161 ceph-mon[91579]: pgmap v16908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:47 smithi188 ceph-mon[92425]: pgmap v16908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:50 smithi161 ceph-mon[91579]: pgmap v16909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:50 smithi188 ceph-mon[92425]: pgmap v16909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:52 smithi161 ceph-mon[91579]: pgmap v16910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:52 smithi188 ceph-mon[92425]: pgmap v16910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:54 smithi161 ceph-mon[91579]: pgmap v16911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:54 smithi188 ceph-mon[92425]: pgmap v16911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:02:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:55 smithi161 ceph-mon[91579]: pgmap v16912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:55 smithi188 ceph-mon[92425]: pgmap v16912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:02:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:57 smithi161 ceph-mon[91579]: pgmap v16913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:57 smithi188 ceph-mon[92425]: pgmap v16913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:02:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:02:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:02:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:02:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:02:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:02:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:02:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:02:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:03:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:00 smithi161 ceph-mon[91579]: pgmap v16914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:00 smithi188 ceph-mon[92425]: pgmap v16914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:02 smithi161 ceph-mon[91579]: pgmap v16915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:02 smithi188 ceph-mon[92425]: pgmap v16915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:04 smithi161 ceph-mon[91579]: pgmap v16916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:04 smithi188 ceph-mon[92425]: pgmap v16916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:05 smithi161 ceph-mon[91579]: pgmap v16917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:05 smithi188 ceph-mon[92425]: pgmap v16917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:07 smithi161 ceph-mon[91579]: pgmap v16918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:07 smithi188 ceph-mon[92425]: pgmap v16918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:10 smithi161 ceph-mon[91579]: pgmap v16919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:10 smithi188 ceph-mon[92425]: pgmap v16919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:12 smithi188 ceph-mon[92425]: pgmap v16920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:12 smithi161 ceph-mon[91579]: pgmap v16920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:14 smithi188 ceph-mon[92425]: pgmap v16921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:14 smithi161 ceph-mon[91579]: pgmap v16921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:15 smithi161 ceph-mon[91579]: pgmap v16922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:15 smithi188 ceph-mon[92425]: pgmap v16922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:17 smithi161 ceph-mon[91579]: pgmap v16923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:17 smithi188 ceph-mon[92425]: pgmap v16923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:20 smithi188 ceph-mon[92425]: pgmap v16924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:20 smithi161 ceph-mon[91579]: pgmap v16924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:22 smithi188 ceph-mon[92425]: pgmap v16925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:22 smithi161 ceph-mon[91579]: pgmap v16925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:24.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:24 smithi188 ceph-mon[92425]: pgmap v16926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:24 smithi161 ceph-mon[91579]: pgmap v16926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:25 smithi161 ceph-mon[91579]: pgmap v16927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:25 smithi188 ceph-mon[92425]: pgmap v16927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:27 smithi161 ceph-mon[91579]: pgmap v16928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:27 smithi188 ceph-mon[92425]: pgmap v16928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:30 smithi188 ceph-mon[92425]: pgmap v16929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:30 smithi161 ceph-mon[91579]: pgmap v16929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:32 smithi188 ceph-mon[92425]: pgmap v16930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:32.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:32 smithi161 ceph-mon[91579]: pgmap v16930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:34 smithi188 ceph-mon[92425]: pgmap v16931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:34.455 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-mon[91579]: pgmap v16931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:34.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:03:34.455+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:03:34.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:03:34.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:03:34.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:03:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:03:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:03:34.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:03:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:03:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:03:34.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:03:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:03:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:03:34.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:03:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:03:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:03:34.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:03:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:03:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:03:34.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:03:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:03:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:03:34.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:03:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:03:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:03:34.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:03:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:03:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:03:34.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:03:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:03:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:03:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:03:34.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:03:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:03:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:03:34.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:03:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:03:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:03:34.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:03:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:03:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:03:34.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:03:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:03:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:03:34.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:03:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:03:34.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1019] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:03:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1019] Local address: 172.21.15.161, port 60302 2022-07-03T20:03:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1019] Peer address: 172.21.15.101, port 22 2022-07-03T20:03:34.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1019] Connection lost 2022-07-03T20:03:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1019] Aborting connection 2022-07-03T20:03:34.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:03:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:03:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:03:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:03:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:03:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:03:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:03:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:03:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:03:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:03:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:03:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:03:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:03:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:03:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:03:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:03:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:03:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:03:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:03:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:03:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:03:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:03:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:03:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:03:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:03:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:03:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:03:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:03:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:03:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:03:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:03:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:03:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:03:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:03:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:03:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:03:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:03:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:03:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:03:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:03:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:03:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:03:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:03:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:03:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:03:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1019] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:03:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1019] Local address: 172.21.15.161, port 60302 2022-07-03T20:03:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1019] Peer address: 172.21.15.101, port 22 2022-07-03T20:03:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1019] Connection lost 2022-07-03T20:03:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1019] Aborting connection 2022-07-03T20:03:35.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:35 smithi161 ceph-mon[91579]: pgmap v16932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:03:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:03:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:03:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:03:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:03:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:03:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:03:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:03:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:03:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:03:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:03:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:03:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:03:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:03:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:03:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:03:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:03:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:03:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:03:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:03:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:03:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:03:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:03:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:03:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:03:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:03:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:03:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:03:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:03:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:03:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:03:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:03:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:03:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:03:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:03:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:03:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:03:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:03:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:03:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:03:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:03:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:03:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:03:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1019] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:03:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1019] Local address: 172.21.15.161, port 60302 2022-07-03T20:03:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1019] Peer address: 172.21.15.101, port 22 2022-07-03T20:03:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1019] Connection lost 2022-07-03T20:03:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1019] Aborting connection 2022-07-03T20:03:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:35 smithi188 ceph-mon[92425]: pgmap v16932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:37 smithi161 ceph-mon[91579]: pgmap v16933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:37 smithi188 ceph-mon[92425]: pgmap v16933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:40 smithi188 ceph-mon[92425]: pgmap v16934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:40 smithi161 ceph-mon[91579]: pgmap v16934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:42 smithi188 ceph-mon[92425]: pgmap v16935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:42.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:42 smithi161 ceph-mon[91579]: pgmap v16935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:42.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:44 smithi188 ceph-mon[92425]: pgmap v16936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:44 smithi161 ceph-mon[91579]: pgmap v16936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:45 smithi161 ceph-mon[91579]: pgmap v16937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:45 smithi188 ceph-mon[92425]: pgmap v16937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:47 smithi161 ceph-mon[91579]: pgmap v16938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:47 smithi188 ceph-mon[92425]: pgmap v16938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:50 smithi188 ceph-mon[92425]: pgmap v16939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:50 smithi161 ceph-mon[91579]: pgmap v16939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:52 smithi188 ceph-mon[92425]: pgmap v16940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:52 smithi161 ceph-mon[91579]: pgmap v16940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:52.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:54 smithi188 ceph-mon[92425]: pgmap v16941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:54 smithi161 ceph-mon[91579]: pgmap v16941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:03:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:55 smithi161 ceph-mon[91579]: pgmap v16942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:55 smithi188 ceph-mon[92425]: pgmap v16942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:03:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:57 smithi161 ceph-mon[91579]: pgmap v16943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:57 smithi188 ceph-mon[92425]: pgmap v16943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:03:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:03:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:03:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:03:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:03:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:03:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:03:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:03:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:04:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:00 smithi161 ceph-mon[91579]: pgmap v16944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:00 smithi188 ceph-mon[92425]: pgmap v16944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:02 smithi188 ceph-mon[92425]: pgmap v16945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:02 smithi161 ceph-mon[91579]: pgmap v16945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:04 smithi188 ceph-mon[92425]: pgmap v16946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:04 smithi161 ceph-mon[91579]: pgmap v16946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:05 smithi161 ceph-mon[91579]: pgmap v16947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:05 smithi188 ceph-mon[92425]: pgmap v16947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:07 smithi161 ceph-mon[91579]: pgmap v16948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:07 smithi188 ceph-mon[92425]: pgmap v16948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:10 smithi188 ceph-mon[92425]: pgmap v16949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:10 smithi161 ceph-mon[91579]: pgmap v16949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:12 smithi188 ceph-mon[92425]: pgmap v16950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:12 smithi161 ceph-mon[91579]: pgmap v16950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:12.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:13 smithi188 ceph-mon[92425]: pgmap v16951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:13 smithi161 ceph-mon[91579]: pgmap v16951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:15 smithi161 ceph-mon[91579]: pgmap v16952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:15 smithi188 ceph-mon[92425]: pgmap v16952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:17 smithi161 ceph-mon[91579]: pgmap v16953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:17 smithi188 ceph-mon[92425]: pgmap v16953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:20 smithi188 ceph-mon[92425]: pgmap v16954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:20 smithi161 ceph-mon[91579]: pgmap v16954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:21 smithi161 ceph-mon[91579]: pgmap v16955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:21 smithi188 ceph-mon[92425]: pgmap v16955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:23 smithi161 ceph-mon[91579]: pgmap v16956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:23 smithi188 ceph-mon[92425]: pgmap v16956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:25 smithi161 ceph-mon[91579]: pgmap v16957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:25 smithi188 ceph-mon[92425]: pgmap v16957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:27 smithi161 ceph-mon[91579]: pgmap v16958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:27 smithi188 ceph-mon[92425]: pgmap v16958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:30 smithi188 ceph-mon[92425]: pgmap v16959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:30 smithi161 ceph-mon[91579]: pgmap v16959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:31 smithi188 ceph-mon[92425]: pgmap v16960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:31 smithi161 ceph-mon[91579]: pgmap v16960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:33 smithi161 ceph-mon[91579]: pgmap v16961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:33 smithi188 ceph-mon[92425]: pgmap v16961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:35.354 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:04:34.954+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:04:35.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:04:35.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:04:35.355 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:04:35.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:04:35.356 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:04:35.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:04:35.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:04:35.357 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:04:35.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:04:35.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:04:35.358 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:04:35.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:04:35.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:04:35.359 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:04:35.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:04:35.360 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:04:35.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:04:35.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:04:35.361 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:04:35.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:04:35.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:04:35.362 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:04:35.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:04:35.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:04:35.363 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:04:35.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:04:35.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:04:35.364 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:04:35.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:04:35.365 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:04:35.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:04:35.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:04:35.366 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:04:35.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:04:35.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:04:35.367 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:04:35.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:04:35.368 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:04:35.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:04:35.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:04:35.369 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:04:35.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:04:35.370 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:04:35.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:04:35.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1021] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:04:35.371 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1021] Local address: 172.21.15.161, port 60306 2022-07-03T20:04:35.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1021] Peer address: 172.21.15.101, port 22 2022-07-03T20:04:35.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1021] Connection lost 2022-07-03T20:04:35.372 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1021] Aborting connection 2022-07-03T20:04:35.373 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:04:34 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:04:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:04:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:04:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:04:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:04:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:04:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:04:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:04:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:04:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:04:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:04:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:04:35.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:04:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:04:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:04:35.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:04:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:04:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:04:35.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:04:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:04:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:04:35.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:04:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:04:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:04:35.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:04:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:04:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:04:35.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:04:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:04:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:04:35.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:04:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:04:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:04:35.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:04:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:04:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:04:35.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:04:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:04:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:04:35.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:04:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:04:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:04:35.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:04:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:04:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:04:35.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:04:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:04:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:04:35.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:04:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:04:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1021] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:04:35.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1021] Local address: 172.21.15.161, port 60306 2022-07-03T20:04:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1021] Peer address: 172.21.15.101, port 22 2022-07-03T20:04:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1021] Connection lost 2022-07-03T20:04:35.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1021] Aborting connection 2022-07-03T20:04:35.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:35 smithi161 ceph-mon[91579]: pgmap v16962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:04:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:04:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:04:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:04:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:04:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:04:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:04:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:04:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:04:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:04:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:04:35.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:04:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:04:35.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:04:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:04:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:04:35.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:04:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:04:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:04:35.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:04:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:04:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:04:35.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:04:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:04:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:04:35.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:04:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:04:35.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:04:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:04:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:04:35.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:04:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:04:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:04:35.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:04:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:04:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:04:35.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:04:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:04:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:04:35.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:04:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:04:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:04:35.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:04:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:04:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:04:35.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:04:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:04:35.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:04:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:04:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1021] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:04:35.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1021] Local address: 172.21.15.161, port 60306 2022-07-03T20:04:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1021] Peer address: 172.21.15.101, port 22 2022-07-03T20:04:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1021] Connection lost 2022-07-03T20:04:35.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1021] Aborting connection 2022-07-03T20:04:35.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:35 smithi188 ceph-mon[92425]: pgmap v16962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:37 smithi161 ceph-mon[91579]: pgmap v16963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:37 smithi188 ceph-mon[92425]: pgmap v16963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:39 smithi161 ceph-mon[91579]: pgmap v16964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:39 smithi188 ceph-mon[92425]: pgmap v16964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:41 smithi161 ceph-mon[91579]: pgmap v16965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:41 smithi188 ceph-mon[92425]: pgmap v16965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:43 smithi161 ceph-mon[91579]: pgmap v16966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:43 smithi188 ceph-mon[92425]: pgmap v16966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:45 smithi161 ceph-mon[91579]: pgmap v16967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:45 smithi188 ceph-mon[92425]: pgmap v16967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:47 smithi161 ceph-mon[91579]: pgmap v16968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:47 smithi188 ceph-mon[92425]: pgmap v16968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:49 smithi161 ceph-mon[91579]: pgmap v16969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:49 smithi188 ceph-mon[92425]: pgmap v16969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:51 smithi161 ceph-mon[91579]: pgmap v16970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:51 smithi188 ceph-mon[92425]: pgmap v16970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:53 smithi161 ceph-mon[91579]: pgmap v16971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:53 smithi188 ceph-mon[92425]: pgmap v16971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:04:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:55 smithi161 ceph-mon[91579]: pgmap v16972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:55 smithi188 ceph-mon[92425]: pgmap v16972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:04:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:57 smithi161 ceph-mon[91579]: pgmap v16973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:57 smithi188 ceph-mon[92425]: pgmap v16973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:04:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:04:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:04:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:04:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:04:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:04:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:04:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:04:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:05:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:00 smithi161 ceph-mon[91579]: pgmap v16974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:00 smithi188 ceph-mon[92425]: pgmap v16974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:01 smithi161 ceph-mon[91579]: pgmap v16975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:01 smithi188 ceph-mon[92425]: pgmap v16975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:03 smithi161 ceph-mon[91579]: pgmap v16976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:03 smithi188 ceph-mon[92425]: pgmap v16976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:05 smithi161 ceph-mon[91579]: pgmap v16977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:05 smithi188 ceph-mon[92425]: pgmap v16977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:07 smithi161 ceph-mon[91579]: pgmap v16978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:07 smithi188 ceph-mon[92425]: pgmap v16978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:09 smithi161 ceph-mon[91579]: pgmap v16979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:09 smithi188 ceph-mon[92425]: pgmap v16979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:11 smithi161 ceph-mon[91579]: pgmap v16980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:11 smithi188 ceph-mon[92425]: pgmap v16980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:13 smithi161 ceph-mon[91579]: pgmap v16981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:13 smithi188 ceph-mon[92425]: pgmap v16981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:15 smithi161 ceph-mon[91579]: pgmap v16982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:15 smithi188 ceph-mon[92425]: pgmap v16982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:17 smithi161 ceph-mon[91579]: pgmap v16983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:17 smithi188 ceph-mon[92425]: pgmap v16983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:19 smithi161 ceph-mon[91579]: pgmap v16984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:19 smithi188 ceph-mon[92425]: pgmap v16984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:21 smithi161 ceph-mon[91579]: pgmap v16985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:21 smithi188 ceph-mon[92425]: pgmap v16985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:23 smithi161 ceph-mon[91579]: pgmap v16986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:23 smithi188 ceph-mon[92425]: pgmap v16986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:25 smithi161 ceph-mon[91579]: pgmap v16987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:25 smithi188 ceph-mon[92425]: pgmap v16987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:28.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:27 smithi161 ceph-mon[91579]: pgmap v16988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:27 smithi188 ceph-mon[92425]: pgmap v16988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:29 smithi161 ceph-mon[91579]: pgmap v16989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:29 smithi188 ceph-mon[92425]: pgmap v16989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:31 smithi161 ceph-mon[91579]: pgmap v16990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:31 smithi188 ceph-mon[92425]: pgmap v16990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:33 smithi161 ceph-mon[91579]: pgmap v16991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:33 smithi188 ceph-mon[92425]: pgmap v16991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-mon[91579]: pgmap v16992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:35.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:05:35.453+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:05:35.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:05:35.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:05:35.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:05:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:05:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:05:35.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:05:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:05:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:05:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:05:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:05:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:05:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:05:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:05:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:05:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:05:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:05:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:05:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:05:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:05:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:05:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:05:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:05:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:05:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:05:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:05:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:05:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:05:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:05:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:05:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1023] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1023] Local address: 172.21.15.161, port 60310 2022-07-03T20:05:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1023] Peer address: 172.21.15.101, port 22 2022-07-03T20:05:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1023] Connection lost 2022-07-03T20:05:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1023] Aborting connection 2022-07-03T20:05:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:05:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:05:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:35 smithi188 ceph-mon[92425]: pgmap v16992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:05:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:05:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:05:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:05:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:05:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:05:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:05:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:05:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:05:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:05:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:05:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:05:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:05:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:05:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:05:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:05:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:05:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:05:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:05:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:05:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:05:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:05:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:05:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:05:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:05:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:05:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:05:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:05:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:05:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:05:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:05:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:05:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:05:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:05:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:05:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:05:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:05:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:05:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:05:36.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:05:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:05:36.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:05:36.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:05:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:05:36.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:05:36.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:05:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:05:36.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1023] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:05:36.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1023] Local address: 172.21.15.161, port 60310 2022-07-03T20:05:36.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1023] Peer address: 172.21.15.101, port 22 2022-07-03T20:05:36.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1023] Connection lost 2022-07-03T20:05:36.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1023] Aborting connection 2022-07-03T20:05:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:05:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:05:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:05:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:05:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:05:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:05:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:05:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:05:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:05:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:05:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:05:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:05:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:05:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:05:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:05:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:05:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:05:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:05:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:05:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:05:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:05:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:05:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:05:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:05:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:05:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:05:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:05:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:05:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:05:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:05:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:05:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:05:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:05:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:05:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:05:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:05:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:05:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:05:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:05:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:05:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:05:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:05:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:05:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:05:36.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:05:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:05:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:05:36.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:05:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:05:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1023] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:05:36.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1023] Local address: 172.21.15.161, port 60310 2022-07-03T20:05:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1023] Peer address: 172.21.15.101, port 22 2022-07-03T20:05:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1023] Connection lost 2022-07-03T20:05:36.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1023] Aborting connection 2022-07-03T20:05:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:37 smithi188 ceph-mon[92425]: pgmap v16993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:37 smithi161 ceph-mon[91579]: pgmap v16993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:39 smithi161 ceph-mon[91579]: pgmap v16994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:39 smithi188 ceph-mon[92425]: pgmap v16994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:41 smithi161 ceph-mon[91579]: pgmap v16995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:41 smithi188 ceph-mon[92425]: pgmap v16995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:43 smithi161 ceph-mon[91579]: pgmap v16996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:43 smithi188 ceph-mon[92425]: pgmap v16996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:45 smithi188 ceph-mon[92425]: pgmap v16997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:45 smithi161 ceph-mon[91579]: pgmap v16997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:47 smithi161 ceph-mon[91579]: pgmap v16998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:47 smithi188 ceph-mon[92425]: pgmap v16998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:49 smithi161 ceph-mon[91579]: pgmap v16999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:49 smithi188 ceph-mon[92425]: pgmap v16999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:51 smithi161 ceph-mon[91579]: pgmap v17000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:51 smithi188 ceph-mon[92425]: pgmap v17000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:53 smithi161 ceph-mon[91579]: pgmap v17001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:53 smithi188 ceph-mon[92425]: pgmap v17001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:05:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:55 smithi161 ceph-mon[91579]: pgmap v17002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:55 smithi188 ceph-mon[92425]: pgmap v17002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:05:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:57 smithi161 ceph-mon[91579]: pgmap v17003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:57 smithi188 ceph-mon[92425]: pgmap v17003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:05:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:05:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:05:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:05:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:05:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:05:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:05:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:05:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:06:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:00 smithi161 ceph-mon[91579]: pgmap v17004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:00 smithi188 ceph-mon[92425]: pgmap v17004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:01 smithi161 ceph-mon[91579]: pgmap v17005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:01 smithi188 ceph-mon[92425]: pgmap v17005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:03 smithi161 ceph-mon[91579]: pgmap v17006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:03 smithi188 ceph-mon[92425]: pgmap v17006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:05 smithi188 ceph-mon[92425]: pgmap v17007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:05 smithi161 ceph-mon[91579]: pgmap v17007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:07 smithi161 ceph-mon[91579]: pgmap v17008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:08.248 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:07 smithi188 ceph-mon[92425]: pgmap v17008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:09 smithi161 ceph-mon[91579]: pgmap v17009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:09 smithi188 ceph-mon[92425]: pgmap v17009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:11 smithi161 ceph-mon[91579]: pgmap v17010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:11 smithi188 ceph-mon[92425]: pgmap v17010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:13 smithi161 ceph-mon[91579]: pgmap v17011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:13 smithi188 ceph-mon[92425]: pgmap v17011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:15 smithi188 ceph-mon[92425]: pgmap v17012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:15.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:15 smithi161 ceph-mon[91579]: pgmap v17012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:17 smithi161 ceph-mon[91579]: pgmap v17013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:17 smithi188 ceph-mon[92425]: pgmap v17013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:19 smithi161 ceph-mon[91579]: pgmap v17014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:19 smithi188 ceph-mon[92425]: pgmap v17014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:21 smithi161 ceph-mon[91579]: pgmap v17015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:21 smithi188 ceph-mon[92425]: pgmap v17015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:23 smithi161 ceph-mon[91579]: pgmap v17016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:23 smithi188 ceph-mon[92425]: pgmap v17016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:25 smithi188 ceph-mon[92425]: pgmap v17017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:25.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:25 smithi161 ceph-mon[91579]: pgmap v17017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:27 smithi161 ceph-mon[91579]: pgmap v17018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:27 smithi188 ceph-mon[92425]: pgmap v17018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:29 smithi161 ceph-mon[91579]: pgmap v17019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:29 smithi188 ceph-mon[92425]: pgmap v17019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:31 smithi161 ceph-mon[91579]: pgmap v17020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:31 smithi188 ceph-mon[92425]: pgmap v17020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:33 smithi161 ceph-mon[91579]: pgmap v17021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:33 smithi188 ceph-mon[92425]: pgmap v17021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:35 smithi188 ceph-mon[92425]: pgmap v17022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:35.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-mon[91579]: pgmap v17022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:36.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:06:35.953+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:06:36.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:06:36.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:06:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:06:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:06:36.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:06:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:06:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:06:36.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:06:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:06:36.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:06:36.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:06:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:06:36.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:06:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:06:36.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:06:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:06:36.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:06:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:06:36.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:06:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:06:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:06:36.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:06:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:06:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:06:36.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:06:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:06:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:06:36.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:06:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:06:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:06:36.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:06:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:06:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:06:36.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:06:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:06:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:06:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:06:36.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:06:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:06:36.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:06:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:06:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:06:36.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:06:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:06:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1025] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:06:36.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1025] Local address: 172.21.15.161, port 60314 2022-07-03T20:06:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1025] Peer address: 172.21.15.101, port 22 2022-07-03T20:06:36.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1025] Connection lost 2022-07-03T20:06:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1025] Aborting connection 2022-07-03T20:06:36.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:06:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:06:36.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:06:36.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:06:36.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:06:36.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:06:36.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:06:36.402 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:06:36.402 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:06:36.403 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:06:36.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:06:36.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:06:36.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:06:36.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:06:36.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:06:36.407 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:06:36.407 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:06:36.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:06:36.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:06:36.409 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:06:36.409 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:06:36.410 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:06:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:06:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:06:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:06:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:06:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:06:36.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:06:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:06:36.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:06:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:06:36.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:06:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:06:36.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:06:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:06:36.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:06:36.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:06:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:06:36.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:06:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:06:36.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:06:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:06:36.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:06:36.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:06:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:06:36.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:06:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:06:36.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:06:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:06:36.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:06:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:06:36.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1025] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:06:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1025] Local address: 172.21.15.161, port 60314 2022-07-03T20:06:36.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1025] Peer address: 172.21.15.101, port 22 2022-07-03T20:06:36.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1025] Connection lost 2022-07-03T20:06:36.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1025] Aborting connection 2022-07-03T20:06:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:06:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:06:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:06:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:06:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:06:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:06:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:06:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:06:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:06:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:06:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:06:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:06:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:06:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:06:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:06:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:06:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:06:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:06:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:06:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:06:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:06:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:06:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:06:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1025] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:06:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1025] Local address: 172.21.15.161, port 60314 2022-07-03T20:06:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1025] Peer address: 172.21.15.101, port 22 2022-07-03T20:06:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1025] Connection lost 2022-07-03T20:06:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1025] Aborting connection 2022-07-03T20:06:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:37 smithi188 ceph-mon[92425]: pgmap v17023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:37 smithi161 ceph-mon[91579]: pgmap v17023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:39 smithi161 ceph-mon[91579]: pgmap v17024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:39 smithi188 ceph-mon[92425]: pgmap v17024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:41 smithi161 ceph-mon[91579]: pgmap v17025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:41 smithi188 ceph-mon[92425]: pgmap v17025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:43 smithi161 ceph-mon[91579]: pgmap v17026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:43 smithi188 ceph-mon[92425]: pgmap v17026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:45 smithi188 ceph-mon[92425]: pgmap v17027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:45 smithi161 ceph-mon[91579]: pgmap v17027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:47 smithi161 ceph-mon[91579]: pgmap v17028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:47 smithi188 ceph-mon[92425]: pgmap v17028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:49 smithi161 ceph-mon[91579]: pgmap v17029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:49 smithi188 ceph-mon[92425]: pgmap v17029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:51 smithi161 ceph-mon[91579]: pgmap v17030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:51 smithi188 ceph-mon[92425]: pgmap v17030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:53 smithi161 ceph-mon[91579]: pgmap v17031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:53 smithi188 ceph-mon[92425]: pgmap v17031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:06:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:55 smithi188 ceph-mon[92425]: pgmap v17032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:55.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:55 smithi161 ceph-mon[91579]: pgmap v17032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:06:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:57 smithi161 ceph-mon[91579]: pgmap v17033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:57 smithi188 ceph-mon[92425]: pgmap v17033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:06:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:06:59.408 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:06:59.409 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:06:59.409 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:06:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:06:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:06:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:06:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:07:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:00 smithi161 ceph-mon[91579]: pgmap v17034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:00 smithi188 ceph-mon[92425]: pgmap v17034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:01 smithi161 ceph-mon[91579]: pgmap v17035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:01 smithi188 ceph-mon[92425]: pgmap v17035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:03 smithi161 ceph-mon[91579]: pgmap v17036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:03 smithi188 ceph-mon[92425]: pgmap v17036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:05 smithi188 ceph-mon[92425]: pgmap v17037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:05 smithi161 ceph-mon[91579]: pgmap v17037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:07 smithi188 ceph-mon[92425]: pgmap v17038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:07 smithi161 ceph-mon[91579]: pgmap v17038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:09 smithi161 ceph-mon[91579]: pgmap v17039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:09 smithi188 ceph-mon[92425]: pgmap v17039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:11 smithi161 ceph-mon[91579]: pgmap v17040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:11 smithi188 ceph-mon[92425]: pgmap v17040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:13 smithi161 ceph-mon[91579]: pgmap v17041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:13 smithi188 ceph-mon[92425]: pgmap v17041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:15 smithi188 ceph-mon[92425]: pgmap v17042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:15.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:15 smithi161 ceph-mon[91579]: pgmap v17042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:17 smithi188 ceph-mon[92425]: pgmap v17043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:17 smithi161 ceph-mon[91579]: pgmap v17043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:19 smithi161 ceph-mon[91579]: pgmap v17044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:19 smithi188 ceph-mon[92425]: pgmap v17044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:21 smithi161 ceph-mon[91579]: pgmap v17045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:21 smithi188 ceph-mon[92425]: pgmap v17045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:23 smithi161 ceph-mon[91579]: pgmap v17046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:23 smithi188 ceph-mon[92425]: pgmap v17046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:25 smithi188 ceph-mon[92425]: pgmap v17047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:25.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:25 smithi161 ceph-mon[91579]: pgmap v17047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:27 smithi188 ceph-mon[92425]: pgmap v17048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:27 smithi161 ceph-mon[91579]: pgmap v17048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:29 smithi161 ceph-mon[91579]: pgmap v17049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:29 smithi188 ceph-mon[92425]: pgmap v17049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:31 smithi161 ceph-mon[91579]: pgmap v17050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:31 smithi188 ceph-mon[92425]: pgmap v17050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:33 smithi161 ceph-mon[91579]: pgmap v17051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:33 smithi188 ceph-mon[92425]: pgmap v17051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:35 smithi188 ceph-mon[92425]: pgmap v17052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:35.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:35 smithi161 ceph-mon[91579]: pgmap v17052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:07:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:07:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:07:36.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:07:36.455+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:07:36.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:07:36.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:07:36.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:07:36.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:07:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:07:36.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:07:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:07:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:07:36.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:07:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:07:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:07:36.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:07:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:07:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:07:36.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:07:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:07:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:07:36.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:07:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:07:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:07:36.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:07:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:07:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:07:36.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:07:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:07:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:07:36.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:07:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:07:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:07:36.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:07:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:07:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:07:36.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:07:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:07:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:07:36.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:07:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:07:36.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:07:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:07:36.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:07:36.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:07:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:07:36.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:07:36.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:07:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1027] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:07:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1027] Local address: 172.21.15.161, port 60318 2022-07-03T20:07:36.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1027] Peer address: 172.21.15.101, port 22 2022-07-03T20:07:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1027] Connection lost 2022-07-03T20:07:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1027] Aborting connection 2022-07-03T20:07:36.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:07:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:07:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:07:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:07:36.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:07:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:07:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:07:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:07:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:07:37.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:07:37.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:07:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:07:37.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:07:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:07:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:07:37.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:07:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:07:37.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:07:37.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:07:37.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:07:37.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:07:37.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:07:37.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:07:37.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:07:37.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:07:37.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:07:37.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:07:37.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1027] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:07:37.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1027] Local address: 172.21.15.161, port 60318 2022-07-03T20:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1027] Peer address: 172.21.15.101, port 22 2022-07-03T20:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1027] Connection lost 2022-07-03T20:07:37.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1027] Aborting connection 2022-07-03T20:07:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:37.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:37 smithi161 ceph-mon[91579]: pgmap v17053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:07:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:07:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:07:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:07:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:07:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:07:37.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:07:37.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:07:37.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:07:37.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:07:37.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:07:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:07:37.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:07:37.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:07:37.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:07:37.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:07:37.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:07:37.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:07:37.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:07:37.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:07:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:07:37.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1027] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1027] Local address: 172.21.15.161, port 60318 2022-07-03T20:07:37.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1027] Peer address: 172.21.15.101, port 22 2022-07-03T20:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1027] Connection lost 2022-07-03T20:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1027] Aborting connection 2022-07-03T20:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:37.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:37 smithi188 ceph-mon[92425]: pgmap v17053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:39 smithi161 ceph-mon[91579]: pgmap v17054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:39 smithi188 ceph-mon[92425]: pgmap v17054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:41 smithi161 ceph-mon[91579]: pgmap v17055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:41 smithi188 ceph-mon[92425]: pgmap v17055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:43 smithi161 ceph-mon[91579]: pgmap v17056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:43 smithi188 ceph-mon[92425]: pgmap v17056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:45 smithi188 ceph-mon[92425]: pgmap v17057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:45 smithi161 ceph-mon[91579]: pgmap v17057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:47 smithi188 ceph-mon[92425]: pgmap v17058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:47 smithi161 ceph-mon[91579]: pgmap v17058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:49 smithi161 ceph-mon[91579]: pgmap v17059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:49 smithi188 ceph-mon[92425]: pgmap v17059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:51 smithi161 ceph-mon[91579]: pgmap v17060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:51 smithi188 ceph-mon[92425]: pgmap v17060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:53 smithi161 ceph-mon[91579]: pgmap v17061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:53 smithi188 ceph-mon[92425]: pgmap v17061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:07:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:55 smithi188 ceph-mon[92425]: pgmap v17062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:55.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:55 smithi161 ceph-mon[91579]: pgmap v17062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:07:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:57 smithi188 ceph-mon[92425]: pgmap v17063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:57 smithi161 ceph-mon[91579]: pgmap v17063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:07:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:07:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:07:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:07:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:07:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:07:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:07:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:07:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:08:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:00 smithi161 ceph-mon[91579]: pgmap v17064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:00 smithi188 ceph-mon[92425]: pgmap v17064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:01 smithi161 ceph-mon[91579]: pgmap v17065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:01 smithi188 ceph-mon[92425]: pgmap v17065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:03 smithi161 ceph-mon[91579]: pgmap v17066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:03 smithi188 ceph-mon[92425]: pgmap v17066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:05 smithi188 ceph-mon[92425]: pgmap v17067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:05 smithi161 ceph-mon[91579]: pgmap v17067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:07 smithi188 ceph-mon[92425]: pgmap v17068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:07 smithi161 ceph-mon[91579]: pgmap v17068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:09 smithi161 ceph-mon[91579]: pgmap v17069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:09 smithi188 ceph-mon[92425]: pgmap v17069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:11 smithi161 ceph-mon[91579]: pgmap v17070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:11 smithi188 ceph-mon[92425]: pgmap v17070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:13 smithi161 ceph-mon[91579]: pgmap v17071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:13 smithi188 ceph-mon[92425]: pgmap v17071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:15 smithi188 ceph-mon[92425]: pgmap v17072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:15.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:15 smithi161 ceph-mon[91579]: pgmap v17072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:17 smithi161 ceph-mon[91579]: pgmap v17073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:17 smithi188 ceph-mon[92425]: pgmap v17073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:19 smithi161 ceph-mon[91579]: pgmap v17074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:19 smithi188 ceph-mon[92425]: pgmap v17074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:21 smithi161 ceph-mon[91579]: pgmap v17075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:21 smithi188 ceph-mon[92425]: pgmap v17075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:23 smithi161 ceph-mon[91579]: pgmap v17076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:23 smithi188 ceph-mon[92425]: pgmap v17076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:25 smithi188 ceph-mon[92425]: pgmap v17077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:25.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:25 smithi161 ceph-mon[91579]: pgmap v17077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:27 smithi161 ceph-mon[91579]: pgmap v17078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:27 smithi188 ceph-mon[92425]: pgmap v17078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:29 smithi161 ceph-mon[91579]: pgmap v17079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:29 smithi188 ceph-mon[92425]: pgmap v17079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:31 smithi161 ceph-mon[91579]: pgmap v17080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:31 smithi188 ceph-mon[92425]: pgmap v17080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:33 smithi161 ceph-mon[91579]: pgmap v17081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:33 smithi188 ceph-mon[92425]: pgmap v17081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:35.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:35 smithi161 ceph-mon[91579]: pgmap v17082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:35.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:35 smithi188 ceph-mon[92425]: pgmap v17082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:37.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:08:36.951+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:08:37.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:08:37.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:08:37.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:08:37.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:08:37.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:08:37.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:08:37.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:08:37.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:08:37.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:08:37.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:08:37.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:08:37.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:08:37.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:08:37.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:08:37.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:08:37.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:08:37.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:08:37.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:08:37.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:08:37.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:08:37.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1029] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:08:37.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1029] Local address: 172.21.15.161, port 60322 2022-07-03T20:08:37.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1029] Peer address: 172.21.15.101, port 22 2022-07-03T20:08:37.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1029] Connection lost 2022-07-03T20:08:37.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1029] Aborting connection 2022-07-03T20:08:37.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:08:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:08:37.396 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:08:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:08:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:08:38.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:08:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:08:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:08:38.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:08:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:08:38.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:08:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:08:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:08:38.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:08:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:08:38.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:08:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:08:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:08:38.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:08:38.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:08:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:08:38.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:08:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:08:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:08:38.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:08:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:08:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:08:38.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:08:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:08:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:08:38.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:08:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:08:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:08:38.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:08:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:08:38.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:08:38.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:08:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:08:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:08:38.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:08:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:08:38.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:08:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:08:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:08:38.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:08:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:08:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:08:38.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1029] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:08:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1029] Local address: 172.21.15.161, port 60322 2022-07-03T20:08:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1029] Peer address: 172.21.15.101, port 22 2022-07-03T20:08:38.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1029] Connection lost 2022-07-03T20:08:38.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1029] Aborting connection 2022-07-03T20:08:38.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: pgmap v17083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:38.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.400 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.401 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:37 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:08:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:08:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:08:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:08:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:08:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:08:38.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:08:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:08:38.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:08:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:08:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:08:38.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:08:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:08:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:08:38.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:08:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:08:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:08:38.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:08:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:08:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:08:38.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:08:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:08:38.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:08:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:08:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:08:38.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:08:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:08:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:08:38.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:08:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:08:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:08:38.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:08:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:08:38.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:08:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:08:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:08:38.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:08:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:08:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:08:38.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:08:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:08:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:08:38.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:08:38.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:08:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:08:38.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1029] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:08:38.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1029] Local address: 172.21.15.161, port 60322 2022-07-03T20:08:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1029] Peer address: 172.21.15.101, port 22 2022-07-03T20:08:38.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1029] Connection lost 2022-07-03T20:08:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1029] Aborting connection 2022-07-03T20:08:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: pgmap v17083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:38.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.432 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:38.433 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:37 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:08:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:39 smithi161 ceph-mon[91579]: pgmap v17084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:39 smithi188 ceph-mon[92425]: pgmap v17084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:41 smithi161 ceph-mon[91579]: pgmap v17085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:41.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:41 smithi188 ceph-mon[92425]: pgmap v17085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:43 smithi161 ceph-mon[91579]: pgmap v17086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:43 smithi188 ceph-mon[92425]: pgmap v17086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:45 smithi161 ceph-mon[91579]: pgmap v17087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:45.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:45 smithi188 ceph-mon[92425]: pgmap v17087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:47 smithi161 ceph-mon[91579]: pgmap v17088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:47 smithi188 ceph-mon[92425]: pgmap v17088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:49 smithi161 ceph-mon[91579]: pgmap v17089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:49 smithi188 ceph-mon[92425]: pgmap v17089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:51 smithi161 ceph-mon[91579]: pgmap v17090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:51 smithi188 ceph-mon[92425]: pgmap v17090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:53 smithi161 ceph-mon[91579]: pgmap v17091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:53 smithi188 ceph-mon[92425]: pgmap v17091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:08:55.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:55 smithi161 ceph-mon[91579]: pgmap v17092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:55.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:55 smithi188 ceph-mon[92425]: pgmap v17092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:08:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:57 smithi161 ceph-mon[91579]: pgmap v17093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:57 smithi188 ceph-mon[92425]: pgmap v17093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:08:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:08:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:08:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:08:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:08:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:08:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:08:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:08:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:09:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:00 smithi161 ceph-mon[91579]: pgmap v17094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:00 smithi188 ceph-mon[92425]: pgmap v17094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:01 smithi161 ceph-mon[91579]: pgmap v17095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:01 smithi188 ceph-mon[92425]: pgmap v17095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:03 smithi161 ceph-mon[91579]: pgmap v17096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:03 smithi188 ceph-mon[92425]: pgmap v17096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:05 smithi161 ceph-mon[91579]: pgmap v17097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:05.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:05 smithi188 ceph-mon[92425]: pgmap v17097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:07 smithi161 ceph-mon[91579]: pgmap v17098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:07 smithi188 ceph-mon[92425]: pgmap v17098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:09 smithi161 ceph-mon[91579]: pgmap v17099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:09 smithi188 ceph-mon[92425]: pgmap v17099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:11 smithi161 ceph-mon[91579]: pgmap v17100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:11 smithi188 ceph-mon[92425]: pgmap v17100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:13 smithi161 ceph-mon[91579]: pgmap v17101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:13 smithi188 ceph-mon[92425]: pgmap v17101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:15.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:15 smithi161 ceph-mon[91579]: pgmap v17102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:15.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:15 smithi188 ceph-mon[92425]: pgmap v17102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:17 smithi161 ceph-mon[91579]: pgmap v17103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:17 smithi188 ceph-mon[92425]: pgmap v17103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:19 smithi161 ceph-mon[91579]: pgmap v17104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:19 smithi188 ceph-mon[92425]: pgmap v17104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:21 smithi161 ceph-mon[91579]: pgmap v17105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:21 smithi188 ceph-mon[92425]: pgmap v17105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:23 smithi161 ceph-mon[91579]: pgmap v17106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:23 smithi188 ceph-mon[92425]: pgmap v17106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:25.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:25 smithi161 ceph-mon[91579]: pgmap v17107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:25.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:25 smithi188 ceph-mon[92425]: pgmap v17107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:27 smithi161 ceph-mon[91579]: pgmap v17108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:27 smithi188 ceph-mon[92425]: pgmap v17108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:29 smithi161 ceph-mon[91579]: pgmap v17109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:29 smithi188 ceph-mon[92425]: pgmap v17109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:31 smithi161 ceph-mon[91579]: pgmap v17110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:31 smithi188 ceph-mon[92425]: pgmap v17110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:33 smithi161 ceph-mon[91579]: pgmap v17111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:33 smithi188 ceph-mon[92425]: pgmap v17111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:35.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:35 smithi161 ceph-mon[91579]: pgmap v17112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:35.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:35 smithi188 ceph-mon[92425]: pgmap v17112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-mon[91579]: pgmap v17113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:09:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:09:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:09:38.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:09:37.875+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:09:38.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:09:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:09:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:09:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:09:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:09:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:09:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:09:38.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:09:38.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:09:38.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:09:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:09:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:09:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:09:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:09:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:09:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:09:38.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:09:38.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:09:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:09:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:09:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:09:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:09:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:09:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:09:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:09:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:09:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:09:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:09:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:09:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:09:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:09:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:09:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:09:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:09:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:09:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:09:38.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:09:38.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:09:38.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:09:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:09:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:09:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:09:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:09:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:09:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1031] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:09:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1031] Local address: 172.21.15.161, port 60352 2022-07-03T20:09:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1031] Peer address: 172.21.15.101, port 22 2022-07-03T20:09:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1031] Connection lost 2022-07-03T20:09:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1031] Aborting connection 2022-07-03T20:09:38.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:09:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:09:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:37 smithi188 ceph-mon[92425]: pgmap v17113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:09:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:09:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:09:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:09:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:09:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:09:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:09:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:09:39.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:09:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:09:39.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:09:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:09:39.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:09:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:09:39.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:09:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:09:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:09:39.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:09:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:09:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:09:39.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:09:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:09:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:09:39.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:09:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:09:39.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:09:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:09:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:09:39.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:09:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:09:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:09:39.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:09:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:09:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:09:39.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:09:39.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:09:39.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:09:39.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:09:39.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:09:39.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:09:39.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:09:39.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:09:39.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:09:39.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:09:39.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:09:39.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:09:39.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:09:39.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:09:39.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:09:39.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1031] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:09:39.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1031] Local address: 172.21.15.161, port 60352 2022-07-03T20:09:39.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1031] Peer address: 172.21.15.101, port 22 2022-07-03T20:09:39.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1031] Connection lost 2022-07-03T20:09:39.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1031] Aborting connection 2022-07-03T20:09:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:09:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:09:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:09:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:09:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:09:39.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:09:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:09:39.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:09:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:09:39.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:09:39.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:09:39.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:09:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:09:39.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:09:39.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:09:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:09:39.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:09:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:09:39.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:09:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:09:39.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:09:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:09:39.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:09:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:09:39.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:09:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:09:39.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:09:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:09:39.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:09:39.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:09:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:09:39.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:09:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:09:39.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:09:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:09:39.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:09:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:09:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:09:39.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:09:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:09:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:09:39.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:09:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:09:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:09:39.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:09:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:09:39.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1031] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:09:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1031] Local address: 172.21.15.161, port 60352 2022-07-03T20:09:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1031] Peer address: 172.21.15.101, port 22 2022-07-03T20:09:39.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1031] Connection lost 2022-07-03T20:09:39.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1031] Aborting connection 2022-07-03T20:09:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:39 smithi161 ceph-mon[91579]: pgmap v17114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:39 smithi188 ceph-mon[92425]: pgmap v17114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:41 smithi161 ceph-mon[91579]: pgmap v17115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:41 smithi188 ceph-mon[92425]: pgmap v17115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:43 smithi161 ceph-mon[91579]: pgmap v17116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:43 smithi188 ceph-mon[92425]: pgmap v17116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:45.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:45 smithi161 ceph-mon[91579]: pgmap v17117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:45.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:45 smithi188 ceph-mon[92425]: pgmap v17117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:46 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:46 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:47 smithi161 ceph-mon[91579]: pgmap v17118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:47 smithi188 ceph-mon[92425]: pgmap v17118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:49 smithi161 ceph-mon[91579]: pgmap v17119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:49 smithi188 ceph-mon[92425]: pgmap v17119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:51 smithi161 ceph-mon[91579]: pgmap v17120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:51 smithi188 ceph-mon[92425]: pgmap v17120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:52.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:52.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:53 smithi161 ceph-mon[91579]: pgmap v17121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:53 smithi188 ceph-mon[92425]: pgmap v17121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:09:55.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:55 smithi161 ceph-mon[91579]: pgmap v17122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:55.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:55 smithi188 ceph-mon[92425]: pgmap v17122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:56 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:56 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:09:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:57 smithi161 ceph-mon[91579]: pgmap v17123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:57 smithi188 ceph-mon[92425]: pgmap v17123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:09:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:09:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:09:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:09:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:09:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:09:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:09:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:09:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:10:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:00 smithi161 ceph-mon[91579]: pgmap v17124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:10:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:00 smithi188 ceph-mon[92425]: pgmap v17124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:10:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:01 smithi161 ceph-mon[91579]: pgmap v17125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:01 smithi188 ceph-mon[92425]: pgmap v17125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:02.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:03 smithi161 ceph-mon[91579]: pgmap v17126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:03 smithi188 ceph-mon[92425]: pgmap v17126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:05.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:05 smithi161 ceph-mon[91579]: pgmap v17127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:05.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:05 smithi188 ceph-mon[92425]: pgmap v17127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:07 smithi161 ceph-mon[91579]: pgmap v17128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:07 smithi188 ceph-mon[92425]: pgmap v17128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:09 smithi161 ceph-mon[91579]: pgmap v17129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:09 smithi188 ceph-mon[92425]: pgmap v17129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:11 smithi161 ceph-mon[91579]: pgmap v17130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:11 smithi188 ceph-mon[92425]: pgmap v17130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:13 smithi161 ceph-mon[91579]: pgmap v17131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:13 smithi188 ceph-mon[92425]: pgmap v17131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:15.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:15 smithi161 ceph-mon[91579]: pgmap v17132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:15.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:15 smithi188 ceph-mon[92425]: pgmap v17132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:16 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:16 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:17 smithi161 ceph-mon[91579]: pgmap v17133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:17 smithi188 ceph-mon[92425]: pgmap v17133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:19 smithi161 ceph-mon[91579]: pgmap v17134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:19 smithi188 ceph-mon[92425]: pgmap v17134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:21 smithi161 ceph-mon[91579]: pgmap v17135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:21 smithi188 ceph-mon[92425]: pgmap v17135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:23 smithi161 ceph-mon[91579]: pgmap v17136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:23 smithi188 ceph-mon[92425]: pgmap v17136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:25.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:25 smithi161 ceph-mon[91579]: pgmap v17137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:25.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:25 smithi188 ceph-mon[92425]: pgmap v17137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:26 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:26 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:27 smithi161 ceph-mon[91579]: pgmap v17138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:27 smithi188 ceph-mon[92425]: pgmap v17138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:29 smithi161 ceph-mon[91579]: pgmap v17139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:29 smithi188 ceph-mon[92425]: pgmap v17139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:31 smithi161 ceph-mon[91579]: pgmap v17140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:31 smithi188 ceph-mon[92425]: pgmap v17140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:33 smithi161 ceph-mon[91579]: pgmap v17141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:33 smithi188 ceph-mon[92425]: pgmap v17141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:36 smithi161 ceph-mon[91579]: pgmap v17142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:36 smithi188 ceph-mon[92425]: pgmap v17142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-mon[91579]: pgmap v17143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:10:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:10:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:10:38.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:10:38.371+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:10:38.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:10:38.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:10:38.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:10:38.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:10:38.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:10:38.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:10:38.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:10:38.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:10:38.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:10:38.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:10:38.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:10:38.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:10:38.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:10:38.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:10:38.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:10:38.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:10:38.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:10:38.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:10:38.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:10:38.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:10:38.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:10:38.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:10:38.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:10:38.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:10:38.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:10:38.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:10:38.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:10:38.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:10:38.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:10:38.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:10:38.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:10:38.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:10:38.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:10:38.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:10:38.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:10:38.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:10:38.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:10:38.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:10:38.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:10:38.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:10:38.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:10:38.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:10:38.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:10:38.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:10:38.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1033] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:10:38.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1033] Local address: 172.21.15.161, port 60356 2022-07-03T20:10:38.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1033] Peer address: 172.21.15.101, port 22 2022-07-03T20:10:38.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1033] Connection lost 2022-07-03T20:10:38.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1033] Aborting connection 2022-07-03T20:10:38.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:10:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:10:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:38 smithi188 ceph-mon[92425]: pgmap v17143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:10:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:10:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:10:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:10:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:10:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:10:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:10:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:10:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:10:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:10:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:10:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:10:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:10:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:10:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:10:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:10:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:10:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:10:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:10:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:10:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:10:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:10:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:10:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:10:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:10:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:10:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:10:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:10:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:10:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:10:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:10:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:10:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1033] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:10:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1033] Local address: 172.21.15.161, port 60356 2022-07-03T20:10:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1033] Peer address: 172.21.15.101, port 22 2022-07-03T20:10:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1033] Connection lost 2022-07-03T20:10:39.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1033] Aborting connection 2022-07-03T20:10:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:10:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:10:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:10:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:10:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:10:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:10:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:10:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:10:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:10:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:10:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:10:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:10:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:10:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:10:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:10:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:10:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:10:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:10:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:10:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:10:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:10:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:10:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:10:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:10:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:10:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:10:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:10:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:10:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:10:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:10:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:10:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:10:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:10:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:10:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:10:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:10:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:10:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1033] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:10:39.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1033] Local address: 172.21.15.161, port 60356 2022-07-03T20:10:39.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1033] Peer address: 172.21.15.101, port 22 2022-07-03T20:10:39.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1033] Connection lost 2022-07-03T20:10:39.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1033] Aborting connection 2022-07-03T20:10:40.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:40 smithi161 ceph-mon[91579]: pgmap v17144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:40.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:40 smithi188 ceph-mon[92425]: pgmap v17144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:42.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:42 smithi161 ceph-mon[91579]: pgmap v17145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:42.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:42 smithi188 ceph-mon[92425]: pgmap v17145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:44.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:44 smithi161 ceph-mon[91579]: pgmap v17146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:44 smithi188 ceph-mon[92425]: pgmap v17146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:46 smithi161 ceph-mon[91579]: pgmap v17147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:46 smithi188 ceph-mon[92425]: pgmap v17147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:48 smithi161 ceph-mon[91579]: pgmap v17148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:48 smithi188 ceph-mon[92425]: pgmap v17148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:50.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:50 smithi161 ceph-mon[91579]: pgmap v17149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:50 smithi188 ceph-mon[92425]: pgmap v17149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:52.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:52 smithi161 ceph-mon[91579]: pgmap v17150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:52.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:52 smithi188 ceph-mon[92425]: pgmap v17150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:54.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:54 smithi161 ceph-mon[91579]: pgmap v17151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:54.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:54 smithi188 ceph-mon[92425]: pgmap v17151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:10:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:56 smithi161 ceph-mon[91579]: pgmap v17152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:56 smithi188 ceph-mon[92425]: pgmap v17152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:10:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:58 smithi161 ceph-mon[91579]: pgmap v17153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:58 smithi188 ceph-mon[92425]: pgmap v17153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:10:59.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:10:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:10:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:10:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:10:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:10:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:10:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:10:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:11:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:00 smithi161 ceph-mon[91579]: pgmap v17154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:00.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:00 smithi188 ceph-mon[92425]: pgmap v17154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:02.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:02 smithi161 ceph-mon[91579]: pgmap v17155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:02.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:02 smithi188 ceph-mon[92425]: pgmap v17155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:04.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:04 smithi161 ceph-mon[91579]: pgmap v17156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:04.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:04 smithi188 ceph-mon[92425]: pgmap v17156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:06 smithi161 ceph-mon[91579]: pgmap v17157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:06 smithi188 ceph-mon[92425]: pgmap v17157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:08 smithi161 ceph-mon[91579]: pgmap v17158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:08.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:08 smithi188 ceph-mon[92425]: pgmap v17158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:10 smithi161 ceph-mon[91579]: pgmap v17159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:10.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:10 smithi188 ceph-mon[92425]: pgmap v17159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:12.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:12 smithi161 ceph-mon[91579]: pgmap v17160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:12.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:12 smithi188 ceph-mon[92425]: pgmap v17160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:12.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:14 smithi161 ceph-mon[91579]: pgmap v17161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:14.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:14 smithi188 ceph-mon[92425]: pgmap v17161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:16 smithi161 ceph-mon[91579]: pgmap v17162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:16 smithi188 ceph-mon[92425]: pgmap v17162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:18 smithi161 ceph-mon[91579]: pgmap v17163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:18 smithi188 ceph-mon[92425]: pgmap v17163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:20.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:20 smithi161 ceph-mon[91579]: pgmap v17164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:20.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:20 smithi188 ceph-mon[92425]: pgmap v17164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:22.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:22 smithi161 ceph-mon[91579]: pgmap v17165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:22.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:22 smithi188 ceph-mon[92425]: pgmap v17165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:23.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:24.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:24 smithi161 ceph-mon[91579]: pgmap v17166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:24.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:24 smithi188 ceph-mon[92425]: pgmap v17166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:26 smithi161 ceph-mon[91579]: pgmap v17167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:26 smithi188 ceph-mon[92425]: pgmap v17167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:28 smithi161 ceph-mon[91579]: pgmap v17168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:28 smithi188 ceph-mon[92425]: pgmap v17168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:30.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:30 smithi161 ceph-mon[91579]: pgmap v17169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:30.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:30 smithi188 ceph-mon[92425]: pgmap v17169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:32.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:32 smithi161 ceph-mon[91579]: pgmap v17170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:32.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:32 smithi188 ceph-mon[92425]: pgmap v17170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:33.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:33.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:34.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:34 smithi161 ceph-mon[91579]: pgmap v17171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:34.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:34 smithi188 ceph-mon[92425]: pgmap v17171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:36 smithi161 ceph-mon[91579]: pgmap v17172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:36 smithi188 ceph-mon[92425]: pgmap v17172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:37.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:38.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-mon[91579]: pgmap v17173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:38.905 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:38 smithi188 ceph-mon[92425]: pgmap v17173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:38.905 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:39.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:11:38.884+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:11:39.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:11:39.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:11:39.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:11:39.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:11:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:11:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:11:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:11:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:11:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:11:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:11:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:11:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:11:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:11:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:11:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:11:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:11:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:11:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:11:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:11:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:11:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:11:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:11:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:11:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:11:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:11:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:11:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:11:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:11:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:11:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:11:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:11:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:11:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:11:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:11:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:11:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:11:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:11:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:11:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:11:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:11:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:11:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:11:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:11:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:11:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1035] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:11:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1035] Local address: 172.21.15.161, port 60370 2022-07-03T20:11:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1035] Peer address: 172.21.15.101, port 22 2022-07-03T20:11:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1035] Connection lost 2022-07-03T20:11:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1035] Aborting connection 2022-07-03T20:11:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:11:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:11:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:11:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:11:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:11:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:11:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:11:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:11:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:11:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:11:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:11:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:11:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:11:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:11:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:11:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:11:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:11:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:11:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:11:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:11:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:11:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:11:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:11:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:11:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:11:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:11:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:11:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:11:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:11:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:11:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:11:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:11:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:11:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:11:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:11:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:11:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:11:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:11:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:11:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:11:39.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:11:39.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:11:39.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:11:39.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:11:39.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:11:39.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:11:39.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:11:39.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:11:39.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:11:39.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:11:39.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:11:39.904 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1035] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:11:39.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1035] Local address: 172.21.15.161, port 60370 2022-07-03T20:11:39.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1035] Peer address: 172.21.15.101, port 22 2022-07-03T20:11:39.905 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1035] Connection lost 2022-07-03T20:11:39.906 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1035] Aborting connection 2022-07-03T20:11:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:11:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:11:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:11:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:11:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:11:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:11:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:11:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:11:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:11:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:11:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:11:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:11:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:11:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:11:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:11:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:11:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:11:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:11:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:11:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:11:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:11:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:11:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:11:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:11:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:11:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:11:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:11:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:11:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:11:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:11:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:11:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:11:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:11:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:11:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:11:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:11:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:11:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:11:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:11:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:11:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:11:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:11:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:11:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:11:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:11:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:11:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:11:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:11:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:11:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1035] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:11:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1035] Local address: 172.21.15.161, port 60370 2022-07-03T20:11:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1035] Peer address: 172.21.15.101, port 22 2022-07-03T20:11:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1035] Connection lost 2022-07-03T20:11:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1035] Aborting connection 2022-07-03T20:11:40.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:40 smithi161 ceph-mon[91579]: pgmap v17174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:40.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:40 smithi188 ceph-mon[92425]: pgmap v17174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:42.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: pgmap v17175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 82443k 2022-07-03T20:11:42.884 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:42 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 84421746: error parsing value: Value '84421746' is below minimum 939524096 2022-07-03T20:11:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: pgmap v17175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:11:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.918 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.919 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:11:42.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 82443k 2022-07-03T20:11:42.920 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:42 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 84421746: error parsing value: Value '84421746' is below minimum 939524096 2022-07-03T20:11:43.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:44.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:44 smithi188 ceph-mon[92425]: pgmap v17176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:44 smithi161 ceph-mon[91579]: pgmap v17176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:46 smithi188 ceph-mon[92425]: pgmap v17177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:46 smithi161 ceph-mon[91579]: pgmap v17177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:48 smithi188 ceph-mon[92425]: pgmap v17178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:48 smithi161 ceph-mon[91579]: pgmap v17178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:50 smithi188 ceph-mon[92425]: pgmap v17179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:50 smithi161 ceph-mon[91579]: pgmap v17179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:52.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:52 smithi188 ceph-mon[92425]: pgmap v17180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:52 smithi161 ceph-mon[91579]: pgmap v17180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:11:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:53.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:54.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:54 smithi188 ceph-mon[92425]: pgmap v17181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:54 smithi161 ceph-mon[91579]: pgmap v17181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:11:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:56 smithi161 ceph-mon[91579]: pgmap v17182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:11:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:56 smithi188 ceph-mon[92425]: pgmap v17182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:11:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:11:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:58 smithi161 ceph-mon[91579]: pgmap v17183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:11:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:11:59.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:58 smithi188 ceph-mon[92425]: pgmap v17183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:11:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:12:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:12:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:12:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:12:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:12:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:12:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:12:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:12:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:00 smithi161 ceph-mon[91579]: pgmap v17184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:00 smithi188 ceph-mon[92425]: pgmap v17184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:03.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:02 smithi161 ceph-mon[91579]: pgmap v17185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:12:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:02 smithi188 ceph-mon[92425]: pgmap v17185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:12:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:04 smithi161 ceph-mon[91579]: pgmap v17186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:04 smithi188 ceph-mon[92425]: pgmap v17186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:06 smithi161 ceph-mon[91579]: pgmap v17187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:06 smithi188 ceph-mon[92425]: pgmap v17187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:08 smithi161 ceph-mon[91579]: pgmap v17188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:08 smithi188 ceph-mon[92425]: pgmap v17188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:10 smithi161 ceph-mon[91579]: pgmap v17189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:11.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:10 smithi188 ceph-mon[92425]: pgmap v17189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:12 smithi161 ceph-mon[91579]: pgmap v17190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:12 smithi188 ceph-mon[92425]: pgmap v17190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:14 smithi161 ceph-mon[91579]: pgmap v17191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:14 smithi188 ceph-mon[92425]: pgmap v17191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:16 smithi161 ceph-mon[91579]: pgmap v17192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:16 smithi188 ceph-mon[92425]: pgmap v17192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:18 smithi161 ceph-mon[91579]: pgmap v17193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:18 smithi188 ceph-mon[92425]: pgmap v17193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:21.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:20 smithi161 ceph-mon[91579]: pgmap v17194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:20 smithi188 ceph-mon[92425]: pgmap v17194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:23.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:22 smithi161 ceph-mon[91579]: pgmap v17195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:23.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:22 smithi188 ceph-mon[92425]: pgmap v17195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:24.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:24 smithi161 ceph-mon[91579]: pgmap v17196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:25.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:24 smithi188 ceph-mon[92425]: pgmap v17196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:26 smithi161 ceph-mon[91579]: pgmap v17197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:26 smithi188 ceph-mon[92425]: pgmap v17197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:28 smithi161 ceph-mon[91579]: pgmap v17198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:28 smithi188 ceph-mon[92425]: pgmap v17198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:31.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:30 smithi161 ceph-mon[91579]: pgmap v17199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:30 smithi188 ceph-mon[92425]: pgmap v17199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:33.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:32 smithi161 ceph-mon[91579]: pgmap v17200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:33.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:32 smithi188 ceph-mon[92425]: pgmap v17200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:34.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:34.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:34 smithi161 ceph-mon[91579]: pgmap v17201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:35.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:34 smithi188 ceph-mon[92425]: pgmap v17201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:36 smithi161 ceph-mon[91579]: pgmap v17202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:36 smithi188 ceph-mon[92425]: pgmap v17202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:39.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:38 smithi161 ceph-mon[91579]: pgmap v17203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:38 smithi188 ceph-mon[92425]: pgmap v17203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:40 smithi161 ceph-mon[91579]: pgmap v17204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:41.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:40 smithi188 ceph-mon[92425]: pgmap v17204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:12:42.153+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:12:42.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:12:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:12:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:12:42.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:12:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:12:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:12:42.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:12:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:12:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:12:42.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:12:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:12:42.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:12:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:12:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:12:42.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:12:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:12:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:12:42.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:12:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:12:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:12:42.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:12:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:12:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:12:42.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:12:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:12:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:12:42.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:12:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:12:42.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:12:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:12:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:12:42.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:12:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:12:42.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:12:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:12:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:12:42.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:12:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:12:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:12:42.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:12:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:12:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:12:42.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:12:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:12:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1037] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:12:42.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1037] Local address: 172.21.15.161, port 60398 2022-07-03T20:12:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1037] Peer address: 172.21.15.101, port 22 2022-07-03T20:12:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1037] Connection lost 2022-07-03T20:12:42.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1037] Aborting connection 2022-07-03T20:12:42.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:12:43.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: pgmap v17205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:12:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:12:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:12:43.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:12:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:42 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:12:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:12:43.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:12:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:12:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:12:43.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:12:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:12:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:12:43.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:12:43.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:12:43.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:12:43.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:12:43.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:12:43.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:12:43.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:12:43.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:12:43.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:12:43.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:12:43.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:12:43.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:12:43.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:12:43.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:12:43.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:12:43.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:12:43.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:12:43.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:12:43.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:12:43.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:12:43.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:12:43.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:12:43.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:12:43.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:12:43.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:12:43.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:12:43.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:12:43.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:12:43.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:12:43.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:12:43.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:12:43.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:12:43.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:12:43.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:12:43.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:12:43.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1037] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:12:43.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1037] Local address: 172.21.15.161, port 60398 2022-07-03T20:12:43.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1037] Peer address: 172.21.15.101, port 22 2022-07-03T20:12:43.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1037] Connection lost 2022-07-03T20:12:43.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1037] Aborting connection 2022-07-03T20:12:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: pgmap v17205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:12:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:12:43.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:12:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:12:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:42 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:12:43.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:12:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:12:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:12:43.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:12:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:12:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:12:43.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:12:43.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:12:43.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:12:43.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:12:43.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:12:43.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:12:43.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:12:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:12:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:12:43.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:12:43.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:12:43.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:12:43.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:12:43.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:12:43.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:12:43.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:12:43.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:12:43.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:12:43.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:12:43.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:12:43.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:12:43.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:12:43.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:12:43.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:12:43.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:12:43.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:12:43.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:12:43.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:12:43.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:12:43.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:12:43.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:12:43.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:12:43.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:12:43.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:12:43.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:12:43.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:12:43.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:12:43.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:12:43.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1037] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:12:43.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1037] Local address: 172.21.15.161, port 60398 2022-07-03T20:12:43.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1037] Peer address: 172.21.15.101, port 22 2022-07-03T20:12:43.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1037] Connection lost 2022-07-03T20:12:43.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1037] Aborting connection 2022-07-03T20:12:44.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:44.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:44 smithi161 ceph-mon[91579]: pgmap v17206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:45.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:44 smithi188 ceph-mon[92425]: pgmap v17206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:46 smithi161 ceph-mon[91579]: pgmap v17207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:46 smithi188 ceph-mon[92425]: pgmap v17207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:48 smithi161 ceph-mon[91579]: pgmap v17208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:49.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:48 smithi188 ceph-mon[92425]: pgmap v17208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:50 smithi161 ceph-mon[91579]: pgmap v17209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:51.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:50 smithi188 ceph-mon[92425]: pgmap v17209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:52 smithi161 ceph-mon[91579]: pgmap v17210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:53.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:52 smithi188 ceph-mon[92425]: pgmap v17210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:54.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:54.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:54 smithi161 ceph-mon[91579]: pgmap v17211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:55.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:54 smithi188 ceph-mon[92425]: pgmap v17211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:12:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:56 smithi161 ceph-mon[91579]: pgmap v17212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:56 smithi188 ceph-mon[92425]: pgmap v17212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:12:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:58 smithi161 ceph-mon[91579]: pgmap v17213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:12:59.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:58 smithi188 ceph-mon[92425]: pgmap v17213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:12:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:13:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:13:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:13:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:13:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:13:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:13:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:13:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:13:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:00 smithi161 ceph-mon[91579]: pgmap v17214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:00 smithi188 ceph-mon[92425]: pgmap v17214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:03.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:02 smithi161 ceph-mon[91579]: pgmap v17215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:02 smithi188 ceph-mon[92425]: pgmap v17215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:04.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:04.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:04 smithi161 ceph-mon[91579]: pgmap v17216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:04 smithi188 ceph-mon[92425]: pgmap v17216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:06 smithi161 ceph-mon[91579]: pgmap v17217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:06 smithi188 ceph-mon[92425]: pgmap v17217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:08 smithi161 ceph-mon[91579]: pgmap v17218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:08 smithi188 ceph-mon[92425]: pgmap v17218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:10 smithi161 ceph-mon[91579]: pgmap v17219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:11.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:10 smithi188 ceph-mon[92425]: pgmap v17219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:12 smithi161 ceph-mon[91579]: pgmap v17220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:12 smithi188 ceph-mon[92425]: pgmap v17220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:14.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:14.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:14 smithi161 ceph-mon[91579]: pgmap v17221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:14 smithi188 ceph-mon[92425]: pgmap v17221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:16 smithi161 ceph-mon[91579]: pgmap v17222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:16 smithi188 ceph-mon[92425]: pgmap v17222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:18 smithi188 ceph-mon[92425]: pgmap v17223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:18 smithi161 ceph-mon[91579]: pgmap v17223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:20 smithi188 ceph-mon[92425]: pgmap v17224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:20 smithi161 ceph-mon[91579]: pgmap v17224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:22 smithi188 ceph-mon[92425]: pgmap v17225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:22 smithi161 ceph-mon[91579]: pgmap v17225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:24.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:25.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:24 smithi188 ceph-mon[92425]: pgmap v17226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:24 smithi161 ceph-mon[91579]: pgmap v17226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:26 smithi188 ceph-mon[92425]: pgmap v17227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:26 smithi161 ceph-mon[91579]: pgmap v17227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:28 smithi161 ceph-mon[91579]: pgmap v17228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:28 smithi188 ceph-mon[92425]: pgmap v17228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:30 smithi161 ceph-mon[91579]: pgmap v17229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:30 smithi188 ceph-mon[92425]: pgmap v17229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:32 smithi161 ceph-mon[91579]: pgmap v17230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:33.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:32 smithi188 ceph-mon[92425]: pgmap v17230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:34 smithi161 ceph-mon[91579]: pgmap v17231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:34 smithi188 ceph-mon[92425]: pgmap v17231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:36 smithi161 ceph-mon[91579]: pgmap v17232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:36 smithi188 ceph-mon[92425]: pgmap v17232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:38 smithi161 ceph-mon[91579]: pgmap v17233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:38 smithi188 ceph-mon[92425]: pgmap v17233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:40 smithi161 ceph-mon[91579]: pgmap v17234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:41.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:40 smithi188 ceph-mon[92425]: pgmap v17234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:42.943 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:13:42.653+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:13:42.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:13:42.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:13:42.944 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:13:42.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:13:42.945 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:13:42.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:13:42.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:13:42.946 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:13:42.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:13:42.947 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:13:42.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:13:42.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:13:42.948 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:13:42.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:13:42.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:13:42.949 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:13:42.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:13:42.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:13:42.950 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:13:42.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:13:42.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:13:42.951 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:13:42.952 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:13:42.952 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:13:42.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:13:42.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:13:42.953 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:13:42.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:13:42.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:13:42.954 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:13:42.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:13:42.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:13:42.955 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:13:42.956 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:13:42.956 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:13:42.956 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:13:42.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:13:42.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:13:42.957 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:13:42.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:13:42.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:13:42.958 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:13:42.959 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:13:42.959 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:13:42.960 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1039] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:13:42.960 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1039] Local address: 172.21.15.161, port 60426 2022-07-03T20:13:42.960 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1039] Peer address: 172.21.15.101, port 22 2022-07-03T20:13:42.961 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1039] Connection lost 2022-07-03T20:13:42.961 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1039] Aborting connection 2022-07-03T20:13:42.961 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:13:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-mon[91579]: pgmap v17235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:13:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:13:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:13:43.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:42 smithi188 ceph-mon[92425]: pgmap v17235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:13:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:13:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:13:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:43 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:13:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:43 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:13:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:13:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:13:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:13:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:13:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:13:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:13:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:13:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:13:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:13:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:13:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:13:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:13:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:13:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:13:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:13:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:13:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:13:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:13:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:13:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:13:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:13:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:13:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:13:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:13:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:13:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:13:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:13:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:13:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:13:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:13:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:13:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:13:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:13:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:13:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:13:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:13:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:13:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:13:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:13:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:13:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:13:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:13:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:13:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:13:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1039] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:13:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1039] Local address: 172.21.15.161, port 60426 2022-07-03T20:13:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1039] Peer address: 172.21.15.101, port 22 2022-07-03T20:13:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1039] Connection lost 2022-07-03T20:13:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1039] Aborting connection 2022-07-03T20:13:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:43 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:13:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:43 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:13:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:13:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:13:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:13:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:13:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:13:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:13:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:13:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:13:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:13:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:13:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:13:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:13:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:13:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:13:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:13:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:13:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:13:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:13:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:13:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:13:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:13:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:13:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:13:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:13:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:13:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:13:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:13:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:13:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:13:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:13:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:13:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:13:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:13:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:13:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:13:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:13:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:13:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:13:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:13:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:13:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:13:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:13:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:13:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:13:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1039] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:13:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1039] Local address: 172.21.15.161, port 60426 2022-07-03T20:13:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1039] Peer address: 172.21.15.101, port 22 2022-07-03T20:13:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1039] Connection lost 2022-07-03T20:13:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1039] Aborting connection 2022-07-03T20:13:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:44 smithi161 ceph-mon[91579]: pgmap v17236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:44 smithi188 ceph-mon[92425]: pgmap v17236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:46 smithi161 ceph-mon[91579]: pgmap v17237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:46 smithi188 ceph-mon[92425]: pgmap v17237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:48 smithi161 ceph-mon[91579]: pgmap v17238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:48 smithi188 ceph-mon[92425]: pgmap v17238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:50 smithi161 ceph-mon[91579]: pgmap v17239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:50 smithi188 ceph-mon[92425]: pgmap v17239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:52 smithi161 ceph-mon[91579]: pgmap v17240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:53.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:52 smithi188 ceph-mon[92425]: pgmap v17240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:54 smithi161 ceph-mon[91579]: pgmap v17241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:54 smithi188 ceph-mon[92425]: pgmap v17241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:13:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:56 smithi161 ceph-mon[91579]: pgmap v17242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:56 smithi188 ceph-mon[92425]: pgmap v17242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:13:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:58 smithi161 ceph-mon[91579]: pgmap v17243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:13:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:58 smithi188 ceph-mon[92425]: pgmap v17243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:13:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:00 smithi161 ceph-mon[91579]: pgmap v17244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:00 smithi188 ceph-mon[92425]: pgmap v17244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:02 smithi161 ceph-mon[91579]: pgmap v17245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:02 smithi188 ceph-mon[92425]: pgmap v17245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:04 smithi161 ceph-mon[91579]: pgmap v17246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:05 smithi188 ceph-mon[92425]: pgmap v17246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:07 smithi161 ceph-mon[91579]: pgmap v17247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:07 smithi188 ceph-mon[92425]: pgmap v17247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:09 smithi161 ceph-mon[91579]: pgmap v17248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:09 smithi188 ceph-mon[92425]: pgmap v17248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:11 smithi161 ceph-mon[91579]: pgmap v17249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:11 smithi188 ceph-mon[92425]: pgmap v17249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:13 smithi161 ceph-mon[91579]: pgmap v17250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:13 smithi188 ceph-mon[92425]: pgmap v17250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:14.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:15 smithi161 ceph-mon[91579]: pgmap v17251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:15 smithi188 ceph-mon[92425]: pgmap v17251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:17 smithi161 ceph-mon[91579]: pgmap v17252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:17 smithi188 ceph-mon[92425]: pgmap v17252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:19 smithi161 ceph-mon[91579]: pgmap v17253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:19 smithi188 ceph-mon[92425]: pgmap v17253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:21 smithi161 ceph-mon[91579]: pgmap v17254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:21.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:21 smithi188 ceph-mon[92425]: pgmap v17254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:22.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:23 smithi161 ceph-mon[91579]: pgmap v17255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:23 smithi188 ceph-mon[92425]: pgmap v17255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:24.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:25 smithi161 ceph-mon[91579]: pgmap v17256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:25 smithi188 ceph-mon[92425]: pgmap v17256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:27 smithi161 ceph-mon[91579]: pgmap v17257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:27 smithi188 ceph-mon[92425]: pgmap v17257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:29 smithi161 ceph-mon[91579]: pgmap v17258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:29 smithi188 ceph-mon[92425]: pgmap v17258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:31 smithi161 ceph-mon[91579]: pgmap v17259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:31 smithi188 ceph-mon[92425]: pgmap v17259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:32.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:33 smithi161 ceph-mon[91579]: pgmap v17260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:33.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:33 smithi188 ceph-mon[92425]: pgmap v17260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:35 smithi161 ceph-mon[91579]: pgmap v17261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:35 smithi188 ceph-mon[92425]: pgmap v17261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:37 smithi161 ceph-mon[91579]: pgmap v17262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:37 smithi188 ceph-mon[92425]: pgmap v17262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:39 smithi161 ceph-mon[91579]: pgmap v17263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:39 smithi188 ceph-mon[92425]: pgmap v17263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:41 smithi161 ceph-mon[91579]: pgmap v17264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:41.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:41 smithi188 ceph-mon[92425]: pgmap v17264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-mon[91579]: pgmap v17265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:43.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:14:43.149+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:14:43.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:14:43.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:14:43.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:14:43.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:14:43.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:14:43.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:14:43.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:14:43.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:14:43.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:14:43.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:14:43.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:14:43.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:14:43.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:14:43.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:14:43.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:14:43.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:14:43.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:14:43.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:14:43.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:14:43.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:14:43.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:14:43.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:14:43.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:14:43.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:14:43.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:14:43.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:14:43.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:14:43.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:14:43.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:14:43.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:14:43.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:14:43.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:14:43.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:14:43.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:14:43.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:14:43.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:14:43.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:14:43.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:14:43.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:14:43.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:14:43.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:14:43.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:14:43.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:14:43.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:14:43.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1041] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:14:43.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1041] Local address: 172.21.15.161, port 60454 2022-07-03T20:14:43.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1041] Peer address: 172.21.15.101, port 22 2022-07-03T20:14:43.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1041] Connection lost 2022-07-03T20:14:43.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1041] Aborting connection 2022-07-03T20:14:43.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:14:43.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:43 smithi188 ceph-mon[92425]: pgmap v17265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:14:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:14:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:14:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:14:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:14:44.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:14:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:14:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:14:44.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:14:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:14:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:14:44.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:14:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:14:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:14:44.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:14:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:14:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:14:44.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:14:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:14:44.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:14:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:14:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:14:44.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:14:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:14:44.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:14:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:14:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:14:44.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:14:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:14:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:14:44.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:14:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:14:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:14:44.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:14:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:14:44.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:14:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:14:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:14:44.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:14:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:14:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:14:44.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:14:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:14:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:14:44.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:14:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:14:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:14:44.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:14:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:14:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1041] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:14:44.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1041] Local address: 172.21.15.161, port 60454 2022-07-03T20:14:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1041] Peer address: 172.21.15.101, port 22 2022-07-03T20:14:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1041] Connection lost 2022-07-03T20:14:44.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1041] Aborting connection 2022-07-03T20:14:44.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:14:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:14:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:14:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:14:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:14:44.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:14:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:14:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:14:44.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:14:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:14:44.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:14:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:14:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:14:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:14:44.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:14:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:14:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:14:44.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:14:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:14:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:14:44.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:14:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:14:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:14:44.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:14:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:14:44.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:14:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:14:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:14:44.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:14:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:14:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:14:44.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:14:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:14:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:14:44.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:14:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:14:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:14:44.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:14:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:14:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:14:44.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:14:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:14:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:14:44.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:14:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:14:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:14:44.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:14:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:14:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:14:44.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1041] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:14:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1041] Local address: 172.21.15.161, port 60454 2022-07-03T20:14:44.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1041] Peer address: 172.21.15.101, port 22 2022-07-03T20:14:44.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1041] Connection lost 2022-07-03T20:14:44.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1041] Aborting connection 2022-07-03T20:14:44.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:45 smithi161 ceph-mon[91579]: pgmap v17266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:45 smithi188 ceph-mon[92425]: pgmap v17266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:47 smithi161 ceph-mon[91579]: pgmap v17267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:14:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:14:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:47 smithi188 ceph-mon[92425]: pgmap v17267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:14:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:14:47.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:48 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T20:14:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:48 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T20:14:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:49 smithi161 ceph-mon[91579]: pgmap v17268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:49 smithi188 ceph-mon[92425]: pgmap v17268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:51 smithi161 ceph-mon[91579]: pgmap v17269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:51 smithi188 ceph-mon[92425]: pgmap v17269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:53 smithi161 ceph-mon[91579]: pgmap v17270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:53.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:53 smithi188 ceph-mon[92425]: pgmap v17270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:55 smithi161 ceph-mon[91579]: pgmap v17271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:55 smithi188 ceph-mon[92425]: pgmap v17271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:14:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:57 smithi188 ceph-mon[92425]: pgmap v17272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:57 smithi161 ceph-mon[91579]: pgmap v17272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:14:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: pgmap v17273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: pgmap v17273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:14:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:14:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:14:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:14:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:15:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:01 smithi188 ceph-mon[92425]: pgmap v17274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:01 smithi161 ceph-mon[91579]: pgmap v17274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:02.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:03 smithi188 ceph-mon[92425]: pgmap v17275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:03 smithi161 ceph-mon[91579]: pgmap v17275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:05 smithi188 ceph-mon[92425]: pgmap v17276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:05 smithi161 ceph-mon[91579]: pgmap v17276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:07 smithi188 ceph-mon[92425]: pgmap v17277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:07 smithi161 ceph-mon[91579]: pgmap v17277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:09 smithi188 ceph-mon[92425]: pgmap v17278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:09 smithi161 ceph-mon[91579]: pgmap v17278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:11 smithi161 ceph-mon[91579]: pgmap v17279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:11 smithi188 ceph-mon[92425]: pgmap v17279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:13 smithi161 ceph-mon[91579]: pgmap v17280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:13 smithi188 ceph-mon[92425]: pgmap v17280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:14.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:15 smithi161 ceph-mon[91579]: pgmap v17281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:15 smithi188 ceph-mon[92425]: pgmap v17281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:17 smithi161 ceph-mon[91579]: pgmap v17282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:17 smithi188 ceph-mon[92425]: pgmap v17282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:19 smithi161 ceph-mon[91579]: pgmap v17283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:19 smithi188 ceph-mon[92425]: pgmap v17283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:21 smithi161 ceph-mon[91579]: pgmap v17284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:21 smithi188 ceph-mon[92425]: pgmap v17284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:23 smithi161 ceph-mon[91579]: pgmap v17285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:23 smithi188 ceph-mon[92425]: pgmap v17285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:25 smithi161 ceph-mon[91579]: pgmap v17286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:25 smithi188 ceph-mon[92425]: pgmap v17286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:27 smithi161 ceph-mon[91579]: pgmap v17287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:27 smithi188 ceph-mon[92425]: pgmap v17287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:29 smithi161 ceph-mon[91579]: pgmap v17288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:29 smithi188 ceph-mon[92425]: pgmap v17288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:31 smithi161 ceph-mon[91579]: pgmap v17289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:31 smithi188 ceph-mon[92425]: pgmap v17289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:33 smithi161 ceph-mon[91579]: pgmap v17290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:33 smithi188 ceph-mon[92425]: pgmap v17290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:35 smithi161 ceph-mon[91579]: pgmap v17291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:35 smithi188 ceph-mon[92425]: pgmap v17291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:37 smithi161 ceph-mon[91579]: pgmap v17292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:37 smithi188 ceph-mon[92425]: pgmap v17292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:39 smithi161 ceph-mon[91579]: pgmap v17293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:39 smithi188 ceph-mon[92425]: pgmap v17293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:41 smithi161 ceph-mon[91579]: pgmap v17294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:41 smithi188 ceph-mon[92425]: pgmap v17294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:43 smithi161 ceph-mon[91579]: pgmap v17295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:43 smithi188 ceph-mon[92425]: pgmap v17295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:44.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:45 smithi161 ceph-mon[91579]: pgmap v17296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:45 smithi188 ceph-mon[92425]: pgmap v17296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:46.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:15:46.528+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:15:46.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:15:46.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:15:46.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:15:46.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:15:46.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:15:46.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:15:46.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:15:46.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:15:46.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:15:46.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:15:46.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:15:46.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:15:46.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:15:46.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:15:46.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:15:46.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:15:46.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:15:46.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:15:46.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:15:46.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:15:46.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:15:46.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:15:46.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:15:46.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:15:46.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:15:46.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:15:46.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:15:46.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:15:46.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:15:46.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:15:46.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:15:46.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:15:46.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:15:46.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:15:46.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:15:46.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:15:46.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:15:46.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:15:46.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:15:46.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:15:46.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:15:46.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:15:46.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:15:46.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:15:46.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1043] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:15:46.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1043] Local address: 172.21.15.161, port 60486 2022-07-03T20:15:46.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1043] Peer address: 172.21.15.101, port 22 2022-07-03T20:15:46.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1043] Connection lost 2022-07-03T20:15:46.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1043] Aborting connection 2022-07-03T20:15:46.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:15:46 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:15:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:47 smithi161 ceph-mon[91579]: pgmap v17297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:15:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:15:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:15:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:47 smithi188 ceph-mon[92425]: pgmap v17297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:15:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:15:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:15:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:15:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:15:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:15:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:15:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:15:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:15:48.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:15:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:15:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:15:48.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:15:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:15:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:15:48.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:15:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:15:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:15:48.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:15:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:15:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:15:48.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:15:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:15:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:15:48.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:15:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:15:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:15:48.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:15:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:15:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:15:48.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:15:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:15:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:15:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:15:48.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:15:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:15:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:15:48.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:15:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:15:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:15:48.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:15:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:15:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:15:48.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:15:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:15:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:15:48.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:15:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:15:48.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:15:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1043] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:15:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1043] Local address: 172.21.15.161, port 60486 2022-07-03T20:15:48.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1043] Peer address: 172.21.15.101, port 22 2022-07-03T20:15:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1043] Connection lost 2022-07-03T20:15:48.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1043] Aborting connection 2022-07-03T20:15:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:15:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:15:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:15:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:15:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:15:48.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:15:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:15:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:15:48.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:15:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:15:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:15:48.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:15:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:15:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:15:48.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:15:48.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:15:48.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:15:48.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:15:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:15:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:15:48.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:15:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:15:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:15:48.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:15:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:15:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:15:48.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:15:48.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:15:48.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:15:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:15:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:15:48.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:15:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:15:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:15:48.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:15:48.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:15:48.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:15:48.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:15:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:15:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:15:48.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:15:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:15:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:15:48.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:15:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:15:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:15:48.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1043] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:15:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1043] Local address: 172.21.15.161, port 60486 2022-07-03T20:15:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1043] Peer address: 172.21.15.101, port 22 2022-07-03T20:15:48.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1043] Connection lost 2022-07-03T20:15:48.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1043] Aborting connection 2022-07-03T20:15:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:49 smithi161 ceph-mon[91579]: pgmap v17298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:49 smithi188 ceph-mon[92425]: pgmap v17298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:50.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:50 smithi161 ceph-mon[91579]: pgmap v17299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:50 smithi188 ceph-mon[92425]: pgmap v17299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:52.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:52 smithi161 ceph-mon[91579]: pgmap v17300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:52.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:52 smithi188 ceph-mon[92425]: pgmap v17300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:54.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:54 smithi161 ceph-mon[91579]: pgmap v17301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:54.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:54 smithi188 ceph-mon[92425]: pgmap v17301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:15:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:56.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:56 smithi161 ceph-mon[91579]: pgmap v17302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:56.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:56 smithi188 ceph-mon[92425]: pgmap v17302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:15:58.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:58 smithi161 ceph-mon[91579]: pgmap v17303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:58.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:58 smithi188 ceph-mon[92425]: pgmap v17303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:15:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:15:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:15:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:15:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:15:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:15:59.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:15:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:15:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:15:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:15:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:00.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:00 smithi161 ceph-mon[91579]: pgmap v17304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:00.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:00 smithi188 ceph-mon[92425]: pgmap v17304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:02.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:02 smithi161 ceph-mon[91579]: pgmap v17305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:02.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:02 smithi188 ceph-mon[92425]: pgmap v17305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:04.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:04 smithi161 ceph-mon[91579]: pgmap v17306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:04.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:04 smithi188 ceph-mon[92425]: pgmap v17306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:06.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:06 smithi161 ceph-mon[91579]: pgmap v17307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:06.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:06 smithi188 ceph-mon[92425]: pgmap v17307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:07.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:07.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:08.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:08 smithi161 ceph-mon[91579]: pgmap v17308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:08.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:08 smithi188 ceph-mon[92425]: pgmap v17308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:09.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:09.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:10 smithi161 ceph-mon[91579]: pgmap v17309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:10.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:10 smithi188 ceph-mon[92425]: pgmap v17309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:12.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:12 smithi161 ceph-mon[91579]: pgmap v17310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:12.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:12.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:12 smithi188 ceph-mon[92425]: pgmap v17310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:14.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:14 smithi161 ceph-mon[91579]: pgmap v17311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:14.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:14 smithi188 ceph-mon[92425]: pgmap v17311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:16.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:16 smithi161 ceph-mon[91579]: pgmap v17312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:16.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:16 smithi188 ceph-mon[92425]: pgmap v17312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:17.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:17.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:18 smithi161 ceph-mon[91579]: pgmap v17313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:18.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:18 smithi188 ceph-mon[92425]: pgmap v17313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:19.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:19.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:20.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:20 smithi161 ceph-mon[91579]: pgmap v17314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:20.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:20 smithi188 ceph-mon[92425]: pgmap v17314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:22.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:22 smithi161 ceph-mon[91579]: pgmap v17315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:22.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:22.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:22 smithi188 ceph-mon[92425]: pgmap v17315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:22.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:24.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:24 smithi161 ceph-mon[91579]: pgmap v17316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:24.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:24 smithi188 ceph-mon[92425]: pgmap v17316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:26.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:26 smithi161 ceph-mon[91579]: pgmap v17317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:26.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:26 smithi188 ceph-mon[92425]: pgmap v17317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:27.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:27.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:28 smithi161 ceph-mon[91579]: pgmap v17318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:28.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:28 smithi188 ceph-mon[92425]: pgmap v17318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:29.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:29.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:30.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:30 smithi161 ceph-mon[91579]: pgmap v17319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:30.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:30 smithi188 ceph-mon[92425]: pgmap v17319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:32.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:32 smithi161 ceph-mon[91579]: pgmap v17320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:32.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:32 smithi188 ceph-mon[92425]: pgmap v17320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:34.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:34 smithi161 ceph-mon[91579]: pgmap v17321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:34.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:34 smithi188 ceph-mon[92425]: pgmap v17321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:36.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:36 smithi161 ceph-mon[91579]: pgmap v17322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:36.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:36 smithi188 ceph-mon[92425]: pgmap v17322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:37.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:38.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:38 smithi188 ceph-mon[92425]: pgmap v17323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:39.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:38 smithi161 ceph-mon[91579]: pgmap v17323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:40.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:40 smithi188 ceph-mon[92425]: pgmap v17324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:40 smithi161 ceph-mon[91579]: pgmap v17324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:42.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:42 smithi188 ceph-mon[92425]: pgmap v17325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:42.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:43.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:42 smithi161 ceph-mon[91579]: pgmap v17325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:44.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:44 smithi188 ceph-mon[92425]: pgmap v17326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:44 smithi161 ceph-mon[91579]: pgmap v17326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:46 smithi188 ceph-mon[92425]: pgmap v17327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:47.031 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:46 smithi161 ceph-mon[91579]: pgmap v17327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:47.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:16:47.030+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:16:47.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:16:47.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:16:47.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:16:47.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:16:47.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:16:47.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:16:47.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:16:47.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:16:47.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:16:47.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:16:47.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:16:47.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:16:47.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:16:47.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:16:47.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:16:47.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:16:47.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:16:47.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:16:47.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:16:47.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:16:47.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:16:47.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:16:47.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:16:47.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:16:47.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:16:47.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:16:47.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:16:47.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:16:47.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:16:47.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:16:47.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:16:47.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:16:47.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:16:47.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:16:47.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:16:47.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:16:47.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:16:47.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:16:47.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:16:47.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:16:47.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:16:47.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:16:47.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:16:47.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:16:47.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1045] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:16:47.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1045] Local address: 172.21.15.161, port 60514 2022-07-03T20:16:47.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1045] Peer address: 172.21.15.101, port 22 2022-07-03T20:16:47.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1045] Connection lost 2022-07-03T20:16:47.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1045] Aborting connection 2022-07-03T20:16:47.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:16:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:16:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:16:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:16:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:16:47.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:16:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:16:47.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:16:47.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:16:47.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:16:47.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:16:47.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:16:47.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:16:47.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:16:47.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:16:47.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:16:47.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:16:47.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:16:47.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:16:47.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:16:47.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:16:47.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:16:47.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:16:47.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:16:47.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:16:47.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:16:47.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:16:47.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:16:47.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:16:47.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:16:47.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:16:47.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:16:47.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:16:47.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:16:47.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:16:47.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:16:47.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:16:47.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:16:47.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:16:47.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:16:47.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:16:47.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:16:47.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:16:47.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:16:47.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:16:47.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:16:47.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:16:47.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:16:47.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:16:47.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:16:47.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1045] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:16:47.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1045] Local address: 172.21.15.161, port 60514 2022-07-03T20:16:47.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1045] Peer address: 172.21.15.101, port 22 2022-07-03T20:16:47.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1045] Connection lost 2022-07-03T20:16:47.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1045] Aborting connection 2022-07-03T20:16:47.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:16:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:16:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:16:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:16:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:16:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:16:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:16:48.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:16:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:16:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:16:48.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:16:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:16:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:16:48.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:16:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:16:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:16:48.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:16:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:16:48.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:16:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:16:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:16:48.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:16:48.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:16:48.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:16:48.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:16:48.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:16:48.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:16:48.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:16:48.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:16:48.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:16:48.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:16:48.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:16:48.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:16:48.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:16:48.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:16:48.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:16:48.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:16:48.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:16:48.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:16:48.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:16:48.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:16:48.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:16:48.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:16:48.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:16:48.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:16:48.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:16:48.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:16:48.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:16:48.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:16:48.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1045] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:16:48.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1045] Local address: 172.21.15.161, port 60514 2022-07-03T20:16:48.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1045] Peer address: 172.21.15.101, port 22 2022-07-03T20:16:48.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1045] Connection lost 2022-07-03T20:16:48.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1045] Aborting connection 2022-07-03T20:16:48.156 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:48 smithi188 ceph-mon[92425]: pgmap v17328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:48 smithi161 ceph-mon[91579]: pgmap v17328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:50.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:50 smithi188 ceph-mon[92425]: pgmap v17329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:50 smithi161 ceph-mon[91579]: pgmap v17329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:52 smithi161 ceph-mon[91579]: pgmap v17330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:53.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:52 smithi188 ceph-mon[92425]: pgmap v17330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:54 smithi161 ceph-mon[91579]: pgmap v17331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:55.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:54 smithi188 ceph-mon[92425]: pgmap v17331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:16:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:16:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:56 smithi161 ceph-mon[91579]: pgmap v17332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:56 smithi188 ceph-mon[92425]: pgmap v17332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:16:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:58 smithi161 ceph-mon[91579]: pgmap v17333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:16:59.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:58 smithi188 ceph-mon[92425]: pgmap v17333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:17:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:17:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:17:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:17:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:17:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:17:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:17:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:17:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:00 smithi161 ceph-mon[91579]: pgmap v17334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:00 smithi188 ceph-mon[92425]: pgmap v17334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:03.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:02 smithi161 ceph-mon[91579]: pgmap v17335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:02 smithi188 ceph-mon[92425]: pgmap v17335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:04 smithi161 ceph-mon[91579]: pgmap v17336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:04 smithi188 ceph-mon[92425]: pgmap v17336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:06 smithi161 ceph-mon[91579]: pgmap v17337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:06 smithi188 ceph-mon[92425]: pgmap v17337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:08 smithi161 ceph-mon[91579]: pgmap v17338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:08 smithi188 ceph-mon[92425]: pgmap v17338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:10.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:10.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:10 smithi161 ceph-mon[91579]: pgmap v17339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:11.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:10 smithi188 ceph-mon[92425]: pgmap v17339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:12 smithi161 ceph-mon[91579]: pgmap v17340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:12 smithi188 ceph-mon[92425]: pgmap v17340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:14 smithi161 ceph-mon[91579]: pgmap v17341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:14 smithi188 ceph-mon[92425]: pgmap v17341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:16 smithi161 ceph-mon[91579]: pgmap v17342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:16 smithi188 ceph-mon[92425]: pgmap v17342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:18 smithi161 ceph-mon[91579]: pgmap v17343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:18 smithi188 ceph-mon[92425]: pgmap v17343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:21.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:20 smithi161 ceph-mon[91579]: pgmap v17344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:20 smithi188 ceph-mon[92425]: pgmap v17344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:23.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:22 smithi161 ceph-mon[91579]: pgmap v17345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:23.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:22 smithi188 ceph-mon[92425]: pgmap v17345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:24 smithi161 ceph-mon[91579]: pgmap v17346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:25.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:24 smithi188 ceph-mon[92425]: pgmap v17346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:26 smithi161 ceph-mon[91579]: pgmap v17347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:26 smithi188 ceph-mon[92425]: pgmap v17347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:28 smithi161 ceph-mon[91579]: pgmap v17348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:28 smithi188 ceph-mon[92425]: pgmap v17348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:30.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:31.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:30 smithi161 ceph-mon[91579]: pgmap v17349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:30 smithi188 ceph-mon[92425]: pgmap v17349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:33.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:32 smithi161 ceph-mon[91579]: pgmap v17350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:33.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:32 smithi188 ceph-mon[92425]: pgmap v17350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:34 smithi161 ceph-mon[91579]: pgmap v17351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:35.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:34 smithi188 ceph-mon[92425]: pgmap v17351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:36 smithi161 ceph-mon[91579]: pgmap v17352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:36 smithi188 ceph-mon[92425]: pgmap v17352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:39.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:38 smithi161 ceph-mon[91579]: pgmap v17353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:38 smithi188 ceph-mon[92425]: pgmap v17353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:40.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:40 smithi161 ceph-mon[91579]: pgmap v17354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:41.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:40 smithi188 ceph-mon[92425]: pgmap v17354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:43.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:42 smithi161 ceph-mon[91579]: pgmap v17355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:43.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:42 smithi188 ceph-mon[92425]: pgmap v17355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:44 smithi161 ceph-mon[91579]: pgmap v17356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:45.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:44 smithi188 ceph-mon[92425]: pgmap v17356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:46 smithi161 ceph-mon[91579]: pgmap v17357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:46 smithi188 ceph-mon[92425]: pgmap v17357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:47.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:17:47.535+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:17:47.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:17:47.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:17:47.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:17:47.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:17:47.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:17:47.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:17:47.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:17:47.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:17:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:17:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:17:47.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:17:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:17:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:17:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:17:47.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:17:47.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:17:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:17:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:17:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:17:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:17:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:17:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:17:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:17:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:17:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:17:47.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:17:47.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:17:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:17:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:17:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:17:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:17:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:17:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:17:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:17:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:17:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:17:47.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:17:47.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:17:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:17:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:17:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:17:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:17:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:17:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:17:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:17:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:17:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:17:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1047] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:17:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1047] Local address: 172.21.15.161, port 60542 2022-07-03T20:17:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1047] Peer address: 172.21.15.101, port 22 2022-07-03T20:17:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1047] Connection lost 2022-07-03T20:17:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1047] Aborting connection 2022-07-03T20:17:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:17:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:17:47.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:17:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:17:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:17:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:48 smithi161 ceph-mon[91579]: pgmap v17358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:17:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:17:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:17:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:17:49.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:17:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:17:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:17:49.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:17:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:17:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:17:49.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:17:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:17:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:17:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:17:49.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:17:49.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:17:49.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:17:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:17:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:17:49.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:17:49.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:17:49.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:17:49.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:17:49.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:17:49.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:17:49.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:17:49.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:17:49.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:17:49.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:17:49.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:17:49.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:17:49.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:17:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:17:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:17:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:17:49.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:17:49.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:17:49.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:17:49.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:17:49.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:17:49.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:17:49.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:17:49.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:17:49.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:17:49.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:17:49.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:17:49.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1047] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:17:49.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1047] Local address: 172.21.15.161, port 60542 2022-07-03T20:17:49.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1047] Peer address: 172.21.15.101, port 22 2022-07-03T20:17:49.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1047] Connection lost 2022-07-03T20:17:49.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1047] Aborting connection 2022-07-03T20:17:49.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:48 smithi188 ceph-mon[92425]: pgmap v17358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:17:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:17:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:17:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:17:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:17:49.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:17:49.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:17:49.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:17:49.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:17:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:17:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:17:49.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:17:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:17:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:17:49.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:17:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:17:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:17:49.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:17:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:17:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:17:49.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:17:49.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:17:49.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:17:49.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:17:49.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:17:49.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:17:49.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:17:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:17:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:17:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:17:49.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:17:49.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:17:49.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:17:49.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:17:49.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:17:49.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:17:49.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:17:49.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:17:49.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:17:49.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:17:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:17:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:17:49.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:17:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:17:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:17:49.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:17:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1047] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:17:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1047] Local address: 172.21.15.161, port 60542 2022-07-03T20:17:49.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1047] Peer address: 172.21.15.101, port 22 2022-07-03T20:17:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1047] Connection lost 2022-07-03T20:17:49.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1047] Aborting connection 2022-07-03T20:17:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:50.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:50 smithi161 ceph-mon[91579]: pgmap v17359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:51.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:50 smithi188 ceph-mon[92425]: pgmap v17359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:52 smithi161 ceph-mon[91579]: pgmap v17360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:53.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:52 smithi188 ceph-mon[92425]: pgmap v17360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:54 smithi161 ceph-mon[91579]: pgmap v17361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:55.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:54 smithi188 ceph-mon[92425]: pgmap v17361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:17:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:17:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:56 smithi161 ceph-mon[91579]: pgmap v17362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:56 smithi188 ceph-mon[92425]: pgmap v17362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:17:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:58 smithi161 ceph-mon[91579]: pgmap v17363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:17:59.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:58 smithi188 ceph-mon[92425]: pgmap v17363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:18:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:18:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:18:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:18:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:18:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:18:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:18:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:18:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:00 smithi161 ceph-mon[91579]: pgmap v17364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:00 smithi188 ceph-mon[92425]: pgmap v17364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:03.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:02 smithi161 ceph-mon[91579]: pgmap v17365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:02 smithi188 ceph-mon[92425]: pgmap v17365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:04 smithi161 ceph-mon[91579]: pgmap v17366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:04 smithi188 ceph-mon[92425]: pgmap v17366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:06 smithi161 ceph-mon[91579]: pgmap v17367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:06 smithi188 ceph-mon[92425]: pgmap v17367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:08 smithi161 ceph-mon[91579]: pgmap v17368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:08 smithi188 ceph-mon[92425]: pgmap v17368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:10.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:10.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:10 smithi161 ceph-mon[91579]: pgmap v17369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:11.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:10 smithi188 ceph-mon[92425]: pgmap v17369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:12 smithi161 ceph-mon[91579]: pgmap v17370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:12 smithi188 ceph-mon[92425]: pgmap v17370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:14 smithi161 ceph-mon[91579]: pgmap v17371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:14 smithi188 ceph-mon[92425]: pgmap v17371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:16 smithi161 ceph-mon[91579]: pgmap v17372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:16 smithi188 ceph-mon[92425]: pgmap v17372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:18 smithi161 ceph-mon[91579]: pgmap v17373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:18 smithi188 ceph-mon[92425]: pgmap v17373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:21.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:20 smithi161 ceph-mon[91579]: pgmap v17374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:20 smithi188 ceph-mon[92425]: pgmap v17374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:23.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:22 smithi161 ceph-mon[91579]: pgmap v17375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:23.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:22 smithi188 ceph-mon[92425]: pgmap v17375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:24 smithi161 ceph-mon[91579]: pgmap v17376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:25.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:24 smithi188 ceph-mon[92425]: pgmap v17376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:26 smithi161 ceph-mon[91579]: pgmap v17377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:26 smithi188 ceph-mon[92425]: pgmap v17377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:28 smithi161 ceph-mon[91579]: pgmap v17378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:28 smithi188 ceph-mon[92425]: pgmap v17378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:30.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:30 smithi188 ceph-mon[92425]: pgmap v17379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:30 smithi161 ceph-mon[91579]: pgmap v17379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:33.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:32 smithi188 ceph-mon[92425]: pgmap v17380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:32 smithi161 ceph-mon[91579]: pgmap v17380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:35.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:34 smithi188 ceph-mon[92425]: pgmap v17381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:34 smithi161 ceph-mon[91579]: pgmap v17381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:37.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:36 smithi188 ceph-mon[92425]: pgmap v17382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:36 smithi161 ceph-mon[91579]: pgmap v17382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:38 smithi188 ceph-mon[92425]: pgmap v17383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:38 smithi161 ceph-mon[91579]: pgmap v17383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:40.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:40 smithi188 ceph-mon[92425]: pgmap v17384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:40 smithi161 ceph-mon[91579]: pgmap v17384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:43.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:42 smithi188 ceph-mon[92425]: pgmap v17385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:42 smithi161 ceph-mon[91579]: pgmap v17385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:44 smithi161 ceph-mon[91579]: pgmap v17386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:44 smithi188 ceph-mon[92425]: pgmap v17386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:46 smithi161 ceph-mon[91579]: pgmap v17387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:46 smithi188 ceph-mon[92425]: pgmap v17387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:48.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:18:48.039+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:18:48.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:18:48.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:18:48.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:18:48.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:18:48.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:18:48.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:18:48.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:18:48.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:18:48.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:18:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:18:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:18:48.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:18:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:18:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:18:48.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:18:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:18:48.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:18:48.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:18:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:18:48.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:18:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:18:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:18:48.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:18:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:18:48.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:18:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:18:48.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:18:48.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:18:48.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:18:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:18:48.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:18:48.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:18:48.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:18:48.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:18:48.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:18:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:18:48.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:18:48.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:18:48.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:18:48.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:18:48.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:18:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:18:48.401 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:18:48.402 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:18:48.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1049] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:18:48.403 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1049] Local address: 172.21.15.161, port 60570 2022-07-03T20:18:48.404 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1049] Peer address: 172.21.15.101, port 22 2022-07-03T20:18:48.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1049] Connection lost 2022-07-03T20:18:48.405 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1049] Aborting connection 2022-07-03T20:18:48.406 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:18:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: pgmap v17388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:18:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:18:49.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:18:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:18:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:18:49.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:18:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:18:49.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:18:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:18:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:18:49.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:18:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:18:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:18:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:18:49.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:18:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:18:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:18:49.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:18:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:18:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:18:49.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:18:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:18:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:18:49.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:18:49.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:18:49.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:18:49.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:18:49.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:18:49.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:18:49.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:18:49.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:18:49.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:18:49.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:18:49.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:18:49.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:18:49.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:18:49.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:18:49.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:18:49.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:18:49.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:18:49.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:18:49.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:18:49.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:18:49.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:18:49.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:18:49.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:18:49.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:18:49.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:18:49.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:18:49.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1049] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:18:49.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1049] Local address: 172.21.15.161, port 60570 2022-07-03T20:18:49.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1049] Peer address: 172.21.15.101, port 22 2022-07-03T20:18:49.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1049] Connection lost 2022-07-03T20:18:49.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1049] Aborting connection 2022-07-03T20:18:49.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: pgmap v17388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:18:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:18:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:18:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:18:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:18:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:18:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:18:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:18:49.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:18:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:18:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:18:49.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:18:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:18:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:18:49.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:18:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:18:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:18:49.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:18:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:18:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:18:49.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:18:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:18:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:18:49.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:18:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:18:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:18:49.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:18:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:18:49.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:18:49.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:18:49.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:18:49.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:18:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:18:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:18:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:18:49.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:18:49.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:18:49.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:18:49.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:18:49.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:18:49.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:18:49.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:18:49.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:18:49.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:18:49.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:18:49.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:18:49.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:18:49.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:18:49.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:18:49.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1049] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:18:49.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1049] Local address: 172.21.15.161, port 60570 2022-07-03T20:18:49.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1049] Peer address: 172.21.15.101, port 22 2022-07-03T20:18:49.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1049] Connection lost 2022-07-03T20:18:49.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1049] Aborting connection 2022-07-03T20:18:49.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:49.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:50.071 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:50 smithi161 ceph-mon[91579]: pgmap v17389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:50 smithi188 ceph-mon[92425]: pgmap v17389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:53.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:52 smithi188 ceph-mon[92425]: pgmap v17390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:52 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T20:18:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:52 smithi161 ceph-mon[91579]: pgmap v17390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:52 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T20:18:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:18:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:55.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:54 smithi188 ceph-mon[92425]: pgmap v17391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:54 smithi161 ceph-mon[91579]: pgmap v17391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:18:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:18:57.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:56 smithi188 ceph-mon[92425]: pgmap v17392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:56 smithi161 ceph-mon[91579]: pgmap v17392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:18:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:58 smithi161 ceph-mon[91579]: pgmap v17393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:18:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:58 smithi188 ceph-mon[92425]: pgmap v17393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:00 smithi161 ceph-mon[91579]: pgmap v17394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:00 smithi188 ceph-mon[92425]: pgmap v17394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:02 smithi161 ceph-mon[91579]: pgmap v17395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:02 smithi188 ceph-mon[92425]: pgmap v17395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:04 smithi161 ceph-mon[91579]: pgmap v17396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:04 smithi188 ceph-mon[92425]: pgmap v17396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:06 smithi161 ceph-mon[91579]: pgmap v17397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:06 smithi188 ceph-mon[92425]: pgmap v17397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:08 smithi161 ceph-mon[91579]: pgmap v17398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:08 smithi188 ceph-mon[92425]: pgmap v17398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:10 smithi161 ceph-mon[91579]: pgmap v17399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:10 smithi188 ceph-mon[92425]: pgmap v17399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:12 smithi161 ceph-mon[91579]: pgmap v17400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:12 smithi188 ceph-mon[92425]: pgmap v17400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:14 smithi161 ceph-mon[91579]: pgmap v17401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:14 smithi188 ceph-mon[92425]: pgmap v17401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:16 smithi161 ceph-mon[91579]: pgmap v17402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:16 smithi188 ceph-mon[92425]: pgmap v17402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:18 smithi161 ceph-mon[91579]: pgmap v17403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:18 smithi188 ceph-mon[92425]: pgmap v17403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:20 smithi161 ceph-mon[91579]: pgmap v17404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:21.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:20 smithi188 ceph-mon[92425]: pgmap v17404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:23.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:22 smithi161 ceph-mon[91579]: pgmap v17405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:22 smithi188 ceph-mon[92425]: pgmap v17405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:24 smithi161 ceph-mon[91579]: pgmap v17406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:24 smithi188 ceph-mon[92425]: pgmap v17406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:26 smithi161 ceph-mon[91579]: pgmap v17407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:26 smithi188 ceph-mon[92425]: pgmap v17407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:28 smithi161 ceph-mon[91579]: pgmap v17408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:28 smithi188 ceph-mon[92425]: pgmap v17408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:31 smithi161 ceph-mon[91579]: pgmap v17409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:31 smithi188 ceph-mon[92425]: pgmap v17409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:33 smithi161 ceph-mon[91579]: pgmap v17410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:33.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:33 smithi188 ceph-mon[92425]: pgmap v17410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:35 smithi161 ceph-mon[91579]: pgmap v17411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:35 smithi188 ceph-mon[92425]: pgmap v17411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:37 smithi161 ceph-mon[91579]: pgmap v17412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:37.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:37 smithi188 ceph-mon[92425]: pgmap v17412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:39 smithi161 ceph-mon[91579]: pgmap v17413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:39.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:39 smithi188 ceph-mon[92425]: pgmap v17413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:41 smithi161 ceph-mon[91579]: pgmap v17414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:41.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:41 smithi188 ceph-mon[92425]: pgmap v17414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:43 smithi161 ceph-mon[91579]: pgmap v17415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:43.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:43 smithi188 ceph-mon[92425]: pgmap v17415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:45 smithi161 ceph-mon[91579]: pgmap v17416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:45 smithi188 ceph-mon[92425]: pgmap v17416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:47 smithi161 ceph-mon[91579]: pgmap v17417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:47 smithi188 ceph-mon[92425]: pgmap v17417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:49 smithi161 ceph-mon[91579]: pgmap v17418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:49 smithi188 ceph-mon[92425]: pgmap v17418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-mon[91579]: pgmap v17419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:51 smithi188 ceph-mon[92425]: pgmap v17419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:19:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:19:52.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:19:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:19:51.939+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:19:52.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:19:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:19:52.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:19:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:19:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:19:52.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:19:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:19:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:19:52.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:19:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:19:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:19:52.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:19:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:19:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:19:52.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:19:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:19:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:19:52.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:19:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:19:52.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:19:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:19:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:19:52.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:19:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:19:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:19:52.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:19:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:19:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:19:52.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:19:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:19:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:19:52.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:19:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:19:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:19:52.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:19:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:19:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:19:52.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:19:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:19:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:19:52.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:19:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:19:52.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:19:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:19:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1051] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:19:52.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1051] Local address: 172.21.15.161, port 60600 2022-07-03T20:19:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1051] Peer address: 172.21.15.101, port 22 2022-07-03T20:19:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1051] Connection lost 2022-07-03T20:19:52.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1051] Aborting connection 2022-07-03T20:19:52.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:19:51 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:19:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:19:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:19:52.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:19:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: pgmap v17420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:19:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:19:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:19:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:19:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:19:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:19:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:19:53.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:19:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:19:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:19:53.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:19:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:19:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:19:53.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:19:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:19:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:19:53.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:19:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:19:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:19:53.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:19:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:19:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:19:53.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:19:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:19:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:19:53.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:19:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:19:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:19:53.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:19:53.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:19:53.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:19:53.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:19:53.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:19:53.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:19:53.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:19:53.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:19:53.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:19:53.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:19:53.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:19:53.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:19:53.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:19:53.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:19:53.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:19:53.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:19:53.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:19:53.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:19:53.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1051] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:19:53.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1051] Local address: 172.21.15.161, port 60600 2022-07-03T20:19:53.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1051] Peer address: 172.21.15.101, port 22 2022-07-03T20:19:53.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1051] Connection lost 2022-07-03T20:19:53.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1051] Aborting connection 2022-07-03T20:19:53.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:53.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: pgmap v17420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:19:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:19:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:19:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:19:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:19:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:19:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:19:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:19:53.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:19:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:19:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:19:53.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:19:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:19:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:19:53.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:19:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:19:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:19:53.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:19:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:19:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:19:53.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:19:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:19:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:19:53.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:19:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:19:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:19:53.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:19:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:19:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:19:53.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:19:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:19:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:19:53.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:19:53.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:19:53.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:19:53.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:19:53.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:19:53.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:19:53.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:19:53.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:19:53.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:19:53.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:19:53.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:19:53.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:19:53.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:19:53.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:19:53.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1051] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:19:53.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1051] Local address: 172.21.15.161, port 60600 2022-07-03T20:19:53.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1051] Peer address: 172.21.15.101, port 22 2022-07-03T20:19:53.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1051] Connection lost 2022-07-03T20:19:53.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1051] Aborting connection 2022-07-03T20:19:53.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:53.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:53.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:19:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:55 smithi161 ceph-mon[91579]: pgmap v17421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:55 smithi188 ceph-mon[92425]: pgmap v17421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:19:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:19:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:57 smithi161 ceph-mon[91579]: pgmap v17422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:57 smithi188 ceph-mon[92425]: pgmap v17422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:19:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:59 smithi161 ceph-mon[91579]: pgmap v17423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:59 smithi188 ceph-mon[92425]: pgmap v17423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:19:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:19:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:19:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:20:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:20:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:20:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:01 smithi161 ceph-mon[91579]: pgmap v17424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:01 smithi188 ceph-mon[92425]: pgmap v17424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:03 smithi161 ceph-mon[91579]: pgmap v17425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:03.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:03 smithi188 ceph-mon[92425]: pgmap v17425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:05 smithi161 ceph-mon[91579]: pgmap v17426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:05 smithi188 ceph-mon[92425]: pgmap v17426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:07 smithi161 ceph-mon[91579]: pgmap v17427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:07 smithi188 ceph-mon[92425]: pgmap v17427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:09 smithi161 ceph-mon[91579]: pgmap v17428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:09 smithi188 ceph-mon[92425]: pgmap v17428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:11 smithi161 ceph-mon[91579]: pgmap v17429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:11 smithi188 ceph-mon[92425]: pgmap v17429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:13.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:13 smithi161 ceph-mon[91579]: pgmap v17430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:13 smithi188 ceph-mon[92425]: pgmap v17430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:15 smithi161 ceph-mon[91579]: pgmap v17431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:15 smithi188 ceph-mon[92425]: pgmap v17431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:17.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:17 smithi161 ceph-mon[91579]: pgmap v17432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:17 smithi188 ceph-mon[92425]: pgmap v17432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:19.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:19 smithi161 ceph-mon[91579]: pgmap v17433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:19 smithi188 ceph-mon[92425]: pgmap v17433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:21.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:21 smithi188 ceph-mon[92425]: pgmap v17434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:21 smithi161 ceph-mon[91579]: pgmap v17434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:23 smithi188 ceph-mon[92425]: pgmap v17435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:23 smithi161 ceph-mon[91579]: pgmap v17435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:25 smithi188 ceph-mon[92425]: pgmap v17436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:25 smithi161 ceph-mon[91579]: pgmap v17436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:27 smithi188 ceph-mon[92425]: pgmap v17437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:27 smithi161 ceph-mon[91579]: pgmap v17437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:29 smithi188 ceph-mon[92425]: pgmap v17438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:29 smithi161 ceph-mon[91579]: pgmap v17438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:31.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:31 smithi188 ceph-mon[92425]: pgmap v17439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:31 smithi161 ceph-mon[91579]: pgmap v17439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:33.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:33 smithi188 ceph-mon[92425]: pgmap v17440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:33 smithi161 ceph-mon[91579]: pgmap v17440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:35 smithi188 ceph-mon[92425]: pgmap v17441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:35 smithi161 ceph-mon[91579]: pgmap v17441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:37 smithi161 ceph-mon[91579]: pgmap v17442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:37 smithi188 ceph-mon[92425]: pgmap v17442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:39 smithi161 ceph-mon[91579]: pgmap v17443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:39 smithi188 ceph-mon[92425]: pgmap v17443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:41 smithi161 ceph-mon[91579]: pgmap v17444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:41 smithi188 ceph-mon[92425]: pgmap v17444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:43 smithi161 ceph-mon[91579]: pgmap v17445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:43 smithi188 ceph-mon[92425]: pgmap v17445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:45 smithi161 ceph-mon[91579]: pgmap v17446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:45 smithi188 ceph-mon[92425]: pgmap v17446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:47 smithi161 ceph-mon[91579]: pgmap v17447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:47 smithi188 ceph-mon[92425]: pgmap v17447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:49 smithi161 ceph-mon[91579]: pgmap v17448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:49.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:49 smithi188 ceph-mon[92425]: pgmap v17448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:51 smithi161 ceph-mon[91579]: pgmap v17449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:51 smithi188 ceph-mon[92425]: pgmap v17449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:53.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:20:52.851+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:20:53.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:20:53.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:20:53.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:20:53.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:20:53.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:20:53.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:20:53.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:20:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:20:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:20:53.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:20:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:20:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:20:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:20:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:20:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:20:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:20:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:20:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:20:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:20:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:20:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:20:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:20:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:20:53.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:20:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:20:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:20:53.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:20:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:20:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:20:53.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:20:53.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:20:53.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:20:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:20:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:20:53.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:20:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:20:53.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:20:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:20:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:20:53.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:20:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:20:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:20:53.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:20:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:20:53.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1053] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:20:53.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1053] Local address: 172.21.15.161, port 60628 2022-07-03T20:20:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1053] Peer address: 172.21.15.101, port 22 2022-07-03T20:20:53.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1053] Connection lost 2022-07-03T20:20:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1053] Aborting connection 2022-07-03T20:20:53.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:20:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:53 smithi161 ceph-mon[91579]: pgmap v17450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:20:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:20:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:20:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:53 smithi188 ceph-mon[92425]: pgmap v17450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:20:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:20:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:20:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:20:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:20:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:20:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:20:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:20:54.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:20:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:20:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:20:54.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:20:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:20:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:20:54.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:20:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:20:54.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:20:54.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:20:54.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:20:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:20:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:20:54.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:20:54.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:20:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:20:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:20:54.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:20:54.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:20:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:20:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:20:54.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:20:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:20:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:20:54.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:20:54.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:20:54.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:20:54.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1053] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1053] Local address: 172.21.15.161, port 60628 2022-07-03T20:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1053] Peer address: 172.21.15.101, port 22 2022-07-03T20:20:54.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1053] Connection lost 2022-07-03T20:20:54.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1053] Aborting connection 2022-07-03T20:20:54.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:20:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:20:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:20:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:20:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:20:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:20:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:20:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:20:54.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:20:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:20:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:20:54.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:20:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:20:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:20:54.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:20:54.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:20:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:20:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:20:54.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:20:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:20:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:20:54.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:20:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:20:54.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:20:54.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:20:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:20:54.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:20:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:20:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:20:54.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:20:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:20:54.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:20:54.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:20:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:20:54.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:20:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:20:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:20:54.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:20:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:20:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1053] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:20:54.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1053] Local address: 172.21.15.161, port 60628 2022-07-03T20:20:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1053] Peer address: 172.21.15.101, port 22 2022-07-03T20:20:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1053] Connection lost 2022-07-03T20:20:54.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1053] Aborting connection 2022-07-03T20:20:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:55 smithi161 ceph-mon[91579]: pgmap v17451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:55 smithi188 ceph-mon[92425]: pgmap v17451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:20:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:20:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:57 smithi161 ceph-mon[91579]: pgmap v17452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:57 smithi188 ceph-mon[92425]: pgmap v17452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:20:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:59 smithi161 ceph-mon[91579]: pgmap v17453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:20:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:20:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:20:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:20:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:59 smithi188 ceph-mon[92425]: pgmap v17453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:20:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:20:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:20:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:20:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:21:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:00.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:01 smithi161 ceph-mon[91579]: pgmap v17454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:01 smithi188 ceph-mon[92425]: pgmap v17454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:02.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:03 smithi161 ceph-mon[91579]: pgmap v17455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:03 smithi188 ceph-mon[92425]: pgmap v17455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:05 smithi161 ceph-mon[91579]: pgmap v17456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:05 smithi188 ceph-mon[92425]: pgmap v17456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:07 smithi161 ceph-mon[91579]: pgmap v17457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:07 smithi188 ceph-mon[92425]: pgmap v17457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:09 smithi161 ceph-mon[91579]: pgmap v17458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:09 smithi188 ceph-mon[92425]: pgmap v17458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:11 smithi161 ceph-mon[91579]: pgmap v17459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:11 smithi188 ceph-mon[92425]: pgmap v17459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:13 smithi161 ceph-mon[91579]: pgmap v17460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:13 smithi188 ceph-mon[92425]: pgmap v17460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:15 smithi161 ceph-mon[91579]: pgmap v17461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:15 smithi188 ceph-mon[92425]: pgmap v17461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:17 smithi161 ceph-mon[91579]: pgmap v17462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:17 smithi188 ceph-mon[92425]: pgmap v17462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:19 smithi161 ceph-mon[91579]: pgmap v17463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:19 smithi188 ceph-mon[92425]: pgmap v17463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:21 smithi161 ceph-mon[91579]: pgmap v17464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:21.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:21 smithi188 ceph-mon[92425]: pgmap v17464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:23 smithi161 ceph-mon[91579]: pgmap v17465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:23 smithi188 ceph-mon[92425]: pgmap v17465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:25 smithi161 ceph-mon[91579]: pgmap v17466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:25 smithi188 ceph-mon[92425]: pgmap v17466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:27 smithi161 ceph-mon[91579]: pgmap v17467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:27 smithi188 ceph-mon[92425]: pgmap v17467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:29 smithi161 ceph-mon[91579]: pgmap v17468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:29 smithi188 ceph-mon[92425]: pgmap v17468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:31 smithi161 ceph-mon[91579]: pgmap v17469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:31 smithi188 ceph-mon[92425]: pgmap v17469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:33 smithi161 ceph-mon[91579]: pgmap v17470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:33 smithi188 ceph-mon[92425]: pgmap v17470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:35 smithi161 ceph-mon[91579]: pgmap v17471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:35 smithi188 ceph-mon[92425]: pgmap v17471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:37 smithi161 ceph-mon[91579]: pgmap v17472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:37 smithi188 ceph-mon[92425]: pgmap v17472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:39 smithi161 ceph-mon[91579]: pgmap v17473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:39 smithi188 ceph-mon[92425]: pgmap v17473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:41 smithi161 ceph-mon[91579]: pgmap v17474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:41 smithi188 ceph-mon[92425]: pgmap v17474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:42.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:42.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:43 smithi161 ceph-mon[91579]: pgmap v17475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:43 smithi188 ceph-mon[92425]: pgmap v17475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:44 smithi161 ceph-mon[91579]: pgmap v17476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:44.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:44 smithi188 ceph-mon[92425]: pgmap v17476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:46 smithi161 ceph-mon[91579]: pgmap v17477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:46 smithi188 ceph-mon[92425]: pgmap v17477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:48 smithi161 ceph-mon[91579]: pgmap v17478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:48 smithi188 ceph-mon[92425]: pgmap v17478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:50 smithi161 ceph-mon[91579]: pgmap v17479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:50 smithi188 ceph-mon[92425]: pgmap v17479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:52.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:52 smithi161 ceph-mon[91579]: pgmap v17480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:52.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:52.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:52 smithi188 ceph-mon[92425]: pgmap v17480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:21:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:21:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:21:53.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:21:53.359+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:21:53.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:21:53.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:21:53.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:21:53.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:21:53.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:21:53.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:21:53.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:21:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:21:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:21:53.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:21:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:21:53.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:21:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:21:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:21:53.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:21:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:21:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:21:53.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:21:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:21:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:21:53.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:21:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:21:53.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:21:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:21:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:21:53.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:21:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:21:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:21:53.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:21:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:21:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:21:53.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:21:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:21:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:21:53.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:21:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:21:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:21:53.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:21:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:21:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:21:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:21:53.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:21:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:21:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:21:53.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1055] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:21:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1055] Local address: 172.21.15.161, port 60656 2022-07-03T20:21:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1055] Peer address: 172.21.15.101, port 22 2022-07-03T20:21:53.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1055] Connection lost 2022-07-03T20:21:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1055] Aborting connection 2022-07-03T20:21:53.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:21:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:21:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:21:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:21:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:21:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:21:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1055] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1055] Local address: 172.21.15.161, port 60656 2022-07-03T20:21:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1055] Peer address: 172.21.15.101, port 22 2022-07-03T20:21:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1055] Connection lost 2022-07-03T20:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1055] Aborting connection 2022-07-03T20:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:54 smithi161 ceph-mon[91579]: pgmap v17481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:54.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:21:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:21:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:21:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:21:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:21:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:21:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:21:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:21:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:21:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:21:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:21:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:21:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:21:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:21:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:21:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:21:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:21:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:21:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:21:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1055] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:21:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1055] Local address: 172.21.15.161, port 60656 2022-07-03T20:21:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1055] Peer address: 172.21.15.101, port 22 2022-07-03T20:21:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1055] Connection lost 2022-07-03T20:21:54.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1055] Aborting connection 2022-07-03T20:21:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:54 smithi188 ceph-mon[92425]: pgmap v17481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:21:54.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:21:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: pgmap v17482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: pgmap v17482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:21:57.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:57.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:21:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:58 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 75365k 2022-07-03T20:21:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:58 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 77173989: error parsing value: Value '77173989' is below minimum 939524096 2022-07-03T20:21:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:58 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 75365k 2022-07-03T20:21:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:58 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 77173989: error parsing value: Value '77173989' is below minimum 939524096 2022-07-03T20:21:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:21:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:59 smithi161 ceph-mon[91579]: pgmap v17483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:21:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:21:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:21:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:21:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:59 smithi188 ceph-mon[92425]: pgmap v17483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:21:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:21:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:21:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:21:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:22:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:01 smithi161 ceph-mon[91579]: pgmap v17484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:01 smithi188 ceph-mon[92425]: pgmap v17484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:03 smithi161 ceph-mon[91579]: pgmap v17485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:03 smithi188 ceph-mon[92425]: pgmap v17485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:05 smithi161 ceph-mon[91579]: pgmap v17486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:05 smithi188 ceph-mon[92425]: pgmap v17486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:07 smithi161 ceph-mon[91579]: pgmap v17487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:07 smithi188 ceph-mon[92425]: pgmap v17487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:09 smithi161 ceph-mon[91579]: pgmap v17488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:09 smithi188 ceph-mon[92425]: pgmap v17488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:11 smithi188 ceph-mon[92425]: pgmap v17489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:11 smithi161 ceph-mon[91579]: pgmap v17489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:13 smithi188 ceph-mon[92425]: pgmap v17490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:13 smithi161 ceph-mon[91579]: pgmap v17490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:15 smithi188 ceph-mon[92425]: pgmap v17491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:15 smithi161 ceph-mon[91579]: pgmap v17491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:17 smithi188 ceph-mon[92425]: pgmap v17492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:17 smithi161 ceph-mon[91579]: pgmap v17492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:19 smithi188 ceph-mon[92425]: pgmap v17493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:19 smithi161 ceph-mon[91579]: pgmap v17493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:20.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:21.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:21 smithi188 ceph-mon[92425]: pgmap v17494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:21 smithi161 ceph-mon[91579]: pgmap v17494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:23 smithi161 ceph-mon[91579]: pgmap v17495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:23 smithi188 ceph-mon[92425]: pgmap v17495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:25 smithi161 ceph-mon[91579]: pgmap v17496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:25 smithi188 ceph-mon[92425]: pgmap v17496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:27 smithi161 ceph-mon[91579]: pgmap v17497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:27 smithi188 ceph-mon[92425]: pgmap v17497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:29 smithi161 ceph-mon[91579]: pgmap v17498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:29 smithi188 ceph-mon[92425]: pgmap v17498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:31 smithi161 ceph-mon[91579]: pgmap v17499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:31 smithi188 ceph-mon[92425]: pgmap v17499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:33 smithi161 ceph-mon[91579]: pgmap v17500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:33 smithi188 ceph-mon[92425]: pgmap v17500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:35 smithi161 ceph-mon[91579]: pgmap v17501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:35 smithi188 ceph-mon[92425]: pgmap v17501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:37 smithi161 ceph-mon[91579]: pgmap v17502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:37 smithi188 ceph-mon[92425]: pgmap v17502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:39 smithi161 ceph-mon[91579]: pgmap v17503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:39 smithi188 ceph-mon[92425]: pgmap v17503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:22:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:41 smithi161 ceph-mon[91579]: pgmap v17504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:41 smithi188 ceph-mon[92425]: pgmap v17504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:43 smithi161 ceph-mon[91579]: pgmap v17505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:43 smithi188 ceph-mon[92425]: pgmap v17505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:45 smithi161 ceph-mon[91579]: pgmap v17506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:45 smithi188 ceph-mon[92425]: pgmap v17506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:47 smithi161 ceph-mon[91579]: pgmap v17507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:47 smithi188 ceph-mon[92425]: pgmap v17507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:49 smithi161 ceph-mon[91579]: pgmap v17508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:49 smithi188 ceph-mon[92425]: pgmap v17508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:51.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:51 smithi161 ceph-mon[91579]: pgmap v17509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:51 smithi188 ceph-mon[92425]: pgmap v17509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:53 smithi161 ceph-mon[91579]: pgmap v17510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:53 smithi188 ceph-mon[92425]: pgmap v17510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:55 smithi161 ceph-mon[91579]: pgmap v17511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:55 smithi188 ceph-mon[92425]: pgmap v17511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:22:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:22:56.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:22:56.607+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:22:56.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:22:56.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:22:56.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:22:56.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:22:56.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:22:56.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:22:56.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:22:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:22:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:22:56.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:22:56.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:22:56.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:22:56.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:22:56.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:22:56.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:22:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:22:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:22:56.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:22:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:22:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:22:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:22:56.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:22:56.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:22:56.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:22:56.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:22:56.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:22:56.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:22:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:22:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:22:56.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:22:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:22:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:22:56.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:22:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:22:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:22:56.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:22:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:22:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:22:56.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:22:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:22:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:22:56.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:22:56.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:22:56.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:22:56.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1057] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:22:56.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1057] Local address: 172.21.15.161, port 60686 2022-07-03T20:22:56.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1057] Peer address: 172.21.15.101, port 22 2022-07-03T20:22:56.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1057] Connection lost 2022-07-03T20:22:56.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1057] Aborting connection 2022-07-03T20:22:56.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:22:56 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:22:57.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:57 smithi161 ceph-mon[91579]: pgmap v17512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:22:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:22:57.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:22:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:57 smithi188 ceph-mon[92425]: pgmap v17512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:22:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:22:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:22:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:22:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:22:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:22:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:22:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:22:58.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:22:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:22:58.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:22:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:22:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:22:58.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:22:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:22:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:22:58.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:22:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:22:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:22:58.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:22:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:22:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:22:58.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:22:58.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:22:58.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:22:58.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:22:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:22:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:22:58.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:22:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:22:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:22:58.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:22:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:22:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:22:58.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:22:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:22:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:22:58.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:22:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:22:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:22:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:22:58.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:22:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:22:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:22:58.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:22:58.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:22:58.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:22:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:22:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:22:58.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1057] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:22:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1057] Local address: 172.21.15.161, port 60686 2022-07-03T20:22:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1057] Peer address: 172.21.15.101, port 22 2022-07-03T20:22:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1057] Connection lost 2022-07-03T20:22:58.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1057] Aborting connection 2022-07-03T20:22:58.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:22:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:22:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:22:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:22:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:22:58.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:22:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:22:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:22:58.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:22:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:22:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:22:58.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:22:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:22:58.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:22:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:22:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:22:58.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:22:58.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:22:58.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:22:58.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:22:58.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:22:58.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:22:58.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:22:58.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:22:58.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:22:58.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:22:58.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:22:58.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:22:58.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:22:58.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:22:58.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:22:58.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:22:58.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:22:58.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:22:58.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:22:58.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:22:58.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:22:58.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:22:58.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:22:58.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:22:58.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:22:58.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:22:58.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:22:58.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:22:58.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:22:58.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:22:58.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1057] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:22:58.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1057] Local address: 172.21.15.161, port 60686 2022-07-03T20:22:58.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1057] Peer address: 172.21.15.101, port 22 2022-07-03T20:22:58.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1057] Connection lost 2022-07-03T20:22:58.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1057] Aborting connection 2022-07-03T20:22:58.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:22:59.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:59 smithi161 ceph-mon[91579]: pgmap v17513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:22:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:22:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:22:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:22:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:59 smithi188 ceph-mon[92425]: pgmap v17513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:22:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:22:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:22:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:22:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:23:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:00.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:01 smithi161 ceph-mon[91579]: pgmap v17514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:01 smithi188 ceph-mon[92425]: pgmap v17514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:03 smithi161 ceph-mon[91579]: pgmap v17515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:03 smithi188 ceph-mon[92425]: pgmap v17515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:05 smithi161 ceph-mon[91579]: pgmap v17516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:05 smithi188 ceph-mon[92425]: pgmap v17516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:07 smithi161 ceph-mon[91579]: pgmap v17517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:07 smithi188 ceph-mon[92425]: pgmap v17517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:09 smithi161 ceph-mon[91579]: pgmap v17518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:09 smithi188 ceph-mon[92425]: pgmap v17518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:11 smithi161 ceph-mon[91579]: pgmap v17519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:11.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:11 smithi188 ceph-mon[92425]: pgmap v17519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:12.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:12.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:13 smithi161 ceph-mon[91579]: pgmap v17520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:13.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:13 smithi188 ceph-mon[92425]: pgmap v17520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:15 smithi161 ceph-mon[91579]: pgmap v17521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:15 smithi188 ceph-mon[92425]: pgmap v17521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:17 smithi161 ceph-mon[91579]: pgmap v17522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:17.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:17 smithi188 ceph-mon[92425]: pgmap v17522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:19 smithi161 ceph-mon[91579]: pgmap v17523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:19.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:19 smithi188 ceph-mon[92425]: pgmap v17523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:21 smithi161 ceph-mon[91579]: pgmap v17524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:21 smithi188 ceph-mon[92425]: pgmap v17524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:22.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:22.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:23 smithi161 ceph-mon[91579]: pgmap v17525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:23.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:23 smithi188 ceph-mon[92425]: pgmap v17525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:25 smithi161 ceph-mon[91579]: pgmap v17526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:25 smithi188 ceph-mon[92425]: pgmap v17526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:27 smithi161 ceph-mon[91579]: pgmap v17527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:27 smithi188 ceph-mon[92425]: pgmap v17527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:29 smithi161 ceph-mon[91579]: pgmap v17528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:29 smithi188 ceph-mon[92425]: pgmap v17528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:31 smithi161 ceph-mon[91579]: pgmap v17529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:31 smithi188 ceph-mon[92425]: pgmap v17529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:33 smithi161 ceph-mon[91579]: pgmap v17530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:33 smithi188 ceph-mon[92425]: pgmap v17530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:35 smithi161 ceph-mon[91579]: pgmap v17531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:35 smithi188 ceph-mon[92425]: pgmap v17531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:36 smithi161 ceph-mon[91579]: pgmap v17532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:36 smithi188 ceph-mon[92425]: pgmap v17532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:38 smithi161 ceph-mon[91579]: pgmap v17533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:38 smithi188 ceph-mon[92425]: pgmap v17533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:39.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:39.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:40.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:40 smithi161 ceph-mon[91579]: pgmap v17534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:40 smithi188 ceph-mon[92425]: pgmap v17534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:42.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:42 smithi161 ceph-mon[91579]: pgmap v17535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:42.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:42.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:42 smithi188 ceph-mon[92425]: pgmap v17535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:42.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:44.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:44 smithi161 ceph-mon[91579]: pgmap v17536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:44.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:44 smithi188 ceph-mon[92425]: pgmap v17536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:46.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:46 smithi161 ceph-mon[91579]: pgmap v17537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:46.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:46 smithi188 ceph-mon[92425]: pgmap v17537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:48.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:48 smithi188 ceph-mon[92425]: pgmap v17538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:48 smithi161 ceph-mon[91579]: pgmap v17538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:49.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:50.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:50.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:50 smithi188 ceph-mon[92425]: pgmap v17539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:50 smithi161 ceph-mon[91579]: pgmap v17539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:52 smithi188 ceph-mon[92425]: pgmap v17540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:52.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:52 smithi161 ceph-mon[91579]: pgmap v17540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:54.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:54 smithi188 ceph-mon[92425]: pgmap v17541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:54 smithi161 ceph-mon[91579]: pgmap v17541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:23:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:23:56.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:56 smithi188 ceph-mon[92425]: pgmap v17542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:57.112 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:56 smithi161 ceph-mon[91579]: pgmap v17542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:57.377 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:23:57.112+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:23:57.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:23:57.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:23:57.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:23:57.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:23:57.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:23:57.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:23:57.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:23:57.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:23:57.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:23:57.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:23:57.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:23:57.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:23:57.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:23:57.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:23:57.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:23:57.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:23:57.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:23:57.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:23:57.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:23:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:23:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:23:57.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:23:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:23:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:23:57.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:23:57.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:23:57.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:23:57.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:23:57.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:23:57.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:23:57.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:23:57.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:23:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:23:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:23:57.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:23:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:23:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:23:57.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:23:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:23:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:23:57.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:23:57.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:23:57.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:23:57.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:23:57.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1059] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:23:57.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1059] Local address: 172.21.15.161, port 60718 2022-07-03T20:23:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1059] Peer address: 172.21.15.101, port 22 2022-07-03T20:23:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1059] Connection lost 2022-07-03T20:23:57.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1059] Aborting connection 2022-07-03T20:23:57.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:23:57.911 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:23:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:23:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:23:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:23:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:23:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:23:57.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:23:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:23:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:23:57.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:23:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:23:57.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:23:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:23:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:23:57.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:23:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:23:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:23:57.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:23:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:23:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:23:57.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:23:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:23:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:23:57.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:23:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:23:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:23:57.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:23:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:23:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:23:57.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:23:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:23:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:23:57.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:23:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:23:57.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:23:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:23:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:23:57.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:23:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:23:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:23:57.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:23:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:23:57.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:23:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:23:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:23:57.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:23:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1059] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:23:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1059] Local address: 172.21.15.161, port 60718 2022-07-03T20:23:57.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1059] Peer address: 172.21.15.101, port 22 2022-07-03T20:23:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1059] Connection lost 2022-07-03T20:23:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1059] Aborting connection 2022-07-03T20:23:57.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:23:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:23:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:23:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:23:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:23:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:23:58.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:23:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:23:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:23:58.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:23:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:23:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:23:58.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:23:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:23:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:23:58.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:23:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:23:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:23:58.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:23:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:23:58.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:23:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:23:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:23:58.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:23:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:23:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:23:58.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:23:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:23:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:23:58.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:23:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:23:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:23:58.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:23:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:23:58.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:23:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:23:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:23:58.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:23:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:23:58.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:23:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:23:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:23:58.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:23:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:23:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:23:58.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:23:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:23:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1059] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:23:58.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1059] Local address: 172.21.15.161, port 60718 2022-07-03T20:23:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1059] Peer address: 172.21.15.101, port 22 2022-07-03T20:23:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1059] Connection lost 2022-07-03T20:23:58.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1059] Aborting connection 2022-07-03T20:23:58.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:23:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:58 smithi161 ceph-mon[91579]: pgmap v17543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:23:59.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:58 smithi188 ceph-mon[92425]: pgmap v17543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:24:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:24:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:24:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:24:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:24:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:24:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:24:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:24:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:00 smithi161 ceph-mon[91579]: pgmap v17544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:00 smithi188 ceph-mon[92425]: pgmap v17544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:03.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:02 smithi161 ceph-mon[91579]: pgmap v17545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:02 smithi188 ceph-mon[92425]: pgmap v17545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:04 smithi161 ceph-mon[91579]: pgmap v17546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:04 smithi188 ceph-mon[92425]: pgmap v17546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:06 smithi161 ceph-mon[91579]: pgmap v17547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:06 smithi188 ceph-mon[92425]: pgmap v17547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:08 smithi161 ceph-mon[91579]: pgmap v17548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:08 smithi188 ceph-mon[92425]: pgmap v17548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:10.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:10 smithi161 ceph-mon[91579]: pgmap v17549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:10 smithi188 ceph-mon[92425]: pgmap v17549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:12 smithi161 ceph-mon[91579]: pgmap v17550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:12 smithi188 ceph-mon[92425]: pgmap v17550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:15.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:14 smithi161 ceph-mon[91579]: pgmap v17551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:14 smithi188 ceph-mon[92425]: pgmap v17551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:15.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:16 smithi161 ceph-mon[91579]: pgmap v17552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:16 smithi188 ceph-mon[92425]: pgmap v17552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:18 smithi161 ceph-mon[91579]: pgmap v17553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:19.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:18 smithi188 ceph-mon[92425]: pgmap v17553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:20.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:21.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:20 smithi161 ceph-mon[91579]: pgmap v17554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:20 smithi188 ceph-mon[92425]: pgmap v17554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:23.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:22 smithi161 ceph-mon[91579]: pgmap v17555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:23.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:22 smithi188 ceph-mon[92425]: pgmap v17555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:25.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:24 smithi161 ceph-mon[91579]: pgmap v17556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:24 smithi188 ceph-mon[92425]: pgmap v17556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:26 smithi161 ceph-mon[91579]: pgmap v17557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:26 smithi188 ceph-mon[92425]: pgmap v17557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:28 smithi161 ceph-mon[91579]: pgmap v17558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:28 smithi188 ceph-mon[92425]: pgmap v17558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:30.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:31.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:30 smithi161 ceph-mon[91579]: pgmap v17559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:30 smithi188 ceph-mon[92425]: pgmap v17559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:33.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:32 smithi161 ceph-mon[91579]: pgmap v17560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:32 smithi188 ceph-mon[92425]: pgmap v17560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:34 smithi161 ceph-mon[91579]: pgmap v17561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:34 smithi188 ceph-mon[92425]: pgmap v17561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:37.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:36 smithi161 ceph-mon[91579]: pgmap v17562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:36 smithi188 ceph-mon[92425]: pgmap v17562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:39.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:38 smithi161 ceph-mon[91579]: pgmap v17563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:38 smithi188 ceph-mon[92425]: pgmap v17563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:40.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:40.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:40 smithi161 ceph-mon[91579]: pgmap v17564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:41.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:40 smithi188 ceph-mon[92425]: pgmap v17564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:43.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:42 smithi161 ceph-mon[91579]: pgmap v17565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:43.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:42 smithi188 ceph-mon[92425]: pgmap v17565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:44 smithi161 ceph-mon[91579]: pgmap v17566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:45.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:44 smithi188 ceph-mon[92425]: pgmap v17566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:46 smithi161 ceph-mon[91579]: pgmap v17567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:47.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:46 smithi188 ceph-mon[92425]: pgmap v17567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:49.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:48 smithi161 ceph-mon[91579]: pgmap v17568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:49.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:48 smithi188 ceph-mon[92425]: pgmap v17568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:50.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:50.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:50 smithi161 ceph-mon[91579]: pgmap v17569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:51.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:50 smithi188 ceph-mon[92425]: pgmap v17569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:53.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:52 smithi161 ceph-mon[91579]: pgmap v17570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:53.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:52 smithi188 ceph-mon[92425]: pgmap v17570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:55.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:54 smithi161 ceph-mon[91579]: pgmap v17571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:54 smithi188 ceph-mon[92425]: pgmap v17571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:24:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:24:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:56 smithi161 ceph-mon[91579]: pgmap v17572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:56 smithi188 ceph-mon[92425]: pgmap v17572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:24:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:24:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:24:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:24:57.612+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:24:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:24:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:24:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:24:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:24:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:24:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:24:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:24:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:24:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:24:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:24:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:24:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:24:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:24:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:24:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:24:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:24:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:24:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:24:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:24:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:24:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:24:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:24:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:24:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:24:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:24:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:24:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:24:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:24:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:24:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:24:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:24:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:24:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:24:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:24:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:24:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:24:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:24:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:24:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:24:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:24:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:24:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:24:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:24:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:24:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1061] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:24:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1061] Local address: 172.21.15.161, port 60770 2022-07-03T20:24:57.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1061] Peer address: 172.21.15.101, port 22 2022-07-03T20:24:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1061] Connection lost 2022-07-03T20:24:57.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1061] Aborting connection 2022-07-03T20:24:57.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:24:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:24:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:24:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:24:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:58 smithi161 ceph-mon[91579]: pgmap v17573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:24:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:24:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:24:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:24:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:24:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:24:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:24:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:24:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:24:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:24:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:24:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:24:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1061] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:24:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1061] Local address: 172.21.15.161, port 60770 2022-07-03T20:24:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1061] Peer address: 172.21.15.101, port 22 2022-07-03T20:24:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1061] Connection lost 2022-07-03T20:24:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1061] Aborting connection 2022-07-03T20:24:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:58 smithi188 ceph-mon[92425]: pgmap v17573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:24:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:24:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:24:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:24:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:24:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:24:59.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:24:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:24:59.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:24:59.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:24:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:24:59.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:24:59.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:24:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:24:59.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:24:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:24:59.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:24:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:24:59.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:24:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:24:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:24:59.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:24:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:24:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:24:59.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:24:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:24:59.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:24:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:24:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:24:59.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:24:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:24:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:24:59.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:24:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:24:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:24:59.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:24:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:24:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:24:59.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:24:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:24:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:24:59.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:24:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:24:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:24:59.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:24:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:24:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:24:59.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1061] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:24:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1061] Local address: 172.21.15.161, port 60770 2022-07-03T20:24:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1061] Peer address: 172.21.15.101, port 22 2022-07-03T20:24:59.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1061] Connection lost 2022-07-03T20:24:59.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1061] Aborting connection 2022-07-03T20:25:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:25:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:00.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:25:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:25:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:01.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:00 smithi161 ceph-mon[91579]: pgmap v17574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:01.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:00 smithi188 ceph-mon[92425]: pgmap v17574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:02 smithi161 ceph-mon[91579]: pgmap v17575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:03.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:02 smithi188 ceph-mon[92425]: pgmap v17575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:05.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:04 smithi161 ceph-mon[91579]: pgmap v17576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:05.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:04 smithi188 ceph-mon[92425]: pgmap v17576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:07.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:06 smithi161 ceph-mon[91579]: pgmap v17577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:07.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:06 smithi188 ceph-mon[92425]: pgmap v17577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:08.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:08.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:09.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:08 smithi161 ceph-mon[91579]: pgmap v17578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:08 smithi188 ceph-mon[92425]: pgmap v17578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:10.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:11.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:10 smithi161 ceph-mon[91579]: pgmap v17579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:11.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:10 smithi188 ceph-mon[92425]: pgmap v17579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:13.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:12 smithi161 ceph-mon[91579]: pgmap v17580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:13.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:12 smithi188 ceph-mon[92425]: pgmap v17580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:14 smithi161 ceph-mon[91579]: pgmap v17581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:15.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:14 smithi188 ceph-mon[92425]: pgmap v17581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:16 smithi161 ceph-mon[91579]: pgmap v17582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:17.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:16 smithi188 ceph-mon[92425]: pgmap v17582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:18.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:18 smithi161 ceph-mon[91579]: pgmap v17583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:18 smithi188 ceph-mon[92425]: pgmap v17583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:20.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:21.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:20 smithi161 ceph-mon[91579]: pgmap v17584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:20 smithi188 ceph-mon[92425]: pgmap v17584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:22 smithi161 ceph-mon[91579]: pgmap v17585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:22 smithi188 ceph-mon[92425]: pgmap v17585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:24 smithi161 ceph-mon[91579]: pgmap v17586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:25.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:24 smithi188 ceph-mon[92425]: pgmap v17586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:27.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:26 smithi161 ceph-mon[91579]: pgmap v17587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:27.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:26 smithi188 ceph-mon[92425]: pgmap v17587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:28.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:28.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:28 smithi161 ceph-mon[91579]: pgmap v17588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:29.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:28 smithi188 ceph-mon[92425]: pgmap v17588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:30.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:30.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:30 smithi188 ceph-mon[92425]: pgmap v17589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:30 smithi161 ceph-mon[91579]: pgmap v17589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:32 smithi188 ceph-mon[92425]: pgmap v17590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:32 smithi161 ceph-mon[91579]: pgmap v17590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:34 smithi188 ceph-mon[92425]: pgmap v17591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:34 smithi161 ceph-mon[91579]: pgmap v17591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:36 smithi188 ceph-mon[92425]: pgmap v17592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:36 smithi161 ceph-mon[91579]: pgmap v17592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:38 smithi188 ceph-mon[92425]: pgmap v17593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:39.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:38 smithi161 ceph-mon[91579]: pgmap v17593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:41.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:40 smithi188 ceph-mon[92425]: pgmap v17594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:40 smithi161 ceph-mon[91579]: pgmap v17594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:43.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:42 smithi188 ceph-mon[92425]: pgmap v17595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:42 smithi161 ceph-mon[91579]: pgmap v17595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:45.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:44 smithi188 ceph-mon[92425]: pgmap v17596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:44 smithi161 ceph-mon[91579]: pgmap v17596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:46 smithi188 ceph-mon[92425]: pgmap v17597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:46 smithi161 ceph-mon[91579]: pgmap v17597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:48 smithi161 ceph-mon[91579]: pgmap v17598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:48 smithi188 ceph-mon[92425]: pgmap v17598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:50 smithi161 ceph-mon[91579]: pgmap v17599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:50 smithi188 ceph-mon[92425]: pgmap v17599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:52 smithi161 ceph-mon[91579]: pgmap v17600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:53.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:52 smithi188 ceph-mon[92425]: pgmap v17600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:54 smithi161 ceph-mon[91579]: pgmap v17601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:54 smithi188 ceph-mon[92425]: pgmap v17601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:25:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:25:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:56 smithi161 ceph-mon[91579]: pgmap v17602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:56 smithi188 ceph-mon[92425]: pgmap v17602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:58.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:25:58.110+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:25:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:25:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:25:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:25:58.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:25:58.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:25:58.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:25:58.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:25:58.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:25:58.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:25:58.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:25:58.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:25:58.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:25:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:25:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:25:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:25:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:25:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:25:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:25:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:25:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:25:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:25:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:25:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:25:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:25:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:25:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:25:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:25:58.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:25:58.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:25:58.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:25:58.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:25:58.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:25:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:25:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:25:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:25:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:25:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:25:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:25:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:25:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:25:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:25:58.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:25:58.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:25:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:25:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1063] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:25:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1063] Local address: 172.21.15.161, port 60798 2022-07-03T20:25:58.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1063] Peer address: 172.21.15.101, port 22 2022-07-03T20:25:58.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1063] Connection lost 2022-07-03T20:25:58.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1063] Aborting connection 2022-07-03T20:25:58.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:25:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:25:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: pgmap v17603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:25:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:25:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:25:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:25:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:25:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:25:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:25:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:25:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:25:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:25:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:25:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:25:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:25:59.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:25:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:25:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:25:59.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:25:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:25:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:25:59.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:25:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:25:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:25:59.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:25:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:25:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:25:59.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:25:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:25:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:25:59.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:25:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:25:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:25:59.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:25:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:25:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:25:59.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:25:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:25:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:25:59.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:25:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:25:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:25:59.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:25:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:25:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:25:59.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:25:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:25:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:25:59.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:25:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1063] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:25:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1063] Local address: 172.21.15.161, port 60798 2022-07-03T20:25:59.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1063] Peer address: 172.21.15.101, port 22 2022-07-03T20:25:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1063] Connection lost 2022-07-03T20:25:59.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1063] Aborting connection 2022-07-03T20:25:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: pgmap v17603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:25:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:25:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:25:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:25:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:25:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:25:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:25:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:25:59.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:25:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:25:59.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:25:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:25:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:25:59.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:25:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:25:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:25:59.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:25:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:25:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:25:59.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:25:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:25:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:25:59.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:25:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:25:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:25:59.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:25:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:25:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:25:59.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:25:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:25:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:25:59.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:25:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:25:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:25:59.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:25:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:25:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:25:59.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:25:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:25:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:25:59.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:25:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:25:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:25:59.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:25:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:25:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:25:59.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:25:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:25:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1063] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:25:59.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1063] Local address: 172.21.15.161, port 60798 2022-07-03T20:25:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1063] Peer address: 172.21.15.101, port 22 2022-07-03T20:25:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1063] Connection lost 2022-07-03T20:25:59.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1063] Aborting connection 2022-07-03T20:26:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:00 smithi161 ceph-mon[91579]: pgmap v17604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:00 smithi188 ceph-mon[92425]: pgmap v17604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:02 smithi161 ceph-mon[91579]: pgmap v17605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:02 smithi188 ceph-mon[92425]: pgmap v17605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:04 smithi161 ceph-mon[91579]: pgmap v17606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:04 smithi188 ceph-mon[92425]: pgmap v17606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:06 smithi161 ceph-mon[91579]: pgmap v17607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:06 smithi188 ceph-mon[92425]: pgmap v17607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:08 smithi161 ceph-mon[91579]: pgmap v17608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:08 smithi188 ceph-mon[92425]: pgmap v17608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:10 smithi161 ceph-mon[91579]: pgmap v17609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:10 smithi188 ceph-mon[92425]: pgmap v17609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:12 smithi161 ceph-mon[91579]: pgmap v17610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:13.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:12 smithi188 ceph-mon[92425]: pgmap v17610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:14 smithi161 ceph-mon[91579]: pgmap v17611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:14 smithi188 ceph-mon[92425]: pgmap v17611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:16 smithi161 ceph-mon[91579]: pgmap v17612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:16 smithi188 ceph-mon[92425]: pgmap v17612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:18 smithi161 ceph-mon[91579]: pgmap v17613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:18 smithi188 ceph-mon[92425]: pgmap v17613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:21.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:20 smithi161 ceph-mon[91579]: pgmap v17614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:21.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:20 smithi188 ceph-mon[92425]: pgmap v17614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:23 smithi161 ceph-mon[91579]: pgmap v17615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:23 smithi188 ceph-mon[92425]: pgmap v17615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:25 smithi161 ceph-mon[91579]: pgmap v17616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:25 smithi188 ceph-mon[92425]: pgmap v17616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:27.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:27 smithi161 ceph-mon[91579]: pgmap v17617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:27.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:27 smithi188 ceph-mon[92425]: pgmap v17617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:29 smithi161 ceph-mon[91579]: pgmap v17618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:29.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:29 smithi188 ceph-mon[92425]: pgmap v17618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:30.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:31.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:31 smithi161 ceph-mon[91579]: pgmap v17619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:31 smithi188 ceph-mon[92425]: pgmap v17619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:33.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:33 smithi161 ceph-mon[91579]: pgmap v17620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:33.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:33 smithi188 ceph-mon[92425]: pgmap v17620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:35 smithi161 ceph-mon[91579]: pgmap v17621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:35 smithi188 ceph-mon[92425]: pgmap v17621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:37.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:37 smithi161 ceph-mon[91579]: pgmap v17622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:37 smithi188 ceph-mon[92425]: pgmap v17622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:39 smithi161 ceph-mon[91579]: pgmap v17623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:39 smithi188 ceph-mon[92425]: pgmap v17623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:40.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:41.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:41 smithi161 ceph-mon[91579]: pgmap v17624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:41.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:41 smithi188 ceph-mon[92425]: pgmap v17624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:43.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:43 smithi161 ceph-mon[91579]: pgmap v17625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:43 smithi188 ceph-mon[92425]: pgmap v17625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:45 smithi161 ceph-mon[91579]: pgmap v17626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:45 smithi188 ceph-mon[92425]: pgmap v17626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:47.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:47 smithi161 ceph-mon[91579]: pgmap v17627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:47.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:47 smithi188 ceph-mon[92425]: pgmap v17627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:49 smithi161 ceph-mon[91579]: pgmap v17628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:49.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:49 smithi188 ceph-mon[92425]: pgmap v17628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:51.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:51 smithi161 ceph-mon[91579]: pgmap v17629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:51 smithi188 ceph-mon[92425]: pgmap v17629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:53.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:53 smithi161 ceph-mon[91579]: pgmap v17630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:53 smithi188 ceph-mon[92425]: pgmap v17630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:55 smithi161 ceph-mon[91579]: pgmap v17631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:55 smithi188 ceph-mon[92425]: pgmap v17631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:26:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:26:57.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:57 smithi161 ceph-mon[91579]: pgmap v17632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:57.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:57 smithi188 ceph-mon[92425]: pgmap v17632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:26:58.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:26:58.511+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:26:58.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:26:58.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:26:58.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:26:58.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:26:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:26:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:26:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:26:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:26:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:26:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:26:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:26:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:26:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:26:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:26:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:26:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:26:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:26:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:26:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:26:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:26:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:26:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:26:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:26:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:26:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:26:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:26:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:26:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:26:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:26:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:26:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:26:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:26:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:26:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:26:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:26:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:26:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:26:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:26:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:26:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:26:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:26:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:26:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:26:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:26:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1065] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:26:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1065] Local address: 172.21.15.161, port 60826 2022-07-03T20:26:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1065] Peer address: 172.21.15.101, port 22 2022-07-03T20:26:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1065] Connection lost 2022-07-03T20:26:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1065] Aborting connection 2022-07-03T20:26:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:26:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: pgmap v17633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:26:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:26:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:26:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:59.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:59.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:59.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: pgmap v17633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:26:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:26:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:26:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:26:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:26:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:26:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:27:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:27:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:27:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:27:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:27:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:27:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:27:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:27:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:27:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:27:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:27:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:27:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:27:00.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:27:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:27:00.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:27:00.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:27:00.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:27:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:27:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:27:00.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:27:00.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:27:00.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:27:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:27:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:27:00.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:27:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:27:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:27:00.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:27:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:27:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:27:00.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:27:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:27:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:27:00.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:27:00.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:27:00.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:27:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:27:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:27:00.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:27:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:27:00.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:27:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:27:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:27:00.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:27:00.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:27:00.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:27:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1065] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:27:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1065] Local address: 172.21.15.161, port 60826 2022-07-03T20:27:00.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1065] Peer address: 172.21.15.101, port 22 2022-07-03T20:27:00.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1065] Connection lost 2022-07-03T20:27:00.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1065] Aborting connection 2022-07-03T20:27:00.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:27:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:27:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:27:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:27:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:27:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:27:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:27:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:27:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:27:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:27:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:27:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:27:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:27:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:27:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:27:00.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:27:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:27:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:27:00.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:27:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:27:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:27:00.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:27:00.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:27:00.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:27:00.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:27:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:27:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:27:00.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:27:00.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:27:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:27:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:27:00.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:27:00.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:27:00.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:27:00.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:27:00.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:27:00.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:27:00.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:27:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:27:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:27:00.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:27:00.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:27:00.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:27:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:27:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:27:00.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:27:00.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1065] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:27:00.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1065] Local address: 172.21.15.161, port 60826 2022-07-03T20:27:00.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1065] Peer address: 172.21.15.101, port 22 2022-07-03T20:27:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1065] Connection lost 2022-07-03T20:27:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1065] Aborting connection 2022-07-03T20:27:00.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:01.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:01 smithi161 ceph-mon[91579]: pgmap v17634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:01.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:01 smithi188 ceph-mon[92425]: pgmap v17634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:03.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:03 smithi161 ceph-mon[91579]: pgmap v17635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:03 smithi188 ceph-mon[92425]: pgmap v17635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:05 smithi161 ceph-mon[91579]: pgmap v17636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:05 smithi188 ceph-mon[92425]: pgmap v17636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:07 smithi161 ceph-mon[91579]: pgmap v17637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:07.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:07 smithi188 ceph-mon[92425]: pgmap v17637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:09.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:09 smithi161 ceph-mon[91579]: pgmap v17638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:09.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:09 smithi188 ceph-mon[92425]: pgmap v17638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:11.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:11 smithi161 ceph-mon[91579]: pgmap v17639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:11.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:11 smithi188 ceph-mon[92425]: pgmap v17639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:13 smithi161 ceph-mon[91579]: pgmap v17640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:13 smithi188 ceph-mon[92425]: pgmap v17640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:15 smithi161 ceph-mon[91579]: pgmap v17641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:15 smithi188 ceph-mon[92425]: pgmap v17641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:17 smithi161 ceph-mon[91579]: pgmap v17642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:17.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:17 smithi188 ceph-mon[92425]: pgmap v17642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:19 smithi188 ceph-mon[92425]: pgmap v17643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:19 smithi161 ceph-mon[91579]: pgmap v17643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:21 smithi188 ceph-mon[92425]: pgmap v17644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:21 smithi161 ceph-mon[91579]: pgmap v17644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:23.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:23 smithi188 ceph-mon[92425]: pgmap v17645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:23.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:23 smithi161 ceph-mon[91579]: pgmap v17645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:25 smithi188 ceph-mon[92425]: pgmap v17646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:25 smithi161 ceph-mon[91579]: pgmap v17646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:27 smithi188 ceph-mon[92425]: pgmap v17647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:27 smithi161 ceph-mon[91579]: pgmap v17647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:28.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:29 smithi161 ceph-mon[91579]: pgmap v17648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:29 smithi188 ceph-mon[92425]: pgmap v17648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:31 smithi161 ceph-mon[91579]: pgmap v17649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:31 smithi188 ceph-mon[92425]: pgmap v17649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:33 smithi161 ceph-mon[91579]: pgmap v17650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:33 smithi188 ceph-mon[92425]: pgmap v17650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:35 smithi161 ceph-mon[91579]: pgmap v17651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:35 smithi188 ceph-mon[92425]: pgmap v17651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:37 smithi161 ceph-mon[91579]: pgmap v17652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:37 smithi188 ceph-mon[92425]: pgmap v17652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:39 smithi161 ceph-mon[91579]: pgmap v17653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:39 smithi188 ceph-mon[92425]: pgmap v17653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:41 smithi161 ceph-mon[91579]: pgmap v17654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:41 smithi188 ceph-mon[92425]: pgmap v17654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:43 smithi161 ceph-mon[91579]: pgmap v17655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:43.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:43 smithi188 ceph-mon[92425]: pgmap v17655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:45 smithi161 ceph-mon[91579]: pgmap v17656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:45 smithi188 ceph-mon[92425]: pgmap v17656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:47 smithi161 ceph-mon[91579]: pgmap v17657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:47 smithi188 ceph-mon[92425]: pgmap v17657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:49 smithi161 ceph-mon[91579]: pgmap v17658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:49 smithi188 ceph-mon[92425]: pgmap v17658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:51 smithi161 ceph-mon[91579]: pgmap v17659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:51 smithi188 ceph-mon[92425]: pgmap v17659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:53.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:53 smithi161 ceph-mon[91579]: pgmap v17660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:53 smithi188 ceph-mon[92425]: pgmap v17660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:55 smithi161 ceph-mon[91579]: pgmap v17661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:55 smithi188 ceph-mon[92425]: pgmap v17661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:27:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:27:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:57 smithi161 ceph-mon[91579]: pgmap v17662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:57 smithi188 ceph-mon[92425]: pgmap v17662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:27:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: pgmap v17663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:27:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:27:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:27:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:27:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:27:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:27:59.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:27:59.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:27:59.010+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:27:59.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:27:59.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:27:59.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:27:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:27:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:27:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:27:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:27:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:27:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:27:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:27:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:27:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:27:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:27:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:27:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:27:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:27:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:27:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:27:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:27:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:27:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:27:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:27:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:27:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:27:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:27:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:27:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:27:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:27:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:27:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:27:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:27:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:27:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:27:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:27:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:27:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:27:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:27:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:27:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:27:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:27:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:27:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:27:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:27:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:27:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1067] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:27:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1067] Local address: 172.21.15.161, port 60854 2022-07-03T20:27:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1067] Peer address: 172.21.15.101, port 22 2022-07-03T20:27:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1067] Connection lost 2022-07-03T20:27:59.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1067] Aborting connection 2022-07-03T20:27:59.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:27:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: pgmap v17663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:27:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:27:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:27:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:27:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:27:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:27:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:27:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:27:59.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:27:59.491+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T20:28:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:28:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:28:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:28:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:28:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:28:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:28:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:28:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:28:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:28:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:28:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:28:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:28:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:28:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:28:00.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:28:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:28:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:28:00.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:28:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:28:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:28:00.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:28:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:28:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:28:00.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:28:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:28:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:28:00.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:28:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:28:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:28:00.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:28:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:28:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:28:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:28:00.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:28:00.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:28:00.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:28:00.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:28:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:28:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:28:00.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:28:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:28:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:28:00.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:28:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:28:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:28:00.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:28:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1067] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:28:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1067] Local address: 172.21.15.161, port 60854 2022-07-03T20:28:00.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1067] Peer address: 172.21.15.101, port 22 2022-07-03T20:28:00.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1067] Connection lost 2022-07-03T20:28:00.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1067] Aborting connection 2022-07-03T20:28:00.644 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:28:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:28:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:28:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:28:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:28:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:28:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:28:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:28:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:28:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:28:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:28:00.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:28:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:28:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:28:00.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:28:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:28:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:28:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:28:00.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:28:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:28:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:28:00.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:28:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:28:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:28:00.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:28:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:28:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:28:00.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:28:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:28:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:28:00.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:28:00.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:28:00.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:28:00.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:28:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:28:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:28:00.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:28:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:28:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:28:00.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:28:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:28:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:28:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:28:00.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:28:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:28:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:28:00.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:28:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1067] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:28:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1067] Local address: 172.21.15.161, port 60854 2022-07-03T20:28:00.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1067] Peer address: 172.21.15.101, port 22 2022-07-03T20:28:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1067] Connection lost 2022-07-03T20:28:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1067] Aborting connection 2022-07-03T20:28:00.678 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:28:01.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:01 smithi161 ceph-mon[91579]: pgmap v17664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:01.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:01 smithi188 ceph-mon[92425]: pgmap v17664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:03 smithi161 ceph-mon[91579]: pgmap v17665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:03 smithi188 ceph-mon[92425]: pgmap v17665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:05 smithi161 ceph-mon[91579]: pgmap v17666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:05 smithi188 ceph-mon[92425]: pgmap v17666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:07 smithi161 ceph-mon[91579]: pgmap v17667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:07 smithi188 ceph-mon[92425]: pgmap v17667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:09 smithi161 ceph-mon[91579]: pgmap v17668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:09.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:09 smithi188 ceph-mon[92425]: pgmap v17668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:11 smithi161 ceph-mon[91579]: pgmap v17669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:11 smithi188 ceph-mon[92425]: pgmap v17669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:13 smithi161 ceph-mon[91579]: pgmap v17670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:13 smithi188 ceph-mon[92425]: pgmap v17670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:15 smithi161 ceph-mon[91579]: pgmap v17671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:15 smithi188 ceph-mon[92425]: pgmap v17671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:17 smithi161 ceph-mon[91579]: pgmap v17672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:17 smithi188 ceph-mon[92425]: pgmap v17672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:19 smithi161 ceph-mon[91579]: pgmap v17673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:19 smithi188 ceph-mon[92425]: pgmap v17673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:21.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:21 smithi161 ceph-mon[91579]: pgmap v17674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:21 smithi188 ceph-mon[92425]: pgmap v17674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:23 smithi161 ceph-mon[91579]: pgmap v17675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:23 smithi188 ceph-mon[92425]: pgmap v17675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:25 smithi161 ceph-mon[91579]: pgmap v17676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:25 smithi188 ceph-mon[92425]: pgmap v17676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:27.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:27 smithi161 ceph-mon[91579]: pgmap v17677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:27.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:27 smithi188 ceph-mon[92425]: pgmap v17677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:28.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:29 smithi161 ceph-mon[91579]: pgmap v17678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:29 smithi188 ceph-mon[92425]: pgmap v17678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:31 smithi161 ceph-mon[91579]: pgmap v17679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:31.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:31 smithi188 ceph-mon[92425]: pgmap v17679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:33.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:33 smithi161 ceph-mon[91579]: pgmap v17680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:33.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:33 smithi188 ceph-mon[92425]: pgmap v17680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:35 smithi161 ceph-mon[91579]: pgmap v17681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:35 smithi188 ceph-mon[92425]: pgmap v17681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:37.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:37 smithi161 ceph-mon[91579]: pgmap v17682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:37.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:37 smithi188 ceph-mon[92425]: pgmap v17682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:39.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:39 smithi161 ceph-mon[91579]: pgmap v17683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:39 smithi188 ceph-mon[92425]: pgmap v17683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:41 smithi161 ceph-mon[91579]: pgmap v17684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:41.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:41 smithi188 ceph-mon[92425]: pgmap v17684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:43.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:43 smithi161 ceph-mon[91579]: pgmap v17685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:43 smithi188 ceph-mon[92425]: pgmap v17685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:45 smithi161 ceph-mon[91579]: pgmap v17686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:45 smithi188 ceph-mon[92425]: pgmap v17686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:47.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:47 smithi161 ceph-mon[91579]: pgmap v17687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:47.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:47 smithi188 ceph-mon[92425]: pgmap v17687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:49.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:49 smithi161 ceph-mon[91579]: pgmap v17688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:49 smithi188 ceph-mon[92425]: pgmap v17688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:51 smithi161 ceph-mon[91579]: pgmap v17689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:51.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:51 smithi188 ceph-mon[92425]: pgmap v17689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:53 smithi161 ceph-mon[91579]: pgmap v17690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:53.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:53 smithi188 ceph-mon[92425]: pgmap v17690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:55 smithi161 ceph-mon[91579]: pgmap v17691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:55 smithi188 ceph-mon[92425]: pgmap v17691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:28:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:57 smithi161 ceph-mon[91579]: pgmap v17692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:57.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:57 smithi188 ceph-mon[92425]: pgmap v17692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-mon[91579]: pgmap v17693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:28:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:28:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:28:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:28:59.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:28:59.514+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:28:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:28:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:28:59.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:28:59.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:28:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:28:59.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:28:59.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:28:59.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:28:59.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:28:59.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:28:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:28:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:28:59.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:28:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:28:59.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:28:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:28:59.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:28:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:28:59.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:28:59.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:28:59.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:28:59.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:28:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:28:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:28:59.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:28:59.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1069] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:28:59.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1069] Local address: 172.21.15.161, port 60860 2022-07-03T20:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1069] Peer address: 172.21.15.101, port 22 2022-07-03T20:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1069] Connection lost 2022-07-03T20:28:59.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1069] Aborting connection 2022-07-03T20:28:59.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:28:59.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:59 smithi188 ceph-mon[92425]: pgmap v17693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:28:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:28:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:28:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:28:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:29:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:29:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:29:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:29:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:29:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:29:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:29:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:29:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:29:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:29:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:29:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:29:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:29:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:29:01.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:29:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:29:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:29:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:29:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:29:01.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:29:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:29:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:29:01.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:29:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:29:01.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:29:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:29:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:29:01.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:29:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:29:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:29:01.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:29:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:29:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:29:01.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:29:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:29:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:29:01.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:29:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:29:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:29:01.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:29:01.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:29:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:29:01.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:29:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:29:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:29:01.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:29:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:29:01.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:29:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:29:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:29:01.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:29:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:29:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:29:01.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:29:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1069] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:29:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1069] Local address: 172.21.15.161, port 60860 2022-07-03T20:29:01.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1069] Peer address: 172.21.15.101, port 22 2022-07-03T20:29:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1069] Connection lost 2022-07-03T20:29:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1069] Aborting connection 2022-07-03T20:29:01.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:01 smithi161 ceph-mon[91579]: pgmap v17694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:29:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:29:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:29:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:29:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:29:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:29:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:29:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:29:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:29:01.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:29:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:29:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:29:01.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:29:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:29:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:29:01.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:29:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:29:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:29:01.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:29:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:29:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:29:01.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:29:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:29:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:29:01.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:29:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:29:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:29:01.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:29:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:29:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:29:01.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:29:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:29:01.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:29:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:29:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:29:01.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:29:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:29:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:29:01.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:29:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:29:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:29:01.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:29:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:29:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:29:01.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:29:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:29:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1069] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:29:01.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1069] Local address: 172.21.15.161, port 60860 2022-07-03T20:29:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1069] Peer address: 172.21.15.101, port 22 2022-07-03T20:29:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1069] Connection lost 2022-07-03T20:29:01.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1069] Aborting connection 2022-07-03T20:29:01.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:01 smithi188 ceph-mon[92425]: pgmap v17694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:03.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:03 smithi161 ceph-mon[91579]: pgmap v17695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:03.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:03 smithi188 ceph-mon[92425]: pgmap v17695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:05 smithi161 ceph-mon[91579]: pgmap v17696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:05 smithi188 ceph-mon[92425]: pgmap v17696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:07.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:07 smithi161 ceph-mon[91579]: pgmap v17697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:07 smithi188 ceph-mon[92425]: pgmap v17697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:09 smithi161 ceph-mon[91579]: pgmap v17698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:09 smithi188 ceph-mon[92425]: pgmap v17698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:11.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:11 smithi161 ceph-mon[91579]: pgmap v17699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:11 smithi188 ceph-mon[92425]: pgmap v17699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:13 smithi161 ceph-mon[91579]: pgmap v17700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:13 smithi188 ceph-mon[92425]: pgmap v17700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:15 smithi161 ceph-mon[91579]: pgmap v17701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:15 smithi188 ceph-mon[92425]: pgmap v17701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:17 smithi161 ceph-mon[91579]: pgmap v17702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:17 smithi188 ceph-mon[92425]: pgmap v17702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:18.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:19 smithi188 ceph-mon[92425]: pgmap v17703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:19 smithi161 ceph-mon[91579]: pgmap v17703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:21 smithi188 ceph-mon[92425]: pgmap v17704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:21 smithi161 ceph-mon[91579]: pgmap v17704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:23 smithi188 ceph-mon[92425]: pgmap v17705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:23.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:23 smithi161 ceph-mon[91579]: pgmap v17705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:25 smithi188 ceph-mon[92425]: pgmap v17706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:25 smithi161 ceph-mon[91579]: pgmap v17706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:27 smithi188 ceph-mon[92425]: pgmap v17707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:27 smithi161 ceph-mon[91579]: pgmap v17707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:28.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:29 smithi188 ceph-mon[92425]: pgmap v17708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:29 smithi161 ceph-mon[91579]: pgmap v17708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:31.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:30 smithi161 ceph-mon[91579]: pgmap v17709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:31.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:30 smithi188 ceph-mon[92425]: pgmap v17709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:32 smithi161 ceph-mon[91579]: pgmap v17710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:32 smithi188 ceph-mon[92425]: pgmap v17710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:34 smithi161 ceph-mon[91579]: pgmap v17711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:35.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:34 smithi188 ceph-mon[92425]: pgmap v17711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:36 smithi161 ceph-mon[91579]: pgmap v17712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:36 smithi188 ceph-mon[92425]: pgmap v17712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:38 smithi161 ceph-mon[91579]: pgmap v17713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:39.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:38 smithi188 ceph-mon[92425]: pgmap v17713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:40 smithi161 ceph-mon[91579]: pgmap v17714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:40 smithi188 ceph-mon[92425]: pgmap v17714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:42 smithi161 ceph-mon[91579]: pgmap v17715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:42 smithi188 ceph-mon[92425]: pgmap v17715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:44 smithi161 ceph-mon[91579]: pgmap v17716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:45.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:44 smithi188 ceph-mon[92425]: pgmap v17716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:46 smithi161 ceph-mon[91579]: pgmap v17717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:46 smithi188 ceph-mon[92425]: pgmap v17717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:48.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:48 smithi161 ceph-mon[91579]: pgmap v17718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:49.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:48 smithi188 ceph-mon[92425]: pgmap v17718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:51.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:50 smithi161 ceph-mon[91579]: pgmap v17719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:50 smithi188 ceph-mon[92425]: pgmap v17719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:52 smithi161 ceph-mon[91579]: pgmap v17720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:52 smithi188 ceph-mon[92425]: pgmap v17720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:54 smithi161 ceph-mon[91579]: pgmap v17721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:54 smithi188 ceph-mon[92425]: pgmap v17721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:29:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:56 smithi161 ceph-mon[91579]: pgmap v17722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:56 smithi188 ceph-mon[92425]: pgmap v17722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:58.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:29:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:58 smithi161 ceph-mon[91579]: pgmap v17723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:29:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:58 smithi188 ceph-mon[92425]: pgmap v17723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:00.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:30:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:30:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:30:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:30:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:30:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:30:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:30:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:30:00.761 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:30:00.447+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:30:00.762 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:30:00.762 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:30:00.764 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:30:00.764 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:30:00.764 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:30:00.765 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:30:00.765 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:30:00.766 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:30:00.767 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:30:00.767 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:30:00.768 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:30:00.769 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:30:00.769 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:30:00.770 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:30:00.770 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:30:00.771 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:30:00.772 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:30:00.772 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:30:00.773 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:30:00.773 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:30:00.774 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:30:00.775 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:30:00.775 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:30:00.776 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:30:00.777 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:30:00.777 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:30:00.778 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:30:00.778 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:30:00.779 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:30:00.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:30:00.780 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:30:00.781 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:30:00.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:30:00.782 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:30:00.783 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:30:00.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:30:00.784 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:30:00.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:30:00.785 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:30:00.786 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:30:00.787 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:30:00.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1071] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:30:00.788 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1071] Local address: 172.21.15.161, port 60864 2022-07-03T20:30:00.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1071] Peer address: 172.21.15.101, port 22 2022-07-03T20:30:00.789 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1071] Connection lost 2022-07-03T20:30:00.790 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1071] Aborting connection 2022-07-03T20:30:00.791 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:30:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-mon[91579]: pgmap v17724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:30:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:30:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:30:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:30:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:00 smithi188 ceph-mon[92425]: pgmap v17724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:30:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:30:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:30:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:30:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:30:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:30:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:30:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:30:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:30:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:30:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:30:02.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:30:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:30:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:30:02.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:30:02.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:30:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:30:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:30:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:30:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:30:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:30:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:30:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:30:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:30:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:30:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:30:02.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:30:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:30:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:30:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:30:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:30:02.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:30:02.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:30:02.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:30:02.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:30:02.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:30:02.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:30:02.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:30:02.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:30:02.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:30:02.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:30:02.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:30:02.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:30:02.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:30:02.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:30:02.154 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:30:02.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:30:02.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1071] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:30:02.155 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1071] Local address: 172.21.15.161, port 60864 2022-07-03T20:30:02.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1071] Peer address: 172.21.15.101, port 22 2022-07-03T20:30:02.156 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1071] Connection lost 2022-07-03T20:30:02.157 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1071] Aborting connection 2022-07-03T20:30:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:30:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:30:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:30:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:30:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:30:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:30:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:30:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:30:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:30:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:30:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:30:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:30:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:30:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:30:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:30:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:30:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:30:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:30:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:30:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:30:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:30:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:30:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:30:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:30:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:30:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:30:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:30:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:30:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:30:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:30:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:30:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:30:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:30:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:30:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:30:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1071] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:30:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1071] Local address: 172.21.15.161, port 60864 2022-07-03T20:30:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1071] Peer address: 172.21.15.101, port 22 2022-07-03T20:30:02.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1071] Connection lost 2022-07-03T20:30:02.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1071] Aborting connection 2022-07-03T20:30:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:02 smithi161 ceph-mon[91579]: pgmap v17725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:02 smithi188 ceph-mon[92425]: pgmap v17725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:04 smithi161 ceph-mon[91579]: pgmap v17726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:04 smithi188 ceph-mon[92425]: pgmap v17726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:06 smithi161 ceph-mon[91579]: pgmap v17727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:06 smithi188 ceph-mon[92425]: pgmap v17727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:08 smithi161 ceph-mon[91579]: pgmap v17728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:09.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:08 smithi188 ceph-mon[92425]: pgmap v17728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:11.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:10 smithi161 ceph-mon[91579]: pgmap v17729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:10 smithi188 ceph-mon[92425]: pgmap v17729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:12 smithi161 ceph-mon[91579]: pgmap v17730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:13.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:12 smithi188 ceph-mon[92425]: pgmap v17730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:14 smithi161 ceph-mon[91579]: pgmap v17731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:14 smithi188 ceph-mon[92425]: pgmap v17731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:17.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:16 smithi161 ceph-mon[91579]: pgmap v17732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:16 smithi188 ceph-mon[92425]: pgmap v17732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:18.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:19.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:18 smithi161 ceph-mon[91579]: pgmap v17733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:18 smithi188 ceph-mon[92425]: pgmap v17733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:20 smithi161 ceph-mon[91579]: pgmap v17734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:21.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:20 smithi188 ceph-mon[92425]: pgmap v17734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:22 smithi161 ceph-mon[91579]: pgmap v17735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:23.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:22 smithi188 ceph-mon[92425]: pgmap v17735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:24 smithi161 ceph-mon[91579]: pgmap v17736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:24 smithi188 ceph-mon[92425]: pgmap v17736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:26 smithi161 ceph-mon[91579]: pgmap v17737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:26 smithi188 ceph-mon[92425]: pgmap v17737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:29.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:28 smithi161 ceph-mon[91579]: pgmap v17738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:28 smithi188 ceph-mon[92425]: pgmap v17738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:30 smithi161 ceph-mon[91579]: pgmap v17739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:30 smithi188 ceph-mon[92425]: pgmap v17739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:33.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:32 smithi161 ceph-mon[91579]: pgmap v17740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:32 smithi188 ceph-mon[92425]: pgmap v17740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:35.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:34 smithi161 ceph-mon[91579]: pgmap v17741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:34 smithi188 ceph-mon[92425]: pgmap v17741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:36 smithi161 ceph-mon[91579]: pgmap v17742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:36 smithi188 ceph-mon[92425]: pgmap v17742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:38.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:38.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:39.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:38 smithi161 ceph-mon[91579]: pgmap v17743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:38 smithi188 ceph-mon[92425]: pgmap v17743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:41.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:40 smithi161 ceph-mon[91579]: pgmap v17744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:40 smithi188 ceph-mon[92425]: pgmap v17744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:43.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:42 smithi161 ceph-mon[91579]: pgmap v17745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:42 smithi188 ceph-mon[92425]: pgmap v17745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:45.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:44 smithi161 ceph-mon[91579]: pgmap v17746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:44 smithi188 ceph-mon[92425]: pgmap v17746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:47.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:46 smithi161 ceph-mon[91579]: pgmap v17747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:46 smithi188 ceph-mon[92425]: pgmap v17747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:48.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:48 smithi161 ceph-mon[91579]: pgmap v17748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:48 smithi188 ceph-mon[92425]: pgmap v17748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:50 smithi161 ceph-mon[91579]: pgmap v17749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:50 smithi188 ceph-mon[92425]: pgmap v17749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:52 smithi161 ceph-mon[91579]: pgmap v17750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:52 smithi188 ceph-mon[92425]: pgmap v17750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:54 smithi161 ceph-mon[91579]: pgmap v17751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:54 smithi188 ceph-mon[92425]: pgmap v17751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:57.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:56 smithi161 ceph-mon[91579]: pgmap v17752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:56 smithi188 ceph-mon[92425]: pgmap v17752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:30:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:30:59.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:58 smithi161 ceph-mon[91579]: pgmap v17753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:30:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:58 smithi188 ceph-mon[92425]: pgmap v17753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:31:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:31:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:31:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:31:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:31:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:31:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:31:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:31:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:00 smithi188 ceph-mon[92425]: pgmap v17754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-mon[91579]: pgmap v17754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:01.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:31:00.952+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:31:01.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:31:01.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:31:01.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:31:01.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:31:01.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:31:01.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:31:01.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:31:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:31:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:31:01.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:31:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:31:01.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:31:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:31:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:31:01.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:31:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:31:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:31:01.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:31:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:31:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:31:01.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:31:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:31:01.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:31:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:31:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:31:01.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:31:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:31:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:31:01.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:31:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:31:01.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:31:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:31:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:31:01.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:31:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:31:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:31:01.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:31:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:31:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:31:01.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:31:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:31:01.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:31:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:31:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:31:01.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1073] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:31:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1073] Local address: 172.21.15.161, port 60868 2022-07-03T20:31:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1073] Peer address: 172.21.15.101, port 22 2022-07-03T20:31:01.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1073] Connection lost 2022-07-03T20:31:01.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1073] Aborting connection 2022-07-03T20:31:01.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:31:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:31:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:31:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:31:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:31:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:31:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:31:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:31:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:31:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:31:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:31:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:31:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:31:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:31:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:31:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:31:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:31:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:31:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:31:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:31:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:31:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:31:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:31:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:31:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:31:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:31:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:31:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:31:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:31:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:31:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:31:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:31:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:31:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:31:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:31:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:31:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:31:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:31:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:31:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:31:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:31:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:31:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:31:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:31:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:31:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:31:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:31:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:31:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:31:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:31:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:31:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1073] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:31:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1073] Local address: 172.21.15.161, port 60868 2022-07-03T20:31:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1073] Peer address: 172.21.15.101, port 22 2022-07-03T20:31:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1073] Connection lost 2022-07-03T20:31:02.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1073] Aborting connection 2022-07-03T20:31:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:31:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:31:02.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:31:02.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:31:02.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:31:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:31:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:31:02.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:31:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:31:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:31:02.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:31:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:31:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:31:02.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:31:02.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:31:02.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:31:02.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:31:02.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:31:02.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:31:02.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:31:02.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:31:02.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:31:02.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:31:02.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:31:02.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:31:02.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:31:02.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:31:02.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:31:02.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:31:02.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:31:02.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:31:02.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:31:02.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:31:02.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:31:02.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:31:02.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:31:02.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:31:02.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:31:02.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:31:02.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:31:02.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:31:02.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:31:02.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:31:02.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:31:02.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:31:02.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:31:02.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:31:02.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:31:02.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:31:02.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1073] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:31:02.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1073] Local address: 172.21.15.161, port 60868 2022-07-03T20:31:02.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1073] Peer address: 172.21.15.101, port 22 2022-07-03T20:31:02.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1073] Connection lost 2022-07-03T20:31:02.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1073] Aborting connection 2022-07-03T20:31:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:02 smithi188 ceph-mon[92425]: pgmap v17755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:02 smithi161 ceph-mon[91579]: pgmap v17755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:04 smithi188 ceph-mon[92425]: pgmap v17756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:04 smithi161 ceph-mon[91579]: pgmap v17756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:06 smithi188 ceph-mon[92425]: pgmap v17757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:07.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:06 smithi161 ceph-mon[91579]: pgmap v17757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:08 smithi188 ceph-mon[92425]: pgmap v17758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:08 smithi161 ceph-mon[91579]: pgmap v17758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:10 smithi188 ceph-mon[92425]: pgmap v17759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:10 smithi161 ceph-mon[91579]: pgmap v17759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:12 smithi188 ceph-mon[92425]: pgmap v17760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:12 smithi161 ceph-mon[91579]: pgmap v17760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:14 smithi188 ceph-mon[92425]: pgmap v17761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:14 smithi161 ceph-mon[91579]: pgmap v17761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:16 smithi161 ceph-mon[91579]: pgmap v17762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:16 smithi188 ceph-mon[92425]: pgmap v17762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:18 smithi161 ceph-mon[91579]: pgmap v17763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:18 smithi188 ceph-mon[92425]: pgmap v17763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:20 smithi161 ceph-mon[91579]: pgmap v17764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:20 smithi188 ceph-mon[92425]: pgmap v17764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:22 smithi161 ceph-mon[91579]: pgmap v17765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:22 smithi188 ceph-mon[92425]: pgmap v17765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:24 smithi161 ceph-mon[91579]: pgmap v17766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:24 smithi188 ceph-mon[92425]: pgmap v17766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:26 smithi161 ceph-mon[91579]: pgmap v17767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:26 smithi188 ceph-mon[92425]: pgmap v17767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:31:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:29.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:28 smithi161 ceph-mon[91579]: pgmap v17768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:28 smithi188 ceph-mon[92425]: pgmap v17768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:30 smithi161 ceph-mon[91579]: pgmap v17769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:30 smithi188 ceph-mon[92425]: pgmap v17769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:32 smithi161 ceph-mon[91579]: pgmap v17770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:32 smithi188 ceph-mon[92425]: pgmap v17770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:34 smithi161 ceph-mon[91579]: pgmap v17771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:34 smithi188 ceph-mon[92425]: pgmap v17771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:36 smithi161 ceph-mon[91579]: pgmap v17772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:36 smithi188 ceph-mon[92425]: pgmap v17772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:38 smithi161 ceph-mon[91579]: pgmap v17773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:38 smithi188 ceph-mon[92425]: pgmap v17773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:40 smithi161 ceph-mon[91579]: pgmap v17774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:40 smithi188 ceph-mon[92425]: pgmap v17774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:42 smithi161 ceph-mon[91579]: pgmap v17775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:42 smithi188 ceph-mon[92425]: pgmap v17775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:44 smithi161 ceph-mon[91579]: pgmap v17776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:44 smithi188 ceph-mon[92425]: pgmap v17776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:46 smithi161 ceph-mon[91579]: pgmap v17777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:46 smithi188 ceph-mon[92425]: pgmap v17777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:49.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:48 smithi161 ceph-mon[91579]: pgmap v17778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:48 smithi188 ceph-mon[92425]: pgmap v17778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:50 smithi161 ceph-mon[91579]: pgmap v17779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:51.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:50 smithi188 ceph-mon[92425]: pgmap v17779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:52 smithi161 ceph-mon[91579]: pgmap v17780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:52 smithi188 ceph-mon[92425]: pgmap v17780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:54 smithi161 ceph-mon[91579]: pgmap v17781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:54 smithi188 ceph-mon[92425]: pgmap v17781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:56 smithi161 ceph-mon[91579]: pgmap v17782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:56 smithi188 ceph-mon[92425]: pgmap v17782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:31:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:31:59.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:31:58 smithi161 ceph-mon[91579]: pgmap v17783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:31:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:31:58 smithi188 ceph-mon[92425]: pgmap v17783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-mon[91579]: pgmap v17784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:01 smithi188 ceph-mon[92425]: pgmap v17784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:01.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:32:01.453+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:32:01.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:32:01.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:32:01.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:32:01.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:32:01.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:32:01.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:32:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:32:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:32:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:32:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:32:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:32:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:32:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:32:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:32:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:32:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:32:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:32:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:32:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:32:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:32:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:32:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:32:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:32:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:32:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:32:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:32:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:32:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:32:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:32:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:32:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:32:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:32:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:32:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:32:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:32:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:32:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:32:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:32:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:32:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:32:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:32:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:32:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:32:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:32:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1075] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:32:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1075] Local address: 172.21.15.161, port 60872 2022-07-03T20:32:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1075] Peer address: 172.21.15.101, port 22 2022-07-03T20:32:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1075] Connection lost 2022-07-03T20:32:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1075] Aborting connection 2022-07-03T20:32:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:32:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:32:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:32:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:32:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:32:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:32:02.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:32:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:32:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:32:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:32:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:32:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:32:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:32:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:32:03.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:32:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:32:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:32:03.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:32:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:32:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:32:03.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:32:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:32:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:32:03.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:32:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:32:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:32:03.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:32:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:32:03.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:32:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:32:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:32:03.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:32:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:32:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:32:03.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:32:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:32:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:32:03.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:32:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:32:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:32:03.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:32:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:32:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:32:03.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:32:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:32:03.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:32:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:32:03.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:32:03.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:32:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:32:03.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:32:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:32:03.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:32:03.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1075] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:32:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1075] Local address: 172.21.15.161, port 60872 2022-07-03T20:32:03.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1075] Peer address: 172.21.15.101, port 22 2022-07-03T20:32:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1075] Connection lost 2022-07-03T20:32:03.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1075] Aborting connection 2022-07-03T20:32:03.399 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:03 smithi161 ceph-mon[91579]: pgmap v17785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:03.400 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:32:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:32:03.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:32:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:32:03.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:32:03.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:32:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:32:03.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:32:03.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:32:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:32:03.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:32:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:32:03.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:32:03.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:32:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:32:03.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:32:03.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:32:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:32:03.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:32:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:32:03.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:32:03.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:32:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:32:03.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:32:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:32:03.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:32:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:32:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:32:03.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:32:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:32:03.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:32:03.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:32:03.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:32:03.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:32:03.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:32:03.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:32:03.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:32:03.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:32:03.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:32:03.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:32:03.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:32:03.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:32:03.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:32:03.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:32:03.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:32:03.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:32:03.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1075] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:32:03.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1075] Local address: 172.21.15.161, port 60872 2022-07-03T20:32:03.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1075] Peer address: 172.21.15.101, port 22 2022-07-03T20:32:03.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1075] Connection lost 2022-07-03T20:32:03.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1075] Aborting connection 2022-07-03T20:32:03.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:03 smithi188 ceph-mon[92425]: pgmap v17785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:03.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: pgmap v17786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:05 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: pgmap v17786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:32:05.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:05.671 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:05 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:32:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:06 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 69073k 2022-07-03T20:32:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:06 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 70731538: error parsing value: Value '70731538' is below minimum 939524096 2022-07-03T20:32:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:06 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 69073k 2022-07-03T20:32:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:06 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 70731538: error parsing value: Value '70731538' is below minimum 939524096 2022-07-03T20:32:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:07 smithi161 ceph-mon[91579]: pgmap v17787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:07.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:07 smithi188 ceph-mon[92425]: pgmap v17787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:09 smithi161 ceph-mon[91579]: pgmap v17788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:09 smithi188 ceph-mon[92425]: pgmap v17788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:11 smithi161 ceph-mon[91579]: pgmap v17789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:11 smithi188 ceph-mon[92425]: pgmap v17789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:13 smithi161 ceph-mon[91579]: pgmap v17790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:13 smithi188 ceph-mon[92425]: pgmap v17790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:15 smithi161 ceph-mon[91579]: pgmap v17791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:15 smithi188 ceph-mon[92425]: pgmap v17791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:17.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:17 smithi161 ceph-mon[91579]: pgmap v17792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:17.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:17 smithi188 ceph-mon[92425]: pgmap v17792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:19 smithi161 ceph-mon[91579]: pgmap v17793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:19 smithi188 ceph-mon[92425]: pgmap v17793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:21 smithi161 ceph-mon[91579]: pgmap v17794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:21 smithi188 ceph-mon[92425]: pgmap v17794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:23 smithi161 ceph-mon[91579]: pgmap v17795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:23 smithi188 ceph-mon[92425]: pgmap v17795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:25 smithi161 ceph-mon[91579]: pgmap v17796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:25 smithi188 ceph-mon[92425]: pgmap v17796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:27 smithi161 ceph-mon[91579]: pgmap v17797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:27 smithi188 ceph-mon[92425]: pgmap v17797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:29 smithi161 ceph-mon[91579]: pgmap v17798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:29 smithi188 ceph-mon[92425]: pgmap v17798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:31 smithi161 ceph-mon[91579]: pgmap v17799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:31 smithi188 ceph-mon[92425]: pgmap v17799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:33 smithi161 ceph-mon[91579]: pgmap v17800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:33 smithi188 ceph-mon[92425]: pgmap v17800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:35 smithi161 ceph-mon[91579]: pgmap v17801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:35 smithi188 ceph-mon[92425]: pgmap v17801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:37 smithi161 ceph-mon[91579]: pgmap v17802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:37 smithi188 ceph-mon[92425]: pgmap v17802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:39 smithi161 ceph-mon[91579]: pgmap v17803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:39 smithi188 ceph-mon[92425]: pgmap v17803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:41 smithi161 ceph-mon[91579]: pgmap v17804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:41 smithi188 ceph-mon[92425]: pgmap v17804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:43 smithi161 ceph-mon[91579]: pgmap v17805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:43 smithi188 ceph-mon[92425]: pgmap v17805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:45 smithi161 ceph-mon[91579]: pgmap v17806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:45 smithi188 ceph-mon[92425]: pgmap v17806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:47.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:47 smithi161 ceph-mon[91579]: pgmap v17807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:47 smithi188 ceph-mon[92425]: pgmap v17807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:49 smithi161 ceph-mon[91579]: pgmap v17808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:49 smithi188 ceph-mon[92425]: pgmap v17808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:51 smithi161 ceph-mon[91579]: pgmap v17809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:51 smithi188 ceph-mon[92425]: pgmap v17809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:53 smithi161 ceph-mon[91579]: pgmap v17810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:53 smithi188 ceph-mon[92425]: pgmap v17810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:55 smithi161 ceph-mon[91579]: pgmap v17811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:55 smithi188 ceph-mon[92425]: pgmap v17811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:57 smithi161 ceph-mon[91579]: pgmap v17812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:57 smithi188 ceph-mon[92425]: pgmap v17812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:32:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:32:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:59 smithi161 ceph-mon[91579]: pgmap v17813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:59 smithi188 ceph-mon[92425]: pgmap v17813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:32:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:32:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:32:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:33:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:01 smithi161 ceph-mon[91579]: pgmap v17814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:01 smithi188 ceph-mon[92425]: pgmap v17814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:03 smithi161 ceph-mon[91579]: pgmap v17815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:03 smithi188 ceph-mon[92425]: pgmap v17815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:05.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:33:04.708+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:33:05.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:33:05.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:33:05.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:33:05.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:33:05.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:33:05.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:33:05.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:33:05.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:33:05.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:33:05.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:33:05.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:33:05.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:33:05.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:33:05.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:33:05.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:33:05.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:33:05.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:33:05.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:33:05.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:33:05.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:33:05.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:33:05.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:33:05.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:33:05.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:33:05.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:33:05.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:33:05.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:33:05.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:33:05.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:33:05.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:33:05.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:33:05.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:33:05.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:33:05.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:33:05.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:33:05.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:33:05.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:33:05.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:33:05.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:33:05.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:33:05.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:33:05.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:33:05.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:33:05.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:33:05.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1077] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:33:05.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1077] Local address: 172.21.15.161, port 60876 2022-07-03T20:33:05.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1077] Peer address: 172.21.15.101, port 22 2022-07-03T20:33:05.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1077] Connection lost 2022-07-03T20:33:05.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1077] Aborting connection 2022-07-03T20:33:05.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:33:04 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:33:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:05 smithi161 ceph-mon[91579]: pgmap v17816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:33:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:33:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:33:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:05 smithi188 ceph-mon[92425]: pgmap v17816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:33:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:33:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:33:06.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:33:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:33:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:33:06.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:33:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:33:06.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:33:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:33:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:33:06.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:33:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:33:06.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:33:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:33:06.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:33:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:33:06.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:33:06.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:33:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:33:06.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:33:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:33:06.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:33:06.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:33:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:33:06.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:33:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:33:06.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:33:06.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:33:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:33:06.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:33:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:33:06.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:33:06.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:33:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:33:06.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:33:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:33:06.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:33:06.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:33:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:33:06.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:33:06.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:33:06.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:33:06.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:33:06.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:33:06.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:33:06.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:33:06.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:33:06.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:33:06.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1077] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:33:06.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1077] Local address: 172.21.15.161, port 60876 2022-07-03T20:33:06.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1077] Peer address: 172.21.15.101, port 22 2022-07-03T20:33:06.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1077] Connection lost 2022-07-03T20:33:06.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1077] Aborting connection 2022-07-03T20:33:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:33:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:33:06.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:33:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:33:06.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:33:06.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:33:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:33:06.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:33:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:33:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:33:06.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:33:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:33:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:33:06.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:33:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:33:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:33:06.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:33:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:33:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:33:06.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:33:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:33:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:33:06.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:33:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:33:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:33:06.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:33:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:33:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:33:06.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:33:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:33:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:33:06.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:33:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:33:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:33:06.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:33:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:33:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:33:06.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:33:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:33:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:33:06.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:33:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:33:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:33:06.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:33:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:33:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:33:06.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1077] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:33:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1077] Local address: 172.21.15.161, port 60876 2022-07-03T20:33:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1077] Peer address: 172.21.15.101, port 22 2022-07-03T20:33:06.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1077] Connection lost 2022-07-03T20:33:06.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1077] Aborting connection 2022-07-03T20:33:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:07 smithi161 ceph-mon[91579]: pgmap v17817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:07 smithi188 ceph-mon[92425]: pgmap v17817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:09.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:09 smithi161 ceph-mon[91579]: pgmap v17818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:09 smithi188 ceph-mon[92425]: pgmap v17818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:11 smithi161 ceph-mon[91579]: pgmap v17819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:11 smithi188 ceph-mon[92425]: pgmap v17819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:13.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:13 smithi161 ceph-mon[91579]: pgmap v17820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:13 smithi188 ceph-mon[92425]: pgmap v17820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:15 smithi161 ceph-mon[91579]: pgmap v17821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:15 smithi188 ceph-mon[92425]: pgmap v17821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:17 smithi161 ceph-mon[91579]: pgmap v17822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:17 smithi188 ceph-mon[92425]: pgmap v17822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:19.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:19 smithi161 ceph-mon[91579]: pgmap v17823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:19 smithi188 ceph-mon[92425]: pgmap v17823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:21 smithi161 ceph-mon[91579]: pgmap v17824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:21 smithi188 ceph-mon[92425]: pgmap v17824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:23 smithi161 ceph-mon[91579]: pgmap v17825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:23 smithi188 ceph-mon[92425]: pgmap v17825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:25 smithi161 ceph-mon[91579]: pgmap v17826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:25 smithi188 ceph-mon[92425]: pgmap v17826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:27 smithi161 ceph-mon[91579]: pgmap v17827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:27 smithi188 ceph-mon[92425]: pgmap v17827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:29 smithi161 ceph-mon[91579]: pgmap v17828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:29 smithi188 ceph-mon[92425]: pgmap v17828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:31 smithi161 ceph-mon[91579]: pgmap v17829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:31 smithi188 ceph-mon[92425]: pgmap v17829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:33 smithi161 ceph-mon[91579]: pgmap v17830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:33 smithi188 ceph-mon[92425]: pgmap v17830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:35 smithi161 ceph-mon[91579]: pgmap v17831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:35 smithi188 ceph-mon[92425]: pgmap v17831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:37 smithi161 ceph-mon[91579]: pgmap v17832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:37 smithi188 ceph-mon[92425]: pgmap v17832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:39 smithi188 ceph-mon[92425]: pgmap v17833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:39 smithi161 ceph-mon[91579]: pgmap v17833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:41 smithi188 ceph-mon[92425]: pgmap v17834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:41.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:41 smithi161 ceph-mon[91579]: pgmap v17834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:43 smithi188 ceph-mon[92425]: pgmap v17835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:43.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:43 smithi161 ceph-mon[91579]: pgmap v17835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:45 smithi188 ceph-mon[92425]: pgmap v17836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:45 smithi161 ceph-mon[91579]: pgmap v17836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:47 smithi188 ceph-mon[92425]: pgmap v17837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:47 smithi161 ceph-mon[91579]: pgmap v17837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:48.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:49 smithi188 ceph-mon[92425]: pgmap v17838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:49.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:49 smithi161 ceph-mon[91579]: pgmap v17838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:51 smithi188 ceph-mon[92425]: pgmap v17839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:51 smithi161 ceph-mon[91579]: pgmap v17839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:53 smithi161 ceph-mon[91579]: pgmap v17840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:53 smithi188 ceph-mon[92425]: pgmap v17840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:55 smithi161 ceph-mon[91579]: pgmap v17841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:55 smithi188 ceph-mon[92425]: pgmap v17841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:57.877 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:57 smithi161 ceph-mon[91579]: pgmap v17842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:57 smithi188 ceph-mon[92425]: pgmap v17842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:33:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:33:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:59 smithi161 ceph-mon[91579]: pgmap v17843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:33:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:33:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:33:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:33:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:59 smithi188 ceph-mon[92425]: pgmap v17843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:33:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:33:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:33:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:33:59.917 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:34:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:01 smithi161 ceph-mon[91579]: pgmap v17844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:01 smithi188 ceph-mon[92425]: pgmap v17844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:03 smithi161 ceph-mon[91579]: pgmap v17845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:03 smithi188 ceph-mon[92425]: pgmap v17845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-mon[91579]: pgmap v17846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:34:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:34:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:34:05.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:34:05.212+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:34:05.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:34:05.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:34:05.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:34:05.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:34:05.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:34:05.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:34:05.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:34:05.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:34:05.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:34:05.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:34:05.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:34:05.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:34:05.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:34:05.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:34:05.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:34:05.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:34:05.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:34:05.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:34:05.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:34:05.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:34:05.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:34:05.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:34:05.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:34:05.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:34:05.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:34:05.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:34:05.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:34:05.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:34:05.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:34:05.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:34:05.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:34:05.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:34:05.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:34:05.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:34:05.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:34:05.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:34:05.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:34:05.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:34:05.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:34:05.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:34:05.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:34:05.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:34:05.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:34:05.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:34:05.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1079] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:34:05.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1079] Local address: 172.21.15.161, port 60880 2022-07-03T20:34:05.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1079] Peer address: 172.21.15.101, port 22 2022-07-03T20:34:05.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1079] Connection lost 2022-07-03T20:34:05.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1079] Aborting connection 2022-07-03T20:34:05.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:34:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:05 smithi188 ceph-mon[92425]: pgmap v17846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:34:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:34:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:34:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:34:06.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:34:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:34:06.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:34:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:34:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:34:06.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:34:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:34:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:34:06.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:34:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:34:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:34:06.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:34:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:34:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:34:06.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:34:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:34:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:34:06.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:34:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:34:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:34:06.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:34:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:34:06.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:34:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:34:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:34:06.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:34:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:34:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:34:06.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:34:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:34:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:34:06.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:34:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:34:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:34:06.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:34:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:34:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:34:06.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:34:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:34:06.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:34:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:34:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:34:06.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:34:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:34:06.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:34:06.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1079] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:34:06.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1079] Local address: 172.21.15.161, port 60880 2022-07-03T20:34:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1079] Peer address: 172.21.15.101, port 22 2022-07-03T20:34:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1079] Connection lost 2022-07-03T20:34:06.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1079] Aborting connection 2022-07-03T20:34:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:34:06.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:34:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:34:06.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:34:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:34:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:34:06.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:34:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:34:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:34:06.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:34:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:34:06.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:34:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:34:06.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:34:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:34:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:34:06.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:34:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:34:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:34:06.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:34:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:34:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:34:06.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:34:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:34:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:34:06.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:34:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:34:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:34:06.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:34:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:34:06.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:34:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:34:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:34:06.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:34:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:34:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:34:06.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:34:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:34:06.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:34:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:34:06.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:34:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:34:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:34:06.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:34:06.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:34:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:34:06.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1079] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:34:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1079] Local address: 172.21.15.161, port 60880 2022-07-03T20:34:06.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1079] Peer address: 172.21.15.101, port 22 2022-07-03T20:34:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1079] Connection lost 2022-07-03T20:34:06.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1079] Aborting connection 2022-07-03T20:34:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:07 smithi161 ceph-mon[91579]: pgmap v17847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:07 smithi188 ceph-mon[92425]: pgmap v17847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:09 smithi161 ceph-mon[91579]: pgmap v17848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:09 smithi188 ceph-mon[92425]: pgmap v17848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:10 smithi161 ceph-mon[91579]: pgmap v17849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:10 smithi188 ceph-mon[92425]: pgmap v17849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:12 smithi161 ceph-mon[91579]: pgmap v17850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:12 smithi188 ceph-mon[92425]: pgmap v17850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:14 smithi161 ceph-mon[91579]: pgmap v17851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:14 smithi188 ceph-mon[92425]: pgmap v17851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:16 smithi161 ceph-mon[91579]: pgmap v17852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:16 smithi188 ceph-mon[92425]: pgmap v17852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:18 smithi161 ceph-mon[91579]: pgmap v17853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:18 smithi188 ceph-mon[92425]: pgmap v17853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:20 smithi161 ceph-mon[91579]: pgmap v17854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:20 smithi188 ceph-mon[92425]: pgmap v17854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:22 smithi161 ceph-mon[91579]: pgmap v17855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:23.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:22 smithi188 ceph-mon[92425]: pgmap v17855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:24 smithi161 ceph-mon[91579]: pgmap v17856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:24 smithi188 ceph-mon[92425]: pgmap v17856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:26 smithi161 ceph-mon[91579]: pgmap v17857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:26 smithi188 ceph-mon[92425]: pgmap v17857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:28 smithi161 ceph-mon[91579]: pgmap v17858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:28 smithi161 ceph-mon[91579]: from='mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]: dispatch 2022-07-03T20:34:29.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:28 smithi161 ceph-mon[91579]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]: dispatch 2022-07-03T20:34:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:28 smithi188 ceph-mon[92425]: pgmap v17858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:28 smithi188 ceph-mon[92425]: from='mds.? [v2:172.21.15.188:6838/204174115,v1:172.21.15.188:6839/204174115]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]: dispatch 2022-07-03T20:34:29.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:28 smithi188 ceph-mon[92425]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]: dispatch 2022-07-03T20:34:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:29 smithi161 ceph-mon[91579]: evicting unresponsive client smithi101:0 (25061), after 304.226 seconds 2022-07-03T20:34:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:29 smithi161 ceph-mon[91579]: Evicting (and blocklisting) client session 25061 (172.21.15.101:0/2170986661) 2022-07-03T20:34:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:29 smithi161 ceph-mon[91579]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]': finished 2022-07-03T20:34:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:29 smithi161 ceph-mon[91579]: osdmap e88: 12 total, 8 up, 8 in 2022-07-03T20:34:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:29 smithi188 ceph-mon[92425]: evicting unresponsive client smithi101:0 (25061), after 304.226 seconds 2022-07-03T20:34:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:29 smithi188 ceph-mon[92425]: Evicting (and blocklisting) client session 25061 (172.21.15.101:0/2170986661) 2022-07-03T20:34:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:29 smithi188 ceph-mon[92425]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.101:0/2170986661"}]': finished 2022-07-03T20:34:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:29 smithi188 ceph-mon[92425]: osdmap e88: 12 total, 8 up, 8 in 2022-07-03T20:34:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:30 smithi161 ceph-mon[91579]: pgmap v17860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T20:34:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:30 smithi188 ceph-mon[92425]: pgmap v17860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T20:34:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:32 smithi161 ceph-mon[91579]: pgmap v17861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T20:34:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:32 smithi188 ceph-mon[92425]: pgmap v17861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 60 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T20:34:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:34 smithi161 ceph-mon[91579]: pgmap v17862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:34 smithi188 ceph-mon[92425]: pgmap v17862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:36 smithi161 ceph-mon[91579]: pgmap v17863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:36 smithi188 ceph-mon[92425]: pgmap v17863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:38 smithi161 ceph-mon[91579]: pgmap v17864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:38 smithi188 ceph-mon[92425]: pgmap v17864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2022-07-03T20:34:41.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:40 smithi161 ceph-mon[91579]: pgmap v17865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 283 B/s wr, 2 op/s 2022-07-03T20:34:41.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:40 smithi188 ceph-mon[92425]: pgmap v17865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 283 B/s wr, 2 op/s 2022-07-03T20:34:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:42 smithi161 ceph-mon[91579]: pgmap v17866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T20:34:43.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:42 smithi188 ceph-mon[92425]: pgmap v17866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2022-07-03T20:34:43.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:44 smithi161 ceph-mon[91579]: pgmap v17867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T20:34:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:44 smithi188 ceph-mon[92425]: pgmap v17867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2022-07-03T20:34:47.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:46 smithi161 ceph-mon[91579]: pgmap v17868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T20:34:47.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:46 smithi188 ceph-mon[92425]: pgmap v17868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T20:34:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:48 smithi161 ceph-mon[91579]: pgmap v17869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2022-07-03T20:34:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:48 smithi188 ceph-mon[92425]: pgmap v17869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2022-07-03T20:34:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:50 smithi161 ceph-mon[91579]: pgmap v17870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T20:34:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:50 smithi188 ceph-mon[92425]: pgmap v17870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2022-07-03T20:34:53.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:52 smithi161 ceph-mon[91579]: pgmap v17871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:53.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:53.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:52 smithi188 ceph-mon[92425]: pgmap v17871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:53.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:54 smithi161 ceph-mon[91579]: pgmap v17872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:54 smithi188 ceph-mon[92425]: pgmap v17872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:57.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:56 smithi161 ceph-mon[91579]: pgmap v17873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:57.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:56 smithi188 ceph-mon[92425]: pgmap v17873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:34:58.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:34:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:58 smithi161 ceph-mon[91579]: pgmap v17874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:34:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:58 smithi188 ceph-mon[92425]: pgmap v17874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:35:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:35:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:35:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:35:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:35:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:35:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:35:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:35:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:00 smithi188 ceph-mon[92425]: pgmap v17875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:00 smithi161 ceph-mon[91579]: pgmap v17875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:02 smithi188 ceph-mon[92425]: pgmap v17876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:02 smithi161 ceph-mon[91579]: pgmap v17876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:04 smithi188 ceph-mon[92425]: pgmap v17877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:04 smithi161 ceph-mon[91579]: pgmap v17877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:35:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:35:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:35:06.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:35:05.712+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:35:06.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:35:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:35:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:35:06.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:35:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:35:06.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:35:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:35:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:35:06.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:35:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:35:06.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:35:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:35:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:35:06.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:35:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:35:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:35:06.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:35:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:35:06.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:35:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:35:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:35:06.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:35:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:35:06.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:35:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:35:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:35:06.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:35:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:35:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:35:06.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:35:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:35:06.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:35:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:35:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:35:06.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:35:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:35:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:35:06.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:35:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:35:06.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:35:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:35:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:35:06.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:35:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:35:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1081] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:35:06.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1081] Local address: 172.21.15.161, port 60884 2022-07-03T20:35:06.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1081] Peer address: 172.21.15.101, port 22 2022-07-03T20:35:06.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1081] Connection lost 2022-07-03T20:35:06.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1081] Aborting connection 2022-07-03T20:35:06.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:35:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:35:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:35:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:35:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:35:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:06 smithi188 ceph-mon[92425]: pgmap v17878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:35:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:35:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:35:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:35:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:35:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:35:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:35:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:35:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:35:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:35:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:35:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:35:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:35:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:35:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:35:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:35:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:35:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:35:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:35:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:35:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:35:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:35:07.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:35:07.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:35:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:35:07.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:35:07.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:35:07.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:35:07.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:35:07.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:35:07.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:35:07.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:35:07.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:35:07.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:35:07.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:35:07.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:35:07.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:35:07.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1081] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:35:07.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1081] Local address: 172.21.15.161, port 60884 2022-07-03T20:35:07.192 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1081] Peer address: 172.21.15.101, port 22 2022-07-03T20:35:07.193 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1081] Connection lost 2022-07-03T20:35:07.194 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1081] Aborting connection 2022-07-03T20:35:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:06 smithi161 ceph-mon[91579]: pgmap v17878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:35:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:35:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:35:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:35:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:35:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:35:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:35:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:35:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:35:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:35:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:35:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:35:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:35:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:35:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:35:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:35:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:35:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:35:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:35:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:35:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:35:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:35:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:35:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:35:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:35:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:35:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:35:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:35:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:35:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:35:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:35:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:35:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:35:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:35:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:35:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:35:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:35:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:35:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:35:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:35:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:35:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:35:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:35:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:35:07.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:35:07.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:35:07.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1081] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:35:07.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1081] Local address: 172.21.15.161, port 60884 2022-07-03T20:35:07.403 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1081] Peer address: 172.21.15.101, port 22 2022-07-03T20:35:07.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1081] Connection lost 2022-07-03T20:35:07.404 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1081] Aborting connection 2022-07-03T20:35:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:08 smithi188 ceph-mon[92425]: pgmap v17879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:08 smithi161 ceph-mon[91579]: pgmap v17879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:10 smithi188 ceph-mon[92425]: pgmap v17880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:10 smithi161 ceph-mon[91579]: pgmap v17880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:12 smithi161 ceph-mon[91579]: pgmap v17881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:12 smithi188 ceph-mon[92425]: pgmap v17881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:14 smithi161 ceph-mon[91579]: pgmap v17882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:14 smithi188 ceph-mon[92425]: pgmap v17882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:16 smithi161 ceph-mon[91579]: pgmap v17883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:16 smithi188 ceph-mon[92425]: pgmap v17883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:18 smithi161 ceph-mon[91579]: pgmap v17884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:18 smithi188 ceph-mon[92425]: pgmap v17884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:20 smithi161 ceph-mon[91579]: pgmap v17885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:20 smithi188 ceph-mon[92425]: pgmap v17885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:22 smithi161 ceph-mon[91579]: pgmap v17886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:22 smithi188 ceph-mon[92425]: pgmap v17886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:24 smithi161 ceph-mon[91579]: pgmap v17887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:24 smithi188 ceph-mon[92425]: pgmap v17887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:26 smithi161 ceph-mon[91579]: pgmap v17888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:26 smithi188 ceph-mon[92425]: pgmap v17888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:28 smithi161 ceph-mon[91579]: pgmap v17889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:28 smithi188 ceph-mon[92425]: pgmap v17889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:30 smithi161 ceph-mon[91579]: pgmap v17890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:30 smithi188 ceph-mon[92425]: pgmap v17890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:32 smithi161 ceph-mon[91579]: pgmap v17891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:32 smithi188 ceph-mon[92425]: pgmap v17891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:34 smithi161 ceph-mon[91579]: pgmap v17892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:34 smithi188 ceph-mon[92425]: pgmap v17892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:36 smithi161 ceph-mon[91579]: pgmap v17893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:36 smithi188 ceph-mon[92425]: pgmap v17893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:38 smithi161 ceph-mon[91579]: pgmap v17894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:38 smithi188 ceph-mon[92425]: pgmap v17894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:40 smithi161 ceph-mon[91579]: pgmap v17895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:40 smithi188 ceph-mon[92425]: pgmap v17895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:42 smithi161 ceph-mon[91579]: pgmap v17896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:42 smithi188 ceph-mon[92425]: pgmap v17896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:44 smithi161 ceph-mon[91579]: pgmap v17897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:44 smithi188 ceph-mon[92425]: pgmap v17897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:46 smithi161 ceph-mon[91579]: pgmap v17898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:46 smithi188 ceph-mon[92425]: pgmap v17898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:48 smithi161 ceph-mon[91579]: pgmap v17899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:48 smithi188 ceph-mon[92425]: pgmap v17899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:51 smithi161 ceph-mon[91579]: pgmap v17900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:51 smithi188 ceph-mon[92425]: pgmap v17900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:53 smithi161 ceph-mon[91579]: pgmap v17901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:53 smithi188 ceph-mon[92425]: pgmap v17901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:55 smithi161 ceph-mon[91579]: pgmap v17902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:55 smithi188 ceph-mon[92425]: pgmap v17902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:57 smithi161 ceph-mon[91579]: pgmap v17903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:57 smithi188 ceph-mon[92425]: pgmap v17903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:35:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:35:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:35:59 smithi161 ceph-mon[91579]: pgmap v17904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:35:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:35:59 smithi188 ceph-mon[92425]: pgmap v17904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:01 smithi161 ceph-mon[91579]: pgmap v17905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:01 smithi188 ceph-mon[92425]: pgmap v17905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:03 smithi161 ceph-mon[91579]: pgmap v17906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:03 smithi188 ceph-mon[92425]: pgmap v17906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:05 smithi161 ceph-mon[91579]: pgmap v17907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:05 smithi188 ceph-mon[92425]: pgmap v17907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:06.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:36:06.210+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:36:06.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:36:06.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:36:06.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:36:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:36:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:36:06.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:36:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:36:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:36:06.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:36:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:36:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:36:06.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:36:06.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:36:06.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:36:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:36:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:36:06.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:36:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:36:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:36:06.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:36:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:36:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:36:06.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:36:06.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:36:06.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:36:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:36:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:36:06.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:36:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:36:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:36:06.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:36:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:36:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:36:06.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:36:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:36:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:36:06.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:36:06.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:36:06.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:36:06.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:36:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:36:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:36:06.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:36:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:36:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1083] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:36:06.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1083] Local address: 172.21.15.161, port 60888 2022-07-03T20:36:06.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1083] Peer address: 172.21.15.101, port 22 2022-07-03T20:36:06.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1083] Connection lost 2022-07-03T20:36:06.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1083] Aborting connection 2022-07-03T20:36:06.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:36:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: pgmap v17908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:36:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:36:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:36:07.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:36:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:36:07.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:36:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:36:07.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:36:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:36:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:36:07.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:36:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:36:07.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:36:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:36:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:36:07.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:36:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:36:07.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:36:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:36:07.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:36:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:36:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:36:07.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:36:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:36:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:36:07.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:36:07.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:36:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:36:07.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:36:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:36:07.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:36:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:36:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:36:07.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:36:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:36:07.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:36:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:36:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:36:07.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:36:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:36:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:36:07.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:36:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:36:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:36:07.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:36:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:36:07.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:36:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:36:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:36:07.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1083] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:36:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1083] Local address: 172.21.15.161, port 60888 2022-07-03T20:36:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1083] Peer address: 172.21.15.101, port 22 2022-07-03T20:36:07.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1083] Connection lost 2022-07-03T20:36:07.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1083] Aborting connection 2022-07-03T20:36:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: pgmap v17908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:36:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:36:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:36:07.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:36:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:36:07.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:36:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:36:07.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:36:07.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:36:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:36:07.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:36:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:36:07.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:36:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:36:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:36:07.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:36:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:36:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:36:07.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:36:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:36:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:36:07.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:36:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:36:07.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:36:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:36:07.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:36:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:36:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:36:07.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:36:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:36:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:36:07.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:36:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:36:07.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:36:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:36:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:36:07.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:36:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:36:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:36:07.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:36:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:36:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:36:07.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:36:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:36:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:36:07.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:36:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:36:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:36:07.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:36:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1083] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:36:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1083] Local address: 172.21.15.161, port 60888 2022-07-03T20:36:07.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1083] Peer address: 172.21.15.101, port 22 2022-07-03T20:36:07.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1083] Connection lost 2022-07-03T20:36:07.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1083] Aborting connection 2022-07-03T20:36:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:09 smithi161 ceph-mon[91579]: pgmap v17909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:09 smithi188 ceph-mon[92425]: pgmap v17909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:11 smithi161 ceph-mon[91579]: pgmap v17910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:11 smithi188 ceph-mon[92425]: pgmap v17910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:13 smithi161 ceph-mon[91579]: pgmap v17911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:13.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:13 smithi188 ceph-mon[92425]: pgmap v17911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:13.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:15 smithi161 ceph-mon[91579]: pgmap v17912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:15 smithi188 ceph-mon[92425]: pgmap v17912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:17 smithi161 ceph-mon[91579]: pgmap v17913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:17 smithi188 ceph-mon[92425]: pgmap v17913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:19 smithi161 ceph-mon[91579]: pgmap v17914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:19 smithi188 ceph-mon[92425]: pgmap v17914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:21 smithi161 ceph-mon[91579]: pgmap v17915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:21 smithi188 ceph-mon[92425]: pgmap v17915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:23 smithi161 ceph-mon[91579]: pgmap v17916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:23 smithi188 ceph-mon[92425]: pgmap v17916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:25 smithi161 ceph-mon[91579]: pgmap v17917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:25 smithi188 ceph-mon[92425]: pgmap v17917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:27 smithi161 ceph-mon[91579]: pgmap v17918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:27 smithi188 ceph-mon[92425]: pgmap v17918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:29 smithi161 ceph-mon[91579]: pgmap v17919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:29 smithi188 ceph-mon[92425]: pgmap v17919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:31 smithi161 ceph-mon[91579]: pgmap v17920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:31 smithi188 ceph-mon[92425]: pgmap v17920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:33 smithi161 ceph-mon[91579]: pgmap v17921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:33 smithi188 ceph-mon[92425]: pgmap v17921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:35 smithi161 ceph-mon[91579]: pgmap v17922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:35 smithi188 ceph-mon[92425]: pgmap v17922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:37 smithi161 ceph-mon[91579]: pgmap v17923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:37 smithi188 ceph-mon[92425]: pgmap v17923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:39 smithi161 ceph-mon[91579]: pgmap v17924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:39 smithi188 ceph-mon[92425]: pgmap v17924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:41 smithi161 ceph-mon[91579]: pgmap v17925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:41 smithi188 ceph-mon[92425]: pgmap v17925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:43 smithi161 ceph-mon[91579]: pgmap v17926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:43 smithi188 ceph-mon[92425]: pgmap v17926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:45 smithi161 ceph-mon[91579]: pgmap v17927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:45 smithi188 ceph-mon[92425]: pgmap v17927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:47 smithi161 ceph-mon[91579]: pgmap v17928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:47 smithi188 ceph-mon[92425]: pgmap v17928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:49 smithi161 ceph-mon[91579]: pgmap v17929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:49 smithi188 ceph-mon[92425]: pgmap v17929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:51 smithi161 ceph-mon[91579]: pgmap v17930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:51 smithi188 ceph-mon[92425]: pgmap v17930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:53 smithi188 ceph-mon[92425]: pgmap v17931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:53 smithi161 ceph-mon[91579]: pgmap v17931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:55 smithi188 ceph-mon[92425]: pgmap v17932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:55 smithi161 ceph-mon[91579]: pgmap v17932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:57 smithi188 ceph-mon[92425]: pgmap v17933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:57 smithi161 ceph-mon[91579]: pgmap v17933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:36:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:36:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:59 smithi188 ceph-mon[92425]: pgmap v17934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:59 smithi161 ceph-mon[91579]: pgmap v17934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:36:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:36:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:36:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:37:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:01 smithi188 ceph-mon[92425]: pgmap v17935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:01 smithi161 ceph-mon[91579]: pgmap v17935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:03 smithi188 ceph-mon[92425]: pgmap v17936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:03 smithi161 ceph-mon[91579]: pgmap v17936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:05 smithi188 ceph-mon[92425]: pgmap v17937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:05 smithi161 ceph-mon[91579]: pgmap v17937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:07.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:37:06.706+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:37:07.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:37:07.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:37:07.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:37:07.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:37:07.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:37:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:37:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:37:07.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:37:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:37:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:37:07.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:37:07.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:37:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:37:07.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:37:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:37:07.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:37:07.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:37:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:37:07.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:37:07.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:37:07.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:37:07.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:37:07.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:37:07.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:37:07.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:37:07.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:37:07.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:37:07.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:37:07.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:37:07.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:37:07.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:37:07.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:37:07.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:37:07.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:37:07.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:37:07.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:37:07.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:37:07.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:37:07.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:37:07.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:37:07.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:37:07.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:37:07.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1085] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:37:07.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1085] Local address: 172.21.15.161, port 60892 2022-07-03T20:37:07.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1085] Peer address: 172.21.15.101, port 22 2022-07-03T20:37:07.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1085] Connection lost 2022-07-03T20:37:07.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1085] Aborting connection 2022-07-03T20:37:07.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:37:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:07 smithi188 ceph-mon[92425]: pgmap v17938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:37:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:37:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:07 smithi161 ceph-mon[91579]: pgmap v17938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:37:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:37:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:37:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:37:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:37:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:37:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:37:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:37:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:37:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:37:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:37:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:37:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:37:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:37:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:37:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:37:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:37:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:37:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:37:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:37:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:37:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:37:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:37:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:37:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:37:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:37:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:37:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:37:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:37:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:37:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:37:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:37:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:37:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:37:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:37:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:37:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:37:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:37:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:37:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:37:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:37:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:37:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:37:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:37:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:37:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:37:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:37:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:37:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:37:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1085] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:37:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1085] Local address: 172.21.15.161, port 60892 2022-07-03T20:37:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1085] Peer address: 172.21.15.101, port 22 2022-07-03T20:37:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1085] Connection lost 2022-07-03T20:37:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1085] Aborting connection 2022-07-03T20:37:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:37:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:37:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:37:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:37:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:37:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:37:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:37:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:37:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:37:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:37:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:37:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:37:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:37:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:37:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:37:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:37:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:37:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:37:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:37:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:37:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:37:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:37:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:37:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:37:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:37:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:37:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:37:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:37:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:37:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:37:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:37:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:37:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:37:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:37:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:37:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:37:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:37:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:37:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:37:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:37:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:37:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:37:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:37:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:37:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:37:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:37:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1085] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:37:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1085] Local address: 172.21.15.161, port 60892 2022-07-03T20:37:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1085] Peer address: 172.21.15.101, port 22 2022-07-03T20:37:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1085] Connection lost 2022-07-03T20:37:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1085] Aborting connection 2022-07-03T20:37:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:09 smithi161 ceph-mon[91579]: pgmap v17939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:09 smithi188 ceph-mon[92425]: pgmap v17939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:11 smithi161 ceph-mon[91579]: pgmap v17940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:11 smithi188 ceph-mon[92425]: pgmap v17940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:13 smithi161 ceph-mon[91579]: pgmap v17941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:13 smithi188 ceph-mon[92425]: pgmap v17941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:15 smithi161 ceph-mon[91579]: pgmap v17942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:15.672 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:15 smithi188 ceph-mon[92425]: pgmap v17942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:17 smithi161 ceph-mon[91579]: pgmap v17943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:17 smithi188 ceph-mon[92425]: pgmap v17943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:19 smithi161 ceph-mon[91579]: pgmap v17944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:19 smithi188 ceph-mon[92425]: pgmap v17944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:21 smithi161 ceph-mon[91579]: pgmap v17945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:21 smithi188 ceph-mon[92425]: pgmap v17945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:23 smithi161 ceph-mon[91579]: pgmap v17946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:23 smithi188 ceph-mon[92425]: pgmap v17946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:25 smithi161 ceph-mon[91579]: pgmap v17947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:25 smithi188 ceph-mon[92425]: pgmap v17947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:27 smithi161 ceph-mon[91579]: pgmap v17948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:27 smithi188 ceph-mon[92425]: pgmap v17948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:29 smithi161 ceph-mon[91579]: pgmap v17949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:29 smithi188 ceph-mon[92425]: pgmap v17949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:31 smithi161 ceph-mon[91579]: pgmap v17950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:31 smithi188 ceph-mon[92425]: pgmap v17950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:33 smithi161 ceph-mon[91579]: pgmap v17951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:33 smithi188 ceph-mon[92425]: pgmap v17951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:35 smithi161 ceph-mon[91579]: pgmap v17952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:35 smithi188 ceph-mon[92425]: pgmap v17952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:37 smithi161 ceph-mon[91579]: pgmap v17953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:37 smithi188 ceph-mon[92425]: pgmap v17953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:39 smithi161 ceph-mon[91579]: pgmap v17954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:39 smithi188 ceph-mon[92425]: pgmap v17954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:41.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:41 smithi161 ceph-mon[91579]: pgmap v17955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:41 smithi188 ceph-mon[92425]: pgmap v17955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:43 smithi161 ceph-mon[91579]: pgmap v17956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:43 smithi188 ceph-mon[92425]: pgmap v17956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:45 smithi161 ceph-mon[91579]: pgmap v17957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:45 smithi188 ceph-mon[92425]: pgmap v17957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:47 smithi161 ceph-mon[91579]: pgmap v17958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:47 smithi188 ceph-mon[92425]: pgmap v17958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:49 smithi161 ceph-mon[91579]: pgmap v17959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:49 smithi188 ceph-mon[92425]: pgmap v17959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:51 smithi161 ceph-mon[91579]: pgmap v17960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:51 smithi188 ceph-mon[92425]: pgmap v17960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:53 smithi161 ceph-mon[91579]: pgmap v17961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:53 smithi188 ceph-mon[92425]: pgmap v17961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:55 smithi161 ceph-mon[91579]: pgmap v17962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:55 smithi188 ceph-mon[92425]: pgmap v17962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:57 smithi161 ceph-mon[91579]: pgmap v17963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:57 smithi188 ceph-mon[92425]: pgmap v17963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:37:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:37:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:59 smithi161 ceph-mon[91579]: pgmap v17964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:37:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:37:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:37:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:37:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:59 smithi188 ceph-mon[92425]: pgmap v17964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:37:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:37:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:37:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:37:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:38:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:01 smithi161 ceph-mon[91579]: pgmap v17965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:01 smithi188 ceph-mon[92425]: pgmap v17965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:03 smithi161 ceph-mon[91579]: pgmap v17966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:03 smithi188 ceph-mon[92425]: pgmap v17966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:05 smithi161 ceph-mon[91579]: pgmap v17967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:05 smithi188 ceph-mon[92425]: pgmap v17967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-mon[91579]: pgmap v17968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:38:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:38:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:38:07.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:38:07.209+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:38:07.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:38:07.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:38:07.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:38:07.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:38:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:38:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:38:07.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:38:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:38:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:38:07.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:38:07.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:38:07.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:38:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:38:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:38:07.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:38:07.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:38:07.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:38:07.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:38:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:38:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:38:07.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:38:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:38:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:38:07.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:38:07.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:38:07.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:38:07.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:38:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:38:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:38:07.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:38:07.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:38:07.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:38:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:38:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:38:07.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:38:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:38:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:38:07.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:38:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:38:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:38:07.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:38:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:38:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:38:07.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:38:07.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1087] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:38:07.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1087] Local address: 172.21.15.161, port 60896 2022-07-03T20:38:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1087] Peer address: 172.21.15.101, port 22 2022-07-03T20:38:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1087] Connection lost 2022-07-03T20:38:07.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1087] Aborting connection 2022-07-03T20:38:07.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:38:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:07 smithi188 ceph-mon[92425]: pgmap v17968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:38:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:38:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:38:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:38:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:38:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:38:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:38:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:38:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:38:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:38:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:38:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:38:08.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:38:08.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:38:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:38:08.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:38:08.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:38:08.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:38:08.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:38:08.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:38:08.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:38:08.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:38:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:38:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:38:08.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:38:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:38:08.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:38:08.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1087] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:38:08.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1087] Local address: 172.21.15.161, port 60896 2022-07-03T20:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1087] Peer address: 172.21.15.101, port 22 2022-07-03T20:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1087] Connection lost 2022-07-03T20:38:08.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1087] Aborting connection 2022-07-03T20:38:08.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:38:08.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:38:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:38:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:38:08.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:38:08.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:38:08.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:38:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:38:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:38:08.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:38:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:38:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:38:08.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:38:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:38:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:38:08.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:38:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:38:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:38:08.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:38:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:38:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:38:08.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:38:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:38:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:38:08.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:38:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:38:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:38:08.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:38:08.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:38:08.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:38:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:38:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:38:08.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:38:08.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:38:08.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1087] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1087] Local address: 172.21.15.161, port 60896 2022-07-03T20:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1087] Peer address: 172.21.15.101, port 22 2022-07-03T20:38:08.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1087] Connection lost 2022-07-03T20:38:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1087] Aborting connection 2022-07-03T20:38:08.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:09 smithi161 ceph-mon[91579]: pgmap v17969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:09 smithi188 ceph-mon[92425]: pgmap v17969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:11 smithi161 ceph-mon[91579]: pgmap v17970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:11 smithi188 ceph-mon[92425]: pgmap v17970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:13 smithi161 ceph-mon[91579]: pgmap v17971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:13 smithi188 ceph-mon[92425]: pgmap v17971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:15 smithi161 ceph-mon[91579]: pgmap v17972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:15 smithi188 ceph-mon[92425]: pgmap v17972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:17 smithi161 ceph-mon[91579]: pgmap v17973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:17 smithi188 ceph-mon[92425]: pgmap v17973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:19 smithi161 ceph-mon[91579]: pgmap v17974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:19 smithi188 ceph-mon[92425]: pgmap v17974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:21 smithi161 ceph-mon[91579]: pgmap v17975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:21 smithi188 ceph-mon[92425]: pgmap v17975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:23 smithi161 ceph-mon[91579]: pgmap v17976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:23 smithi188 ceph-mon[92425]: pgmap v17976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:25 smithi161 ceph-mon[91579]: pgmap v17977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:25 smithi188 ceph-mon[92425]: pgmap v17977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:27 smithi161 ceph-mon[91579]: pgmap v17978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:27 smithi188 ceph-mon[92425]: pgmap v17978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:29 smithi161 ceph-mon[91579]: pgmap v17979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:29 smithi188 ceph-mon[92425]: pgmap v17979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:31 smithi161 ceph-mon[91579]: pgmap v17980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:31 smithi188 ceph-mon[92425]: pgmap v17980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:33 smithi161 ceph-mon[91579]: pgmap v17981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:33 smithi188 ceph-mon[92425]: pgmap v17981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:35 smithi161 ceph-mon[91579]: pgmap v17982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:35 smithi188 ceph-mon[92425]: pgmap v17982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:37 smithi161 ceph-mon[91579]: pgmap v17983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:37 smithi188 ceph-mon[92425]: pgmap v17983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:39 smithi161 ceph-mon[91579]: pgmap v17984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:39 smithi188 ceph-mon[92425]: pgmap v17984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:41 smithi161 ceph-mon[91579]: pgmap v17985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:41 smithi188 ceph-mon[92425]: pgmap v17985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:43 smithi161 ceph-mon[91579]: pgmap v17986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:43 smithi188 ceph-mon[92425]: pgmap v17986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:45 smithi161 ceph-mon[91579]: pgmap v17987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:45 smithi188 ceph-mon[92425]: pgmap v17987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:47 smithi161 ceph-mon[91579]: pgmap v17988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:47 smithi188 ceph-mon[92425]: pgmap v17988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:49 smithi161 ceph-mon[91579]: pgmap v17989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:49 smithi188 ceph-mon[92425]: pgmap v17989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:51 smithi161 ceph-mon[91579]: pgmap v17990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:51 smithi188 ceph-mon[92425]: pgmap v17990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:53 smithi161 ceph-mon[91579]: pgmap v17991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:53 smithi188 ceph-mon[92425]: pgmap v17991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:55 smithi161 ceph-mon[91579]: pgmap v17992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:55 smithi188 ceph-mon[92425]: pgmap v17992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:57 smithi161 ceph-mon[91579]: pgmap v17993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:57 smithi188 ceph-mon[92425]: pgmap v17993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:38:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:38:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:59 smithi188 ceph-mon[92425]: pgmap v17994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:38:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:38:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:38:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:38:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:59 smithi161 ceph-mon[91579]: pgmap v17994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:38:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:38:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:38:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:38:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:39:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:01 smithi188 ceph-mon[92425]: pgmap v17995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:01 smithi161 ceph-mon[91579]: pgmap v17995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:03 smithi188 ceph-mon[92425]: pgmap v17996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:03 smithi161 ceph-mon[91579]: pgmap v17996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:05 smithi188 ceph-mon[92425]: pgmap v17997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:05 smithi161 ceph-mon[91579]: pgmap v17997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:07 smithi188 ceph-mon[92425]: pgmap v17998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:07.707 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-mon[91579]: pgmap v17998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:08.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:39:07.707+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:39:08.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:39:08.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:39:08.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:39:08.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:39:08.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:39:08.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:39:08.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:39:08.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:39:08.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:39:08.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:39:08.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:39:08.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:39:08.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:39:08.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:39:08.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:39:08.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:39:08.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:39:08.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:39:08.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:39:08.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:39:08.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:39:08.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:39:08.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:39:08.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:39:08.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:39:08.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:39:08.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:39:08.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:39:08.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:39:08.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:39:08.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:39:08.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:39:08.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:39:08.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:39:08.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:39:08.152 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:39:08.153 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:39:08.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:39:08.154 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:39:08.155 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:39:08.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:39:08.156 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:39:08.157 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:39:08.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:39:08.158 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1089] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:39:08.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1089] Local address: 172.21.15.161, port 60930 2022-07-03T20:39:08.159 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1089] Peer address: 172.21.15.101, port 22 2022-07-03T20:39:08.160 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1089] Connection lost 2022-07-03T20:39:08.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1089] Aborting connection 2022-07-03T20:39:08.161 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:39:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:39:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:39:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:08.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:39:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:39:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:39:08.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:08.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:39:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:39:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:39:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:39:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:39:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:39:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:39:09.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:39:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:39:09.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:39:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:39:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:39:09.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:39:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:39:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:39:09.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:39:09.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:39:09.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:39:09.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:39:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:39:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:39:09.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:39:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:39:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:39:09.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:39:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:39:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:39:09.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:39:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:39:09.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:39:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:39:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:39:09.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:39:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:39:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:39:09.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:39:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:39:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:39:09.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:39:09.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:39:09.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:39:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:39:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:39:09.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:39:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:39:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:39:09.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:39:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1089] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:39:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1089] Local address: 172.21.15.161, port 60930 2022-07-03T20:39:09.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1089] Peer address: 172.21.15.101, port 22 2022-07-03T20:39:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1089] Connection lost 2022-07-03T20:39:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1089] Aborting connection 2022-07-03T20:39:09.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:09 smithi188 ceph-mon[92425]: pgmap v17999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:39:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:39:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:39:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:39:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:39:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:39:09.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:39:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:39:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:39:09.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:39:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:39:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:39:09.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:39:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:39:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:39:09.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:39:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:39:09.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:39:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:39:09.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:39:09.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:39:09.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:39:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:39:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:39:09.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:39:09.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:39:09.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:39:09.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:39:09.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:39:09.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:39:09.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:39:09.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:39:09.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:39:09.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:39:09.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:39:09.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:39:09.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:39:09.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:39:09.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:39:09.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:39:09.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:39:09.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:39:09.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:39:09.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:39:09.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:39:09.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:39:09.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1089] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:39:09.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1089] Local address: 172.21.15.161, port 60930 2022-07-03T20:39:09.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1089] Peer address: 172.21.15.101, port 22 2022-07-03T20:39:09.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1089] Connection lost 2022-07-03T20:39:09.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1089] Aborting connection 2022-07-03T20:39:09.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:09 smithi161 ceph-mon[91579]: pgmap v17999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:11 smithi188 ceph-mon[92425]: pgmap v18000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:11 smithi161 ceph-mon[91579]: pgmap v18000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:13 smithi188 ceph-mon[92425]: pgmap v18001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:13 smithi161 ceph-mon[91579]: pgmap v18001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:15 smithi161 ceph-mon[91579]: pgmap v18002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:15 smithi188 ceph-mon[92425]: pgmap v18002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:17 smithi161 ceph-mon[91579]: pgmap v18003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:17 smithi188 ceph-mon[92425]: pgmap v18003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:18.921 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:19 smithi161 ceph-mon[91579]: pgmap v18004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:19 smithi188 ceph-mon[92425]: pgmap v18004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:21 smithi161 ceph-mon[91579]: pgmap v18005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:21 smithi188 ceph-mon[92425]: pgmap v18005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:23 smithi161 ceph-mon[91579]: pgmap v18006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:23 smithi188 ceph-mon[92425]: pgmap v18006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:25 smithi161 ceph-mon[91579]: pgmap v18007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:25 smithi188 ceph-mon[92425]: pgmap v18007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:27 smithi161 ceph-mon[91579]: pgmap v18008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:27 smithi188 ceph-mon[92425]: pgmap v18008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:29 smithi161 ceph-mon[91579]: pgmap v18009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:29 smithi188 ceph-mon[92425]: pgmap v18009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:31 smithi161 ceph-mon[91579]: pgmap v18010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:31 smithi188 ceph-mon[92425]: pgmap v18010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:33 smithi161 ceph-mon[91579]: pgmap v18011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:33 smithi188 ceph-mon[92425]: pgmap v18011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:35 smithi161 ceph-mon[91579]: pgmap v18012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:35 smithi188 ceph-mon[92425]: pgmap v18012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:37 smithi161 ceph-mon[91579]: pgmap v18013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:37 smithi188 ceph-mon[92425]: pgmap v18013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:39 smithi161 ceph-mon[91579]: pgmap v18014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:39 smithi188 ceph-mon[92425]: pgmap v18014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:41 smithi161 ceph-mon[91579]: pgmap v18015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:41 smithi188 ceph-mon[92425]: pgmap v18015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:43 smithi161 ceph-mon[91579]: pgmap v18016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:43 smithi188 ceph-mon[92425]: pgmap v18016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:45 smithi161 ceph-mon[91579]: pgmap v18017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:45 smithi188 ceph-mon[92425]: pgmap v18017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:47 smithi161 ceph-mon[91579]: pgmap v18018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:47 smithi188 ceph-mon[92425]: pgmap v18018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:49 smithi161 ceph-mon[91579]: pgmap v18019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:49 smithi188 ceph-mon[92425]: pgmap v18019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:51 smithi161 ceph-mon[91579]: pgmap v18020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:51 smithi188 ceph-mon[92425]: pgmap v18020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:53 smithi161 ceph-mon[91579]: pgmap v18021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:53 smithi188 ceph-mon[92425]: pgmap v18021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:55 smithi161 ceph-mon[91579]: pgmap v18022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:55 smithi188 ceph-mon[92425]: pgmap v18022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:57 smithi161 ceph-mon[91579]: pgmap v18023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:57 smithi188 ceph-mon[92425]: pgmap v18023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:39:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:39:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:59 smithi161 ceph-mon[91579]: pgmap v18024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:39:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:39:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:39:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:39:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:59 smithi188 ceph-mon[92425]: pgmap v18024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:39:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:39:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:39:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:39:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:40:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:00 smithi161 ceph-mon[91579]: pgmap v18025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:40:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:00 smithi188 ceph-mon[92425]: pgmap v18025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:40:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:02 smithi161 ceph-mon[91579]: pgmap v18026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:03.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:03.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:02 smithi188 ceph-mon[92425]: pgmap v18026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:03.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:04 smithi161 ceph-mon[91579]: pgmap v18027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:04 smithi188 ceph-mon[92425]: pgmap v18027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:06 smithi161 ceph-mon[91579]: pgmap v18028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:06 smithi188 ceph-mon[92425]: pgmap v18028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-mon[91579]: pgmap v18029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:40:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:40:08.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:40:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:40:08.624+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:40:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:40:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:40:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:40:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:40:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:40:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:40:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:40:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:40:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:40:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:40:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:40:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:40:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:40:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:40:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:40:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:40:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:40:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:40:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:40:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:40:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:40:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:40:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:40:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1091] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:40:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1091] Local address: 172.21.15.161, port 60934 2022-07-03T20:40:08.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1091] Peer address: 172.21.15.101, port 22 2022-07-03T20:40:08.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1091] Connection lost 2022-07-03T20:40:08.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1091] Aborting connection 2022-07-03T20:40:08.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:40:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:08 smithi188 ceph-mon[92425]: pgmap v18029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:40:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:40:08.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:40:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:40:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:40:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:40:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:40:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:40:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:40:10.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:40:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:40:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:40:10.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:40:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:40:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:40:10.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:40:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:40:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:40:10.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:40:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:40:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:40:10.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:40:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:40:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:40:10.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:40:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:40:10.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:40:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:40:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:40:10.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:40:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:40:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:40:10.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:40:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:40:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:40:10.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:40:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:40:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:40:10.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:40:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:40:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:40:10.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:40:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:40:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:40:10.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:40:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:40:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:40:10.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:40:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:40:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1091] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:40:10.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1091] Local address: 172.21.15.161, port 60934 2022-07-03T20:40:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1091] Peer address: 172.21.15.101, port 22 2022-07-03T20:40:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1091] Connection lost 2022-07-03T20:40:10.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1091] Aborting connection 2022-07-03T20:40:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:40:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:40:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:40:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:40:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:40:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:40:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:40:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:40:10.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:40:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:40:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:40:10.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:40:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:40:10.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:40:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:40:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:40:10.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:40:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:40:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:40:10.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:40:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:40:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:40:10.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:40:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:40:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:40:10.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:40:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:40:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:40:10.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:40:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:40:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:40:10.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:40:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:40:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:40:10.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:40:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:40:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:40:10.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:40:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:40:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:40:10.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:40:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:40:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:40:10.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:40:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:40:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:40:10.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1091] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:40:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1091] Local address: 172.21.15.161, port 60934 2022-07-03T20:40:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1091] Peer address: 172.21.15.101, port 22 2022-07-03T20:40:10.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1091] Connection lost 2022-07-03T20:40:10.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1091] Aborting connection 2022-07-03T20:40:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:10 smithi161 ceph-mon[91579]: pgmap v18030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:10 smithi188 ceph-mon[92425]: pgmap v18030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:12 smithi161 ceph-mon[91579]: pgmap v18031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:12 smithi188 ceph-mon[92425]: pgmap v18031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:13.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:14 smithi161 ceph-mon[91579]: pgmap v18032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:14 smithi188 ceph-mon[92425]: pgmap v18032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:16 smithi161 ceph-mon[91579]: pgmap v18033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:16 smithi188 ceph-mon[92425]: pgmap v18033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:18 smithi188 ceph-mon[92425]: pgmap v18034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:18 smithi161 ceph-mon[91579]: pgmap v18034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:20 smithi188 ceph-mon[92425]: pgmap v18035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:20 smithi161 ceph-mon[91579]: pgmap v18035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:22 smithi188 ceph-mon[92425]: pgmap v18036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:22 smithi161 ceph-mon[91579]: pgmap v18036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:24 smithi188 ceph-mon[92425]: pgmap v18037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:24 smithi161 ceph-mon[91579]: pgmap v18037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:26 smithi188 ceph-mon[92425]: pgmap v18038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:26 smithi161 ceph-mon[91579]: pgmap v18038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:29.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:28 smithi188 ceph-mon[92425]: pgmap v18039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:28 smithi161 ceph-mon[91579]: pgmap v18039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:30 smithi188 ceph-mon[92425]: pgmap v18040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:30 smithi161 ceph-mon[91579]: pgmap v18040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:33.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:32 smithi188 ceph-mon[92425]: pgmap v18041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:33.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:32 smithi161 ceph-mon[91579]: pgmap v18041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:34 smithi188 ceph-mon[92425]: pgmap v18042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:34 smithi161 ceph-mon[91579]: pgmap v18042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:36 smithi161 ceph-mon[91579]: pgmap v18043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:36 smithi188 ceph-mon[92425]: pgmap v18043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:38 smithi161 ceph-mon[91579]: pgmap v18044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:38 smithi188 ceph-mon[92425]: pgmap v18044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:40 smithi161 ceph-mon[91579]: pgmap v18045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:40 smithi188 ceph-mon[92425]: pgmap v18045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:42 smithi161 ceph-mon[91579]: pgmap v18046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:42 smithi188 ceph-mon[92425]: pgmap v18046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:44 smithi161 ceph-mon[91579]: pgmap v18047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:44 smithi188 ceph-mon[92425]: pgmap v18047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:46 smithi161 ceph-mon[91579]: pgmap v18048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:46 smithi188 ceph-mon[92425]: pgmap v18048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:48 smithi161 ceph-mon[91579]: pgmap v18049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:48 smithi188 ceph-mon[92425]: pgmap v18049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:50 smithi161 ceph-mon[91579]: pgmap v18050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:50 smithi188 ceph-mon[92425]: pgmap v18050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:52 smithi161 ceph-mon[91579]: pgmap v18051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:52 smithi188 ceph-mon[92425]: pgmap v18051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:54 smithi161 ceph-mon[91579]: pgmap v18052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:54 smithi188 ceph-mon[92425]: pgmap v18052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:56 smithi161 ceph-mon[91579]: pgmap v18053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:56 smithi188 ceph-mon[92425]: pgmap v18053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:40:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:40:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:58 smithi161 ceph-mon[91579]: pgmap v18054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:40:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:58 smithi188 ceph-mon[92425]: pgmap v18054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:00 smithi161 ceph-mon[91579]: pgmap v18055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:00 smithi188 ceph-mon[92425]: pgmap v18055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:02 smithi161 ceph-mon[91579]: pgmap v18056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:02 smithi188 ceph-mon[92425]: pgmap v18056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:04 smithi161 ceph-mon[91579]: pgmap v18057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:04 smithi188 ceph-mon[92425]: pgmap v18057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:06 smithi161 ceph-mon[91579]: pgmap v18058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:06 smithi188 ceph-mon[92425]: pgmap v18058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:08 smithi161 ceph-mon[91579]: pgmap v18059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:09.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:41:09.129+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:41:09.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:41:09.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:41:09.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:41:09.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:41:09.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:41:09.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:41:09.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:41:09.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:41:09.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:41:09.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:41:09.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:41:09.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:41:09.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:41:09.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:41:09.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:41:09.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:41:09.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:41:09.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:41:09.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:41:09.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:41:09.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:41:09.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:41:09.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:41:09.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:41:09.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:41:09.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:41:09.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:41:09.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:41:09.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:41:09.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:41:09.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:41:09.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:41:09.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:41:09.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:41:09.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:41:09.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:41:09.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:41:09.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:41:09.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:41:09.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:41:09.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:41:09.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:41:09.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:41:09.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:41:09.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1093] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:41:09.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1093] Local address: 172.21.15.161, port 60938 2022-07-03T20:41:09.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1093] Peer address: 172.21.15.101, port 22 2022-07-03T20:41:09.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1093] Connection lost 2022-07-03T20:41:09.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1093] Aborting connection 2022-07-03T20:41:09.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:41:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:08 smithi188 ceph-mon[92425]: pgmap v18059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:41:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:41:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:41:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:41:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:41:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:41:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:41:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:41:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:41:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:41:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:41:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:41:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:41:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:41:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:41:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:41:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:41:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:41:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:41:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:41:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:41:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:41:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:41:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:41:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:41:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:41:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:41:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:41:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:41:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:41:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:41:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:41:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:41:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:41:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:41:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:41:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:41:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:41:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:41:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:41:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:41:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:41:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:41:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:41:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:41:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1093] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:41:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1093] Local address: 172.21.15.161, port 60938 2022-07-03T20:41:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1093] Peer address: 172.21.15.101, port 22 2022-07-03T20:41:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1093] Connection lost 2022-07-03T20:41:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1093] Aborting connection 2022-07-03T20:41:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:41:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:41:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:41:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:41:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:41:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:41:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:41:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:41:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:41:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:41:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:41:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:41:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:41:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:41:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:41:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:41:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:41:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:41:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:41:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:41:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:41:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:41:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:41:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:41:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:41:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:41:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:41:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:41:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:41:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:41:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:41:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:41:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:41:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:41:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:41:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:41:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:41:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:41:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:41:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:41:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:41:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:41:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:41:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:41:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:41:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:41:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1093] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:41:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1093] Local address: 172.21.15.161, port 60938 2022-07-03T20:41:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1093] Peer address: 172.21.15.101, port 22 2022-07-03T20:41:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1093] Connection lost 2022-07-03T20:41:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1093] Aborting connection 2022-07-03T20:41:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:10 smithi161 ceph-mon[91579]: pgmap v18060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:10 smithi188 ceph-mon[92425]: pgmap v18060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:12 smithi161 ceph-mon[91579]: pgmap v18061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:12 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:12 smithi188 ceph-mon[92425]: pgmap v18061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:12 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:14 smithi161 ceph-mon[91579]: pgmap v18062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:14 smithi188 ceph-mon[92425]: pgmap v18062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:17 smithi161 ceph-mon[91579]: pgmap v18063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:17 smithi188 ceph-mon[92425]: pgmap v18063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:41:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:19 smithi161 ceph-mon[91579]: pgmap v18064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:19 smithi188 ceph-mon[92425]: pgmap v18064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:21 smithi161 ceph-mon[91579]: pgmap v18065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:21 smithi188 ceph-mon[92425]: pgmap v18065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:23 smithi161 ceph-mon[91579]: pgmap v18066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:23 smithi188 ceph-mon[92425]: pgmap v18066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:25 smithi161 ceph-mon[91579]: pgmap v18067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:25 smithi188 ceph-mon[92425]: pgmap v18067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:27 smithi161 ceph-mon[91579]: pgmap v18068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:27 smithi188 ceph-mon[92425]: pgmap v18068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:29 smithi161 ceph-mon[91579]: pgmap v18069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:29 smithi188 ceph-mon[92425]: pgmap v18069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:31 smithi161 ceph-mon[91579]: pgmap v18070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:31 smithi188 ceph-mon[92425]: pgmap v18070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:33 smithi161 ceph-mon[91579]: pgmap v18071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:33 smithi188 ceph-mon[92425]: pgmap v18071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:35 smithi161 ceph-mon[91579]: pgmap v18072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:35 smithi188 ceph-mon[92425]: pgmap v18072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:37 smithi161 ceph-mon[91579]: pgmap v18073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:37 smithi188 ceph-mon[92425]: pgmap v18073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:39 smithi161 ceph-mon[91579]: pgmap v18074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:39 smithi188 ceph-mon[92425]: pgmap v18074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:41 smithi161 ceph-mon[91579]: pgmap v18075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:41 smithi188 ceph-mon[92425]: pgmap v18075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:43 smithi161 ceph-mon[91579]: pgmap v18076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:43 smithi188 ceph-mon[92425]: pgmap v18076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:45 smithi161 ceph-mon[91579]: pgmap v18077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:45 smithi188 ceph-mon[92425]: pgmap v18077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:47 smithi161 ceph-mon[91579]: pgmap v18078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:47 smithi188 ceph-mon[92425]: pgmap v18078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:49 smithi161 ceph-mon[91579]: pgmap v18079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:49 smithi188 ceph-mon[92425]: pgmap v18079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:51 smithi161 ceph-mon[91579]: pgmap v18080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:51 smithi188 ceph-mon[92425]: pgmap v18080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:53 smithi161 ceph-mon[91579]: pgmap v18081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:53.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:53 smithi188 ceph-mon[92425]: pgmap v18081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:55 smithi161 ceph-mon[91579]: pgmap v18082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:55 smithi188 ceph-mon[92425]: pgmap v18082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:57 smithi161 ceph-mon[91579]: pgmap v18083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:57 smithi188 ceph-mon[92425]: pgmap v18083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:41:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:41:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:59 smithi161 ceph-mon[91579]: pgmap v18084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:59 smithi188 ceph-mon[92425]: pgmap v18084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:41:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:41:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:41:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:42:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:01 smithi161 ceph-mon[91579]: pgmap v18085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:01 smithi188 ceph-mon[92425]: pgmap v18085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:03 smithi161 ceph-mon[91579]: pgmap v18086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:03.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:03 smithi188 ceph-mon[92425]: pgmap v18086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:05 smithi161 ceph-mon[91579]: pgmap v18087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:05 smithi188 ceph-mon[92425]: pgmap v18087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:07 smithi161 ceph-mon[91579]: pgmap v18088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:07 smithi188 ceph-mon[92425]: pgmap v18088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:09 smithi188 ceph-mon[92425]: pgmap v18089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-mon[91579]: pgmap v18089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:10.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:42:09.637+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:42:10.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:42:10.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:42:10.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:42:10.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:42:10.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:42:10.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:42:10.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:42:10.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:42:10.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:42:10.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:42:10.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:42:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:42:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:42:10.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:42:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:42:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:42:10.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:42:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:42:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:42:10.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:42:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:42:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:42:10.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:42:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:42:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:42:10.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:42:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:42:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:42:10.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:42:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:42:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:42:10.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:42:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:42:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:42:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:42:10.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:42:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:42:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:42:10.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:42:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:42:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:42:10.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:42:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:42:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:42:10.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1095] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:42:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1095] Local address: 172.21.15.161, port 60942 2022-07-03T20:42:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1095] Peer address: 172.21.15.101, port 22 2022-07-03T20:42:10.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1095] Connection lost 2022-07-03T20:42:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1095] Aborting connection 2022-07-03T20:42:10.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:42:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:42:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:42:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:42:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:42:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:42:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:42:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:42:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:42:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:42:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:42:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:42:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:42:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:42:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:42:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:42:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:42:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:42:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:42:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:42:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:42:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:42:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:42:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:42:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:42:11.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:42:11.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:42:11.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:42:11.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:42:11.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:42:11.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:42:11.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:42:11.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:42:11.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:42:11.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:42:11.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:42:11.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1095] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:42:11.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1095] Local address: 172.21.15.161, port 60942 2022-07-03T20:42:11.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1095] Peer address: 172.21.15.101, port 22 2022-07-03T20:42:11.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1095] Connection lost 2022-07-03T20:42:11.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1095] Aborting connection 2022-07-03T20:42:11.435 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:11 smithi188 ceph-mon[92425]: pgmap v18090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:42:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:42:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:42:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:42:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:42:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:42:11.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:42:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:42:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:42:11.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:42:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:42:11.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:42:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:42:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:42:11.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:42:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:42:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:42:11.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:42:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:42:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:42:11.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:42:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:42:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:42:11.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:42:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:42:11.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:42:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:42:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:42:11.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:42:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:42:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:42:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:42:11.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:42:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:42:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:42:11.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:42:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:42:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:42:11.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:42:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:42:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:42:11.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:42:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:42:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:42:11.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:42:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:42:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1095] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:42:11.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1095] Local address: 172.21.15.161, port 60942 2022-07-03T20:42:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1095] Peer address: 172.21.15.101, port 22 2022-07-03T20:42:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1095] Connection lost 2022-07-03T20:42:11.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1095] Aborting connection 2022-07-03T20:42:11.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:11 smithi161 ceph-mon[91579]: pgmap v18090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: pgmap v18091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: pgmap v18091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:13.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:42:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:14 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 63044k 2022-07-03T20:42:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:14 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 64557523: error parsing value: Value '64557523' is below minimum 939524096 2022-07-03T20:42:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:14 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 63044k 2022-07-03T20:42:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:14 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 64557523: error parsing value: Value '64557523' is below minimum 939524096 2022-07-03T20:42:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:15 smithi161 ceph-mon[91579]: pgmap v18092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:15 smithi188 ceph-mon[92425]: pgmap v18092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:17 smithi161 ceph-mon[91579]: pgmap v18093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:17 smithi188 ceph-mon[92425]: pgmap v18093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:19 smithi161 ceph-mon[91579]: pgmap v18094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:19 smithi188 ceph-mon[92425]: pgmap v18094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:21 smithi161 ceph-mon[91579]: pgmap v18095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:21 smithi188 ceph-mon[92425]: pgmap v18095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:23 smithi161 ceph-mon[91579]: pgmap v18096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:23 smithi188 ceph-mon[92425]: pgmap v18096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:25 smithi161 ceph-mon[91579]: pgmap v18097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:25 smithi188 ceph-mon[92425]: pgmap v18097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:27 smithi161 ceph-mon[91579]: pgmap v18098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:27 smithi188 ceph-mon[92425]: pgmap v18098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:29 smithi161 ceph-mon[91579]: pgmap v18099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:29 smithi188 ceph-mon[92425]: pgmap v18099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:31 smithi161 ceph-mon[91579]: pgmap v18100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:31 smithi188 ceph-mon[92425]: pgmap v18100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:33 smithi161 ceph-mon[91579]: pgmap v18101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:33 smithi188 ceph-mon[92425]: pgmap v18101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:35 smithi161 ceph-mon[91579]: pgmap v18102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:35 smithi188 ceph-mon[92425]: pgmap v18102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:37 smithi161 ceph-mon[91579]: pgmap v18103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:37 smithi188 ceph-mon[92425]: pgmap v18103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:39 smithi161 ceph-mon[91579]: pgmap v18104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:39 smithi188 ceph-mon[92425]: pgmap v18104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:41 smithi161 ceph-mon[91579]: pgmap v18105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:41 smithi188 ceph-mon[92425]: pgmap v18105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:43 smithi161 ceph-mon[91579]: pgmap v18106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:43 smithi188 ceph-mon[92425]: pgmap v18106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:45 smithi161 ceph-mon[91579]: pgmap v18107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:45 smithi188 ceph-mon[92425]: pgmap v18107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:47 smithi161 ceph-mon[91579]: pgmap v18108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:47 smithi188 ceph-mon[92425]: pgmap v18108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:49 smithi161 ceph-mon[91579]: pgmap v18109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:49 smithi188 ceph-mon[92425]: pgmap v18109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:51 smithi161 ceph-mon[91579]: pgmap v18110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:51 smithi188 ceph-mon[92425]: pgmap v18110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:53 smithi161 ceph-mon[91579]: pgmap v18111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:53 smithi188 ceph-mon[92425]: pgmap v18111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:55 smithi161 ceph-mon[91579]: pgmap v18112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:55 smithi188 ceph-mon[92425]: pgmap v18112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:57 smithi161 ceph-mon[91579]: pgmap v18113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:57 smithi188 ceph-mon[92425]: pgmap v18113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:42:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:42:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:59 smithi161 ceph-mon[91579]: pgmap v18114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:42:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:42:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:42:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:42:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:59 smithi188 ceph-mon[92425]: pgmap v18114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:42:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:42:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:42:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:42:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:43:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:01 smithi161 ceph-mon[91579]: pgmap v18115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:01 smithi188 ceph-mon[92425]: pgmap v18115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:03 smithi188 ceph-mon[92425]: pgmap v18116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:03 smithi161 ceph-mon[91579]: pgmap v18116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:05 smithi188 ceph-mon[92425]: pgmap v18117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:05 smithi161 ceph-mon[91579]: pgmap v18117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:07 smithi188 ceph-mon[92425]: pgmap v18118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:07 smithi161 ceph-mon[91579]: pgmap v18118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:09 smithi188 ceph-mon[92425]: pgmap v18119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:09 smithi161 ceph-mon[91579]: pgmap v18119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:11 smithi188 ceph-mon[92425]: pgmap v18120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:11 smithi161 ceph-mon[91579]: pgmap v18120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:13.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:43:12.898+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:43:13.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:43:13.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:43:13.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:43:13.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:43:13.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:43:13.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:43:13.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:43:13.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:43:13.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:43:13.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:43:13.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:43:13.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:43:13.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:43:13.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:43:13.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:43:13.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:43:13.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:43:13.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:43:13.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:43:13.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:43:13.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:43:13.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:43:13.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:43:13.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:43:13.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:43:13.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:43:13.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:43:13.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:43:13.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:43:13.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:43:13.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:43:13.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:43:13.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:43:13.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:43:13.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:43:13.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:43:13.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:43:13.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:43:13.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:43:13.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:43:13.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:43:13.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:43:13.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:43:13.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:43:13.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1097] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:43:13.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1097] Local address: 172.21.15.161, port 60952 2022-07-03T20:43:13.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1097] Peer address: 172.21.15.101, port 22 2022-07-03T20:43:13.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1097] Connection lost 2022-07-03T20:43:13.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1097] Aborting connection 2022-07-03T20:43:13.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:43:12 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:43:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:13 smithi188 ceph-mon[92425]: pgmap v18121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:43:13.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:43:13.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:43:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:13 smithi161 ceph-mon[91579]: pgmap v18121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:43:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:43:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:43:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:43:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:43:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:43:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:43:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:43:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:43:14.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:43:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:43:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:43:14.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:43:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:43:14.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:43:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:43:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:43:14.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:43:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:43:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:43:14.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:43:14.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:43:14.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:43:14.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:43:14.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:43:14.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:43:14.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:43:14.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:43:14.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:43:14.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:43:14.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:43:14.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:43:14.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:43:14.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:43:14.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:43:14.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:43:14.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:43:14.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:43:14.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:43:14.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:43:14.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:43:14.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:43:14.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:43:14.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:43:14.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:43:14.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1097] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1097] Local address: 172.21.15.161, port 60952 2022-07-03T20:43:14.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1097] Peer address: 172.21.15.101, port 22 2022-07-03T20:43:14.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1097] Connection lost 2022-07-03T20:43:14.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1097] Aborting connection 2022-07-03T20:43:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:43:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:43:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:43:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:43:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:43:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:43:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:43:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:43:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:43:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:43:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:43:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:43:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:43:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:43:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:43:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:43:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:43:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:43:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:43:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:43:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:43:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:43:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:43:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:43:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:43:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:43:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:43:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:43:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:43:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:43:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:43:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:43:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:43:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:43:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:43:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:43:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:43:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:43:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:43:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:43:14.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1097] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:43:14.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1097] Local address: 172.21.15.161, port 60952 2022-07-03T20:43:14.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1097] Peer address: 172.21.15.101, port 22 2022-07-03T20:43:14.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1097] Connection lost 2022-07-03T20:43:14.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1097] Aborting connection 2022-07-03T20:43:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:15 smithi188 ceph-mon[92425]: pgmap v18122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:15 smithi161 ceph-mon[91579]: pgmap v18122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:17 smithi188 ceph-mon[92425]: pgmap v18123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:17 smithi161 ceph-mon[91579]: pgmap v18123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:18.686 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:18.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:19.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:19 smithi188 ceph-mon[92425]: pgmap v18124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:19 smithi161 ceph-mon[91579]: pgmap v18124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:20.703 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:21 smithi188 ceph-mon[92425]: pgmap v18125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:21 smithi161 ceph-mon[91579]: pgmap v18125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:23 smithi161 ceph-mon[91579]: pgmap v18126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:23 smithi188 ceph-mon[92425]: pgmap v18126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:25.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:25 smithi188 ceph-mon[92425]: pgmap v18127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:25.719 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:25 smithi161 ceph-mon[91579]: pgmap v18127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:27 smithi161 ceph-mon[91579]: pgmap v18128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:27 smithi188 ceph-mon[92425]: pgmap v18128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:28.722 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:29 smithi161 ceph-mon[91579]: pgmap v18129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:29 smithi188 ceph-mon[92425]: pgmap v18129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:31 smithi161 ceph-mon[91579]: pgmap v18130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:31 smithi188 ceph-mon[92425]: pgmap v18130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:33 smithi161 ceph-mon[91579]: pgmap v18131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:33 smithi188 ceph-mon[92425]: pgmap v18131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:35 smithi161 ceph-mon[91579]: pgmap v18132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:35 smithi188 ceph-mon[92425]: pgmap v18132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:37 smithi161 ceph-mon[91579]: pgmap v18133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:37 smithi188 ceph-mon[92425]: pgmap v18133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:39 smithi161 ceph-mon[91579]: pgmap v18134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:39 smithi188 ceph-mon[92425]: pgmap v18134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:40.732 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:41 smithi161 ceph-mon[91579]: pgmap v18135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:41 smithi188 ceph-mon[92425]: pgmap v18135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:43 smithi161 ceph-mon[91579]: pgmap v18136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:43 smithi188 ceph-mon[92425]: pgmap v18136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:45 smithi161 ceph-mon[91579]: pgmap v18137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:45 smithi188 ceph-mon[92425]: pgmap v18137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:47 smithi161 ceph-mon[91579]: pgmap v18138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:47 smithi188 ceph-mon[92425]: pgmap v18138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:49 smithi161 ceph-mon[91579]: pgmap v18139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:49 smithi188 ceph-mon[92425]: pgmap v18139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:51 smithi161 ceph-mon[91579]: pgmap v18140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:51 smithi188 ceph-mon[92425]: pgmap v18140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:53 smithi161 ceph-mon[91579]: pgmap v18141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:53 smithi188 ceph-mon[92425]: pgmap v18141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:55 smithi161 ceph-mon[91579]: pgmap v18142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:55 smithi188 ceph-mon[92425]: pgmap v18142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:43:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:57 smithi161 ceph-mon[91579]: pgmap v18143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:57 smithi188 ceph-mon[92425]: pgmap v18143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:43:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:43:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:59 smithi161 ceph-mon[91579]: pgmap v18144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:43:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:43:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:43:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:43:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:43:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:59 smithi188 ceph-mon[92425]: pgmap v18144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:43:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:43:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:43:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:43:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:43:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:44:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:01 smithi161 ceph-mon[91579]: pgmap v18145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:01 smithi188 ceph-mon[92425]: pgmap v18145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:03 smithi161 ceph-mon[91579]: pgmap v18146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:03 smithi188 ceph-mon[92425]: pgmap v18146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:05 smithi161 ceph-mon[91579]: pgmap v18147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:05 smithi188 ceph-mon[92425]: pgmap v18147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:07 smithi161 ceph-mon[91579]: pgmap v18148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:07 smithi188 ceph-mon[92425]: pgmap v18148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:09 smithi161 ceph-mon[91579]: pgmap v18149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:09 smithi188 ceph-mon[92425]: pgmap v18149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:11 smithi161 ceph-mon[91579]: pgmap v18150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:11 smithi188 ceph-mon[92425]: pgmap v18150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-mon[91579]: pgmap v18151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:44:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:44:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:44:13.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:44:13.408+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:44:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:44:13.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:44:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:44:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:44:13.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:44:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:44:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:44:13.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:44:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:44:13.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:44:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:44:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:44:13.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:44:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:44:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:44:13.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:44:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:44:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:44:13.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:44:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:44:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:44:13.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:44:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:44:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:44:13.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:44:13.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:44:13.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:44:13.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:44:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:44:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:44:13.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:44:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:44:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:44:13.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:44:13.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:44:13.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:44:13.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:44:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:44:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:44:13.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:44:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:44:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:44:13.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:44:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:44:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1099] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:44:13.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1099] Local address: 172.21.15.161, port 60980 2022-07-03T20:44:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1099] Peer address: 172.21.15.101, port 22 2022-07-03T20:44:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1099] Connection lost 2022-07-03T20:44:13.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1099] Aborting connection 2022-07-03T20:44:13.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:44:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:13 smithi188 ceph-mon[92425]: pgmap v18151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:44:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:44:13.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:44:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:44:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:44:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:44:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:44:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:44:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:44:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:44:14.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:44:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:44:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:44:14.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:44:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:44:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:44:14.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:44:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:44:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:44:14.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:44:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:44:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:44:14.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:44:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:44:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:44:14.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:44:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:44:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:44:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:44:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:44:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:44:14.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:44:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:44:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:44:14.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:44:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:44:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:44:14.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:44:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:44:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:44:14.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:44:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:44:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:44:14.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:44:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:44:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:44:14.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:44:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:44:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:44:14.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1099] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:44:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1099] Local address: 172.21.15.161, port 60980 2022-07-03T20:44:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1099] Peer address: 172.21.15.101, port 22 2022-07-03T20:44:14.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1099] Connection lost 2022-07-03T20:44:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1099] Aborting connection 2022-07-03T20:44:14.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:44:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:44:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:44:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:44:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:44:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:44:14.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:44:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:44:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:44:14.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:44:14.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:44:14.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:44:14.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:44:14.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:44:14.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:44:14.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:44:14.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:44:14.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:44:14.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:44:14.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:44:14.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:44:14.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:44:14.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:44:14.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:44:14.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:44:14.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:44:14.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:44:14.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:44:14.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:44:14.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:44:14.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:44:14.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:44:14.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:44:14.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:44:14.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:44:14.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:44:14.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:44:14.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:44:14.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:44:14.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:44:14.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:44:14.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:44:14.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:44:14.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:44:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:44:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:44:14.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1099] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:44:14.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1099] Local address: 172.21.15.161, port 60980 2022-07-03T20:44:14.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1099] Peer address: 172.21.15.101, port 22 2022-07-03T20:44:14.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1099] Connection lost 2022-07-03T20:44:14.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1099] Aborting connection 2022-07-03T20:44:14.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:15 smithi161 ceph-mon[91579]: pgmap v18152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:15 smithi188 ceph-mon[92425]: pgmap v18152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:17 smithi161 ceph-mon[91579]: pgmap v18153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:17 smithi188 ceph-mon[92425]: pgmap v18153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:19 smithi161 ceph-mon[91579]: pgmap v18154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:19 smithi188 ceph-mon[92425]: pgmap v18154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:20 smithi188 ceph-mon[92425]: pgmap v18155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:20 smithi161 ceph-mon[91579]: pgmap v18155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:23.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:22 smithi188 ceph-mon[92425]: pgmap v18156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:23.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:22 smithi161 ceph-mon[91579]: pgmap v18156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:23.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:24 smithi161 ceph-mon[91579]: pgmap v18157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:24 smithi188 ceph-mon[92425]: pgmap v18157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:26 smithi161 ceph-mon[91579]: pgmap v18158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:26 smithi188 ceph-mon[92425]: pgmap v18158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:28 smithi161 ceph-mon[91579]: pgmap v18159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:28 smithi188 ceph-mon[92425]: pgmap v18159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:30 smithi161 ceph-mon[91579]: pgmap v18160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:30 smithi188 ceph-mon[92425]: pgmap v18160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:32 smithi161 ceph-mon[91579]: pgmap v18161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:32 smithi188 ceph-mon[92425]: pgmap v18161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:34 smithi161 ceph-mon[91579]: pgmap v18162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:34 smithi188 ceph-mon[92425]: pgmap v18162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:36 smithi161 ceph-mon[91579]: pgmap v18163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:36 smithi188 ceph-mon[92425]: pgmap v18163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:38 smithi161 ceph-mon[91579]: pgmap v18164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:38 smithi188 ceph-mon[92425]: pgmap v18164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:40 smithi161 ceph-mon[91579]: pgmap v18165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:40 smithi188 ceph-mon[92425]: pgmap v18165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:42 smithi161 ceph-mon[91579]: pgmap v18166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:43.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:42 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:42 smithi188 ceph-mon[92425]: pgmap v18166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:42 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:44 smithi161 ceph-mon[91579]: pgmap v18167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:44 smithi188 ceph-mon[92425]: pgmap v18167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:47.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:46 smithi161 ceph-mon[91579]: pgmap v18168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:46 smithi188 ceph-mon[92425]: pgmap v18168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:47 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:47 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:49.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:48 smithi161 ceph-mon[91579]: pgmap v18169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:48 smithi188 ceph-mon[92425]: pgmap v18169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:51.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:50 smithi161 ceph-mon[91579]: pgmap v18170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:50 smithi188 ceph-mon[92425]: pgmap v18170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:52 smithi161 ceph-mon[91579]: pgmap v18171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:53.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:52 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:52 smithi188 ceph-mon[92425]: pgmap v18171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:52 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:54 smithi161 ceph-mon[91579]: pgmap v18172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:54 smithi188 ceph-mon[92425]: pgmap v18172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:44:57.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:56 smithi161 ceph-mon[91579]: pgmap v18173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:56 smithi188 ceph-mon[92425]: pgmap v18173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:44:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:44:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:44:59 smithi161 ceph-mon[91579]: pgmap v18174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:44:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:44:59 smithi188 ceph-mon[92425]: pgmap v18174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:01 smithi161 ceph-mon[91579]: pgmap v18175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:01 smithi188 ceph-mon[92425]: pgmap v18175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:03 smithi161 ceph-mon[91579]: pgmap v18176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:03.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:03 smithi188 ceph-mon[92425]: pgmap v18176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:03.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:05 smithi161 ceph-mon[91579]: pgmap v18177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:05 smithi188 ceph-mon[92425]: pgmap v18177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:05.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:07 smithi161 ceph-mon[91579]: pgmap v18178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:07 smithi188 ceph-mon[92425]: pgmap v18178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:09 smithi161 ceph-mon[91579]: pgmap v18179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:09 smithi188 ceph-mon[92425]: pgmap v18179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:11 smithi161 ceph-mon[91579]: pgmap v18180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:11 smithi188 ceph-mon[92425]: pgmap v18180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-mon[91579]: pgmap v18181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:13 smithi188 ceph-mon[92425]: pgmap v18181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:13.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:45:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:45:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:45:14.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:45:13.914+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:45:14.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:45:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:45:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:45:14.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:45:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:45:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:45:14.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:45:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:45:14.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:45:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:45:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:45:14.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:45:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:45:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:45:14.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:45:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:45:14.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:45:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:45:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:45:14.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:45:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:45:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:45:14.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:45:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:45:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:45:14.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:45:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:45:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:45:14.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:45:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:45:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:45:14.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:45:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:45:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:45:14.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:45:14.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:45:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:45:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:45:14.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:45:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:45:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:45:14.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:45:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:45:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:45:14.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:45:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1101] Local address: 172.21.15.161, port 32776 2022-07-03T20:45:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1101] Peer address: 172.21.15.101, port 22 2022-07-03T20:45:14.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1101] Connection lost 2022-07-03T20:45:14.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1101] Aborting connection 2022-07-03T20:45:14.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:45:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:45:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:45:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:45:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:45:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:15 smithi161 ceph-mon[91579]: pgmap v18182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:15 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:15 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:45:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:45:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:45:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:45:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:45:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:45:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:45:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:45:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:45:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:45:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:45:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:45:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:45:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:45:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:45:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:45:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:45:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:45:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:45:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:45:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:45:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:45:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:45:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:45:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:45:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:45:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:45:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:45:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:45:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:45:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:45:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:45:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:45:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:45:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:45:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:45:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:45:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:45:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:45:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:45:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:45:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:45:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:45:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:45:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:45:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:45:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1101] Local address: 172.21.15.161, port 32776 2022-07-03T20:45:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1101] Peer address: 172.21.15.101, port 22 2022-07-03T20:45:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1101] Connection lost 2022-07-03T20:45:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1101] Aborting connection 2022-07-03T20:45:15.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:15 smithi188 ceph-mon[92425]: pgmap v18182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:15 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:45:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:15 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:45:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:45:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:45:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:45:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:45:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:45:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:45:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:45:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:45:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:45:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:45:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:45:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:45:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:45:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:45:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:45:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:45:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:45:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:45:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:45:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:45:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:45:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:45:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:45:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:45:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:45:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:45:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:45:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:45:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:45:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:45:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:45:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:45:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:45:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:45:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:45:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:45:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:45:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:45:15.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:45:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:45:15.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:45:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:45:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:45:15.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:45:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1101] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:45:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1101] Local address: 172.21.15.161, port 32776 2022-07-03T20:45:15.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1101] Peer address: 172.21.15.101, port 22 2022-07-03T20:45:15.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1101] Connection lost 2022-07-03T20:45:15.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1101] Aborting connection 2022-07-03T20:45:15.434 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:17 smithi161 ceph-mon[91579]: pgmap v18183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:17 smithi188 ceph-mon[92425]: pgmap v18183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:18 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T20:45:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:45:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:45:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:18 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T20:45:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:45:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:45:18.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:19 smithi161 ceph-mon[91579]: pgmap v18184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:19 smithi188 ceph-mon[92425]: pgmap v18184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:21 smithi161 ceph-mon[91579]: pgmap v18185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:21 smithi188 ceph-mon[92425]: pgmap v18185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:23 smithi161 ceph-mon[91579]: pgmap v18186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:23 smithi188 ceph-mon[92425]: pgmap v18186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:23.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:25 smithi161 ceph-mon[91579]: pgmap v18187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:25 smithi188 ceph-mon[92425]: pgmap v18187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:27 smithi161 ceph-mon[91579]: pgmap v18188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:27 smithi188 ceph-mon[92425]: pgmap v18188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:29 smithi161 ceph-mon[91579]: pgmap v18189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:29 smithi188 ceph-mon[92425]: pgmap v18189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:31 smithi161 ceph-mon[91579]: pgmap v18190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:31 smithi188 ceph-mon[92425]: pgmap v18190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:33 smithi161 ceph-mon[91579]: pgmap v18191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:33.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:33 smithi188 ceph-mon[92425]: pgmap v18191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:33.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:35 smithi161 ceph-mon[91579]: pgmap v18192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:35 smithi188 ceph-mon[92425]: pgmap v18192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:37 smithi161 ceph-mon[91579]: pgmap v18193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:37 smithi188 ceph-mon[92425]: pgmap v18193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:39 smithi161 ceph-mon[91579]: pgmap v18194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:39 smithi188 ceph-mon[92425]: pgmap v18194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:41.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:41 smithi161 ceph-mon[91579]: pgmap v18195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:41 smithi188 ceph-mon[92425]: pgmap v18195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:43 smithi161 ceph-mon[91579]: pgmap v18196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:43.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:43 smithi188 ceph-mon[92425]: pgmap v18196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:43.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:45 smithi161 ceph-mon[91579]: pgmap v18197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:45 smithi188 ceph-mon[92425]: pgmap v18197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:47.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:47 smithi188 ceph-mon[92425]: pgmap v18198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:47.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:47 smithi161 ceph-mon[91579]: pgmap v18198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:49 smithi188 ceph-mon[92425]: pgmap v18199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:49 smithi161 ceph-mon[91579]: pgmap v18199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:51 smithi188 ceph-mon[92425]: pgmap v18200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:51 smithi161 ceph-mon[91579]: pgmap v18200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:53.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:53 smithi188 ceph-mon[92425]: pgmap v18201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:53.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:53 smithi161 ceph-mon[91579]: pgmap v18201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:55 smithi188 ceph-mon[92425]: pgmap v18202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:55.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:55 smithi161 ceph-mon[91579]: pgmap v18202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:45:57.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:57 smithi188 ceph-mon[92425]: pgmap v18203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:57 smithi161 ceph-mon[91579]: pgmap v18203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:45:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:45:59.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:59 smithi188 ceph-mon[92425]: pgmap v18204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:59.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:45:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:59 smithi161 ceph-mon[91579]: pgmap v18204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:45:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:45:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:45:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:45:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:46:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:01 smithi161 ceph-mon[91579]: pgmap v18205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:01 smithi188 ceph-mon[92425]: pgmap v18205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:03 smithi161 ceph-mon[91579]: pgmap v18206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:03 smithi188 ceph-mon[92425]: pgmap v18206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:05 smithi161 ceph-mon[91579]: pgmap v18207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:05 smithi188 ceph-mon[92425]: pgmap v18207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:07 smithi161 ceph-mon[91579]: pgmap v18208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:07 smithi188 ceph-mon[92425]: pgmap v18208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:09 smithi161 ceph-mon[91579]: pgmap v18209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:09 smithi188 ceph-mon[92425]: pgmap v18209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:11 smithi161 ceph-mon[91579]: pgmap v18210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:11 smithi188 ceph-mon[92425]: pgmap v18210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:13 smithi161 ceph-mon[91579]: pgmap v18211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:13.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:13 smithi188 ceph-mon[92425]: pgmap v18211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:15 smithi161 ceph-mon[91579]: pgmap v18212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:15 smithi188 ceph-mon[92425]: pgmap v18212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-mon[91579]: pgmap v18213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:17.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:46:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:46:17.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:46:17.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:46:17.098+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:46:17.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:46:17.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:46:17.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:46:17.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:46:17.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:46:17.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:46:17.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:46:17.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:46:17.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:46:17.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:46:17.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:46:17.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:46:17.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:46:17.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:46:17.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:46:17.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:46:17.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:46:17.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:46:17.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:46:17.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:46:17.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:46:17.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:46:17.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:46:17.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:46:17.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:46:17.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:46:17.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:46:17.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:46:17.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:46:17.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:46:17.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:46:17.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:46:17.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:46:17.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:46:17.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:46:17.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:46:17.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:46:17.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:46:17.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:46:17.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:46:17.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:46:17.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:46:17.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:46:17.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:46:17.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:46:17.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1103] Local address: 172.21.15.161, port 32806 2022-07-03T20:46:17.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1103] Peer address: 172.21.15.101, port 22 2022-07-03T20:46:17.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1103] Connection lost 2022-07-03T20:46:17.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1103] Aborting connection 2022-07-03T20:46:17.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:46:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:46:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:17 smithi188 ceph-mon[92425]: pgmap v18213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:46:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:46:17.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:17 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:46:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:46:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:18 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:46:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:46:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:46:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:46:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:46:18.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:46:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:46:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:46:18.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:46:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:46:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:46:18.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:46:18.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:46:18.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:46:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:46:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:46:18.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:46:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:46:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:46:18.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:46:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:46:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:46:18.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:46:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:46:18.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:46:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:46:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:46:18.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:46:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:46:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:46:18.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:46:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:46:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:46:18.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:46:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:46:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:46:18.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:46:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:46:18.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:46:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:46:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:46:18.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:46:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:46:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:46:18.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:46:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:46:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1103] Local address: 172.21.15.161, port 32806 2022-07-03T20:46:18.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1103] Peer address: 172.21.15.101, port 22 2022-07-03T20:46:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1103] Connection lost 2022-07-03T20:46:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1103] Aborting connection 2022-07-03T20:46:18.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:46:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:18 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:46:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:46:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:46:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:46:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:46:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:46:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:46:18.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:46:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:46:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:46:18.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:46:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:46:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:46:18.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:46:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:46:18.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:46:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:46:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:46:18.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:46:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:46:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:46:18.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:46:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:46:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:46:18.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:46:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:46:18.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:46:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:46:18.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:46:18.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:46:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:46:18.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:46:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:46:18.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:46:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:46:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:46:18.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:46:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:46:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:46:18.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:46:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:46:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:46:18.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:46:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:46:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:46:18.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1103] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:46:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1103] Local address: 172.21.15.161, port 32806 2022-07-03T20:46:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1103] Peer address: 172.21.15.101, port 22 2022-07-03T20:46:18.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1103] Connection lost 2022-07-03T20:46:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1103] Aborting connection 2022-07-03T20:46:18.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:19 smithi161 ceph-mon[91579]: pgmap v18214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:19 smithi188 ceph-mon[92425]: pgmap v18214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:21 smithi161 ceph-mon[91579]: pgmap v18215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:21 smithi188 ceph-mon[92425]: pgmap v18215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:23 smithi161 ceph-mon[91579]: pgmap v18216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:23 smithi188 ceph-mon[92425]: pgmap v18216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:25 smithi161 ceph-mon[91579]: pgmap v18217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:25 smithi188 ceph-mon[92425]: pgmap v18217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:27 smithi161 ceph-mon[91579]: pgmap v18218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:27 smithi188 ceph-mon[92425]: pgmap v18218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:29 smithi161 ceph-mon[91579]: pgmap v18219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:29 smithi188 ceph-mon[92425]: pgmap v18219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:31 smithi161 ceph-mon[91579]: pgmap v18220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:31 smithi188 ceph-mon[92425]: pgmap v18220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:33 smithi161 ceph-mon[91579]: pgmap v18221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:33 smithi188 ceph-mon[92425]: pgmap v18221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:35 smithi161 ceph-mon[91579]: pgmap v18222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:35 smithi188 ceph-mon[92425]: pgmap v18222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:37 smithi161 ceph-mon[91579]: pgmap v18223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:37 smithi188 ceph-mon[92425]: pgmap v18223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:39 smithi161 ceph-mon[91579]: pgmap v18224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:39 smithi188 ceph-mon[92425]: pgmap v18224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:41 smithi161 ceph-mon[91579]: pgmap v18225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:41 smithi188 ceph-mon[92425]: pgmap v18225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:43 smithi161 ceph-mon[91579]: pgmap v18226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:43 smithi188 ceph-mon[92425]: pgmap v18226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:45 smithi161 ceph-mon[91579]: pgmap v18227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:45 smithi188 ceph-mon[92425]: pgmap v18227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:47 smithi161 ceph-mon[91579]: pgmap v18228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:47 smithi188 ceph-mon[92425]: pgmap v18228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:49 smithi161 ceph-mon[91579]: pgmap v18229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:49 smithi188 ceph-mon[92425]: pgmap v18229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:51 smithi161 ceph-mon[91579]: pgmap v18230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:51 smithi188 ceph-mon[92425]: pgmap v18230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:53 smithi161 ceph-mon[91579]: pgmap v18231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:53 smithi188 ceph-mon[92425]: pgmap v18231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:55 smithi161 ceph-mon[91579]: pgmap v18232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:55 smithi188 ceph-mon[92425]: pgmap v18232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:46:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:57 smithi161 ceph-mon[91579]: pgmap v18233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:57 smithi188 ceph-mon[92425]: pgmap v18233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:46:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:46:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:59 smithi161 ceph-mon[91579]: pgmap v18234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:46:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:46:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:46:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:46:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:46:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:59 smithi188 ceph-mon[92425]: pgmap v18234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:46:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:46:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:46:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:46:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:46:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:47:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:01 smithi161 ceph-mon[91579]: pgmap v18235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:01 smithi188 ceph-mon[92425]: pgmap v18235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:03 smithi161 ceph-mon[91579]: pgmap v18236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:03 smithi188 ceph-mon[92425]: pgmap v18236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:05 smithi161 ceph-mon[91579]: pgmap v18237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:05 smithi188 ceph-mon[92425]: pgmap v18237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:07 smithi161 ceph-mon[91579]: pgmap v18238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:07 smithi188 ceph-mon[92425]: pgmap v18238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:09 smithi161 ceph-mon[91579]: pgmap v18239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:09 smithi188 ceph-mon[92425]: pgmap v18239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:11 smithi161 ceph-mon[91579]: pgmap v18240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:11 smithi188 ceph-mon[92425]: pgmap v18240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:13 smithi161 ceph-mon[91579]: pgmap v18241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:13 smithi188 ceph-mon[92425]: pgmap v18241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:15 smithi161 ceph-mon[91579]: pgmap v18242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:15 smithi188 ceph-mon[92425]: pgmap v18242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:17.596 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-mon[91579]: pgmap v18243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:17.597 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:47:17.584+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:47:17.598 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:47:17.598 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:47:17.598 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:47:17.599 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:47:17.600 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:47:17.600 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:47:17.600 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:47:17.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:47:17.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:47:17.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:47:17.602 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:47:17.602 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:47:17.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:47:17.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:47:17.603 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:47:17.604 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:47:17.604 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:47:17.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:47:17.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:47:17.605 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:47:17.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:47:17.606 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:47:17.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:47:17.607 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:47:17.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:47:17.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:47:17.608 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:47:17.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:47:17.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:47:17.609 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:47:17.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:47:17.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:47:17.610 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:47:17.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:47:17.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:47:17.611 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:47:17.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:47:17.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:47:17.612 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:47:17.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:47:17.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:47:17.613 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:47:17.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:47:17.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:47:17.614 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:47:17.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1105] Local address: 172.21.15.161, port 32834 2022-07-03T20:47:17.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1105] Peer address: 172.21.15.101, port 22 2022-07-03T20:47:17.615 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1105] Connection lost 2022-07-03T20:47:17.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1105] Aborting connection 2022-07-03T20:47:17.616 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:47:17.617 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:17 smithi188 ceph-mon[92425]: pgmap v18243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:47:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:47:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:47:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:47:18.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:47:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:47:18.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:47:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:18 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:47:18.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:19 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:47:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:47:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:47:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:47:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:47:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:47:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:47:19.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:47:19.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:47:19.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:47:19.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:47:19.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:47:19.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:47:19.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:47:19.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:47:19.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:47:19.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:47:19.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:47:19.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:47:19.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:47:19.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:47:19.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:47:19.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:47:19.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:47:19.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:47:19.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:47:19.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:47:19.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:47:19.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:47:19.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:47:19.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:47:19.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:47:19.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:47:19.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:47:19.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:47:19.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:47:19.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:47:19.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:47:19.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:47:19.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:47:19.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:47:19.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:47:19.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:47:19.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:47:19.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:47:19.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:47:19.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1105] Local address: 172.21.15.161, port 32834 2022-07-03T20:47:19.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1105] Peer address: 172.21.15.101, port 22 2022-07-03T20:47:19.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1105] Connection lost 2022-07-03T20:47:19.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1105] Aborting connection 2022-07-03T20:47:19.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:19 smithi161 ceph-mon[91579]: pgmap v18244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:47:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:47:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:47:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:47:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:47:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:47:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:47:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:47:19.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:47:19.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:47:19.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:47:19.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:47:19.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:47:19.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:47:19.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:47:19.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:47:19.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:47:19.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:47:19.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:47:19.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:47:19.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:47:19.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:47:19.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:47:19.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:47:19.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:47:19.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:47:19.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:47:19.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:47:19.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:47:19.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:47:19.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:47:19.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:47:19.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:47:19.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:47:19.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:47:19.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:47:19.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:47:19.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:47:19.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:47:19.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:47:19.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:47:19.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:47:19.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:47:19.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:47:19.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:47:19.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1105] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:47:19.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1105] Local address: 172.21.15.161, port 32834 2022-07-03T20:47:19.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1105] Peer address: 172.21.15.101, port 22 2022-07-03T20:47:19.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1105] Connection lost 2022-07-03T20:47:19.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1105] Aborting connection 2022-07-03T20:47:19.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:19 smithi188 ceph-mon[92425]: pgmap v18244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:21 smithi161 ceph-mon[91579]: pgmap v18245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:21 smithi188 ceph-mon[92425]: pgmap v18245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:23 smithi161 ceph-mon[91579]: pgmap v18246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:23 smithi188 ceph-mon[92425]: pgmap v18246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:25 smithi161 ceph-mon[91579]: pgmap v18247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:25 smithi188 ceph-mon[92425]: pgmap v18247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:27 smithi161 ceph-mon[91579]: pgmap v18248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:27 smithi188 ceph-mon[92425]: pgmap v18248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:29 smithi188 ceph-mon[92425]: pgmap v18249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:29 smithi161 ceph-mon[91579]: pgmap v18249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:31 smithi188 ceph-mon[92425]: pgmap v18250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:31 smithi161 ceph-mon[91579]: pgmap v18250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:33 smithi188 ceph-mon[92425]: pgmap v18251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:33 smithi161 ceph-mon[91579]: pgmap v18251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:35 smithi188 ceph-mon[92425]: pgmap v18252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:35 smithi161 ceph-mon[91579]: pgmap v18252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:37 smithi188 ceph-mon[92425]: pgmap v18253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:37 smithi161 ceph-mon[91579]: pgmap v18253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:39 smithi188 ceph-mon[92425]: pgmap v18254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:39 smithi161 ceph-mon[91579]: pgmap v18254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:41 smithi188 ceph-mon[92425]: pgmap v18255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:41 smithi161 ceph-mon[91579]: pgmap v18255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:43 smithi188 ceph-mon[92425]: pgmap v18256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:43 smithi161 ceph-mon[91579]: pgmap v18256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:45 smithi161 ceph-mon[91579]: pgmap v18257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:45 smithi188 ceph-mon[92425]: pgmap v18257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:47 smithi161 ceph-mon[91579]: pgmap v18258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:47 smithi188 ceph-mon[92425]: pgmap v18258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:49 smithi161 ceph-mon[91579]: pgmap v18259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:49 smithi188 ceph-mon[92425]: pgmap v18259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:50.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:51 smithi161 ceph-mon[91579]: pgmap v18260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:51 smithi188 ceph-mon[92425]: pgmap v18260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:53 smithi161 ceph-mon[91579]: pgmap v18261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:53 smithi188 ceph-mon[92425]: pgmap v18261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:55 smithi161 ceph-mon[91579]: pgmap v18262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:55 smithi188 ceph-mon[92425]: pgmap v18262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:47:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:57 smithi161 ceph-mon[91579]: pgmap v18263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:57 smithi188 ceph-mon[92425]: pgmap v18263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:47:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:47:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:59 smithi161 ceph-mon[91579]: pgmap v18264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:47:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:47:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:47:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:47:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:47:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:59 smithi188 ceph-mon[92425]: pgmap v18264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:47:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:47:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:47:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:47:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:47:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:48:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:01 smithi161 ceph-mon[91579]: pgmap v18265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:01 smithi188 ceph-mon[92425]: pgmap v18265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:03 smithi161 ceph-mon[91579]: pgmap v18266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:03 smithi188 ceph-mon[92425]: pgmap v18266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:05 smithi161 ceph-mon[91579]: pgmap v18267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:05 smithi188 ceph-mon[92425]: pgmap v18267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:07 smithi161 ceph-mon[91579]: pgmap v18268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:07 smithi188 ceph-mon[92425]: pgmap v18268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:09 smithi161 ceph-mon[91579]: pgmap v18269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:09 smithi188 ceph-mon[92425]: pgmap v18269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:11 smithi161 ceph-mon[91579]: pgmap v18270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:11 smithi188 ceph-mon[92425]: pgmap v18270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:13 smithi161 ceph-mon[91579]: pgmap v18271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:13 smithi188 ceph-mon[92425]: pgmap v18271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:15 smithi161 ceph-mon[91579]: pgmap v18272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:15 smithi188 ceph-mon[92425]: pgmap v18272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:17 smithi161 ceph-mon[91579]: pgmap v18273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:17 smithi188 ceph-mon[92425]: pgmap v18273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:18.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:48:18.079+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:48:18.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:48:18.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:48:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:48:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:48:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:48:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:48:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:48:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:48:18.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:48:18.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:48:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:48:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:48:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:48:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:48:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:48:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:48:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:48:18.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:48:18.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:48:18.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:48:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:48:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:48:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:48:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:48:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:48:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:48:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:48:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:48:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:48:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:48:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:48:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:48:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:48:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:48:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:48:18.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:48:18.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:48:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:48:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:48:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:48:18.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:48:18.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:48:18.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:48:18.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:48:18.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:48:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1107] Local address: 172.21.15.161, port 32862 2022-07-03T20:48:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1107] Peer address: 172.21.15.101, port 22 2022-07-03T20:48:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1107] Connection lost 2022-07-03T20:48:18.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1107] Aborting connection 2022-07-03T20:48:18.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:48:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:48:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:48:18.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:48:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:48:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:48:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:48:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:19 smithi161 ceph-mon[91579]: pgmap v18274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:19 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:48:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:19 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:48:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:48:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:48:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:48:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:48:19.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:48:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:48:19.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:48:19.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:48:19.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:48:19.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:48:19.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:48:19.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:48:19.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:48:19.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:48:19.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:48:19.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:48:19.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:48:19.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:48:19.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:48:19.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:48:19.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:48:19.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:48:19.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:48:19.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:48:19.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:48:19.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:48:19.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:48:19.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:48:19.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:48:19.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:48:19.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:48:19.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:48:19.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:48:19.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:48:19.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:48:19.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:48:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:48:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:48:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:48:19.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:48:19.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:48:19.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:48:19.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:48:19.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:48:19.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:48:19.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1107] Local address: 172.21.15.161, port 32862 2022-07-03T20:48:19.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1107] Peer address: 172.21.15.101, port 22 2022-07-03T20:48:19.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1107] Connection lost 2022-07-03T20:48:19.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1107] Aborting connection 2022-07-03T20:48:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:19 smithi188 ceph-mon[92425]: pgmap v18274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:19 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:48:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:48:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:48:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:48:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:48:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:48:19.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:48:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:48:19.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:48:19.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:48:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:48:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:48:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:48:19.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:48:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:48:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:48:19.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:48:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:48:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:48:19.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:48:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:48:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:48:19.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:48:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:48:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:48:19.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:48:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:48:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:48:19.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:48:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:48:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:48:19.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:48:19.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:48:19.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:48:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:48:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:48:19.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:48:19.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:48:19.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:48:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:48:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:48:19.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:48:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:48:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:48:19.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:48:19.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:48:19.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1107] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:48:19.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1107] Local address: 172.21.15.161, port 32862 2022-07-03T20:48:19.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1107] Peer address: 172.21.15.101, port 22 2022-07-03T20:48:19.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1107] Connection lost 2022-07-03T20:48:19.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1107] Aborting connection 2022-07-03T20:48:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:21 smithi161 ceph-mon[91579]: pgmap v18275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:21 smithi188 ceph-mon[92425]: pgmap v18275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:23 smithi161 ceph-mon[91579]: pgmap v18276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:23 smithi188 ceph-mon[92425]: pgmap v18276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:25 smithi161 ceph-mon[91579]: pgmap v18277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:25 smithi188 ceph-mon[92425]: pgmap v18277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:27 smithi161 ceph-mon[91579]: pgmap v18278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:27 smithi188 ceph-mon[92425]: pgmap v18278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:29 smithi161 ceph-mon[91579]: pgmap v18279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:29 smithi188 ceph-mon[92425]: pgmap v18279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:31 smithi161 ceph-mon[91579]: pgmap v18280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:31 smithi188 ceph-mon[92425]: pgmap v18280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:33 smithi161 ceph-mon[91579]: pgmap v18281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:33 smithi188 ceph-mon[92425]: pgmap v18281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:35 smithi161 ceph-mon[91579]: pgmap v18282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:35 smithi188 ceph-mon[92425]: pgmap v18282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:37 smithi161 ceph-mon[91579]: pgmap v18283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:37 smithi188 ceph-mon[92425]: pgmap v18283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:39 smithi161 ceph-mon[91579]: pgmap v18284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:39 smithi188 ceph-mon[92425]: pgmap v18284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:41 smithi161 ceph-mon[91579]: pgmap v18285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:41 smithi188 ceph-mon[92425]: pgmap v18285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:43 smithi161 ceph-mon[91579]: pgmap v18286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:43 smithi188 ceph-mon[92425]: pgmap v18286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:45 smithi161 ceph-mon[91579]: pgmap v18287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:45 smithi188 ceph-mon[92425]: pgmap v18287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:47 smithi161 ceph-mon[91579]: pgmap v18288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:47 smithi188 ceph-mon[92425]: pgmap v18288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:49 smithi161 ceph-mon[91579]: pgmap v18289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:49 smithi188 ceph-mon[92425]: pgmap v18289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:51 smithi161 ceph-mon[91579]: pgmap v18290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:51 smithi188 ceph-mon[92425]: pgmap v18290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:53 smithi161 ceph-mon[91579]: pgmap v18291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:53 smithi188 ceph-mon[92425]: pgmap v18291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:55 smithi161 ceph-mon[91579]: pgmap v18292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:55 smithi188 ceph-mon[92425]: pgmap v18292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:48:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:57 smithi161 ceph-mon[91579]: pgmap v18293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:57 smithi188 ceph-mon[92425]: pgmap v18293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:48:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:48:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:59 smithi161 ceph-mon[91579]: pgmap v18294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:48:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:48:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:48:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:48:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:48:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:59 smithi188 ceph-mon[92425]: pgmap v18294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:48:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:48:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:48:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:48:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:48:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:49:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:01 smithi161 ceph-mon[91579]: pgmap v18295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:01 smithi188 ceph-mon[92425]: pgmap v18295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:03 smithi161 ceph-mon[91579]: pgmap v18296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:03 smithi188 ceph-mon[92425]: pgmap v18296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:04 smithi188 ceph-mon[92425]: pgmap v18297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:04 smithi161 ceph-mon[91579]: pgmap v18297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:06 smithi161 ceph-mon[91579]: pgmap v18298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:06 smithi188 ceph-mon[92425]: pgmap v18298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:08 smithi161 ceph-mon[91579]: pgmap v18299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:08 smithi188 ceph-mon[92425]: pgmap v18299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:10 smithi161 ceph-mon[91579]: pgmap v18300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:10 smithi188 ceph-mon[92425]: pgmap v18300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:13.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:12 smithi161 ceph-mon[91579]: pgmap v18301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:13.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:12 smithi188 ceph-mon[92425]: pgmap v18301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:14 smithi161 ceph-mon[91579]: pgmap v18302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:14 smithi188 ceph-mon[92425]: pgmap v18302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:17.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:16 smithi161 ceph-mon[91579]: pgmap v18303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:17.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:16 smithi188 ceph-mon[92425]: pgmap v18303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:18.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:49:18.583+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:49:18.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:49:18.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:49:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:49:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:49:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:49:18.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:49:18.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:49:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:49:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:49:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:49:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:49:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:49:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:49:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:49:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:49:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:49:18.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:49:18.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:49:18.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:49:18.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:49:18.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:49:18.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:49:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:49:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:49:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:49:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:49:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:49:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:49:18.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:49:18.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:49:18.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:49:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:49:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:49:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:49:18.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:49:18.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:49:18.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:49:18.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:49:18.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:49:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:49:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:49:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:49:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:49:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:49:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:49:18.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1109] Local address: 172.21.15.161, port 32890 2022-07-03T20:49:18.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1109] Peer address: 172.21.15.101, port 22 2022-07-03T20:49:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1109] Connection lost 2022-07-03T20:49:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1109] Aborting connection 2022-07-03T20:49:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:49:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: pgmap v18304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:49:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:49:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:49:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:49:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:49:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:49:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:49:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:49:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:49:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:49:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:49:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:49:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:49:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:49:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:49:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:49:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:49:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:49:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:49:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:49:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:49:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:49:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:49:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:49:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:49:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:49:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:49:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:49:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:49:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:49:19.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:49:19.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:49:19.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:49:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:49:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:49:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:49:19.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:49:19.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:49:19.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:49:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:49:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:49:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:49:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:49:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:49:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:49:19.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:49:19.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:49:19.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:49:19.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:49:19.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:49:19.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1109] Local address: 172.21.15.161, port 32890 2022-07-03T20:49:19.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1109] Peer address: 172.21.15.101, port 22 2022-07-03T20:49:19.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1109] Connection lost 2022-07-03T20:49:19.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1109] Aborting connection 2022-07-03T20:49:19.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:19.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:19.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: pgmap v18304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:19.399 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:19.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:49:19.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:49:19.400 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:49:19.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:49:19.401 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:49:19.401 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:49:19.402 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:49:19.402 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:49:19.402 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:49:19.403 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:49:19.403 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:49:19.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:49:19.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:49:19.404 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:49:19.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:49:19.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:49:19.405 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:49:19.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:49:19.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:49:19.406 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:49:19.407 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:49:19.407 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:49:19.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:49:19.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:49:19.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:49:19.408 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:49:19.409 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:49:19.409 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:49:19.409 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:49:19.410 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:49:19.410 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:49:19.410 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:49:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:49:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:49:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:49:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:49:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:49:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:49:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:49:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:49:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:49:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:49:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:49:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:49:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:49:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:49:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:49:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:49:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:49:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1109] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:49:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1109] Local address: 172.21.15.161, port 32890 2022-07-03T20:49:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1109] Peer address: 172.21.15.101, port 22 2022-07-03T20:49:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1109] Connection lost 2022-07-03T20:49:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1109] Aborting connection 2022-07-03T20:49:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:20.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:20.380 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:20.381 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:21 smithi161 ceph-mon[91579]: pgmap v18305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:21 smithi188 ceph-mon[92425]: pgmap v18305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:23 smithi161 ceph-mon[91579]: pgmap v18306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:23 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T20:49:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:23.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:23 smithi188 ceph-mon[92425]: pgmap v18306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:23 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T20:49:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:49:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:25 smithi161 ceph-mon[91579]: pgmap v18307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:25 smithi188 ceph-mon[92425]: pgmap v18307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:27 smithi161 ceph-mon[91579]: pgmap v18308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:27 smithi188 ceph-mon[92425]: pgmap v18308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:29 smithi161 ceph-mon[91579]: pgmap v18309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:29 smithi188 ceph-mon[92425]: pgmap v18309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:31 smithi161 ceph-mon[91579]: pgmap v18310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:31 smithi188 ceph-mon[92425]: pgmap v18310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:33 smithi161 ceph-mon[91579]: pgmap v18311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:33 smithi188 ceph-mon[92425]: pgmap v18311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:35 smithi161 ceph-mon[91579]: pgmap v18312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:35 smithi188 ceph-mon[92425]: pgmap v18312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:37 smithi161 ceph-mon[91579]: pgmap v18313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:37 smithi188 ceph-mon[92425]: pgmap v18313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:39 smithi161 ceph-mon[91579]: pgmap v18314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:39 smithi188 ceph-mon[92425]: pgmap v18314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:41 smithi161 ceph-mon[91579]: pgmap v18315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:41 smithi188 ceph-mon[92425]: pgmap v18315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:43 smithi161 ceph-mon[91579]: pgmap v18316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:43 smithi188 ceph-mon[92425]: pgmap v18316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:45 smithi161 ceph-mon[91579]: pgmap v18317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:45 smithi188 ceph-mon[92425]: pgmap v18317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:47 smithi161 ceph-mon[91579]: pgmap v18318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:47 smithi188 ceph-mon[92425]: pgmap v18318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:49 smithi161 ceph-mon[91579]: pgmap v18319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:49 smithi188 ceph-mon[92425]: pgmap v18319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:51 smithi161 ceph-mon[91579]: pgmap v18320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:51 smithi188 ceph-mon[92425]: pgmap v18320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:53 smithi161 ceph-mon[91579]: pgmap v18321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:53 smithi188 ceph-mon[92425]: pgmap v18321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:55 smithi161 ceph-mon[91579]: pgmap v18322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:55 smithi188 ceph-mon[92425]: pgmap v18322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:49:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:57 smithi161 ceph-mon[91579]: pgmap v18323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:57 smithi188 ceph-mon[92425]: pgmap v18323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:49:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:49:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:59 smithi161 ceph-mon[91579]: pgmap v18324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:49:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:49:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:49:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:49:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:49:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:59 smithi188 ceph-mon[92425]: pgmap v18324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:49:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:49:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:49:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:49:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:49:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:50:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:50:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T20:50:01.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:01 smithi161 ceph-mon[91579]: pgmap v18325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:01 smithi188 ceph-mon[92425]: pgmap v18325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:03 smithi161 ceph-mon[91579]: pgmap v18326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:03 smithi188 ceph-mon[92425]: pgmap v18326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:05 smithi161 ceph-mon[91579]: pgmap v18327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:05 smithi188 ceph-mon[92425]: pgmap v18327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:07 smithi161 ceph-mon[91579]: pgmap v18328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:07 smithi188 ceph-mon[92425]: pgmap v18328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:09 smithi161 ceph-mon[91579]: pgmap v18329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:09 smithi188 ceph-mon[92425]: pgmap v18329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:10.670 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:11 smithi161 ceph-mon[91579]: pgmap v18330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:11 smithi188 ceph-mon[92425]: pgmap v18330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:13 smithi161 ceph-mon[91579]: pgmap v18331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:13 smithi188 ceph-mon[92425]: pgmap v18331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:15 smithi161 ceph-mon[91579]: pgmap v18332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:15 smithi188 ceph-mon[92425]: pgmap v18332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:17 smithi161 ceph-mon[91579]: pgmap v18333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:17 smithi188 ceph-mon[92425]: pgmap v18333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:19 smithi188 ceph-mon[92425]: pgmap v18334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:19 smithi161 ceph-mon[91579]: pgmap v18334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:21 smithi188 ceph-mon[92425]: pgmap v18335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:21 smithi161 ceph-mon[91579]: pgmap v18335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:22.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:50:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:50:22.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:50:22.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:50:22.295+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:50:22.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:50:22.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:50:22.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:50:22.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:50:22.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:50:22.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:50:22.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:50:22.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:50:22.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:50:22.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:50:22.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:50:22.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:50:22.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:50:22.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:50:22.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:50:22.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:50:22.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:50:22.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:50:22.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:50:22.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:50:22.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:50:22.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:50:22.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:50:22.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:50:22.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:50:22.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:50:22.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:50:22.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:50:22.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:50:22.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:50:22.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:50:22.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:50:22.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:50:22.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:50:22.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:50:22.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:50:22.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:50:22.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:50:22.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:50:22.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:50:22.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:50:22.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:50:22.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:50:22.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:50:22.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:50:22.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1111] Local address: 172.21.15.161, port 32920 2022-07-03T20:50:22.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1111] Peer address: 172.21.15.101, port 22 2022-07-03T20:50:22.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1111] Connection lost 2022-07-03T20:50:22.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1111] Aborting connection 2022-07-03T20:50:22.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:50:22 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:50:22.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:50:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:50:22.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:22 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:50:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: pgmap v18336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:50:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:50:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:50:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:50:23.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:50:23.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:50:23.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:50:23.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:50:23.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:50:23.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:50:23.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:50:23.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:50:23.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:50:23.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:50:23.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:50:23.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:50:23.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:50:23.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:50:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:50:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:50:23.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:50:23.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:50:23.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:50:23.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:50:23.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:50:23.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:50:23.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:50:23.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:50:23.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1111] Local address: 172.21.15.161, port 32920 2022-07-03T20:50:23.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1111] Peer address: 172.21.15.101, port 22 2022-07-03T20:50:23.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1111] Connection lost 2022-07-03T20:50:23.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1111] Aborting connection 2022-07-03T20:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:23 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: pgmap v18336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:50:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:50:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:50:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:50:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:50:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:50:23.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:50:23.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:50:23.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:50:23.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:50:23.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:50:23.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:50:23.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:50:23.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:50:23.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:50:23.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:50:23.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:50:23.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:50:23.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:50:23.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:50:23.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:50:23.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:50:23.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:50:23.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:50:23.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:50:23.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:50:23.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:50:23.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:50:23.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:50:23.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:50:23.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:50:23.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:50:23.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:50:23.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:50:23.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:50:23.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:50:23.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:50:23.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:50:23.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:50:23.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:50:23.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:50:23.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:50:23.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:50:23.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:50:23.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:50:23.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:50:23.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1111] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:50:23.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1111] Local address: 172.21.15.161, port 32920 2022-07-03T20:50:23.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1111] Peer address: 172.21.15.101, port 22 2022-07-03T20:50:23.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1111] Connection lost 2022-07-03T20:50:23.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1111] Aborting connection 2022-07-03T20:50:23.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:23.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:23.904 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:23 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:50:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:25 smithi188 ceph-mon[92425]: pgmap v18337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:25 smithi161 ceph-mon[91579]: pgmap v18337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:27 smithi188 ceph-mon[92425]: pgmap v18338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:27 smithi161 ceph-mon[91579]: pgmap v18338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:29 smithi188 ceph-mon[92425]: pgmap v18339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:29 smithi161 ceph-mon[91579]: pgmap v18339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:31 smithi188 ceph-mon[92425]: pgmap v18340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:31 smithi161 ceph-mon[91579]: pgmap v18340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:33 smithi188 ceph-mon[92425]: pgmap v18341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:33 smithi161 ceph-mon[91579]: pgmap v18341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:35 smithi161 ceph-mon[91579]: pgmap v18342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:35 smithi188 ceph-mon[92425]: pgmap v18342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:37 smithi161 ceph-mon[91579]: pgmap v18343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:37 smithi188 ceph-mon[92425]: pgmap v18343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:39 smithi161 ceph-mon[91579]: pgmap v18344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:39 smithi188 ceph-mon[92425]: pgmap v18344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:41 smithi161 ceph-mon[91579]: pgmap v18345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:41 smithi188 ceph-mon[92425]: pgmap v18345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:43 smithi161 ceph-mon[91579]: pgmap v18346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:43 smithi188 ceph-mon[92425]: pgmap v18346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:45 smithi161 ceph-mon[91579]: pgmap v18347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:45 smithi188 ceph-mon[92425]: pgmap v18347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:47 smithi161 ceph-mon[91579]: pgmap v18348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:47 smithi188 ceph-mon[92425]: pgmap v18348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:49 smithi161 ceph-mon[91579]: pgmap v18349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:49 smithi188 ceph-mon[92425]: pgmap v18349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:51 smithi161 ceph-mon[91579]: pgmap v18350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:51 smithi188 ceph-mon[92425]: pgmap v18350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:53 smithi161 ceph-mon[91579]: pgmap v18351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:53 smithi188 ceph-mon[92425]: pgmap v18351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:55 smithi161 ceph-mon[91579]: pgmap v18352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:55 smithi188 ceph-mon[92425]: pgmap v18352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:50:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:57 smithi161 ceph-mon[91579]: pgmap v18353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:57 smithi188 ceph-mon[92425]: pgmap v18353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:50:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:50:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:59 smithi161 ceph-mon[91579]: pgmap v18354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:50:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:50:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:50:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:50:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:50:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:59 smithi188 ceph-mon[92425]: pgmap v18354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:50:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:50:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:50:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:51:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:01 smithi161 ceph-mon[91579]: pgmap v18355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:01 smithi188 ceph-mon[92425]: pgmap v18355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:03 smithi161 ceph-mon[91579]: pgmap v18356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:03 smithi188 ceph-mon[92425]: pgmap v18356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:05 smithi161 ceph-mon[91579]: pgmap v18357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:05 smithi188 ceph-mon[92425]: pgmap v18357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:07 smithi161 ceph-mon[91579]: pgmap v18358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:07 smithi188 ceph-mon[92425]: pgmap v18358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:09 smithi161 ceph-mon[91579]: pgmap v18359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:09 smithi188 ceph-mon[92425]: pgmap v18359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:11 smithi161 ceph-mon[91579]: pgmap v18360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:11 smithi188 ceph-mon[92425]: pgmap v18360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:13 smithi161 ceph-mon[91579]: pgmap v18361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:13 smithi188 ceph-mon[92425]: pgmap v18361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:15 smithi161 ceph-mon[91579]: pgmap v18362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:15 smithi188 ceph-mon[92425]: pgmap v18362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:17 smithi161 ceph-mon[91579]: pgmap v18363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:17 smithi188 ceph-mon[92425]: pgmap v18363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:18.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:19 smithi161 ceph-mon[91579]: pgmap v18364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:19 smithi188 ceph-mon[92425]: pgmap v18364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:21 smithi161 ceph-mon[91579]: pgmap v18365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:21 smithi188 ceph-mon[92425]: pgmap v18365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:23.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:51:23.213+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:51:23.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:51:23.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:51:23.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:51:23.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:51:23.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:51:23.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:51:23.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:51:23.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:51:23.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:51:23.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:51:23.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:51:23.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:51:23.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:51:23.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:51:23.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:51:23.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:51:23.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:51:23.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:51:23.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:51:23.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:51:23.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:51:23.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:51:23.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:51:23.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:51:23.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:51:23.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:51:23.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:51:23.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:51:23.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:51:23.540 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:51:23.541 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:51:23.541 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:51:23.542 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:51:23.542 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:51:23.542 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:51:23.543 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:51:23.543 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:51:23.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:51:23.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:51:23.544 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:51:23.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:51:23.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:51:23.545 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:51:23.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:51:23.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:51:23.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1113] Local address: 172.21.15.161, port 32948 2022-07-03T20:51:23.546 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1113] Peer address: 172.21.15.101, port 22 2022-07-03T20:51:23.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1113] Connection lost 2022-07-03T20:51:23.547 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1113] Aborting connection 2022-07-03T20:51:23.548 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:51:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-mon[91579]: pgmap v18366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:51:23.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:51:23.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:51:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:23 smithi188 ceph-mon[92425]: pgmap v18366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:51:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:51:23.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:51:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:24 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:51:24.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:24 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:51:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:51:24.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:51:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:51:24.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:51:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:51:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:51:24.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:51:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:51:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:51:24.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:51:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:51:24.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:51:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:51:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:51:24.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:51:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:51:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:51:24.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:51:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:51:24.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:51:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:51:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:51:24.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:51:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:51:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:51:24.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:51:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:51:24.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:51:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:51:24.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:51:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:51:24.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:51:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:51:24.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:51:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:51:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:51:24.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:51:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:51:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:51:24.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:51:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:51:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:51:24.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:51:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:51:24.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:51:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1113] Local address: 172.21.15.161, port 32948 2022-07-03T20:51:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1113] Peer address: 172.21.15.101, port 22 2022-07-03T20:51:24.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1113] Connection lost 2022-07-03T20:51:24.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1113] Aborting connection 2022-07-03T20:51:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:24 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:51:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:24 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:51:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:51:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:51:24.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:51:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:51:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:51:24.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:51:24.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:51:24.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:51:24.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:51:24.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:51:24.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:51:24.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:51:24.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:51:24.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:51:24.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:51:24.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:51:24.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:51:24.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:51:24.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:51:24.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:51:24.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:51:24.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:51:24.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:51:24.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:51:24.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:51:24.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:51:24.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:51:24.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:51:24.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:51:24.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:51:24.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:51:24.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:51:24.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:51:24.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:51:24.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:51:24.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:51:24.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:51:24.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:51:24.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:51:24.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:51:24.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:51:24.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:51:24.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:51:24.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:51:24.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1113] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:51:24.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1113] Local address: 172.21.15.161, port 32948 2022-07-03T20:51:24.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1113] Peer address: 172.21.15.101, port 22 2022-07-03T20:51:24.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1113] Connection lost 2022-07-03T20:51:24.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1113] Aborting connection 2022-07-03T20:51:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:25 smithi161 ceph-mon[91579]: pgmap v18367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:25 smithi188 ceph-mon[92425]: pgmap v18367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:27 smithi161 ceph-mon[91579]: pgmap v18368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:27 smithi188 ceph-mon[92425]: pgmap v18368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:29 smithi161 ceph-mon[91579]: pgmap v18369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:29 smithi188 ceph-mon[92425]: pgmap v18369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:31 smithi161 ceph-mon[91579]: pgmap v18370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:31 smithi188 ceph-mon[92425]: pgmap v18370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:33 smithi161 ceph-mon[91579]: pgmap v18371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:33 smithi188 ceph-mon[92425]: pgmap v18371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:35 smithi161 ceph-mon[91579]: pgmap v18372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:35 smithi188 ceph-mon[92425]: pgmap v18372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:37 smithi161 ceph-mon[91579]: pgmap v18373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:37 smithi188 ceph-mon[92425]: pgmap v18373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:39 smithi161 ceph-mon[91579]: pgmap v18374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:39 smithi188 ceph-mon[92425]: pgmap v18374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:41 smithi161 ceph-mon[91579]: pgmap v18375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:41 smithi188 ceph-mon[92425]: pgmap v18375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:43 smithi161 ceph-mon[91579]: pgmap v18376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:43 smithi188 ceph-mon[92425]: pgmap v18376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:45 smithi161 ceph-mon[91579]: pgmap v18377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:45 smithi188 ceph-mon[92425]: pgmap v18377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:47 smithi161 ceph-mon[91579]: pgmap v18378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:47 smithi188 ceph-mon[92425]: pgmap v18378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:51:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:49 smithi161 ceph-mon[91579]: pgmap v18379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:49 smithi188 ceph-mon[92425]: pgmap v18379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:51 smithi161 ceph-mon[91579]: pgmap v18380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:51 smithi188 ceph-mon[92425]: pgmap v18380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:53 smithi161 ceph-mon[91579]: pgmap v18381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:53 smithi188 ceph-mon[92425]: pgmap v18381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:55 smithi161 ceph-mon[91579]: pgmap v18382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:55 smithi188 ceph-mon[92425]: pgmap v18382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:51:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:57 smithi161 ceph-mon[91579]: pgmap v18383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:57 smithi188 ceph-mon[92425]: pgmap v18383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T20:51:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:51:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:59 smithi161 ceph-mon[91579]: pgmap v18384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:51:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:51:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:51:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:51:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:51:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:59 smithi188 ceph-mon[92425]: pgmap v18384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:51:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:51:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:51:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:51:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:51:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:52:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:01 smithi161 ceph-mon[91579]: pgmap v18385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:01 smithi188 ceph-mon[92425]: pgmap v18385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:03 smithi161 ceph-mon[91579]: pgmap v18386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:03 smithi188 ceph-mon[92425]: pgmap v18386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:05 smithi161 ceph-mon[91579]: pgmap v18387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:05 smithi188 ceph-mon[92425]: pgmap v18387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:07 smithi161 ceph-mon[91579]: pgmap v18388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:07 smithi188 ceph-mon[92425]: pgmap v18388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:09 smithi188 ceph-mon[92425]: pgmap v18389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:09 smithi161 ceph-mon[91579]: pgmap v18389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:11 smithi188 ceph-mon[92425]: pgmap v18390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:11 smithi161 ceph-mon[91579]: pgmap v18390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:12.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:12 smithi188 ceph-mon[92425]: pgmap v18391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:12 smithi161 ceph-mon[91579]: pgmap v18391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:14 smithi188 ceph-mon[92425]: pgmap v18392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:15.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:14 smithi161 ceph-mon[91579]: pgmap v18392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:16.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:16 smithi188 ceph-mon[92425]: pgmap v18393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:16 smithi161 ceph-mon[91579]: pgmap v18393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:19 smithi161 ceph-mon[91579]: pgmap v18394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:19 smithi188 ceph-mon[92425]: pgmap v18394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:21 smithi161 ceph-mon[91579]: pgmap v18395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:21 smithi188 ceph-mon[92425]: pgmap v18395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-mon[91579]: pgmap v18396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:23 smithi188 ceph-mon[92425]: pgmap v18396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:24.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:52:23.720+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:52:24.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:52:24.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:52:24.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:52:24.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:52:24.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:52:24.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:52:24.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:52:24.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:52:24.054 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:52:24.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:52:24.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:52:24.055 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:52:24.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:52:24.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:52:24.056 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:52:24.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:52:24.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:52:24.057 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:52:24.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:52:24.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:52:24.058 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:52:24.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:52:24.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:52:24.059 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:52:24.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:52:24.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:52:24.060 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:52:24.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:52:24.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:52:24.061 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:52:24.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:52:24.062 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:52:24.063 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:52:24.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:52:24.064 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:52:24.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:52:24.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:52:24.065 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:52:24.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:52:24.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:52:24.066 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:52:24.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:52:24.067 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:52:24.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:52:24.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:52:24.068 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1115] Local address: 172.21.15.161, port 32976 2022-07-03T20:52:24.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1115] Peer address: 172.21.15.101, port 22 2022-07-03T20:52:24.069 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1115] Connection lost 2022-07-03T20:52:24.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1115] Aborting connection 2022-07-03T20:52:24.070 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:52:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:52:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:52:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:52:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:52:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:52:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:52:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:52:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:52:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:52:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:52:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:52:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:52:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:52:25.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:52:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:52:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:52:25.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:52:25.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:52:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:52:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:52:25.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:52:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:52:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:52:25.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:52:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:52:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:52:25.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:52:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:52:25.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:52:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:52:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:52:25.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:52:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:52:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:52:25.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:52:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:52:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:52:25.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:52:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:52:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:52:25.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:52:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:52:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:52:25.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:52:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:52:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:52:25.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:52:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:52:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:52:25.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:52:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:52:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:52:25.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:52:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:52:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1115] Local address: 172.21.15.161, port 32976 2022-07-03T20:52:25.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1115] Peer address: 172.21.15.101, port 22 2022-07-03T20:52:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1115] Connection lost 2022-07-03T20:52:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1115] Aborting connection 2022-07-03T20:52:25.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:25 smithi161 ceph-mon[91579]: pgmap v18397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:25.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:52:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:52:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1115] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1115] Local address: 172.21.15.161, port 32976 2022-07-03T20:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1115] Peer address: 172.21.15.101, port 22 2022-07-03T20:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1115] Connection lost 2022-07-03T20:52:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1115] Aborting connection 2022-07-03T20:52:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:25 smithi188 ceph-mon[92425]: pgmap v18397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: pgmap v18398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:27 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: pgmap v18398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:52:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:27.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:27 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T20:52:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:28 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 55704k 2022-07-03T20:52:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:28 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 57041330: error parsing value: Value '57041330' is below minimum 939524096 2022-07-03T20:52:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:28 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 55704k 2022-07-03T20:52:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:28 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 57041330: error parsing value: Value '57041330' is below minimum 939524096 2022-07-03T20:52:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:29 smithi161 ceph-mon[91579]: pgmap v18399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:29 smithi188 ceph-mon[92425]: pgmap v18399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:31 smithi188 ceph-mon[92425]: pgmap v18400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:31 smithi161 ceph-mon[91579]: pgmap v18400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:33 smithi188 ceph-mon[92425]: pgmap v18401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:33 smithi161 ceph-mon[91579]: pgmap v18401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:35 smithi188 ceph-mon[92425]: pgmap v18402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:35 smithi161 ceph-mon[91579]: pgmap v18402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:37 smithi188 ceph-mon[92425]: pgmap v18403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:37 smithi161 ceph-mon[91579]: pgmap v18403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:39 smithi188 ceph-mon[92425]: pgmap v18404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:39 smithi161 ceph-mon[91579]: pgmap v18404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:41 smithi188 ceph-mon[92425]: pgmap v18405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:41 smithi161 ceph-mon[91579]: pgmap v18405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:43 smithi188 ceph-mon[92425]: pgmap v18406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:43 smithi161 ceph-mon[91579]: pgmap v18406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:45 smithi188 ceph-mon[92425]: pgmap v18407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:45 smithi161 ceph-mon[91579]: pgmap v18407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:47 smithi161 ceph-mon[91579]: pgmap v18408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:47 smithi188 ceph-mon[92425]: pgmap v18408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:49 smithi161 ceph-mon[91579]: pgmap v18409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:49 smithi188 ceph-mon[92425]: pgmap v18409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:51 smithi161 ceph-mon[91579]: pgmap v18410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:51 smithi188 ceph-mon[92425]: pgmap v18410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:53 smithi161 ceph-mon[91579]: pgmap v18411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:53 smithi188 ceph-mon[92425]: pgmap v18411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:55 smithi161 ceph-mon[91579]: pgmap v18412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:55 smithi188 ceph-mon[92425]: pgmap v18412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:52:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:57 smithi161 ceph-mon[91579]: pgmap v18413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:57 smithi188 ceph-mon[92425]: pgmap v18413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:52:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:52:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:59 smithi161 ceph-mon[91579]: pgmap v18414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:52:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:52:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:52:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:52:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:59 smithi188 ceph-mon[92425]: pgmap v18414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:52:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:52:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:52:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:52:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:52:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:53:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:01 smithi161 ceph-mon[91579]: pgmap v18415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:01 smithi188 ceph-mon[92425]: pgmap v18415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:03 smithi161 ceph-mon[91579]: pgmap v18416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:03 smithi188 ceph-mon[92425]: pgmap v18416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:05 smithi161 ceph-mon[91579]: pgmap v18417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:05 smithi188 ceph-mon[92425]: pgmap v18417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:07 smithi161 ceph-mon[91579]: pgmap v18418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:07 smithi188 ceph-mon[92425]: pgmap v18418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:09 smithi161 ceph-mon[91579]: pgmap v18419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:09 smithi188 ceph-mon[92425]: pgmap v18419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:11 smithi161 ceph-mon[91579]: pgmap v18420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:11 smithi188 ceph-mon[92425]: pgmap v18420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:13 smithi161 ceph-mon[91579]: pgmap v18421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:13 smithi188 ceph-mon[92425]: pgmap v18421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:15 smithi161 ceph-mon[91579]: pgmap v18422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:15 smithi188 ceph-mon[92425]: pgmap v18422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:17 smithi161 ceph-mon[91579]: pgmap v18423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:17 smithi188 ceph-mon[92425]: pgmap v18423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:19 smithi161 ceph-mon[91579]: pgmap v18424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:19 smithi188 ceph-mon[92425]: pgmap v18424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:21 smithi161 ceph-mon[91579]: pgmap v18425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:21 smithi188 ceph-mon[92425]: pgmap v18425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:23 smithi161 ceph-mon[91579]: pgmap v18426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:23 smithi188 ceph-mon[92425]: pgmap v18426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:25 smithi161 ceph-mon[91579]: pgmap v18427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:25 smithi188 ceph-mon[92425]: pgmap v18427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:27.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:53:26.965+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:53:27.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:53:27.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:53:27.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:53:27.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:53:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:53:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:53:27.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:53:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:53:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:53:27.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:53:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:53:27.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:53:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:53:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:53:27.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:53:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:53:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:53:27.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:53:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:53:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:53:27.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:53:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:53:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:53:27.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:53:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:53:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:53:27.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:53:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:53:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:53:27.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:53:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:53:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:53:27.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:53:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:53:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:53:27.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:53:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:53:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:53:27.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:53:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:53:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:53:27.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:53:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:53:27.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:53:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:53:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1117] Local address: 172.21.15.161, port 33006 2022-07-03T20:53:27.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1117] Peer address: 172.21.15.101, port 22 2022-07-03T20:53:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1117] Connection lost 2022-07-03T20:53:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1117] Aborting connection 2022-07-03T20:53:27.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:53:26 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:53:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:27 smithi161 ceph-mon[91579]: pgmap v18428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:53:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:53:27.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:27 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:53:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:27 smithi188 ceph-mon[92425]: pgmap v18428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:53:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:53:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:27 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:53:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:53:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:53:28.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:53:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:53:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:53:28.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:53:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:53:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:53:28.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:53:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:53:28.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:53:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:53:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:53:28.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:53:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:53:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:53:28.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:53:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:53:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:53:28.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:53:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:53:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:53:28.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:53:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:53:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:53:28.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:53:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:53:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:53:28.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:53:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:53:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:53:28.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:53:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:53:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:53:28.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:53:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:53:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:53:28.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:53:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:53:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:53:28.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:53:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:53:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:53:28.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:53:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:53:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:53:28.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:53:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1117] Local address: 172.21.15.161, port 33006 2022-07-03T20:53:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1117] Peer address: 172.21.15.101, port 22 2022-07-03T20:53:28.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1117] Connection lost 2022-07-03T20:53:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1117] Aborting connection 2022-07-03T20:53:28.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:53:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:53:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:53:28.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:53:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:53:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:53:28.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:53:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:53:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:53:28.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:53:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:53:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:53:28.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:53:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:53:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:53:28.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:53:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:53:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:53:28.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:53:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:53:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:53:28.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:53:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:53:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:53:28.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:53:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:53:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:53:28.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:53:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:53:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:53:28.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:53:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:53:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:53:28.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:53:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:53:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:53:28.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:53:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:53:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:53:28.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:53:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:53:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:53:28.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:53:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:53:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:53:28.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:53:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1117] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:53:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1117] Local address: 172.21.15.161, port 33006 2022-07-03T20:53:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1117] Peer address: 172.21.15.101, port 22 2022-07-03T20:53:28.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1117] Connection lost 2022-07-03T20:53:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1117] Aborting connection 2022-07-03T20:53:28.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:29 smithi161 ceph-mon[91579]: pgmap v18429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:29 smithi188 ceph-mon[92425]: pgmap v18429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:31 smithi161 ceph-mon[91579]: pgmap v18430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:31 smithi188 ceph-mon[92425]: pgmap v18430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:33 smithi161 ceph-mon[91579]: pgmap v18431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:33 smithi188 ceph-mon[92425]: pgmap v18431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:35 smithi161 ceph-mon[91579]: pgmap v18432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:35 smithi188 ceph-mon[92425]: pgmap v18432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:37 smithi161 ceph-mon[91579]: pgmap v18433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:37 smithi188 ceph-mon[92425]: pgmap v18433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:39 smithi161 ceph-mon[91579]: pgmap v18434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:39 smithi188 ceph-mon[92425]: pgmap v18434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:41 smithi161 ceph-mon[91579]: pgmap v18435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:41 smithi188 ceph-mon[92425]: pgmap v18435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:43 smithi161 ceph-mon[91579]: pgmap v18436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:43 smithi188 ceph-mon[92425]: pgmap v18436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:45 smithi161 ceph-mon[91579]: pgmap v18437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:45 smithi188 ceph-mon[92425]: pgmap v18437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:47 smithi161 ceph-mon[91579]: pgmap v18438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:47 smithi188 ceph-mon[92425]: pgmap v18438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:49 smithi161 ceph-mon[91579]: pgmap v18439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:49 smithi188 ceph-mon[92425]: pgmap v18439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:51 smithi161 ceph-mon[91579]: pgmap v18440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:51 smithi188 ceph-mon[92425]: pgmap v18440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:53 smithi161 ceph-mon[91579]: pgmap v18441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:53.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:53 smithi188 ceph-mon[92425]: pgmap v18441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:53.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:55 smithi161 ceph-mon[91579]: pgmap v18442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:55 smithi188 ceph-mon[92425]: pgmap v18442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:53:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:57 smithi161 ceph-mon[91579]: pgmap v18443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:57 smithi188 ceph-mon[92425]: pgmap v18443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:53:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:53:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:59 smithi161 ceph-mon[91579]: pgmap v18444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:53:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:53:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:53:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:53:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:59 smithi188 ceph-mon[92425]: pgmap v18444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:53:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:53:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:53:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:54:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:01 smithi161 ceph-mon[91579]: pgmap v18445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:01 smithi188 ceph-mon[92425]: pgmap v18445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:03 smithi161 ceph-mon[91579]: pgmap v18446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:03 smithi188 ceph-mon[92425]: pgmap v18446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:05 smithi161 ceph-mon[91579]: pgmap v18447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:05 smithi188 ceph-mon[92425]: pgmap v18447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:07 smithi188 ceph-mon[92425]: pgmap v18448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:07 smithi161 ceph-mon[91579]: pgmap v18448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:09 smithi188 ceph-mon[92425]: pgmap v18449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:09 smithi161 ceph-mon[91579]: pgmap v18449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:11 smithi188 ceph-mon[92425]: pgmap v18450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:11 smithi161 ceph-mon[91579]: pgmap v18450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:13 smithi188 ceph-mon[92425]: pgmap v18451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:13 smithi161 ceph-mon[91579]: pgmap v18451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:15 smithi188 ceph-mon[92425]: pgmap v18452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:15 smithi161 ceph-mon[91579]: pgmap v18452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:17 smithi188 ceph-mon[92425]: pgmap v18453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:17 smithi161 ceph-mon[91579]: pgmap v18453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:19 smithi188 ceph-mon[92425]: pgmap v18454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:19 smithi161 ceph-mon[91579]: pgmap v18454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:21.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:20 smithi161 ceph-mon[91579]: pgmap v18455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:20 smithi188 ceph-mon[92425]: pgmap v18455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:23.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:23 smithi161 ceph-mon[91579]: pgmap v18456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:23.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:23 smithi188 ceph-mon[92425]: pgmap v18456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:25 smithi161 ceph-mon[91579]: pgmap v18457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:25 smithi188 ceph-mon[92425]: pgmap v18457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:27.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-mon[91579]: pgmap v18458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:27.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:27 smithi188 ceph-mon[92425]: pgmap v18458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:54:27.469+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:54:27.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:54:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:54:27.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:54:27.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:54:27.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:54:27.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:54:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:54:27.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:54:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:54:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:54:27.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:54:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:54:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:54:27.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:54:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:54:27.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:54:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:54:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:54:27.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:54:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:54:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:54:27.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:54:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:54:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:54:27.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:54:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:54:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:54:27.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:54:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:54:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:54:27.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:54:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:54:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:54:27.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:54:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:54:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:54:27.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:54:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:54:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:54:27.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:54:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:54:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:54:27.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:54:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:54:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:54:27.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1119] Local address: 172.21.15.161, port 33062 2022-07-03T20:54:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1119] Peer address: 172.21.15.101, port 22 2022-07-03T20:54:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1119] Connection lost 2022-07-03T20:54:27.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1119] Aborting connection 2022-07-03T20:54:27.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:54:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:54:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:54:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:54:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:28 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:54:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:28 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:54:28.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:54:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:54:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:54:28.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:54:28.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:54:28.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:54:28.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:54:28.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:54:28.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:54:28.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:54:28.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:54:28.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:54:28.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:54:28.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:54:28.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:54:28.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:54:28.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:54:28.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:54:28.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:54:28.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:54:28.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:54:28.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:54:28.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:54:28.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:54:28.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:54:28.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:54:28.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:54:28.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:54:28.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:54:28.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:54:28.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:54:28.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:54:28.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:54:28.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:54:28.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:54:28.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:54:28.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:54:28.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:54:28.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:54:28.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:54:28.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:54:28.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:54:28.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:54:28.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:54:28.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:54:28.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1119] Local address: 172.21.15.161, port 33062 2022-07-03T20:54:28.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1119] Peer address: 172.21.15.101, port 22 2022-07-03T20:54:28.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1119] Connection lost 2022-07-03T20:54:28.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1119] Aborting connection 2022-07-03T20:54:28.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:54:28.419 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:54:28.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:54:28.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:28 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:54:28.420 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:28 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:54:28.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:54:28.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:54:28.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:54:28.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:54:28.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:54:28.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:54:28.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:54:28.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:54:28.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:54:28.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:54:28.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:54:28.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:54:28.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:54:28.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:54:28.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:54:28.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:54:28.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:54:28.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:54:28.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:54:28.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:54:28.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:54:28.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:54:28.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:54:28.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:54:28.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:54:28.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:54:28.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:54:28.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:54:28.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:54:28.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:54:28.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:54:28.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:54:28.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:54:28.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:54:28.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:54:28.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:54:28.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:54:28.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:54:28.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:54:28.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:54:28.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:54:28.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:54:28.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:54:28.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:54:28.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1119] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:54:28.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1119] Local address: 172.21.15.161, port 33062 2022-07-03T20:54:28.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1119] Peer address: 172.21.15.101, port 22 2022-07-03T20:54:28.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1119] Connection lost 2022-07-03T20:54:28.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1119] Aborting connection 2022-07-03T20:54:29.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:29 smithi161 ceph-mon[91579]: pgmap v18459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:29.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:29.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:29 smithi188 ceph-mon[92425]: pgmap v18459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:29.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:31.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:31 smithi161 ceph-mon[91579]: pgmap v18460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:31.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:31 smithi188 ceph-mon[92425]: pgmap v18460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:33.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:33 smithi161 ceph-mon[91579]: pgmap v18461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:33.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:33 smithi188 ceph-mon[92425]: pgmap v18461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:35 smithi161 ceph-mon[91579]: pgmap v18462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:35 smithi188 ceph-mon[92425]: pgmap v18462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:35.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:37.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:37 smithi161 ceph-mon[91579]: pgmap v18463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:37.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:37 smithi188 ceph-mon[92425]: pgmap v18463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:39.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:39 smithi161 ceph-mon[91579]: pgmap v18464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:39.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:39.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:39 smithi188 ceph-mon[92425]: pgmap v18464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:39.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:41.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:41 smithi188 ceph-mon[92425]: pgmap v18465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:41 smithi161 ceph-mon[91579]: pgmap v18465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:43.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:43 smithi188 ceph-mon[92425]: pgmap v18466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:43 smithi161 ceph-mon[91579]: pgmap v18466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:45 smithi188 ceph-mon[92425]: pgmap v18467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:45.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:45 smithi161 ceph-mon[91579]: pgmap v18467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:47.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:47 smithi188 ceph-mon[92425]: pgmap v18468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:47 smithi161 ceph-mon[91579]: pgmap v18468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:49.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:49 smithi188 ceph-mon[92425]: pgmap v18469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:49.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:49 smithi161 ceph-mon[91579]: pgmap v18469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:51.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:51 smithi188 ceph-mon[92425]: pgmap v18470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:51 smithi161 ceph-mon[91579]: pgmap v18470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:53 smithi161 ceph-mon[91579]: pgmap v18471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:53 smithi188 ceph-mon[92425]: pgmap v18471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:55 smithi161 ceph-mon[91579]: pgmap v18472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:55 smithi188 ceph-mon[92425]: pgmap v18472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:55.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:54:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:57 smithi161 ceph-mon[91579]: pgmap v18473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:57 smithi188 ceph-mon[92425]: pgmap v18473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:54:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: pgmap v18474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:54:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:54:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:54:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:54:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:54:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: pgmap v18474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:54:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:54:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:54:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:54:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:55:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:01 smithi161 ceph-mon[91579]: pgmap v18475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:01 smithi188 ceph-mon[92425]: pgmap v18475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:03 smithi161 ceph-mon[91579]: pgmap v18476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:03 smithi188 ceph-mon[92425]: pgmap v18476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:05 smithi161 ceph-mon[91579]: pgmap v18477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:05 smithi188 ceph-mon[92425]: pgmap v18477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:07 smithi161 ceph-mon[91579]: pgmap v18478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:07 smithi188 ceph-mon[92425]: pgmap v18478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:09 smithi161 ceph-mon[91579]: pgmap v18479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:09 smithi188 ceph-mon[92425]: pgmap v18479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:11 smithi161 ceph-mon[91579]: pgmap v18480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:11 smithi188 ceph-mon[92425]: pgmap v18480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:13 smithi161 ceph-mon[91579]: pgmap v18481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:13 smithi188 ceph-mon[92425]: pgmap v18481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:15 smithi161 ceph-mon[91579]: pgmap v18482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:15 smithi188 ceph-mon[92425]: pgmap v18482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:17 smithi161 ceph-mon[91579]: pgmap v18483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:17 smithi188 ceph-mon[92425]: pgmap v18483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:19 smithi161 ceph-mon[91579]: pgmap v18484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:19 smithi188 ceph-mon[92425]: pgmap v18484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:21 smithi161 ceph-mon[91579]: pgmap v18485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:21 smithi188 ceph-mon[92425]: pgmap v18485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:23 smithi161 ceph-mon[91579]: pgmap v18486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:23 smithi188 ceph-mon[92425]: pgmap v18486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:25 smithi161 ceph-mon[91579]: pgmap v18487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:25 smithi188 ceph-mon[92425]: pgmap v18487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-mon[91579]: pgmap v18488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:27 smithi188 ceph-mon[92425]: pgmap v18488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:28.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:55:27.972+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:55:28.248 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:55:28.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:55:28.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:55:28.249 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:55:28.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:55:28.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:55:28.250 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:55:28.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:55:28.251 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:55:28.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:55:28.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:55:28.252 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:55:28.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:55:28.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:55:28.253 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:55:28.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:55:28.254 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:55:28.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:55:28.255 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:55:28.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:55:28.256 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:55:28.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:55:28.257 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:55:28.258 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:55:28.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:55:28.259 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:55:28.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:55:28.260 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:55:28.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:55:28.261 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:55:28.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:55:28.262 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:55:28.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:55:28.263 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:55:28.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:55:28.264 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:55:28.265 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:55:28.266 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:55:28.266 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:55:28.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:55:28.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:55:28.267 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:55:28.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:55:28.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:55:28.268 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:55:28.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1121] Local address: 172.21.15.161, port 33090 2022-07-03T20:55:28.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1121] Peer address: 172.21.15.101, port 22 2022-07-03T20:55:28.269 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1121] Connection lost 2022-07-03T20:55:28.270 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1121] Aborting connection 2022-07-03T20:55:28.270 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:55:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:55:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:55:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:55:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:55:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:55:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:55:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:55:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:55:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:55:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:55:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:55:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:55:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:55:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:55:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:55:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:55:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:55:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:55:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:55:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:55:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:55:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:55:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:55:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:55:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:55:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:55:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:55:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:55:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:55:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:55:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:55:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:55:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:55:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:55:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:55:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:55:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:55:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:55:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:55:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:55:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:55:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:55:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:55:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:55:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:55:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:55:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:55:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:55:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:55:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:55:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:55:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1121] Local address: 172.21.15.161, port 33090 2022-07-03T20:55:29.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1121] Peer address: 172.21.15.101, port 22 2022-07-03T20:55:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1121] Connection lost 2022-07-03T20:55:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1121] Aborting connection 2022-07-03T20:55:29.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:29 smithi161 ceph-mon[91579]: pgmap v18489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:55:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:55:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:55:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:55:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:55:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:55:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:55:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:55:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:55:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:55:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:55:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:55:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:55:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:55:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:55:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:55:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:55:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:55:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:55:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:55:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:55:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:55:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:55:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:55:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:55:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:55:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:55:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:55:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:55:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:55:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:55:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:55:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:55:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:55:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:55:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:55:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:55:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:55:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:55:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:55:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:55:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:55:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:55:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:55:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:55:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:55:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1121] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:55:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1121] Local address: 172.21.15.161, port 33090 2022-07-03T20:55:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1121] Peer address: 172.21.15.101, port 22 2022-07-03T20:55:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1121] Connection lost 2022-07-03T20:55:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1121] Aborting connection 2022-07-03T20:55:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:29 smithi188 ceph-mon[92425]: pgmap v18489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:31 smithi161 ceph-mon[91579]: pgmap v18490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:31 smithi188 ceph-mon[92425]: pgmap v18490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:33 smithi161 ceph-mon[91579]: pgmap v18491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:33 smithi188 ceph-mon[92425]: pgmap v18491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:35 smithi161 ceph-mon[91579]: pgmap v18492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:35 smithi188 ceph-mon[92425]: pgmap v18492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:37 smithi161 ceph-mon[91579]: pgmap v18493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:37 smithi188 ceph-mon[92425]: pgmap v18493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:38.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:39 smithi161 ceph-mon[91579]: pgmap v18494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:39 smithi188 ceph-mon[92425]: pgmap v18494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:41 smithi161 ceph-mon[91579]: pgmap v18495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:41 smithi188 ceph-mon[92425]: pgmap v18495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:43 smithi161 ceph-mon[91579]: pgmap v18496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:43 smithi188 ceph-mon[92425]: pgmap v18496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:45 smithi161 ceph-mon[91579]: pgmap v18497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:45 smithi188 ceph-mon[92425]: pgmap v18497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:47 smithi161 ceph-mon[91579]: pgmap v18498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:47 smithi188 ceph-mon[92425]: pgmap v18498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:48.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:49 smithi161 ceph-mon[91579]: pgmap v18499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:49 smithi188 ceph-mon[92425]: pgmap v18499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:51 smithi161 ceph-mon[91579]: pgmap v18500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:51 smithi188 ceph-mon[92425]: pgmap v18500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:53 smithi161 ceph-mon[91579]: pgmap v18501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:53 smithi188 ceph-mon[92425]: pgmap v18501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:55 smithi161 ceph-mon[91579]: pgmap v18502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:55 smithi188 ceph-mon[92425]: pgmap v18502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:55:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:57 smithi161 ceph-mon[91579]: pgmap v18503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:57 smithi188 ceph-mon[92425]: pgmap v18503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:55:58.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:55:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:59 smithi161 ceph-mon[91579]: pgmap v18504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:55:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:55:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:55:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:55:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:55:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:59 smithi188 ceph-mon[92425]: pgmap v18504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:55:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:55:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:55:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:55:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:55:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:56:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:01 smithi161 ceph-mon[91579]: pgmap v18505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:01 smithi188 ceph-mon[92425]: pgmap v18505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:03 smithi161 ceph-mon[91579]: pgmap v18506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:03 smithi188 ceph-mon[92425]: pgmap v18506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:05 smithi161 ceph-mon[91579]: pgmap v18507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:05 smithi188 ceph-mon[92425]: pgmap v18507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:07 smithi161 ceph-mon[91579]: pgmap v18508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:07 smithi188 ceph-mon[92425]: pgmap v18508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:08.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:09 smithi161 ceph-mon[91579]: pgmap v18509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:09 smithi188 ceph-mon[92425]: pgmap v18509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:11 smithi161 ceph-mon[91579]: pgmap v18510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:11 smithi188 ceph-mon[92425]: pgmap v18510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:13 smithi161 ceph-mon[91579]: pgmap v18511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:13 smithi188 ceph-mon[92425]: pgmap v18511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:15 smithi161 ceph-mon[91579]: pgmap v18512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:15 smithi188 ceph-mon[92425]: pgmap v18512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:17 smithi161 ceph-mon[91579]: pgmap v18513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:17 smithi188 ceph-mon[92425]: pgmap v18513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:18.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:19 smithi161 ceph-mon[91579]: pgmap v18514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:19 smithi188 ceph-mon[92425]: pgmap v18514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:21 smithi161 ceph-mon[91579]: pgmap v18515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:21 smithi188 ceph-mon[92425]: pgmap v18515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:23 smithi161 ceph-mon[91579]: pgmap v18516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:23.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:23 smithi188 ceph-mon[92425]: pgmap v18516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:25 smithi161 ceph-mon[91579]: pgmap v18517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:25 smithi188 ceph-mon[92425]: pgmap v18517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:27 smithi188 ceph-mon[92425]: pgmap v18518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:27 smithi161 ceph-mon[91579]: pgmap v18518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:28.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:56:28.468+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:56:28.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:56:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:56:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:56:28.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:56:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:56:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:56:28.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:56:28.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:56:28.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:56:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:56:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:56:28.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:56:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:56:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:56:28.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:56:28.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:56:28.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:56:28.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:56:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:56:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:56:28.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:56:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:56:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:56:28.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:56:28.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:56:28.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:56:28.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:56:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:56:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:56:28.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:56:28.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:56:28.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:56:28.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:56:28.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:56:28.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:56:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:56:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:56:28.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:56:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:56:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:56:28.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:56:28.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:56:28.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:56:28.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:56:28.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:56:28.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1123] Local address: 172.21.15.161, port 33118 2022-07-03T20:56:28.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1123] Peer address: 172.21.15.101, port 22 2022-07-03T20:56:28.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1123] Connection lost 2022-07-03T20:56:28.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1123] Aborting connection 2022-07-03T20:56:28.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:56:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: pgmap v18519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:56:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:56:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:56:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:56:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:56:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:56:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:56:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:56:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:56:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:56:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:56:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:56:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:56:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:56:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:56:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:56:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:56:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:56:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:56:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:56:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:56:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:56:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:56:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:56:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:56:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:56:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:56:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:56:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:56:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:56:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:56:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:56:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:56:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:56:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:56:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:56:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:56:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:56:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:56:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:56:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:56:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:56:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:56:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:56:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:56:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:56:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:56:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:56:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:56:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:56:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1123] Local address: 172.21.15.161, port 33118 2022-07-03T20:56:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1123] Peer address: 172.21.15.101, port 22 2022-07-03T20:56:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1123] Connection lost 2022-07-03T20:56:29.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1123] Aborting connection 2022-07-03T20:56:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: pgmap v18519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:56:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:56:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:56:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:56:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:56:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:56:29.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:56:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:56:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:56:29.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:56:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:56:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:56:29.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:56:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:56:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:56:29.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:56:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:56:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:56:29.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:56:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:56:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:56:29.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:56:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:56:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:56:29.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:56:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:56:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:56:29.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:56:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:56:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:56:29.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:56:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:56:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:56:29.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:56:29.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:56:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:56:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:56:29.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:56:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:56:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:56:29.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:56:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:56:29.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:56:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:56:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:56:29.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:56:29.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:56:29.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:56:29.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1123] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:56:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1123] Local address: 172.21.15.161, port 33118 2022-07-03T20:56:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1123] Peer address: 172.21.15.101, port 22 2022-07-03T20:56:29.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1123] Connection lost 2022-07-03T20:56:29.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1123] Aborting connection 2022-07-03T20:56:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:31 smithi188 ceph-mon[92425]: pgmap v18520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:31 smithi161 ceph-mon[91579]: pgmap v18520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:33 smithi188 ceph-mon[92425]: pgmap v18521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:33 smithi161 ceph-mon[91579]: pgmap v18521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:35 smithi188 ceph-mon[92425]: pgmap v18522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:35 smithi161 ceph-mon[91579]: pgmap v18522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:37 smithi188 ceph-mon[92425]: pgmap v18523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:37 smithi161 ceph-mon[91579]: pgmap v18523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:39 smithi188 ceph-mon[92425]: pgmap v18524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:39 smithi161 ceph-mon[91579]: pgmap v18524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:41 smithi161 ceph-mon[91579]: pgmap v18525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:41 smithi188 ceph-mon[92425]: pgmap v18525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:43 smithi161 ceph-mon[91579]: pgmap v18526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:43 smithi188 ceph-mon[92425]: pgmap v18526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:45 smithi161 ceph-mon[91579]: pgmap v18527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:45 smithi188 ceph-mon[92425]: pgmap v18527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:47 smithi161 ceph-mon[91579]: pgmap v18528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:47 smithi188 ceph-mon[92425]: pgmap v18528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:49 smithi161 ceph-mon[91579]: pgmap v18529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:49 smithi188 ceph-mon[92425]: pgmap v18529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:51 smithi161 ceph-mon[91579]: pgmap v18530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:51 smithi188 ceph-mon[92425]: pgmap v18530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:53 smithi161 ceph-mon[91579]: pgmap v18531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:53 smithi188 ceph-mon[92425]: pgmap v18531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:55 smithi161 ceph-mon[91579]: pgmap v18532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:55 smithi188 ceph-mon[92425]: pgmap v18532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:56:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:57 smithi161 ceph-mon[91579]: pgmap v18533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:57 smithi188 ceph-mon[92425]: pgmap v18533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:56:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:56:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:59 smithi161 ceph-mon[91579]: pgmap v18534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:56:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:56:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:56:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:56:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:56:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:59 smithi188 ceph-mon[92425]: pgmap v18534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:56:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:56:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:56:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:56:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:56:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:57:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:01 smithi161 ceph-mon[91579]: pgmap v18535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:01 smithi188 ceph-mon[92425]: pgmap v18535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:03 smithi161 ceph-mon[91579]: pgmap v18536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:03 smithi188 ceph-mon[92425]: pgmap v18536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:05 smithi161 ceph-mon[91579]: pgmap v18537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:05 smithi188 ceph-mon[92425]: pgmap v18537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:07 smithi161 ceph-mon[91579]: pgmap v18538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:07 smithi188 ceph-mon[92425]: pgmap v18538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:09 smithi161 ceph-mon[91579]: pgmap v18539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:09 smithi188 ceph-mon[92425]: pgmap v18539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:11 smithi161 ceph-mon[91579]: pgmap v18540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:11 smithi188 ceph-mon[92425]: pgmap v18540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:13 smithi161 ceph-mon[91579]: pgmap v18541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:13.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:13 smithi188 ceph-mon[92425]: pgmap v18541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:15 smithi161 ceph-mon[91579]: pgmap v18542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:15 smithi188 ceph-mon[92425]: pgmap v18542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:17 smithi161 ceph-mon[91579]: pgmap v18543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:17 smithi188 ceph-mon[92425]: pgmap v18543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:19 smithi161 ceph-mon[91579]: pgmap v18544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:19 smithi188 ceph-mon[92425]: pgmap v18544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:21 smithi161 ceph-mon[91579]: pgmap v18545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:21 smithi188 ceph-mon[92425]: pgmap v18545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:23 smithi161 ceph-mon[91579]: pgmap v18546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:23 smithi188 ceph-mon[92425]: pgmap v18546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:25 smithi161 ceph-mon[91579]: pgmap v18547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:25 smithi188 ceph-mon[92425]: pgmap v18547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:27 smithi161 ceph-mon[91579]: pgmap v18548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:27 smithi188 ceph-mon[92425]: pgmap v18548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:29.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:57:28.966+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:57:29.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:57:29.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:57:29.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:57:29.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:57:29.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:57:29.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:57:29.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:57:29.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:57:29.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:57:29.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:57:29.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:57:29.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:57:29.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:57:29.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:57:29.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:57:29.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:57:29.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:57:29.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:57:29.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:57:29.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:57:29.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:57:29.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:57:29.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:57:29.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:57:29.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:57:29.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:57:29.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:57:29.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:57:29.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:57:29.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:57:29.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:57:29.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:57:29.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:57:29.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:57:29.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:57:29.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:57:29.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:57:29.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:57:29.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:57:29.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:57:29.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:57:29.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:57:29.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:57:29.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:57:29.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:57:29.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1125] Local address: 172.21.15.161, port 33146 2022-07-03T20:57:29.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1125] Peer address: 172.21.15.101, port 22 2022-07-03T20:57:29.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1125] Connection lost 2022-07-03T20:57:29.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1125] Aborting connection 2022-07-03T20:57:29.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:57:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:57:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:29 smithi161 ceph-mon[91579]: pgmap v18549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:57:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:57:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:57:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:29 smithi188 ceph-mon[92425]: pgmap v18549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:57:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:57:29.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:57:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:57:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:57:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:57:30.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:57:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:57:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:57:30.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:57:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:57:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:57:30.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:57:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:57:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:57:30.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:57:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:57:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:57:30.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:57:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:57:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:57:30.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:57:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:57:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:57:30.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:57:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:57:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:57:30.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:57:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:57:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:57:30.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:57:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:57:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:57:30.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:57:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:57:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:57:30.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:57:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:57:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:57:30.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:57:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:57:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:57:30.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:57:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:57:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:57:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:57:30.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:57:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:57:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:57:30.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:57:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1125] Local address: 172.21.15.161, port 33146 2022-07-03T20:57:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1125] Peer address: 172.21.15.101, port 22 2022-07-03T20:57:30.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1125] Connection lost 2022-07-03T20:57:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1125] Aborting connection 2022-07-03T20:57:30.895 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:57:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:57:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:57:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:57:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:57:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:57:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:57:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:57:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:57:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:57:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:57:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:57:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:57:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:57:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:57:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:57:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:57:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:57:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:57:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:57:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:57:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:57:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:57:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:57:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:57:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:57:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:57:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:57:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:57:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:57:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:57:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:57:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:57:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:57:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:57:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:57:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:57:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:57:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:57:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:57:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:57:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:57:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:57:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:57:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:57:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:57:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1125] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:57:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1125] Local address: 172.21.15.161, port 33146 2022-07-03T20:57:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1125] Peer address: 172.21.15.101, port 22 2022-07-03T20:57:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1125] Connection lost 2022-07-03T20:57:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1125] Aborting connection 2022-07-03T20:57:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:31 smithi161 ceph-mon[91579]: pgmap v18550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:31 smithi188 ceph-mon[92425]: pgmap v18550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:33 smithi161 ceph-mon[91579]: pgmap v18551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:33 smithi188 ceph-mon[92425]: pgmap v18551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:35 smithi161 ceph-mon[91579]: pgmap v18552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:35 smithi188 ceph-mon[92425]: pgmap v18552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:37 smithi161 ceph-mon[91579]: pgmap v18553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:37 smithi188 ceph-mon[92425]: pgmap v18553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:39 smithi161 ceph-mon[91579]: pgmap v18554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:39 smithi188 ceph-mon[92425]: pgmap v18554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:41 smithi161 ceph-mon[91579]: pgmap v18555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:41 smithi188 ceph-mon[92425]: pgmap v18555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:43 smithi161 ceph-mon[91579]: pgmap v18556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:43 smithi188 ceph-mon[92425]: pgmap v18556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:45 smithi161 ceph-mon[91579]: pgmap v18557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:45 smithi188 ceph-mon[92425]: pgmap v18557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:47 smithi161 ceph-mon[91579]: pgmap v18558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:47 smithi188 ceph-mon[92425]: pgmap v18558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:49 smithi161 ceph-mon[91579]: pgmap v18559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:49 smithi188 ceph-mon[92425]: pgmap v18559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:51 smithi161 ceph-mon[91579]: pgmap v18560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:51 smithi188 ceph-mon[92425]: pgmap v18560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:53 smithi161 ceph-mon[91579]: pgmap v18561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:53 smithi188 ceph-mon[92425]: pgmap v18561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:55 smithi161 ceph-mon[91579]: pgmap v18562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:55 smithi188 ceph-mon[92425]: pgmap v18562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:57:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:57 smithi161 ceph-mon[91579]: pgmap v18563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:57 smithi188 ceph-mon[92425]: pgmap v18563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:57:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:57:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:59 smithi161 ceph-mon[91579]: pgmap v18564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:57:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:57:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:57:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:57:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:57:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:59 smithi188 ceph-mon[92425]: pgmap v18564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:57:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:57:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:57:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:57:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:57:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:58:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:01 smithi161 ceph-mon[91579]: pgmap v18565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:01 smithi188 ceph-mon[92425]: pgmap v18565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:03 smithi161 ceph-mon[91579]: pgmap v18566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:03 smithi188 ceph-mon[92425]: pgmap v18566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:05 smithi161 ceph-mon[91579]: pgmap v18567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:05 smithi188 ceph-mon[92425]: pgmap v18567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:07 smithi161 ceph-mon[91579]: pgmap v18568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:07 smithi188 ceph-mon[92425]: pgmap v18568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:09 smithi161 ceph-mon[91579]: pgmap v18569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:09 smithi188 ceph-mon[92425]: pgmap v18569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:11 smithi161 ceph-mon[91579]: pgmap v18570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:11 smithi188 ceph-mon[92425]: pgmap v18570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:13 smithi161 ceph-mon[91579]: pgmap v18571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:13 smithi188 ceph-mon[92425]: pgmap v18571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:15 smithi161 ceph-mon[91579]: pgmap v18572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:15 smithi188 ceph-mon[92425]: pgmap v18572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:17 smithi161 ceph-mon[91579]: pgmap v18573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:17 smithi188 ceph-mon[92425]: pgmap v18573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:18.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:19 smithi161 ceph-mon[91579]: pgmap v18574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:19 smithi188 ceph-mon[92425]: pgmap v18574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:21 smithi188 ceph-mon[92425]: pgmap v18575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:21 smithi161 ceph-mon[91579]: pgmap v18575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:23 smithi188 ceph-mon[92425]: pgmap v18576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:23 smithi161 ceph-mon[91579]: pgmap v18576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:25 smithi188 ceph-mon[92425]: pgmap v18577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:25 smithi161 ceph-mon[91579]: pgmap v18577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:27 smithi188 ceph-mon[92425]: pgmap v18578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:27 smithi161 ceph-mon[91579]: pgmap v18578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-mon[91579]: pgmap v18579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:58:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:58:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:58:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:58:29.472+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:58:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:58:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:58:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:58:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:58:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:58:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:58:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:58:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:58:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:58:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:58:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:58:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:58:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:58:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:58:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:58:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:58:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:58:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:58:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:58:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:58:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:58:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:58:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:58:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:58:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:58:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:58:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:58:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:58:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:58:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:58:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:58:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:58:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:58:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:58:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:58:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:58:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:58:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:58:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:58:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:58:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:58:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:58:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:58:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:58:29.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:58:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1127] Local address: 172.21.15.161, port 33174 2022-07-03T20:58:29.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1127] Peer address: 172.21.15.101, port 22 2022-07-03T20:58:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1127] Connection lost 2022-07-03T20:58:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1127] Aborting connection 2022-07-03T20:58:29.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:58:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:29 smithi188 ceph-mon[92425]: pgmap v18579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:58:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:58:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:58:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:58:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:58:30.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:58:30.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:58:30.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:58:30.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:58:30.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:58:30.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:58:30.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:58:30.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:58:30.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:58:30.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:58:30.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:58:30.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:58:30.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:58:30.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:58:30.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1127] Local address: 172.21.15.161, port 33174 2022-07-03T20:58:30.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1127] Peer address: 172.21.15.101, port 22 2022-07-03T20:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1127] Connection lost 2022-07-03T20:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1127] Aborting connection 2022-07-03T20:58:30.929 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:58:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:58:31.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:58:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:58:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:58:31.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:58:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:58:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:58:31.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:58:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:58:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:58:31.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:58:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:58:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:58:31.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:58:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:58:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:58:31.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:58:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:58:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:58:31.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:58:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:58:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:58:31.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:58:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:58:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:58:31.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:58:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:58:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:58:31.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:58:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:58:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:58:31.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:58:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:58:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:58:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:58:31.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:58:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:58:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:58:31.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:58:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:58:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:58:31.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:58:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:58:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:58:31.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:58:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1127] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:58:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1127] Local address: 172.21.15.161, port 33174 2022-07-03T20:58:31.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1127] Peer address: 172.21.15.101, port 22 2022-07-03T20:58:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1127] Connection lost 2022-07-03T20:58:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1127] Aborting connection 2022-07-03T20:58:31.145 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:31 smithi188 ceph-mon[92425]: pgmap v18580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:31 smithi161 ceph-mon[91579]: pgmap v18580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:33 smithi188 ceph-mon[92425]: pgmap v18581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:33 smithi161 ceph-mon[91579]: pgmap v18581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:35 smithi188 ceph-mon[92425]: pgmap v18582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:35 smithi161 ceph-mon[91579]: pgmap v18582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:37 smithi188 ceph-mon[92425]: pgmap v18583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:37 smithi161 ceph-mon[91579]: pgmap v18583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:39.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:39 smithi161 ceph-mon[91579]: pgmap v18584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:39 smithi188 ceph-mon[92425]: pgmap v18584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:41 smithi161 ceph-mon[91579]: pgmap v18585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:41 smithi188 ceph-mon[92425]: pgmap v18585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:43 smithi161 ceph-mon[91579]: pgmap v18586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:43 smithi188 ceph-mon[92425]: pgmap v18586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:45 smithi161 ceph-mon[91579]: pgmap v18587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:45 smithi188 ceph-mon[92425]: pgmap v18587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:47 smithi161 ceph-mon[91579]: pgmap v18588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:47 smithi188 ceph-mon[92425]: pgmap v18588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:49 smithi161 ceph-mon[91579]: pgmap v18589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:49 smithi188 ceph-mon[92425]: pgmap v18589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:51 smithi161 ceph-mon[91579]: pgmap v18590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:51 smithi188 ceph-mon[92425]: pgmap v18590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:53 smithi161 ceph-mon[91579]: pgmap v18591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:53 smithi188 ceph-mon[92425]: pgmap v18591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:55 smithi161 ceph-mon[91579]: pgmap v18592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:55 smithi188 ceph-mon[92425]: pgmap v18592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:58:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:57 smithi161 ceph-mon[91579]: pgmap v18593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:57 smithi188 ceph-mon[92425]: pgmap v18593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:58:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:58 smithi161 ceph-mon[91579]: pgmap v18594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:58:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:58 smithi188 ceph-mon[92425]: pgmap v18594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:58:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:58:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:58:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:00 smithi161 ceph-mon[91579]: pgmap v18595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:00 smithi188 ceph-mon[92425]: pgmap v18595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:03.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:03 smithi188 ceph-mon[92425]: pgmap v18596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:03 smithi161 ceph-mon[91579]: pgmap v18596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:05 smithi161 ceph-mon[91579]: pgmap v18597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:05 smithi188 ceph-mon[92425]: pgmap v18597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:07 smithi161 ceph-mon[91579]: pgmap v18598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:07 smithi188 ceph-mon[92425]: pgmap v18598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:09 smithi161 ceph-mon[91579]: pgmap v18599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:09 smithi188 ceph-mon[92425]: pgmap v18599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:11 smithi161 ceph-mon[91579]: pgmap v18600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:11 smithi188 ceph-mon[92425]: pgmap v18600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:13 smithi161 ceph-mon[91579]: pgmap v18601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:13 smithi188 ceph-mon[92425]: pgmap v18601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:15 smithi161 ceph-mon[91579]: pgmap v18602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:15 smithi188 ceph-mon[92425]: pgmap v18602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:17 smithi161 ceph-mon[91579]: pgmap v18603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:17 smithi188 ceph-mon[92425]: pgmap v18603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:19 smithi161 ceph-mon[91579]: pgmap v18604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:19 smithi188 ceph-mon[92425]: pgmap v18604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:21 smithi161 ceph-mon[91579]: pgmap v18605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:21 smithi188 ceph-mon[92425]: pgmap v18605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:23 smithi161 ceph-mon[91579]: pgmap v18606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:23 smithi188 ceph-mon[92425]: pgmap v18606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:25 smithi161 ceph-mon[91579]: pgmap v18607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:25 smithi188 ceph-mon[92425]: pgmap v18607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:27 smithi161 ceph-mon[91579]: pgmap v18608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:27 smithi188 ceph-mon[92425]: pgmap v18608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-mon[91579]: pgmap v18609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:29 smithi188 ceph-mon[92425]: pgmap v18609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:59:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:59:30.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:59:30.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:59:29.976+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:59:30.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:59:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:59:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T20:59:30.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:59:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:59:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:59:30.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T20:59:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:59:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T20:59:30.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:59:30.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T20:59:30.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:59:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:59:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T20:59:30.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:59:30.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T20:59:30.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:59:30.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T20:59:30.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:59:30.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T20:59:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:59:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:59:30.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:59:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:59:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:59:30.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T20:59:30.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:59:30.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:59:30.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:59:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T20:59:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:59:30.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T20:59:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:59:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:59:30.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:59:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T20:59:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:59:30.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T20:59:30.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:59:30.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T20:59:30.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:59:30.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T20:59:30.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:59:30.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:59:30.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:59:30.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1129] Local address: 172.21.15.161, port 33204 2022-07-03T20:59:30.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1129] Peer address: 172.21.15.101, port 22 2022-07-03T20:59:30.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1129] Connection lost 2022-07-03T20:59:30.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1129] Aborting connection 2022-07-03T20:59:30.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 20:59:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T20:59:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T20:59:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T20:59:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T20:59:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:59:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:59:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:59:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:59:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T20:59:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:59:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:59:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:59:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T20:59:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:59:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T20:59:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:59:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T20:59:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:59:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:59:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:59:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T20:59:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T20:59:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:59:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T20:59:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:59:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T20:59:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:59:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:59:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:59:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:59:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:59:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:59:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:59:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:59:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:59:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T20:59:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:59:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T20:59:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:59:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:59:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:59:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:59:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:59:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T20:59:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:59:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:59:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:59:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T20:59:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:59:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:59:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:59:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1129] Local address: 172.21.15.161, port 33204 2022-07-03T20:59:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1129] Peer address: 172.21.15.101, port 22 2022-07-03T20:59:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1129] Connection lost 2022-07-03T20:59:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1129] Aborting connection 2022-07-03T20:59:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: pgmap v18610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.682 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:31 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T20:59:31.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T20:59:31.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:59:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T20:59:31.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T20:59:31.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T20:59:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T20:59:31.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T20:59:31.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T20:59:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T20:59:31.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T20:59:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T20:59:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T20:59:31.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T20:59:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:59:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T20:59:31.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T20:59:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T20:59:31.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T20:59:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T20:59:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T20:59:31.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T20:59:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T20:59:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T20:59:31.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T20:59:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T20:59:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T20:59:31.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T20:59:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T20:59:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T20:59:31.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T20:59:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T20:59:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T20:59:31.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T20:59:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T20:59:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T20:59:31.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T20:59:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T20:59:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T20:59:31.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T20:59:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T20:59:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T20:59:31.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T20:59:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T20:59:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T20:59:31.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T20:59:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1129] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T20:59:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1129] Local address: 172.21.15.161, port 33204 2022-07-03T20:59:31.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1129] Peer address: 172.21.15.101, port 22 2022-07-03T20:59:31.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1129] Connection lost 2022-07-03T20:59:31.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1129] Aborting connection 2022-07-03T20:59:31.901 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: pgmap v18610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:31.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.902 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:31.903 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:31 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T20:59:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:33 smithi188 ceph-mon[92425]: pgmap v18611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:33 smithi161 ceph-mon[91579]: pgmap v18611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:35 smithi188 ceph-mon[92425]: pgmap v18612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:35 smithi161 ceph-mon[91579]: pgmap v18612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:37 smithi161 ceph-mon[91579]: pgmap v18613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:37 smithi188 ceph-mon[92425]: pgmap v18613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:38.883 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:39 smithi161 ceph-mon[91579]: pgmap v18614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:39 smithi188 ceph-mon[92425]: pgmap v18614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:41 smithi161 ceph-mon[91579]: pgmap v18615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:41 smithi188 ceph-mon[92425]: pgmap v18615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:43 smithi161 ceph-mon[91579]: pgmap v18616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:43 smithi188 ceph-mon[92425]: pgmap v18616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:45 smithi161 ceph-mon[91579]: pgmap v18617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:45 smithi188 ceph-mon[92425]: pgmap v18617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:47 smithi161 ceph-mon[91579]: pgmap v18618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:47 smithi188 ceph-mon[92425]: pgmap v18618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:49 smithi161 ceph-mon[91579]: pgmap v18619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:49 smithi188 ceph-mon[92425]: pgmap v18619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:51 smithi161 ceph-mon[91579]: pgmap v18620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:51 smithi188 ceph-mon[92425]: pgmap v18620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:53 smithi161 ceph-mon[91579]: pgmap v18621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:53 smithi188 ceph-mon[92425]: pgmap v18621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:55 smithi161 ceph-mon[91579]: pgmap v18622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:55 smithi188 ceph-mon[92425]: pgmap v18622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T20:59:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:57 smithi161 ceph-mon[91579]: pgmap v18623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:57 smithi188 ceph-mon[92425]: pgmap v18623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T20:59:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T20:59:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:59 smithi161 ceph-mon[91579]: pgmap v18624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 20:59:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:59 smithi188 ceph-mon[92425]: pgmap v18624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T20:59:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T20:59:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T20:59:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 20:59:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:00:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:00.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:00:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:00.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:00:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:01 smithi161 ceph-mon[91579]: pgmap v18625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:01 smithi188 ceph-mon[92425]: pgmap v18625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:03 smithi161 ceph-mon[91579]: pgmap v18626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:03 smithi188 ceph-mon[92425]: pgmap v18626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:05 smithi161 ceph-mon[91579]: pgmap v18627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:05 smithi188 ceph-mon[92425]: pgmap v18627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:07 smithi161 ceph-mon[91579]: pgmap v18628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:07 smithi188 ceph-mon[92425]: pgmap v18628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:09 smithi161 ceph-mon[91579]: pgmap v18629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:09 smithi188 ceph-mon[92425]: pgmap v18629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:11 smithi161 ceph-mon[91579]: pgmap v18630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:11 smithi188 ceph-mon[92425]: pgmap v18630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:13 smithi161 ceph-mon[91579]: pgmap v18631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:13 smithi188 ceph-mon[92425]: pgmap v18631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:15 smithi161 ceph-mon[91579]: pgmap v18632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:15 smithi188 ceph-mon[92425]: pgmap v18632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:17 smithi161 ceph-mon[91579]: pgmap v18633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:17 smithi188 ceph-mon[92425]: pgmap v18633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:19 smithi161 ceph-mon[91579]: pgmap v18634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:19 smithi188 ceph-mon[92425]: pgmap v18634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:21 smithi161 ceph-mon[91579]: pgmap v18635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:21 smithi188 ceph-mon[92425]: pgmap v18635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:23 smithi161 ceph-mon[91579]: pgmap v18636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:23 smithi188 ceph-mon[92425]: pgmap v18636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:25 smithi161 ceph-mon[91579]: pgmap v18637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:25 smithi188 ceph-mon[92425]: pgmap v18637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:27 smithi161 ceph-mon[91579]: pgmap v18638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:27 smithi188 ceph-mon[92425]: pgmap v18638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:28.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:29 smithi161 ceph-mon[91579]: pgmap v18639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:29 smithi188 ceph-mon[92425]: pgmap v18639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:30.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:31.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:00:30.909+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:00:31.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:00:31.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:00:31.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:00:31.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:00:31.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:00:31.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:00:31.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:00:31.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:00:31.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:00:31.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:00:31.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:00:31.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:00:31.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:00:31.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:00:31.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:00:31.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:00:31.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:00:31.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:00:31.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:00:31.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:00:31.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:00:31.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:00:31.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:00:31.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:00:31.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:00:31.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:00:31.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:00:31.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:00:31.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:00:31.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:00:31.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:00:31.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:00:31.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:00:31.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:00:31.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:00:31.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:00:31.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:00:31.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:00:31.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:00:31.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:00:31.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:00:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:00:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:00:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:00:31.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:00:31.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1131] Local address: 172.21.15.161, port 33232 2022-07-03T21:00:31.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1131] Peer address: 172.21.15.101, port 22 2022-07-03T21:00:31.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1131] Connection lost 2022-07-03T21:00:31.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1131] Aborting connection 2022-07-03T21:00:31.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:00:30 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:00:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:31 smithi161 ceph-mon[91579]: pgmap v18640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:00:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:00:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:00:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:31 smithi188 ceph-mon[92425]: pgmap v18640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:00:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:00:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:00:32.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:00:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:00:32.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:00:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:00:32.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:00:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:00:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:00:32.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:00:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:00:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:00:32.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:00:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:00:32.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:00:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:00:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:00:32.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:00:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:00:32.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:00:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:00:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:00:32.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:00:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:00:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:00:32.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:00:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:00:32.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:00:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:00:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:00:32.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:00:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:00:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:00:32.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:00:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:00:32.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:00:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:00:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:00:32.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:00:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:00:32.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:00:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:00:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:00:32.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:00:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:00:32.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:00:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:00:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:00:32.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:00:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1131] Local address: 172.21.15.161, port 33232 2022-07-03T21:00:32.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1131] Peer address: 172.21.15.101, port 22 2022-07-03T21:00:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1131] Connection lost 2022-07-03T21:00:32.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1131] Aborting connection 2022-07-03T21:00:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:00:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:00:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:00:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:00:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:00:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:00:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:00:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:00:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:00:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:00:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:00:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:00:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:00:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:00:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:00:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:00:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:00:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:00:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:00:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:00:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:00:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:00:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:00:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:00:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:00:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:00:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:00:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:00:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:00:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:00:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:00:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:00:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:00:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:00:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:00:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:00:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:00:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:00:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:00:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:00:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:00:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:00:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:00:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:00:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:00:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:00:32.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1131] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:00:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1131] Local address: 172.21.15.161, port 33232 2022-07-03T21:00:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1131] Peer address: 172.21.15.101, port 22 2022-07-03T21:00:32.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1131] Connection lost 2022-07-03T21:00:32.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1131] Aborting connection 2022-07-03T21:00:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:33 smithi161 ceph-mon[91579]: pgmap v18641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:33 smithi188 ceph-mon[92425]: pgmap v18641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:35 smithi161 ceph-mon[91579]: pgmap v18642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:35 smithi188 ceph-mon[92425]: pgmap v18642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:37 smithi161 ceph-mon[91579]: pgmap v18643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:37 smithi188 ceph-mon[92425]: pgmap v18643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:38.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:39 smithi161 ceph-mon[91579]: pgmap v18644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:39 smithi188 ceph-mon[92425]: pgmap v18644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:41 smithi161 ceph-mon[91579]: pgmap v18645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:41 smithi188 ceph-mon[92425]: pgmap v18645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:43 smithi161 ceph-mon[91579]: pgmap v18646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:43 smithi188 ceph-mon[92425]: pgmap v18646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:45 smithi161 ceph-mon[91579]: pgmap v18647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:45 smithi188 ceph-mon[92425]: pgmap v18647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:47 smithi161 ceph-mon[91579]: pgmap v18648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:47 smithi188 ceph-mon[92425]: pgmap v18648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:49 smithi161 ceph-mon[91579]: pgmap v18649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:49 smithi188 ceph-mon[92425]: pgmap v18649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:51 smithi161 ceph-mon[91579]: pgmap v18650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:51 smithi188 ceph-mon[92425]: pgmap v18650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:53 smithi161 ceph-mon[91579]: pgmap v18651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:53.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:53 smithi188 ceph-mon[92425]: pgmap v18651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:55 smithi161 ceph-mon[91579]: pgmap v18652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:55 smithi188 ceph-mon[92425]: pgmap v18652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:00:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:57 smithi161 ceph-mon[91579]: pgmap v18653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:57 smithi188 ceph-mon[92425]: pgmap v18653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:00:58.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:00:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:59 smithi161 ceph-mon[91579]: pgmap v18654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:00:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:00:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:00:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:00:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:00:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:59 smithi188 ceph-mon[92425]: pgmap v18654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:00:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:00:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:00:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:00:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:00:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:01:00.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:00.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:01 smithi161 ceph-mon[91579]: pgmap v18655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:01 smithi188 ceph-mon[92425]: pgmap v18655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:03 smithi161 ceph-mon[91579]: pgmap v18656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:03.926 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:03 smithi188 ceph-mon[92425]: pgmap v18656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:03.927 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:05 smithi161 ceph-mon[91579]: pgmap v18657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:05 smithi188 ceph-mon[92425]: pgmap v18657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:07 smithi161 ceph-mon[91579]: pgmap v18658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:07 smithi188 ceph-mon[92425]: pgmap v18658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:08.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:08.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:09 smithi161 ceph-mon[91579]: pgmap v18659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:09 smithi188 ceph-mon[92425]: pgmap v18659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:10.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:10.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:11 smithi161 ceph-mon[91579]: pgmap v18660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:11 smithi188 ceph-mon[92425]: pgmap v18660: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:13 smithi161 ceph-mon[91579]: pgmap v18661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:13 smithi188 ceph-mon[92425]: pgmap v18661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:15 smithi161 ceph-mon[91579]: pgmap v18662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:15 smithi188 ceph-mon[92425]: pgmap v18662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:17 smithi161 ceph-mon[91579]: pgmap v18663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:17 smithi188 ceph-mon[92425]: pgmap v18663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:18.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:18.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:19 smithi161 ceph-mon[91579]: pgmap v18664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:19 smithi188 ceph-mon[92425]: pgmap v18664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:20.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:20.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:21 smithi161 ceph-mon[91579]: pgmap v18665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:21 smithi188 ceph-mon[92425]: pgmap v18665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:23 smithi161 ceph-mon[91579]: pgmap v18666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:23 smithi188 ceph-mon[92425]: pgmap v18666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:25 smithi161 ceph-mon[91579]: pgmap v18667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:25 smithi188 ceph-mon[92425]: pgmap v18667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:27 smithi161 ceph-mon[91579]: pgmap v18668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:27 smithi188 ceph-mon[92425]: pgmap v18668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:28.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:29.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:29 smithi188 ceph-mon[92425]: pgmap v18669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:29 smithi161 ceph-mon[91579]: pgmap v18669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:30.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-mon[91579]: pgmap v18670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:01:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:01:31.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:01:31.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:01:31.418+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:01:31.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:01:31.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:01:31.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:01:31.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:01:31.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:01:31.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:01:31.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:01:31.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:01:31.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:01:31.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:01:31.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:01:31.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:01:31.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:01:31.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:01:31.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:01:31.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:01:31.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:01:31.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:01:31.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:01:31.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:01:31.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:01:31.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:01:31.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:01:31.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:01:31.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:01:31.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:01:31.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:01:31.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:01:31.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:01:31.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:01:31.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:01:31.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:01:31.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:01:31.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:01:31.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:01:31.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:01:31.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:01:31.901 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:01:31.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:01:31.902 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:01:31.903 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:01:31.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:01:31.904 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:01:31.905 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:01:31.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:01:31.906 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1133] Local address: 172.21.15.161, port 33260 2022-07-03T21:01:31.907 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1133] Peer address: 172.21.15.101, port 22 2022-07-03T21:01:31.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1133] Connection lost 2022-07-03T21:01:31.908 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1133] Aborting connection 2022-07-03T21:01:31.909 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:01:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:01:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:31 smithi188 ceph-mon[92425]: pgmap v18670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:01:31.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:01:31.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:31 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:01:32.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:32 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:01:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:32 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:01:32.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:01:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:01:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:01:32.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:01:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:01:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:01:32.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:01:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:01:32.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:01:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:01:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:01:32.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:01:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:01:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:01:32.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:01:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:01:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:01:32.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:01:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:01:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:01:32.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:01:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:01:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:01:32.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:01:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:01:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:01:32.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:01:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:01:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:01:32.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:01:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:01:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:01:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:01:32.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:01:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:01:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:01:32.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:01:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:01:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:01:32.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:01:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:01:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:01:32.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:01:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:01:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:01:32.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1133] Local address: 172.21.15.161, port 33260 2022-07-03T21:01:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1133] Peer address: 172.21.15.101, port 22 2022-07-03T21:01:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1133] Connection lost 2022-07-03T21:01:32.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1133] Aborting connection 2022-07-03T21:01:33.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:32 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:01:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:32 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:01:33.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:01:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:01:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:01:33.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:01:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:01:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:01:33.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:01:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:01:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:01:33.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:01:33.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:01:33.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:01:33.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:01:33.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:01:33.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:01:33.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:01:33.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:01:33.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:01:33.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:01:33.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:01:33.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:01:33.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:01:33.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:01:33.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:01:33.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:01:33.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:01:33.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:01:33.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:01:33.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:01:33.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:01:33.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:01:33.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:01:33.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:01:33.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:01:33.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:01:33.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:01:33.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:01:33.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:01:33.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:01:33.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:01:33.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:01:33.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:01:33.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:01:33.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:01:33.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1133] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:01:33.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1133] Local address: 172.21.15.161, port 33260 2022-07-03T21:01:33.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1133] Peer address: 172.21.15.101, port 22 2022-07-03T21:01:33.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1133] Connection lost 2022-07-03T21:01:33.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1133] Aborting connection 2022-07-03T21:01:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:33 smithi188 ceph-mon[92425]: pgmap v18671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:33 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:33 smithi161 ceph-mon[91579]: pgmap v18671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:33 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:35 smithi188 ceph-mon[92425]: pgmap v18672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:35 smithi161 ceph-mon[91579]: pgmap v18672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:37 smithi188 ceph-mon[92425]: pgmap v18673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:37 smithi161 ceph-mon[91579]: pgmap v18673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:38.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:39 smithi188 ceph-mon[92425]: pgmap v18674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:39 smithi161 ceph-mon[91579]: pgmap v18674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:40.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:41 smithi188 ceph-mon[92425]: pgmap v18675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:41 smithi161 ceph-mon[91579]: pgmap v18675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:43 smithi188 ceph-mon[92425]: pgmap v18676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:43 smithi161 ceph-mon[91579]: pgmap v18676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:01:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:45 smithi188 ceph-mon[92425]: pgmap v18677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:45 smithi161 ceph-mon[91579]: pgmap v18677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:46.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:47 smithi161 ceph-mon[91579]: pgmap v18678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:47 smithi188 ceph-mon[92425]: pgmap v18678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:49 smithi161 ceph-mon[91579]: pgmap v18679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:49 smithi188 ceph-mon[92425]: pgmap v18679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:51 smithi161 ceph-mon[91579]: pgmap v18680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:51 smithi188 ceph-mon[92425]: pgmap v18680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:53 smithi161 ceph-mon[91579]: pgmap v18681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:53 smithi188 ceph-mon[92425]: pgmap v18681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:55 smithi161 ceph-mon[91579]: pgmap v18682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:55 smithi188 ceph-mon[92425]: pgmap v18682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:01:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:01:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:57 smithi161 ceph-mon[91579]: pgmap v18683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:57 smithi188 ceph-mon[92425]: pgmap v18683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:01:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:01:59.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:59 smithi161 ceph-mon[91579]: pgmap v18684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:02:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:02:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:02:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:01:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:02:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:59 smithi188 ceph-mon[92425]: pgmap v18684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:02:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:02:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:02:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:01:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:02:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:01 smithi161 ceph-mon[91579]: pgmap v18685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:01 smithi188 ceph-mon[92425]: pgmap v18685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:03 smithi161 ceph-mon[91579]: pgmap v18686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:03 smithi188 ceph-mon[92425]: pgmap v18686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:05 smithi161 ceph-mon[91579]: pgmap v18687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:05 smithi188 ceph-mon[92425]: pgmap v18687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:07 smithi161 ceph-mon[91579]: pgmap v18688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:07 smithi188 ceph-mon[92425]: pgmap v18688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:09.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:09 smithi161 ceph-mon[91579]: pgmap v18689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:09 smithi188 ceph-mon[92425]: pgmap v18689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:11 smithi161 ceph-mon[91579]: pgmap v18690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:11 smithi188 ceph-mon[92425]: pgmap v18690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:13.715 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:13 smithi188 ceph-mon[92425]: pgmap v18691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:13.716 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:13 smithi161 ceph-mon[91579]: pgmap v18691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:15 smithi161 ceph-mon[91579]: pgmap v18692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:15 smithi188 ceph-mon[92425]: pgmap v18692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:17 smithi161 ceph-mon[91579]: pgmap v18693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:17 smithi188 ceph-mon[92425]: pgmap v18693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:19.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:19.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:19 smithi161 ceph-mon[91579]: pgmap v18694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:19 smithi188 ceph-mon[92425]: pgmap v18694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:21.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:21 smithi161 ceph-mon[91579]: pgmap v18695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:21 smithi188 ceph-mon[92425]: pgmap v18695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:23 smithi161 ceph-mon[91579]: pgmap v18696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:23 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:23 smithi188 ceph-mon[92425]: pgmap v18696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:23 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:25 smithi161 ceph-mon[91579]: pgmap v18697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:25 smithi188 ceph-mon[92425]: pgmap v18697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:27.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:26 smithi161 ceph-mon[91579]: pgmap v18698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:27.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:26 smithi188 ceph-mon[92425]: pgmap v18698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:29 smithi161 ceph-mon[91579]: pgmap v18699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:29 smithi188 ceph-mon[92425]: pgmap v18699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-mon[91579]: pgmap v18700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:31 smithi188 ceph-mon[92425]: pgmap v18700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:32.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:02:31.932+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:02:32.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:02:32.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:02:32.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:02:32.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:02:32.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:02:32.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:02:32.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:02:32.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:02:32.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:02:32.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:02:32.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:02:32.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:02:32.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:02:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:02:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:02:32.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:02:32.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:02:32.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:02:32.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:02:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:02:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:02:32.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:02:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:02:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:02:32.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:02:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:02:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:02:32.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:02:32.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:02:32.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:02:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:02:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:02:32.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:02:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:02:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:02:32.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:02:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:02:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:02:32.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:02:32.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:02:32.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:02:32.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:02:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:02:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:02:32.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:02:32.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1135] Local address: 172.21.15.161, port 33288 2022-07-03T21:02:32.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1135] Peer address: 172.21.15.101, port 22 2022-07-03T21:02:32.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1135] Connection lost 2022-07-03T21:02:32.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1135] Aborting connection 2022-07-03T21:02:32.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:02:31 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:02:32.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:02:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:02:32.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:32 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:02:32.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:02:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:02:32.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:32 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:02:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:33 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:02:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:33 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:02:33.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:02:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:02:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:02:33.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:02:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:02:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:02:33.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:02:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:02:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:02:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:02:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:02:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:02:33.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:02:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:02:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:02:33.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:02:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:02:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:02:33.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:02:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:02:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:02:33.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:02:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:02:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:02:33.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:02:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:02:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:02:33.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:02:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:02:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:02:33.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:02:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:02:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:02:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:02:33.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:02:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:02:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:02:33.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:02:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:02:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:02:33.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:02:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:02:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:02:33.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:02:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:02:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1135] Local address: 172.21.15.161, port 33288 2022-07-03T21:02:33.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1135] Peer address: 172.21.15.101, port 22 2022-07-03T21:02:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1135] Connection lost 2022-07-03T21:02:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1135] Aborting connection 2022-07-03T21:02:33.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:33 smithi161 ceph-mon[91579]: pgmap v18701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:33 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:02:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:33 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:02:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:02:33.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:02:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:02:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:02:33.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:02:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:02:33.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:02:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:02:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:02:33.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:02:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:02:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:02:33.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:02:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:02:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:02:33.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:02:33.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:02:33.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:02:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:02:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:02:33.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:02:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:02:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:02:33.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:02:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:02:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:02:33.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:02:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:02:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:02:33.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:02:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:02:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:02:33.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:02:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:02:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:02:33.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:02:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:02:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:02:33.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:02:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:02:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:02:33.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:02:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:02:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:02:33.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1135] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:02:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1135] Local address: 172.21.15.161, port 33288 2022-07-03T21:02:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1135] Peer address: 172.21.15.101, port 22 2022-07-03T21:02:33.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1135] Connection lost 2022-07-03T21:02:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1135] Aborting connection 2022-07-03T21:02:33.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:33 smithi188 ceph-mon[92425]: pgmap v18701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: pgmap v18702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:35 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: pgmap v18702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:02:36.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:36.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:35 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:02:37.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:36 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 49150k 2022-07-03T21:02:37.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:36 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 50330443: error parsing value: Value '50330443' is below minimum 939524096 2022-07-03T21:02:37.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:36 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 49150k 2022-07-03T21:02:37.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:36 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 50330443: error parsing value: Value '50330443' is below minimum 939524096 2022-07-03T21:02:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:37 smithi161 ceph-mon[91579]: pgmap v18703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:37 smithi188 ceph-mon[92425]: pgmap v18703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:39.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:39.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:39 smithi161 ceph-mon[91579]: pgmap v18704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:39 smithi188 ceph-mon[92425]: pgmap v18704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:42.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:41 smithi161 ceph-mon[91579]: pgmap v18705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:41 smithi188 ceph-mon[92425]: pgmap v18705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:43 smithi161 ceph-mon[91579]: pgmap v18706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:43 smithi188 ceph-mon[92425]: pgmap v18706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:45 smithi161 ceph-mon[91579]: pgmap v18707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:46.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:45 smithi188 ceph-mon[92425]: pgmap v18707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:47 smithi161 ceph-mon[91579]: pgmap v18708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:47 smithi188 ceph-mon[92425]: pgmap v18708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:49.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:49 smithi161 ceph-mon[91579]: pgmap v18709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:49 smithi188 ceph-mon[92425]: pgmap v18709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:51.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:51.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:51 smithi161 ceph-mon[91579]: pgmap v18710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:51 smithi188 ceph-mon[92425]: pgmap v18710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:53 smithi161 ceph-mon[91579]: pgmap v18711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:53 smithi188 ceph-mon[92425]: pgmap v18711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:55 smithi161 ceph-mon[91579]: pgmap v18712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:56.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:55 smithi188 ceph-mon[92425]: pgmap v18712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:02:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:02:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:57 smithi161 ceph-mon[91579]: pgmap v18713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:57 smithi188 ceph-mon[92425]: pgmap v18713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:02:59.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:02:59.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:59 smithi161 ceph-mon[91579]: pgmap v18714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:02:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:59 smithi188 ceph-mon[92425]: pgmap v18714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:02:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:01.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:01 smithi161 ceph-mon[91579]: pgmap v18715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:01 smithi188 ceph-mon[92425]: pgmap v18715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:03 smithi161 ceph-mon[91579]: pgmap v18716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:03 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:03 smithi188 ceph-mon[92425]: pgmap v18716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:03 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:05 smithi161 ceph-mon[91579]: pgmap v18717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:05 smithi188 ceph-mon[92425]: pgmap v18717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:07 smithi161 ceph-mon[91579]: pgmap v18718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:07 smithi188 ceph-mon[92425]: pgmap v18718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:09.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:09 smithi161 ceph-mon[91579]: pgmap v18719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:09 smithi188 ceph-mon[92425]: pgmap v18719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:11.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:11 smithi161 ceph-mon[91579]: pgmap v18720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:11 smithi188 ceph-mon[92425]: pgmap v18720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:13.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:12 smithi161 ceph-mon[91579]: pgmap v18721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:13.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:12 smithi188 ceph-mon[92425]: pgmap v18721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:14 smithi161 ceph-mon[91579]: pgmap v18722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:14 smithi188 ceph-mon[92425]: pgmap v18722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:17.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:16 smithi161 ceph-mon[91579]: pgmap v18723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:17.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:16 smithi188 ceph-mon[92425]: pgmap v18723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:19 smithi161 ceph-mon[91579]: pgmap v18724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:19 smithi188 ceph-mon[92425]: pgmap v18724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:21 smithi161 ceph-mon[91579]: pgmap v18725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:21 smithi188 ceph-mon[92425]: pgmap v18725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:23 smithi161 ceph-mon[91579]: pgmap v18726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:23 smithi188 ceph-mon[92425]: pgmap v18726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:25 smithi161 ceph-mon[91579]: pgmap v18727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:25 smithi188 ceph-mon[92425]: pgmap v18727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:27 smithi161 ceph-mon[91579]: pgmap v18728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:27 smithi188 ceph-mon[92425]: pgmap v18728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:29 smithi161 ceph-mon[91579]: pgmap v18729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:29 smithi188 ceph-mon[92425]: pgmap v18729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:31 smithi161 ceph-mon[91579]: pgmap v18730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:31 smithi188 ceph-mon[92425]: pgmap v18730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:33 smithi161 ceph-mon[91579]: pgmap v18731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:33 smithi188 ceph-mon[92425]: pgmap v18731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-mon[91579]: pgmap v18732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:35.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:03:35.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:03:35.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:03:35.194+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:03:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:03:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:03:35.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:03:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:03:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:03:35.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:03:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:03:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:03:35.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:03:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:03:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:03:35.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:03:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:03:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:03:35.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:03:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:03:35.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:03:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:03:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:03:35.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:03:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:03:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:03:35.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:03:35.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:03:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:03:35.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:03:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:03:35.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:03:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:03:35.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:03:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:03:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:03:35.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:03:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:03:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:03:35.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:03:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:03:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:03:35.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:03:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:03:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:03:35.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:03:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:03:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:03:35.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1137] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:03:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1137] Local address: 172.21.15.161, port 33318 2022-07-03T21:03:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1137] Peer address: 172.21.15.101, port 22 2022-07-03T21:03:35.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1137] Connection lost 2022-07-03T21:03:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1137] Aborting connection 2022-07-03T21:03:35.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:03:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:35 smithi188 ceph-mon[92425]: pgmap v18732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:03:35.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:03:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:03:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:03:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:03:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:03:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:03:36.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:03:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:03:36.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:03:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:03:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:03:36.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:03:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:03:36.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:03:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:03:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:03:36.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:03:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:03:36.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:03:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:03:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:03:36.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:03:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:03:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:03:36.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:03:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:03:36.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:03:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:03:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:03:36.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:03:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:03:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:03:36.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:03:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:03:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:03:36.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:03:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:03:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:03:36.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:03:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:03:36.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:03:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:03:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:03:36.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:03:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:03:36.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:03:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:03:36.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1137] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:03:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1137] Local address: 172.21.15.161, port 33318 2022-07-03T21:03:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1137] Peer address: 172.21.15.101, port 22 2022-07-03T21:03:36.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1137] Connection lost 2022-07-03T21:03:36.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1137] Aborting connection 2022-07-03T21:03:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:03:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:03:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:03:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:03:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:03:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:03:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:03:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:03:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:03:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:03:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:03:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:03:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:03:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:03:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:03:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:03:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:03:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:03:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:03:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:03:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:03:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:03:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:03:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:03:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:03:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:03:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:03:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:03:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:03:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:03:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:03:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:03:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:03:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:03:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:03:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:03:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:03:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:03:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:03:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:03:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:03:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:03:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:03:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:03:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:03:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:03:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1137] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:03:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1137] Local address: 172.21.15.161, port 33318 2022-07-03T21:03:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1137] Peer address: 172.21.15.101, port 22 2022-07-03T21:03:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1137] Connection lost 2022-07-03T21:03:36.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1137] Aborting connection 2022-07-03T21:03:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:37 smithi161 ceph-mon[91579]: pgmap v18733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:37 smithi188 ceph-mon[92425]: pgmap v18733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:39 smithi161 ceph-mon[91579]: pgmap v18734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:39 smithi188 ceph-mon[92425]: pgmap v18734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:41 smithi161 ceph-mon[91579]: pgmap v18735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:41 smithi188 ceph-mon[92425]: pgmap v18735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:43 smithi161 ceph-mon[91579]: pgmap v18736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:43 smithi188 ceph-mon[92425]: pgmap v18736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:45 smithi161 ceph-mon[91579]: pgmap v18737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:45.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:45 smithi188 ceph-mon[92425]: pgmap v18737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:47 smithi161 ceph-mon[91579]: pgmap v18738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:47 smithi188 ceph-mon[92425]: pgmap v18738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:49 smithi161 ceph-mon[91579]: pgmap v18739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:49 smithi188 ceph-mon[92425]: pgmap v18739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:51 smithi161 ceph-mon[91579]: pgmap v18740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:51 smithi188 ceph-mon[92425]: pgmap v18740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:53 smithi161 ceph-mon[91579]: pgmap v18741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:53 smithi188 ceph-mon[92425]: pgmap v18741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:55 smithi161 ceph-mon[91579]: pgmap v18742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:55.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:55 smithi188 ceph-mon[92425]: pgmap v18742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:03:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:57 smithi161 ceph-mon[91579]: pgmap v18743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:57 smithi188 ceph-mon[92425]: pgmap v18743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:03:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: pgmap v18744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:03:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:59.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: pgmap v18744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:03:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:03:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:03:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:03:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:03:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:04:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:01 smithi161 ceph-mon[91579]: pgmap v18745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:01 smithi188 ceph-mon[92425]: pgmap v18745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:03 smithi161 ceph-mon[91579]: pgmap v18746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:03 smithi188 ceph-mon[92425]: pgmap v18746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:05 smithi161 ceph-mon[91579]: pgmap v18747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:05.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:05 smithi188 ceph-mon[92425]: pgmap v18747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:07 smithi161 ceph-mon[91579]: pgmap v18748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:07 smithi188 ceph-mon[92425]: pgmap v18748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:09 smithi161 ceph-mon[91579]: pgmap v18749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:09 smithi188 ceph-mon[92425]: pgmap v18749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:11 smithi161 ceph-mon[91579]: pgmap v18750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:11 smithi188 ceph-mon[92425]: pgmap v18750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:13 smithi161 ceph-mon[91579]: pgmap v18751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:13 smithi188 ceph-mon[92425]: pgmap v18751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:15 smithi161 ceph-mon[91579]: pgmap v18752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:15.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:15 smithi188 ceph-mon[92425]: pgmap v18752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:17 smithi161 ceph-mon[91579]: pgmap v18753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:17 smithi188 ceph-mon[92425]: pgmap v18753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:19 smithi161 ceph-mon[91579]: pgmap v18754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:19 smithi188 ceph-mon[92425]: pgmap v18754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:20.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:21 smithi161 ceph-mon[91579]: pgmap v18755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:21 smithi188 ceph-mon[92425]: pgmap v18755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:23 smithi161 ceph-mon[91579]: pgmap v18756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:23 smithi188 ceph-mon[92425]: pgmap v18756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:25 smithi161 ceph-mon[91579]: pgmap v18757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:25 smithi188 ceph-mon[92425]: pgmap v18757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:27 smithi161 ceph-mon[91579]: pgmap v18758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:27 smithi188 ceph-mon[92425]: pgmap v18758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:29 smithi161 ceph-mon[91579]: pgmap v18759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:29 smithi188 ceph-mon[92425]: pgmap v18759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:31 smithi161 ceph-mon[91579]: pgmap v18760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:31 smithi188 ceph-mon[92425]: pgmap v18760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:33 smithi161 ceph-mon[91579]: pgmap v18761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:33 smithi188 ceph-mon[92425]: pgmap v18761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:35 smithi188 ceph-mon[92425]: pgmap v18762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:35.698 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-mon[91579]: pgmap v18762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:35.699 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:36.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:04:35.698+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:04:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:04:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:04:36.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:04:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:04:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:04:36.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:04:36.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:04:36.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:04:36.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:04:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:04:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:04:36.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:04:36.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:04:36.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:04:36.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:04:36.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:04:36.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:04:36.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:04:36.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:04:36.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:04:36.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:04:36.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:04:36.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:04:36.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:04:36.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:04:36.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:04:36.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:04:36.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:04:36.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:04:36.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:04:36.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:04:36.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:04:36.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:04:36.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:04:36.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:04:36.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:04:36.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:04:36.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:04:36.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:04:36.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:04:36.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:04:36.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:04:36.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:04:36.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:04:36.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1139] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:04:36.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1139] Local address: 172.21.15.161, port 33346 2022-07-03T21:04:36.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1139] Peer address: 172.21.15.101, port 22 2022-07-03T21:04:36.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1139] Connection lost 2022-07-03T21:04:36.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1139] Aborting connection 2022-07-03T21:04:36.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:35 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:04:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:04:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:04:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:04:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:36 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:04:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:36 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:04:36.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:04:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:04:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:04:36.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:04:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:04:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:04:36.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:04:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:04:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:04:36.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:04:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:04:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:04:36.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:04:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:04:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:04:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:04:36.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:04:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:04:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:04:36.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:04:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:04:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:04:36.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:04:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:04:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:04:36.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:04:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:04:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:04:36.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:04:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:04:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:04:36.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:04:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:04:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:04:36.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:04:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:04:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:04:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:04:36.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:04:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:04:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:04:36.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:04:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:04:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:04:36.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1139] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:04:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1139] Local address: 172.21.15.161, port 33346 2022-07-03T21:04:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1139] Peer address: 172.21.15.101, port 22 2022-07-03T21:04:36.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1139] Connection lost 2022-07-03T21:04:36.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1139] Aborting connection 2022-07-03T21:04:36.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:04:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:04:36.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:04:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:36 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:04:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:36 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:04:36.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:04:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:04:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:04:36.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:04:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:04:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:04:36.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:04:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:04:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:04:36.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:04:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:04:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:04:36.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:04:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:04:36.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:04:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:04:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:04:36.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:04:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:04:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:04:36.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:04:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:04:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:04:36.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:04:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:04:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:04:36.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:04:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:04:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:04:36.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:04:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:04:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:04:36.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:04:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:04:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:04:36.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:04:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:04:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:04:36.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:04:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:04:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:04:36.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:04:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:04:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:04:36.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1139] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:04:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1139] Local address: 172.21.15.161, port 33346 2022-07-03T21:04:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1139] Peer address: 172.21.15.101, port 22 2022-07-03T21:04:36.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1139] Connection lost 2022-07-03T21:04:36.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1139] Aborting connection 2022-07-03T21:04:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:37 smithi188 ceph-mon[92425]: pgmap v18763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:37 smithi161 ceph-mon[91579]: pgmap v18763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:39 smithi188 ceph-mon[92425]: pgmap v18764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:39 smithi161 ceph-mon[91579]: pgmap v18764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:41 smithi188 ceph-mon[92425]: pgmap v18765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:41 smithi161 ceph-mon[91579]: pgmap v18765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:43 smithi188 ceph-mon[92425]: pgmap v18766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:43 smithi161 ceph-mon[91579]: pgmap v18766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:45 smithi161 ceph-mon[91579]: pgmap v18767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:45 smithi188 ceph-mon[92425]: pgmap v18767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:47 smithi161 ceph-mon[91579]: pgmap v18768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:47 smithi188 ceph-mon[92425]: pgmap v18768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:49 smithi161 ceph-mon[91579]: pgmap v18769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:49 smithi188 ceph-mon[92425]: pgmap v18769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:50.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:51 smithi161 ceph-mon[91579]: pgmap v18770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:51 smithi188 ceph-mon[92425]: pgmap v18770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:53 smithi161 ceph-mon[91579]: pgmap v18771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:53 smithi188 ceph-mon[92425]: pgmap v18771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:55.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:04:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:04:54.951+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T21:04:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:55 smithi161 ceph-mon[91579]: pgmap v18772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:55 smithi188 ceph-mon[92425]: pgmap v18772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:04:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:57 smithi161 ceph-mon[91579]: pgmap v18773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:57 smithi188 ceph-mon[92425]: pgmap v18773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:04:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: pgmap v18774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:04:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:04:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:04:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:04:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: pgmap v18774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:04:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:04:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:04:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:04:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:04:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:04:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:05:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:01 smithi161 ceph-mon[91579]: pgmap v18775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:01 smithi188 ceph-mon[92425]: pgmap v18775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:03 smithi161 ceph-mon[91579]: pgmap v18776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:03 smithi188 ceph-mon[92425]: pgmap v18776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:05 smithi161 ceph-mon[91579]: pgmap v18777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:05 smithi188 ceph-mon[92425]: pgmap v18777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:07 smithi161 ceph-mon[91579]: pgmap v18778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:07 smithi188 ceph-mon[92425]: pgmap v18778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:09 smithi161 ceph-mon[91579]: pgmap v18779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:09 smithi188 ceph-mon[92425]: pgmap v18779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:11 smithi161 ceph-mon[91579]: pgmap v18780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:11 smithi188 ceph-mon[92425]: pgmap v18780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:13 smithi161 ceph-mon[91579]: pgmap v18781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:13 smithi188 ceph-mon[92425]: pgmap v18781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:15 smithi161 ceph-mon[91579]: pgmap v18782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:15 smithi188 ceph-mon[92425]: pgmap v18782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:17 smithi161 ceph-mon[91579]: pgmap v18783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:17 smithi188 ceph-mon[92425]: pgmap v18783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:19 smithi161 ceph-mon[91579]: pgmap v18784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:19 smithi188 ceph-mon[92425]: pgmap v18784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:21 smithi161 ceph-mon[91579]: pgmap v18785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:21 smithi188 ceph-mon[92425]: pgmap v18785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:23 smithi161 ceph-mon[91579]: pgmap v18786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:23 smithi188 ceph-mon[92425]: pgmap v18786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:25 smithi161 ceph-mon[91579]: pgmap v18787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:25 smithi188 ceph-mon[92425]: pgmap v18787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:27 smithi161 ceph-mon[91579]: pgmap v18788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:27 smithi188 ceph-mon[92425]: pgmap v18788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:29 smithi161 ceph-mon[91579]: pgmap v18789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:29 smithi188 ceph-mon[92425]: pgmap v18789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:31 smithi161 ceph-mon[91579]: pgmap v18790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:31 smithi188 ceph-mon[92425]: pgmap v18790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:33 smithi161 ceph-mon[91579]: pgmap v18791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:33 smithi188 ceph-mon[92425]: pgmap v18791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:35 smithi161 ceph-mon[91579]: pgmap v18792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:35 smithi188 ceph-mon[92425]: pgmap v18792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:36.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:05:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:05:36.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:05:36.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:05:36.190+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:05:36.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:05:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:05:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:05:36.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:05:36.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:05:36.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:05:36.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:05:36.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:05:36.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:05:36.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:05:36.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:05:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:05:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:05:36.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:05:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:05:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:05:36.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:05:36.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:05:36.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:05:36.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:05:36.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:05:36.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:05:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:05:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:05:36.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:05:36.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:05:36.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:05:36.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:05:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:05:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:05:36.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:05:36.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:05:36.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:05:36.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:05:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:05:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:05:36.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:05:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:05:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:05:36.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:05:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:05:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:05:36.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:05:36.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:05:36.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:05:36.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1141] Local address: 172.21.15.161, port 33358 2022-07-03T21:05:36.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1141] Peer address: 172.21.15.101, port 22 2022-07-03T21:05:36.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1141] Connection lost 2022-07-03T21:05:36.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1141] Aborting connection 2022-07-03T21:05:36.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:05:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:05:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:05:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:05:36.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:36 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:05:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:05:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:05:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:05:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:05:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:05:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:05:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:05:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:05:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:05:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:05:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:05:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:05:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:05:37.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:05:37.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:05:37.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:05:37.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:05:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:05:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:05:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:05:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:05:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:05:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:05:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:05:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:05:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:05:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:05:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:05:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:05:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:05:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:05:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:05:37.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:05:37.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:05:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:05:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:05:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:05:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:05:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:05:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:05:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:05:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:05:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:05:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:05:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:05:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:05:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:05:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1141] Local address: 172.21.15.161, port 33358 2022-07-03T21:05:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1141] Peer address: 172.21.15.101, port 22 2022-07-03T21:05:37.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1141] Connection lost 2022-07-03T21:05:37.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1141] Aborting connection 2022-07-03T21:05:37.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:37 smithi161 ceph-mon[91579]: pgmap v18793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:05:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:05:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:05:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:05:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:05:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:05:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:05:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:05:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:05:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:05:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:05:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:05:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:05:37.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:05:37.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:05:37.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:05:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:05:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:05:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:05:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:05:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:05:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:05:37.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:05:37.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:05:37.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:05:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:05:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:05:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:05:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:05:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:05:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:05:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:05:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:05:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:05:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:05:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:05:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:05:37.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:05:37.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:05:37.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:05:37.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:05:37.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:05:37.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:05:37.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:05:37.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:05:37.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:05:37.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1141] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:05:37.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1141] Local address: 172.21.15.161, port 33358 2022-07-03T21:05:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1141] Peer address: 172.21.15.101, port 22 2022-07-03T21:05:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1141] Connection lost 2022-07-03T21:05:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1141] Aborting connection 2022-07-03T21:05:37.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:37 smithi188 ceph-mon[92425]: pgmap v18793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:39 smithi161 ceph-mon[91579]: pgmap v18794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:39 smithi188 ceph-mon[92425]: pgmap v18794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:41 smithi161 ceph-mon[91579]: pgmap v18795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:41 smithi188 ceph-mon[92425]: pgmap v18795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:43 smithi161 ceph-mon[91579]: pgmap v18796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:43 smithi188 ceph-mon[92425]: pgmap v18796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:45 smithi161 ceph-mon[91579]: pgmap v18797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:45 smithi188 ceph-mon[92425]: pgmap v18797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:47 smithi161 ceph-mon[91579]: pgmap v18798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:47 smithi188 ceph-mon[92425]: pgmap v18798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:49 smithi161 ceph-mon[91579]: pgmap v18799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:49 smithi188 ceph-mon[92425]: pgmap v18799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:51 smithi161 ceph-mon[91579]: pgmap v18800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:51 smithi188 ceph-mon[92425]: pgmap v18800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:53 smithi161 ceph-mon[91579]: pgmap v18801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:53 smithi188 ceph-mon[92425]: pgmap v18801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:55 smithi161 ceph-mon[91579]: pgmap v18802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:55 smithi188 ceph-mon[92425]: pgmap v18802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:57 smithi161 ceph-mon[91579]: pgmap v18803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:57 smithi188 ceph-mon[92425]: pgmap v18803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:05:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: pgmap v18804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:05:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:05:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:05:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:05:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:05:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: pgmap v18804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:05:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:05:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:05:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:05:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:05:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:05:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:06:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:01 smithi161 ceph-mon[91579]: pgmap v18805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:01 smithi188 ceph-mon[92425]: pgmap v18805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:03 smithi161 ceph-mon[91579]: pgmap v18806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:03 smithi188 ceph-mon[92425]: pgmap v18806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:05 smithi161 ceph-mon[91579]: pgmap v18807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:05 smithi188 ceph-mon[92425]: pgmap v18807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:07.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:07 smithi161 ceph-mon[91579]: pgmap v18808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:07 smithi188 ceph-mon[92425]: pgmap v18808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:09 smithi161 ceph-mon[91579]: pgmap v18809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:09 smithi188 ceph-mon[92425]: pgmap v18809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:11 smithi161 ceph-mon[91579]: pgmap v18810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:11 smithi188 ceph-mon[92425]: pgmap v18810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:13 smithi161 ceph-mon[91579]: pgmap v18811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:13 smithi188 ceph-mon[92425]: pgmap v18811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:15 smithi161 ceph-mon[91579]: pgmap v18812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:15 smithi188 ceph-mon[92425]: pgmap v18812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:17 smithi161 ceph-mon[91579]: pgmap v18813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:17 smithi188 ceph-mon[92425]: pgmap v18813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:19 smithi161 ceph-mon[91579]: pgmap v18814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:19 smithi188 ceph-mon[92425]: pgmap v18814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:21 smithi161 ceph-mon[91579]: pgmap v18815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:21 smithi188 ceph-mon[92425]: pgmap v18815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:23 smithi161 ceph-mon[91579]: pgmap v18816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:23 smithi188 ceph-mon[92425]: pgmap v18816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:25 smithi161 ceph-mon[91579]: pgmap v18817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:25 smithi188 ceph-mon[92425]: pgmap v18817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:27 smithi161 ceph-mon[91579]: pgmap v18818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:27 smithi188 ceph-mon[92425]: pgmap v18818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:29 smithi161 ceph-mon[91579]: pgmap v18819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:29 smithi188 ceph-mon[92425]: pgmap v18819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:31 smithi161 ceph-mon[91579]: pgmap v18820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:31 smithi188 ceph-mon[92425]: pgmap v18820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:33 smithi161 ceph-mon[91579]: pgmap v18821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:33 smithi188 ceph-mon[92425]: pgmap v18821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:35 smithi161 ceph-mon[91579]: pgmap v18822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:35 smithi188 ceph-mon[92425]: pgmap v18822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:37.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:06:36.681+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:06:37.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:06:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:06:37.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:06:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:06:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:06:37.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:06:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:06:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:06:37.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:06:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:06:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:06:37.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:06:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:06:37.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:06:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:06:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:06:37.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:06:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:06:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:06:37.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:06:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:06:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:06:37.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:06:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:06:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:06:37.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:06:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:06:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:06:37.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:06:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:06:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:06:37.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:06:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:06:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:06:37.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:06:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:06:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:06:37.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:06:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:06:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:06:37.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:06:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:06:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:06:37.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:06:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:06:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1143] Local address: 172.21.15.161, port 33362 2022-07-03T21:06:37.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1143] Peer address: 172.21.15.101, port 22 2022-07-03T21:06:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1143] Connection lost 2022-07-03T21:06:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1143] Aborting connection 2022-07-03T21:06:37.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:06:36 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:06:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: pgmap v18823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:06:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:06:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:06:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:06:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:37 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:06:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:06:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:06:37.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:06:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:06:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:06:37.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:06:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:06:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:06:37.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:06:37.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:06:37.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:06:37.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:06:37.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:06:37.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:06:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:06:37.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:06:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:06:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:06:37.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:06:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:06:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:06:37.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:06:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:06:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:06:37.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:06:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:06:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:06:37.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:06:37.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:06:37.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:06:37.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:06:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:06:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:06:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:06:37.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:06:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:06:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:06:37.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:06:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:06:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:06:37.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:06:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:06:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:06:37.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:06:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:06:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1143] Local address: 172.21.15.161, port 33362 2022-07-03T21:06:37.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1143] Peer address: 172.21.15.101, port 22 2022-07-03T21:06:37.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1143] Connection lost 2022-07-03T21:06:37.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1143] Aborting connection 2022-07-03T21:06:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: pgmap v18823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:06:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:06:37.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:06:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:06:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:37 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:06:37.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:06:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:06:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:06:37.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:06:37.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:06:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:06:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:06:37.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:06:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:06:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:06:37.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:06:37.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:06:37.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:06:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:06:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:06:37.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:06:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:06:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:06:37.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:06:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:06:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:06:37.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:06:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:06:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:06:37.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:06:37.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:06:37.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:06:37.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:06:37.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:06:37.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:06:37.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:06:37.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:06:37.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:06:37.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:06:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:06:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:06:37.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:06:37.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:06:37.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:06:37.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:06:37.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:06:37.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:06:37.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:06:37.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:06:37.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1143] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:06:37.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1143] Local address: 172.21.15.161, port 33362 2022-07-03T21:06:37.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1143] Peer address: 172.21.15.101, port 22 2022-07-03T21:06:37.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1143] Connection lost 2022-07-03T21:06:37.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1143] Aborting connection 2022-07-03T21:06:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:39 smithi161 ceph-mon[91579]: pgmap v18824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:39 smithi188 ceph-mon[92425]: pgmap v18824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:41 smithi161 ceph-mon[91579]: pgmap v18825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:41 smithi188 ceph-mon[92425]: pgmap v18825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:43 smithi161 ceph-mon[91579]: pgmap v18826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:43 smithi188 ceph-mon[92425]: pgmap v18826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:45 smithi161 ceph-mon[91579]: pgmap v18827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:45 smithi188 ceph-mon[92425]: pgmap v18827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:47 smithi161 ceph-mon[91579]: pgmap v18828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:47 smithi188 ceph-mon[92425]: pgmap v18828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:49 smithi161 ceph-mon[91579]: pgmap v18829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:49 smithi188 ceph-mon[92425]: pgmap v18829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:51 smithi161 ceph-mon[91579]: pgmap v18830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:51 smithi188 ceph-mon[92425]: pgmap v18830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:53 smithi161 ceph-mon[91579]: pgmap v18831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:53 smithi188 ceph-mon[92425]: pgmap v18831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:55 smithi161 ceph-mon[91579]: pgmap v18832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:55 smithi188 ceph-mon[92425]: pgmap v18832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:57 smithi161 ceph-mon[91579]: pgmap v18833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:57 smithi188 ceph-mon[92425]: pgmap v18833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:06:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: pgmap v18834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:06:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:06:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:06:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:06:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:06:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: pgmap v18834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:06:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:06:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:06:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:06:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:06:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:06:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:07:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:01 smithi161 ceph-mon[91579]: pgmap v18835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:01 smithi188 ceph-mon[92425]: pgmap v18835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:03 smithi161 ceph-mon[91579]: pgmap v18836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:03 smithi188 ceph-mon[92425]: pgmap v18836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:05 smithi161 ceph-mon[91579]: pgmap v18837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:05 smithi188 ceph-mon[92425]: pgmap v18837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:07 smithi161 ceph-mon[91579]: pgmap v18838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:07 smithi188 ceph-mon[92425]: pgmap v18838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:09 smithi161 ceph-mon[91579]: pgmap v18839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:09 smithi188 ceph-mon[92425]: pgmap v18839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:11 smithi161 ceph-mon[91579]: pgmap v18840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:11 smithi188 ceph-mon[92425]: pgmap v18840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:13 smithi161 ceph-mon[91579]: pgmap v18841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:13 smithi188 ceph-mon[92425]: pgmap v18841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:15 smithi161 ceph-mon[91579]: pgmap v18842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:15 smithi188 ceph-mon[92425]: pgmap v18842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:17 smithi161 ceph-mon[91579]: pgmap v18843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:17 smithi188 ceph-mon[92425]: pgmap v18843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:19 smithi161 ceph-mon[91579]: pgmap v18844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:19 smithi188 ceph-mon[92425]: pgmap v18844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:21 smithi161 ceph-mon[91579]: pgmap v18845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:21 smithi188 ceph-mon[92425]: pgmap v18845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:23 smithi161 ceph-mon[91579]: pgmap v18846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:23 smithi188 ceph-mon[92425]: pgmap v18846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:25 smithi161 ceph-mon[91579]: pgmap v18847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:25 smithi188 ceph-mon[92425]: pgmap v18847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:27 smithi161 ceph-mon[91579]: pgmap v18848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:27 smithi188 ceph-mon[92425]: pgmap v18848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:29 smithi161 ceph-mon[91579]: pgmap v18849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:29 smithi188 ceph-mon[92425]: pgmap v18849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:31 smithi161 ceph-mon[91579]: pgmap v18850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:31 smithi188 ceph-mon[92425]: pgmap v18850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:33 smithi161 ceph-mon[91579]: pgmap v18851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:33 smithi188 ceph-mon[92425]: pgmap v18851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:35 smithi161 ceph-mon[91579]: pgmap v18852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:35 smithi188 ceph-mon[92425]: pgmap v18852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-mon[91579]: pgmap v18853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:07:37.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:07:37.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:07:37.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:07:37.183+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:07:37.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:07:37.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:07:37.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:07:37.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:07:37.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:07:37.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:07:37.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:07:37.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:07:37.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:07:37.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:07:37.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:07:37.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:07:37.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:07:37.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:07:37.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:07:37.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:07:37.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:07:37.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:07:37.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:07:37.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:07:37.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:07:37.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:07:37.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:07:37.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:07:37.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:07:37.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:07:37.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:07:37.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:07:37.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:07:37.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:07:37.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:07:37.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:07:37.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:07:37.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:07:37.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:07:37.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:07:37.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:07:37.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:07:37.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:07:37.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:07:37.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:07:37.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:07:37.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:07:37.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:07:37.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:07:37.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1145] Local address: 172.21.15.161, port 33366 2022-07-03T21:07:37.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1145] Peer address: 172.21.15.101, port 22 2022-07-03T21:07:37.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1145] Connection lost 2022-07-03T21:07:37.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1145] Aborting connection 2022-07-03T21:07:37.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:07:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:07:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:37 smithi188 ceph-mon[92425]: pgmap v18853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:07:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:07:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:37 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:07:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:07:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:07:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:07:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:07:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:07:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:07:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:07:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:07:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:07:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:07:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:07:38.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:07:38.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:07:38.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:07:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:07:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:07:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:07:38.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:07:38.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:07:38.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:07:38.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:07:38.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:07:38.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:07:38.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:07:38.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:07:38.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:07:38.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:07:38.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:07:38.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:07:38.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:07:38.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:07:38.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:07:38.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:07:38.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:07:38.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:07:38.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:07:38.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:07:38.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:07:38.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:07:38.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:07:38.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:07:38.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:07:38.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:07:38.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:07:38.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:07:38.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:07:38.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:07:38.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1145] Local address: 172.21.15.161, port 33366 2022-07-03T21:07:38.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1145] Peer address: 172.21.15.101, port 22 2022-07-03T21:07:38.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1145] Connection lost 2022-07-03T21:07:38.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1145] Aborting connection 2022-07-03T21:07:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:07:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:07:38.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:07:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:07:38.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:07:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:07:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:07:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:07:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:07:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:07:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:07:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:07:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:07:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:07:38.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:07:38.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:07:38.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:07:38.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:07:38.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:07:38.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:07:38.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:07:38.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:07:38.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:07:38.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:07:38.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:07:38.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:07:38.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:07:38.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:07:38.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:07:38.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:07:38.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:07:38.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:07:38.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:07:38.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:07:38.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:07:38.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:07:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:07:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:07:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:07:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:07:38.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:07:38.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:07:38.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:07:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:07:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:07:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:07:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1145] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:07:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1145] Local address: 172.21.15.161, port 33366 2022-07-03T21:07:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1145] Peer address: 172.21.15.101, port 22 2022-07-03T21:07:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1145] Connection lost 2022-07-03T21:07:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1145] Aborting connection 2022-07-03T21:07:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:39 smithi161 ceph-mon[91579]: pgmap v18854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:39.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:39 smithi188 ceph-mon[92425]: pgmap v18854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:41 smithi161 ceph-mon[91579]: pgmap v18855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:41 smithi188 ceph-mon[92425]: pgmap v18855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:43 smithi161 ceph-mon[91579]: pgmap v18856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:43.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:43 smithi188 ceph-mon[92425]: pgmap v18856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:45 smithi161 ceph-mon[91579]: pgmap v18857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:45 smithi188 ceph-mon[92425]: pgmap v18857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:47 smithi161 ceph-mon[91579]: pgmap v18858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:47 smithi188 ceph-mon[92425]: pgmap v18858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:49 smithi161 ceph-mon[91579]: pgmap v18859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:49 smithi188 ceph-mon[92425]: pgmap v18859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:51 smithi161 ceph-mon[91579]: pgmap v18860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:51 smithi188 ceph-mon[92425]: pgmap v18860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:53 smithi161 ceph-mon[91579]: pgmap v18861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:53 smithi188 ceph-mon[92425]: pgmap v18861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:55 smithi161 ceph-mon[91579]: pgmap v18862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:55 smithi188 ceph-mon[92425]: pgmap v18862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:57 smithi161 ceph-mon[91579]: pgmap v18863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:57 smithi188 ceph-mon[92425]: pgmap v18863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:07:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: pgmap v18864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:07:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:07:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:07:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:07:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:07:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: pgmap v18864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:07:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:07:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:07:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:07:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:07:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:07:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:08:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:01 smithi161 ceph-mon[91579]: pgmap v18865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:01 smithi188 ceph-mon[92425]: pgmap v18865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:03.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:03 smithi161 ceph-mon[91579]: pgmap v18866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:03 smithi188 ceph-mon[92425]: pgmap v18866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:05 smithi161 ceph-mon[91579]: pgmap v18867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:05 smithi188 ceph-mon[92425]: pgmap v18867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:07 smithi161 ceph-mon[91579]: pgmap v18868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:07.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:07 smithi188 ceph-mon[92425]: pgmap v18868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:09 smithi161 ceph-mon[91579]: pgmap v18869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:09 smithi188 ceph-mon[92425]: pgmap v18869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:11 smithi161 ceph-mon[91579]: pgmap v18870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:11 smithi188 ceph-mon[92425]: pgmap v18870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:13 smithi161 ceph-mon[91579]: pgmap v18871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:13 smithi188 ceph-mon[92425]: pgmap v18871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:15 smithi161 ceph-mon[91579]: pgmap v18872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:15 smithi188 ceph-mon[92425]: pgmap v18872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:17 smithi161 ceph-mon[91579]: pgmap v18873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:17 smithi188 ceph-mon[92425]: pgmap v18873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:19 smithi161 ceph-mon[91579]: pgmap v18874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:19.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:19 smithi188 ceph-mon[92425]: pgmap v18874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:21 smithi161 ceph-mon[91579]: pgmap v18875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:21 smithi188 ceph-mon[92425]: pgmap v18875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:23 smithi161 ceph-mon[91579]: pgmap v18876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:23 smithi188 ceph-mon[92425]: pgmap v18876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:25 smithi161 ceph-mon[91579]: pgmap v18877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:25 smithi188 ceph-mon[92425]: pgmap v18877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:27 smithi161 ceph-mon[91579]: pgmap v18878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:27 smithi188 ceph-mon[92425]: pgmap v18878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:29 smithi161 ceph-mon[91579]: pgmap v18879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:29 smithi188 ceph-mon[92425]: pgmap v18879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:31 smithi161 ceph-mon[91579]: pgmap v18880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:31 smithi188 ceph-mon[92425]: pgmap v18880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:33 smithi161 ceph-mon[91579]: pgmap v18881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:33 smithi188 ceph-mon[92425]: pgmap v18881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:35 smithi161 ceph-mon[91579]: pgmap v18882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:35 smithi188 ceph-mon[92425]: pgmap v18882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-mon[91579]: pgmap v18883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:37 smithi188 ceph-mon[92425]: pgmap v18883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:38.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:08:37.676+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:08:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:08:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:08:38.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:08:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:08:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:08:38.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:08:38.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:08:38.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:08:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:08:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:08:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:08:38.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:08:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:08:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:08:38.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:08:38.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:08:38.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:08:38.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:08:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:08:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:08:38.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:08:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:08:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:08:38.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:08:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:08:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:08:38.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:08:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:08:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:08:38.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:08:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:08:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:08:38.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:08:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:08:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:08:38.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:08:38.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:08:38.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:08:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:08:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:08:38.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:08:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:08:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:08:38.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:08:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:08:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1147] Local address: 172.21.15.161, port 33370 2022-07-03T21:08:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1147] Peer address: 172.21.15.101, port 22 2022-07-03T21:08:38.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1147] Connection lost 2022-07-03T21:08:38.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1147] Aborting connection 2022-07-03T21:08:38.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:08:37 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:08:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:08:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:08:38.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:08:38.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:38 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:08:38.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:38 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:08:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:08:38.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:08:38.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:08:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:08:38.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:08:38.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:08:38.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:08:38.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:08:38.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:08:38.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:08:38.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:08:38.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:08:38.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:08:38.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:08:38.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:08:38.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:08:38.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:08:38.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:08:38.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:08:38.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:08:38.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:08:38.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:08:38.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:08:38.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:08:38.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:08:38.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:08:38.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:08:38.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:08:38.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:08:38.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:08:38.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:08:38.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:08:38.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:08:38.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:08:38.654 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:08:38.655 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:08:38.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:08:38.656 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:08:38.657 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:08:38.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:08:38.658 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:08:38.659 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:08:38.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:08:38.660 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:08:38.661 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:08:38.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1147] Local address: 172.21.15.161, port 33370 2022-07-03T21:08:38.662 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1147] Peer address: 172.21.15.101, port 22 2022-07-03T21:08:38.663 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1147] Connection lost 2022-07-03T21:08:38.664 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1147] Aborting connection 2022-07-03T21:08:38.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:08:38.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:08:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:08:38.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:38 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:08:38.668 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:38 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:08:38.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:08:38.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:08:38.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:08:38.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:08:38.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:08:38.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:08:38.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:08:38.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:08:38.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:08:38.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:08:38.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:08:38.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:08:38.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:08:38.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:08:38.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:08:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:08:38.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:08:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:08:38.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:08:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:08:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:08:38.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:08:38.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:08:38.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:08:38.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:08:38.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:08:38.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:08:38.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:08:38.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:08:38.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:08:38.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:08:38.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:08:38.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:08:38.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:08:38.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:08:38.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:08:38.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:08:38.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:08:38.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:08:38.687 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:08:38.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:08:38.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:08:38.688 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:08:38.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:08:38.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1147] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:08:38.689 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1147] Local address: 172.21.15.161, port 33370 2022-07-03T21:08:38.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1147] Peer address: 172.21.15.101, port 22 2022-07-03T21:08:38.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1147] Connection lost 2022-07-03T21:08:38.690 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1147] Aborting connection 2022-07-03T21:08:39.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:39 smithi161 ceph-mon[91579]: pgmap v18884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:39.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:39 smithi188 ceph-mon[92425]: pgmap v18884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:41 smithi161 ceph-mon[91579]: pgmap v18885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:41 smithi188 ceph-mon[92425]: pgmap v18885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:43 smithi161 ceph-mon[91579]: pgmap v18886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:43 smithi188 ceph-mon[92425]: pgmap v18886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:44.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:45 smithi161 ceph-mon[91579]: pgmap v18887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:45.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:45 smithi188 ceph-mon[92425]: pgmap v18887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:47 smithi161 ceph-mon[91579]: pgmap v18888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:47 smithi188 ceph-mon[92425]: pgmap v18888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:49 smithi161 ceph-mon[91579]: pgmap v18889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:49 smithi188 ceph-mon[92425]: pgmap v18889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:51 smithi161 ceph-mon[91579]: pgmap v18890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:51 smithi188 ceph-mon[92425]: pgmap v18890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:53 smithi161 ceph-mon[91579]: pgmap v18891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:53 smithi188 ceph-mon[92425]: pgmap v18891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:55 smithi161 ceph-mon[91579]: pgmap v18892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:55 smithi188 ceph-mon[92425]: pgmap v18892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:57 smithi161 ceph-mon[91579]: pgmap v18893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:57 smithi188 ceph-mon[92425]: pgmap v18893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:08:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: pgmap v18894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:08:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:08:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:08:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:08:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:08:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: pgmap v18894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:08:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:08:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:08:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:08:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:08:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:08:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:09:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:01 smithi161 ceph-mon[91579]: pgmap v18895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:01 smithi188 ceph-mon[92425]: pgmap v18895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:03 smithi161 ceph-mon[91579]: pgmap v18896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:03 smithi188 ceph-mon[92425]: pgmap v18896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:04.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:04.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:05.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:05 smithi161 ceph-mon[91579]: pgmap v18897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:05 smithi188 ceph-mon[92425]: pgmap v18897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:07.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:07 smithi161 ceph-mon[91579]: pgmap v18898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:07 smithi188 ceph-mon[92425]: pgmap v18898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:09.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:09 smithi161 ceph-mon[91579]: pgmap v18899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:09.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:09 smithi188 ceph-mon[92425]: pgmap v18899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:11 smithi161 ceph-mon[91579]: pgmap v18900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:11 smithi188 ceph-mon[92425]: pgmap v18900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:13.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:13 smithi161 ceph-mon[91579]: pgmap v18901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:13.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:13 smithi188 ceph-mon[92425]: pgmap v18901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:15 smithi161 ceph-mon[91579]: pgmap v18902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:15 smithi188 ceph-mon[92425]: pgmap v18902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:17 smithi161 ceph-mon[91579]: pgmap v18903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:17.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:17 smithi188 ceph-mon[92425]: pgmap v18903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:19 smithi161 ceph-mon[91579]: pgmap v18904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:19.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:19 smithi188 ceph-mon[92425]: pgmap v18904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:21 smithi161 ceph-mon[91579]: pgmap v18905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:21 smithi188 ceph-mon[92425]: pgmap v18905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:23 smithi161 ceph-mon[91579]: pgmap v18906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:23.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:23 smithi188 ceph-mon[92425]: pgmap v18906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:25 smithi161 ceph-mon[91579]: pgmap v18907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:25 smithi188 ceph-mon[92425]: pgmap v18907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:27 smithi161 ceph-mon[91579]: pgmap v18908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:27 smithi188 ceph-mon[92425]: pgmap v18908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:29 smithi161 ceph-mon[91579]: pgmap v18909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:29 smithi188 ceph-mon[92425]: pgmap v18909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:31 smithi161 ceph-mon[91579]: pgmap v18910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:31 smithi188 ceph-mon[92425]: pgmap v18910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:33.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:33 smithi161 ceph-mon[91579]: pgmap v18911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:33 smithi188 ceph-mon[92425]: pgmap v18911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:35 smithi161 ceph-mon[91579]: pgmap v18912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:35 smithi188 ceph-mon[92425]: pgmap v18912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:37.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:37 smithi161 ceph-mon[91579]: pgmap v18913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:37.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:37 smithi188 ceph-mon[92425]: pgmap v18913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:38.517 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:09:38.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:09:38.518 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:09:38.519 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:09:38.174+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:09:38.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:09:38.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:09:38.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:09:38.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:09:38.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:09:38.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:09:38.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:09:38.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:09:38.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:09:38.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:09:38.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:09:38.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:09:38.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:09:38.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:09:38.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:09:38.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:09:38.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:09:38.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:09:38.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:09:38.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:09:38.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:09:38.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:09:38.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:09:38.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:09:38.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:09:38.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:09:38.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:09:38.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:09:38.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:09:38.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:09:38.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:09:38.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:09:38.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:09:38.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:09:38.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:09:38.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:09:38.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:09:38.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:09:38.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:09:38.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:09:38.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:09:38.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:09:38.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:09:38.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:09:38.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:09:38.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1149] Local address: 172.21.15.161, port 33400 2022-07-03T21:09:38.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1149] Peer address: 172.21.15.101, port 22 2022-07-03T21:09:38.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1149] Connection lost 2022-07-03T21:09:38.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1149] Aborting connection 2022-07-03T21:09:38.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:09:38 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:09:38.625 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:09:38.626 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:09:38.626 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:38 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:09:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:09:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:09:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:09:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:09:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:09:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:09:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:09:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:09:39.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:09:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:09:39.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:09:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:09:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:09:39.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:09:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:09:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:09:39.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:09:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:09:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:09:39.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:09:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:09:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:09:39.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:09:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:09:39.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:09:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:09:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:09:39.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:09:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:09:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:09:39.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:09:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:09:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:09:39.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:09:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:09:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:09:39.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:09:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:09:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:09:39.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:09:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:09:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:09:39.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:09:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:09:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:09:39.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:09:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:09:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1149] Local address: 172.21.15.161, port 33400 2022-07-03T21:09:39.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1149] Peer address: 172.21.15.101, port 22 2022-07-03T21:09:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1149] Connection lost 2022-07-03T21:09:39.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1149] Aborting connection 2022-07-03T21:09:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: pgmap v18914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:39.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:39 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:09:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:09:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:09:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:09:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:09:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:09:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:09:39.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:09:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:09:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:09:39.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:09:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:09:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:09:39.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:09:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:09:39.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:09:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:09:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:09:39.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:09:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:09:39.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:09:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:09:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:09:39.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:09:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:09:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:09:39.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:09:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:09:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:09:39.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:09:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:09:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:09:39.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:09:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:09:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:09:39.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:09:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:09:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:09:39.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:09:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:09:39.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:09:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:09:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:09:39.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:09:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:09:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:09:39.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1149] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:09:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1149] Local address: 172.21.15.161, port 33400 2022-07-03T21:09:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1149] Peer address: 172.21.15.101, port 22 2022-07-03T21:09:39.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1149] Connection lost 2022-07-03T21:09:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1149] Aborting connection 2022-07-03T21:09:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: pgmap v18914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:39.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:39.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:39.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:39 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:09:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:41 smithi161 ceph-mon[91579]: pgmap v18915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:41 smithi188 ceph-mon[92425]: pgmap v18915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:43 smithi161 ceph-mon[91579]: pgmap v18916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:43 smithi188 ceph-mon[92425]: pgmap v18916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:45.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:45 smithi161 ceph-mon[91579]: pgmap v18917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:45 smithi188 ceph-mon[92425]: pgmap v18917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:47.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:47 smithi161 ceph-mon[91579]: pgmap v18918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:47.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:47 smithi188 ceph-mon[92425]: pgmap v18918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:49.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:49 smithi161 ceph-mon[91579]: pgmap v18919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:49.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:49.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:49 smithi188 ceph-mon[92425]: pgmap v18919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:51 smithi161 ceph-mon[91579]: pgmap v18920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:51 smithi188 ceph-mon[92425]: pgmap v18920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:53 smithi161 ceph-mon[91579]: pgmap v18921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:53 smithi188 ceph-mon[92425]: pgmap v18921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:54.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:55.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:55 smithi161 ceph-mon[91579]: pgmap v18922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:55 smithi188 ceph-mon[92425]: pgmap v18922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:57.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:57 smithi161 ceph-mon[91579]: pgmap v18923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:57.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:57 smithi188 ceph-mon[92425]: pgmap v18923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:09:59.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: pgmap v18924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:59.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:09:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:09:59.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:09:59.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:09:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:09:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: pgmap v18924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:09:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:09:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:09:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:09:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:09:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:09:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:10:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:10:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:10:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:01 smithi161 ceph-mon[91579]: pgmap v18925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:01 smithi188 ceph-mon[92425]: pgmap v18925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:03.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:03 smithi161 ceph-mon[91579]: pgmap v18926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:03 smithi188 ceph-mon[92425]: pgmap v18926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:05 smithi188 ceph-mon[92425]: pgmap v18927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:05 smithi161 ceph-mon[91579]: pgmap v18927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:07 smithi188 ceph-mon[92425]: pgmap v18928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:07.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:07 smithi161 ceph-mon[91579]: pgmap v18928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:09 smithi188 ceph-mon[92425]: pgmap v18929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:09 smithi161 ceph-mon[91579]: pgmap v18929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:11 smithi161 ceph-mon[91579]: pgmap v18930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:11 smithi188 ceph-mon[92425]: pgmap v18930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:13.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:13 smithi161 ceph-mon[91579]: pgmap v18931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:13 smithi188 ceph-mon[92425]: pgmap v18931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:15 smithi161 ceph-mon[91579]: pgmap v18932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:15 smithi188 ceph-mon[92425]: pgmap v18932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:17.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:17 smithi161 ceph-mon[91579]: pgmap v18933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:17 smithi188 ceph-mon[92425]: pgmap v18933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:19.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:19 smithi161 ceph-mon[91579]: pgmap v18934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:19.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:19 smithi188 ceph-mon[92425]: pgmap v18934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:21 smithi161 ceph-mon[91579]: pgmap v18935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:21 smithi188 ceph-mon[92425]: pgmap v18935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:23.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:23 smithi161 ceph-mon[91579]: pgmap v18936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:23 smithi188 ceph-mon[92425]: pgmap v18936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:25 smithi161 ceph-mon[91579]: pgmap v18937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:25 smithi188 ceph-mon[92425]: pgmap v18937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:27.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:27 smithi161 ceph-mon[91579]: pgmap v18938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:27.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:27 smithi188 ceph-mon[92425]: pgmap v18938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:29 smithi188 ceph-mon[92425]: pgmap v18939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:29 smithi161 ceph-mon[91579]: pgmap v18939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:31 smithi161 ceph-mon[91579]: pgmap v18940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:31 smithi188 ceph-mon[92425]: pgmap v18940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:33.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:33 smithi161 ceph-mon[91579]: pgmap v18941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:33.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:33 smithi188 ceph-mon[92425]: pgmap v18941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:35 smithi161 ceph-mon[91579]: pgmap v18942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:35 smithi188 ceph-mon[92425]: pgmap v18942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:37.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:37 smithi188 ceph-mon[92425]: pgmap v18943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:37 smithi161 ceph-mon[91579]: pgmap v18943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:39.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:10:39.096+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:10:39.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:10:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:10:39.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:10:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:10:39.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:10:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:10:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:10:39.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:10:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:10:39.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:10:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:10:39.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:10:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:10:39.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:10:39.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:10:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:10:39.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:10:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:10:39.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:10:39.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:10:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:10:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:10:39.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:10:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:10:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:10:39.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:10:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:10:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:10:39.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:10:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:10:39.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:10:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:10:39.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:10:39.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:10:39.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:10:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:10:39.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:10:39.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:10:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:10:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1151] Local address: 172.21.15.161, port 33404 2022-07-03T21:10:39.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1151] Peer address: 172.21.15.101, port 22 2022-07-03T21:10:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1151] Connection lost 2022-07-03T21:10:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1151] Aborting connection 2022-07-03T21:10:39.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:10:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:39 smithi188 ceph-mon[92425]: pgmap v18944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:39.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:10:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:10:39.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:10:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-mon[91579]: pgmap v18944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:10:39.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:10:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:10:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:10:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:10:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:10:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:10:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:10:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:10:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:10:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:10:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:10:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:10:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:10:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:10:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:10:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:10:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:10:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:10:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:10:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:10:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:10:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:10:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:10:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:10:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:10:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:10:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:10:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:10:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:10:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:10:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:10:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:10:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:10:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:10:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:10:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:10:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:10:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:10:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:10:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:10:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:10:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:10:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:10:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:10:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:10:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:10:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:10:40.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:10:40.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1151] Local address: 172.21.15.161, port 33404 2022-07-03T21:10:40.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1151] Peer address: 172.21.15.101, port 22 2022-07-03T21:10:40.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1151] Connection lost 2022-07-03T21:10:40.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1151] Aborting connection 2022-07-03T21:10:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:10:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:10:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:10:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:10:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:10:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:10:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:10:40.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:10:40.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:10:40.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:10:40.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:10:40.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:10:40.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:10:40.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:10:40.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:10:40.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:10:40.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:10:40.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:10:40.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:10:40.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:10:40.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:10:40.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:10:40.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:10:40.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:10:40.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:10:40.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:10:40.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:10:40.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:10:40.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:10:40.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:10:40.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:10:40.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:10:40.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:10:40.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:10:40.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:10:40.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:10:40.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:10:40.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:10:40.899 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:10:40.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:10:40.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:10:40.900 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:10:40.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:10:40.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:10:40.901 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:10:40.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:10:40.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1151] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:10:40.902 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1151] Local address: 172.21.15.161, port 33404 2022-07-03T21:10:40.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1151] Peer address: 172.21.15.101, port 22 2022-07-03T21:10:40.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1151] Connection lost 2022-07-03T21:10:40.903 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1151] Aborting connection 2022-07-03T21:10:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:41 smithi188 ceph-mon[92425]: pgmap v18945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:41 smithi161 ceph-mon[91579]: pgmap v18945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:43 smithi188 ceph-mon[92425]: pgmap v18946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:43 smithi161 ceph-mon[91579]: pgmap v18946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:45 smithi188 ceph-mon[92425]: pgmap v18947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:45 smithi161 ceph-mon[91579]: pgmap v18947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:47.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:47 smithi188 ceph-mon[92425]: pgmap v18948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:47 smithi161 ceph-mon[91579]: pgmap v18948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:49 smithi188 ceph-mon[92425]: pgmap v18949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:49.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:49 smithi161 ceph-mon[91579]: pgmap v18949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:51 smithi161 ceph-mon[91579]: pgmap v18950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:51 smithi188 ceph-mon[92425]: pgmap v18950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:53.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:53 smithi161 ceph-mon[91579]: pgmap v18951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:53 smithi188 ceph-mon[92425]: pgmap v18951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:55 smithi188 ceph-mon[92425]: pgmap v18952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:55 smithi161 ceph-mon[91579]: pgmap v18952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:57 smithi188 ceph-mon[92425]: pgmap v18953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:57 smithi161 ceph-mon[91579]: pgmap v18953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:10:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: pgmap v18954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:10:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:10:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:10:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:10:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:10:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: pgmap v18954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:10:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:10:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:10:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:10:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:10:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:10:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:11:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:01 smithi161 ceph-mon[91579]: pgmap v18955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:01 smithi188 ceph-mon[92425]: pgmap v18955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:03.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:03 smithi188 ceph-mon[92425]: pgmap v18956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:03 smithi161 ceph-mon[91579]: pgmap v18956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:05 smithi188 ceph-mon[92425]: pgmap v18957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:05 smithi161 ceph-mon[91579]: pgmap v18957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:07 smithi188 ceph-mon[92425]: pgmap v18958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:07 smithi161 ceph-mon[91579]: pgmap v18958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:09 smithi188 ceph-mon[92425]: pgmap v18959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:09 smithi161 ceph-mon[91579]: pgmap v18959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:11 smithi161 ceph-mon[91579]: pgmap v18960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:11 smithi188 ceph-mon[92425]: pgmap v18960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:13 smithi188 ceph-mon[92425]: pgmap v18961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:13 smithi161 ceph-mon[91579]: pgmap v18961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:14.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:15 smithi188 ceph-mon[92425]: pgmap v18962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:15 smithi161 ceph-mon[91579]: pgmap v18962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:17 smithi188 ceph-mon[92425]: pgmap v18963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:17 smithi161 ceph-mon[91579]: pgmap v18963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:19.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:19 smithi188 ceph-mon[92425]: pgmap v18964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:19.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:19 smithi161 ceph-mon[91579]: pgmap v18964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:21 smithi161 ceph-mon[91579]: pgmap v18965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:21 smithi188 ceph-mon[92425]: pgmap v18965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:23 smithi188 ceph-mon[92425]: pgmap v18966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:23 smithi161 ceph-mon[91579]: pgmap v18966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:25 smithi188 ceph-mon[92425]: pgmap v18967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:25 smithi161 ceph-mon[91579]: pgmap v18967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:27.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:27 smithi188 ceph-mon[92425]: pgmap v18968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:27 smithi161 ceph-mon[91579]: pgmap v18968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:29 smithi188 ceph-mon[92425]: pgmap v18969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:29 smithi161 ceph-mon[91579]: pgmap v18969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:31 smithi161 ceph-mon[91579]: pgmap v18970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:31 smithi188 ceph-mon[92425]: pgmap v18970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:33 smithi188 ceph-mon[92425]: pgmap v18971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:33 smithi161 ceph-mon[91579]: pgmap v18971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:34.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:35.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:35 smithi188 ceph-mon[92425]: pgmap v18972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:35 smithi161 ceph-mon[91579]: pgmap v18972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:37.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:37 smithi188 ceph-mon[92425]: pgmap v18973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:37 smithi161 ceph-mon[91579]: pgmap v18973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:39.660 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:39 smithi188 ceph-mon[92425]: pgmap v18974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:39.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:39.666 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-mon[91579]: pgmap v18974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:39.667 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:39.667 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:11:39.600+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:11:39.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:11:39.668 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:11:39.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:11:39.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:11:39.669 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:11:39.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:11:39.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:11:39.670 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:11:39.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:11:39.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:11:39.671 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:11:39.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:11:39.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:11:39.672 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:11:39.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:11:39.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:11:39.673 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:11:39.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:11:39.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:11:39.674 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:11:39.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:11:39.675 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:11:39.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:11:39.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:11:39.676 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:11:39.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:11:39.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:11:39.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:11:39.677 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:11:39.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:11:39.678 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:11:39.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:11:39.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:11:39.679 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:11:39.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:11:39.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:11:39.680 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:11:39.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:11:39.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:11:39.681 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:11:39.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:11:39.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:11:39.682 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:11:39.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:11:39.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:11:39.683 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1153] Local address: 172.21.15.161, port 33408 2022-07-03T21:11:39.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1153] Peer address: 172.21.15.101, port 22 2022-07-03T21:11:39.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1153] Connection lost 2022-07-03T21:11:39.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1153] Aborting connection 2022-07-03T21:11:39.684 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:11:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:11:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:11:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:11:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:11:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:40 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:11:40.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:40 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:11:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:11:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:11:40.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:11:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:11:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:11:40.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:11:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:11:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:11:40.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:11:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:11:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:11:40.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:11:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:11:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:11:40.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:11:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:11:40.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:11:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:11:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:11:40.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:11:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:11:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:11:40.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:11:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:11:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:11:40.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:11:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:11:40.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:11:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:11:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:11:40.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:11:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:11:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:11:40.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:11:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:11:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:11:40.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:11:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:11:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:11:40.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:11:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:11:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:11:40.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:11:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:11:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:11:40.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1153] Local address: 172.21.15.161, port 33408 2022-07-03T21:11:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1153] Peer address: 172.21.15.101, port 22 2022-07-03T21:11:40.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1153] Connection lost 2022-07-03T21:11:40.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1153] Aborting connection 2022-07-03T21:11:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:11:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:11:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:11:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:40 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:11:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:40 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:11:40.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:11:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:11:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:11:40.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:11:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:11:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:11:40.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:11:40.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:11:40.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:11:40.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:11:40.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:11:40.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:11:40.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:11:40.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:11:40.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:11:40.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:11:40.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:11:40.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:11:40.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:11:40.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:11:40.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:11:40.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:11:40.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:11:40.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:11:40.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:11:40.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:11:40.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:11:40.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:11:40.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:11:40.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:11:40.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:11:40.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:11:40.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:11:40.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:11:40.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:11:40.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:11:40.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:11:40.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:11:40.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:11:40.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:11:40.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:11:40.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:11:40.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:11:40.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:11:40.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1153] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:11:40.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1153] Local address: 172.21.15.161, port 33408 2022-07-03T21:11:40.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1153] Peer address: 172.21.15.101, port 22 2022-07-03T21:11:40.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1153] Connection lost 2022-07-03T21:11:40.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1153] Aborting connection 2022-07-03T21:11:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:41 smithi188 ceph-mon[92425]: pgmap v18975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:41 smithi161 ceph-mon[91579]: pgmap v18975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:43 smithi188 ceph-mon[92425]: pgmap v18976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:43 smithi161 ceph-mon[91579]: pgmap v18976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:45 smithi161 ceph-mon[91579]: pgmap v18977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:45 smithi188 ceph-mon[92425]: pgmap v18977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:47 smithi161 ceph-mon[91579]: pgmap v18978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:47 smithi188 ceph-mon[92425]: pgmap v18978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:49 smithi161 ceph-mon[91579]: pgmap v18979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:49 smithi188 ceph-mon[92425]: pgmap v18979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:51 smithi161 ceph-mon[91579]: pgmap v18980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:51 smithi188 ceph-mon[92425]: pgmap v18980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:53.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:53 smithi188 ceph-mon[92425]: pgmap v18981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:53 smithi161 ceph-mon[91579]: pgmap v18981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:54.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:55.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:55 smithi188 ceph-mon[92425]: pgmap v18982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:55 smithi161 ceph-mon[91579]: pgmap v18982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:57.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:57 smithi188 ceph-mon[92425]: pgmap v18983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:57 smithi161 ceph-mon[91579]: pgmap v18983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:11:59.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: pgmap v18984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:59.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:11:59.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:11:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:11:59.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:11:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:11:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: pgmap v18984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:11:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:11:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:11:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:11:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:11:59.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:11:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:12:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:01 smithi161 ceph-mon[91579]: pgmap v18985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:01 smithi188 ceph-mon[92425]: pgmap v18985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:03.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:03 smithi188 ceph-mon[92425]: pgmap v18986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:03 smithi161 ceph-mon[91579]: pgmap v18986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:05.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:05 smithi188 ceph-mon[92425]: pgmap v18987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:05 smithi161 ceph-mon[91579]: pgmap v18987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:07.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:07 smithi188 ceph-mon[92425]: pgmap v18988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:07 smithi161 ceph-mon[91579]: pgmap v18988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:09.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:09 smithi188 ceph-mon[92425]: pgmap v18989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:09.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:09 smithi161 ceph-mon[91579]: pgmap v18989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:11 smithi161 ceph-mon[91579]: pgmap v18990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:11 smithi188 ceph-mon[92425]: pgmap v18990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:13.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:13 smithi188 ceph-mon[92425]: pgmap v18991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:13 smithi161 ceph-mon[91579]: pgmap v18991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:15.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:15 smithi188 ceph-mon[92425]: pgmap v18992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:15 smithi161 ceph-mon[91579]: pgmap v18992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:17.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:17 smithi188 ceph-mon[92425]: pgmap v18993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:17 smithi161 ceph-mon[91579]: pgmap v18993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:19 smithi161 ceph-mon[91579]: pgmap v18994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:19 smithi188 ceph-mon[92425]: pgmap v18994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:21 smithi161 ceph-mon[91579]: pgmap v18995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:21 smithi188 ceph-mon[92425]: pgmap v18995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:23.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:23 smithi188 ceph-mon[92425]: pgmap v18996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:23 smithi161 ceph-mon[91579]: pgmap v18996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:12:24.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:25 smithi188 ceph-mon[92425]: pgmap v18997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:25 smithi161 ceph-mon[91579]: pgmap v18997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:27 smithi161 ceph-mon[91579]: pgmap v18998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:27 smithi188 ceph-mon[92425]: pgmap v18998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:29 smithi161 ceph-mon[91579]: pgmap v18999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:29 smithi188 ceph-mon[92425]: pgmap v18999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:31 smithi161 ceph-mon[91579]: pgmap v19000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:31 smithi188 ceph-mon[92425]: pgmap v19000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:33.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:33 smithi188 ceph-mon[92425]: pgmap v19001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:33 smithi161 ceph-mon[91579]: pgmap v19001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:35 smithi161 ceph-mon[91579]: pgmap v19002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:35 smithi188 ceph-mon[92425]: pgmap v19002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:37 smithi161 ceph-mon[91579]: pgmap v19003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:37 smithi188 ceph-mon[92425]: pgmap v19003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:39 smithi161 ceph-mon[91579]: pgmap v19004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:39 smithi188 ceph-mon[92425]: pgmap v19004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:40.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:12:40.099+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:12:40.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:12:40.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:12:40.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:12:40.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:12:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:12:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:12:40.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:12:40.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:12:40.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:12:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:12:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:12:40.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:12:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:12:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:12:40.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:12:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:12:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:12:40.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:12:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:12:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:12:40.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:12:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:12:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:12:40.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:12:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:12:40.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:12:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:12:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:12:40.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:12:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:12:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:12:40.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:12:40.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:12:40.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:12:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:12:40.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:12:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:12:40.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:12:40.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:12:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:12:40.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:12:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:12:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:12:40.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:12:40.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:12:40.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1155] Local address: 172.21.15.161, port 33412 2022-07-03T21:12:40.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1155] Peer address: 172.21.15.101, port 22 2022-07-03T21:12:40.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1155] Connection lost 2022-07-03T21:12:40.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1155] Aborting connection 2022-07-03T21:12:40.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:12:40.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:12:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:12:40.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:12:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:12:40.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:12:40.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:12:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:41 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:12:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:41 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:12:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:12:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:12:41.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:12:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:12:41.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:12:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:12:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:12:41.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:12:41.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:12:41.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:12:41.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:12:41.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:12:41.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:12:41.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:12:41.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:12:41.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:12:41.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:12:41.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:12:41.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:12:41.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:12:41.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:12:41.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:12:41.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:12:41.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:12:41.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:12:41.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:12:41.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:12:41.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:12:41.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:12:41.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:12:41.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:12:41.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:12:41.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:12:41.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:12:41.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:12:41.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:12:41.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:12:41.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:12:41.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:12:41.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:12:41.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:12:41.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:12:41.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:12:41.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:12:41.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:12:41.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1155] Local address: 172.21.15.161, port 33412 2022-07-03T21:12:41.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1155] Peer address: 172.21.15.101, port 22 2022-07-03T21:12:41.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1155] Connection lost 2022-07-03T21:12:41.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1155] Aborting connection 2022-07-03T21:12:41.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:41 smithi161 ceph-mon[91579]: pgmap v19005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:41 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:12:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:41 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:12:41.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:12:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:12:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:12:41.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:12:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:12:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:12:41.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:12:41.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:12:41.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:12:41.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:12:41.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:12:41.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:12:41.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:12:41.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:12:41.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:12:41.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:12:41.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:12:41.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:12:41.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:12:41.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:12:41.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:12:41.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:12:41.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:12:41.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:12:41.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:12:41.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:12:41.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:12:41.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:12:41.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:12:41.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:12:41.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:12:41.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:12:41.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:12:41.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:12:41.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:12:41.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:12:41.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:12:41.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:12:41.933 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:12:41.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:12:41.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:12:41.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:12:41.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:12:41.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:12:41.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1155] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:12:41.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1155] Local address: 172.21.15.161, port 33412 2022-07-03T21:12:41.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1155] Peer address: 172.21.15.101, port 22 2022-07-03T21:12:41.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1155] Connection lost 2022-07-03T21:12:41.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1155] Aborting connection 2022-07-03T21:12:41.937 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:41 smithi188 ceph-mon[92425]: pgmap v19005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: pgmap v19006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.134 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.135 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.136 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.137 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.138 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.139 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: pgmap v19006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:12:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.169 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.170 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:12:44.171 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:44 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 43383k 2022-07-03T21:12:45.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:44 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 44424863: error parsing value: Value '44424863' is below minimum 939524096 2022-07-03T21:12:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:44 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 43383k 2022-07-03T21:12:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:44 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 44424863: error parsing value: Value '44424863' is below minimum 939524096 2022-07-03T21:12:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:45 smithi161 ceph-mon[91579]: pgmap v19007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:45 smithi188 ceph-mon[92425]: pgmap v19007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:47 smithi161 ceph-mon[91579]: pgmap v19008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:47 smithi188 ceph-mon[92425]: pgmap v19008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:49.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:49.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:49 smithi161 ceph-mon[91579]: pgmap v19009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:49 smithi188 ceph-mon[92425]: pgmap v19009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:51 smithi161 ceph-mon[91579]: pgmap v19010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:51 smithi188 ceph-mon[92425]: pgmap v19010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:53 smithi161 ceph-mon[91579]: pgmap v19011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:53 smithi188 ceph-mon[92425]: pgmap v19011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:55 smithi161 ceph-mon[91579]: pgmap v19012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:55 smithi188 ceph-mon[92425]: pgmap v19012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:57 smithi161 ceph-mon[91579]: pgmap v19013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:57 smithi188 ceph-mon[92425]: pgmap v19013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:12:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: pgmap v19014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:12:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:12:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:12:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:12:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:12:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: pgmap v19014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:12:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:12:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:12:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:12:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:12:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:12:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:13:01.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:01 smithi161 ceph-mon[91579]: pgmap v19015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:01.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:01 smithi188 ceph-mon[92425]: pgmap v19015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:03 smithi161 ceph-mon[91579]: pgmap v19016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:03 smithi188 ceph-mon[92425]: pgmap v19016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:05 smithi161 ceph-mon[91579]: pgmap v19017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:05 smithi188 ceph-mon[92425]: pgmap v19017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:07 smithi161 ceph-mon[91579]: pgmap v19018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:07 smithi188 ceph-mon[92425]: pgmap v19018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:09 smithi161 ceph-mon[91579]: pgmap v19019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:09 smithi188 ceph-mon[92425]: pgmap v19019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:11 smithi161 ceph-mon[91579]: pgmap v19020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:11 smithi188 ceph-mon[92425]: pgmap v19020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:13 smithi161 ceph-mon[91579]: pgmap v19021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:13 smithi188 ceph-mon[92425]: pgmap v19021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:15 smithi161 ceph-mon[91579]: pgmap v19022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:15 smithi188 ceph-mon[92425]: pgmap v19022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:17 smithi161 ceph-mon[91579]: pgmap v19023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:17 smithi188 ceph-mon[92425]: pgmap v19023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:19 smithi161 ceph-mon[91579]: pgmap v19024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:19 smithi188 ceph-mon[92425]: pgmap v19024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:21 smithi161 ceph-mon[91579]: pgmap v19025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:21 smithi188 ceph-mon[92425]: pgmap v19025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:23 smithi161 ceph-mon[91579]: pgmap v19026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:23 smithi188 ceph-mon[92425]: pgmap v19026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:24.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:25 smithi161 ceph-mon[91579]: pgmap v19027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:25 smithi188 ceph-mon[92425]: pgmap v19027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:27 smithi161 ceph-mon[91579]: pgmap v19028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:27 smithi188 ceph-mon[92425]: pgmap v19028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:29 smithi161 ceph-mon[91579]: pgmap v19029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:29 smithi188 ceph-mon[92425]: pgmap v19029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:31 smithi161 ceph-mon[91579]: pgmap v19030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:31 smithi188 ceph-mon[92425]: pgmap v19030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:33 smithi161 ceph-mon[91579]: pgmap v19031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:33 smithi188 ceph-mon[92425]: pgmap v19031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:35 smithi161 ceph-mon[91579]: pgmap v19032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:35 smithi188 ceph-mon[92425]: pgmap v19032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:37 smithi161 ceph-mon[91579]: pgmap v19033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:37 smithi188 ceph-mon[92425]: pgmap v19033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:39 smithi161 ceph-mon[91579]: pgmap v19034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:39 smithi188 ceph-mon[92425]: pgmap v19034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:41 smithi161 ceph-mon[91579]: pgmap v19035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:41.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:41 smithi188 ceph-mon[92425]: pgmap v19035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:43.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-mon[91579]: pgmap v19036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:13:43.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:13:43.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:13:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:13:43.360+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:13:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:13:43.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:13:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:13:43.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:13:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:13:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:13:43.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:13:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:13:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:13:43.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:13:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:13:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:13:43.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:13:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:13:43.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:13:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:13:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:13:43.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:13:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:13:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:13:43.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:13:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:13:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:13:43.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:13:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:13:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:13:43.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:13:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:13:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:13:43.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:13:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:13:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:13:43.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:13:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:13:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:13:43.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:13:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:13:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:13:43.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:13:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:13:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:13:43.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:13:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:13:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:13:43.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:13:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1157] Local address: 172.21.15.161, port 33416 2022-07-03T21:13:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1157] Peer address: 172.21.15.101, port 22 2022-07-03T21:13:43.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1157] Connection lost 2022-07-03T21:13:43.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1157] Aborting connection 2022-07-03T21:13:43.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:13:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:43 smithi188 ceph-mon[92425]: pgmap v19036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:13:43.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:13:43.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:43 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:13:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:13:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:44 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:13:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:13:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:13:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:13:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:13:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:13:44.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:13:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:13:44.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:13:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:13:44.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:13:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:13:44.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:13:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:13:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:13:44.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:13:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:13:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:13:44.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:13:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:13:44.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:13:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:13:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:13:44.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:13:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:13:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:13:44.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:13:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:13:44.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:13:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:13:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:13:44.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:13:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:13:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:13:44.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:13:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:13:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:13:44.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:13:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:13:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:13:44.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:13:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:13:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:13:44.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:13:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:13:44.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:13:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1157] Local address: 172.21.15.161, port 33416 2022-07-03T21:13:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1157] Peer address: 172.21.15.101, port 22 2022-07-03T21:13:44.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1157] Connection lost 2022-07-03T21:13:44.898 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1157] Aborting connection 2022-07-03T21:13:44.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:13:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:44 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:13:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:13:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:13:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:13:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:13:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:13:44.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:13:44.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:13:44.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:13:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:13:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:13:44.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:13:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:13:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:13:44.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:13:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:13:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:13:44.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:13:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:13:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:13:44.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:13:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:13:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:13:44.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:13:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:13:44.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:13:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:13:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:13:44.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:13:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:13:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:13:44.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:13:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:13:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:13:44.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:13:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:13:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:13:44.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:13:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:13:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:13:44.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:13:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1157] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:13:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1157] Local address: 172.21.15.161, port 33416 2022-07-03T21:13:44.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1157] Peer address: 172.21.15.101, port 22 2022-07-03T21:13:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1157] Connection lost 2022-07-03T21:13:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1157] Aborting connection 2022-07-03T21:13:44.930 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:45 smithi161 ceph-mon[91579]: pgmap v19037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:45 smithi188 ceph-mon[92425]: pgmap v19037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:47 smithi161 ceph-mon[91579]: pgmap v19038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:47 smithi188 ceph-mon[92425]: pgmap v19038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:49 smithi161 ceph-mon[91579]: pgmap v19039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:49 smithi188 ceph-mon[92425]: pgmap v19039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:51 smithi161 ceph-mon[91579]: pgmap v19040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:51 smithi188 ceph-mon[92425]: pgmap v19040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:53 smithi161 ceph-mon[91579]: pgmap v19041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:53 smithi188 ceph-mon[92425]: pgmap v19041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:55 smithi161 ceph-mon[91579]: pgmap v19042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:55 smithi188 ceph-mon[92425]: pgmap v19042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:57 smithi161 ceph-mon[91579]: pgmap v19043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:57 smithi188 ceph-mon[92425]: pgmap v19043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:13:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: pgmap v19044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:13:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:13:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:13:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:13:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:13:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: pgmap v19044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:13:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:13:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:13:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:13:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:13:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:13:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:14:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:01 smithi161 ceph-mon[91579]: pgmap v19045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:01 smithi188 ceph-mon[92425]: pgmap v19045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:03 smithi161 ceph-mon[91579]: pgmap v19046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:03 smithi188 ceph-mon[92425]: pgmap v19046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:05 smithi161 ceph-mon[91579]: pgmap v19047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:05 smithi188 ceph-mon[92425]: pgmap v19047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:07 smithi161 ceph-mon[91579]: pgmap v19048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:07 smithi188 ceph-mon[92425]: pgmap v19048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:09 smithi161 ceph-mon[91579]: pgmap v19049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:09 smithi188 ceph-mon[92425]: pgmap v19049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:11 smithi161 ceph-mon[91579]: pgmap v19050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:11 smithi188 ceph-mon[92425]: pgmap v19050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:13 smithi161 ceph-mon[91579]: pgmap v19051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:13 smithi188 ceph-mon[92425]: pgmap v19051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:15 smithi161 ceph-mon[91579]: pgmap v19052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:15 smithi188 ceph-mon[92425]: pgmap v19052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:17 smithi161 ceph-mon[91579]: pgmap v19053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:17 smithi188 ceph-mon[92425]: pgmap v19053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:19 smithi161 ceph-mon[91579]: pgmap v19054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:19 smithi188 ceph-mon[92425]: pgmap v19054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:21.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:21 smithi161 ceph-mon[91579]: pgmap v19055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:21 smithi188 ceph-mon[92425]: pgmap v19055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:23 smithi161 ceph-mon[91579]: pgmap v19056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:23 smithi188 ceph-mon[92425]: pgmap v19056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:25 smithi161 ceph-mon[91579]: pgmap v19057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:25 smithi188 ceph-mon[92425]: pgmap v19057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:27 smithi161 ceph-mon[91579]: pgmap v19058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:27 smithi188 ceph-mon[92425]: pgmap v19058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:29 smithi161 ceph-mon[91579]: pgmap v19059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:29 smithi188 ceph-mon[92425]: pgmap v19059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:31 smithi161 ceph-mon[91579]: pgmap v19060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:31 smithi188 ceph-mon[92425]: pgmap v19060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:33 smithi161 ceph-mon[91579]: pgmap v19061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:33 smithi188 ceph-mon[92425]: pgmap v19061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:35 smithi161 ceph-mon[91579]: pgmap v19062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:35 smithi188 ceph-mon[92425]: pgmap v19062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:37 smithi161 ceph-mon[91579]: pgmap v19063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:37 smithi188 ceph-mon[92425]: pgmap v19063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:39 smithi161 ceph-mon[91579]: pgmap v19064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:39 smithi188 ceph-mon[92425]: pgmap v19064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:41 smithi161 ceph-mon[91579]: pgmap v19065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:41 smithi188 ceph-mon[92425]: pgmap v19065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:43.867 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-mon[91579]: pgmap v19066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:43.878 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:43 smithi188 ceph-mon[92425]: pgmap v19066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:44.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:14:43.867+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:14:44.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:14:44.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:14:44.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:14:44.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:14:44.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:14:44.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:14:44.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:14:44.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:14:44.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:14:44.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:14:44.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:14:44.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:14:44.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:14:44.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:14:44.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:14:44.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:14:44.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:14:44.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:14:44.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:14:44.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:14:44.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:14:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:14:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:14:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:14:44.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:14:44.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:14:44.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:14:44.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:14:44.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:14:44.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:14:44.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:14:44.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:14:44.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:14:44.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:14:44.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:14:44.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:14:44.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:14:44.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:14:44.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:14:44.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:14:44.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:14:44.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:14:44.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:14:44.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:14:44.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:14:44.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1159] Local address: 172.21.15.161, port 33420 2022-07-03T21:14:44.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1159] Peer address: 172.21.15.101, port 22 2022-07-03T21:14:44.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1159] Connection lost 2022-07-03T21:14:44.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1159] Aborting connection 2022-07-03T21:14:44.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:14:43 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:14:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:14:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:14:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:14:44.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:44 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:14:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:14:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:14:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:14:44.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:44 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:14:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:14:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:14:45.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:14:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:14:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:14:45.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:14:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:14:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:14:45.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:14:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:14:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:14:45.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:14:45.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:14:45.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:14:45.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:14:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:14:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:14:45.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:14:45.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:14:45.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:14:45.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:14:45.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:14:45.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:14:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:14:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:14:45.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:14:45.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:14:45.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:14:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:14:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:14:45.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:14:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:14:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:14:45.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:14:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:14:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:14:45.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:14:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:14:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:14:45.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:14:45.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:14:45.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:14:45.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:14:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:14:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:14:45.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:14:45.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1159] Local address: 172.21.15.161, port 33420 2022-07-03T21:14:45.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1159] Peer address: 172.21.15.101, port 22 2022-07-03T21:14:45.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1159] Connection lost 2022-07-03T21:14:45.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1159] Aborting connection 2022-07-03T21:14:45.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:45 smithi161 ceph-mon[91579]: pgmap v19067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:14:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:14:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:14:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:14:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:14:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:14:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:14:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:14:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:14:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:14:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:14:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:14:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:14:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:14:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:14:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:14:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:14:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:14:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:14:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:14:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:14:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1159] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:14:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1159] Local address: 172.21.15.161, port 33420 2022-07-03T21:14:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1159] Peer address: 172.21.15.101, port 22 2022-07-03T21:14:45.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1159] Connection lost 2022-07-03T21:14:45.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1159] Aborting connection 2022-07-03T21:14:45.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:45 smithi188 ceph-mon[92425]: pgmap v19067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:47 smithi161 ceph-mon[91579]: pgmap v19068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:47 smithi188 ceph-mon[92425]: pgmap v19068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:49 smithi161 ceph-mon[91579]: pgmap v19069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:49 smithi188 ceph-mon[92425]: pgmap v19069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:51 smithi161 ceph-mon[91579]: pgmap v19070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:51 smithi188 ceph-mon[92425]: pgmap v19070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:53 smithi161 ceph-mon[91579]: pgmap v19071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:53 smithi188 ceph-mon[92425]: pgmap v19071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:55 smithi161 ceph-mon[91579]: pgmap v19072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:55 smithi188 ceph-mon[92425]: pgmap v19072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:57 smithi161 ceph-mon[91579]: pgmap v19073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:57 smithi188 ceph-mon[92425]: pgmap v19073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:14:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: pgmap v19074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:14:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:14:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:14:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:14:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:14:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: pgmap v19074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:14:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:14:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:14:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:14:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:14:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:14:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:15:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:01 smithi161 ceph-mon[91579]: pgmap v19075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:01 smithi188 ceph-mon[92425]: pgmap v19075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:03 smithi161 ceph-mon[91579]: pgmap v19076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:03 smithi188 ceph-mon[92425]: pgmap v19076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:05 smithi161 ceph-mon[91579]: pgmap v19077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:05 smithi188 ceph-mon[92425]: pgmap v19077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:07 smithi161 ceph-mon[91579]: pgmap v19078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:07 smithi188 ceph-mon[92425]: pgmap v19078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:09 smithi161 ceph-mon[91579]: pgmap v19079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:09 smithi188 ceph-mon[92425]: pgmap v19079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:11 smithi161 ceph-mon[91579]: pgmap v19080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:11 smithi188 ceph-mon[92425]: pgmap v19080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:13 smithi161 ceph-mon[91579]: pgmap v19081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:13 smithi188 ceph-mon[92425]: pgmap v19081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:15 smithi161 ceph-mon[91579]: pgmap v19082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:15 smithi188 ceph-mon[92425]: pgmap v19082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:17 smithi161 ceph-mon[91579]: pgmap v19083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:17 smithi188 ceph-mon[92425]: pgmap v19083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:19 smithi161 ceph-mon[91579]: pgmap v19084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:19 smithi188 ceph-mon[92425]: pgmap v19084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:21 smithi161 ceph-mon[91579]: pgmap v19085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:21.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:21 smithi188 ceph-mon[92425]: pgmap v19085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:23 smithi161 ceph-mon[91579]: pgmap v19086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:23 smithi188 ceph-mon[92425]: pgmap v19086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:25 smithi161 ceph-mon[91579]: pgmap v19087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:25 smithi188 ceph-mon[92425]: pgmap v19087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:27 smithi161 ceph-mon[91579]: pgmap v19088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:27 smithi188 ceph-mon[92425]: pgmap v19088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:29 smithi161 ceph-mon[91579]: pgmap v19089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:29 smithi188 ceph-mon[92425]: pgmap v19089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:31 smithi161 ceph-mon[91579]: pgmap v19090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:31 smithi188 ceph-mon[92425]: pgmap v19090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:33 smithi161 ceph-mon[91579]: pgmap v19091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:33 smithi188 ceph-mon[92425]: pgmap v19091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:35 smithi161 ceph-mon[91579]: pgmap v19092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:35 smithi188 ceph-mon[92425]: pgmap v19092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:37 smithi161 ceph-mon[91579]: pgmap v19093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:37.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:37 smithi188 ceph-mon[92425]: pgmap v19093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:39 smithi161 ceph-mon[91579]: pgmap v19094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:39 smithi188 ceph-mon[92425]: pgmap v19094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:41 smithi161 ceph-mon[91579]: pgmap v19095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:41 smithi188 ceph-mon[92425]: pgmap v19095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:43 smithi161 ceph-mon[91579]: pgmap v19096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:43 smithi188 ceph-mon[92425]: pgmap v19096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:44.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:15:44.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:15:44.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:15:44.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:15:44.365+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:15:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:15:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:15:44.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:15:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:15:44.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:15:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:15:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:15:44.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:15:44.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:15:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:15:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:15:44.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:15:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:15:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:15:44.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:15:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:15:44.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:15:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:15:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:15:44.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:15:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:15:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:15:44.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:15:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:15:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:15:44.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:15:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:15:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:15:44.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:15:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:15:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:15:44.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:15:44.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:15:44.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:15:44.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:15:44.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:15:44.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:15:44.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:15:44.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:15:44.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:15:44.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:15:44.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:15:44.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:15:44.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:15:44.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:15:44.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1161] Local address: 172.21.15.161, port 33424 2022-07-03T21:15:44.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1161] Peer address: 172.21.15.101, port 22 2022-07-03T21:15:44.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1161] Connection lost 2022-07-03T21:15:44.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1161] Aborting connection 2022-07-03T21:15:44.655 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:15:44 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:15:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:15:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:15:44.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:15:45.850 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:45 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:15:45.850 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:45 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:15:45.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:15:45.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:15:45.851 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:15:45.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:15:45.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:15:45.852 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:15:45.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:15:45.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:15:45.853 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:15:45.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:15:45.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:15:45.854 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:15:45.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:15:45.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:15:45.855 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:15:45.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:15:45.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:15:45.856 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:15:45.857 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:15:45.857 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:15:45.858 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:15:45.858 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:15:45.858 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:15:45.859 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:15:45.859 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:15:45.859 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:15:45.859 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:15:45.860 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:15:45.860 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:15:45.860 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:15:45.861 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:15:45.861 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:15:45.861 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:15:45.862 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:15:45.862 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:15:45.863 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:15:45.863 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:15:45.863 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:15:45.864 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:15:45.864 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:15:45.864 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:15:45.865 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:15:45.865 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:15:45.865 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:15:45.866 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:15:45.866 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1161] Local address: 172.21.15.161, port 33424 2022-07-03T21:15:45.866 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1161] Peer address: 172.21.15.101, port 22 2022-07-03T21:15:45.867 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1161] Connection lost 2022-07-03T21:15:45.867 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1161] Aborting connection 2022-07-03T21:15:45.867 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:45 smithi161 ceph-mon[91579]: pgmap v19097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:45 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:15:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:45 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:15:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:15:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:15:45.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:15:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:15:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:15:45.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:15:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:15:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:15:45.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:15:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:15:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:15:45.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:15:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:15:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:15:45.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:15:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:15:45.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:15:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:15:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:15:45.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:15:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:15:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:15:45.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:15:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:15:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:15:45.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:15:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:15:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:15:45.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:15:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:15:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:15:45.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:15:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:15:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:15:45.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:15:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:15:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:15:45.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:15:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:15:45.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:15:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:15:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:15:45.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:15:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:15:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1161] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:15:45.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1161] Local address: 172.21.15.161, port 33424 2022-07-03T21:15:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1161] Peer address: 172.21.15.101, port 22 2022-07-03T21:15:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1161] Connection lost 2022-07-03T21:15:45.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1161] Aborting connection 2022-07-03T21:15:45.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:45 smithi188 ceph-mon[92425]: pgmap v19097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:47 smithi161 ceph-mon[91579]: pgmap v19098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:47 smithi188 ceph-mon[92425]: pgmap v19098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:48 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T21:15:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:15:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:48 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:15:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:48 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T21:15:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:15:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:48 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:15:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:49 smithi161 ceph-mon[91579]: pgmap v19099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:49 smithi188 ceph-mon[92425]: pgmap v19099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:51 smithi161 ceph-mon[91579]: pgmap v19100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:51 smithi188 ceph-mon[92425]: pgmap v19100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:53 smithi161 ceph-mon[91579]: pgmap v19101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:53 smithi188 ceph-mon[92425]: pgmap v19101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:55 smithi161 ceph-mon[91579]: pgmap v19102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:55 smithi188 ceph-mon[92425]: pgmap v19102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:57 smithi161 ceph-mon[91579]: pgmap v19103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:57 smithi188 ceph-mon[92425]: pgmap v19103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:15:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: pgmap v19104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:15:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:15:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:15:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:15:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:15:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: pgmap v19104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:15:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:15:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:15:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:15:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:15:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:15:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:16:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:01 smithi161 ceph-mon[91579]: pgmap v19105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:01 smithi188 ceph-mon[92425]: pgmap v19105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:03 smithi161 ceph-mon[91579]: pgmap v19106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:03 smithi188 ceph-mon[92425]: pgmap v19106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:05 smithi161 ceph-mon[91579]: pgmap v19107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:05 smithi188 ceph-mon[92425]: pgmap v19107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:07 smithi161 ceph-mon[91579]: pgmap v19108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:07 smithi188 ceph-mon[92425]: pgmap v19108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:09 smithi161 ceph-mon[91579]: pgmap v19109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:09 smithi188 ceph-mon[92425]: pgmap v19109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:11 smithi161 ceph-mon[91579]: pgmap v19110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:11 smithi188 ceph-mon[92425]: pgmap v19110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:13 smithi161 ceph-mon[91579]: pgmap v19111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:13 smithi188 ceph-mon[92425]: pgmap v19111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:14.886 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:15 smithi161 ceph-mon[91579]: pgmap v19112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:15 smithi188 ceph-mon[92425]: pgmap v19112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:17 smithi161 ceph-mon[91579]: pgmap v19113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:17 smithi188 ceph-mon[92425]: pgmap v19113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:19 smithi161 ceph-mon[91579]: pgmap v19114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:19 smithi188 ceph-mon[92425]: pgmap v19114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:21 smithi161 ceph-mon[91579]: pgmap v19115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:21 smithi188 ceph-mon[92425]: pgmap v19115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:23 smithi161 ceph-mon[91579]: pgmap v19116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:23 smithi188 ceph-mon[92425]: pgmap v19116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:25 smithi161 ceph-mon[91579]: pgmap v19117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:25 smithi188 ceph-mon[92425]: pgmap v19117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:27 smithi161 ceph-mon[91579]: pgmap v19118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:27 smithi188 ceph-mon[92425]: pgmap v19118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:29 smithi161 ceph-mon[91579]: pgmap v19119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:29 smithi188 ceph-mon[92425]: pgmap v19119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:31 smithi161 ceph-mon[91579]: pgmap v19120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:31 smithi188 ceph-mon[92425]: pgmap v19120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:33 smithi161 ceph-mon[91579]: pgmap v19121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:33 smithi188 ceph-mon[92425]: pgmap v19121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:35.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:35 smithi161 ceph-mon[91579]: pgmap v19122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:35 smithi188 ceph-mon[92425]: pgmap v19122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:37 smithi161 ceph-mon[91579]: pgmap v19123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:37 smithi188 ceph-mon[92425]: pgmap v19123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:39 smithi161 ceph-mon[91579]: pgmap v19124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:39 smithi188 ceph-mon[92425]: pgmap v19124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:41 smithi161 ceph-mon[91579]: pgmap v19125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:41 smithi188 ceph-mon[92425]: pgmap v19125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:43 smithi161 ceph-mon[91579]: pgmap v19126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:43 smithi188 ceph-mon[92425]: pgmap v19126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:45 smithi161 ceph-mon[91579]: pgmap v19127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:45 smithi188 ceph-mon[92425]: pgmap v19127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-mon[91579]: pgmap v19128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:47.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:16:47.653+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:16:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:16:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:16:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:16:47.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:16:47.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:16:47.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:16:47.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:16:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:16:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:16:47.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:16:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:16:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:16:47.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:16:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:16:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:16:47.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:16:47.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:16:47.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:16:47.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:16:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:16:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:16:47.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:16:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:16:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:16:47.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:16:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:16:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:16:47.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:16:47.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:16:47.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:16:47.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:16:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:16:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:16:47.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:16:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:16:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:16:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:16:47.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:16:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:16:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:16:47.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:16:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:16:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:16:47.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:16:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:16:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1163] Local address: 172.21.15.161, port 33428 2022-07-03T21:16:47.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1163] Peer address: 172.21.15.101, port 22 2022-07-03T21:16:47.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1163] Connection lost 2022-07-03T21:16:47.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1163] Aborting connection 2022-07-03T21:16:47.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:16:47 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:16:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:47 smithi188 ceph-mon[92425]: pgmap v19128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:16:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:16:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:16:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:48 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:16:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:48 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:16:48.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:16:48.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:16:48.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:16:48.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:16:48.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:16:48.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:16:48.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:16:48.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:16:48.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:16:48.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:16:48.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:16:48.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:16:48.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:16:48.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:16:48.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:16:48.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:16:48.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:16:48.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:16:48.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:16:48.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:16:48.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:16:48.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:16:48.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:16:48.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:16:48.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:16:48.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:16:48.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:16:48.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:16:48.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:16:48.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:16:48.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:16:48.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:16:48.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:16:48.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:16:48.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:16:48.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:16:48.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:16:48.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:16:48.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:16:48.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:16:48.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:16:48.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:16:48.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:16:48.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:16:48.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:16:48.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1163] Local address: 172.21.15.161, port 33428 2022-07-03T21:16:48.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1163] Peer address: 172.21.15.101, port 22 2022-07-03T21:16:48.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1163] Connection lost 2022-07-03T21:16:48.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1163] Aborting connection 2022-07-03T21:16:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:16:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:16:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:16:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:48 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:16:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:48 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:16:48.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:16:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:16:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:16:48.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:16:48.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:16:48.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:16:48.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:16:48.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:16:48.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:16:48.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:16:48.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:16:48.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:16:48.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:16:48.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:16:48.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:16:48.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:16:48.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:16:48.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:16:48.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:16:48.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:16:48.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:16:48.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:16:48.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:16:48.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:16:48.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:16:48.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:16:48.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:16:48.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:16:48.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:16:48.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:16:48.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:16:48.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:16:48.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:16:48.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:16:48.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:16:48.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:16:48.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:16:48.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:16:48.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:16:48.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:16:48.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:16:48.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:16:48.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:16:48.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:16:48.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1163] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:16:48.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1163] Local address: 172.21.15.161, port 33428 2022-07-03T21:16:48.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1163] Peer address: 172.21.15.101, port 22 2022-07-03T21:16:48.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1163] Connection lost 2022-07-03T21:16:48.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1163] Aborting connection 2022-07-03T21:16:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:49 smithi161 ceph-mon[91579]: pgmap v19129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:49 smithi188 ceph-mon[92425]: pgmap v19129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:51.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:51 smithi161 ceph-mon[91579]: pgmap v19130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:51 smithi188 ceph-mon[92425]: pgmap v19130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:53 smithi161 ceph-mon[91579]: pgmap v19131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:53 smithi188 ceph-mon[92425]: pgmap v19131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:55 smithi161 ceph-mon[91579]: pgmap v19132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:55 smithi188 ceph-mon[92425]: pgmap v19132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:57 smithi161 ceph-mon[91579]: pgmap v19133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:57 smithi188 ceph-mon[92425]: pgmap v19133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:16:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: pgmap v19134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:16:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:16:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:16:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:16:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:16:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: pgmap v19134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:16:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:16:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:16:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:16:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:16:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:16:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:17:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:01 smithi161 ceph-mon[91579]: pgmap v19135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:01 smithi188 ceph-mon[92425]: pgmap v19135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:03 smithi161 ceph-mon[91579]: pgmap v19136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:03 smithi188 ceph-mon[92425]: pgmap v19136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:05 smithi161 ceph-mon[91579]: pgmap v19137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:05 smithi188 ceph-mon[92425]: pgmap v19137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:07 smithi161 ceph-mon[91579]: pgmap v19138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:07 smithi188 ceph-mon[92425]: pgmap v19138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:09 smithi161 ceph-mon[91579]: pgmap v19139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:09 smithi188 ceph-mon[92425]: pgmap v19139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:11.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:11 smithi161 ceph-mon[91579]: pgmap v19140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:11.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:11 smithi188 ceph-mon[92425]: pgmap v19140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:13 smithi161 ceph-mon[91579]: pgmap v19141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:13 smithi188 ceph-mon[92425]: pgmap v19141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:15 smithi161 ceph-mon[91579]: pgmap v19142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:15 smithi188 ceph-mon[92425]: pgmap v19142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:17 smithi161 ceph-mon[91579]: pgmap v19143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:17.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:17 smithi188 ceph-mon[92425]: pgmap v19143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:19 smithi161 ceph-mon[91579]: pgmap v19144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:19 smithi188 ceph-mon[92425]: pgmap v19144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:19.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:21 smithi161 ceph-mon[91579]: pgmap v19145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:21 smithi188 ceph-mon[92425]: pgmap v19145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:23 smithi161 ceph-mon[91579]: pgmap v19146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:23 smithi188 ceph-mon[92425]: pgmap v19146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:25.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:25 smithi161 ceph-mon[91579]: pgmap v19147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:25 smithi188 ceph-mon[92425]: pgmap v19147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:27 smithi161 ceph-mon[91579]: pgmap v19148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:27.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:27 smithi188 ceph-mon[92425]: pgmap v19148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:29 smithi161 ceph-mon[91579]: pgmap v19149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:29 smithi188 ceph-mon[92425]: pgmap v19149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:31 smithi161 ceph-mon[91579]: pgmap v19150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:31 smithi188 ceph-mon[92425]: pgmap v19150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:33 smithi161 ceph-mon[91579]: pgmap v19151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:33 smithi188 ceph-mon[92425]: pgmap v19151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:35 smithi161 ceph-mon[91579]: pgmap v19152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:35 smithi188 ceph-mon[92425]: pgmap v19152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:37 smithi161 ceph-mon[91579]: pgmap v19153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:37 smithi188 ceph-mon[92425]: pgmap v19153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:39 smithi161 ceph-mon[91579]: pgmap v19154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:39 smithi188 ceph-mon[92425]: pgmap v19154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:41.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:41 smithi161 ceph-mon[91579]: pgmap v19155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:41 smithi188 ceph-mon[92425]: pgmap v19155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:43 smithi161 ceph-mon[91579]: pgmap v19156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:43 smithi188 ceph-mon[92425]: pgmap v19156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:45 smithi161 ceph-mon[91579]: pgmap v19157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:45 smithi188 ceph-mon[92425]: pgmap v19157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:47 smithi161 ceph-mon[91579]: pgmap v19158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:47 smithi188 ceph-mon[92425]: pgmap v19158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:48.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:17:48.168+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:17:48.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:17:48.520 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:17:48.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:17:48.521 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:17:48.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:17:48.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:17:48.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:17:48.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:17:48.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:17:48.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:17:48.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:17:48.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:17:48.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:17:48.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:17:48.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:17:48.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:17:48.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:17:48.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:17:48.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:17:48.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:17:48.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:17:48.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:17:48.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:17:48.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:17:48.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:17:48.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:17:48.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:17:48.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:17:48.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:17:48.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:17:48.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:17:48.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:17:48.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:17:48.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:17:48.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:17:48.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:17:48.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:17:48.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:17:48.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:17:48.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:17:48.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:17:48.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:17:48.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:17:48.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:17:48.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:17:48.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1165] Local address: 172.21.15.161, port 33432 2022-07-03T21:17:48.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1165] Peer address: 172.21.15.101, port 22 2022-07-03T21:17:48.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1165] Connection lost 2022-07-03T21:17:48.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1165] Aborting connection 2022-07-03T21:17:48.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:17:48.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:17:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:17:48.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:48 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:17:48.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:17:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:17:48.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:48 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:17:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:17:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:17:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:17:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:17:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:17:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:17:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:17:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:17:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:17:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:17:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:17:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:17:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:17:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:17:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:17:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:17:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:17:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:17:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:17:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:17:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:17:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:17:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:17:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:17:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:17:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:17:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:17:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:17:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:17:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:17:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:17:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:17:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:17:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:17:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:17:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:17:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:17:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:17:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:17:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:17:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:17:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:17:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:17:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:17:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:17:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:17:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:17:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1165] Local address: 172.21.15.161, port 33432 2022-07-03T21:17:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1165] Peer address: 172.21.15.101, port 22 2022-07-03T21:17:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1165] Connection lost 2022-07-03T21:17:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1165] Aborting connection 2022-07-03T21:17:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:49 smithi161 ceph-mon[91579]: pgmap v19159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:49.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:17:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:17:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:17:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:17:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:17:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:17:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:17:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:17:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:17:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:17:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:17:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:17:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:17:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:17:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:17:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:17:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:17:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:17:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:17:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:17:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:17:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:17:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:17:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:17:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:17:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:17:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:17:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:17:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:17:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:17:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:17:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:17:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:17:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:17:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:17:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:17:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:17:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:17:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:17:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:17:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:17:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:17:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:17:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:17:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:17:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:17:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1165] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:17:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1165] Local address: 172.21.15.161, port 33432 2022-07-03T21:17:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1165] Peer address: 172.21.15.101, port 22 2022-07-03T21:17:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1165] Connection lost 2022-07-03T21:17:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1165] Aborting connection 2022-07-03T21:17:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:49 smithi188 ceph-mon[92425]: pgmap v19159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:51.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:51 smithi161 ceph-mon[91579]: pgmap v19160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:51.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:51 smithi188 ceph-mon[92425]: pgmap v19160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:53 smithi161 ceph-mon[91579]: pgmap v19161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:53 smithi188 ceph-mon[92425]: pgmap v19161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:55 smithi161 ceph-mon[91579]: pgmap v19162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:55 smithi188 ceph-mon[92425]: pgmap v19162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:57 smithi161 ceph-mon[91579]: pgmap v19163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:57 smithi188 ceph-mon[92425]: pgmap v19163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:17:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: pgmap v19164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:17:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:17:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:17:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:17:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:17:59.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: pgmap v19164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:17:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:17:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:17:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:17:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:17:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:17:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:18:01.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:01 smithi161 ceph-mon[91579]: pgmap v19165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:01 smithi188 ceph-mon[92425]: pgmap v19165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:03 smithi161 ceph-mon[91579]: pgmap v19166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:03 smithi188 ceph-mon[92425]: pgmap v19166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:05 smithi161 ceph-mon[91579]: pgmap v19167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:05 smithi188 ceph-mon[92425]: pgmap v19167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:07 smithi161 ceph-mon[91579]: pgmap v19168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:07 smithi188 ceph-mon[92425]: pgmap v19168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:09 smithi161 ceph-mon[91579]: pgmap v19169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:09.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:09 smithi188 ceph-mon[92425]: pgmap v19169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:11.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:11 smithi161 ceph-mon[91579]: pgmap v19170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:11 smithi188 ceph-mon[92425]: pgmap v19170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:13 smithi161 ceph-mon[91579]: pgmap v19171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:13 smithi188 ceph-mon[92425]: pgmap v19171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:15.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:15 smithi161 ceph-mon[91579]: pgmap v19172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:15 smithi188 ceph-mon[92425]: pgmap v19172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:17 smithi161 ceph-mon[91579]: pgmap v19173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:17 smithi188 ceph-mon[92425]: pgmap v19173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:19 smithi161 ceph-mon[91579]: pgmap v19174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:19 smithi188 ceph-mon[92425]: pgmap v19174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:21.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:21 smithi161 ceph-mon[91579]: pgmap v19175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:21 smithi188 ceph-mon[92425]: pgmap v19175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:23 smithi161 ceph-mon[91579]: pgmap v19176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:23 smithi188 ceph-mon[92425]: pgmap v19176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:25 smithi161 ceph-mon[91579]: pgmap v19177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:25.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:25 smithi188 ceph-mon[92425]: pgmap v19177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:27 smithi161 ceph-mon[91579]: pgmap v19178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:27 smithi188 ceph-mon[92425]: pgmap v19178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:29 smithi161 ceph-mon[91579]: pgmap v19179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:29 smithi188 ceph-mon[92425]: pgmap v19179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:31 smithi188 ceph-mon[92425]: pgmap v19180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:31 smithi161 ceph-mon[91579]: pgmap v19180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:33 smithi161 ceph-mon[91579]: pgmap v19181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:33 smithi188 ceph-mon[92425]: pgmap v19181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:35 smithi161 ceph-mon[91579]: pgmap v19182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:35 smithi188 ceph-mon[92425]: pgmap v19182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:37 smithi161 ceph-mon[91579]: pgmap v19183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:37 smithi188 ceph-mon[92425]: pgmap v19183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:39 smithi161 ceph-mon[91579]: pgmap v19184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:39 smithi188 ceph-mon[92425]: pgmap v19184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:41.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:41 smithi188 ceph-mon[92425]: pgmap v19185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:41 smithi161 ceph-mon[91579]: pgmap v19185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:43 smithi161 ceph-mon[91579]: pgmap v19186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:43 smithi188 ceph-mon[92425]: pgmap v19186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:45 smithi161 ceph-mon[91579]: pgmap v19187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:45.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:45 smithi188 ceph-mon[92425]: pgmap v19187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:47 smithi161 ceph-mon[91579]: pgmap v19188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:47 smithi188 ceph-mon[92425]: pgmap v19188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:49.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:18:48.666+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:18:49.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:18:49.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:18:49.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:18:49.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:18:49.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:18:49.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:18:49.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:18:49.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:18:49.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:18:49.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:18:49.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:18:49.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:18:49.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:18:49.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:18:49.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:18:49.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:18:49.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:18:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:18:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:18:49.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:18:49.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:18:49.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:18:49.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:18:49.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:18:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:18:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:18:49.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:18:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:18:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:18:49.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:18:49.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:18:49.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:18:49.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:18:49.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:18:49.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:18:49.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:18:49.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:18:49.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:18:49.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:18:49.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:18:49.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:18:49.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:18:49.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:18:49.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:18:49.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:18:49.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1167] Local address: 172.21.15.161, port 33436 2022-07-03T21:18:49.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1167] Peer address: 172.21.15.101, port 22 2022-07-03T21:18:49.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1167] Connection lost 2022-07-03T21:18:49.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1167] Aborting connection 2022-07-03T21:18:49.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:18:48 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:18:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: pgmap v19189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:18:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:18:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:18:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:18:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:18:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:18:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:18:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:18:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:18:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:18:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:18:49.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:18:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:18:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:18:49.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:18:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:18:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:18:49.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:18:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:18:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:18:49.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:18:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:18:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:18:49.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:18:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:18:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:18:49.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:18:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:18:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:18:49.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:18:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:18:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:18:49.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:18:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:18:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:18:49.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:18:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:18:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:18:49.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:18:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:18:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:18:49.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:18:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:18:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:18:49.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:18:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:18:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:18:49.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:18:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:18:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:18:49.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1167] Local address: 172.21.15.161, port 33436 2022-07-03T21:18:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1167] Peer address: 172.21.15.101, port 22 2022-07-03T21:18:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1167] Connection lost 2022-07-03T21:18:49.897 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1167] Aborting connection 2022-07-03T21:18:49.898 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: pgmap v19189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:18:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:18:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:18:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:18:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:18:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:18:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:18:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:18:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:18:49.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:18:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:18:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:18:49.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:18:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:18:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:18:49.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:18:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:18:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:18:49.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:18:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:18:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:18:49.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:18:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:18:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:18:49.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:18:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:18:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:18:49.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:18:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:18:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:18:49.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:18:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:18:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:18:49.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:18:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:18:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:18:49.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:18:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:18:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:18:49.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:18:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:18:49.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:18:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:18:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:18:49.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:18:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:18:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:18:49.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:18:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:18:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1167] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:18:49.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1167] Local address: 172.21.15.161, port 33436 2022-07-03T21:18:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1167] Peer address: 172.21.15.101, port 22 2022-07-03T21:18:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1167] Connection lost 2022-07-03T21:18:49.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1167] Aborting connection 2022-07-03T21:18:49.932 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:51 smithi188 ceph-mon[92425]: pgmap v19190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:51 smithi161 ceph-mon[91579]: pgmap v19190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:53 smithi161 ceph-mon[91579]: pgmap v19191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:53 smithi188 ceph-mon[92425]: pgmap v19191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:54.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:55 smithi161 ceph-mon[91579]: pgmap v19192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:55 smithi188 ceph-mon[92425]: pgmap v19192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:57 smithi161 ceph-mon[91579]: pgmap v19193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:57 smithi188 ceph-mon[92425]: pgmap v19193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:18:59.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: pgmap v19194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:18:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:18:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:18:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:18:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:18:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: pgmap v19194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:18:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:18:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:18:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:18:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:18:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:18:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:19:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:01 smithi188 ceph-mon[92425]: pgmap v19195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:01 smithi161 ceph-mon[91579]: pgmap v19195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:03 smithi161 ceph-mon[91579]: pgmap v19196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:03 smithi188 ceph-mon[92425]: pgmap v19196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:05 smithi161 ceph-mon[91579]: pgmap v19197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:05 smithi188 ceph-mon[92425]: pgmap v19197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:07 smithi161 ceph-mon[91579]: pgmap v19198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:07 smithi188 ceph-mon[92425]: pgmap v19198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:09 smithi161 ceph-mon[91579]: pgmap v19199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:09 smithi188 ceph-mon[92425]: pgmap v19199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:11.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:11 smithi188 ceph-mon[92425]: pgmap v19200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:11 smithi161 ceph-mon[91579]: pgmap v19200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:13 smithi161 ceph-mon[91579]: pgmap v19201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:13 smithi188 ceph-mon[92425]: pgmap v19201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:14.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:15 smithi161 ceph-mon[91579]: pgmap v19202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:15 smithi188 ceph-mon[92425]: pgmap v19202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:17 smithi161 ceph-mon[91579]: pgmap v19203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:17 smithi188 ceph-mon[92425]: pgmap v19203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:19 smithi161 ceph-mon[91579]: pgmap v19204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:19 smithi188 ceph-mon[92425]: pgmap v19204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:21.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:21 smithi188 ceph-mon[92425]: pgmap v19205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:21 smithi161 ceph-mon[91579]: pgmap v19205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:23 smithi161 ceph-mon[91579]: pgmap v19206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:23 smithi188 ceph-mon[92425]: pgmap v19206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:25 smithi161 ceph-mon[91579]: pgmap v19207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:25 smithi188 ceph-mon[92425]: pgmap v19207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:27 smithi161 ceph-mon[91579]: pgmap v19208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:27 smithi188 ceph-mon[92425]: pgmap v19208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:29 smithi161 ceph-mon[91579]: pgmap v19209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:29.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:29 smithi188 ceph-mon[92425]: pgmap v19209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:31 smithi188 ceph-mon[92425]: pgmap v19210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:31 smithi161 ceph-mon[91579]: pgmap v19210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:33.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:33 smithi161 ceph-mon[91579]: pgmap v19211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:33 smithi188 ceph-mon[92425]: pgmap v19211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:35 smithi161 ceph-mon[91579]: pgmap v19212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:35 smithi188 ceph-mon[92425]: pgmap v19212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:37 smithi161 ceph-mon[91579]: pgmap v19213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:37 smithi188 ceph-mon[92425]: pgmap v19213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:39 smithi161 ceph-mon[91579]: pgmap v19214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:39.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:39 smithi188 ceph-mon[92425]: pgmap v19214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:41.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:41 smithi188 ceph-mon[92425]: pgmap v19215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:41 smithi161 ceph-mon[91579]: pgmap v19215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:43 smithi161 ceph-mon[91579]: pgmap v19216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:43 smithi188 ceph-mon[92425]: pgmap v19216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:45 smithi161 ceph-mon[91579]: pgmap v19217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:45 smithi188 ceph-mon[92425]: pgmap v19217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:47 smithi161 ceph-mon[91579]: pgmap v19218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:47 smithi188 ceph-mon[92425]: pgmap v19218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:49.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:19:49.165+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:19:49.522 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:19:49.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:19:49.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:19:49.523 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:19:49.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:19:49.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:19:49.524 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:19:49.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:19:49.525 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:19:49.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:19:49.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:19:49.526 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:19:49.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:19:49.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:19:49.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:19:49.527 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:19:49.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:19:49.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:19:49.528 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:19:49.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:19:49.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:19:49.529 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:19:49.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:19:49.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:19:49.530 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:19:49.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:19:49.531 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:19:49.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:19:49.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:19:49.532 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:19:49.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:19:49.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:19:49.533 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:19:49.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:19:49.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:19:49.534 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:19:49.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:19:49.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:19:49.535 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:19:49.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:19:49.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:19:49.536 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:19:49.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:19:49.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:19:49.537 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:19:49.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1169] Local address: 172.21.15.161, port 33440 2022-07-03T21:19:49.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1169] Peer address: 172.21.15.101, port 22 2022-07-03T21:19:49.538 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1169] Connection lost 2022-07-03T21:19:49.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1169] Aborting connection 2022-07-03T21:19:49.539 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:19:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: pgmap v19219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:19:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:19:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:19:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.882 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:49 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: pgmap v19219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:19:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:19:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:19:49.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:49.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:49 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:50.822 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:50 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:19:50.823 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:50 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:19:50.823 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:19:50.824 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:19:50.824 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:19:50.824 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:19:50.825 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:19:50.825 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:19:50.825 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:19:50.826 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:19:50.826 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:19:50.826 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:19:50.827 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:19:50.827 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:19:50.827 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:19:50.828 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:19:50.828 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:19:50.828 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:19:50.829 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:19:50.829 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:19:50.829 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:19:50.830 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:19:50.830 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:19:50.830 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:19:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:19:50.831 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:19:50.832 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:19:50.832 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:19:50.832 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:19:50.833 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:19:50.833 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:19:50.833 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:19:50.834 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:19:50.834 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:19:50.834 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:19:50.835 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:19:50.835 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:19:50.835 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:19:50.836 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:19:50.836 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:19:50.836 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:19:50.837 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:19:50.837 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:19:50.837 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:19:50.838 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:19:50.838 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:19:50.838 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:19:50.839 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1169] Local address: 172.21.15.161, port 33440 2022-07-03T21:19:50.839 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1169] Peer address: 172.21.15.101, port 22 2022-07-03T21:19:50.839 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1169] Connection lost 2022-07-03T21:19:50.840 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1169] Aborting connection 2022-07-03T21:19:50.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:50 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:19:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:50 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:19:50.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:19:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:19:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:19:50.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:19:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:19:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:19:50.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:19:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:19:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:19:50.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:19:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:19:50.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:19:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:19:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:19:50.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:19:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:19:50.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:19:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:19:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:19:50.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:19:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:19:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:19:50.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:19:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:19:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:19:50.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:19:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:19:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:19:50.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:19:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:19:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:19:50.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:19:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:19:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:19:50.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:19:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:19:50.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:19:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:19:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:19:50.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:19:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:19:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:19:50.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:19:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:19:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1169] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:19:50.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1169] Local address: 172.21.15.161, port 33440 2022-07-03T21:19:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1169] Peer address: 172.21.15.101, port 22 2022-07-03T21:19:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1169] Connection lost 2022-07-03T21:19:50.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1169] Aborting connection 2022-07-03T21:19:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:51 smithi161 ceph-mon[91579]: pgmap v19220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:51 smithi188 ceph-mon[92425]: pgmap v19220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:53 smithi161 ceph-mon[91579]: pgmap v19221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:53 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T21:19:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:54.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:53 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:53 smithi188 ceph-mon[92425]: pgmap v19221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:53 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T21:19:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:54.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:53 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:19:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:19:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:55 smithi161 ceph-mon[91579]: pgmap v19222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:55 smithi188 ceph-mon[92425]: pgmap v19222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:19:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:57 smithi161 ceph-mon[91579]: pgmap v19223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:19:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:57 smithi188 ceph-mon[92425]: pgmap v19223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: pgmap v19224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:20:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:20:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:20:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:19:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:20:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: pgmap v19224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:20:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:20:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:20:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:19:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:20:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:20:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:20:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:01 smithi161 ceph-mon[91579]: pgmap v19225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:01 smithi188 ceph-mon[92425]: pgmap v19225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:03 smithi161 ceph-mon[91579]: pgmap v19226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:03 smithi188 ceph-mon[92425]: pgmap v19226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:05 smithi161 ceph-mon[91579]: pgmap v19227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:05 smithi188 ceph-mon[92425]: pgmap v19227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:07 smithi161 ceph-mon[91579]: pgmap v19228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:07 smithi188 ceph-mon[92425]: pgmap v19228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:09 smithi161 ceph-mon[91579]: pgmap v19229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:09 smithi188 ceph-mon[92425]: pgmap v19229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:11.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:11.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:12.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:11 smithi161 ceph-mon[91579]: pgmap v19230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:12.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:11 smithi188 ceph-mon[92425]: pgmap v19230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:13 smithi161 ceph-mon[91579]: pgmap v19231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:13 smithi188 ceph-mon[92425]: pgmap v19231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:15 smithi161 ceph-mon[91579]: pgmap v19232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:16.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:15 smithi188 ceph-mon[92425]: pgmap v19232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:16.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:17 smithi161 ceph-mon[91579]: pgmap v19233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:17 smithi188 ceph-mon[92425]: pgmap v19233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:19 smithi161 ceph-mon[91579]: pgmap v19234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:19 smithi188 ceph-mon[92425]: pgmap v19234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:21.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:21.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:21 smithi188 ceph-mon[92425]: pgmap v19235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:21 smithi161 ceph-mon[91579]: pgmap v19235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:23 smithi188 ceph-mon[92425]: pgmap v19236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:23 smithi161 ceph-mon[91579]: pgmap v19236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:25.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:20:24.963+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T21:20:25.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:25 smithi188 ceph-mon[92425]: pgmap v19237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:25 smithi161 ceph-mon[91579]: pgmap v19237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:26.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:20:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:27 smithi188 ceph-mon[92425]: pgmap v19238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:27 smithi161 ceph-mon[91579]: pgmap v19238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:29 smithi188 ceph-mon[92425]: pgmap v19239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:29 smithi161 ceph-mon[91579]: pgmap v19239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:31 smithi188 ceph-mon[92425]: pgmap v19240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:31 smithi161 ceph-mon[91579]: pgmap v19240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:33 smithi161 ceph-mon[91579]: pgmap v19241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:33 smithi188 ceph-mon[92425]: pgmap v19241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:35 smithi161 ceph-mon[91579]: pgmap v19242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:35.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:35 smithi188 ceph-mon[92425]: pgmap v19242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:37 smithi161 ceph-mon[91579]: pgmap v19243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:37 smithi188 ceph-mon[92425]: pgmap v19243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:39 smithi161 ceph-mon[91579]: pgmap v19244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:39 smithi188 ceph-mon[92425]: pgmap v19244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:41 smithi161 ceph-mon[91579]: pgmap v19245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:41 smithi188 ceph-mon[92425]: pgmap v19245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:43 smithi161 ceph-mon[91579]: pgmap v19246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:43 smithi188 ceph-mon[92425]: pgmap v19246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:45 smithi161 ceph-mon[91579]: pgmap v19247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:45 smithi188 ceph-mon[92425]: pgmap v19247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:47 smithi161 ceph-mon[91579]: pgmap v19248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:47 smithi188 ceph-mon[92425]: pgmap v19248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:49 smithi161 ceph-mon[91579]: pgmap v19249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:49.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:49 smithi188 ceph-mon[92425]: pgmap v19249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:51.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:51 smithi188 ceph-mon[92425]: pgmap v19250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:51 smithi161 ceph-mon[91579]: pgmap v19250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:52.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:20:52.869+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:20:52.873 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:20:52.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:20:52.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:20:52.874 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:20:52.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:20:52.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:20:52.875 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:20:52.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:20:52.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:20:52.876 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:20:52.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:20:52.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:20:52.877 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:20:52.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:20:52.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:20:52.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:20:52.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:20:52.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:20:52.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:20:52.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:20:52.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:20:52.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:20:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:20:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:20:52.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:20:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:20:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:20:52.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:20:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:20:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:20:52.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:20:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:20:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:20:52.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:20:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:20:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:20:52.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:20:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:20:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:20:52.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:20:53.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:20:53.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:20:53.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:20:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:20:53.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:20:53.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1171] Local address: 172.21.15.161, port 33458 2022-07-03T21:20:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1171] Peer address: 172.21.15.101, port 22 2022-07-03T21:20:53.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1171] Connection lost 2022-07-03T21:20:53.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1171] Aborting connection 2022-07-03T21:20:53.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:20:52 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:20:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: pgmap v19251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:53.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:20:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:20:53.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:53 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:20:53.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:20:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:20:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:20:53.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:20:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:20:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:20:53.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:20:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:20:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:20:53.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:20:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:20:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:20:53.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:20:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:20:53.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:20:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:20:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:20:53.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:20:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:20:53.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:20:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:20:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:20:53.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:20:53.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:20:53.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:20:53.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:20:53.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:20:53.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:20:53.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:20:53.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:20:53.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:20:53.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:20:53.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:20:53.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:20:53.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:20:53.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:20:53.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:20:53.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:20:53.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:20:53.647 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:20:53.648 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:20:53.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:20:53.649 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:20:53.650 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:20:53.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:20:53.651 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1171] Local address: 172.21.15.161, port 33458 2022-07-03T21:20:53.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1171] Peer address: 172.21.15.101, port 22 2022-07-03T21:20:53.652 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1171] Connection lost 2022-07-03T21:20:53.653 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1171] Aborting connection 2022-07-03T21:20:53.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: pgmap v19251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:20:53.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:20:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:20:53.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:20:53.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:53 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:20:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:20:53.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:20:53.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:20:53.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:20:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:20:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:20:53.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:20:53.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:20:53.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:20:53.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:20:53.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:20:53.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:20:53.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:20:53.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:20:53.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:20:53.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:20:53.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:20:53.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:20:53.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:20:53.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:20:53.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:20:53.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:20:53.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:20:53.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:20:53.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:20:53.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:20:53.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:20:53.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:20:53.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:20:53.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:20:53.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:20:53.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:20:53.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:20:53.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:20:53.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:20:53.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:20:53.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:20:53.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:20:53.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:20:53.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:20:53.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:20:53.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:20:53.683 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:20:53.684 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:20:53.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1171] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:20:53.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1171] Local address: 172.21.15.161, port 33458 2022-07-03T21:20:53.685 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1171] Peer address: 172.21.15.101, port 22 2022-07-03T21:20:53.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1171] Connection lost 2022-07-03T21:20:53.686 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1171] Aborting connection 2022-07-03T21:20:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:20:55.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:20:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:55 smithi161 ceph-mon[91579]: pgmap v19252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:55 smithi188 ceph-mon[92425]: pgmap v19252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:20:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:57 smithi161 ceph-mon[91579]: pgmap v19253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:20:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:57 smithi188 ceph-mon[92425]: pgmap v19253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: pgmap v19254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:20:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: pgmap v19254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:20:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:01 smithi161 ceph-mon[91579]: pgmap v19255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:01 smithi188 ceph-mon[92425]: pgmap v19255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:03 smithi161 ceph-mon[91579]: pgmap v19256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:03 smithi188 ceph-mon[92425]: pgmap v19256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:05 smithi161 ceph-mon[91579]: pgmap v19257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:05 smithi188 ceph-mon[92425]: pgmap v19257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:07 smithi161 ceph-mon[91579]: pgmap v19258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:07.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:07 smithi188 ceph-mon[92425]: pgmap v19258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:09 smithi161 ceph-mon[91579]: pgmap v19259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:09 smithi188 ceph-mon[92425]: pgmap v19259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:11 smithi161 ceph-mon[91579]: pgmap v19260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:11 smithi188 ceph-mon[92425]: pgmap v19260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:13 smithi161 ceph-mon[91579]: pgmap v19261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:13.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:13 smithi188 ceph-mon[92425]: pgmap v19261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:15 smithi161 ceph-mon[91579]: pgmap v19262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:15.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:15 smithi188 ceph-mon[92425]: pgmap v19262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:17.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:17 smithi161 ceph-mon[91579]: pgmap v19263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:17 smithi188 ceph-mon[92425]: pgmap v19263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:19 smithi161 ceph-mon[91579]: pgmap v19264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:19 smithi188 ceph-mon[92425]: pgmap v19264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:21 smithi161 ceph-mon[91579]: pgmap v19265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:21 smithi188 ceph-mon[92425]: pgmap v19265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:23 smithi161 ceph-mon[91579]: pgmap v19266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:23.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:23 smithi188 ceph-mon[92425]: pgmap v19266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:25 smithi161 ceph-mon[91579]: pgmap v19267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:25 smithi188 ceph-mon[92425]: pgmap v19267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:27 smithi161 ceph-mon[91579]: pgmap v19268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:27 smithi188 ceph-mon[92425]: pgmap v19268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:29 smithi161 ceph-mon[91579]: pgmap v19269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:29 smithi188 ceph-mon[92425]: pgmap v19269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:31 smithi161 ceph-mon[91579]: pgmap v19270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:31 smithi188 ceph-mon[92425]: pgmap v19270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:33 smithi161 ceph-mon[91579]: pgmap v19271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:33 smithi188 ceph-mon[92425]: pgmap v19271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:35 smithi161 ceph-mon[91579]: pgmap v19272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:35 smithi188 ceph-mon[92425]: pgmap v19272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:37 smithi161 ceph-mon[91579]: pgmap v19273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:37 smithi188 ceph-mon[92425]: pgmap v19273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:39 smithi161 ceph-mon[91579]: pgmap v19274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:39 smithi188 ceph-mon[92425]: pgmap v19274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:41 smithi161 ceph-mon[91579]: pgmap v19275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:41 smithi188 ceph-mon[92425]: pgmap v19275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:43 smithi161 ceph-mon[91579]: pgmap v19276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:43 smithi188 ceph-mon[92425]: pgmap v19276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:45 smithi161 ceph-mon[91579]: pgmap v19277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:45 smithi188 ceph-mon[92425]: pgmap v19277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:47.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:47 smithi161 ceph-mon[91579]: pgmap v19278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:47.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:47 smithi188 ceph-mon[92425]: pgmap v19278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:49 smithi161 ceph-mon[91579]: pgmap v19279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:49 smithi188 ceph-mon[92425]: pgmap v19279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:51 smithi161 ceph-mon[91579]: pgmap v19280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:51 smithi188 ceph-mon[92425]: pgmap v19280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:53.836 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:53 smithi188 ceph-mon[92425]: pgmap v19281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:53.843 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-mon[91579]: pgmap v19281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:53.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:21:53.778+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:21:53.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:21:53.844 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:21:53.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:21:53.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:21:53.845 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:21:53.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:21:53.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:21:53.846 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:21:53.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:21:53.847 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:21:53.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:21:53.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:21:53.848 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:21:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:21:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:21:53.849 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:21:53.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:21:53.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:21:53.850 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:21:53.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:21:53.851 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:21:53.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:21:53.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:21:53.852 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:21:53.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:21:53.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:21:53.853 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:21:53.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:21:53.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:21:53.854 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:21:53.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:21:53.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:21:53.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:21:53.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:21:53.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:21:53.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:21:53.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:21:53.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:21:53.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:21:53.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:21:53.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:21:53.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:21:53.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:21:53.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:21:53.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:21:53.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1173] Local address: 172.21.15.161, port 33462 2022-07-03T21:21:53.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1173] Peer address: 172.21.15.101, port 22 2022-07-03T21:21:53.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1173] Connection lost 2022-07-03T21:21:53.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1173] Aborting connection 2022-07-03T21:21:53.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:21:53 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:21:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:21:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:21:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:21:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:21:54.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:21:54.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:21:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:21:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:21:54.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:21:54.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:21:54.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:21:54.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:21:54.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:21:54.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:21:54.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:21:54.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:21:54.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:21:54.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:21:54.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:21:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:21:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1173] Local address: 172.21.15.161, port 33462 2022-07-03T21:21:54.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1173] Peer address: 172.21.15.101, port 22 2022-07-03T21:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1173] Connection lost 2022-07-03T21:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1173] Aborting connection 2022-07-03T21:21:54.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:21:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:21:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:21:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:21:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:21:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:21:54.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:21:54.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:21:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:21:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:21:54.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:21:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:21:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:21:54.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:21:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:21:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:21:54.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:21:54.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:21:54.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:21:54.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:21:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:21:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:21:54.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:21:54.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:21:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:21:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:21:54.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:21:54.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:21:54.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:21:54.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:21:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:21:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:21:54.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:21:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1173] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:21:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1173] Local address: 172.21.15.161, port 33462 2022-07-03T21:21:54.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1173] Peer address: 172.21.15.101, port 22 2022-07-03T21:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1173] Connection lost 2022-07-03T21:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1173] Aborting connection 2022-07-03T21:21:54.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:55 smithi161 ceph-mon[91579]: pgmap v19282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:55 smithi188 ceph-mon[92425]: pgmap v19282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:57 smithi161 ceph-mon[91579]: pgmap v19283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:57 smithi188 ceph-mon[92425]: pgmap v19283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:21:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: pgmap v19284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:59.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:59.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:59.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:21:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: pgmap v19284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:21:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:21:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:21:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:21:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:21:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:22:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:01 smithi161 ceph-mon[91579]: pgmap v19285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:01 smithi188 ceph-mon[92425]: pgmap v19285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:03 smithi161 ceph-mon[91579]: pgmap v19286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:03 smithi188 ceph-mon[92425]: pgmap v19286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:05 smithi161 ceph-mon[91579]: pgmap v19287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:05 smithi188 ceph-mon[92425]: pgmap v19287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:07.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:07 smithi161 ceph-mon[91579]: pgmap v19288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:07 smithi188 ceph-mon[92425]: pgmap v19288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:09.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:09 smithi161 ceph-mon[91579]: pgmap v19289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:09 smithi188 ceph-mon[92425]: pgmap v19289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:11 smithi161 ceph-mon[91579]: pgmap v19290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:11 smithi188 ceph-mon[92425]: pgmap v19290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:13 smithi161 ceph-mon[91579]: pgmap v19291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:13 smithi188 ceph-mon[92425]: pgmap v19291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:14.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:15 smithi161 ceph-mon[91579]: pgmap v19292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:15 smithi188 ceph-mon[92425]: pgmap v19292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:17 smithi161 ceph-mon[91579]: pgmap v19293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:17 smithi188 ceph-mon[92425]: pgmap v19293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:19 smithi161 ceph-mon[91579]: pgmap v19294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:19 smithi188 ceph-mon[92425]: pgmap v19294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:21 smithi161 ceph-mon[91579]: pgmap v19295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:21 smithi188 ceph-mon[92425]: pgmap v19295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:23.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:23 smithi161 ceph-mon[91579]: pgmap v19296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:23 smithi188 ceph-mon[92425]: pgmap v19296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:25 smithi161 ceph-mon[91579]: pgmap v19297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:25 smithi188 ceph-mon[92425]: pgmap v19297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:27 smithi161 ceph-mon[91579]: pgmap v19298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:27 smithi188 ceph-mon[92425]: pgmap v19298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:29 smithi161 ceph-mon[91579]: pgmap v19299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:29.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:29 smithi188 ceph-mon[92425]: pgmap v19299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:31 smithi161 ceph-mon[91579]: pgmap v19300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:31 smithi188 ceph-mon[92425]: pgmap v19300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:33 smithi161 ceph-mon[91579]: pgmap v19301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:33 smithi188 ceph-mon[92425]: pgmap v19301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:35 smithi161 ceph-mon[91579]: pgmap v19302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:35 smithi188 ceph-mon[92425]: pgmap v19302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:37 smithi161 ceph-mon[91579]: pgmap v19303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:37 smithi188 ceph-mon[92425]: pgmap v19303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:39 smithi161 ceph-mon[91579]: pgmap v19304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:39 smithi188 ceph-mon[92425]: pgmap v19304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:39.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:41 smithi161 ceph-mon[91579]: pgmap v19305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:41 smithi188 ceph-mon[92425]: pgmap v19305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:43.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:43 smithi161 ceph-mon[91579]: pgmap v19306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:43.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:43 smithi188 ceph-mon[92425]: pgmap v19306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:44.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:45 smithi161 ceph-mon[91579]: pgmap v19307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:45 smithi188 ceph-mon[92425]: pgmap v19307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:47 smithi161 ceph-mon[91579]: pgmap v19308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:47 smithi188 ceph-mon[92425]: pgmap v19308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:49 smithi161 ceph-mon[91579]: pgmap v19309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:49 smithi188 ceph-mon[92425]: pgmap v19309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:51 smithi161 ceph-mon[91579]: pgmap v19310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:51 smithi188 ceph-mon[92425]: pgmap v19310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:53 smithi161 ceph-mon[91579]: pgmap v19311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:53.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:53 smithi188 ceph-mon[92425]: pgmap v19311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:22:54.601 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:22:54.284+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:22:54.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:22:54.619 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:22:54.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:22:54.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:22:54.620 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:22:54.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:22:54.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:22:54.621 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:22:54.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:22:54.622 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:22:54.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:22:54.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:22:54.623 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:22:54.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:22:54.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:22:54.624 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:22:54.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:22:54.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:22:54.625 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:22:54.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:22:54.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:22:54.626 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:22:54.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:22:54.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:22:54.627 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:22:54.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:22:54.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:22:54.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:22:54.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:22:54.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:22:54.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:22:54.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:22:54.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:22:54.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:22:54.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:22:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:22:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:22:54.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:22:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:22:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:22:54.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:22:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:22:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:22:54.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:22:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:22:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1175] Local address: 172.21.15.161, port 33466 2022-07-03T21:22:54.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1175] Peer address: 172.21.15.101, port 22 2022-07-03T21:22:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1175] Connection lost 2022-07-03T21:22:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1175] Aborting connection 2022-07-03T21:22:54.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:22:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:54.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:22:54.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:22:54.881 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:22:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:22:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:22:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:22:54.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:22:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:55 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:22:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:55 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:22:55.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:22:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:22:55.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:22:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:22:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:22:55.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:22:55.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:22:55.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:22:55.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:22:55.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:22:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:22:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:22:55.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:22:55.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:22:55.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:22:55.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:22:55.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:22:55.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:22:55.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:22:55.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:22:55.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1175] Local address: 172.21.15.161, port 33466 2022-07-03T21:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1175] Peer address: 172.21.15.101, port 22 2022-07-03T21:22:55.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1175] Connection lost 2022-07-03T21:22:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1175] Aborting connection 2022-07-03T21:22:55.896 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:55 smithi161 ceph-mon[91579]: pgmap v19312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:55 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:22:55.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:55 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:22:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:22:55.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:22:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:22:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:22:55.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:22:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:22:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:22:55.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:22:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:22:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:22:55.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:22:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:22:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:22:55.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:22:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:22:55.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:22:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:22:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:22:55.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:22:55.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:22:55.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:22:55.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:22:55.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:22:55.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:22:55.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:22:55.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:22:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:22:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:22:55.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1175] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:22:55.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1175] Local address: 172.21.15.161, port 33466 2022-07-03T21:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1175] Peer address: 172.21.15.101, port 22 2022-07-03T21:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1175] Connection lost 2022-07-03T21:22:55.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1175] Aborting connection 2022-07-03T21:22:55.931 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:55 smithi188 ceph-mon[92425]: pgmap v19312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:22:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: pgmap v19313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:22:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.384 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:57 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: pgmap v19313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:22:58.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:22:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:58.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:57 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:22:59.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:58 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 36305k 2022-07-03T21:22:59.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:58 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 37177106: error parsing value: Value '37177106' is below minimum 939524096 2022-07-03T21:22:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:58 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 36305k 2022-07-03T21:22:59.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:58 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 37177106: error parsing value: Value '37177106' is below minimum 939524096 2022-07-03T21:23:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: pgmap v19314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:23:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:23:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:22:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: pgmap v19314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:23:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:23:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:22:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:01 smithi161 ceph-mon[91579]: pgmap v19315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:23:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:01 smithi188 ceph-mon[92425]: pgmap v19315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:23:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:03 smithi161 ceph-mon[91579]: pgmap v19316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:23:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:03 smithi188 ceph-mon[92425]: pgmap v19316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:23:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:05.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:05 smithi161 ceph-mon[91579]: pgmap v19317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:05 smithi188 ceph-mon[92425]: pgmap v19317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:07.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:07 smithi161 ceph-mon[91579]: pgmap v19318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:07 smithi188 ceph-mon[92425]: pgmap v19318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:09 smithi161 ceph-mon[91579]: pgmap v19319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:09.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:09 smithi188 ceph-mon[92425]: pgmap v19319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:11 smithi161 ceph-mon[91579]: pgmap v19320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:11 smithi188 ceph-mon[92425]: pgmap v19320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:13.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:13 smithi161 ceph-mon[91579]: pgmap v19321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:13 smithi188 ceph-mon[92425]: pgmap v19321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:14.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:15.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:15 smithi161 ceph-mon[91579]: pgmap v19322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:15 smithi188 ceph-mon[92425]: pgmap v19322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:17.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:17 smithi161 ceph-mon[91579]: pgmap v19323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:17 smithi188 ceph-mon[92425]: pgmap v19323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:19.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:19 smithi161 ceph-mon[91579]: pgmap v19324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:19.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:19 smithi188 ceph-mon[92425]: pgmap v19324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:21 smithi161 ceph-mon[91579]: pgmap v19325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:21 smithi188 ceph-mon[92425]: pgmap v19325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:23 smithi161 ceph-mon[91579]: pgmap v19326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:23 smithi188 ceph-mon[92425]: pgmap v19326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:24.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:25.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:25 smithi161 ceph-mon[91579]: pgmap v19327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:25 smithi188 ceph-mon[92425]: pgmap v19327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:27.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:27 smithi161 ceph-mon[91579]: pgmap v19328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:27 smithi188 ceph-mon[92425]: pgmap v19328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:29.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:29 smithi161 ceph-mon[91579]: pgmap v19329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:29.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:29 smithi188 ceph-mon[92425]: pgmap v19329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:31 smithi161 ceph-mon[91579]: pgmap v19330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:31 smithi188 ceph-mon[92425]: pgmap v19330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:33.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:33 smithi161 ceph-mon[91579]: pgmap v19331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:33.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:33 smithi188 ceph-mon[92425]: pgmap v19331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:34.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:35.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:35 smithi161 ceph-mon[91579]: pgmap v19332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:35 smithi188 ceph-mon[92425]: pgmap v19332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:37.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:37 smithi161 ceph-mon[91579]: pgmap v19333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:37 smithi188 ceph-mon[92425]: pgmap v19333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:39.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:39 smithi161 ceph-mon[91579]: pgmap v19334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:39.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:39 smithi188 ceph-mon[92425]: pgmap v19334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:41 smithi161 ceph-mon[91579]: pgmap v19335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:41 smithi188 ceph-mon[92425]: pgmap v19335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:43.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:43 smithi161 ceph-mon[91579]: pgmap v19336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:43 smithi188 ceph-mon[92425]: pgmap v19336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:44.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:45.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:45 smithi161 ceph-mon[91579]: pgmap v19337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:45 smithi188 ceph-mon[92425]: pgmap v19337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:47.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:47 smithi161 ceph-mon[91579]: pgmap v19338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:47 smithi188 ceph-mon[92425]: pgmap v19338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:49.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:49 smithi161 ceph-mon[91579]: pgmap v19339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:49.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:49 smithi188 ceph-mon[92425]: pgmap v19339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:51 smithi161 ceph-mon[91579]: pgmap v19340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:51 smithi188 ceph-mon[92425]: pgmap v19340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:53.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:53 smithi161 ceph-mon[91579]: pgmap v19341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:53 smithi188 ceph-mon[92425]: pgmap v19341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:54.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:55.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:55 smithi161 ceph-mon[91579]: pgmap v19342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:55 smithi188 ceph-mon[92425]: pgmap v19342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:57.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-mon[91579]: pgmap v19343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:23:57.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:23:57.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:23:57.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:23:57.437+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:23:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:23:57.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:23:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:23:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:23:57.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:23:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:23:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:23:57.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:23:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:23:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:23:57.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:23:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:23:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:23:57.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:23:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:23:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:23:57.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:23:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:23:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:23:57.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:23:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:23:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:23:57.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:23:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:23:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:23:57.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:23:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:23:57.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:23:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:23:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:23:57.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:23:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:23:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:23:57.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:23:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:23:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:23:57.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:23:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:23:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:23:57.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:23:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:23:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:23:57.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:23:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:23:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:23:57.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1177] Local address: 172.21.15.161, port 33474 2022-07-03T21:23:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1177] Peer address: 172.21.15.101, port 22 2022-07-03T21:23:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1177] Connection lost 2022-07-03T21:23:57.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1177] Aborting connection 2022-07-03T21:23:57.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:23:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:57 smithi188 ceph-mon[92425]: pgmap v19343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:23:57.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:23:57.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:57 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:23:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:23:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:23:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:23:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:23:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:23:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:23:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:23:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:23:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:23:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:23:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:23:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:23:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:23:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:23:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:23:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:23:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:23:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:23:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:23:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:23:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:23:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:23:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:23:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:23:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:23:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:23:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:23:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:23:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:23:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:23:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:23:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:23:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:23:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:23:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:23:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:23:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:23:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:23:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:23:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:23:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:23:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:23:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:23:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:23:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:23:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1177] Local address: 172.21.15.161, port 33474 2022-07-03T21:23:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1177] Peer address: 172.21.15.101, port 22 2022-07-03T21:23:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1177] Connection lost 2022-07-03T21:23:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1177] Aborting connection 2022-07-03T21:23:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:23:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:23:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:23:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:23:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:23:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:23:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:23:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:23:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:23:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:23:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:23:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:23:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:23:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:23:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:23:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:23:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:23:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:23:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:23:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:23:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:23:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:23:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:23:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:23:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:23:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:23:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:23:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:23:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:23:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:23:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:23:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:23:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:23:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:23:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:23:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:23:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:23:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:23:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:23:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:23:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:23:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:23:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:23:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:23:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:23:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:23:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1177] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:23:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1177] Local address: 172.21.15.161, port 33474 2022-07-03T21:23:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1177] Peer address: 172.21.15.101, port 22 2022-07-03T21:23:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1177] Connection lost 2022-07-03T21:23:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1177] Aborting connection 2022-07-03T21:23:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: pgmap v19344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:23:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:23:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:23:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:23:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:23:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:24:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: pgmap v19344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:24:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:24:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:24:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:23:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:24:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:01 smithi161 ceph-mon[91579]: pgmap v19345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:01 smithi188 ceph-mon[92425]: pgmap v19345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:03.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:03 smithi161 ceph-mon[91579]: pgmap v19346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:03 smithi188 ceph-mon[92425]: pgmap v19346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:04.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:05 smithi188 ceph-mon[92425]: pgmap v19347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:05 smithi161 ceph-mon[91579]: pgmap v19347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:07 smithi188 ceph-mon[92425]: pgmap v19348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:07 smithi161 ceph-mon[91579]: pgmap v19348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:09 smithi188 ceph-mon[92425]: pgmap v19349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:09 smithi161 ceph-mon[91579]: pgmap v19349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:11 smithi161 ceph-mon[91579]: pgmap v19350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:11 smithi188 ceph-mon[92425]: pgmap v19350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:13.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:13 smithi161 ceph-mon[91579]: pgmap v19351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:13 smithi188 ceph-mon[92425]: pgmap v19351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:15 smithi188 ceph-mon[92425]: pgmap v19352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:15 smithi161 ceph-mon[91579]: pgmap v19352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:17 smithi188 ceph-mon[92425]: pgmap v19353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:17 smithi161 ceph-mon[91579]: pgmap v19353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:19.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:19 smithi188 ceph-mon[92425]: pgmap v19354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:19 smithi161 ceph-mon[91579]: pgmap v19354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:21 smithi161 ceph-mon[91579]: pgmap v19355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:21 smithi188 ceph-mon[92425]: pgmap v19355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:23.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:23 smithi161 ceph-mon[91579]: pgmap v19356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:23 smithi188 ceph-mon[92425]: pgmap v19356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:25 smithi188 ceph-mon[92425]: pgmap v19357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:25 smithi161 ceph-mon[91579]: pgmap v19357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:27.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:27 smithi188 ceph-mon[92425]: pgmap v19358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:27 smithi161 ceph-mon[91579]: pgmap v19358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:29 smithi188 ceph-mon[92425]: pgmap v19359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:29.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:29 smithi161 ceph-mon[91579]: pgmap v19359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:31 smithi161 ceph-mon[91579]: pgmap v19360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:31 smithi188 ceph-mon[92425]: pgmap v19360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:33 smithi188 ceph-mon[92425]: pgmap v19361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:33 smithi161 ceph-mon[91579]: pgmap v19361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:34.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:35 smithi188 ceph-mon[92425]: pgmap v19362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:35 smithi161 ceph-mon[91579]: pgmap v19362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:37 smithi188 ceph-mon[92425]: pgmap v19363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:37 smithi161 ceph-mon[91579]: pgmap v19363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:39 smithi188 ceph-mon[92425]: pgmap v19364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:39.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:39 smithi161 ceph-mon[91579]: pgmap v19364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:41 smithi161 ceph-mon[91579]: pgmap v19365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:41 smithi188 ceph-mon[92425]: pgmap v19365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:43 smithi188 ceph-mon[92425]: pgmap v19366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:43 smithi161 ceph-mon[91579]: pgmap v19366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:44.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:45 smithi188 ceph-mon[92425]: pgmap v19367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:45 smithi161 ceph-mon[91579]: pgmap v19367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:47.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:47 smithi188 ceph-mon[92425]: pgmap v19368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:47 smithi161 ceph-mon[91579]: pgmap v19368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:49 smithi188 ceph-mon[92425]: pgmap v19369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:49.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:49 smithi161 ceph-mon[91579]: pgmap v19369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:51.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:51 smithi161 ceph-mon[91579]: pgmap v19370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:51 smithi188 ceph-mon[92425]: pgmap v19370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:53 smithi188 ceph-mon[92425]: pgmap v19371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:53 smithi161 ceph-mon[91579]: pgmap v19371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:55.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:55 smithi188 ceph-mon[92425]: pgmap v19372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:55 smithi161 ceph-mon[91579]: pgmap v19372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:57.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:57 smithi188 ceph-mon[92425]: pgmap v19373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:57.947 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-mon[91579]: pgmap v19373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:24:58.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:24:57.945+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:24:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:24:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:24:58.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:24:58.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:24:58.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:24:58.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:24:58.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:24:58.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:24:58.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:24:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:24:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:24:58.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:24:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:24:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:24:58.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:24:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:24:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:24:58.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:24:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:24:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:24:58.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:24:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:24:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:24:58.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:24:58.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:24:58.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:24:58.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:24:58.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:24:58.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:24:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:24:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:24:58.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:24:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:24:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:24:58.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:24:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:24:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:24:58.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:24:58.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:24:58.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:24:58.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:24:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:24:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:24:58.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:24:58.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:24:58.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1179] Local address: 172.21.15.161, port 33502 2022-07-03T21:24:58.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1179] Peer address: 172.21.15.101, port 22 2022-07-03T21:24:58.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1179] Connection lost 2022-07-03T21:24:58.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1179] Aborting connection 2022-07-03T21:24:58.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:24:57 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:24:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:24:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:24:58.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:58 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:58 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:24:58.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:24:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:24:58.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:24:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:24:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:24:58.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:24:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:24:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:24:58.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:24:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:24:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:24:58.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:24:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:24:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:24:58.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:24:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:24:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:24:58.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:24:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:24:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:24:58.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:24:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:24:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:24:58.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:24:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:24:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:24:58.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:24:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:24:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:24:58.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:24:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:24:58.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:24:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:24:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:24:58.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:24:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:24:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:24:58.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:24:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:24:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:24:58.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:24:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:24:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:24:58.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:24:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:24:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1179] Local address: 172.21.15.161, port 33502 2022-07-03T21:24:58.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1179] Peer address: 172.21.15.101, port 22 2022-07-03T21:24:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1179] Connection lost 2022-07-03T21:24:58.932 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1179] Aborting connection 2022-07-03T21:24:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:24:59.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:58 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:24:59.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:58 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:24:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:24:59.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:24:59.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:24:59.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:24:59.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:24:59.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:24:59.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:24:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:24:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:24:59.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:24:59.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:24:59.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:24:59.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:24:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:24:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:24:59.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:24:59.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:24:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:24:59.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:24:59.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:24:59.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:24:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:24:59.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:24:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:24:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1179] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:24:59.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1179] Local address: 172.21.15.161, port 33502 2022-07-03T21:24:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1179] Peer address: 172.21.15.101, port 22 2022-07-03T21:24:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1179] Connection lost 2022-07-03T21:24:59.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1179] Aborting connection 2022-07-03T21:24:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: pgmap v19374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:24:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:24:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:24:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:24:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:24:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:24:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:25:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: pgmap v19374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:25:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:25:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:25:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:24:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:25:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:01 smithi161 ceph-mon[91579]: pgmap v19375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:01 smithi188 ceph-mon[92425]: pgmap v19375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:03 smithi188 ceph-mon[92425]: pgmap v19376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:03 smithi161 ceph-mon[91579]: pgmap v19376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:05.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:05 smithi188 ceph-mon[92425]: pgmap v19377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:05 smithi161 ceph-mon[91579]: pgmap v19377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:07.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:07 smithi188 ceph-mon[92425]: pgmap v19378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:07 smithi161 ceph-mon[91579]: pgmap v19378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:09 smithi188 ceph-mon[92425]: pgmap v19379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:09.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:09 smithi161 ceph-mon[91579]: pgmap v19379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:11 smithi161 ceph-mon[91579]: pgmap v19380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:11 smithi188 ceph-mon[92425]: pgmap v19380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:13.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:13 smithi188 ceph-mon[92425]: pgmap v19381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:13 smithi161 ceph-mon[91579]: pgmap v19381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:14.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:15.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:15 smithi188 ceph-mon[92425]: pgmap v19382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:15 smithi161 ceph-mon[91579]: pgmap v19382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:17.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:17 smithi188 ceph-mon[92425]: pgmap v19383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:17 smithi161 ceph-mon[91579]: pgmap v19383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:19 smithi188 ceph-mon[92425]: pgmap v19384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:19.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:19 smithi161 ceph-mon[91579]: pgmap v19384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:21 smithi161 ceph-mon[91579]: pgmap v19385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:21 smithi188 ceph-mon[92425]: pgmap v19385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:23.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:23 smithi188 ceph-mon[92425]: pgmap v19386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:23 smithi161 ceph-mon[91579]: pgmap v19386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:24.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:25.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:25 smithi188 ceph-mon[92425]: pgmap v19387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:25 smithi161 ceph-mon[91579]: pgmap v19387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:27 smithi161 ceph-mon[91579]: pgmap v19388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:27 smithi188 ceph-mon[92425]: pgmap v19388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:29 smithi161 ceph-mon[91579]: pgmap v19389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:29 smithi188 ceph-mon[92425]: pgmap v19389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:31 smithi161 ceph-mon[91579]: pgmap v19390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:31 smithi188 ceph-mon[92425]: pgmap v19390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:33.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:33 smithi188 ceph-mon[92425]: pgmap v19391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:33 smithi161 ceph-mon[91579]: pgmap v19391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:34.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:35.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:35 smithi188 ceph-mon[92425]: pgmap v19392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:35 smithi161 ceph-mon[91579]: pgmap v19392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:37.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:37 smithi188 ceph-mon[92425]: pgmap v19393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:37 smithi161 ceph-mon[91579]: pgmap v19393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:39 smithi161 ceph-mon[91579]: pgmap v19394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:39 smithi188 ceph-mon[92425]: pgmap v19394: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:41 smithi161 ceph-mon[91579]: pgmap v19395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:41 smithi188 ceph-mon[92425]: pgmap v19395: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:43.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:43 smithi188 ceph-mon[92425]: pgmap v19396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:43 smithi161 ceph-mon[91579]: pgmap v19396: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:44.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:45.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:45 smithi188 ceph-mon[92425]: pgmap v19397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:45 smithi161 ceph-mon[91579]: pgmap v19397: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:47 smithi161 ceph-mon[91579]: pgmap v19398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:47 smithi188 ceph-mon[92425]: pgmap v19398: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:49 smithi161 ceph-mon[91579]: pgmap v19399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:49 smithi188 ceph-mon[92425]: pgmap v19399: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:51 smithi161 ceph-mon[91579]: pgmap v19400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:51 smithi188 ceph-mon[92425]: pgmap v19400: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:53.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:53 smithi188 ceph-mon[92425]: pgmap v19401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:53 smithi161 ceph-mon[91579]: pgmap v19401: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:25:54.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:25:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:55 smithi161 ceph-mon[91579]: pgmap v19402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:55 smithi188 ceph-mon[92425]: pgmap v19402: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:58.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:57 smithi161 ceph-mon[91579]: pgmap v19403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:57 smithi188 ceph-mon[92425]: pgmap v19403: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:25:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:25:58.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:25:58.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:25:58.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:25:58.442+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:25:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:25:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:25:58.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:25:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:25:58.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:25:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:25:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:25:58.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:25:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:25:58.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:25:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:25:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:25:58.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:25:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:25:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:25:58.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:25:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:25:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:25:58.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:25:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:25:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:25:58.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:25:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:25:58.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:25:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:25:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:25:58.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:25:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:25:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:25:58.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:25:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:25:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:25:58.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:25:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:25:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:25:58.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:25:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:25:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:25:58.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:25:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:25:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:25:58.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:25:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:25:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:25:58.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:25:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1181] Local address: 172.21.15.161, port 33506 2022-07-03T21:25:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1181] Peer address: 172.21.15.101, port 22 2022-07-03T21:25:58.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1181] Connection lost 2022-07-03T21:25:58.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1181] Aborting connection 2022-07-03T21:25:58.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:25:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:25:58.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:25:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:25:58.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:58 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:26:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:26:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:26:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:26:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:26:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:26:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:26:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:26:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:26:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:26:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:26:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:26:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:26:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:26:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:26:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:26:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:26:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:26:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:26:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:26:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:26:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:26:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:26:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:26:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:26:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:26:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:26:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:26:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:26:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:26:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:26:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:26:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:26:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:26:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:26:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:26:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:26:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:26:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:26:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:26:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:26:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:26:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:26:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:26:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:26:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:26:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:26:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:26:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1181] Local address: 172.21.15.161, port 33506 2022-07-03T21:26:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1181] Peer address: 172.21.15.101, port 22 2022-07-03T21:26:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1181] Connection lost 2022-07-03T21:26:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1181] Aborting connection 2022-07-03T21:26:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: pgmap v19404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:26:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:26:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:26:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:25:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:26:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:26:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:26:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:26:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:26:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:26:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:26:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:26:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:26:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:26:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:26:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:26:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:26:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:26:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:26:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:26:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:26:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:26:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:26:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:26:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:26:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:26:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:26:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:26:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:26:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:26:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:26:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:26:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:26:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:26:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:26:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:26:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:26:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:26:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:26:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:26:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:26:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:26:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:26:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:26:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:26:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:26:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:26:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:26:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:26:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:26:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:26:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1181] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:26:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1181] Local address: 172.21.15.161, port 33506 2022-07-03T21:26:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1181] Peer address: 172.21.15.101, port 22 2022-07-03T21:26:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1181] Connection lost 2022-07-03T21:26:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1181] Aborting connection 2022-07-03T21:26:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: pgmap v19404: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:26:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:26:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:26:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:25:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:26:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:01 smithi161 ceph-mon[91579]: pgmap v19405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:01 smithi188 ceph-mon[92425]: pgmap v19405: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:03 smithi188 ceph-mon[92425]: pgmap v19406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:03 smithi161 ceph-mon[91579]: pgmap v19406: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:05 smithi161 ceph-mon[91579]: pgmap v19407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:05 smithi188 ceph-mon[92425]: pgmap v19407: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:07 smithi161 ceph-mon[91579]: pgmap v19408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:07 smithi188 ceph-mon[92425]: pgmap v19408: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:09 smithi161 ceph-mon[91579]: pgmap v19409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:09 smithi188 ceph-mon[92425]: pgmap v19409: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:11 smithi161 ceph-mon[91579]: pgmap v19410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:11 smithi188 ceph-mon[92425]: pgmap v19410: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:13 smithi161 ceph-mon[91579]: pgmap v19411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:13 smithi188 ceph-mon[92425]: pgmap v19411: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:15 smithi161 ceph-mon[91579]: pgmap v19412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:15 smithi188 ceph-mon[92425]: pgmap v19412: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:17 smithi161 ceph-mon[91579]: pgmap v19413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:17 smithi188 ceph-mon[92425]: pgmap v19413: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:19 smithi161 ceph-mon[91579]: pgmap v19414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:19 smithi188 ceph-mon[92425]: pgmap v19414: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:21 smithi161 ceph-mon[91579]: pgmap v19415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:21 smithi188 ceph-mon[92425]: pgmap v19415: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:23 smithi161 ceph-mon[91579]: pgmap v19416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:23 smithi188 ceph-mon[92425]: pgmap v19416: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:25.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:25 smithi161 ceph-mon[91579]: pgmap v19417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:25 smithi188 ceph-mon[92425]: pgmap v19417: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:27 smithi161 ceph-mon[91579]: pgmap v19418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:27 smithi188 ceph-mon[92425]: pgmap v19418: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:29 smithi161 ceph-mon[91579]: pgmap v19419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:29 smithi188 ceph-mon[92425]: pgmap v19419: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:31 smithi161 ceph-mon[91579]: pgmap v19420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:31 smithi188 ceph-mon[92425]: pgmap v19420: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:33 smithi161 ceph-mon[91579]: pgmap v19421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:33 smithi188 ceph-mon[92425]: pgmap v19421: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:35.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:35 smithi161 ceph-mon[91579]: pgmap v19422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:35 smithi188 ceph-mon[92425]: pgmap v19422: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:37 smithi161 ceph-mon[91579]: pgmap v19423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:37 smithi188 ceph-mon[92425]: pgmap v19423: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:39 smithi161 ceph-mon[91579]: pgmap v19424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:39 smithi188 ceph-mon[92425]: pgmap v19424: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:41 smithi161 ceph-mon[91579]: pgmap v19425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:41 smithi188 ceph-mon[92425]: pgmap v19425: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:43 smithi161 ceph-mon[91579]: pgmap v19426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:43 smithi188 ceph-mon[92425]: pgmap v19426: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:45 smithi161 ceph-mon[91579]: pgmap v19427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:45 smithi188 ceph-mon[92425]: pgmap v19427: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:47 smithi161 ceph-mon[91579]: pgmap v19428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:47 smithi188 ceph-mon[92425]: pgmap v19428: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:49 smithi161 ceph-mon[91579]: pgmap v19429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:49 smithi188 ceph-mon[92425]: pgmap v19429: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:51 smithi161 ceph-mon[91579]: pgmap v19430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:51 smithi188 ceph-mon[92425]: pgmap v19430: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:53 smithi161 ceph-mon[91579]: pgmap v19431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:53 smithi188 ceph-mon[92425]: pgmap v19431: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:26:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:26:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:55 smithi161 ceph-mon[91579]: pgmap v19432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:55 smithi188 ceph-mon[92425]: pgmap v19432: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:57 smithi161 ceph-mon[91579]: pgmap v19433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:57 smithi188 ceph-mon[92425]: pgmap v19433: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:26:59.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:26:58.945+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:26:59.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:26:59.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:26:59.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:26:59.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:26:59.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:26:59.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:26:59.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:26:59.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:26:59.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:26:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:26:59.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:26:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:26:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:26:59.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:26:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:26:59.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:26:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:26:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:26:59.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:26:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:26:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:26:59.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:26:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:26:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:26:59.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:26:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:26:59.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:26:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:26:59.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:26:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:26:59.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:26:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:26:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:26:59.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:26:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:26:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:26:59.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:26:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:26:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:26:59.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:26:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:26:59.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:26:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:26:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:26:59.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:26:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1183] Local address: 172.21.15.161, port 33510 2022-07-03T21:26:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1183] Peer address: 172.21.15.101, port 22 2022-07-03T21:26:59.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1183] Connection lost 2022-07-03T21:26:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1183] Aborting connection 2022-07-03T21:26:59.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:26:58 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:27:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: pgmap v19434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:27:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:27:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:27:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:27:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:27:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:27:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:27:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:27:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:27:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:27:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:27:00.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:27:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:27:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:27:00.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:27:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:27:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:27:00.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:27:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:27:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:27:00.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:27:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:27:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:27:00.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:27:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:27:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:27:00.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:27:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:27:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:27:00.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:27:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:27:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:27:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:27:00.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:27:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:27:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:27:00.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:27:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:27:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:27:00.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:27:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:27:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:27:00.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:27:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:27:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:27:00.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:27:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:27:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:27:00.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:27:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:27:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1183] Local address: 172.21.15.161, port 33510 2022-07-03T21:27:00.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1183] Peer address: 172.21.15.101, port 22 2022-07-03T21:27:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1183] Connection lost 2022-07-03T21:27:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1183] Aborting connection 2022-07-03T21:27:00.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:00.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:00.149 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:26:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: pgmap v19434: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:27:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:27:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:27:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:27:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:27:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:27:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:27:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:27:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:27:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:27:00.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:27:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:27:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:27:00.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:27:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:27:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:27:00.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:27:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:27:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:27:00.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:27:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:27:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:27:00.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:27:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:27:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:27:00.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:27:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:27:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:27:00.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:27:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:27:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:27:00.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:27:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:27:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:27:00.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:27:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:27:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:27:00.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:27:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:27:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:27:00.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:27:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:27:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:27:00.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:27:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:27:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:27:00.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:27:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:27:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:27:00.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1183] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:27:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1183] Local address: 172.21.15.161, port 33510 2022-07-03T21:27:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1183] Peer address: 172.21.15.101, port 22 2022-07-03T21:27:00.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1183] Connection lost 2022-07-03T21:27:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1183] Aborting connection 2022-07-03T21:27:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:00.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:00.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:26:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:01 smithi161 ceph-mon[91579]: pgmap v19435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:01.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:01 smithi188 ceph-mon[92425]: pgmap v19435: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:03 smithi161 ceph-mon[91579]: pgmap v19436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:03 smithi188 ceph-mon[92425]: pgmap v19436: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:05 smithi161 ceph-mon[91579]: pgmap v19437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:05 smithi188 ceph-mon[92425]: pgmap v19437: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:07 smithi161 ceph-mon[91579]: pgmap v19438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:07 smithi188 ceph-mon[92425]: pgmap v19438: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:09 smithi161 ceph-mon[91579]: pgmap v19439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:09 smithi188 ceph-mon[92425]: pgmap v19439: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:11 smithi161 ceph-mon[91579]: pgmap v19440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:11 smithi188 ceph-mon[92425]: pgmap v19440: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:13 smithi161 ceph-mon[91579]: pgmap v19441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:13 smithi188 ceph-mon[92425]: pgmap v19441: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:15 smithi161 ceph-mon[91579]: pgmap v19442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:15 smithi188 ceph-mon[92425]: pgmap v19442: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:17 smithi161 ceph-mon[91579]: pgmap v19443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:17 smithi188 ceph-mon[92425]: pgmap v19443: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:19 smithi161 ceph-mon[91579]: pgmap v19444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:19 smithi188 ceph-mon[92425]: pgmap v19444: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:21 smithi161 ceph-mon[91579]: pgmap v19445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:21 smithi188 ceph-mon[92425]: pgmap v19445: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:23 smithi161 ceph-mon[91579]: pgmap v19446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:23 smithi188 ceph-mon[92425]: pgmap v19446: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:25 smithi161 ceph-mon[91579]: pgmap v19447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:25 smithi188 ceph-mon[92425]: pgmap v19447: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:27 smithi161 ceph-mon[91579]: pgmap v19448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:27 smithi188 ceph-mon[92425]: pgmap v19448: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:29 smithi161 ceph-mon[91579]: pgmap v19449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:29 smithi188 ceph-mon[92425]: pgmap v19449: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:31 smithi161 ceph-mon[91579]: pgmap v19450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:31 smithi188 ceph-mon[92425]: pgmap v19450: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:33 smithi161 ceph-mon[91579]: pgmap v19451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:33 smithi188 ceph-mon[92425]: pgmap v19451: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:36.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:35 smithi161 ceph-mon[91579]: pgmap v19452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:35 smithi188 ceph-mon[92425]: pgmap v19452: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:37 smithi161 ceph-mon[91579]: pgmap v19453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:37 smithi188 ceph-mon[92425]: pgmap v19453: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:39 smithi161 ceph-mon[91579]: pgmap v19454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:39 smithi188 ceph-mon[92425]: pgmap v19454: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:41 smithi161 ceph-mon[91579]: pgmap v19455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:41 smithi188 ceph-mon[92425]: pgmap v19455: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:43 smithi161 ceph-mon[91579]: pgmap v19456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:43 smithi188 ceph-mon[92425]: pgmap v19456: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:45 smithi161 ceph-mon[91579]: pgmap v19457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:45 smithi188 ceph-mon[92425]: pgmap v19457: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:47 smithi161 ceph-mon[91579]: pgmap v19458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:47 smithi188 ceph-mon[92425]: pgmap v19458: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:49 smithi161 ceph-mon[91579]: pgmap v19459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:49 smithi188 ceph-mon[92425]: pgmap v19459: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:51 smithi161 ceph-mon[91579]: pgmap v19460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:51 smithi188 ceph-mon[92425]: pgmap v19460: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:53 smithi161 ceph-mon[91579]: pgmap v19461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:53 smithi188 ceph-mon[92425]: pgmap v19461: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:55 smithi161 ceph-mon[91579]: pgmap v19462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:55 smithi188 ceph-mon[92425]: pgmap v19462: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:57 smithi161 ceph-mon[91579]: pgmap v19463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:57 smithi188 ceph-mon[92425]: pgmap v19463: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:27:59.706 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:27:59.448+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:27:59.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:27:59.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:27:59.707 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:27:59.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:27:59.708 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:27:59.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:27:59.709 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:27:59.710 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:27:59.710 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:27:59.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:27:59.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:27:59.711 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:27:59.712 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:27:59.712 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:27:59.712 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:27:59.713 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:27:59.713 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:27:59.713 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:27:59.714 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:27:59.714 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:27:59.714 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:27:59.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:27:59.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:27:59.715 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:27:59.716 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:27:59.716 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:27:59.717 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:27:59.717 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:27:59.717 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:27:59.718 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:27:59.718 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:27:59.718 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:27:59.719 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:27:59.719 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:27:59.719 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:27:59.720 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:27:59.720 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:27:59.720 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:27:59.721 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:27:59.721 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:27:59.722 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:27:59.722 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:27:59.722 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:27:59.723 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:27:59.723 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:27:59.723 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1185] Local address: 172.21.15.161, port 33514 2022-07-03T21:27:59.724 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1185] Peer address: 172.21.15.101, port 22 2022-07-03T21:27:59.724 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1185] Connection lost 2022-07-03T21:27:59.724 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1185] Aborting connection 2022-07-03T21:27:59.725 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:27:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: pgmap v19464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:27:59.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:27:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:59.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:27:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:27:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:27:59.915 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:27:59.916 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:27:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: pgmap v19464: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:28:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:28:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:28:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:28:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:27:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:28:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:28:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:28:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:28:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:28:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:28:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:28:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:28:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:28:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:28:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:28:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:28:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:28:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:28:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:28:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:28:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:28:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:28:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:28:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:28:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:28:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:28:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:28:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:28:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:28:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:28:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:28:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:28:01.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:28:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:28:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:28:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:28:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:28:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:28:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:28:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:28:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:28:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:28:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:28:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:28:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:28:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:28:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:28:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:28:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:28:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:28:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:28:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:28:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1185] Local address: 172.21.15.161, port 33514 2022-07-03T21:28:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1185] Peer address: 172.21.15.101, port 22 2022-07-03T21:28:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1185] Connection lost 2022-07-03T21:28:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1185] Aborting connection 2022-07-03T21:28:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:28:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:28:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:28:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:28:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:28:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:28:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:28:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:28:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:28:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:28:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:28:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:28:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:28:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:28:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:28:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:28:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:28:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:28:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:28:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:28:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:28:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:28:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:28:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:28:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:28:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:28:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:28:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:28:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:28:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:28:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:28:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:28:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:28:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:28:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:28:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:28:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:28:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:28:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:28:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:28:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:28:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:28:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:28:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:28:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:28:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:28:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1185] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:28:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1185] Local address: 172.21.15.161, port 33514 2022-07-03T21:28:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1185] Peer address: 172.21.15.101, port 22 2022-07-03T21:28:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1185] Connection lost 2022-07-03T21:28:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1185] Aborting connection 2022-07-03T21:28:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:01 smithi161 ceph-mon[91579]: pgmap v19465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:01 smithi188 ceph-mon[92425]: pgmap v19465: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:03 smithi161 ceph-mon[91579]: pgmap v19466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:03 smithi188 ceph-mon[92425]: pgmap v19466: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:05 smithi161 ceph-mon[91579]: pgmap v19467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:05 smithi188 ceph-mon[92425]: pgmap v19467: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:07 smithi161 ceph-mon[91579]: pgmap v19468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:07 smithi188 ceph-mon[92425]: pgmap v19468: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:09 smithi161 ceph-mon[91579]: pgmap v19469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:09 smithi188 ceph-mon[92425]: pgmap v19469: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:10.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:11 smithi161 ceph-mon[91579]: pgmap v19470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:11 smithi188 ceph-mon[92425]: pgmap v19470: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:13 smithi161 ceph-mon[91579]: pgmap v19471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:13 smithi188 ceph-mon[92425]: pgmap v19471: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:15 smithi161 ceph-mon[91579]: pgmap v19472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:15 smithi188 ceph-mon[92425]: pgmap v19472: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:17 smithi161 ceph-mon[91579]: pgmap v19473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:17 smithi188 ceph-mon[92425]: pgmap v19473: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:19 smithi161 ceph-mon[91579]: pgmap v19474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:19 smithi188 ceph-mon[92425]: pgmap v19474: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:20.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:21 smithi161 ceph-mon[91579]: pgmap v19475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:21 smithi188 ceph-mon[92425]: pgmap v19475: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:23 smithi161 ceph-mon[91579]: pgmap v19476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:23 smithi188 ceph-mon[92425]: pgmap v19476: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:25 smithi161 ceph-mon[91579]: pgmap v19477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:25 smithi188 ceph-mon[92425]: pgmap v19477: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:27 smithi161 ceph-mon[91579]: pgmap v19478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:27 smithi188 ceph-mon[92425]: pgmap v19478: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:29 smithi161 ceph-mon[91579]: pgmap v19479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:29 smithi188 ceph-mon[92425]: pgmap v19479: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:31 smithi161 ceph-mon[91579]: pgmap v19480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:31 smithi188 ceph-mon[92425]: pgmap v19480: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:34.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:33 smithi161 ceph-mon[91579]: pgmap v19481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:33 smithi188 ceph-mon[92425]: pgmap v19481: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:35 smithi161 ceph-mon[91579]: pgmap v19482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:35 smithi188 ceph-mon[92425]: pgmap v19482: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:37 smithi161 ceph-mon[91579]: pgmap v19483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:38.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:37 smithi188 ceph-mon[92425]: pgmap v19483: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:39 smithi161 ceph-mon[91579]: pgmap v19484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:39 smithi188 ceph-mon[92425]: pgmap v19484: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:41 smithi161 ceph-mon[91579]: pgmap v19485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:41 smithi188 ceph-mon[92425]: pgmap v19485: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:43 smithi161 ceph-mon[91579]: pgmap v19486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:43 smithi188 ceph-mon[92425]: pgmap v19486: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:45 smithi161 ceph-mon[91579]: pgmap v19487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:45 smithi188 ceph-mon[92425]: pgmap v19487: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:47 smithi161 ceph-mon[91579]: pgmap v19488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:47 smithi188 ceph-mon[92425]: pgmap v19488: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:49 smithi161 ceph-mon[91579]: pgmap v19489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:49 smithi188 ceph-mon[92425]: pgmap v19489: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:51 smithi161 ceph-mon[91579]: pgmap v19490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:51 smithi188 ceph-mon[92425]: pgmap v19490: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:53 smithi161 ceph-mon[91579]: pgmap v19491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:53 smithi188 ceph-mon[92425]: pgmap v19491: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:28:55.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:28:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:55 smithi161 ceph-mon[91579]: pgmap v19492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:55 smithi188 ceph-mon[92425]: pgmap v19492: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:57 smithi161 ceph-mon[91579]: pgmap v19493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:28:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:57 smithi188 ceph-mon[92425]: pgmap v19493: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:00.008 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: pgmap v19494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:00.009 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:00.009 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:29:00.010 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:29:00.010 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:29:00.010 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:29:00.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:28:59.948+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:29:00.011 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:29:00.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:29:00.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:29:00.012 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:29:00.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:29:00.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:29:00.013 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:29:00.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:29:00.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:29:00.014 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:29:00.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:29:00.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:29:00.015 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:29:00.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:29:00.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:29:00.016 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:29:00.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:29:00.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:29:00.017 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:29:00.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:29:00.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:29:00.018 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:29:00.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:29:00.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:29:00.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:29:00.019 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:29:00.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:29:00.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:29:00.020 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:29:00.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:29:00.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:29:00.021 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:29:00.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:29:00.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:29:00.022 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:29:00.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:29:00.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:29:00.023 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:29:00.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:29:00.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:29:00.024 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:29:00.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:29:00.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:29:00.025 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:29:00.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:29:00.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1187] Local address: 172.21.15.161, port 33518 2022-07-03T21:29:00.026 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1187] Peer address: 172.21.15.101, port 22 2022-07-03T21:29:00.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1187] Connection lost 2022-07-03T21:29:00.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1187] Aborting connection 2022-07-03T21:29:00.027 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:28:59 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:29:00.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: pgmap v19494: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:00.028 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:00.029 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:29:00.029 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:29:00.029 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:29:00.030 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:28:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:29:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:29:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:29:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:29:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:00 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:29:01.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:00 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:29:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:29:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:29:01.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:29:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:29:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:29:01.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:29:01.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:29:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:29:01.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:29:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:29:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:29:01.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:29:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:29:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:29:01.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:29:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:29:01.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:29:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:29:01.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:29:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:29:01.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:29:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:29:01.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:29:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:29:01.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:29:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:29:01.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:29:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:29:01.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:29:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:29:01.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:29:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:29:01.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:29:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:29:01.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:29:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:29:01.148 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:29:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:29:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:29:01.149 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:29:01.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:29:01.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:29:01.150 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:29:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:29:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:29:01.151 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1187] Local address: 172.21.15.161, port 33518 2022-07-03T21:29:01.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1187] Peer address: 172.21.15.101, port 22 2022-07-03T21:29:01.152 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1187] Connection lost 2022-07-03T21:29:01.153 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1187] Aborting connection 2022-07-03T21:29:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:29:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:29:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:29:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:00 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:29:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:00 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:29:01.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:29:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:29:01.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:29:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:29:01.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:29:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:29:01.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:29:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:29:01.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:29:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:29:01.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:29:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:29:01.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:29:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:29:01.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:29:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:29:01.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:29:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:29:01.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:29:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:29:01.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:29:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:29:01.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:29:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:29:01.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:29:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:29:01.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:29:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:29:01.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:29:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:29:01.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:29:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:29:01.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:29:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:29:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:29:01.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:29:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:29:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:29:01.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:29:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:29:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:29:01.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:29:01.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:29:01.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:29:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1187] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:29:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1187] Local address: 172.21.15.161, port 33518 2022-07-03T21:29:01.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1187] Peer address: 172.21.15.101, port 22 2022-07-03T21:29:01.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1187] Connection lost 2022-07-03T21:29:01.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1187] Aborting connection 2022-07-03T21:29:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:01 smithi161 ceph-mon[91579]: pgmap v19495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:01 smithi188 ceph-mon[92425]: pgmap v19495: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:03 smithi161 ceph-mon[91579]: pgmap v19496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:03 smithi188 ceph-mon[92425]: pgmap v19496: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:05.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:05 smithi161 ceph-mon[91579]: pgmap v19497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:05 smithi188 ceph-mon[92425]: pgmap v19497: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:07 smithi161 ceph-mon[91579]: pgmap v19498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:07 smithi188 ceph-mon[92425]: pgmap v19498: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:09 smithi161 ceph-mon[91579]: pgmap v19499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:09 smithi188 ceph-mon[92425]: pgmap v19499: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:11 smithi161 ceph-mon[91579]: pgmap v19500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:11 smithi188 ceph-mon[92425]: pgmap v19500: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:13 smithi161 ceph-mon[91579]: pgmap v19501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:13 smithi188 ceph-mon[92425]: pgmap v19501: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:15.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:15 smithi161 ceph-mon[91579]: pgmap v19502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:16.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:15 smithi188 ceph-mon[92425]: pgmap v19502: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:18.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:17 smithi161 ceph-mon[91579]: pgmap v19503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:17 smithi188 ceph-mon[92425]: pgmap v19503: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:19 smithi161 ceph-mon[91579]: pgmap v19504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:19 smithi188 ceph-mon[92425]: pgmap v19504: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:21.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:21 smithi161 ceph-mon[91579]: pgmap v19505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:21 smithi188 ceph-mon[92425]: pgmap v19505: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:24.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:23 smithi161 ceph-mon[91579]: pgmap v19506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:23 smithi188 ceph-mon[92425]: pgmap v19506: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:26.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:25 smithi161 ceph-mon[91579]: pgmap v19507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:25 smithi188 ceph-mon[92425]: pgmap v19507: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:28.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:27 smithi161 ceph-mon[91579]: pgmap v19508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:28.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:27 smithi188 ceph-mon[92425]: pgmap v19508: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:29 smithi161 ceph-mon[91579]: pgmap v19509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:29 smithi188 ceph-mon[92425]: pgmap v19509: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:31 smithi161 ceph-mon[91579]: pgmap v19510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:31 smithi188 ceph-mon[92425]: pgmap v19510: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:33 smithi161 ceph-mon[91579]: pgmap v19511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:33 smithi188 ceph-mon[92425]: pgmap v19511: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:35 smithi161 ceph-mon[91579]: pgmap v19512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:35 smithi188 ceph-mon[92425]: pgmap v19512: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:37 smithi161 ceph-mon[91579]: pgmap v19513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:37 smithi188 ceph-mon[92425]: pgmap v19513: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:39 smithi161 ceph-mon[91579]: pgmap v19514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:39 smithi188 ceph-mon[92425]: pgmap v19514: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:41.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:41 smithi161 ceph-mon[91579]: pgmap v19515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:41.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:41 smithi188 ceph-mon[92425]: pgmap v19515: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:44.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:43 smithi161 ceph-mon[91579]: pgmap v19516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:44.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:43 smithi188 ceph-mon[92425]: pgmap v19516: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:45 smithi161 ceph-mon[91579]: pgmap v19517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:45 smithi188 ceph-mon[92425]: pgmap v19517: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:48.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:47 smithi161 ceph-mon[91579]: pgmap v19518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:47 smithi188 ceph-mon[92425]: pgmap v19518: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:49 smithi161 ceph-mon[91579]: pgmap v19519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:49 smithi188 ceph-mon[92425]: pgmap v19519: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:51 smithi161 ceph-mon[91579]: pgmap v19520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:51 smithi188 ceph-mon[92425]: pgmap v19520: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:53 smithi161 ceph-mon[91579]: pgmap v19521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:53 smithi188 ceph-mon[92425]: pgmap v19521: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:29:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:29:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:55 smithi161 ceph-mon[91579]: pgmap v19522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:56.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:55 smithi188 ceph-mon[92425]: pgmap v19522: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:57 smithi161 ceph-mon[91579]: pgmap v19523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:29:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:57 smithi188 ceph-mon[92425]: pgmap v19523: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: pgmap v19524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:30:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:30:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:30:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:29:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:30:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: pgmap v19524: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:30:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:30:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:30:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:29:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:30:00.747 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:30:00.447+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:30:00.747 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:30:00.747 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:30:00.748 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:30:00.748 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:30:00.748 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:30:00.749 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:30:00.749 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:30:00.750 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:30:00.750 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:30:00.750 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:30:00.751 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:30:00.751 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:30:00.751 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:30:00.752 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:30:00.752 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:30:00.752 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:30:00.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:30:00.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:30:00.753 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:30:00.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:30:00.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:30:00.754 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:30:00.755 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:30:00.755 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:30:00.755 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:30:00.756 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:30:00.756 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:30:00.756 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:30:00.757 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:30:00.757 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:30:00.757 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:30:00.758 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:30:00.758 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:30:00.758 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:30:00.759 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:30:00.759 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:30:00.759 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:30:00.760 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:30:00.760 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:30:00.760 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:30:00.761 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:30:00.761 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:30:00.761 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:30:00.762 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:30:00.762 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:30:00.762 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1189] Local address: 172.21.15.161, port 33522 2022-07-03T21:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1189] Peer address: 172.21.15.101, port 22 2022-07-03T21:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1189] Connection lost 2022-07-03T21:30:00.763 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1189] Aborting connection 2022-07-03T21:30:00.764 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:30:01.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:30:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:30:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:30:01.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:30:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:30:01.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:30:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:30:01.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:30:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:30:02.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:30:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:30:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:30:02.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:30:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:30:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:30:02.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:30:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:30:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:30:02.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:30:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:30:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:30:02.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:30:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:30:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:30:02.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:30:02.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:30:02.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:30:02.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:30:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:30:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:30:02.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:30:02.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:30:02.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:30:02.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:30:02.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:30:02.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:30:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:30:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:30:02.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:30:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:30:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:30:02.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:30:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:30:02.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:30:02.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:30:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:30:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:30:02.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:30:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:30:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:30:02.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:30:02.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:30:02.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:30:02.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1189] Local address: 172.21.15.161, port 33522 2022-07-03T21:30:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1189] Peer address: 172.21.15.101, port 22 2022-07-03T21:30:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1189] Connection lost 2022-07-03T21:30:02.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1189] Aborting connection 2022-07-03T21:30:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: pgmap v19525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.147 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.148 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:01 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:30:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:30:02.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:30:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:30:02.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:30:02.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:30:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:30:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:30:02.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:30:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:30:02.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:30:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:30:02.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:30:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:30:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:30:02.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:30:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:30:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:30:02.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:30:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:30:02.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:30:02.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:30:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:30:02.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:30:02.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:30:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:30:02.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:30:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:30:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:30:02.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:30:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:30:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:30:02.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:30:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:30:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:30:02.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:30:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:30:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:30:02.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1189] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:30:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1189] Local address: 172.21.15.161, port 33522 2022-07-03T21:30:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1189] Peer address: 172.21.15.101, port 22 2022-07-03T21:30:02.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1189] Connection lost 2022-07-03T21:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1189] Aborting connection 2022-07-03T21:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: pgmap v19525: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:02.181 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.182 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:02.183 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:01 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:30:04.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:03 smithi161 ceph-mon[91579]: pgmap v19526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:04.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:03 smithi188 ceph-mon[92425]: pgmap v19526: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:05 smithi161 ceph-mon[91579]: pgmap v19527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:05 smithi188 ceph-mon[92425]: pgmap v19527: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:07 smithi188 ceph-mon[92425]: pgmap v19528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:07 smithi161 ceph-mon[91579]: pgmap v19528: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:09 smithi188 ceph-mon[92425]: pgmap v19529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:09 smithi161 ceph-mon[91579]: pgmap v19529: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:11.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:11 smithi161 ceph-mon[91579]: pgmap v19530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:11 smithi188 ceph-mon[92425]: pgmap v19530: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:13 smithi161 ceph-mon[91579]: pgmap v19531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:13 smithi188 ceph-mon[92425]: pgmap v19531: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:15 smithi161 ceph-mon[91579]: pgmap v19532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:15 smithi188 ceph-mon[92425]: pgmap v19532: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:17 smithi161 ceph-mon[91579]: pgmap v19533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:18.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:17 smithi188 ceph-mon[92425]: pgmap v19533: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:19 smithi161 ceph-mon[91579]: pgmap v19534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:19 smithi188 ceph-mon[92425]: pgmap v19534: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:21 smithi161 ceph-mon[91579]: pgmap v19535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:21 smithi188 ceph-mon[92425]: pgmap v19535: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:23 smithi161 ceph-mon[91579]: pgmap v19536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:24.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:23 smithi188 ceph-mon[92425]: pgmap v19536: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:25.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:25 smithi161 ceph-mon[91579]: pgmap v19537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:26.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:25 smithi188 ceph-mon[92425]: pgmap v19537: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:27 smithi161 ceph-mon[91579]: pgmap v19538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:27 smithi188 ceph-mon[92425]: pgmap v19538: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:29 smithi161 ceph-mon[91579]: pgmap v19539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:29 smithi188 ceph-mon[92425]: pgmap v19539: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:30.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:31 smithi161 ceph-mon[91579]: pgmap v19540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:31 smithi188 ceph-mon[92425]: pgmap v19540: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:33 smithi161 ceph-mon[91579]: pgmap v19541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:33 smithi188 ceph-mon[92425]: pgmap v19541: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:35.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:35 smithi161 ceph-mon[91579]: pgmap v19542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:35 smithi188 ceph-mon[92425]: pgmap v19542: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:37 smithi161 ceph-mon[91579]: pgmap v19543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:37 smithi188 ceph-mon[92425]: pgmap v19543: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:39 smithi161 ceph-mon[91579]: pgmap v19544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:39 smithi188 ceph-mon[92425]: pgmap v19544: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:41 smithi161 ceph-mon[91579]: pgmap v19545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:41 smithi188 ceph-mon[92425]: pgmap v19545: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:43 smithi161 ceph-mon[91579]: pgmap v19546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:43 smithi188 ceph-mon[92425]: pgmap v19546: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:45.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:45 smithi161 ceph-mon[91579]: pgmap v19547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:46.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:45 smithi188 ceph-mon[92425]: pgmap v19547: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:47 smithi161 ceph-mon[91579]: pgmap v19548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:48.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:47 smithi188 ceph-mon[92425]: pgmap v19548: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:49 smithi161 ceph-mon[91579]: pgmap v19549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:49 smithi188 ceph-mon[92425]: pgmap v19549: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:51 smithi161 ceph-mon[91579]: pgmap v19550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:51.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:51 smithi188 ceph-mon[92425]: pgmap v19550: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:54.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:53 smithi161 ceph-mon[91579]: pgmap v19551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:54.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:53 smithi188 ceph-mon[92425]: pgmap v19551: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:30:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:55.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:30:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:55 smithi161 ceph-mon[91579]: pgmap v19552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:55 smithi188 ceph-mon[92425]: pgmap v19552: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:57 smithi161 ceph-mon[91579]: pgmap v19553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:30:58.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:57 smithi188 ceph-mon[92425]: pgmap v19553: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: pgmap v19554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:31:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:31:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:31:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:30:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:31:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: pgmap v19554: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:31:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:31:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:31:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:30:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:31:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:01 smithi188 ceph-mon[92425]: pgmap v19555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:01.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:31:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:31:01.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:01 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:31:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-mon[91579]: pgmap v19555: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:31:01.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:31:01.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:31:01.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:31:01.379+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:31:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:31:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:31:01.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:31:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:31:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:31:01.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:31:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:31:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:31:01.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:31:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:31:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:31:01.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:31:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:31:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:31:01.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:31:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:31:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:31:01.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:31:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:31:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:31:01.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:31:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:31:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:31:01.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:31:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:31:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:31:01.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:31:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:31:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:31:01.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:31:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:31:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:31:01.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:31:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:31:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:31:01.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:31:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:31:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:31:01.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:31:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:31:01.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:31:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:31:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:31:01.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:31:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:31:01.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1191] Local address: 172.21.15.161, port 33526 2022-07-03T21:31:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1191] Peer address: 172.21.15.101, port 22 2022-07-03T21:31:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1191] Connection lost 2022-07-03T21:31:01.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1191] Aborting connection 2022-07-03T21:31:01.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:31:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:31:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:31:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:31:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:31:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:31:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:31:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:31:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:31:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:31:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:31:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:31:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:31:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:31:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:31:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:31:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:31:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:31:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:31:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:31:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:31:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:31:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:31:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:31:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:31:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:31:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:31:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:31:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:31:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:31:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:31:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:31:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:31:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:31:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:31:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:31:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:31:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:31:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:31:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:31:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:31:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:31:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:31:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:31:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:31:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:31:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:31:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:31:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:31:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1191] Local address: 172.21.15.161, port 33526 2022-07-03T21:31:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1191] Peer address: 172.21.15.101, port 22 2022-07-03T21:31:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1191] Connection lost 2022-07-03T21:31:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1191] Aborting connection 2022-07-03T21:31:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:31:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:31:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:31:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:31:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:31:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:31:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:31:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:31:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:31:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:31:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:31:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:31:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:31:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:31:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:31:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:31:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:31:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:31:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:31:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:31:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:31:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:31:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:31:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:31:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:31:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:31:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:31:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:31:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:31:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:31:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:31:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:31:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:31:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:31:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:31:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:31:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:31:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:31:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:31:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:31:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:31:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:31:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:31:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:31:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:31:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:31:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1191] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:31:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1191] Local address: 172.21.15.161, port 33526 2022-07-03T21:31:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1191] Peer address: 172.21.15.101, port 22 2022-07-03T21:31:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1191] Connection lost 2022-07-03T21:31:02.931 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1191] Aborting connection 2022-07-03T21:31:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:03 smithi161 ceph-mon[91579]: pgmap v19556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:03.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:03 smithi188 ceph-mon[92425]: pgmap v19556: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:05 smithi161 ceph-mon[91579]: pgmap v19557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:05 smithi188 ceph-mon[92425]: pgmap v19557: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:07 smithi161 ceph-mon[91579]: pgmap v19558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:07 smithi188 ceph-mon[92425]: pgmap v19558: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:09 smithi161 ceph-mon[91579]: pgmap v19559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:09 smithi188 ceph-mon[92425]: pgmap v19559: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:11 smithi161 ceph-mon[91579]: pgmap v19560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:11 smithi188 ceph-mon[92425]: pgmap v19560: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:13 smithi161 ceph-mon[91579]: pgmap v19561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:13 smithi188 ceph-mon[92425]: pgmap v19561: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:15 smithi161 ceph-mon[91579]: pgmap v19562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:15 smithi188 ceph-mon[92425]: pgmap v19562: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:17 smithi161 ceph-mon[91579]: pgmap v19563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:17 smithi188 ceph-mon[92425]: pgmap v19563: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:19 smithi161 ceph-mon[91579]: pgmap v19564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:19 smithi188 ceph-mon[92425]: pgmap v19564: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:21 smithi161 ceph-mon[91579]: pgmap v19565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:21 smithi188 ceph-mon[92425]: pgmap v19565: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:23 smithi161 ceph-mon[91579]: pgmap v19566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:23 smithi188 ceph-mon[92425]: pgmap v19566: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:25 smithi161 ceph-mon[91579]: pgmap v19567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:25 smithi188 ceph-mon[92425]: pgmap v19567: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:27 smithi161 ceph-mon[91579]: pgmap v19568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:27 smithi188 ceph-mon[92425]: pgmap v19568: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:29 smithi161 ceph-mon[91579]: pgmap v19569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:29 smithi188 ceph-mon[92425]: pgmap v19569: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:31 smithi161 ceph-mon[91579]: pgmap v19570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:31 smithi188 ceph-mon[92425]: pgmap v19570: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:33 smithi161 ceph-mon[91579]: pgmap v19571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:34.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:33 smithi188 ceph-mon[92425]: pgmap v19571: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:35 smithi161 ceph-mon[91579]: pgmap v19572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:35 smithi188 ceph-mon[92425]: pgmap v19572: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:37 smithi161 ceph-mon[91579]: pgmap v19573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:37 smithi188 ceph-mon[92425]: pgmap v19573: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:39 smithi161 ceph-mon[91579]: pgmap v19574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:39 smithi188 ceph-mon[92425]: pgmap v19574: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:41 smithi161 ceph-mon[91579]: pgmap v19575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:41 smithi188 ceph-mon[92425]: pgmap v19575: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:43 smithi161 ceph-mon[91579]: pgmap v19576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:43 smithi188 ceph-mon[92425]: pgmap v19576: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:45 smithi161 ceph-mon[91579]: pgmap v19577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:45 smithi188 ceph-mon[92425]: pgmap v19577: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:47 smithi161 ceph-mon[91579]: pgmap v19578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:47 smithi188 ceph-mon[92425]: pgmap v19578: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:49 smithi161 ceph-mon[91579]: pgmap v19579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:49 smithi188 ceph-mon[92425]: pgmap v19579: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:51 smithi161 ceph-mon[91579]: pgmap v19580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:51 smithi188 ceph-mon[92425]: pgmap v19580: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:31:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:53 smithi161 ceph-mon[91579]: pgmap v19581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:53 smithi188 ceph-mon[92425]: pgmap v19581: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:31:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:31:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:55 smithi161 ceph-mon[91579]: pgmap v19582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:55 smithi188 ceph-mon[92425]: pgmap v19582: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:57 smithi161 ceph-mon[91579]: pgmap v19583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:31:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:57 smithi188 ceph-mon[92425]: pgmap v19583: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: pgmap v19584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:32:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:32:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:32:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:32:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:31:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: pgmap v19584: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:32:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:32:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:32:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:32:00.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:31:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-mon[91579]: pgmap v19585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:01.891 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:01 smithi188 ceph-mon[92425]: pgmap v19585: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:02.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:32:01.879+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:32:02.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:32:02.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:32:02.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:32:02.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:32:02.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:32:02.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:32:02.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:32:02.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:32:02.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:32:02.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:32:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:32:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:32:02.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:32:02.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:32:02.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:32:02.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:32:02.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:32:02.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:32:02.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:32:02.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:32:02.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:32:02.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:32:02.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:32:02.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:32:02.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:32:02.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:32:02.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:32:02.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:32:02.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:32:02.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:32:02.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:32:02.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:32:02.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:32:02.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:32:02.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:32:02.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:32:02.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:32:02.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:32:02.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:32:02.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:32:02.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:32:02.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:32:02.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:32:02.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:32:02.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:32:02.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1193] Local address: 172.21.15.161, port 33530 2022-07-03T21:32:02.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1193] Peer address: 172.21.15.101, port 22 2022-07-03T21:32:02.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1193] Connection lost 2022-07-03T21:32:02.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1193] Aborting connection 2022-07-03T21:32:02.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:32:01 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:32:02.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:32:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:32:02.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:32:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:02 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:32:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:02 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:32:02.880 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:32:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:32:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:32:02.881 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:32:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:32:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:32:02.882 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:32:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:32:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:32:02.883 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:32:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:32:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:32:02.884 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:32:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:32:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:32:02.885 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:32:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:32:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:32:02.886 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:32:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:32:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:32:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:32:02.887 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:32:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:32:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:32:02.888 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:32:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:32:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:32:02.889 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:32:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:32:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:32:02.890 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:32:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:32:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:32:02.891 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:32:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:32:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:32:02.892 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:32:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:32:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:32:02.893 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:32:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:32:02.894 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:32:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:32:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:32:02.895 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1193] Local address: 172.21.15.161, port 33530 2022-07-03T21:32:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1193] Peer address: 172.21.15.101, port 22 2022-07-03T21:32:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1193] Connection lost 2022-07-03T21:32:02.896 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1193] Aborting connection 2022-07-03T21:32:02.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:32:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:32:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:32:02.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:02 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:32:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:02 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:32:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:32:02.914 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:32:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:32:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:32:02.915 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:32:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:32:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:32:02.916 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:32:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:32:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:32:02.917 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:32:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:32:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:32:02.918 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:32:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:32:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:32:02.919 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:32:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:32:02.920 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:32:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:32:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:32:02.921 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:32:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:32:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:32:02.922 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:32:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:32:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:32:02.923 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:32:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:32:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:32:02.924 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:32:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:32:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:32:02.925 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:32:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:32:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:32:02.926 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:32:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:32:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:32:02.927 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:32:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:32:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:32:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:32:02.928 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:32:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1193] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:32:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1193] Local address: 172.21.15.161, port 33530 2022-07-03T21:32:02.929 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1193] Peer address: 172.21.15.101, port 22 2022-07-03T21:32:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1193] Connection lost 2022-07-03T21:32:02.930 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1193] Aborting connection 2022-07-03T21:32:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:03 smithi161 ceph-mon[91579]: pgmap v19586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:03.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:03 smithi188 ceph-mon[92425]: pgmap v19586: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:04.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:05.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:05 smithi161 ceph-mon[91579]: pgmap v19587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:05.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:05 smithi188 ceph-mon[92425]: pgmap v19587: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:07 smithi161 ceph-mon[91579]: pgmap v19588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:07 smithi188 ceph-mon[92425]: pgmap v19588: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:09 smithi161 ceph-mon[91579]: pgmap v19589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:09 smithi188 ceph-mon[92425]: pgmap v19589: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:11 smithi161 ceph-mon[91579]: pgmap v19590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:11.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:11 smithi188 ceph-mon[92425]: pgmap v19590: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:13 smithi161 ceph-mon[91579]: pgmap v19591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:13 smithi188 ceph-mon[92425]: pgmap v19591: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:15 smithi161 ceph-mon[91579]: pgmap v19592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:15 smithi188 ceph-mon[92425]: pgmap v19592: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:17 smithi161 ceph-mon[91579]: pgmap v19593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:17 smithi188 ceph-mon[92425]: pgmap v19593: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:19 smithi161 ceph-mon[91579]: pgmap v19594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:19 smithi188 ceph-mon[92425]: pgmap v19594: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:21 smithi161 ceph-mon[91579]: pgmap v19595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:21.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:21 smithi188 ceph-mon[92425]: pgmap v19595: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:23 smithi161 ceph-mon[91579]: pgmap v19596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:23 smithi188 ceph-mon[92425]: pgmap v19596: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:25 smithi161 ceph-mon[91579]: pgmap v19597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:25 smithi188 ceph-mon[92425]: pgmap v19597: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:27 smithi161 ceph-mon[91579]: pgmap v19598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:27 smithi188 ceph-mon[92425]: pgmap v19598: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:29 smithi161 ceph-mon[91579]: pgmap v19599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:29 smithi188 ceph-mon[92425]: pgmap v19599: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:31 smithi161 ceph-mon[91579]: pgmap v19600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:31 smithi188 ceph-mon[92425]: pgmap v19600: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:33 smithi161 ceph-mon[91579]: pgmap v19601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:33 smithi188 ceph-mon[92425]: pgmap v19601: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:35 smithi161 ceph-mon[91579]: pgmap v19602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:35 smithi188 ceph-mon[92425]: pgmap v19602: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:37 smithi161 ceph-mon[91579]: pgmap v19603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:37 smithi188 ceph-mon[92425]: pgmap v19603: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:39 smithi161 ceph-mon[91579]: pgmap v19604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:39 smithi188 ceph-mon[92425]: pgmap v19604: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:41 smithi161 ceph-mon[91579]: pgmap v19605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:41 smithi188 ceph-mon[92425]: pgmap v19605: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:43 smithi161 ceph-mon[91579]: pgmap v19606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:43 smithi188 ceph-mon[92425]: pgmap v19606: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:45 smithi161 ceph-mon[91579]: pgmap v19607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:45 smithi188 ceph-mon[92425]: pgmap v19607: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:47 smithi161 ceph-mon[91579]: pgmap v19608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:47 smithi188 ceph-mon[92425]: pgmap v19608: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:49 smithi161 ceph-mon[91579]: pgmap v19609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:49 smithi188 ceph-mon[92425]: pgmap v19609: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:51 smithi161 ceph-mon[91579]: pgmap v19610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:51 smithi188 ceph-mon[92425]: pgmap v19610: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:54.133 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:53 smithi161 ceph-mon[91579]: pgmap v19611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:53 smithi188 ceph-mon[92425]: pgmap v19611: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:32:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:32:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:55 smithi161 ceph-mon[91579]: pgmap v19612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:55 smithi188 ceph-mon[92425]: pgmap v19612: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:57 smithi161 ceph-mon[91579]: pgmap v19613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:32:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:57 smithi188 ceph-mon[92425]: pgmap v19613: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: pgmap v19614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:33:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:33:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:33:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:33:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:32:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: pgmap v19614: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:33:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:33:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:33:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:33:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:32:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:01 smithi161 ceph-mon[91579]: pgmap v19615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:01 smithi188 ceph-mon[92425]: pgmap v19615: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:02.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:33:02.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:33:02.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:33:02.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:33:02.372+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:33:02.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:33:02.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:33:02.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:33:02.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:33:02.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:33:02.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:33:02.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:33:02.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:33:02.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:33:02.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:33:02.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:33:02.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:33:02.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:33:02.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:33:02.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:33:02.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:33:02.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:33:02.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:33:02.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:33:02.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:33:02.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:33:02.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:33:02.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:33:02.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:33:02.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:33:02.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:33:02.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:33:02.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:33:02.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:33:02.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:33:02.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:33:02.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:33:02.647 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:33:02.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:33:02.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:33:02.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:33:02.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:33:02.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:33:02.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:33:02.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:33:02.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:33:02.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:33:02.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:33:02.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:33:02.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:33:02.652 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1195] Local address: 172.21.15.161, port 33534 2022-07-03T21:33:02.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1195] Peer address: 172.21.15.101, port 22 2022-07-03T21:33:02.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1195] Connection lost 2022-07-03T21:33:02.653 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1195] Aborting connection 2022-07-03T21:33:02.654 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:33:02 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:33:02.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:33:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:33:02.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:02 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:33:03.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:03 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:33:03.916 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:03 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:33:03.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:33:03.916 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:33:03.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:33:03.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:33:03.917 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:33:03.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:33:03.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:33:03.918 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:33:03.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:33:03.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:33:03.919 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:33:03.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:33:03.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:33:03.920 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:33:03.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:33:03.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:33:03.921 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:33:03.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:33:03.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:33:03.922 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:33:03.923 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:33:03.923 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:33:03.923 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:33:03.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:33:03.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:33:03.924 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:33:03.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:33:03.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:33:03.925 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:33:03.926 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:33:03.926 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:33:03.926 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:33:03.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:33:03.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:33:03.927 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:33:03.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:33:03.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:33:03.928 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:33:03.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:33:03.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:33:03.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:33:03.929 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:33:03.930 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:33:03.930 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:33:03.930 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:33:03.931 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1195] Local address: 172.21.15.161, port 33534 2022-07-03T21:33:03.931 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1195] Peer address: 172.21.15.101, port 22 2022-07-03T21:33:03.932 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1195] Connection lost 2022-07-03T21:33:03.932 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1195] Aborting connection 2022-07-03T21:33:03.932 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:03 smithi161 ceph-mon[91579]: pgmap v19616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:03.933 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:03 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:33:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:03 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:33:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:33:03.934 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:33:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:33:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:33:03.935 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:33:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:33:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:33:03.936 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:33:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:33:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:33:03.937 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:33:03.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:33:03.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:33:03.938 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:33:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:33:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:33:03.939 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:33:03.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:33:03.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:33:03.940 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:33:03.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:33:03.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:33:03.941 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:33:03.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:33:03.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:33:03.942 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:33:03.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:33:03.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:33:03.943 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:33:03.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:33:03.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:33:03.944 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:33:03.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:33:03.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:33:03.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:33:03.945 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:33:03.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:33:03.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:33:03.946 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:33:03.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:33:03.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:33:03.947 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:33:03.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:33:03.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:33:03.948 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1195] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:33:03.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1195] Local address: 172.21.15.161, port 33534 2022-07-03T21:33:03.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1195] Peer address: 172.21.15.101, port 22 2022-07-03T21:33:03.949 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1195] Connection lost 2022-07-03T21:33:03.950 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1195] Aborting connection 2022-07-03T21:33:03.950 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:03 smithi188 ceph-mon[92425]: pgmap v19616: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:04.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:04.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: pgmap v19617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:06 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: pgmap v19617: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:33:06.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:06.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:06 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:33:07.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:07 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 30014k 2022-07-03T21:33:07.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:07 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 30734655: error parsing value: Value '30734655' is below minimum 939524096 2022-07-03T21:33:07.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:07 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 30014k 2022-07-03T21:33:07.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:07 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 30734655: error parsing value: Value '30734655' is below minimum 939524096 2022-07-03T21:33:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:08 smithi161 ceph-mon[91579]: pgmap v19618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:08 smithi188 ceph-mon[92425]: pgmap v19618: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:10 smithi161 ceph-mon[91579]: pgmap v19619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:10 smithi188 ceph-mon[92425]: pgmap v19619: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:11 smithi161 ceph-mon[91579]: pgmap v19620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:11 smithi188 ceph-mon[92425]: pgmap v19620: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:13 smithi161 ceph-mon[91579]: pgmap v19621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:13 smithi188 ceph-mon[92425]: pgmap v19621: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:15 smithi161 ceph-mon[91579]: pgmap v19622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:15 smithi188 ceph-mon[92425]: pgmap v19622: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:17 smithi161 ceph-mon[91579]: pgmap v19623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:17 smithi188 ceph-mon[92425]: pgmap v19623: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:19 smithi161 ceph-mon[91579]: pgmap v19624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:19 smithi188 ceph-mon[92425]: pgmap v19624: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:21 smithi161 ceph-mon[91579]: pgmap v19625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:21 smithi188 ceph-mon[92425]: pgmap v19625: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:23 smithi161 ceph-mon[91579]: pgmap v19626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:23 smithi188 ceph-mon[92425]: pgmap v19626: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:25 smithi161 ceph-mon[91579]: pgmap v19627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:25 smithi188 ceph-mon[92425]: pgmap v19627: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:27 smithi161 ceph-mon[91579]: pgmap v19628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:27 smithi188 ceph-mon[92425]: pgmap v19628: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:29 smithi161 ceph-mon[91579]: pgmap v19629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:29 smithi188 ceph-mon[92425]: pgmap v19629: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:31 smithi161 ceph-mon[91579]: pgmap v19630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:31 smithi188 ceph-mon[92425]: pgmap v19630: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:33 smithi161 ceph-mon[91579]: pgmap v19631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:34.167 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:33 smithi188 ceph-mon[92425]: pgmap v19631: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:35.168 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:35 smithi161 ceph-mon[91579]: pgmap v19632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:35 smithi188 ceph-mon[92425]: pgmap v19632: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:37 smithi161 ceph-mon[91579]: pgmap v19633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:37 smithi188 ceph-mon[92425]: pgmap v19633: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:39 smithi161 ceph-mon[91579]: pgmap v19634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:39 smithi188 ceph-mon[92425]: pgmap v19634: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:41 smithi161 ceph-mon[91579]: pgmap v19635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:41 smithi188 ceph-mon[92425]: pgmap v19635: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:43 smithi161 ceph-mon[91579]: pgmap v19636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:43 smithi188 ceph-mon[92425]: pgmap v19636: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:45 smithi161 ceph-mon[91579]: pgmap v19637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:45 smithi188 ceph-mon[92425]: pgmap v19637: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:47 smithi161 ceph-mon[91579]: pgmap v19638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:47 smithi188 ceph-mon[92425]: pgmap v19638: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:49 smithi161 ceph-mon[91579]: pgmap v19639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:50.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:49 smithi188 ceph-mon[92425]: pgmap v19639: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:51 smithi161 ceph-mon[91579]: pgmap v19640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:51 smithi188 ceph-mon[92425]: pgmap v19640: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:53 smithi161 ceph-mon[91579]: pgmap v19641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:53 smithi188 ceph-mon[92425]: pgmap v19641: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:33:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:33:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:55 smithi161 ceph-mon[91579]: pgmap v19642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:55 smithi188 ceph-mon[92425]: pgmap v19642: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:57 smithi161 ceph-mon[91579]: pgmap v19643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:33:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:57 smithi188 ceph-mon[92425]: pgmap v19643: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: pgmap v19644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:34:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:34:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:34:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:34:00.132 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:33:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: pgmap v19644: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:34:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:34:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:34:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:34:00.165 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:33:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:01 smithi161 ceph-mon[91579]: pgmap v19645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:01 smithi188 ceph-mon[92425]: pgmap v19645: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:03 smithi161 ceph-mon[91579]: pgmap v19646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:03 smithi188 ceph-mon[92425]: pgmap v19646: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:05.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:05.814 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:34:05.534+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:34:05.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:34:05.815 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:34:05.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:34:05.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:34:05.816 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:34:05.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:34:05.817 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:34:05.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:34:05.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:34:05.818 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:34:05.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:34:05.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:34:05.819 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:34:05.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:34:05.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:34:05.820 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:34:05.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:34:05.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:34:05.821 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:34:05.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:34:05.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:34:05.822 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:34:05.823 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:34:05.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:34:05.824 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:34:05.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:34:05.825 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:34:05.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:34:05.826 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:34:05.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:34:05.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:34:05.827 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:34:05.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:34:05.828 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:34:05.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:34:05.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:34:05.829 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:34:05.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:34:05.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:34:05.830 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:34:05.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:34:05.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:34:05.831 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:34:05.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:34:05.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:34:05.832 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1197] Local address: 172.21.15.161, port 33538 2022-07-03T21:34:05.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1197] Peer address: 172.21.15.101, port 22 2022-07-03T21:34:05.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1197] Connection lost 2022-07-03T21:34:05.833 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1197] Aborting connection 2022-07-03T21:34:05.834 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:34:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-mon[91579]: pgmap v19647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:34:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:34:06.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:34:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:05 smithi188 ceph-mon[92425]: pgmap v19647: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:34:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:34:06.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:34:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:34:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:34:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:34:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:34:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:34:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:34:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:34:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:34:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:34:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:34:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:34:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:34:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:34:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:34:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:34:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:34:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:34:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:34:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:34:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:34:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:34:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:34:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:34:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:34:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:34:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:34:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:34:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:34:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:34:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:34:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:34:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:34:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:34:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:34:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:34:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:34:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:34:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:34:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:34:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:34:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:34:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:34:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:34:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:34:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:34:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:34:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:34:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1197] Local address: 172.21.15.161, port 33538 2022-07-03T21:34:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1197] Peer address: 172.21.15.101, port 22 2022-07-03T21:34:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1197] Connection lost 2022-07-03T21:34:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1197] Aborting connection 2022-07-03T21:34:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:34:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:34:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:34:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:34:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:34:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:34:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:34:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:34:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:34:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:34:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:34:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:34:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:34:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:34:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:34:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:34:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:34:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:34:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:34:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:34:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:34:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:34:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:34:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:34:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:34:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:34:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:34:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:34:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:34:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:34:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:34:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:34:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:34:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:34:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:34:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:34:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:34:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:34:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:34:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:34:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:34:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:34:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:34:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:34:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:34:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:34:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1197] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:34:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1197] Local address: 172.21.15.161, port 33538 2022-07-03T21:34:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1197] Peer address: 172.21.15.101, port 22 2022-07-03T21:34:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1197] Connection lost 2022-07-03T21:34:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1197] Aborting connection 2022-07-03T21:34:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:07 smithi161 ceph-mon[91579]: pgmap v19648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:07 smithi188 ceph-mon[92425]: pgmap v19648: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:09 smithi161 ceph-mon[91579]: pgmap v19649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:09 smithi188 ceph-mon[92425]: pgmap v19649: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:11 smithi161 ceph-mon[91579]: pgmap v19650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:11 smithi188 ceph-mon[92425]: pgmap v19650: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:13 smithi161 ceph-mon[91579]: pgmap v19651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:13 smithi188 ceph-mon[92425]: pgmap v19651: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:15 smithi161 ceph-mon[91579]: pgmap v19652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:15 smithi188 ceph-mon[92425]: pgmap v19652: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:17 smithi161 ceph-mon[91579]: pgmap v19653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:17 smithi188 ceph-mon[92425]: pgmap v19653: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:19 smithi161 ceph-mon[91579]: pgmap v19654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:19 smithi188 ceph-mon[92425]: pgmap v19654: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:21 smithi161 ceph-mon[91579]: pgmap v19655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:21 smithi188 ceph-mon[92425]: pgmap v19655: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:23 smithi161 ceph-mon[91579]: pgmap v19656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:23 smithi188 ceph-mon[92425]: pgmap v19656: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:25 smithi161 ceph-mon[91579]: pgmap v19657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:25 smithi188 ceph-mon[92425]: pgmap v19657: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:27 smithi161 ceph-mon[91579]: pgmap v19658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:27 smithi188 ceph-mon[92425]: pgmap v19658: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:29 smithi161 ceph-mon[91579]: pgmap v19659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:29 smithi188 ceph-mon[92425]: pgmap v19659: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:31.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:31 smithi161 ceph-mon[91579]: osdmap e89: 12 total, 8 up, 8 in 2022-07-03T21:34:31.879 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:31 smithi161 ceph-mon[91579]: pgmap v19661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:31 smithi188 ceph-mon[92425]: osdmap e89: 12 total, 8 up, 8 in 2022-07-03T21:34:31.913 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:31 smithi188 ceph-mon[92425]: pgmap v19661: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:33 smithi161 ceph-mon[91579]: pgmap v19662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:33 smithi188 ceph-mon[92425]: pgmap v19662: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:35 smithi161 ceph-mon[91579]: pgmap v19663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:35 smithi188 ceph-mon[92425]: pgmap v19663: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:37 smithi161 ceph-mon[91579]: pgmap v19664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:37 smithi188 ceph-mon[92425]: pgmap v19664: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:39 smithi161 ceph-mon[91579]: pgmap v19665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:39 smithi188 ceph-mon[92425]: pgmap v19665: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2022-07-03T21:34:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:41.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:41 smithi161 ceph-mon[91579]: pgmap v19666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 1 op/s 2022-07-03T21:34:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:41 smithi188 ceph-mon[92425]: pgmap v19666: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 1 op/s 2022-07-03T21:34:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:43 smithi161 ceph-mon[91579]: pgmap v19667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:43 smithi188 ceph-mon[92425]: pgmap v19667: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:45 smithi161 ceph-mon[91579]: pgmap v19668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:45 smithi188 ceph-mon[92425]: pgmap v19668: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:47 smithi161 ceph-mon[91579]: pgmap v19669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:47 smithi188 ceph-mon[92425]: pgmap v19669: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:49 smithi161 ceph-mon[91579]: pgmap v19670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:49 smithi188 ceph-mon[92425]: pgmap v19670: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:51 smithi161 ceph-mon[91579]: pgmap v19671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:51 smithi188 ceph-mon[92425]: pgmap v19671: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:53 smithi161 ceph-mon[91579]: pgmap v19672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:53 smithi188 ceph-mon[92425]: pgmap v19672: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:34:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:34:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:55 smithi161 ceph-mon[91579]: pgmap v19673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:55 smithi188 ceph-mon[92425]: pgmap v19673: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:57 smithi161 ceph-mon[91579]: pgmap v19674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:34:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:57 smithi188 ceph-mon[92425]: pgmap v19674: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: pgmap v19675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:35:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:35:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:35:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:35:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:34:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: pgmap v19675: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:35:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:35:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:35:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:35:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:34:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:01 smithi161 ceph-mon[91579]: pgmap v19676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:01 smithi188 ceph-mon[92425]: pgmap v19676: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:03 smithi161 ceph-mon[91579]: pgmap v19677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:03 smithi188 ceph-mon[92425]: pgmap v19677: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:06.096 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:05 smithi161 ceph-mon[91579]: pgmap v19678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:06.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:35:06.038+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:35:06.097 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:35:06.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:35:06.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:35:06.098 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:35:06.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:35:06.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:35:06.099 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:35:06.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:35:06.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:35:06.100 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:35:06.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:35:06.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:35:06.101 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:35:06.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:35:06.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:35:06.102 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:35:06.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:35:06.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:35:06.103 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:35:06.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:35:06.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:35:06.104 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:35:06.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:35:06.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:35:06.105 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:35:06.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:35:06.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:35:06.106 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:35:06.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:35:06.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:35:06.107 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:35:06.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:35:06.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:35:06.108 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:35:06.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:35:06.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:35:06.109 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:35:06.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:35:06.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:35:06.110 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:35:06.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:35:06.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:35:06.111 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:35:06.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:35:06.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:35:06.112 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1199] Local address: 172.21.15.161, port 33542 2022-07-03T21:35:06.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1199] Peer address: 172.21.15.101, port 22 2022-07-03T21:35:06.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1199] Connection lost 2022-07-03T21:35:06.113 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1199] Aborting connection 2022-07-03T21:35:06.114 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:35:06.114 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:05 smithi188 ceph-mon[92425]: pgmap v19678: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:35:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:35:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:35:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:35:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:06 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:35:07.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:35:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:35:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:35:07.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:35:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:35:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:35:07.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:35:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:35:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:35:07.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:35:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:35:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:35:07.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:35:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:35:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:35:07.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:35:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:35:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:35:07.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:35:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:35:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:35:07.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:35:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:35:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:35:07.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:35:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:35:07.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:35:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:35:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:35:07.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:35:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:35:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:35:07.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:35:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:35:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:35:07.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:35:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:35:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:35:07.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:35:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:35:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:35:07.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:35:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:35:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:35:07.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:35:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1199] Local address: 172.21.15.161, port 33542 2022-07-03T21:35:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1199] Peer address: 172.21.15.101, port 22 2022-07-03T21:35:07.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1199] Connection lost 2022-07-03T21:35:07.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1199] Aborting connection 2022-07-03T21:35:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:35:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:06 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:06 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:35:07.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:35:07.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:35:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:35:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:35:07.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:35:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:35:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:35:07.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:35:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:35:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:35:07.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:35:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:35:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:35:07.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:35:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:35:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:35:07.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:35:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:35:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:35:07.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:35:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:35:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:35:07.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:35:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:35:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:35:07.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:35:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:35:07.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:35:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:35:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:35:07.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:35:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:35:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:35:07.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:35:07.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:35:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:35:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:35:07.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:35:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1199] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:35:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1199] Local address: 172.21.15.161, port 33542 2022-07-03T21:35:07.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1199] Peer address: 172.21.15.101, port 22 2022-07-03T21:35:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1199] Connection lost 2022-07-03T21:35:07.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1199] Aborting connection 2022-07-03T21:35:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:07 smithi161 ceph-mon[91579]: pgmap v19679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:07 smithi188 ceph-mon[92425]: pgmap v19679: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:09 smithi161 ceph-mon[91579]: pgmap v19680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:09 smithi188 ceph-mon[92425]: pgmap v19680: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:11.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:11 smithi161 ceph-mon[91579]: pgmap v19681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:11 smithi188 ceph-mon[92425]: pgmap v19681: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:13 smithi161 ceph-mon[91579]: pgmap v19682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:13 smithi188 ceph-mon[92425]: pgmap v19682: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:15 smithi161 ceph-mon[91579]: pgmap v19683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:15 smithi188 ceph-mon[92425]: pgmap v19683: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:17 smithi161 ceph-mon[91579]: pgmap v19684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:17 smithi188 ceph-mon[92425]: pgmap v19684: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:19 smithi161 ceph-mon[91579]: pgmap v19685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:19 smithi188 ceph-mon[92425]: pgmap v19685: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:21.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:21 smithi161 ceph-mon[91579]: pgmap v19686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:21 smithi188 ceph-mon[92425]: pgmap v19686: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:23 smithi161 ceph-mon[91579]: pgmap v19687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:23 smithi188 ceph-mon[92425]: pgmap v19687: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:25 smithi161 ceph-mon[91579]: pgmap v19688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:25 smithi188 ceph-mon[92425]: pgmap v19688: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:27 smithi161 ceph-mon[91579]: pgmap v19689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:27 smithi188 ceph-mon[92425]: pgmap v19689: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:29 smithi161 ceph-mon[91579]: pgmap v19690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:29 smithi188 ceph-mon[92425]: pgmap v19690: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:31.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:31.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:31 smithi161 ceph-mon[91579]: pgmap v19691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:31 smithi188 ceph-mon[92425]: pgmap v19691: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:33 smithi161 ceph-mon[91579]: pgmap v19692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:33 smithi188 ceph-mon[92425]: pgmap v19692: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:35 smithi161 ceph-mon[91579]: pgmap v19693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:36.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:35 smithi188 ceph-mon[92425]: pgmap v19693: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:36.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:37 smithi161 ceph-mon[91579]: pgmap v19694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:37 smithi188 ceph-mon[92425]: pgmap v19694: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:40.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:35:39 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:35:39.971+0000 7f7276074700 -1 mgr.server handle_report got status from non-daemon mon.b 2022-07-03T21:35:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:39 smithi161 ceph-mon[91579]: pgmap v19695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:40.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:39 smithi188 ceph-mon[92425]: pgmap v19695: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:40.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:41.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:41.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-07-03T21:35:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:41 smithi161 ceph-mon[91579]: pgmap v19696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:41 smithi188 ceph-mon[92425]: pgmap v19696: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:43 smithi161 ceph-mon[91579]: pgmap v19697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:43 smithi188 ceph-mon[92425]: pgmap v19697: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:45 smithi188 ceph-mon[92425]: pgmap v19698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:45 smithi161 ceph-mon[91579]: pgmap v19698: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:47 smithi188 ceph-mon[92425]: pgmap v19699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:47 smithi161 ceph-mon[91579]: pgmap v19699: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:49 smithi188 ceph-mon[92425]: pgmap v19700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:49 smithi161 ceph-mon[91579]: pgmap v19700: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:51.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:51 smithi161 ceph-mon[91579]: pgmap v19701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:51 smithi188 ceph-mon[92425]: pgmap v19701: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:53 smithi161 ceph-mon[91579]: pgmap v19702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:53 smithi188 ceph-mon[92425]: pgmap v19702: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:35:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:35:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:55 smithi161 ceph-mon[91579]: pgmap v19703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:55 smithi188 ceph-mon[92425]: pgmap v19703: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:57 smithi161 ceph-mon[91579]: pgmap v19704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:35:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:57 smithi188 ceph-mon[92425]: pgmap v19704: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: pgmap v19705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:36:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:36:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:36:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:36:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:35:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: pgmap v19705: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:36:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:36:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:36:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:36:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:35:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:01.878 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:01 smithi161 ceph-mon[91579]: pgmap v19706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:01 smithi188 ceph-mon[92425]: pgmap v19706: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:03 smithi161 ceph-mon[91579]: pgmap v19707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:03 smithi188 ceph-mon[92425]: pgmap v19707: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:05 smithi161 ceph-mon[91579]: pgmap v19708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:05 smithi188 ceph-mon[92425]: pgmap v19708: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:06.855 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:36:06.537+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:36:06.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:36:06.856 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:36:06.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:36:06.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:36:06.857 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:36:06.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:36:06.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:36:06.858 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:36:06.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:36:06.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:36:06.859 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:36:06.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:36:06.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:36:06.860 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:36:06.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:36:06.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:36:06.861 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:36:06.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:36:06.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:36:06.862 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:36:06.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:36:06.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:36:06.863 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:36:06.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:36:06.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:36:06.864 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:36:06.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:36:06.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:36:06.865 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:36:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:36:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:36:06.866 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:36:06.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:36:06.867 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:36:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:36:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:36:06.868 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:36:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:36:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:36:06.869 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:36:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:36:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:36:06.870 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:36:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:36:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:36:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1201] Local address: 172.21.15.161, port 33554 2022-07-03T21:36:06.871 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1201] Peer address: 172.21.15.101, port 22 2022-07-03T21:36:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1201] Connection lost 2022-07-03T21:36:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1201] Aborting connection 2022-07-03T21:36:06.872 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:36:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:36:07.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:36:07.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:06 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:36:07.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:36:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:36:07.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:06 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:36:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:36:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:36:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:36:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:36:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:36:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:36:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:36:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:36:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:36:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:36:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:36:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:36:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:36:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:36:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:36:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:36:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:36:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:36:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:36:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:36:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:36:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:36:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:36:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:36:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:36:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:36:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:36:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:36:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:36:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:36:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:36:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:36:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:36:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:36:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:36:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:36:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:36:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:36:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:36:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:36:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:36:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:36:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:36:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:36:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:36:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:36:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:36:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1201] Local address: 172.21.15.161, port 33554 2022-07-03T21:36:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1201] Peer address: 172.21.15.101, port 22 2022-07-03T21:36:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1201] Connection lost 2022-07-03T21:36:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1201] Aborting connection 2022-07-03T21:36:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:07 smithi161 ceph-mon[91579]: pgmap v19709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:36:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:36:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:36:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:36:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:36:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:36:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:36:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:36:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:36:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:36:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:36:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:36:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:36:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:36:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:36:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:36:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:36:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:36:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:36:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:36:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:36:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:36:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:36:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:36:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:36:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:36:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:36:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:36:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:36:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:36:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:36:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:36:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:36:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:36:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:36:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:36:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:36:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:36:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:36:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:36:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:36:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:36:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:36:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:36:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:36:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:36:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1201] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:36:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1201] Local address: 172.21.15.161, port 33554 2022-07-03T21:36:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1201] Peer address: 172.21.15.101, port 22 2022-07-03T21:36:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1201] Connection lost 2022-07-03T21:36:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1201] Aborting connection 2022-07-03T21:36:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:07 smithi188 ceph-mon[92425]: pgmap v19709: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:09 smithi161 ceph-mon[91579]: pgmap v19710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:10.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:09 smithi188 ceph-mon[92425]: pgmap v19710: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:11 smithi188 ceph-mon[92425]: pgmap v19711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:11 smithi161 ceph-mon[91579]: pgmap v19711: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:13 smithi161 ceph-mon[91579]: pgmap v19712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:13 smithi188 ceph-mon[92425]: pgmap v19712: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:15 smithi161 ceph-mon[91579]: pgmap v19713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:15 smithi188 ceph-mon[92425]: pgmap v19713: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:17 smithi161 ceph-mon[91579]: pgmap v19714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:17 smithi188 ceph-mon[92425]: pgmap v19714: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:20.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:19 smithi161 ceph-mon[91579]: pgmap v19715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:20.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:19 smithi188 ceph-mon[92425]: pgmap v19715: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:21 smithi188 ceph-mon[92425]: pgmap v19716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:21 smithi161 ceph-mon[91579]: pgmap v19716: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:23 smithi161 ceph-mon[91579]: pgmap v19717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:23 smithi188 ceph-mon[92425]: pgmap v19717: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:25 smithi161 ceph-mon[91579]: pgmap v19718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:25 smithi188 ceph-mon[92425]: pgmap v19718: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:28.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:27 smithi161 ceph-mon[91579]: pgmap v19719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:27 smithi188 ceph-mon[92425]: pgmap v19719: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:30.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:29 smithi161 ceph-mon[91579]: pgmap v19720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:30.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:29 smithi188 ceph-mon[92425]: pgmap v19720: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:31 smithi188 ceph-mon[92425]: pgmap v19721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:31 smithi161 ceph-mon[91579]: pgmap v19721: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:33 smithi161 ceph-mon[91579]: pgmap v19722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:33 smithi188 ceph-mon[92425]: pgmap v19722: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:36.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:35 smithi161 ceph-mon[91579]: pgmap v19723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:35 smithi188 ceph-mon[92425]: pgmap v19723: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:38.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:37 smithi161 ceph-mon[91579]: pgmap v19724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:37 smithi188 ceph-mon[92425]: pgmap v19724: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:39 smithi161 ceph-mon[91579]: pgmap v19725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:40.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:39 smithi188 ceph-mon[92425]: pgmap v19725: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:41 smithi188 ceph-mon[92425]: pgmap v19726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:41 smithi161 ceph-mon[91579]: pgmap v19726: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:43 smithi161 ceph-mon[91579]: pgmap v19727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:43 smithi188 ceph-mon[92425]: pgmap v19727: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:45.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:46.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:45 smithi161 ceph-mon[91579]: pgmap v19728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:45 smithi188 ceph-mon[92425]: pgmap v19728: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:48.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:47 smithi161 ceph-mon[91579]: pgmap v19729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:47 smithi188 ceph-mon[92425]: pgmap v19729: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:49 smithi161 ceph-mon[91579]: pgmap v19730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:50.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:49 smithi188 ceph-mon[92425]: pgmap v19730: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:51 smithi188 ceph-mon[92425]: pgmap v19731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:51 smithi161 ceph-mon[91579]: pgmap v19731: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:54.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:53 smithi161 ceph-mon[91579]: pgmap v19732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:53 smithi188 ceph-mon[92425]: pgmap v19732: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:36:55.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:36:56.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:55 smithi161 ceph-mon[91579]: pgmap v19733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:55 smithi188 ceph-mon[92425]: pgmap v19733: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:58.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:57 smithi161 ceph-mon[91579]: pgmap v19734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:36:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:57 smithi188 ceph-mon[92425]: pgmap v19734: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:00.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: pgmap v19735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:37:00.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:37:00.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:37:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:37:00.131 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:36:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: pgmap v19735: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:37:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:37:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:37:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:37:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:36:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:01 smithi188 ceph-mon[92425]: pgmap v19736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:01 smithi161 ceph-mon[91579]: pgmap v19736: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:04.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:03 smithi161 ceph-mon[91579]: pgmap v19737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:03 smithi188 ceph-mon[92425]: pgmap v19737: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:05.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:06.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:05 smithi161 ceph-mon[91579]: pgmap v19738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:05 smithi188 ceph-mon[92425]: pgmap v19738: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:07.378 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:37:07.037+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:37:07.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:37:07.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:37:07.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:37:07.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:37:07.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:37:07.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:37:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:37:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:37:07.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:37:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:37:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:37:07.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:37:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:37:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:37:07.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:37:07.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:37:07.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:37:07.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:37:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:37:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:37:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:37:07.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:37:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:37:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:37:07.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:37:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:37:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:37:07.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:37:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:37:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:37:07.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:37:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:37:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:37:07.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:37:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:37:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:37:07.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:37:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:37:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:37:07.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:37:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:37:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:37:07.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:37:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:37:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:37:07.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1203] Local address: 172.21.15.161, port 33558 2022-07-03T21:37:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1203] Peer address: 172.21.15.101, port 22 2022-07-03T21:37:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1203] Connection lost 2022-07-03T21:37:07.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1203] Aborting connection 2022-07-03T21:37:07.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:37:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: pgmap v19739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:37:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:37:08.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:37:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:37:08.130 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:07 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:37:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:37:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:37:08.131 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:37:08.132 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:37:08.133 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:37:08.134 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:37:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:37:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:37:08.135 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:37:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:37:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:37:08.136 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:37:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:37:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:37:08.137 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:37:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:37:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:37:08.138 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:37:08.139 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:37:08.140 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:37:08.141 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:37:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:37:08.142 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:37:08.143 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:37:08.144 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:37:08.145 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:37:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:37:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1203] Local address: 172.21.15.161, port 33558 2022-07-03T21:37:08.146 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1203] Peer address: 172.21.15.101, port 22 2022-07-03T21:37:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1203] Connection lost 2022-07-03T21:37:08.147 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1203] Aborting connection 2022-07-03T21:37:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: pgmap v19739: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:37:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:37:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:37:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:37:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:07 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:37:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:37:08.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:37:08.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:37:08.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:37:08.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:37:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:37:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:37:08.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:37:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:37:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:37:08.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:37:08.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:37:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:37:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:37:08.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:37:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:37:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:37:08.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:37:08.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:37:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:37:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:37:08.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:37:08.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:37:08.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:37:08.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:37:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:37:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:37:08.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1203] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:37:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1203] Local address: 172.21.15.161, port 33558 2022-07-03T21:37:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1203] Peer address: 172.21.15.101, port 22 2022-07-03T21:37:08.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1203] Connection lost 2022-07-03T21:37:08.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1203] Aborting connection 2022-07-03T21:37:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:09 smithi161 ceph-mon[91579]: pgmap v19740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:10.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:09 smithi188 ceph-mon[92425]: pgmap v19740: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:11 smithi188 ceph-mon[92425]: pgmap v19741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:11 smithi161 ceph-mon[91579]: pgmap v19741: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:13 smithi161 ceph-mon[91579]: pgmap v19742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:13 smithi188 ceph-mon[92425]: pgmap v19742: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:15.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:16.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:15 smithi161 ceph-mon[91579]: pgmap v19743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:15 smithi188 ceph-mon[92425]: pgmap v19743: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:18.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:17 smithi161 ceph-mon[91579]: pgmap v19744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:17 smithi188 ceph-mon[92425]: pgmap v19744: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:19 smithi188 ceph-mon[92425]: pgmap v19745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:19 smithi161 ceph-mon[91579]: pgmap v19745: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:21 smithi188 ceph-mon[92425]: pgmap v19746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:21 smithi161 ceph-mon[91579]: pgmap v19746: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:24.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:23 smithi161 ceph-mon[91579]: pgmap v19747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:23 smithi188 ceph-mon[92425]: pgmap v19747: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:25.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:26.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:25 smithi161 ceph-mon[91579]: pgmap v19748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:25 smithi188 ceph-mon[92425]: pgmap v19748: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:27 smithi188 ceph-mon[92425]: pgmap v19749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:27 smithi161 ceph-mon[91579]: pgmap v19749: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:29 smithi188 ceph-mon[92425]: pgmap v19750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:29 smithi161 ceph-mon[91579]: pgmap v19750: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:31 smithi188 ceph-mon[92425]: pgmap v19751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:31 smithi161 ceph-mon[91579]: pgmap v19751: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:34.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:33 smithi161 ceph-mon[91579]: pgmap v19752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:33 smithi188 ceph-mon[92425]: pgmap v19752: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:35.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:35 smithi188 ceph-mon[92425]: pgmap v19753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:35 smithi161 ceph-mon[91579]: pgmap v19753: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:37 smithi188 ceph-mon[92425]: pgmap v19754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:37 smithi161 ceph-mon[91579]: pgmap v19754: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:39 smithi188 ceph-mon[92425]: pgmap v19755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:39 smithi161 ceph-mon[91579]: pgmap v19755: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:41.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:41 smithi188 ceph-mon[92425]: pgmap v19756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:41 smithi161 ceph-mon[91579]: pgmap v19756: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:44.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:43 smithi161 ceph-mon[91579]: pgmap v19757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:43 smithi188 ceph-mon[92425]: pgmap v19757: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:45 smithi188 ceph-mon[92425]: pgmap v19758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:45 smithi161 ceph-mon[91579]: pgmap v19758: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:47 smithi188 ceph-mon[92425]: pgmap v19759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:47 smithi161 ceph-mon[91579]: pgmap v19759: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:49 smithi188 ceph-mon[92425]: pgmap v19760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:49 smithi161 ceph-mon[91579]: pgmap v19760: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:51.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:51 smithi188 ceph-mon[92425]: pgmap v19761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:51 smithi161 ceph-mon[91579]: pgmap v19761: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:53 smithi188 ceph-mon[92425]: pgmap v19762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:54.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:53 smithi161 ceph-mon[91579]: pgmap v19762: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:37:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:37:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:55 smithi188 ceph-mon[92425]: pgmap v19763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:55 smithi161 ceph-mon[91579]: pgmap v19763: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:58.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:57 smithi188 ceph-mon[92425]: pgmap v19764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:37:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:57 smithi161 ceph-mon[91579]: pgmap v19764: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:00.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: pgmap v19765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:38:00.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:38:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:38:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:38:00.164 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:37:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: pgmap v19765: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:38:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:38:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:38:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:38:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:37:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:01.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:01 smithi188 ceph-mon[92425]: pgmap v19766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:01 smithi161 ceph-mon[91579]: pgmap v19766: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:03 smithi188 ceph-mon[92425]: pgmap v19767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:03 smithi161 ceph-mon[91579]: pgmap v19767: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:05 smithi188 ceph-mon[92425]: pgmap v19768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:05 smithi161 ceph-mon[91579]: pgmap v19768: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:07.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:38:07.537+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:38:07.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:38:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:38:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:38:07.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:38:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:38:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:38:07.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:38:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:38:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:38:07.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:38:07.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:38:07.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:38:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:38:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:38:07.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:38:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:38:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:38:07.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:38:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:38:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:38:07.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:38:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:38:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:38:07.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:38:07.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:38:07.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:38:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:38:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:38:07.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:38:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:38:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:38:07.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:38:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:38:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:38:07.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:38:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:38:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:38:07.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:38:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:38:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:38:07.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:38:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:38:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:38:07.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:38:07.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:38:07.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1205] Local address: 172.21.15.161, port 33562 2022-07-03T21:38:07.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1205] Peer address: 172.21.15.101, port 22 2022-07-03T21:38:07.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1205] Connection lost 2022-07-03T21:38:07.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1205] Aborting connection 2022-07-03T21:38:07.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:38:08.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:07 smithi188 ceph-mon[92425]: pgmap v19769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:38:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:38:08.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:07 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:38:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-mon[91579]: pgmap v19769: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:38:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:38:08.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:07 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:38:09.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:38:09.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:38:09.164 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:38:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:38:09.165 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:38:09.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:38:09.166 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:38:09.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:38:09.167 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:38:09.168 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:38:09.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:38:09.169 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:38:09.170 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:38:09.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:38:09.171 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:38:09.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:38:09.172 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:38:09.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:38:09.173 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:38:09.174 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:38:09.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:38:09.175 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:38:09.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:38:09.176 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:38:09.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:38:09.177 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:38:09.178 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:38:09.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:38:09.179 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:38:09.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:38:09.180 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:38:09.181 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:38:09.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:38:09.182 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:38:09.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:38:09.183 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:38:09.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:38:09.184 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:38:09.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:38:09.185 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:38:09.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:38:09.186 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:38:09.187 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:38:09.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:38:09.188 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:38:09.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:38:09.189 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:38:09.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1205] Local address: 172.21.15.161, port 33562 2022-07-03T21:38:09.190 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1205] Peer address: 172.21.15.101, port 22 2022-07-03T21:38:09.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1205] Connection lost 2022-07-03T21:38:09.191 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1205] Aborting connection 2022-07-03T21:38:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:38:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:38:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:38:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:38:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:38:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:38:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:38:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:38:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:38:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:38:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:38:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:38:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:38:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:38:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:38:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:38:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:38:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:38:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:38:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:38:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:38:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:38:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:38:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:38:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:38:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:38:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:38:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:38:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:38:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:38:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:38:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:38:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:38:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:38:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:38:09.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:38:09.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:38:09.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:38:09.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:38:09.397 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:38:09.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:38:09.398 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:38:09.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:38:09.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:38:09.399 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:38:09.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:38:09.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1205] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:38:09.400 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1205] Local address: 172.21.15.161, port 33562 2022-07-03T21:38:09.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1205] Peer address: 172.21.15.101, port 22 2022-07-03T21:38:09.401 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1205] Connection lost 2022-07-03T21:38:09.402 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1205] Aborting connection 2022-07-03T21:38:10.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:09 smithi188 ceph-mon[92425]: pgmap v19770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:10.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:09 smithi161 ceph-mon[91579]: pgmap v19770: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:11.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:11 smithi188 ceph-mon[92425]: pgmap v19771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:11 smithi161 ceph-mon[91579]: pgmap v19771: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:13 smithi188 ceph-mon[92425]: pgmap v19772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:13 smithi161 ceph-mon[91579]: pgmap v19772: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:15 smithi188 ceph-mon[92425]: pgmap v19773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:15 smithi161 ceph-mon[91579]: pgmap v19773: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:18.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:17 smithi188 ceph-mon[92425]: pgmap v19774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:17 smithi161 ceph-mon[91579]: pgmap v19774: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:20.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:19 smithi188 ceph-mon[92425]: pgmap v19775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:20.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:19 smithi161 ceph-mon[91579]: pgmap v19775: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:21.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:21 smithi188 ceph-mon[92425]: pgmap v19776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:21 smithi161 ceph-mon[91579]: pgmap v19776: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:23 smithi188 ceph-mon[92425]: pgmap v19777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:23 smithi161 ceph-mon[91579]: pgmap v19777: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:25.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:26.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:25 smithi188 ceph-mon[92425]: pgmap v19778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:25 smithi161 ceph-mon[91579]: pgmap v19778: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:28.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:27 smithi188 ceph-mon[92425]: pgmap v19779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:27 smithi161 ceph-mon[91579]: pgmap v19779: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:30.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:29 smithi188 ceph-mon[92425]: pgmap v19780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:30.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:29 smithi161 ceph-mon[91579]: pgmap v19780: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:31.912 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:31 smithi188 ceph-mon[92425]: pgmap v19781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:31 smithi161 ceph-mon[91579]: pgmap v19781: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:33 smithi188 ceph-mon[92425]: pgmap v19782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:33 smithi161 ceph-mon[91579]: pgmap v19782: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:35.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:36.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:35 smithi188 ceph-mon[92425]: pgmap v19783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:35 smithi161 ceph-mon[91579]: pgmap v19783: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:38.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:37 smithi188 ceph-mon[92425]: pgmap v19784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:37 smithi161 ceph-mon[91579]: pgmap v19784: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:40.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:39 smithi188 ceph-mon[92425]: pgmap v19785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:40.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:39 smithi161 ceph-mon[91579]: pgmap v19785: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:41 smithi161 ceph-mon[91579]: pgmap v19786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:41 smithi188 ceph-mon[92425]: pgmap v19786: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:44.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:43 smithi188 ceph-mon[92425]: pgmap v19787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:43 smithi161 ceph-mon[91579]: pgmap v19787: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:45.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:46.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:45 smithi188 ceph-mon[92425]: pgmap v19788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:45 smithi161 ceph-mon[91579]: pgmap v19788: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:48.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:47 smithi188 ceph-mon[92425]: pgmap v19789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:47 smithi161 ceph-mon[91579]: pgmap v19789: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:50.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:49 smithi188 ceph-mon[92425]: pgmap v19790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:50.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:49 smithi161 ceph-mon[91579]: pgmap v19790: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:51 smithi161 ceph-mon[91579]: pgmap v19791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:51 smithi188 ceph-mon[92425]: pgmap v19791: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:54.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:53 smithi188 ceph-mon[92425]: pgmap v19792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:53 smithi161 ceph-mon[91579]: pgmap v19792: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:38:55.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:38:56.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:55 smithi188 ceph-mon[92425]: pgmap v19793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:55 smithi161 ceph-mon[91579]: pgmap v19793: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:57 smithi161 ceph-mon[91579]: pgmap v19794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:38:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:57 smithi188 ceph-mon[92425]: pgmap v19794: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: pgmap v19795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:39:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:39:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:39:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:39:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:38:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: pgmap v19795: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:39:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:39:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:39:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:39:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:38:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:01 smithi161 ceph-mon[91579]: pgmap v19796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:01 smithi188 ceph-mon[92425]: pgmap v19796: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:04.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:03 smithi188 ceph-mon[92425]: pgmap v19797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:03 smithi161 ceph-mon[91579]: pgmap v19797: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:05.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:06.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:05 smithi188 ceph-mon[92425]: pgmap v19798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:05 smithi161 ceph-mon[91579]: pgmap v19798: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:07 smithi161 ceph-mon[91579]: pgmap v19799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:39:08.031+0000 7f7278839700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:39:08.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:39:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:39:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:39:08.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:39:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:39:08.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:39:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:39:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:39:08.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:39:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:39:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:39:08.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:39:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:39:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:39:08.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:39:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:39:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:39:08.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:39:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:39:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:39:08.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:39:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:39:08.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:39:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:39:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:39:08.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:39:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:39:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:39:08.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:39:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:39:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:39:08.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:39:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:39:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:39:08.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:39:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:39:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:39:08.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:39:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:39:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:39:08.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:39:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:39:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:39:08.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:39:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:39:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1207] Local address: 172.21.15.161, port 33596 2022-07-03T21:39:08.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1207] Peer address: 172.21.15.101, port 22 2022-07-03T21:39:08.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1207] Connection lost 2022-07-03T21:39:08.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1207] Aborting connection 2022-07-03T21:39:08.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:39:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:07 smithi188 ceph-mon[92425]: pgmap v19799: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:39:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:39:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:39:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:39:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:08 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:39:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:39:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:39:09.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:39:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:39:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:39:09.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:39:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:39:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:39:09.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:39:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:39:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:39:09.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:39:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:39:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:39:09.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:39:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:39:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:39:09.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:39:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:39:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:39:09.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:39:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:39:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:39:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:39:09.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:39:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:39:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:39:09.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:39:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:39:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:39:09.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:39:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:39:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:39:09.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:39:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:39:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:39:09.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:39:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:39:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:39:09.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:39:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:39:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:39:09.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:39:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:39:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:39:09.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1207] Local address: 172.21.15.161, port 33596 2022-07-03T21:39:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1207] Peer address: 172.21.15.101, port 22 2022-07-03T21:39:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1207] Connection lost 2022-07-03T21:39:09.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1207] Aborting connection 2022-07-03T21:39:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:39:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:39:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:39:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:08 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:39:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:08 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:39:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:39:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:39:09.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:39:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:39:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:39:09.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:39:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:39:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:39:09.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:39:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:39:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:39:09.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:39:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:39:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:39:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:39:09.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:39:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:39:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:39:09.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:39:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:39:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:39:09.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:39:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:39:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:39:09.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:39:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:39:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:39:09.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:39:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:39:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:39:09.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:39:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:39:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:39:09.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:39:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:39:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:39:09.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:39:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:39:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:39:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:39:09.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:39:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:39:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:39:09.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:39:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1207] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:39:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1207] Local address: 172.21.15.161, port 33596 2022-07-03T21:39:09.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1207] Peer address: 172.21.15.101, port 22 2022-07-03T21:39:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1207] Connection lost 2022-07-03T21:39:09.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1207] Aborting connection 2022-07-03T21:39:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:09 smithi161 ceph-mon[91579]: pgmap v19800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:09 smithi188 ceph-mon[92425]: pgmap v19800: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:11 smithi161 ceph-mon[91579]: pgmap v19801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:11 smithi188 ceph-mon[92425]: pgmap v19801: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:13 smithi188 ceph-mon[92425]: pgmap v19802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:13 smithi161 ceph-mon[91579]: pgmap v19802: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:16.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:15 smithi188 ceph-mon[92425]: pgmap v19803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:15 smithi161 ceph-mon[91579]: pgmap v19803: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:17 smithi161 ceph-mon[91579]: pgmap v19804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:17 smithi188 ceph-mon[92425]: pgmap v19804: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:19 smithi161 ceph-mon[91579]: pgmap v19805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:19 smithi188 ceph-mon[92425]: pgmap v19805: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:21 smithi161 ceph-mon[91579]: pgmap v19806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:21 smithi188 ceph-mon[92425]: pgmap v19806: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:24.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:23 smithi188 ceph-mon[92425]: pgmap v19807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:23 smithi161 ceph-mon[91579]: pgmap v19807: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:25 smithi161 ceph-mon[91579]: pgmap v19808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:25 smithi188 ceph-mon[92425]: pgmap v19808: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:27 smithi161 ceph-mon[91579]: pgmap v19809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:27 smithi188 ceph-mon[92425]: pgmap v19809: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:29 smithi161 ceph-mon[91579]: pgmap v19810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:29 smithi188 ceph-mon[92425]: pgmap v19810: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:31 smithi161 ceph-mon[91579]: pgmap v19811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:31 smithi188 ceph-mon[92425]: pgmap v19811: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:34.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:33 smithi188 ceph-mon[92425]: pgmap v19812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:33 smithi161 ceph-mon[91579]: pgmap v19812: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:35 smithi161 ceph-mon[91579]: pgmap v19813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:35 smithi188 ceph-mon[92425]: pgmap v19813: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:37 smithi161 ceph-mon[91579]: pgmap v19814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:37 smithi188 ceph-mon[92425]: pgmap v19814: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:39 smithi161 ceph-mon[91579]: pgmap v19815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:39 smithi188 ceph-mon[92425]: pgmap v19815: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:41 smithi161 ceph-mon[91579]: pgmap v19816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:41 smithi188 ceph-mon[92425]: pgmap v19816: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:43 smithi161 ceph-mon[91579]: pgmap v19817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:43 smithi188 ceph-mon[92425]: pgmap v19817: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:45 smithi161 ceph-mon[91579]: pgmap v19818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:45 smithi188 ceph-mon[92425]: pgmap v19818: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:47 smithi161 ceph-mon[91579]: pgmap v19819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:47 smithi188 ceph-mon[92425]: pgmap v19819: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:49 smithi161 ceph-mon[91579]: pgmap v19820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:49 smithi188 ceph-mon[92425]: pgmap v19820: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:51 smithi161 ceph-mon[91579]: pgmap v19821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:52.166 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:51 smithi188 ceph-mon[92425]: pgmap v19821: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:53 smithi161 ceph-mon[91579]: pgmap v19822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:53 smithi188 ceph-mon[92425]: pgmap v19822: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:39:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:39:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:55 smithi161 ceph-mon[91579]: pgmap v19823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:55 smithi188 ceph-mon[92425]: pgmap v19823: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:57 smithi161 ceph-mon[91579]: pgmap v19824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:39:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:57 smithi188 ceph-mon[92425]: pgmap v19824: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: pgmap v19825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:40:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:40:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:40:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:40:00.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:39:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: pgmap v19825: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:40:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:40:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:40:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:40:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:39:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:01.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:40:01.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:40:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:01 smithi161 ceph-mon[91579]: pgmap v19826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:01 smithi188 ceph-mon[92425]: pgmap v19826: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:03 smithi161 ceph-mon[91579]: pgmap v19827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:03 smithi188 ceph-mon[92425]: pgmap v19827: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:05 smithi161 ceph-mon[91579]: pgmap v19828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:05 smithi188 ceph-mon[92425]: pgmap v19828: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:08.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:07 smithi161 ceph-mon[91579]: pgmap v19829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:07 smithi188 ceph-mon[92425]: pgmap v19829: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:08.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:40:08.532+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:40:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:40:08.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:40:08.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:40:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:40:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:40:08.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:40:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:40:08.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:40:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:40:08.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:40:08.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:40:08.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:40:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:40:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:40:08.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:40:08.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:40:08.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:40:08.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:40:08.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:40:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:40:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:40:08.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:40:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:40:08.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:40:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:40:08.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:40:08.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:40:08.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:40:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:40:08.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:40:08.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:40:08.898 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1209] Local address: 172.21.15.161, port 33614 2022-07-03T21:40:08.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1209] Peer address: 172.21.15.101, port 22 2022-07-03T21:40:08.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1209] Connection lost 2022-07-03T21:40:08.899 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1209] Aborting connection 2022-07-03T21:40:08.900 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:40:09.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:40:09.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:08 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:40:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:40:09.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:40:09.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:08 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:40:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:40:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:40:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:40:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:40:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:40:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:40:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:40:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:40:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:40:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:40:10.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:40:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:40:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:40:10.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:40:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:40:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:40:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:40:10.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:40:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:40:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:40:10.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:40:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:40:10.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:40:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:40:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:40:10.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:40:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:40:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:40:10.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:40:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:40:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:40:10.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:40:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:40:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:40:10.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:40:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:40:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:40:10.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:40:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:40:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:40:10.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:40:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:40:10.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:40:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:40:10.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:40:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1209] Local address: 172.21.15.161, port 33614 2022-07-03T21:40:10.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1209] Peer address: 172.21.15.101, port 22 2022-07-03T21:40:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1209] Connection lost 2022-07-03T21:40:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1209] Aborting connection 2022-07-03T21:40:10.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: pgmap v19830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.397 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.398 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:40:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:40:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:40:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:40:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:40:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:40:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:40:10.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:40:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:40:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:40:10.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:40:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:40:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:40:10.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:40:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:40:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:40:10.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:40:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:40:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:40:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:40:10.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:40:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:40:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:40:10.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:40:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:40:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:40:10.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:40:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:40:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:40:10.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:40:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:40:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:40:10.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:40:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:40:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:40:10.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:40:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:40:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:40:10.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:40:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:40:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:40:10.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:40:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:40:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:40:10.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1209] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:40:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1209] Local address: 172.21.15.161, port 33614 2022-07-03T21:40:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1209] Peer address: 172.21.15.101, port 22 2022-07-03T21:40:10.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1209] Connection lost 2022-07-03T21:40:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1209] Aborting connection 2022-07-03T21:40:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: pgmap v19830: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:10.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:40:10.431 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:11 smithi161 ceph-mon[91579]: pgmap v19831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:11 smithi188 ceph-mon[92425]: pgmap v19831: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:14.236 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:13 smithi161 ceph-mon[91579]: pgmap v19832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:13 smithi188 ceph-mon[92425]: pgmap v19832: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:15 smithi161 ceph-mon[91579]: pgmap v19833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:15 smithi188 ceph-mon[92425]: pgmap v19833: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:17 smithi161 ceph-mon[91579]: pgmap v19834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:17 smithi188 ceph-mon[92425]: pgmap v19834: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:19 smithi161 ceph-mon[91579]: pgmap v19835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:19 smithi188 ceph-mon[92425]: pgmap v19835: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:21 smithi161 ceph-mon[91579]: pgmap v19836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:21 smithi188 ceph-mon[92425]: pgmap v19836: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:23 smithi161 ceph-mon[91579]: pgmap v19837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:23 smithi188 ceph-mon[92425]: pgmap v19837: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:25 smithi161 ceph-mon[91579]: pgmap v19838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:25 smithi188 ceph-mon[92425]: pgmap v19838: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:27 smithi161 ceph-mon[91579]: pgmap v19839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:27 smithi188 ceph-mon[92425]: pgmap v19839: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:29 smithi161 ceph-mon[91579]: pgmap v19840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:29 smithi188 ceph-mon[92425]: pgmap v19840: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:31 smithi161 ceph-mon[91579]: pgmap v19841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:31 smithi188 ceph-mon[92425]: pgmap v19841: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:33 smithi161 ceph-mon[91579]: pgmap v19842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:33 smithi188 ceph-mon[92425]: pgmap v19842: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:35 smithi161 ceph-mon[91579]: pgmap v19843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:35 smithi188 ceph-mon[92425]: pgmap v19843: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:37 smithi161 ceph-mon[91579]: pgmap v19844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:37 smithi188 ceph-mon[92425]: pgmap v19844: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:39 smithi161 ceph-mon[91579]: pgmap v19845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:39 smithi188 ceph-mon[92425]: pgmap v19845: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:41 smithi161 ceph-mon[91579]: pgmap v19846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:41 smithi188 ceph-mon[92425]: pgmap v19846: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:43 smithi161 ceph-mon[91579]: pgmap v19847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:43 smithi188 ceph-mon[92425]: pgmap v19847: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:45 smithi161 ceph-mon[91579]: pgmap v19848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:45 smithi188 ceph-mon[92425]: pgmap v19848: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:47 smithi161 ceph-mon[91579]: pgmap v19849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:47 smithi188 ceph-mon[92425]: pgmap v19849: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:49 smithi161 ceph-mon[91579]: pgmap v19850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:49 smithi188 ceph-mon[92425]: pgmap v19850: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:51 smithi161 ceph-mon[91579]: pgmap v19851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:51 smithi188 ceph-mon[92425]: pgmap v19851: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:53 smithi161 ceph-mon[91579]: pgmap v19852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:53 smithi188 ceph-mon[92425]: pgmap v19852: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:40:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:40:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:55 smithi161 ceph-mon[91579]: pgmap v19853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:55 smithi188 ceph-mon[92425]: pgmap v19853: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:57 smithi161 ceph-mon[91579]: pgmap v19854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:40:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:57 smithi188 ceph-mon[92425]: pgmap v19854: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: pgmap v19855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:41:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:41:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:40:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: pgmap v19855: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:41:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:41:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:40:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:01 smithi161 ceph-mon[91579]: pgmap v19856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:01 smithi188 ceph-mon[92425]: pgmap v19856: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:03 smithi161 ceph-mon[91579]: pgmap v19857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:03 smithi188 ceph-mon[92425]: pgmap v19857: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:05 smithi161 ceph-mon[91579]: pgmap v19858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:05 smithi188 ceph-mon[92425]: pgmap v19858: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:07 smithi161 ceph-mon[91579]: pgmap v19859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:07 smithi188 ceph-mon[92425]: pgmap v19859: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:09.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:41:09.512+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:41:09.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:41:09.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:41:09.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:41:09.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:41:09.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:41:09.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:41:09.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:41:09.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:41:09.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:41:09.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:41:09.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:41:09.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:41:09.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:41:09.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:41:09.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:41:09.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:41:09.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:41:09.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:41:09.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:41:09.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:41:09.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:41:09.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:41:09.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:41:09.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:41:09.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:41:09.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:41:09.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:41:09.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:41:09.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:41:09.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:41:09.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:41:09.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:41:09.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:41:09.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:41:09.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:41:09.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:41:09.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:41:09.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:41:09.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:41:09.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:41:09.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:41:09.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:41:09.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:41:09.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:41:09.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:41:09.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1211] Local address: 172.21.15.161, port 33620 2022-07-03T21:41:09.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1211] Peer address: 172.21.15.101, port 22 2022-07-03T21:41:09.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1211] Connection lost 2022-07-03T21:41:09.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1211] Aborting connection 2022-07-03T21:41:09.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:41:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-mon[91579]: pgmap v19860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:10.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:41:10.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:41:10.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:09 smithi188 ceph-mon[92425]: pgmap v19860: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:41:10.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:41:10.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:41:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:41:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:41:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:41:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:41:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:41:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:41:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:41:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:41:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:41:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:41:11.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:41:11.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:41:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:41:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:41:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:41:11.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:41:11.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:41:11.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:41:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:41:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:41:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:41:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:41:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:41:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:41:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:41:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:41:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:41:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:41:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:41:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:41:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:41:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:41:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:41:11.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:41:11.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:41:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:41:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:41:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:41:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:41:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:41:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:41:11.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:41:11.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:41:11.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:41:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:41:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:41:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1211] Local address: 172.21.15.161, port 33620 2022-07-03T21:41:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1211] Peer address: 172.21.15.101, port 22 2022-07-03T21:41:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1211] Connection lost 2022-07-03T21:41:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1211] Aborting connection 2022-07-03T21:41:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:41:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:41:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:41:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:41:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:41:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:41:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:41:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:41:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:41:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:41:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:41:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:41:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:41:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:41:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:41:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:41:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:41:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:41:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:41:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:41:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:41:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:41:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:41:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:41:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:41:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:41:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:41:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:41:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:41:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:41:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:41:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:41:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:41:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:41:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:41:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:41:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:41:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:41:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:41:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:41:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:41:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:41:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:41:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:41:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:41:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:41:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1211] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:41:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1211] Local address: 172.21.15.161, port 33620 2022-07-03T21:41:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1211] Peer address: 172.21.15.101, port 22 2022-07-03T21:41:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1211] Connection lost 2022-07-03T21:41:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1211] Aborting connection 2022-07-03T21:41:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:11 smithi161 ceph-mon[91579]: pgmap v19861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:11 smithi188 ceph-mon[92425]: pgmap v19861: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:13 smithi161 ceph-mon[91579]: pgmap v19862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:13 smithi188 ceph-mon[92425]: pgmap v19862: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:15 smithi161 ceph-mon[91579]: pgmap v19863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:15 smithi188 ceph-mon[92425]: pgmap v19863: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:17 smithi161 ceph-mon[91579]: pgmap v19864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:17 smithi188 ceph-mon[92425]: pgmap v19864: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:19 smithi161 ceph-mon[91579]: pgmap v19865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:19 smithi188 ceph-mon[92425]: pgmap v19865: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:21 smithi161 ceph-mon[91579]: pgmap v19866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:21 smithi188 ceph-mon[92425]: pgmap v19866: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:23 smithi161 ceph-mon[91579]: pgmap v19867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:23 smithi188 ceph-mon[92425]: pgmap v19867: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:25 smithi161 ceph-mon[91579]: pgmap v19868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:25 smithi188 ceph-mon[92425]: pgmap v19868: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:27 smithi161 ceph-mon[91579]: pgmap v19869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:27 smithi188 ceph-mon[92425]: pgmap v19869: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:29 smithi161 ceph-mon[91579]: pgmap v19870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:29 smithi188 ceph-mon[92425]: pgmap v19870: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:31 smithi161 ceph-mon[91579]: pgmap v19871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:31 smithi188 ceph-mon[92425]: pgmap v19871: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:41:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:33 smithi161 ceph-mon[91579]: pgmap v19872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:33 smithi188 ceph-mon[92425]: pgmap v19872: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:35 smithi161 ceph-mon[91579]: pgmap v19873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:35 smithi188 ceph-mon[92425]: pgmap v19873: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:37 smithi161 ceph-mon[91579]: pgmap v19874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:37 smithi188 ceph-mon[92425]: pgmap v19874: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:39 smithi161 ceph-mon[91579]: pgmap v19875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:39 smithi188 ceph-mon[92425]: pgmap v19875: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:41 smithi161 ceph-mon[91579]: pgmap v19876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:41 smithi188 ceph-mon[92425]: pgmap v19876: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:43 smithi161 ceph-mon[91579]: pgmap v19877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:43 smithi188 ceph-mon[92425]: pgmap v19877: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:45 smithi161 ceph-mon[91579]: pgmap v19878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:45 smithi188 ceph-mon[92425]: pgmap v19878: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:47 smithi161 ceph-mon[91579]: pgmap v19879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:47 smithi188 ceph-mon[92425]: pgmap v19879: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:49 smithi161 ceph-mon[91579]: pgmap v19880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:49 smithi188 ceph-mon[92425]: pgmap v19880: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:51 smithi161 ceph-mon[91579]: pgmap v19881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:51 smithi188 ceph-mon[92425]: pgmap v19881: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:53 smithi161 ceph-mon[91579]: pgmap v19882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:53 smithi188 ceph-mon[92425]: pgmap v19882: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:41:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:41:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:55 smithi161 ceph-mon[91579]: pgmap v19883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:55 smithi188 ceph-mon[92425]: pgmap v19883: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:57 smithi161 ceph-mon[91579]: pgmap v19884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:41:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:57 smithi188 ceph-mon[92425]: pgmap v19884: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: pgmap v19885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:42:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:42:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:42:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:42:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:41:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: pgmap v19885: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:42:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:42:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:42:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:42:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:41:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:01 smithi161 ceph-mon[91579]: pgmap v19886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:01 smithi188 ceph-mon[92425]: pgmap v19886: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:03 smithi161 ceph-mon[91579]: pgmap v19887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:03 smithi188 ceph-mon[92425]: pgmap v19887: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:05 smithi161 ceph-mon[91579]: pgmap v19888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:05 smithi188 ceph-mon[92425]: pgmap v19888: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:07 smithi161 ceph-mon[91579]: pgmap v19889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:07 smithi188 ceph-mon[92425]: pgmap v19889: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:09 smithi161 ceph-mon[91579]: pgmap v19890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:10.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:42:10.014+0000 7f7274070700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:42:10.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:42:10.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:42:10.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:42:10.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:42:10.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:42:10.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:42:10.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:42:10.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:42:10.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:42:10.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:42:10.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:42:10.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:42:10.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:42:10.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:42:10.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:42:10.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:42:10.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:42:10.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:42:10.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:42:10.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:42:10.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:42:10.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:42:10.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:42:10.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:42:10.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:42:10.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:42:10.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:42:10.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:42:10.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:42:10.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:42:10.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:42:10.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:42:10.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:42:10.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:42:10.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:42:10.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:42:10.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:42:10.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:42:10.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:42:10.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:42:10.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:42:10.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:42:10.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:42:10.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:42:10.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:42:10.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1213] Local address: 172.21.15.161, port 33624 2022-07-03T21:42:10.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1213] Peer address: 172.21.15.101, port 22 2022-07-03T21:42:10.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1213] Connection lost 2022-07-03T21:42:10.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1213] Aborting connection 2022-07-03T21:42:10.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:42:10.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:09 smithi188 ceph-mon[92425]: pgmap v19890: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:42:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:42:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:42:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:42:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:10 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:42:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:42:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:42:11.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:42:11.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:42:11.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:42:11.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:42:11.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:42:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:42:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:42:11.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:42:11.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:42:11.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:42:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:42:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:42:11.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:42:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:42:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:42:11.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:42:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:42:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:42:11.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:42:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:42:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:42:11.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:42:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:42:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:42:11.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:42:11.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:42:11.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:42:11.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:42:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:42:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:42:11.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:42:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:42:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:42:11.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:42:11.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:42:11.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:42:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:42:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:42:11.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:42:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:42:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:42:11.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1213] Local address: 172.21.15.161, port 33624 2022-07-03T21:42:11.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1213] Peer address: 172.21.15.101, port 22 2022-07-03T21:42:11.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1213] Connection lost 2022-07-03T21:42:11.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1213] Aborting connection 2022-07-03T21:42:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:42:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:42:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:42:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:10 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:42:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:10 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:42:11.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:42:11.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:42:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:42:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:42:11.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:42:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:42:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:42:11.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:42:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:42:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:42:11.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:42:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:42:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:42:11.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:42:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:42:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:42:11.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:42:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:42:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:42:11.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:42:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:42:11.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:42:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:42:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:42:11.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:42:11.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:42:11.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:42:11.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:42:11.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:42:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:42:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:42:11.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1213] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1213] Local address: 172.21.15.161, port 33624 2022-07-03T21:42:11.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1213] Peer address: 172.21.15.101, port 22 2022-07-03T21:42:11.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1213] Connection lost 2022-07-03T21:42:11.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1213] Aborting connection 2022-07-03T21:42:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:11 smithi161 ceph-mon[91579]: pgmap v19891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:11 smithi188 ceph-mon[92425]: pgmap v19891: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:13 smithi161 ceph-mon[91579]: pgmap v19892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:13 smithi188 ceph-mon[92425]: pgmap v19892: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:15 smithi161 ceph-mon[91579]: pgmap v19893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:15 smithi188 ceph-mon[92425]: pgmap v19893: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:17 smithi161 ceph-mon[91579]: pgmap v19894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:17 smithi188 ceph-mon[92425]: pgmap v19894: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:19 smithi161 ceph-mon[91579]: pgmap v19895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:19 smithi188 ceph-mon[92425]: pgmap v19895: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:21 smithi161 ceph-mon[91579]: pgmap v19896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:21 smithi188 ceph-mon[92425]: pgmap v19896: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:23 smithi161 ceph-mon[91579]: pgmap v19897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:23 smithi188 ceph-mon[92425]: pgmap v19897: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:25 smithi161 ceph-mon[91579]: pgmap v19898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:25 smithi188 ceph-mon[92425]: pgmap v19898: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:27 smithi161 ceph-mon[91579]: pgmap v19899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:27 smithi188 ceph-mon[92425]: pgmap v19899: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:29 smithi161 ceph-mon[91579]: pgmap v19900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:29 smithi188 ceph-mon[92425]: pgmap v19900: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:31 smithi161 ceph-mon[91579]: pgmap v19901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:31 smithi188 ceph-mon[92425]: pgmap v19901: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:33 smithi161 ceph-mon[91579]: pgmap v19902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:33 smithi188 ceph-mon[92425]: pgmap v19902: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:35 smithi161 ceph-mon[91579]: pgmap v19903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:35 smithi188 ceph-mon[92425]: pgmap v19903: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:37 smithi161 ceph-mon[91579]: pgmap v19904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:37 smithi188 ceph-mon[92425]: pgmap v19904: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:39 smithi161 ceph-mon[91579]: pgmap v19905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:39 smithi188 ceph-mon[92425]: pgmap v19905: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:41 smithi161 ceph-mon[91579]: pgmap v19906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:41 smithi188 ceph-mon[92425]: pgmap v19906: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:43 smithi161 ceph-mon[91579]: pgmap v19907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:43 smithi188 ceph-mon[92425]: pgmap v19907: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:45 smithi161 ceph-mon[91579]: pgmap v19908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:45 smithi188 ceph-mon[92425]: pgmap v19908: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:47 smithi161 ceph-mon[91579]: pgmap v19909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:47 smithi188 ceph-mon[92425]: pgmap v19909: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:49 smithi161 ceph-mon[91579]: pgmap v19910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:49 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:49 smithi188 ceph-mon[92425]: pgmap v19910: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:49 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:51 smithi161 ceph-mon[91579]: pgmap v19911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:51 smithi188 ceph-mon[92425]: pgmap v19911: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:53 smithi161 ceph-mon[91579]: pgmap v19912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:53 smithi188 ceph-mon[92425]: pgmap v19912: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:42:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:42:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:55 smithi161 ceph-mon[91579]: pgmap v19913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:55 smithi188 ceph-mon[92425]: pgmap v19913: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:57 smithi161 ceph-mon[91579]: pgmap v19914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:42:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:57 smithi188 ceph-mon[92425]: pgmap v19914: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: pgmap v19915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:43:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:43:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:43:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:43:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:42:59 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: pgmap v19915: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:43:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:43:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:43:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:43:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:42:59 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:01 smithi161 ceph-mon[91579]: pgmap v19916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:01 smithi188 ceph-mon[92425]: pgmap v19916: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:03 smithi161 ceph-mon[91579]: pgmap v19917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:03 smithi188 ceph-mon[92425]: pgmap v19917: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:05 smithi161 ceph-mon[91579]: pgmap v19918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:05 smithi188 ceph-mon[92425]: pgmap v19918: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:07 smithi161 ceph-mon[91579]: pgmap v19919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:07 smithi188 ceph-mon[92425]: pgmap v19919: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:09 smithi161 ceph-mon[91579]: pgmap v19920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:09 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:09 smithi188 ceph-mon[92425]: pgmap v19920: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:09 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:10.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:43:10.520+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:43:10.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:43:10.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:43:10.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:43:10.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:43:10.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:43:10.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:43:10.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:43:10.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:43:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:43:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:43:10.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:43:10.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:43:10.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:43:10.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:43:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:43:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:43:10.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:43:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:43:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:43:10.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:43:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:43:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:43:10.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:43:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:43:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:43:10.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:43:10.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:43:10.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:43:10.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:43:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:43:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:43:10.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:43:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:43:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:43:10.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:43:10.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:43:10.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:43:10.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:43:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:43:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:43:10.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:43:10.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:43:10.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:43:10.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:43:10.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:43:10.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1215] Local address: 172.21.15.161, port 33628 2022-07-03T21:43:10.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1215] Peer address: 172.21.15.101, port 22 2022-07-03T21:43:10.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1215] Connection lost 2022-07-03T21:43:10.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1215] Aborting connection 2022-07-03T21:43:10.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:43:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:43:11.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:43:11.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:43:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:43:11.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:43:11.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:43:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:11 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:43:12.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:11 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:43:12.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:43:12.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:43:12.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:43:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:43:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:43:12.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:43:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:43:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:43:12.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:43:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:43:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:43:12.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:43:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:43:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:43:12.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:43:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:43:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:43:12.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:43:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:43:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:43:12.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:43:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:43:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:43:12.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:43:12.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:43:12.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:43:12.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:43:12.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:43:12.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:43:12.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:43:12.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:43:12.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:43:12.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:43:12.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:43:12.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:43:12.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:43:12.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:43:12.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:43:12.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:43:12.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:43:12.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:43:12.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:43:12.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:43:12.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:43:12.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:43:12.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1215] Local address: 172.21.15.161, port 33628 2022-07-03T21:43:12.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1215] Peer address: 172.21.15.101, port 22 2022-07-03T21:43:12.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1215] Connection lost 2022-07-03T21:43:12.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1215] Aborting connection 2022-07-03T21:43:12.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:11 smithi161 ceph-mon[91579]: pgmap v19921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:11 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:43:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:11 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:43:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:43:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:43:12.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:43:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:43:12.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:43:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:43:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:43:12.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:43:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:43:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:43:12.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:43:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:43:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:43:12.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:43:12.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:43:12.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:43:12.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:43:12.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:43:12.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:43:12.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:43:12.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:43:12.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:43:12.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:43:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:43:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:43:12.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:43:12.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:43:12.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:43:12.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:43:12.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:43:12.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:43:12.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:43:12.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:43:12.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:43:12.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:43:12.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:43:12.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:43:12.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:43:12.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:43:12.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:43:12.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:43:12.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:43:12.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:43:12.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:43:12.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1215] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:43:12.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1215] Local address: 172.21.15.161, port 33628 2022-07-03T21:43:12.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1215] Peer address: 172.21.15.101, port 22 2022-07-03T21:43:12.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1215] Connection lost 2022-07-03T21:43:12.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1215] Aborting connection 2022-07-03T21:43:12.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:11 smithi188 ceph-mon[92425]: pgmap v19921: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: pgmap v19922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.382 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.383 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:13 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: pgmap v19922: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:43:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:14.417 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:13 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:43:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:14 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 23985k 2022-07-03T21:43:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:14 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 24560640: error parsing value: Value '24560640' is below minimum 939524096 2022-07-03T21:43:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:14 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 23985k 2022-07-03T21:43:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:14 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 24560640: error parsing value: Value '24560640' is below minimum 939524096 2022-07-03T21:43:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:15 smithi161 ceph-mon[91579]: pgmap v19923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:16 smithi188 ceph-mon[92425]: pgmap v19923: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:18 smithi161 ceph-mon[91579]: pgmap v19924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:18 smithi188 ceph-mon[92425]: pgmap v19924: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:20 smithi161 ceph-mon[91579]: pgmap v19925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:20 smithi188 ceph-mon[92425]: pgmap v19925: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:21 smithi161 ceph-mon[91579]: pgmap v19926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:21 smithi188 ceph-mon[92425]: pgmap v19926: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:23 smithi161 ceph-mon[91579]: pgmap v19927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:23 smithi188 ceph-mon[92425]: pgmap v19927: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:25 smithi161 ceph-mon[91579]: pgmap v19928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:25 smithi188 ceph-mon[92425]: pgmap v19928: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:27 smithi161 ceph-mon[91579]: pgmap v19929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:27 smithi188 ceph-mon[92425]: pgmap v19929: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:29 smithi161 ceph-mon[91579]: pgmap v19930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:30.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:29 smithi188 ceph-mon[92425]: pgmap v19930: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:30.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:31 smithi161 ceph-mon[91579]: pgmap v19931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:31 smithi188 ceph-mon[92425]: pgmap v19931: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:33 smithi161 ceph-mon[91579]: pgmap v19932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:33 smithi188 ceph-mon[92425]: pgmap v19932: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:34 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:34 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:35 smithi161 ceph-mon[91579]: pgmap v19933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:35 smithi188 ceph-mon[92425]: pgmap v19933: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:37 smithi161 ceph-mon[91579]: pgmap v19934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:37 smithi188 ceph-mon[92425]: pgmap v19934: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:39 smithi161 ceph-mon[91579]: pgmap v19935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:39 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:39 smithi188 ceph-mon[92425]: pgmap v19935: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:39 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:41 smithi161 ceph-mon[91579]: pgmap v19936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:41 smithi188 ceph-mon[92425]: pgmap v19936: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:43 smithi161 ceph-mon[91579]: pgmap v19937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:43 smithi188 ceph-mon[92425]: pgmap v19937: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:44 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:44 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:45 smithi161 ceph-mon[91579]: pgmap v19938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:45 smithi188 ceph-mon[92425]: pgmap v19938: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:47 smithi161 ceph-mon[91579]: pgmap v19939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:47 smithi188 ceph-mon[92425]: pgmap v19939: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:50 smithi161 ceph-mon[91579]: pgmap v19940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:50 smithi188 ceph-mon[92425]: pgmap v19940: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:51 smithi161 ceph-mon[91579]: pgmap v19941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:51 smithi188 ceph-mon[92425]: pgmap v19941: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:53 smithi161 ceph-mon[91579]: pgmap v19942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:53 smithi188 ceph-mon[92425]: pgmap v19942: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:43:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:54 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:54 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:43:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:55 smithi161 ceph-mon[91579]: pgmap v19943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:55 smithi188 ceph-mon[92425]: pgmap v19943: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:43:57 smithi161 ceph-mon[91579]: pgmap v19944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:43:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:43:57 smithi188 ceph-mon[92425]: pgmap v19944: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: pgmap v19945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:44:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:44:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:44:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:44:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: pgmap v19945: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:44:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:44:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:44:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:44:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:01 smithi161 ceph-mon[91579]: pgmap v19946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:01 smithi188 ceph-mon[92425]: pgmap v19946: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:04.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:03 smithi161 ceph-mon[91579]: pgmap v19947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:03 smithi188 ceph-mon[92425]: pgmap v19947: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:04 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:04 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:05 smithi161 ceph-mon[91579]: pgmap v19948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:05 smithi188 ceph-mon[92425]: pgmap v19948: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:08 smithi161 ceph-mon[91579]: pgmap v19949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:08 smithi188 ceph-mon[92425]: pgmap v19949: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:10.010 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:10 smithi161 ceph-mon[91579]: pgmap v19950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:10 smithi188 ceph-mon[92425]: pgmap v19950: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:11 smithi161 ceph-mon[91579]: pgmap v19951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:11 smithi188 ceph-mon[92425]: pgmap v19951: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-mon[91579]: pgmap v19952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:14.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:44:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:44:14.129 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:44:14.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:44:13.775+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:44:14.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:44:14.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:44:14.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:44:14.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:44:14.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:44:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:44:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:44:14.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:44:14.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:44:14.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:44:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:44:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:44:14.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:44:14.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:44:14.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:44:14.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:44:14.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:44:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:44:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:44:14.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:44:14.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:44:14.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:44:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:44:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:44:14.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:44:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:44:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:44:14.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:44:14.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:44:14.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:44:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:44:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:44:14.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:44:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:44:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:44:14.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:44:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:44:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:44:14.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:44:14.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:44:14.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:44:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:44:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:44:14.147 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:44:14.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:44:14.148 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1217] Local address: 172.21.15.161, port 33632 2022-07-03T21:44:14.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1217] Peer address: 172.21.15.101, port 22 2022-07-03T21:44:14.149 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1217] Connection lost 2022-07-03T21:44:14.150 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1217] Aborting connection 2022-07-03T21:44:14.151 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:44:13 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:13 smithi188 ceph-mon[92425]: pgmap v19952: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:44:14.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:44:14.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:13 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:44:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:14 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:44:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:14 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:44:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:44:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:44:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:44:15.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:44:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:44:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:44:15.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:44:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:44:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:44:15.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:44:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:44:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:44:15.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:44:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:44:15.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:44:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:44:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:44:15.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:44:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:44:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:44:15.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:44:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:44:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:44:15.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:44:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:44:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:44:15.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:44:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:44:15.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:44:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:44:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:44:15.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:44:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:44:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:44:15.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:44:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:44:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:44:15.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:44:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:44:15.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:44:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:44:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:44:15.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:44:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1217] Local address: 172.21.15.161, port 33632 2022-07-03T21:44:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1217] Peer address: 172.21.15.101, port 22 2022-07-03T21:44:15.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1217] Connection lost 2022-07-03T21:44:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1217] Aborting connection 2022-07-03T21:44:15.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:14 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:14 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:44:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:14 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:44:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:44:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:44:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:44:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:44:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:44:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:44:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:44:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:44:15.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:44:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:44:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:44:15.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:44:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:44:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:44:15.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:44:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:44:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:44:15.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:44:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:44:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:44:15.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:44:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:44:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:44:15.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:44:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:44:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:44:15.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:44:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:44:15.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:44:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:44:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:44:15.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:44:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:44:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:44:15.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:44:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:44:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:44:15.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:44:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:44:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:44:15.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:44:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:44:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:44:15.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:44:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1217] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:44:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1217] Local address: 172.21.15.161, port 33632 2022-07-03T21:44:15.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1217] Peer address: 172.21.15.101, port 22 2022-07-03T21:44:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1217] Connection lost 2022-07-03T21:44:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1217] Aborting connection 2022-07-03T21:44:15.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:14 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:15 smithi161 ceph-mon[91579]: pgmap v19953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:16 smithi188 ceph-mon[92425]: pgmap v19953: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:18 smithi161 ceph-mon[91579]: pgmap v19954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:18 smithi188 ceph-mon[92425]: pgmap v19954: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:20 smithi161 ceph-mon[91579]: pgmap v19955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:20 smithi188 ceph-mon[92425]: pgmap v19955: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:21 smithi161 ceph-mon[91579]: pgmap v19956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:21 smithi188 ceph-mon[92425]: pgmap v19956: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:23 smithi161 ceph-mon[91579]: pgmap v19957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:23 smithi188 ceph-mon[92425]: pgmap v19957: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:25.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:26 smithi161 ceph-mon[91579]: pgmap v19958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:26 smithi188 ceph-mon[92425]: pgmap v19958: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:28 smithi161 ceph-mon[91579]: pgmap v19959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:28 smithi188 ceph-mon[92425]: pgmap v19959: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:30 smithi161 ceph-mon[91579]: pgmap v19960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:30 smithi188 ceph-mon[92425]: pgmap v19960: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:31 smithi161 ceph-mon[91579]: pgmap v19961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:31 smithi188 ceph-mon[92425]: pgmap v19961: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:34 smithi161 ceph-mon[91579]: pgmap v19962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:34 smithi188 ceph-mon[92425]: pgmap v19962: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:36.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:36 smithi161 ceph-mon[91579]: pgmap v19963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:36 smithi188 ceph-mon[92425]: pgmap v19963: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:38 smithi161 ceph-mon[91579]: pgmap v19964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:38 smithi188 ceph-mon[92425]: pgmap v19964: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:40 smithi161 ceph-mon[91579]: pgmap v19965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:40 smithi188 ceph-mon[92425]: pgmap v19965: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:41 smithi161 ceph-mon[91579]: pgmap v19966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:41 smithi188 ceph-mon[92425]: pgmap v19966: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:44 smithi161 ceph-mon[91579]: pgmap v19967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:44 smithi188 ceph-mon[92425]: pgmap v19967: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:46 smithi161 ceph-mon[91579]: pgmap v19968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:46 smithi188 ceph-mon[92425]: pgmap v19968: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:48 smithi161 ceph-mon[91579]: pgmap v19969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:48 smithi188 ceph-mon[92425]: pgmap v19969: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:50 smithi161 ceph-mon[91579]: pgmap v19970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:50 smithi188 ceph-mon[92425]: pgmap v19970: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:51 smithi161 ceph-mon[91579]: pgmap v19971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:51 smithi188 ceph-mon[92425]: pgmap v19971: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:54.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:54 smithi161 ceph-mon[91579]: pgmap v19972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:54 smithi188 ceph-mon[92425]: pgmap v19972: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:44:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:44:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:56 smithi161 ceph-mon[91579]: pgmap v19973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:56 smithi188 ceph-mon[92425]: pgmap v19973: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:44:58 smithi161 ceph-mon[91579]: pgmap v19974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:44:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:44:58 smithi188 ceph-mon[92425]: pgmap v19974: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: pgmap v19975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:45:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:45:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:45:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: pgmap v19975: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:45:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:45:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:45:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:01 smithi161 ceph-mon[91579]: pgmap v19976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:01 smithi188 ceph-mon[92425]: pgmap v19976: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:04 smithi161 ceph-mon[91579]: pgmap v19977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:04 smithi188 ceph-mon[92425]: pgmap v19977: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:06.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:06 smithi161 ceph-mon[91579]: pgmap v19978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:06 smithi188 ceph-mon[92425]: pgmap v19978: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:08 smithi161 ceph-mon[91579]: pgmap v19979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:08 smithi188 ceph-mon[92425]: pgmap v19979: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:10 smithi161 ceph-mon[91579]: pgmap v19980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:10 smithi188 ceph-mon[92425]: pgmap v19980: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:11 smithi161 ceph-mon[91579]: pgmap v19981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:11 smithi188 ceph-mon[92425]: pgmap v19981: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:14.280 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-mon[91579]: pgmap v19982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:14.289 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:14 smithi188 ceph-mon[92425]: pgmap v19982: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:14.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:45:14.279+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:45:14.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:45:14.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:45:14.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:45:14.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:45:14.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:45:14.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:45:14.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:45:14.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:45:14.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:45:14.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:45:14.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:45:14.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:45:14.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:45:14.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:45:14.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:45:14.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:45:14.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:45:14.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:45:14.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:45:14.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:45:14.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:45:14.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:45:14.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:45:14.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:45:14.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:45:14.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:45:14.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:45:14.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:45:14.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:45:14.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:45:14.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:45:14.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:45:14.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:45:14.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:45:14.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:45:14.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:45:14.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:45:14.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:45:14.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:45:14.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:45:14.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:45:14.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:45:14.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:45:14.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:45:14.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:45:14.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1219] Local address: 172.21.15.161, port 33636 2022-07-03T21:45:14.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1219] Peer address: 172.21.15.101, port 22 2022-07-03T21:45:14.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1219] Connection lost 2022-07-03T21:45:14.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1219] Aborting connection 2022-07-03T21:45:14.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:45:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:45:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:45:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:45:15.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:45:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:45:15.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:45:15.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:45:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:45:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:45:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:45:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:45:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:45:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:45:16.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:45:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:45:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:45:16.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:45:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:45:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:45:16.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:45:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:45:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:45:16.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:45:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:45:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:45:16.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:45:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:45:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:45:16.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:45:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:45:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:45:16.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:45:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:45:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:45:16.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:45:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:45:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:45:16.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:45:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:45:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:45:16.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:45:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:45:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:45:16.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:45:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:45:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:45:16.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:45:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:45:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:45:16.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:45:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:45:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:45:16.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:45:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1219] Local address: 172.21.15.161, port 33636 2022-07-03T21:45:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1219] Peer address: 172.21.15.101, port 22 2022-07-03T21:45:16.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1219] Connection lost 2022-07-03T21:45:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1219] Aborting connection 2022-07-03T21:45:16.395 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:16 smithi161 ceph-mon[91579]: pgmap v19983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:45:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:45:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:45:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:45:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:45:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:45:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:45:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:45:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:45:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:45:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:45:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:45:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:45:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:45:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:45:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:45:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:45:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:45:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:45:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:45:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:45:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:45:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:45:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:45:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:45:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:45:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:45:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:45:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:45:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:45:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:45:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:45:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:45:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:45:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:45:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:45:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:45:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:45:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:45:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:45:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:45:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:45:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:45:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:45:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:45:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:45:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1219] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:45:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1219] Local address: 172.21.15.161, port 33636 2022-07-03T21:45:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1219] Peer address: 172.21.15.101, port 22 2022-07-03T21:45:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1219] Connection lost 2022-07-03T21:45:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1219] Aborting connection 2022-07-03T21:45:16.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:16 smithi188 ceph-mon[92425]: pgmap v19983: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:18 smithi161 ceph-mon[91579]: pgmap v19984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:18 smithi188 ceph-mon[92425]: pgmap v19984: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:20 smithi161 ceph-mon[91579]: pgmap v19985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:20 smithi188 ceph-mon[92425]: pgmap v19985: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:21 smithi161 ceph-mon[91579]: pgmap v19986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:21 smithi188 ceph-mon[92425]: pgmap v19986: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:24 smithi161 ceph-mon[91579]: pgmap v19987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:24 smithi188 ceph-mon[92425]: pgmap v19987: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:26 smithi161 ceph-mon[91579]: pgmap v19988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:26 smithi188 ceph-mon[92425]: pgmap v19988: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:28 smithi161 ceph-mon[91579]: pgmap v19989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:28 smithi188 ceph-mon[92425]: pgmap v19989: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:30 smithi161 ceph-mon[91579]: pgmap v19990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:30 smithi188 ceph-mon[92425]: pgmap v19990: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:31 smithi161 ceph-mon[91579]: pgmap v19991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:31 smithi188 ceph-mon[92425]: pgmap v19991: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:34 smithi161 ceph-mon[91579]: pgmap v19992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:34 smithi188 ceph-mon[92425]: pgmap v19992: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:36 smithi161 ceph-mon[91579]: pgmap v19993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:36 smithi188 ceph-mon[92425]: pgmap v19993: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:38 smithi161 ceph-mon[91579]: pgmap v19994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:38 smithi188 ceph-mon[92425]: pgmap v19994: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:40 smithi161 ceph-mon[91579]: pgmap v19995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:40 smithi188 ceph-mon[92425]: pgmap v19995: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:41 smithi161 ceph-mon[91579]: pgmap v19996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:41 smithi188 ceph-mon[92425]: pgmap v19996: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:44 smithi161 ceph-mon[91579]: pgmap v19997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:44 smithi188 ceph-mon[92425]: pgmap v19997: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:46 smithi161 ceph-mon[91579]: pgmap v19998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:46 smithi188 ceph-mon[92425]: pgmap v19998: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:48 smithi161 ceph-mon[91579]: pgmap v19999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:48 smithi188 ceph-mon[92425]: pgmap v19999: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:50 smithi161 ceph-mon[91579]: pgmap v20000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:50 smithi188 ceph-mon[92425]: pgmap v20000: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:51 smithi161 ceph-mon[91579]: pgmap v20001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:51 smithi188 ceph-mon[92425]: pgmap v20001: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:54 smithi161 ceph-mon[91579]: pgmap v20002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:54 smithi188 ceph-mon[92425]: pgmap v20002: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:45:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:45:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:56 smithi161 ceph-mon[91579]: pgmap v20003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:56 smithi188 ceph-mon[92425]: pgmap v20003: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:45:58 smithi161 ceph-mon[91579]: pgmap v20004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:45:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:45:58 smithi188 ceph-mon[92425]: pgmap v20004: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: pgmap v20005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:46:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:46:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:46:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:46:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: pgmap v20005: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:46:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:46:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:46:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:46:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:01 smithi161 ceph-mon[91579]: pgmap v20006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:01 smithi188 ceph-mon[92425]: pgmap v20006: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:04 smithi161 ceph-mon[91579]: pgmap v20007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:04 smithi188 ceph-mon[92425]: pgmap v20007: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:05.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:06 smithi161 ceph-mon[91579]: pgmap v20008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:06 smithi188 ceph-mon[92425]: pgmap v20008: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:08 smithi161 ceph-mon[91579]: pgmap v20009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:08 smithi188 ceph-mon[92425]: pgmap v20009: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:10 smithi161 ceph-mon[91579]: pgmap v20010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:10 smithi188 ceph-mon[92425]: pgmap v20010: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:11 smithi161 ceph-mon[91579]: pgmap v20011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:11 smithi188 ceph-mon[92425]: pgmap v20011: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-mon[91579]: pgmap v20012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:14 smithi188 ceph-mon[92425]: pgmap v20012: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:15.036 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:46:14.781+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:46:15.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:46:15.037 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:46:15.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:46:15.038 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:46:15.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:46:15.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:46:15.039 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:46:15.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:46:15.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:46:15.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:46:15.040 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:46:15.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:46:15.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:46:15.041 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:46:15.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:46:15.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:46:15.042 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:46:15.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:46:15.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:46:15.043 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:46:15.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:46:15.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:46:15.044 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:46:15.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:46:15.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:46:15.045 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:46:15.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:46:15.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:46:15.046 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:46:15.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:46:15.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:46:15.047 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:46:15.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:46:15.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:46:15.048 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:46:15.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:46:15.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:46:15.049 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:46:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:46:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:46:15.050 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:46:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:46:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:46:15.051 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:46:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:46:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1221] Local address: 172.21.15.161, port 33640 2022-07-03T21:46:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1221] Peer address: 172.21.15.101, port 22 2022-07-03T21:46:15.052 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1221] Connection lost 2022-07-03T21:46:15.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1221] Aborting connection 2022-07-03T21:46:15.053 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:46:14 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:46:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:15.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:46:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:46:15.163 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:46:15.304 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:15.305 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:46:15.305 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:46:15.305 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:46:16.361 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:16 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:46:16.362 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:16 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:46:16.362 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:46:16.362 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:46:16.363 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:46:16.363 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:46:16.363 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:46:16.364 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:46:16.364 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:46:16.364 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:46:16.365 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:46:16.365 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:46:16.365 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:46:16.366 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:46:16.366 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:46:16.367 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:46:16.367 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:46:16.367 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:46:16.368 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:46:16.368 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:46:16.368 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:46:16.369 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:46:16.369 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:46:16.369 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:46:16.370 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:46:16.370 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:46:16.370 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:46:16.371 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:46:16.371 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:46:16.371 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:46:16.372 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:46:16.372 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:46:16.372 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:46:16.373 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:46:16.373 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:46:16.373 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:46:16.374 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:46:16.374 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:46:16.374 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:46:16.375 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:46:16.375 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:46:16.375 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:46:16.376 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:46:16.376 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:46:16.376 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:46:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:46:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:46:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1221] Local address: 172.21.15.161, port 33640 2022-07-03T21:46:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1221] Peer address: 172.21.15.101, port 22 2022-07-03T21:46:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1221] Connection lost 2022-07-03T21:46:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1221] Aborting connection 2022-07-03T21:46:16.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:16 smithi161 ceph-mon[91579]: pgmap v20013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:16 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:46:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:16 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:46:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:46:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:46:16.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:46:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:46:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:46:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:46:16.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:46:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:46:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:46:16.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:46:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:46:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:46:16.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:46:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:46:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:46:16.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:46:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:46:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:46:16.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:46:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:46:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:46:16.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:46:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:46:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:46:16.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:46:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:46:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:46:16.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:46:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:46:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:46:16.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:46:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:46:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:46:16.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:46:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:46:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:46:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:46:16.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:46:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:46:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:46:16.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:46:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:46:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:46:16.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:46:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1221] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:46:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1221] Local address: 172.21.15.161, port 33640 2022-07-03T21:46:16.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1221] Peer address: 172.21.15.101, port 22 2022-07-03T21:46:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1221] Connection lost 2022-07-03T21:46:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1221] Aborting connection 2022-07-03T21:46:16.428 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:16 smithi188 ceph-mon[92425]: pgmap v20013: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:18 smithi161 ceph-mon[91579]: pgmap v20014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:46:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:18 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:46:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:18 smithi188 ceph-mon[92425]: pgmap v20014: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:46:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:18 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:46:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:19 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi161 2022-07-03T21:46:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:19 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi161 2022-07-03T21:46:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:20 smithi161 ceph-mon[91579]: pgmap v20015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:20 smithi188 ceph-mon[92425]: pgmap v20015: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:21 smithi161 ceph-mon[91579]: pgmap v20016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:21 smithi188 ceph-mon[92425]: pgmap v20016: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:24 smithi161 ceph-mon[91579]: pgmap v20017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:24 smithi188 ceph-mon[92425]: pgmap v20017: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:26 smithi161 ceph-mon[91579]: pgmap v20018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:26 smithi188 ceph-mon[92425]: pgmap v20018: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:28 smithi161 ceph-mon[91579]: pgmap v20019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:28 smithi188 ceph-mon[92425]: pgmap v20019: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:30 smithi161 ceph-mon[91579]: pgmap v20020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:30 smithi188 ceph-mon[92425]: pgmap v20020: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:31 smithi161 ceph-mon[91579]: pgmap v20021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:31 smithi188 ceph-mon[92425]: pgmap v20021: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:34 smithi161 ceph-mon[91579]: pgmap v20022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:34 smithi188 ceph-mon[92425]: pgmap v20022: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:36 smithi161 ceph-mon[91579]: pgmap v20023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:36 smithi188 ceph-mon[92425]: pgmap v20023: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:38.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:38 smithi161 ceph-mon[91579]: pgmap v20024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:38 smithi188 ceph-mon[92425]: pgmap v20024: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:40 smithi161 ceph-mon[91579]: pgmap v20025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:40.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:40 smithi188 ceph-mon[92425]: pgmap v20025: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:40.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:41 smithi161 ceph-mon[91579]: pgmap v20026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:41 smithi188 ceph-mon[92425]: pgmap v20026: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:44 smithi161 ceph-mon[91579]: pgmap v20027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:44 smithi188 ceph-mon[92425]: pgmap v20027: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:46 smithi161 ceph-mon[91579]: pgmap v20028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:46 smithi188 ceph-mon[92425]: pgmap v20028: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:48 smithi161 ceph-mon[91579]: pgmap v20029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:48 smithi188 ceph-mon[92425]: pgmap v20029: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:50 smithi161 ceph-mon[91579]: pgmap v20030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:50.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:50 smithi188 ceph-mon[92425]: pgmap v20030: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:50.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:51 smithi161 ceph-mon[91579]: pgmap v20031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:51 smithi188 ceph-mon[92425]: pgmap v20031: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:54 smithi161 ceph-mon[91579]: pgmap v20032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:54 smithi188 ceph-mon[92425]: pgmap v20032: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:46:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:46:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:56 smithi161 ceph-mon[91579]: pgmap v20033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:56 smithi188 ceph-mon[92425]: pgmap v20033: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:46:58 smithi161 ceph-mon[91579]: pgmap v20034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:46:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:46:58 smithi188 ceph-mon[92425]: pgmap v20034: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: pgmap v20035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:47:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:47:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:47:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:47:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: pgmap v20035: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:47:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:47:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:47:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:47:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:01 smithi161 ceph-mon[91579]: pgmap v20036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:01 smithi188 ceph-mon[92425]: pgmap v20036: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:04 smithi161 ceph-mon[91579]: pgmap v20037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:04 smithi188 ceph-mon[92425]: pgmap v20037: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:06 smithi161 ceph-mon[91579]: pgmap v20038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:06 smithi188 ceph-mon[92425]: pgmap v20038: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:08.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:08 smithi161 ceph-mon[91579]: pgmap v20039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:08 smithi188 ceph-mon[92425]: pgmap v20039: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:10 smithi161 ceph-mon[91579]: pgmap v20040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:10 smithi188 ceph-mon[92425]: pgmap v20040: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:11 smithi161 ceph-mon[91579]: pgmap v20041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:11 smithi188 ceph-mon[92425]: pgmap v20041: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:14 smithi161 ceph-mon[91579]: pgmap v20042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:14 smithi188 ceph-mon[92425]: pgmap v20042: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:15.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:16 smithi161 ceph-mon[91579]: pgmap v20043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:16 smithi188 ceph-mon[92425]: pgmap v20043: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:18 smithi161 ceph-mon[91579]: pgmap v20044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:18.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:47:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:47:18.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:18 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:47:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:47:17.972+0000 7f727286d700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:47:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:47:18.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:47:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:47:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:47:18.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:47:18.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:47:18.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:47:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:47:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:47:18.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:47:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:47:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:47:18.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:47:18.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:47:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:47:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:47:18.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:47:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:47:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:47:18.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:47:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:47:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:47:18.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:47:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:47:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:47:18.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:47:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:47:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:47:18.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:47:18.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:47:18.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:47:18.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:47:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:47:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:47:18.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:47:18.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:47:18.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:47:18.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:47:18.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:47:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:47:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:47:18.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:47:18.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:47:18.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:47:18.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:47:18.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1223] Local address: 172.21.15.161, port 33644 2022-07-03T21:47:18.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1223] Peer address: 172.21.15.101, port 22 2022-07-03T21:47:18.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1223] Connection lost 2022-07-03T21:47:18.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1223] Aborting connection 2022-07-03T21:47:18.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:47:17 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:47:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:18 smithi188 ceph-mon[92425]: pgmap v20044: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:47:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:47:18.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:18 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:47:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:19 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:47:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:19 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:47:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:47:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:47:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:47:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:47:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:47:19.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:47:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:47:19.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:47:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:47:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:47:19.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:47:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:47:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:47:19.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:47:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:47:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:47:19.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:47:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:47:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:47:19.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:47:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:47:19.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:47:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:47:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:47:19.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:47:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:47:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:47:19.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:47:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:47:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:47:19.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:47:19.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:47:19.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:47:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:47:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:47:19.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:47:19.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:47:19.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:47:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:47:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:47:19.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:47:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:47:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:47:19.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:47:19.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:47:19.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1223] Local address: 172.21.15.161, port 33644 2022-07-03T21:47:19.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1223] Peer address: 172.21.15.101, port 22 2022-07-03T21:47:19.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1223] Connection lost 2022-07-03T21:47:19.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1223] Aborting connection 2022-07-03T21:47:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:19 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:47:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:19 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:47:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:47:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:47:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:47:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:47:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:47:19.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:47:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:47:19.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:47:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:47:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:47:19.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:47:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:47:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:47:19.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:47:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:47:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:47:19.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:47:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:47:19.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:47:19.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:47:19.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:47:19.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:47:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:47:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:47:19.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:47:19.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:47:19.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:47:19.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:47:19.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:47:19.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:47:19.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:47:19.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:47:19.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:47:19.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:47:19.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:47:19.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:47:19.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:47:19.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:47:19.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:47:19.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:47:19.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:47:19.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:47:19.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:47:19.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:47:19.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1223] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:47:19.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1223] Local address: 172.21.15.161, port 33644 2022-07-03T21:47:19.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1223] Peer address: 172.21.15.101, port 22 2022-07-03T21:47:19.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1223] Connection lost 2022-07-03T21:47:19.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1223] Aborting connection 2022-07-03T21:47:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:20 smithi161 ceph-mon[91579]: pgmap v20045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:20 smithi188 ceph-mon[92425]: pgmap v20045: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:21 smithi161 ceph-mon[91579]: pgmap v20046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:21 smithi188 ceph-mon[92425]: pgmap v20046: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:24 smithi161 ceph-mon[91579]: pgmap v20047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:24 smithi188 ceph-mon[92425]: pgmap v20047: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:26 smithi161 ceph-mon[91579]: pgmap v20048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:26 smithi188 ceph-mon[92425]: pgmap v20048: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:28 smithi161 ceph-mon[91579]: pgmap v20049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:28 smithi188 ceph-mon[92425]: pgmap v20049: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:30 smithi161 ceph-mon[91579]: pgmap v20050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:30 smithi188 ceph-mon[92425]: pgmap v20050: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:31 smithi161 ceph-mon[91579]: pgmap v20051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:31 smithi188 ceph-mon[92425]: pgmap v20051: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:34 smithi161 ceph-mon[91579]: pgmap v20052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:34 smithi188 ceph-mon[92425]: pgmap v20052: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:35.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:36 smithi161 ceph-mon[91579]: pgmap v20053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:36 smithi188 ceph-mon[92425]: pgmap v20053: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:38 smithi161 ceph-mon[91579]: pgmap v20054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:38 smithi188 ceph-mon[92425]: pgmap v20054: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:40 smithi161 ceph-mon[91579]: pgmap v20055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:40 smithi188 ceph-mon[92425]: pgmap v20055: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:42.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:41 smithi161 ceph-mon[91579]: pgmap v20056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:41 smithi188 ceph-mon[92425]: pgmap v20056: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:44.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:44 smithi161 ceph-mon[91579]: pgmap v20057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:44 smithi188 ceph-mon[92425]: pgmap v20057: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:46 smithi161 ceph-mon[91579]: pgmap v20058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:46 smithi188 ceph-mon[92425]: pgmap v20058: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:48.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:48 smithi161 ceph-mon[91579]: pgmap v20059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:48 smithi188 ceph-mon[92425]: pgmap v20059: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:50 smithi161 ceph-mon[91579]: pgmap v20060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:50 smithi188 ceph-mon[92425]: pgmap v20060: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:51 smithi161 ceph-mon[91579]: pgmap v20061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:51 smithi188 ceph-mon[92425]: pgmap v20061: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:54 smithi161 ceph-mon[91579]: pgmap v20062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:54 smithi188 ceph-mon[92425]: pgmap v20062: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:47:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:55.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:47:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:56 smithi161 ceph-mon[91579]: pgmap v20063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:56 smithi188 ceph-mon[92425]: pgmap v20063: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:47:58 smithi161 ceph-mon[91579]: pgmap v20064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:47:58.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:47:58 smithi188 ceph-mon[92425]: pgmap v20064: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: pgmap v20065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:48:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:48:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:48:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:48:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: pgmap v20065: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:48:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:48:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:48:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:48:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:01 smithi161 ceph-mon[91579]: pgmap v20066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:01 smithi188 ceph-mon[92425]: pgmap v20066: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:04 smithi161 ceph-mon[91579]: pgmap v20067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:04 smithi188 ceph-mon[92425]: pgmap v20067: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:06 smithi161 ceph-mon[91579]: pgmap v20068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:06 smithi188 ceph-mon[92425]: pgmap v20068: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:08 smithi161 ceph-mon[91579]: pgmap v20069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:08 smithi188 ceph-mon[92425]: pgmap v20069: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:10 smithi161 ceph-mon[91579]: pgmap v20070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:10 smithi188 ceph-mon[92425]: pgmap v20070: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:11 smithi161 ceph-mon[91579]: pgmap v20071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:11 smithi188 ceph-mon[92425]: pgmap v20071: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:14 smithi161 ceph-mon[91579]: pgmap v20072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:14 smithi188 ceph-mon[92425]: pgmap v20072: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:16.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:16 smithi161 ceph-mon[91579]: pgmap v20073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:16 smithi188 ceph-mon[92425]: pgmap v20073: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-mon[91579]: pgmap v20074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:18 smithi188 ceph-mon[92425]: pgmap v20074: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:18.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:48:18.486+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:48:18.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:48:18.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:48:18.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:48:18.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:48:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:48:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:48:18.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:48:18.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:48:18.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:48:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:48:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:48:18.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:48:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:48:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:48:18.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:48:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:48:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:48:18.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:48:18.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:48:18.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:48:18.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:48:18.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:48:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:48:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:48:18.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:48:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:48:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:48:18.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:48:18.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:48:18.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:48:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:48:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:48:18.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:48:18.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:48:18.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:48:18.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:48:18.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:48:18.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:48:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:48:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:48:18.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:48:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:48:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:48:18.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:48:18.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:48:18.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1225] Local address: 172.21.15.161, port 33648 2022-07-03T21:48:18.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1225] Peer address: 172.21.15.101, port 22 2022-07-03T21:48:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1225] Connection lost 2022-07-03T21:48:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1225] Aborting connection 2022-07-03T21:48:18.897 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:48:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:48:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:48:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:48:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:48:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:48:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:48:19.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:48:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:48:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:48:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:48:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:48:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:48:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:48:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:48:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:48:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:48:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:48:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:48:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:48:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:48:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:48:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:48:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:48:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:48:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:48:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:48:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:48:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:48:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:48:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:48:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:48:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:48:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:48:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:48:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:48:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:48:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:48:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:48:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:48:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:48:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:48:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:48:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:48:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:48:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:48:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:48:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:48:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:48:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:48:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:48:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:48:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:48:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:48:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:48:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1225] Local address: 172.21.15.161, port 33648 2022-07-03T21:48:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1225] Peer address: 172.21.15.101, port 22 2022-07-03T21:48:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1225] Connection lost 2022-07-03T21:48:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1225] Aborting connection 2022-07-03T21:48:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:20 smithi161 ceph-mon[91579]: pgmap v20075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:48:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:48:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:48:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:48:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:48:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:48:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:48:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:48:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:48:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:48:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:48:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:48:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:48:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:48:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:48:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:48:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:48:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:48:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:48:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:48:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:48:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:48:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:48:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:48:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:48:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:48:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:48:20.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:48:20.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:48:20.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:48:20.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:48:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:48:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:48:20.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:48:20.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:48:20.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:48:20.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:48:20.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:48:20.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:48:20.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:48:20.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:48:20.431 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:48:20.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:48:20.432 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:48:20.433 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:48:20.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:48:20.434 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1225] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:48:20.435 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1225] Local address: 172.21.15.161, port 33648 2022-07-03T21:48:20.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1225] Peer address: 172.21.15.101, port 22 2022-07-03T21:48:20.436 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1225] Connection lost 2022-07-03T21:48:20.437 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1225] Aborting connection 2022-07-03T21:48:20.437 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:20 smithi188 ceph-mon[92425]: pgmap v20075: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:20.438 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:21 smithi161 ceph-mon[91579]: pgmap v20076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:21 smithi188 ceph-mon[92425]: pgmap v20076: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:24 smithi161 ceph-mon[91579]: pgmap v20077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:24 smithi188 ceph-mon[92425]: pgmap v20077: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:26.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:26 smithi161 ceph-mon[91579]: pgmap v20078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:26 smithi188 ceph-mon[92425]: pgmap v20078: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:28 smithi161 ceph-mon[91579]: pgmap v20079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:28 smithi188 ceph-mon[92425]: pgmap v20079: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:30 smithi161 ceph-mon[91579]: pgmap v20080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:30 smithi188 ceph-mon[92425]: pgmap v20080: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:31 smithi161 ceph-mon[91579]: pgmap v20081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:31 smithi188 ceph-mon[92425]: pgmap v20081: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:34 smithi161 ceph-mon[91579]: pgmap v20082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:34 smithi188 ceph-mon[92425]: pgmap v20082: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:36 smithi161 ceph-mon[91579]: pgmap v20083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:36 smithi188 ceph-mon[92425]: pgmap v20083: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:38 smithi161 ceph-mon[91579]: pgmap v20084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:38 smithi188 ceph-mon[92425]: pgmap v20084: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:40 smithi161 ceph-mon[91579]: pgmap v20085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:40 smithi188 ceph-mon[92425]: pgmap v20085: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:41 smithi161 ceph-mon[91579]: pgmap v20086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:41 smithi188 ceph-mon[92425]: pgmap v20086: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:44 smithi161 ceph-mon[91579]: pgmap v20087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:44 smithi188 ceph-mon[92425]: pgmap v20087: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:45.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:46 smithi161 ceph-mon[91579]: pgmap v20088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:46 smithi188 ceph-mon[92425]: pgmap v20088: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:48 smithi161 ceph-mon[91579]: pgmap v20089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:48.418 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:48 smithi188 ceph-mon[92425]: pgmap v20089: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:50 smithi161 ceph-mon[91579]: pgmap v20090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:50 smithi188 ceph-mon[92425]: pgmap v20090: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:51 smithi161 ceph-mon[91579]: pgmap v20091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:51 smithi188 ceph-mon[92425]: pgmap v20091: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:54 smithi161 ceph-mon[91579]: pgmap v20092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:54 smithi188 ceph-mon[92425]: pgmap v20092: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:48:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:48:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:56 smithi161 ceph-mon[91579]: pgmap v20093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:56 smithi188 ceph-mon[92425]: pgmap v20093: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:48:58 smithi161 ceph-mon[91579]: pgmap v20094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:48:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:48:58 smithi188 ceph-mon[92425]: pgmap v20094: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:00.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: pgmap v20095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:49:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:49:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:49:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:49:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:00.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: pgmap v20095: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:49:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:49:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:49:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:49:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:01 smithi161 ceph-mon[91579]: pgmap v20096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:01 smithi188 ceph-mon[92425]: pgmap v20096: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:04 smithi161 ceph-mon[91579]: pgmap v20097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:04 smithi188 ceph-mon[92425]: pgmap v20097: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:06 smithi161 ceph-mon[91579]: pgmap v20098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:06 smithi188 ceph-mon[92425]: pgmap v20098: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:08 smithi161 ceph-mon[91579]: pgmap v20099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:08 smithi188 ceph-mon[92425]: pgmap v20099: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:10 smithi161 ceph-mon[91579]: pgmap v20100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:10 smithi188 ceph-mon[92425]: pgmap v20100: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:11 smithi161 ceph-mon[91579]: pgmap v20101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:11 smithi188 ceph-mon[92425]: pgmap v20101: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:14 smithi161 ceph-mon[91579]: pgmap v20102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:14 smithi188 ceph-mon[92425]: pgmap v20102: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:16 smithi161 ceph-mon[91579]: pgmap v20103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:16 smithi188 ceph-mon[92425]: pgmap v20103: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-mon[91579]: pgmap v20104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:18 smithi188 ceph-mon[92425]: pgmap v20104: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:49:19.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:49:19.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:19 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:49:19.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:49:18.986+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:49:19.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:49:19.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:49:19.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:49:19.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:49:19.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:49:19.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:49:19.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:49:19.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:49:19.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:49:19.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:49:19.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:49:19.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:49:19.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:49:19.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:49:19.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:49:19.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:49:19.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:49:19.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:49:19.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:49:19.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:49:19.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:49:19.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:49:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:49:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:49:19.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:49:19.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:49:19.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:49:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:49:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:49:19.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:49:19.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:49:19.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:49:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:49:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:49:19.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:49:19.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:49:19.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:49:19.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:49:19.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:49:19.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:49:19.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:49:19.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:49:19.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:49:19.397 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:49:19.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:49:19.398 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1227] Local address: 172.21.15.161, port 33652 2022-07-03T21:49:19.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1227] Peer address: 172.21.15.101, port 22 2022-07-03T21:49:19.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1227] Connection lost 2022-07-03T21:49:19.399 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1227] Aborting connection 2022-07-03T21:49:19.400 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:49:18 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:49:19.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:49:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:49:19.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:19 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:49:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:20 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:49:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:20 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:49:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:49:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:49:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:49:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:49:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:49:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:49:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:49:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:49:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:49:20.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:49:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:49:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:49:20.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:49:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:49:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:49:20.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:49:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:49:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:49:20.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:49:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:49:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:49:20.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:49:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:49:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:49:20.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:49:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:49:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:49:20.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:49:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:49:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:49:20.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:49:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:49:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:49:20.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:49:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:49:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:49:20.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:49:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:49:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:49:20.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:49:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:49:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:49:20.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:49:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:49:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:49:20.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1227] Local address: 172.21.15.161, port 33652 2022-07-03T21:49:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1227] Peer address: 172.21.15.101, port 22 2022-07-03T21:49:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1227] Connection lost 2022-07-03T21:49:20.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1227] Aborting connection 2022-07-03T21:49:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:20 smithi161 ceph-mon[91579]: pgmap v20105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:20.396 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:20 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:49:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:20 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:49:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:49:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:49:20.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:49:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:49:20.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:49:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:49:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:49:20.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:49:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:49:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:49:20.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:49:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:49:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:49:20.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:49:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:49:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:49:20.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:49:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:49:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:49:20.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:49:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:49:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:49:20.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:49:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:49:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:49:20.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:49:20.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:49:20.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:49:20.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:49:20.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:49:20.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:49:20.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:49:20.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:49:20.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:49:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:49:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:49:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:49:20.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:49:20.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:49:20.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:49:20.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:49:20.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:49:20.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:49:20.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:49:20.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1227] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:49:20.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1227] Local address: 172.21.15.161, port 33652 2022-07-03T21:49:20.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1227] Peer address: 172.21.15.101, port 22 2022-07-03T21:49:20.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1227] Connection lost 2022-07-03T21:49:20.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1227] Aborting connection 2022-07-03T21:49:20.429 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:20 smithi188 ceph-mon[92425]: pgmap v20105: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:20.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:21 smithi161 ceph-mon[91579]: pgmap v20106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:21 smithi188 ceph-mon[92425]: pgmap v20106: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:24 smithi161 ceph-mon[91579]: pgmap v20107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:24 smithi188 ceph-mon[92425]: pgmap v20107: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:25.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:25.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:26.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:26 smithi161 ceph-mon[91579]: pgmap v20108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:26.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:26 smithi188 ceph-mon[92425]: pgmap v20108: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:28 smithi161 ceph-mon[91579]: pgmap v20109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:28 smithi188 ceph-mon[92425]: pgmap v20109: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:30.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:30 smithi161 ceph-mon[91579]: pgmap v20110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:30.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:30 smithi188 ceph-mon[92425]: pgmap v20110: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:31 smithi161 ceph-mon[91579]: pgmap v20111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:31 smithi188 ceph-mon[92425]: pgmap v20111: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:34.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:34 smithi161 ceph-mon[91579]: pgmap v20112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:34 smithi188 ceph-mon[92425]: pgmap v20112: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:35.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:36 smithi161 ceph-mon[91579]: pgmap v20113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:36.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:36 smithi188 ceph-mon[92425]: pgmap v20113: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:38 smithi161 ceph-mon[91579]: pgmap v20114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:38 smithi188 ceph-mon[92425]: pgmap v20114: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:40 smithi161 ceph-mon[91579]: pgmap v20115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:40 smithi188 ceph-mon[92425]: pgmap v20115: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:41 smithi161 ceph-mon[91579]: pgmap v20116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:41 smithi188 ceph-mon[92425]: pgmap v20116: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:44 smithi161 ceph-mon[91579]: pgmap v20117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:44 smithi188 ceph-mon[92425]: pgmap v20117: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:46.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:46 smithi161 ceph-mon[91579]: pgmap v20118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:46 smithi188 ceph-mon[92425]: pgmap v20118: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:48 smithi161 ceph-mon[91579]: pgmap v20119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:48 smithi188 ceph-mon[92425]: pgmap v20119: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:50 smithi161 ceph-mon[91579]: pgmap v20120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:50 smithi188 ceph-mon[92425]: pgmap v20120: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:51 smithi161 ceph-mon[91579]: pgmap v20121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:51 smithi188 ceph-mon[92425]: pgmap v20121: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:54 smithi161 ceph-mon[91579]: pgmap v20122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:54.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:54 smithi188 ceph-mon[92425]: pgmap v20122: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:49:55.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:49:56.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:56 smithi161 ceph-mon[91579]: pgmap v20123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:56 smithi188 ceph-mon[92425]: pgmap v20123: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:58.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:49:58 smithi161 ceph-mon[91579]: pgmap v20124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:49:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:49:58 smithi188 ceph-mon[92425]: pgmap v20124: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: pgmap v20125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:00.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:50:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:50:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:50:00.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:50:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:00.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:00 smithi161 ceph-mon[91579]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:50:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: pgmap v20125: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:50:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:50:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:50:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:50:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:00 smithi188 ceph-mon[92425]: overall HEALTH_WARN 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive 2022-07-03T21:50:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:01 smithi161 ceph-mon[91579]: pgmap v20126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:01 smithi188 ceph-mon[92425]: pgmap v20126: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:04 smithi161 ceph-mon[91579]: pgmap v20127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:04 smithi188 ceph-mon[92425]: pgmap v20127: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:06.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:06 smithi161 ceph-mon[91579]: pgmap v20128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:06 smithi188 ceph-mon[92425]: pgmap v20128: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:08.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:08 smithi161 ceph-mon[91579]: pgmap v20129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:08 smithi188 ceph-mon[92425]: pgmap v20129: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:10.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:10 smithi161 ceph-mon[91579]: pgmap v20130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:10.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:10 smithi188 ceph-mon[92425]: pgmap v20130: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:11 smithi161 ceph-mon[91579]: pgmap v20131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:11 smithi188 ceph-mon[92425]: pgmap v20131: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:14.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:14 smithi161 ceph-mon[91579]: pgmap v20132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:14 smithi188 ceph-mon[92425]: pgmap v20132: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:15.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:16.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:16 smithi161 ceph-mon[91579]: pgmap v20133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:16 smithi188 ceph-mon[92425]: pgmap v20133: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:18.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:18 smithi161 ceph-mon[91579]: pgmap v20134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:18.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:18 smithi188 ceph-mon[92425]: pgmap v20134: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:19.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:50:19.488+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:50:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:50:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:50:19.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:50:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:50:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:50:19.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:50:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:50:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:50:19.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:50:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:50:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:50:19.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:50:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:50:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:50:19.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:50:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:50:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:50:19.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:50:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:50:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:50:19.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:50:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:50:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:50:19.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:50:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:50:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:50:19.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:50:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:50:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:50:19.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:50:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:50:19.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:50:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:50:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:50:19.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:50:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:50:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:50:19.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:50:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:50:19.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:50:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:50:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:50:19.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:50:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:50:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:50:19.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1229] Local address: 172.21.15.161, port 33656 2022-07-03T21:50:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1229] Peer address: 172.21.15.101, port 22 2022-07-03T21:50:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1229] Connection lost 2022-07-03T21:50:19.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1229] Aborting connection 2022-07-03T21:50:19.896 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:50:19 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:50:20.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: pgmap v20135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:50:20.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:50:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:50:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:20.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.380 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.381 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:20 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.403 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: pgmap v20135: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:20.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:50:20.404 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:50:20.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:50:20.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:20.405 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:20.406 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:20 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:21 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:50:21.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:21 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:50:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:50:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:50:21.379 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:50:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:50:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:50:21.380 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:50:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:50:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:50:21.381 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:50:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:50:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:50:21.382 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:50:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:50:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:50:21.383 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:50:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:50:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:50:21.384 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:50:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:50:21.385 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:50:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:50:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:50:21.386 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:50:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:50:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:50:21.387 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:50:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:50:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:50:21.388 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:50:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:50:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:50:21.389 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:50:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:50:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:50:21.390 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:50:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:50:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:50:21.391 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:50:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:50:21.392 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:50:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:50:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:50:21.393 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:50:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:50:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:50:21.394 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1229] Local address: 172.21.15.161, port 33656 2022-07-03T21:50:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1229] Peer address: 172.21.15.101, port 22 2022-07-03T21:50:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1229] Connection lost 2022-07-03T21:50:21.395 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1229] Aborting connection 2022-07-03T21:50:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:21 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:50:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:21 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:50:21.412 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:50:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:50:21.413 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:50:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:50:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:50:21.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:50:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:50:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:50:21.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:50:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:50:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:50:21.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:50:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:50:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:50:21.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:50:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:50:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:50:21.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:50:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:50:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:50:21.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:50:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:50:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:50:21.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:50:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:50:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:50:21.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:50:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:50:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:50:21.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:50:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:50:21.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:50:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:50:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:50:21.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:50:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:50:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:50:21.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:50:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:50:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:50:21.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:50:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:50:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:50:21.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:50:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1229] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:50:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1229] Local address: 172.21.15.161, port 33656 2022-07-03T21:50:21.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1229] Peer address: 172.21.15.101, port 22 2022-07-03T21:50:21.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1229] Connection lost 2022-07-03T21:50:21.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1229] Aborting connection 2022-07-03T21:50:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:22 smithi161 ceph-mon[91579]: pgmap v20136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:22 smithi188 ceph-mon[92425]: pgmap v20136: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:24 smithi188 ceph-mon[92425]: pgmap v20137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:24 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:24 smithi161 ceph-mon[91579]: pgmap v20137: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:24.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:24 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:50:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:25 smithi188 ceph-mon[92425]: Detected new or changed devices on smithi188 2022-07-03T21:50:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:25 smithi161 ceph-mon[91579]: Detected new or changed devices on smithi188 2022-07-03T21:50:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:26 smithi188 ceph-mon[92425]: pgmap v20138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:26 smithi161 ceph-mon[91579]: pgmap v20138: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:28 smithi188 ceph-mon[92425]: pgmap v20139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:28 smithi161 ceph-mon[91579]: pgmap v20139: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:30 smithi188 ceph-mon[92425]: pgmap v20140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:30 smithi161 ceph-mon[91579]: pgmap v20140: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:31 smithi161 ceph-mon[91579]: pgmap v20141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:31 smithi188 ceph-mon[92425]: pgmap v20141: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:34.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:34 smithi161 ceph-mon[91579]: pgmap v20142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:34.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:34 smithi188 ceph-mon[92425]: pgmap v20142: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:35.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:36.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:36 smithi161 ceph-mon[91579]: pgmap v20143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:36 smithi188 ceph-mon[92425]: pgmap v20143: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:38.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:38 smithi161 ceph-mon[91579]: pgmap v20144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:38 smithi188 ceph-mon[92425]: pgmap v20144: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:40.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:40 smithi161 ceph-mon[91579]: pgmap v20145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:40.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:40 smithi188 ceph-mon[92425]: pgmap v20145: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:41 smithi161 ceph-mon[91579]: pgmap v20146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:41 smithi188 ceph-mon[92425]: pgmap v20146: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:44.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:44 smithi161 ceph-mon[91579]: pgmap v20147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:44 smithi188 ceph-mon[92425]: pgmap v20147: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:45.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:46.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:46 smithi161 ceph-mon[91579]: pgmap v20148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:46.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:46 smithi188 ceph-mon[92425]: pgmap v20148: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:48.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:48 smithi161 ceph-mon[91579]: pgmap v20149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:48.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:48 smithi188 ceph-mon[92425]: pgmap v20149: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:50.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:50 smithi161 ceph-mon[91579]: pgmap v20150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:50.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:50.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:50 smithi188 ceph-mon[92425]: pgmap v20150: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:51 smithi161 ceph-mon[91579]: pgmap v20151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:51 smithi188 ceph-mon[92425]: pgmap v20151: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:54.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:54 smithi161 ceph-mon[91579]: pgmap v20152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:54 smithi188 ceph-mon[92425]: pgmap v20152: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:50:55.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:50:56.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:56 smithi161 ceph-mon[91579]: pgmap v20153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:56 smithi188 ceph-mon[92425]: pgmap v20153: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:58.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:50:58 smithi161 ceph-mon[91579]: pgmap v20154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:50:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:50:58 smithi188 ceph-mon[92425]: pgmap v20154: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: pgmap v20155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:51:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:51:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:51:00.415 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:51:00.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: pgmap v20155: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:51:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:51:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:51:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:51:00.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:01 smithi161 ceph-mon[91579]: pgmap v20156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:01 smithi188 ceph-mon[92425]: pgmap v20156: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:04.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:04 smithi161 ceph-mon[91579]: pgmap v20157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:04 smithi188 ceph-mon[92425]: pgmap v20157: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:05.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:05.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:06 smithi188 ceph-mon[92425]: pgmap v20158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:06 smithi161 ceph-mon[91579]: pgmap v20158: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:08.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:08 smithi188 ceph-mon[92425]: pgmap v20159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:08 smithi161 ceph-mon[91579]: pgmap v20159: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:10 smithi188 ceph-mon[92425]: pgmap v20160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:10 smithi161 ceph-mon[91579]: pgmap v20160: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:11 smithi161 ceph-mon[91579]: pgmap v20161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:11 smithi188 ceph-mon[92425]: pgmap v20161: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:14.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:14 smithi161 ceph-mon[91579]: pgmap v20162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:14 smithi188 ceph-mon[92425]: pgmap v20162: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:15.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:16 smithi188 ceph-mon[92425]: pgmap v20163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:16 smithi161 ceph-mon[91579]: pgmap v20163: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:18 smithi188 ceph-mon[92425]: pgmap v20164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:18 smithi161 ceph-mon[91579]: pgmap v20164: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:20 smithi188 ceph-mon[92425]: pgmap v20165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:20 smithi161 ceph-mon[91579]: pgmap v20165: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:21 smithi161 ceph-mon[91579]: pgmap v20166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:21 smithi188 ceph-mon[92425]: pgmap v20166: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:23.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:51:23.302+0000 7f7274871700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:51:23.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:51:23.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:51:23.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:51:23.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:51:23.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:51:23.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:51:23.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:51:23.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:51:23.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:51:23.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:51:23.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:51:23.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:51:23.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:51:23.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:51:23.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:51:23.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:51:23.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:51:23.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:51:23.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:51:23.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:51:23.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:51:23.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:51:23.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:51:23.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:51:23.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:51:23.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:51:23.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:51:23.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:51:23.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:51:23.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:51:23.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:51:23.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:51:23.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:51:23.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:51:23.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:51:23.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:51:23.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:51:23.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:51:23.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:51:23.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:51:23.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:51:23.648 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:51:23.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:51:23.649 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:51:23.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:51:23.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1231] Local address: 172.21.15.161, port 33660 2022-07-03T21:51:23.650 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1231] Peer address: 172.21.15.101, port 22 2022-07-03T21:51:23.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1231] Connection lost 2022-07-03T21:51:23.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1231] Aborting connection 2022-07-03T21:51:23.651 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:51:23 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:51:24.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:24 smithi161 ceph-mon[91579]: pgmap v20167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:51:24.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:51:24.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:24 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:51:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:24 smithi188 ceph-mon[92425]: pgmap v20167: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:51:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:51:24.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:24 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:51:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:51:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:51:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:51:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:51:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:51:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:51:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:51:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:51:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:51:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:51:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:51:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:51:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:51:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:51:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:51:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:51:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:51:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:51:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:51:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:51:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:51:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:51:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:51:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:51:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:51:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:51:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:51:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:51:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:51:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:51:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:51:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:51:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:51:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:51:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:51:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:51:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:51:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:51:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:51:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:51:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:51:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:51:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:51:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:51:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:51:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1231] Local address: 172.21.15.161, port 33660 2022-07-03T21:51:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1231] Peer address: 172.21.15.101, port 22 2022-07-03T21:51:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1231] Connection lost 2022-07-03T21:51:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1231] Aborting connection 2022-07-03T21:51:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.647 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:25.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:51:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:51:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:51:25.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:51:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:51:25.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:51:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:51:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:51:25.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:51:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:51:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:51:25.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:51:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:51:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:51:25.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:51:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:51:25.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:51:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:51:25.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:51:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:51:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:51:25.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:51:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:51:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:51:25.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:51:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:51:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:51:25.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:51:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:51:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:51:25.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:51:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:51:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:51:25.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:51:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:51:25.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:51:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:51:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:51:25.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:51:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:51:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:51:25.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:51:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:51:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:51:25.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:51:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:51:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1231] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:51:25.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1231] Local address: 172.21.15.161, port 33660 2022-07-03T21:51:25.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1231] Peer address: 172.21.15.101, port 22 2022-07-03T21:51:25.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1231] Connection lost 2022-07-03T21:51:25.682 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1231] Aborting connection 2022-07-03T21:51:25.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.683 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:51:25.684 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:26 smithi161 ceph-mon[91579]: pgmap v20168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:26 smithi188 ceph-mon[92425]: pgmap v20168: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:28 smithi161 ceph-mon[91579]: pgmap v20169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:28 smithi188 ceph-mon[92425]: pgmap v20169: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:30 smithi161 ceph-mon[91579]: pgmap v20170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:30 smithi188 ceph-mon[92425]: pgmap v20170: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:31 smithi161 ceph-mon[91579]: pgmap v20171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:31 smithi188 ceph-mon[92425]: pgmap v20171: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:34 smithi188 ceph-mon[92425]: pgmap v20172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:34 smithi161 ceph-mon[91579]: pgmap v20172: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:36 smithi188 ceph-mon[92425]: pgmap v20173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:36 smithi161 ceph-mon[91579]: pgmap v20173: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:38.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:38 smithi188 ceph-mon[92425]: pgmap v20174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:38 smithi161 ceph-mon[91579]: pgmap v20174: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:40 smithi188 ceph-mon[92425]: pgmap v20175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:40.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:40 smithi161 ceph-mon[91579]: pgmap v20175: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:41 smithi161 ceph-mon[91579]: pgmap v20176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:41 smithi188 ceph-mon[92425]: pgmap v20176: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:44.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:44 smithi188 ceph-mon[92425]: pgmap v20177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:44 smithi161 ceph-mon[91579]: pgmap v20177: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:45.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:46.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:46 smithi188 ceph-mon[92425]: pgmap v20178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:46 smithi161 ceph-mon[91579]: pgmap v20178: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:48 smithi188 ceph-mon[92425]: pgmap v20179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:48 smithi161 ceph-mon[91579]: pgmap v20179: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:50 smithi188 ceph-mon[92425]: pgmap v20180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:50.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:50 smithi161 ceph-mon[91579]: pgmap v20180: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:51 smithi161 ceph-mon[91579]: pgmap v20181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:52.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:51 smithi188 ceph-mon[92425]: pgmap v20181: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:54 smithi188 ceph-mon[92425]: pgmap v20182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:54 smithi161 ceph-mon[91579]: pgmap v20182: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:51:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:51:56.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:56 smithi188 ceph-mon[92425]: pgmap v20183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:56 smithi161 ceph-mon[91579]: pgmap v20183: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:58.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:51:58 smithi188 ceph-mon[92425]: pgmap v20184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:51:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:51:58 smithi161 ceph-mon[91579]: pgmap v20184: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:00.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: pgmap v20185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:52:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:52:00.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:52:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:52:00.414 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: pgmap v20185: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:52:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:52:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:52:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:52:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:02.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:01 smithi161 ceph-mon[91579]: pgmap v20186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:01 smithi188 ceph-mon[92425]: pgmap v20186: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:04.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:04 smithi188 ceph-mon[92425]: pgmap v20187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:04 smithi161 ceph-mon[91579]: pgmap v20187: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:05.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:06.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:06 smithi188 ceph-mon[92425]: pgmap v20188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:06 smithi161 ceph-mon[91579]: pgmap v20188: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:08.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:08 smithi188 ceph-mon[92425]: pgmap v20189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:08 smithi161 ceph-mon[91579]: pgmap v20189: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2022-07-03T21:52:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:10 smithi188 ceph-mon[92425]: pgmap v20190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:10.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:10 smithi161 ceph-mon[91579]: pgmap v20190: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:11 smithi161 ceph-mon[91579]: pgmap v20191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:11 smithi188 ceph-mon[92425]: pgmap v20191: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:14 smithi188 ceph-mon[92425]: pgmap v20192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:14 smithi161 ceph-mon[91579]: pgmap v20192: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:15.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:16.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:16 smithi188 ceph-mon[92425]: pgmap v20193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:16 smithi161 ceph-mon[91579]: pgmap v20193: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:18.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:18 smithi188 ceph-mon[92425]: pgmap v20194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:18 smithi161 ceph-mon[91579]: pgmap v20194: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:20 smithi188 ceph-mon[92425]: pgmap v20195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:20.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:20 smithi161 ceph-mon[91579]: pgmap v20195: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:22.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:21 smithi161 ceph-mon[91579]: pgmap v20196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:21 smithi188 ceph-mon[92425]: pgmap v20196: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:24.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:24 smithi188 ceph-mon[92425]: pgmap v20197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-mon[91579]: pgmap v20197: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:24.628 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:52:24.214+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:52:24.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:52:24.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:52:24.629 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:52:24.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:52:24.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:52:24.630 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:52:24.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:52:24.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:52:24.631 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:52:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:52:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:52:24.632 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:52:24.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:52:24.633 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:52:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:52:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:52:24.634 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:52:24.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:52:24.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:52:24.635 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:52:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:52:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:52:24.636 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:52:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:52:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:52:24.637 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:52:24.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:52:24.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:52:24.638 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:52:24.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:52:24.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:52:24.639 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:52:24.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:52:24.640 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:52:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:52:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:52:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:52:24.641 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:52:24.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:52:24.642 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:52:24.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:52:24.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:52:24.643 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:52:24.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:52:24.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:52:24.644 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1233] Local address: 172.21.15.161, port 33664 2022-07-03T21:52:24.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1233] Peer address: 172.21.15.101, port 22 2022-07-03T21:52:24.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1233] Connection lost 2022-07-03T21:52:24.645 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1233] Aborting connection 2022-07-03T21:52:24.646 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:52:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:52:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:52:25.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:52:25.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:52:25.414 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:52:25.415 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:52:25.416 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:52:25.417 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:52:25.418 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:52:25.419 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:52:25.420 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:52:25.421 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:52:25.422 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:52:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:52:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:52:25.423 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:52:25.424 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:52:25.425 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:52:25.426 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:52:25.427 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:52:25.428 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1233] Local address: 172.21.15.161, port 33664 2022-07-03T21:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1233] Peer address: 172.21.15.101, port 22 2022-07-03T21:52:25.429 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1233] Connection lost 2022-07-03T21:52:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1233] Aborting connection 2022-07-03T21:52:25.430 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:52:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:52:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:52:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:52:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:52:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:52:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:52:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:52:25.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:52:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:52:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:52:25.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:52:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:52:25.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:52:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:52:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:52:25.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:52:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:52:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:52:25.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:52:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:52:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:52:25.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:52:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:52:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:52:25.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:52:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:52:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:52:25.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:52:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:52:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:52:25.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:52:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:52:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:52:25.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:52:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:52:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:52:25.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:52:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:52:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:52:25.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:52:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:52:25.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:52:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:52:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:52:25.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:52:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:52:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:52:25.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:52:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1233] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:52:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1233] Local address: 172.21.15.161, port 33664 2022-07-03T21:52:25.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1233] Peer address: 172.21.15.101, port 22 2022-07-03T21:52:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1233] Connection lost 2022-07-03T21:52:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1233] Aborting connection 2022-07-03T21:52:25.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:26.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:26 smithi188 ceph-mon[92425]: pgmap v20198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:26 smithi161 ceph-mon[91579]: pgmap v20198: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:28 smithi188 ceph-mon[92425]: pgmap v20199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:28 smithi161 ceph-mon[91579]: pgmap v20199: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:30 smithi188 ceph-mon[92425]: pgmap v20200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:30.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:30 smithi161 ceph-mon[91579]: pgmap v20200: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:32.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:31 smithi161 ceph-mon[91579]: pgmap v20201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:31 smithi188 ceph-mon[92425]: pgmap v20201: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:34.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:34 smithi188 ceph-mon[92425]: pgmap v20202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:34 smithi161 ceph-mon[91579]: pgmap v20202: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:35.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:35.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:36.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:36 smithi188 ceph-mon[92425]: pgmap v20203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:36 smithi161 ceph-mon[91579]: pgmap v20203: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:38.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:38 smithi188 ceph-mon[92425]: pgmap v20204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:38 smithi161 ceph-mon[91579]: pgmap v20204: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:40 smithi188 ceph-mon[92425]: pgmap v20205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:40.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:40 smithi161 ceph-mon[91579]: pgmap v20205: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:41 smithi161 ceph-mon[91579]: pgmap v20206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:41 smithi188 ceph-mon[92425]: pgmap v20206: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:44.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:44 smithi188 ceph-mon[92425]: pgmap v20207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:44 smithi161 ceph-mon[91579]: pgmap v20207: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:45.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:46.416 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:46 smithi188 ceph-mon[92425]: pgmap v20208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:46 smithi161 ceph-mon[91579]: pgmap v20208: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:48.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:48 smithi188 ceph-mon[92425]: pgmap v20209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:48 smithi161 ceph-mon[91579]: pgmap v20209: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:50 smithi161 ceph-mon[91579]: pgmap v20210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:50 smithi188 ceph-mon[92425]: pgmap v20210: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:52.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:51 smithi161 ceph-mon[91579]: pgmap v20211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:51 smithi188 ceph-mon[92425]: pgmap v20211: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:54.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:54 smithi188 ceph-mon[92425]: pgmap v20212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:54 smithi161 ceph-mon[91579]: pgmap v20212: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:52:55.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:52:56.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:56 smithi188 ceph-mon[92425]: pgmap v20213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:56 smithi161 ceph-mon[91579]: pgmap v20213: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:52:58 smithi161 ceph-mon[91579]: pgmap v20214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:52:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:52:58 smithi188 ceph-mon[92425]: pgmap v20214: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: pgmap v20215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:53:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:53:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:53:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:53:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:00.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: pgmap v20215: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:53:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:53:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:53:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:01 smithi161 ceph-mon[91579]: pgmap v20216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:01 smithi188 ceph-mon[92425]: pgmap v20216: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:04.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:04 smithi188 ceph-mon[92425]: pgmap v20217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:04 smithi161 ceph-mon[91579]: pgmap v20217: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:06 smithi161 ceph-mon[91579]: pgmap v20218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:06 smithi188 ceph-mon[92425]: pgmap v20218: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:08 smithi161 ceph-mon[91579]: pgmap v20219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:08 smithi188 ceph-mon[92425]: pgmap v20219: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:10 smithi161 ceph-mon[91579]: pgmap v20220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:10 smithi188 ceph-mon[92425]: pgmap v20220: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:12.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:11 smithi161 ceph-mon[91579]: pgmap v20221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:11 smithi188 ceph-mon[92425]: pgmap v20221: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:14.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:14 smithi188 ceph-mon[92425]: pgmap v20222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:14 smithi161 ceph-mon[91579]: pgmap v20222: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:16 smithi161 ceph-mon[91579]: pgmap v20223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:16 smithi188 ceph-mon[92425]: pgmap v20223: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:18 smithi161 ceph-mon[91579]: pgmap v20224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:18 smithi188 ceph-mon[92425]: pgmap v20224: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:20 smithi161 ceph-mon[91579]: pgmap v20225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:20 smithi188 ceph-mon[92425]: pgmap v20225: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:22.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:21 smithi161 ceph-mon[91579]: pgmap v20226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:21 smithi188 ceph-mon[92425]: pgmap v20226: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-mon[91579]: pgmap v20227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:24 smithi188 ceph-mon[92425]: pgmap v20227: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:25.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:53:24.719+0000 7f727106a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:53:25.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:53:25.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:53:25.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:53:25.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:53:25.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:53:25.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:53:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:53:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:53:25.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:53:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:53:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:53:25.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:53:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:53:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:53:25.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:53:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:53:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:53:25.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:53:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:53:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:53:25.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:53:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:53:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:53:25.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:53:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:53:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:53:25.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:53:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:53:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:53:25.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:53:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:53:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:53:25.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:53:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:53:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:53:25.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:53:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:53:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:53:25.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:53:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:53:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:53:25.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:53:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:53:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:53:25.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:53:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1235] Local address: 172.21.15.161, port 33668 2022-07-03T21:53:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1235] Peer address: 172.21.15.101, port 22 2022-07-03T21:53:25.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1235] Connection lost 2022-07-03T21:53:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1235] Aborting connection 2022-07-03T21:53:25.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:53:24 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:53:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:53:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:53:25.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:53:25.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:53:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:53:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:53:25.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:26 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:53:26.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:26 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:53:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:53:26.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:53:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:53:26.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:53:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:53:26.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:53:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:53:26.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:53:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:53:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:53:26.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:53:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:53:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:53:26.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:53:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:53:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:53:26.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:53:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:53:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:53:26.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:53:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:53:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:53:26.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:53:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:53:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:53:26.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:53:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:53:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:53:26.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:53:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:53:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:53:26.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:53:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:53:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:53:26.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:53:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:53:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:53:26.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:53:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:53:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:53:26.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:53:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:53:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:53:26.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:53:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:53:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1235] Local address: 172.21.15.161, port 33668 2022-07-03T21:53:26.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1235] Peer address: 172.21.15.101, port 22 2022-07-03T21:53:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1235] Connection lost 2022-07-03T21:53:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1235] Aborting connection 2022-07-03T21:53:26.646 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:26 smithi161 ceph-mon[91579]: pgmap v20228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:26 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:53:26.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:26 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:53:26.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:53:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:53:26.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:53:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:53:26.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:53:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:53:26.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:53:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:53:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:53:26.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:53:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:53:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:53:26.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:53:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:53:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:53:26.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:53:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:53:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:53:26.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:53:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:53:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:53:26.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:53:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:53:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:53:26.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:53:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:53:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:53:26.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:53:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:53:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:53:26.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:53:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:53:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:53:26.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:53:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:53:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:53:26.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:53:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:53:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:53:26.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:53:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:53:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:53:26.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:53:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:53:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1235] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:53:26.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1235] Local address: 172.21.15.161, port 33668 2022-07-03T21:53:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1235] Peer address: 172.21.15.101, port 22 2022-07-03T21:53:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1235] Connection lost 2022-07-03T21:53:26.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1235] Aborting connection 2022-07-03T21:53:26.681 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:26 smithi188 ceph-mon[92425]: pgmap v20228: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: pgmap v20229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.631 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.632 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:28 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: pgmap v20229: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' 2022-07-03T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.665 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.666 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:28.667 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:28 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi161", "name": "osd_memory_target"}]: dispatch 2022-07-03T21:53:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:29 smithi161 ceph-mon[91579]: Adjusting osd_memory_target on smithi188 to 17955k 2022-07-03T21:53:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:29 smithi161 ceph-mon[91579]: Unable to set osd_memory_target on smithi188 to 18386624: error parsing value: Value '18386624' is below minimum 939524096 2022-07-03T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:29 smithi188 ceph-mon[92425]: Adjusting osd_memory_target on smithi188 to 17955k 2022-07-03T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:29 smithi188 ceph-mon[92425]: Unable to set osd_memory_target on smithi188 to 18386624: error parsing value: Value '18386624' is below minimum 939524096 2022-07-03T21:53:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:30 smithi161 ceph-mon[91579]: pgmap v20230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:30 smithi188 ceph-mon[92425]: pgmap v20230: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:32.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:31 smithi161 ceph-mon[91579]: pgmap v20231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:31 smithi188 ceph-mon[92425]: pgmap v20231: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:34 smithi161 ceph-mon[91579]: pgmap v20232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:34 smithi188 ceph-mon[92425]: pgmap v20232: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:36 smithi161 ceph-mon[91579]: pgmap v20233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:36 smithi188 ceph-mon[92425]: pgmap v20233: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:38 smithi161 ceph-mon[91579]: pgmap v20234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:38 smithi188 ceph-mon[92425]: pgmap v20234: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:40 smithi161 ceph-mon[91579]: pgmap v20235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:40 smithi188 ceph-mon[92425]: pgmap v20235: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:40.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:42.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:41 smithi161 ceph-mon[91579]: pgmap v20236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:41 smithi188 ceph-mon[92425]: pgmap v20236: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:44 smithi161 ceph-mon[91579]: pgmap v20237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:44 smithi188 ceph-mon[92425]: pgmap v20237: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:46 smithi161 ceph-mon[91579]: pgmap v20238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:46 smithi188 ceph-mon[92425]: pgmap v20238: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:48 smithi161 ceph-mon[91579]: pgmap v20239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:48.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:48 smithi188 ceph-mon[92425]: pgmap v20239: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:50 smithi161 ceph-mon[91579]: pgmap v20240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:50 smithi188 ceph-mon[92425]: pgmap v20240: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:52.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:51 smithi161 ceph-mon[91579]: pgmap v20241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:51 smithi188 ceph-mon[92425]: pgmap v20241: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:54 smithi161 ceph-mon[91579]: pgmap v20242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:54 smithi188 ceph-mon[92425]: pgmap v20242: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:53:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:53:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:56 smithi161 ceph-mon[91579]: pgmap v20243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:56 smithi188 ceph-mon[92425]: pgmap v20243: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:53:58 smithi161 ceph-mon[91579]: pgmap v20244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:53:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:53:58 smithi188 ceph-mon[92425]: pgmap v20244: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: pgmap v20245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:54:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:54:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:54:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:54:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: pgmap v20245: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:54:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:54:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:54:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:54:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:02.127 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:01 smithi161 ceph-mon[91579]: pgmap v20246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:01 smithi188 ceph-mon[92425]: pgmap v20246: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:04 smithi161 ceph-mon[91579]: pgmap v20247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:04.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:04 smithi188 ceph-mon[92425]: pgmap v20247: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:06 smithi161 ceph-mon[91579]: pgmap v20248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:06 smithi188 ceph-mon[92425]: pgmap v20248: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:08 smithi161 ceph-mon[91579]: pgmap v20249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:08 smithi188 ceph-mon[92425]: pgmap v20249: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:10 smithi161 ceph-mon[91579]: pgmap v20250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:10 smithi188 ceph-mon[92425]: pgmap v20250: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:12.128 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:11 smithi161 ceph-mon[91579]: pgmap v20251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:11 smithi188 ceph-mon[92425]: pgmap v20251: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:14 smithi161 ceph-mon[91579]: pgmap v20252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:14.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:14 smithi188 ceph-mon[92425]: pgmap v20252: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:16 smithi161 ceph-mon[91579]: pgmap v20253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:16.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:16 smithi188 ceph-mon[92425]: pgmap v20253: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:18 smithi161 ceph-mon[91579]: pgmap v20254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:18.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:18 smithi188 ceph-mon[92425]: pgmap v20254: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:20 smithi161 ceph-mon[91579]: pgmap v20255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:20 smithi188 ceph-mon[92425]: pgmap v20255: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:21 smithi188 ceph-mon[92425]: pgmap v20256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:21 smithi161 ceph-mon[91579]: pgmap v20256: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:24 smithi161 ceph-mon[91579]: pgmap v20257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:24.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:24 smithi188 ceph-mon[92425]: pgmap v20257: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:26 smithi161 ceph-mon[91579]: pgmap v20258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:26 smithi188 ceph-mon[92425]: pgmap v20258: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:28.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:28 smithi161 ceph-mon[91579]: pgmap v20259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:54:28.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:54:28.379 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:28 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:54:28.379 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:54:27.953+0000 7f727186b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:54:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:54:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:54:28.380 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:54:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:54:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:54:28.381 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:54:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:54:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:54:28.382 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:54:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:54:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:54:28.383 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:54:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:54:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:54:28.384 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:54:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:54:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:54:28.385 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:54:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:54:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:54:28.386 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:54:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:54:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:54:28.387 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:54:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:54:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:54:28.388 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:54:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:54:28.389 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:54:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:54:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:54:28.390 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:54:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:54:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:54:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:54:28.391 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:54:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:54:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:54:28.392 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:54:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:54:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:54:28.393 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:54:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:54:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:54:28.394 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:54:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1237] Local address: 172.21.15.161, port 33700 2022-07-03T21:54:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1237] Peer address: 172.21.15.101, port 22 2022-07-03T21:54:28.395 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1237] Connection lost 2022-07-03T21:54:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1237] Aborting connection 2022-07-03T21:54:28.396 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:54:27 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:54:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:28 smithi188 ceph-mon[92425]: pgmap v20259: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:54:28.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:54:28.413 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:28 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:54:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:29 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:54:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:29 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:54:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:54:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:54:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:54:29.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:54:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:54:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:54:29.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:54:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:54:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:54:29.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:54:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:54:29.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:54:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:54:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:54:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:54:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:54:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:54:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:54:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:54:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:54:29.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:54:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:54:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:54:29.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:54:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:54:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:54:29.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:54:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:54:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:54:29.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:54:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:54:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:54:29.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:54:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:54:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:54:29.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:54:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:54:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:54:29.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:54:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:54:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:54:29.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:54:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:54:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:54:29.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:54:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1237] Local address: 172.21.15.161, port 33700 2022-07-03T21:54:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1237] Peer address: 172.21.15.101, port 22 2022-07-03T21:54:29.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1237] Connection lost 2022-07-03T21:54:29.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1237] Aborting connection 2022-07-03T21:54:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:29 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:29 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:54:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:54:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:54:29.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:54:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:54:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:54:29.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:54:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:54:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:54:29.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:54:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:54:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:54:29.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:54:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:54:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:54:29.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:54:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:54:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:54:29.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:54:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:54:29.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:54:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:54:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:54:29.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:54:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:54:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:54:29.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:54:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:54:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:54:29.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:54:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:54:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:54:29.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:54:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:54:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:54:29.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:54:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:54:29.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:54:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:54:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:54:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:54:29.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:54:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:54:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:54:29.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1237] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:54:29.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1237] Local address: 172.21.15.161, port 33700 2022-07-03T21:54:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1237] Peer address: 172.21.15.101, port 22 2022-07-03T21:54:29.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1237] Connection lost 2022-07-03T21:54:29.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1237] Aborting connection 2022-07-03T21:54:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:30 smithi161 ceph-mon[91579]: pgmap v20260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:30 smithi188 ceph-mon[92425]: pgmap v20260: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:31 smithi188 ceph-mon[92425]: pgmap v20261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:31 smithi161 ceph-mon[91579]: pgmap v20261: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:34 smithi161 ceph-mon[91579]: pgmap v20262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:34 smithi188 ceph-mon[92425]: pgmap v20262: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:36 smithi161 ceph-mon[91579]: pgmap v20263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:36 smithi188 ceph-mon[92425]: pgmap v20263: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:38 smithi161 ceph-mon[91579]: pgmap v20264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:38 smithi188 ceph-mon[92425]: pgmap v20264: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:40 smithi161 ceph-mon[91579]: pgmap v20265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:40 smithi188 ceph-mon[92425]: pgmap v20265: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:41 smithi188 ceph-mon[92425]: pgmap v20266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:41 smithi161 ceph-mon[91579]: pgmap v20266: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:44 smithi161 ceph-mon[91579]: pgmap v20267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:44 smithi188 ceph-mon[92425]: pgmap v20267: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:46 smithi161 ceph-mon[91579]: pgmap v20268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:46 smithi188 ceph-mon[92425]: pgmap v20268: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:48 smithi161 ceph-mon[91579]: pgmap v20269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:48 smithi188 ceph-mon[92425]: pgmap v20269: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:50 smithi161 ceph-mon[91579]: pgmap v20270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:50 smithi188 ceph-mon[92425]: pgmap v20270: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:50.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:51 smithi188 ceph-mon[92425]: pgmap v20271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:51 smithi161 ceph-mon[91579]: pgmap v20271: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:54 smithi161 ceph-mon[91579]: pgmap v20272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:54.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:54 smithi188 ceph-mon[92425]: pgmap v20272: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:54:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:55.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:54:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:56 smithi161 ceph-mon[91579]: pgmap v20273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:56 smithi188 ceph-mon[92425]: pgmap v20273: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:54:58 smithi161 ceph-mon[91579]: pgmap v20274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:54:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:54:58 smithi188 ceph-mon[92425]: pgmap v20274: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: pgmap v20275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:55:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:55:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:55:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:55:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: pgmap v20275: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:55:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:55:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:55:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:55:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:02.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:01 smithi188 ceph-mon[92425]: pgmap v20276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:01 smithi161 ceph-mon[91579]: pgmap v20276: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:04 smithi161 ceph-mon[91579]: pgmap v20277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:04.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:04 smithi188 ceph-mon[92425]: pgmap v20277: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:06 smithi161 ceph-mon[91579]: pgmap v20278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:06 smithi188 ceph-mon[92425]: pgmap v20278: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:08 smithi161 ceph-mon[91579]: pgmap v20279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:08 smithi188 ceph-mon[92425]: pgmap v20279: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:10 smithi161 ceph-mon[91579]: pgmap v20280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:10 smithi188 ceph-mon[92425]: pgmap v20280: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:12.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:11 smithi188 ceph-mon[92425]: pgmap v20281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:11 smithi161 ceph-mon[91579]: pgmap v20281: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:14.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:14 smithi161 ceph-mon[91579]: pgmap v20282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:14.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:14 smithi188 ceph-mon[92425]: pgmap v20282: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:15.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:16 smithi161 ceph-mon[91579]: pgmap v20283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:16 smithi188 ceph-mon[92425]: pgmap v20283: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:18 smithi161 ceph-mon[91579]: pgmap v20284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:18 smithi188 ceph-mon[92425]: pgmap v20284: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:20 smithi161 ceph-mon[91579]: pgmap v20285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:20 smithi188 ceph-mon[92425]: pgmap v20285: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:22.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:21 smithi188 ceph-mon[92425]: pgmap v20286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:22.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:21 smithi161 ceph-mon[91579]: pgmap v20286: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:24 smithi161 ceph-mon[91579]: pgmap v20287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:24 smithi188 ceph-mon[92425]: pgmap v20287: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:26 smithi161 ceph-mon[91579]: pgmap v20288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:26 smithi188 ceph-mon[92425]: pgmap v20288: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:28.466 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-mon[91579]: pgmap v20289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:28.467 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:55:28.457+0000 7f727386f700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:55:28.468 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:55:28.468 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:55:28.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:55:28.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:55:28.469 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:55:28.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:55:28.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:55:28.470 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:55:28.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:55:28.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:55:28.471 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:55:28.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:55:28.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:55:28.472 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:55:28.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:55:28.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:55:28.473 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:55:28.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:55:28.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:55:28.474 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:55:28.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:55:28.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:55:28.475 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:55:28.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:55:28.476 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:55:28.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:55:28.477 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:55:28.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:55:28.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:55:28.478 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:55:28.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:55:28.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:55:28.479 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:55:28.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:55:28.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:55:28.480 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:55:28.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:55:28.481 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:55:28.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:55:28.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:55:28.482 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:55:28.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:55:28.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:55:28.483 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:55:28.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:55:28.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1239] Local address: 172.21.15.161, port 33704 2022-07-03T21:55:28.484 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1239] Peer address: 172.21.15.101, port 22 2022-07-03T21:55:28.485 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1239] Connection lost 2022-07-03T21:55:28.485 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1239] Aborting connection 2022-07-03T21:55:28.486 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:55:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:55:28.486 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:28 smithi188 ceph-mon[92425]: pgmap v20289: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:29.633 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:55:29.634 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:55:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:55:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:55:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:55:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:55:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:55:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:55:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:55:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:55:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:55:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:55:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:55:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:55:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:55:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:55:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:55:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:55:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:55:30.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:55:30.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:55:30.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:55:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:55:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:55:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:55:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:55:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:55:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:55:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:55:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:55:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:55:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:55:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:55:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:55:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:55:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:55:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:55:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:55:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:55:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:55:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:55:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:55:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:55:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:55:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:55:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:55:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:55:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:55:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:55:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:55:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:55:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:55:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1239] Local address: 172.21.15.161, port 33704 2022-07-03T21:55:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1239] Peer address: 172.21.15.101, port 22 2022-07-03T21:55:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1239] Connection lost 2022-07-03T21:55:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1239] Aborting connection 2022-07-03T21:55:30.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:30 smithi161 ceph-mon[91579]: pgmap v20290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:30.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:55:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:55:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:55:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:55:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:55:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:55:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:55:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:55:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:55:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:55:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:55:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:55:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:55:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:55:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:55:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:55:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:55:30.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:55:30.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:55:30.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:55:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:55:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:55:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:55:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:55:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:55:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:55:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:55:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:55:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:55:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:55:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:55:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:55:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:55:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:55:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:55:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:55:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:55:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:55:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:55:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:55:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:55:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:55:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:55:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:55:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:55:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:55:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1239] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:55:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1239] Local address: 172.21.15.161, port 33704 2022-07-03T21:55:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1239] Peer address: 172.21.15.101, port 22 2022-07-03T21:55:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1239] Connection lost 2022-07-03T21:55:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1239] Aborting connection 2022-07-03T21:55:30.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:30 smithi188 ceph-mon[92425]: pgmap v20290: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:30.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:32.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:31 smithi188 ceph-mon[92425]: pgmap v20291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:31 smithi161 ceph-mon[91579]: pgmap v20291: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:34.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:34 smithi161 ceph-mon[91579]: pgmap v20292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:34 smithi188 ceph-mon[92425]: pgmap v20292: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:36 smithi161 ceph-mon[91579]: pgmap v20293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:36 smithi188 ceph-mon[92425]: pgmap v20293: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:38 smithi161 ceph-mon[91579]: pgmap v20294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:38.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:38 smithi188 ceph-mon[92425]: pgmap v20294: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:40 smithi161 ceph-mon[91579]: pgmap v20295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:40 smithi188 ceph-mon[92425]: pgmap v20295: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:42.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:41 smithi188 ceph-mon[92425]: pgmap v20296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:41 smithi161 ceph-mon[91579]: pgmap v20296: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:44 smithi161 ceph-mon[91579]: pgmap v20297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:44 smithi188 ceph-mon[92425]: pgmap v20297: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:46 smithi161 ceph-mon[91579]: pgmap v20298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:46 smithi188 ceph-mon[92425]: pgmap v20298: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:48 smithi161 ceph-mon[91579]: pgmap v20299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:48 smithi188 ceph-mon[92425]: pgmap v20299: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:50 smithi161 ceph-mon[91579]: pgmap v20300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:50 smithi188 ceph-mon[92425]: pgmap v20300: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:51 smithi188 ceph-mon[92425]: pgmap v20301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:51 smithi161 ceph-mon[91579]: pgmap v20301: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:54 smithi161 ceph-mon[91579]: pgmap v20302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:54 smithi188 ceph-mon[92425]: pgmap v20302: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:55:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:55:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:56 smithi161 ceph-mon[91579]: pgmap v20303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:56 smithi188 ceph-mon[92425]: pgmap v20303: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:55:58 smithi161 ceph-mon[91579]: pgmap v20304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:55:58.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:55:58 smithi188 ceph-mon[92425]: pgmap v20304: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: pgmap v20305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:56:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:56:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:56:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:56:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: pgmap v20305: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:56:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:56:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:56:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:56:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:02.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:01 smithi188 ceph-mon[92425]: pgmap v20306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:01 smithi161 ceph-mon[91579]: pgmap v20306: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:04 smithi161 ceph-mon[91579]: pgmap v20307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:04.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:04 smithi188 ceph-mon[92425]: pgmap v20307: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:06 smithi161 ceph-mon[91579]: pgmap v20308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:06.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:06 smithi188 ceph-mon[92425]: pgmap v20308: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:08 smithi161 ceph-mon[91579]: pgmap v20309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:08 smithi188 ceph-mon[92425]: pgmap v20309: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:10 smithi161 ceph-mon[91579]: pgmap v20310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:10 smithi188 ceph-mon[92425]: pgmap v20310: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:12.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:11 smithi188 ceph-mon[92425]: pgmap v20311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:11 smithi161 ceph-mon[91579]: pgmap v20311: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:14 smithi161 ceph-mon[91579]: pgmap v20312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:14.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:14 smithi188 ceph-mon[92425]: pgmap v20312: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:16 smithi161 ceph-mon[91579]: pgmap v20313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:16 smithi188 ceph-mon[92425]: pgmap v20313: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:18 smithi161 ceph-mon[91579]: pgmap v20314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:18 smithi188 ceph-mon[92425]: pgmap v20314: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:20 smithi161 ceph-mon[91579]: pgmap v20315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:20 smithi188 ceph-mon[92425]: pgmap v20315: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:22.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:21 smithi188 ceph-mon[92425]: pgmap v20316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:21 smithi161 ceph-mon[91579]: pgmap v20316: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:24 smithi161 ceph-mon[91579]: pgmap v20317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:24 smithi188 ceph-mon[92425]: pgmap v20317: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:25.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:26 smithi161 ceph-mon[91579]: pgmap v20318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:26 smithi188 ceph-mon[92425]: pgmap v20318: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-mon[91579]: pgmap v20319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:28 smithi188 ceph-mon[92425]: pgmap v20319: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:29.230 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:56:28.959+0000 7f727206c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:56:29.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:56:29.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:56:29.231 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:56:29.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:56:29.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:56:29.232 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:56:29.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:56:29.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:56:29.233 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:56:29.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:56:29.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:56:29.234 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:56:29.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:56:29.235 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:56:29.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:56:29.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:56:29.236 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:56:29.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:56:29.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:56:29.237 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:56:29.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:56:29.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:56:29.238 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:56:29.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:56:29.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:56:29.239 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:56:29.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:56:29.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:56:29.240 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:56:29.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:56:29.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:56:29.241 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:56:29.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:56:29.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:56:29.242 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:56:29.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:56:29.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:56:29.243 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:56:29.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:56:29.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:56:29.244 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:56:29.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:56:29.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:56:29.245 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:56:29.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:56:29.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1241] Local address: 172.21.15.161, port 33708 2022-07-03T21:56:29.246 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1241] Peer address: 172.21.15.101, port 22 2022-07-03T21:56:29.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1241] Connection lost 2022-07-03T21:56:29.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1241] Aborting connection 2022-07-03T21:56:29.247 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:56:28 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:56:29.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:56:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:56:29.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:29 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:56:29.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:56:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:56:29.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:29 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:56:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:30 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:56:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:30 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:56:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:56:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:56:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:56:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:56:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:56:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:56:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:56:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:56:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:56:30.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:56:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:56:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:56:30.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:56:30.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:56:30.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:56:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:56:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:56:30.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:56:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:56:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:56:30.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:56:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:56:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:56:30.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:56:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:56:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:56:30.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:56:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:56:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:56:30.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:56:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:56:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:56:30.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:56:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:56:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:56:30.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:56:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:56:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:56:30.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:56:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:56:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:56:30.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:56:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:56:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:56:30.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:56:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1241] Local address: 172.21.15.161, port 33708 2022-07-03T21:56:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1241] Peer address: 172.21.15.101, port 22 2022-07-03T21:56:30.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1241] Connection lost 2022-07-03T21:56:30.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1241] Aborting connection 2022-07-03T21:56:30.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:30 smithi161 ceph-mon[91579]: pgmap v20320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:30.645 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:30 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:56:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:30 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:56:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:56:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:56:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:56:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:56:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:56:30.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:56:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:56:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:56:30.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:56:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:56:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:56:30.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:56:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:56:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:56:30.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:56:30.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:56:30.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:56:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:56:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:56:30.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:56:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:56:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:56:30.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:56:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:56:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:56:30.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:56:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:56:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:56:30.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:56:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:56:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:56:30.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:56:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:56:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:56:30.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:56:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:56:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:56:30.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:56:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:56:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:56:30.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:56:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:56:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:56:30.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:56:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1241] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:56:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1241] Local address: 172.21.15.161, port 33708 2022-07-03T21:56:30.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1241] Peer address: 172.21.15.101, port 22 2022-07-03T21:56:30.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1241] Connection lost 2022-07-03T21:56:30.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1241] Aborting connection 2022-07-03T21:56:30.679 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:30 smithi188 ceph-mon[92425]: pgmap v20320: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:30.680 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:32.161 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:31 smithi188 ceph-mon[92425]: pgmap v20321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:32.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:31 smithi161 ceph-mon[91579]: pgmap v20321: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:34 smithi161 ceph-mon[91579]: pgmap v20322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:34 smithi188 ceph-mon[92425]: pgmap v20322: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:36 smithi161 ceph-mon[91579]: pgmap v20323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:36 smithi188 ceph-mon[92425]: pgmap v20323: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:38 smithi161 ceph-mon[91579]: pgmap v20324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:38 smithi188 ceph-mon[92425]: pgmap v20324: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:40 smithi161 ceph-mon[91579]: pgmap v20325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:40 smithi188 ceph-mon[92425]: pgmap v20325: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:42.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:41 smithi188 ceph-mon[92425]: pgmap v20326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:41 smithi161 ceph-mon[91579]: pgmap v20326: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:44 smithi161 ceph-mon[91579]: pgmap v20327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:44.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:44 smithi188 ceph-mon[92425]: pgmap v20327: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:45.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:46 smithi161 ceph-mon[91579]: pgmap v20328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:46 smithi188 ceph-mon[92425]: pgmap v20328: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:48 smithi161 ceph-mon[91579]: pgmap v20329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:48 smithi188 ceph-mon[92425]: pgmap v20329: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:50 smithi161 ceph-mon[91579]: pgmap v20330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:50 smithi188 ceph-mon[92425]: pgmap v20330: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:52.162 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:51 smithi188 ceph-mon[92425]: pgmap v20331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:51 smithi161 ceph-mon[91579]: pgmap v20331: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:54 smithi161 ceph-mon[91579]: pgmap v20332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:54.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:54 smithi188 ceph-mon[92425]: pgmap v20332: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:56:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:56:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:56 smithi161 ceph-mon[91579]: pgmap v20333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:56.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:56 smithi188 ceph-mon[92425]: pgmap v20333: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:56:58 smithi161 ceph-mon[91579]: pgmap v20334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:56:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:56:58 smithi188 ceph-mon[92425]: pgmap v20334: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: pgmap v20335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:57:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:57:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:57:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:57:00.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: pgmap v20335: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:57:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:57:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:57:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:57:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:02.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:01 smithi161 ceph-mon[91579]: pgmap v20336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:01 smithi188 ceph-mon[92425]: pgmap v20336: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:04 smithi161 ceph-mon[91579]: pgmap v20337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:04.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:04 smithi188 ceph-mon[92425]: pgmap v20337: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:05.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:06 smithi161 ceph-mon[91579]: pgmap v20338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:06 smithi188 ceph-mon[92425]: pgmap v20338: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:08 smithi161 ceph-mon[91579]: pgmap v20339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:08.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:08 smithi188 ceph-mon[92425]: pgmap v20339: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:10.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:10 smithi161 ceph-mon[91579]: pgmap v20340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:10.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:10 smithi188 ceph-mon[92425]: pgmap v20340: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:11 smithi161 ceph-mon[91579]: pgmap v20341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:12.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:11 smithi188 ceph-mon[92425]: pgmap v20341: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:14 smithi161 ceph-mon[91579]: pgmap v20342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:14.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:14 smithi188 ceph-mon[92425]: pgmap v20342: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:15.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:16 smithi161 ceph-mon[91579]: pgmap v20343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:16 smithi188 ceph-mon[92425]: pgmap v20343: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:18 smithi161 ceph-mon[91579]: pgmap v20344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:18 smithi188 ceph-mon[92425]: pgmap v20344: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:20 smithi161 ceph-mon[91579]: pgmap v20345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:20 smithi188 ceph-mon[92425]: pgmap v20345: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:21 smithi161 ceph-mon[91579]: pgmap v20346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:21 smithi188 ceph-mon[92425]: pgmap v20346: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:24 smithi161 ceph-mon[91579]: pgmap v20347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:24 smithi188 ceph-mon[92425]: pgmap v20347: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:26 smithi161 ceph-mon[91579]: pgmap v20348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:26 smithi188 ceph-mon[92425]: pgmap v20348: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:28.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:28 smithi161 ceph-mon[91579]: pgmap v20349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:28 smithi188 ceph-mon[92425]: pgmap v20349: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:29.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:57:29.452+0000 7f727903a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:57:29.878 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:57:29.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:57:29.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:57:29.879 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:57:29.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:57:29.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:57:29.880 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:57:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:57:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:57:29.881 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:57:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:57:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:57:29.882 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:57:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:57:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:57:29.883 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:57:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:57:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:57:29.884 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:57:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:57:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:57:29.885 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:57:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:57:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:57:29.886 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:57:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:57:29.887 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:57:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:57:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:57:29.888 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:57:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:57:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:57:29.889 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:57:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:57:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:57:29.890 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:57:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:57:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:57:29.891 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:57:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:57:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:57:29.892 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:57:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:57:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:57:29.893 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1243] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:57:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1243] Local address: 172.21.15.161, port 33712 2022-07-03T21:57:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1243] Peer address: 172.21.15.101, port 22 2022-07-03T21:57:29.894 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1243] Connection lost 2022-07-03T21:57:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1243] Aborting connection 2022-07-03T21:57:29.895 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:57:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:57:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:30 smithi161 ceph-mon[91579]: pgmap v20350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:57:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:57:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:57:30.630 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:30 smithi188 ceph-mon[92425]: pgmap v20350: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:57:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:57:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:57:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:57:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:57:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:57:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:57:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:57:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:57:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:57:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:57:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:57:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:57:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:57:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:57:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:57:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:57:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:57:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:57:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:57:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:57:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:57:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:57:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:57:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:57:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:57:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:57:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:57:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:57:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:57:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:57:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:57:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:57:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:57:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:57:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:57:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:57:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:57:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:57:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:57:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:57:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:57:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:57:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:57:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:57:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:57:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:57:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1243] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:57:31.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1243] Local address: 172.21.15.161, port 33712 2022-07-03T21:57:31.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1243] Peer address: 172.21.15.101, port 22 2022-07-03T21:57:31.645 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1243] Connection lost 2022-07-03T21:57:31.646 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1243] Aborting connection 2022-07-03T21:57:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:57:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:57:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:57:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:57:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:57:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:57:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:57:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:57:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:57:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:57:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:57:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:57:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:57:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:57:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:57:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:57:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:57:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:57:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:57:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:57:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:57:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:57:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:57:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:57:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:57:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:57:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:57:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:57:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:57:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:57:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:57:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:57:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:57:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:57:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:57:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:57:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:57:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:57:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:57:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:57:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:57:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:57:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:57:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:57:31.679 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:57:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:57:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1243] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:57:31.680 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1243] Local address: 172.21.15.161, port 33712 2022-07-03T21:57:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1243] Peer address: 172.21.15.101, port 22 2022-07-03T21:57:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1243] Connection lost 2022-07-03T21:57:31.681 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1243] Aborting connection 2022-07-03T21:57:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:32 smithi161 ceph-mon[91579]: pgmap v20351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:32 smithi188 ceph-mon[92425]: pgmap v20351: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:34 smithi161 ceph-mon[91579]: pgmap v20352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:34.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:34 smithi188 ceph-mon[92425]: pgmap v20352: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:35.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:36 smithi161 ceph-mon[91579]: pgmap v20353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:36.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:36 smithi188 ceph-mon[92425]: pgmap v20353: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:38 smithi161 ceph-mon[91579]: pgmap v20354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:38 smithi188 ceph-mon[92425]: pgmap v20354: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:40 smithi161 ceph-mon[91579]: pgmap v20355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:40 smithi188 ceph-mon[92425]: pgmap v20355: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:41 smithi161 ceph-mon[91579]: pgmap v20356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:41 smithi188 ceph-mon[92425]: pgmap v20356: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:44 smithi161 ceph-mon[91579]: pgmap v20357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:44.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:44 smithi188 ceph-mon[92425]: pgmap v20357: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:46 smithi161 ceph-mon[91579]: pgmap v20358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:46.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:46 smithi188 ceph-mon[92425]: pgmap v20358: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:48 smithi161 ceph-mon[91579]: pgmap v20359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:48 smithi188 ceph-mon[92425]: pgmap v20359: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:50 smithi161 ceph-mon[91579]: pgmap v20360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:50 smithi188 ceph-mon[92425]: pgmap v20360: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:51 smithi161 ceph-mon[91579]: pgmap v20361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:51 smithi188 ceph-mon[92425]: pgmap v20361: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:54 smithi161 ceph-mon[91579]: pgmap v20362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:54.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:54 smithi188 ceph-mon[92425]: pgmap v20362: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:57:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:57:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:56 smithi161 ceph-mon[91579]: pgmap v20363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:56 smithi188 ceph-mon[92425]: pgmap v20363: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:58.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:57:58 smithi161 ceph-mon[91579]: pgmap v20364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:57:58.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:57:58 smithi188 ceph-mon[92425]: pgmap v20364: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:00.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: pgmap v20365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:58:00.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:58:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:58:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:58:00.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:00 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: pgmap v20365: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:00.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:58:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:58:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-07-03T21:58:00.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-07-03T21:58:00.664 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:00 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:02.378 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:01 smithi161 ceph-mon[91579]: pgmap v20366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:02.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:01 smithi188 ceph-mon[92425]: pgmap v20366: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:04.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:04 smithi161 ceph-mon[91579]: pgmap v20367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:04.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:04 smithi188 ceph-mon[92425]: pgmap v20367: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:05.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:05 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:05.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:05 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:06.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:06 smithi161 ceph-mon[91579]: pgmap v20368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:06.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:06 smithi188 ceph-mon[92425]: pgmap v20368: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:08.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:08 smithi161 ceph-mon[91579]: pgmap v20369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:08.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:08 smithi188 ceph-mon[92425]: pgmap v20369: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:10.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:10 smithi161 ceph-mon[91579]: pgmap v20370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:10.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:10 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:10 smithi188 ceph-mon[92425]: pgmap v20370: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:10.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:10 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:12.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:11 smithi161 ceph-mon[91579]: pgmap v20371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:12.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:11 smithi188 ceph-mon[92425]: pgmap v20371: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:14.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:14 smithi161 ceph-mon[91579]: pgmap v20372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:14.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:14 smithi188 ceph-mon[92425]: pgmap v20372: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:15.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:15 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:15.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:15 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:16.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:16 smithi161 ceph-mon[91579]: pgmap v20373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:16.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:16 smithi188 ceph-mon[92425]: pgmap v20373: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:18.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:18 smithi161 ceph-mon[91579]: pgmap v20374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:18.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:18 smithi188 ceph-mon[92425]: pgmap v20374: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:20.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:20 smithi161 ceph-mon[91579]: pgmap v20375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:20.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:20 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:20.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:20 smithi188 ceph-mon[92425]: pgmap v20375: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:20.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:20 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:22.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:21 smithi161 ceph-mon[91579]: pgmap v20376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:22.412 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:21 smithi188 ceph-mon[92425]: pgmap v20376: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:24.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:24 smithi161 ceph-mon[91579]: pgmap v20377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:24.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:24 smithi188 ceph-mon[92425]: pgmap v20377: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:25.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:25 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:25.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:25 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:26.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:26 smithi161 ceph-mon[91579]: pgmap v20378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:26.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:26 smithi188 ceph-mon[92425]: pgmap v20378: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:28.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:28 smithi161 ceph-mon[91579]: pgmap v20379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:28.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:28 smithi188 ceph-mon[92425]: pgmap v20379: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:30.128 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:58:29.850+0000 7f727306e700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:58:30.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:58:30.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:58:30.129 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: yield 2022-07-03T21:58:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:58:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:58:30.130 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:58:30.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 'Opening SSH connection to') 2022-07-03T21:58:30.131 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:58:30.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await conn.wait_established() 2022-07-03T21:58:30.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:58:30.132 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: await self._waiter 2022-07-03T21:58:30.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:58:30.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:58:30.133 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: During handling of the above exception, another exception occurred: 2022-07-03T21:58:30.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:58:30.134 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Traceback (most recent call last): 2022-07-03T21:58:30.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:58:30.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return f(*arg) 2022-07-03T21:58:30.135 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:58:30.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self._write_client_files(client_files, host) 2022-07-03T21:58:30.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:58:30.136 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:58:30.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:58:30.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:58:30.137 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:58:30.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.event_loop.get_result(coro) 2022-07-03T21:58:30.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:58:30.138 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:58:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:58:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: return self.__get_result() 2022-07-03T21:58:30.139 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:58:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise self._exception 2022-07-03T21:58:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:58:30.140 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:58:30.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:58:30.141 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: conn = await self._remote_connection(host, addr) 2022-07-03T21:58:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:58:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise 2022-07-03T21:58:30.142 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:58:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: self.gen.throw(type, value, traceback) 2022-07-03T21:58:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:58:30.143 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: raise OrchestratorError(msg) 2022-07-03T21:58:30.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:58:30.144 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:58:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:58:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1245] Local address: 172.21.15.161, port 33716 2022-07-03T21:58:30.145 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1245] Peer address: 172.21.15.101, port 22 2022-07-03T21:58:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1245] Connection lost 2022-07-03T21:58:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: [conn=1245] Aborting connection 2022-07-03T21:58:30.146 INFO:journalctl@ceph.mgr.y.smithi161.stdout:Jul 03 21:58:29 smithi161 ceph-23a851ae-fab9-11ec-842c-001a4aab830c-mgr-y[92871]: 2022-07-03T21:58:30.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:30 smithi161 ceph-mon[91579]: pgmap v20380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:58:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:58:30.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:58:30.629 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:30 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:30.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:30 smithi188 ceph-mon[92425]: pgmap v20380: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-07-03T21:58:30.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-07-03T21:58:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-07-03T21:58:30.663 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:30 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:31.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:31 smithi161 ceph-mon[91579]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:58:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:31 smithi161 ceph-mon[91579]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:58:31.628 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:58:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:58:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: yield 2022-07-03T21:58:31.629 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:58:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:58:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:58:31.630 INFO:journalctl@ceph.mon.b.smithi161.stdout: 'Opening SSH connection to') 2022-07-03T21:58:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:58:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: await conn.wait_established() 2022-07-03T21:58:31.631 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:58:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: await self._waiter 2022-07-03T21:58:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:58:31.632 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:58:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:58:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: 2022-07-03T21:58:31.633 INFO:journalctl@ceph.mon.b.smithi161.stdout: Traceback (most recent call last): 2022-07-03T21:58:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:58:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: return f(*arg) 2022-07-03T21:58:31.634 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:58:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self._write_client_files(client_files, host) 2022-07-03T21:58:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:58:31.635 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:58:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:58:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:58:31.636 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:58:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:58:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:58:31.637 INFO:journalctl@ceph.mon.b.smithi161.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:58:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:58:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: return self.__get_result() 2022-07-03T21:58:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:58:31.638 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise self._exception 2022-07-03T21:58:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:58:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:58:31.639 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:58:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:58:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:58:31.640 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise 2022-07-03T21:58:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:58:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:58:31.641 INFO:journalctl@ceph.mon.b.smithi161.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:58:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: raise OrchestratorError(msg) 2022-07-03T21:58:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:58:31.642 INFO:journalctl@ceph.mon.b.smithi161.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:58:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:58:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1245] Local address: 172.21.15.161, port 33716 2022-07-03T21:58:31.643 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1245] Peer address: 172.21.15.101, port 22 2022-07-03T21:58:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1245] Connection lost 2022-07-03T21:58:31.644 INFO:journalctl@ceph.mon.b.smithi161.stdout: [conn=1245] Aborting connection 2022-07-03T21:58:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:31 smithi188 ceph-mon[92425]: Removing smithi101:/etc/ceph/ceph.conf 2022-07-03T21:58:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:31 smithi188 ceph-mon[92425]: executing refresh((['smithi101', 'smithi161', 'smithi188'],)) failed. 2022-07-03T21:58:31.662 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:58:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-07-03T21:58:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: yield 2022-07-03T21:58:31.663 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-07-03T21:58:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-07-03T21:58:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-07-03T21:58:31.664 INFO:journalctl@ceph.mon.c.smithi188.stdout: 'Opening SSH connection to') 2022-07-03T21:58:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect 2022-07-03T21:58:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: await conn.wait_established() 2022-07-03T21:58:31.665 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established 2022-07-03T21:58:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: await self._waiter 2022-07-03T21:58:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: asyncssh.misc.ConnectionLost: Connection lost 2022-07-03T21:58:31.666 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:58:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: During handling of the above exception, another exception occurred: 2022-07-03T21:58:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: 2022-07-03T21:58:31.667 INFO:journalctl@ceph.mon.c.smithi188.stdout: Traceback (most recent call last): 2022-07-03T21:58:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-07-03T21:58:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: return f(*arg) 2022-07-03T21:58:31.668 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 270, in refresh 2022-07-03T21:58:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self._write_client_files(client_files, host) 2022-07-03T21:58:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1074, in _write_client_files 2022-07-03T21:58:31.669 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-07-03T21:58:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 196, in check_execute_command 2022-07-03T21:58:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-07-03T21:58:31.670 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 590, in wait_async 2022-07-03T21:58:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.event_loop.get_result(coro) 2022-07-03T21:58:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-07-03T21:58:31.671 INFO:journalctl@ceph.mon.c.smithi188.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-07-03T21:58:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-07-03T21:58:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: return self.__get_result() 2022-07-03T21:58:31.672 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-07-03T21:58:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise self._exception 2022-07-03T21:58:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 183, in _check_execute_command 2022-07-03T21:58:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-07-03T21:58:31.673 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-07-03T21:58:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: conn = await self._remote_connection(host, addr) 2022-07-03T21:58:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-07-03T21:58:31.674 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise 2022-07-03T21:58:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-07-03T21:58:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: self.gen.throw(type, value, traceback) 2022-07-03T21:58:31.675 INFO:journalctl@ceph.mon.c.smithi188.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 115, in redirect_log 2022-07-03T21:58:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: raise OrchestratorError(msg) 2022-07-03T21:58:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: orchestrator._interface.OrchestratorError: Failed to connect to smithi101 (172.21.15.101). Connection lost 2022-07-03T21:58:31.676 INFO:journalctl@ceph.mon.c.smithi188.stdout: Log: Opening SSH connection to 172.21.15.101, port 22 2022-07-03T21:58:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1245] Connected to SSH server at 172.21.15.101, port 22 2022-07-03T21:58:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1245] Local address: 172.21.15.161, port 33716 2022-07-03T21:58:31.677 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1245] Peer address: 172.21.15.101, port 22 2022-07-03T21:58:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1245] Connection lost 2022-07-03T21:58:31.678 INFO:journalctl@ceph.mon.c.smithi188.stdout: [conn=1245] Aborting connection 2022-07-03T21:58:32.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:32 smithi161 ceph-mon[91579]: pgmap v20381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:32.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:32 smithi188 ceph-mon[92425]: pgmap v20381: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:34.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:34 smithi161 ceph-mon[91579]: pgmap v20382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:34.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:34 smithi188 ceph-mon[92425]: pgmap v20382: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:35.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:35 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:35.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:35 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:36.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:36 smithi161 ceph-mon[91579]: pgmap v20383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:36.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:36 smithi188 ceph-mon[92425]: pgmap v20383: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:38.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:38 smithi161 ceph-mon[91579]: pgmap v20384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:38.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:38 smithi188 ceph-mon[92425]: pgmap v20384: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:40.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:40 smithi161 ceph-mon[91579]: pgmap v20385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:40.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:40 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:40.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:40 smithi188 ceph-mon[92425]: pgmap v20385: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:40.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:40 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:42.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:41 smithi161 ceph-mon[91579]: pgmap v20386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:42.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:41 smithi188 ceph-mon[92425]: pgmap v20386: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:44.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:44 smithi161 ceph-mon[91579]: pgmap v20387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:44.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:44 smithi188 ceph-mon[92425]: pgmap v20387: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:45.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:45 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:45.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:45 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:46.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:46 smithi161 ceph-mon[91579]: pgmap v20388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:46.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:46 smithi188 ceph-mon[92425]: pgmap v20388: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:48.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:48 smithi161 ceph-mon[91579]: pgmap v20389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:48.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:48 smithi188 ceph-mon[92425]: pgmap v20389: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:50.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:50 smithi161 ceph-mon[91579]: pgmap v20390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:50.628 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:50 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:50.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:50 smithi188 ceph-mon[92425]: pgmap v20390: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:50.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:50 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:52.377 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:51 smithi161 ceph-mon[91579]: pgmap v20391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:52.411 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:51 smithi188 ceph-mon[92425]: pgmap v20391: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:54.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:54 smithi161 ceph-mon[91579]: pgmap v20392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:54.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:54 smithi188 ceph-mon[92425]: pgmap v20392: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2022-07-03T21:58:55.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:55 smithi161 ceph-mon[91579]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:55.661 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:55 smithi188 ceph-mon[92425]: from='mgr.24112 172.21.15.161:0/3302867460' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-07-03T21:58:56.627 INFO:journalctl@ceph.mon.b.smithi161.stdout:Jul 03 21:58:56 smithi161 ceph-mon[91579]: pgmap v20393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:56.662 INFO:journalctl@ceph.mon.c.smithi188.stdout:Jul 03 21:58:56 smithi188 ceph-mon[92425]: pgmap v20393: 97 pgs: 96 active+clean, 1 unknown; 2.5 MiB data, 61 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1 op/s 2022-07-03T21:58:57.412 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-03T21:58:57.414 DEBUG:teuthology.task.console_log:Killing console logger for smithi101 2022-07-03T21:58:57.416 DEBUG:teuthology.task.console_log:Killing console logger for smithi161 2022-07-03T21:58:57.416 DEBUG:teuthology.task.console_log:Killing console logger for smithi188 2022-07-03T21:58:57.417 DEBUG:teuthology.exit:Finished running handlers